Lots of the comments here are pointing at details of the markets and whether it's possible to profit off of knowing that transformative AI is coming. Which is all fine and good, but I think there's a simple  way to look at it that's very illuminating.

The stock market is good at predicting company success because there are a lot of people trading in it who think hard about which companies will succeed, doing things like writing documents about  those companies' target markets, products, and leadership. Traders who do a good job at this sort of analysis get more funds to trade with, which makes their trading activity have a larger impact on the prices.

Now, when you say that:

the market is decisively rejecting – i.e., putting very low probability on – the development of transformative AI in the very near term, say within the next ten years.

I think what you're claiming is that market prices are substantially controlled by traders who have a probability like that in their heads. Or traders who are following an algorithm which had a probability like that in the spreadsheet. Or something thing like that. Some sort of serious cognition, serious in the way that traders treat company revenue forecasts.

And I think that this is false. I think their heads don't contain any probability for transformative AI at all. I think that if you could peer into the internal communications of trading firms, and you went looking for their thoughts about AI timelines affecting interest rates, you wouldn't find thoughts like that. And if you did find an occasional trader who had such thoughts, and quantified how much impact they would have on the prices if they went all-in on trading based on that theory, you would find their impact was infinitesimal.

Market prices aren't mystical, they're aggregations of traders' cognition. If the cognition isn't there, then the market price can't tell you anything. If the cognition is there but it doesn't control enough of the capital to move the price, then the price can't tell you anything.

I think this post is a trap for people who think of market prices as a slightly mystical source of information, who don't have much of a model of what cognition is behind those prices.

While this is a very valuable post, I don't think the core argument quite holds, for the following reasons:

1. Markets work well as information aggregation algorithms when it is possible to profit a lot from being the first to realize something (e.g., as portrayed in "The Big Short" about the Financial Crisis).
2. In this case, there is no way for the first movers to profit big. Sure, you can take your capital out of the market and spend it before the world ends (or everyone becomes super-rich post-singularity), but that's not the same as making a billion bucks. 
3. You can argue that one could take a short position on interest rates (e.g., in the form of a loan) if you believe that they will rise at some point, but that is a different bet from short timelines - what you're betting on then, is when the world will realize that timelines are short, since that's what it will take before many people choose to pull out of the market, and thus drive interest rates up. It is entirely possible to believe both that timelines are short, and that the world won't realize AI is near for a while yet, in which case you wouldn't do this. Furthermore, counterparty risks tend to get in the way of taking up very big loans, and so they would dominate your cost of capital.
4. All that said, it is possible that the strategy of "people with a high x-risk estimate should use long-term loans to fund their work" is indeed a feasible funding mechanism for such work, since this would not be a bet intending to make the borrower rich - it would just be a bet to survive, although you could get poor in the process.

I trade global rates for a large hedge fund so I think i can give the inside view on how financial market participants think about this. 

First, the essential claim is true - no one in rates markets talks about the theme of AI driving a massive increase in potential growth. 

However, even if this did become accepted as a potential scenario it would be very unlikely to show up in government bond yields so using yields as evidence of the likelihood of the scenario is, imho, a mistake. I'll give a number of reasons.

1. Rates markets don't price in events (even ones that are fully known) more than one or two years ahead of time (Y2K, contentious elections in Italy or France, etc). This is generally outside participants time horizons but also...
2. A lot can happen in two years (much less ten years). Major terrorist attack, pandemic, nuclear war to name three possibilities all of which would fully torpedo any bet you would make on AI, no matter how certain you are of the outcome.
3. The premise is not obviously true that higher growth leads to higher real yields. That is one heuristic among many when thinking about what real yields should do. It's important to think about the mechanism here - if real yields are well below the economic growth rate then businesses are incentivized to borrow money to invest in projects which earn that growth rate. But this is really a return on capital phenomenon, not a growth rate phenomenon. It's much more applicable to a scenario of humanoid robots where you have to invest large amounts of money up front to yield a return in wage savings over time. Zero marginal cost software doesn't require the same kind of capital investment so there is no surge in borrowing necessary to attain the economic growth. 
4. The heuristic of rates=growth is really better thought of as what rate is necessary to equilibrate savings and investment. So even a massive demand for investment funds can be offset by willing savers. In the case where gains from AI were evenly distributed it's plausible to say the income effect is such that people generally will  feel richer and thus want to consume more today (and will require a high interest rate to  push consumption from today to tomorrow). But in reality the wealth gains from AI will probably accrue to a small slice of the population. Rich people have very low marginal propensities to consume (and high marginal propensities to save). So a surge in wealth to rich people is generally thought to push interest rates down.
5. Saving the most practical point for last: Interest rates in developed economies are heavily influenced by where the Fed wants them to be in order to fulfill their mandate of full employment and 2% inflation. If there were a huge speculative increase in interest rates in anticipation of a growth surge ten years from now that would torpedo current borrowing and thus the economy. So the Fed would fight back by cutting the overnight interest rate and, if necessary, purchasing bonds. The Fed's ammo is essentially unlimited and they would get their way and you would lose money.

To try to group/summarize the discussion in the comments and offer some replies:

1. ‘Traders are not thinking about AGI, the inferential distance is too large’; or ‘a short can only profit if other people take the short position too’

(a) Anyone who thinks they have an edge in markets thinks they've noticed something which requires such a large inferential distance that no one else has seen it.

• Any trade requires that the market price eventually converges to the ‘correct’ price
• ⇒ This argument proves too much – it’s a general argument against ever betting that the market will correct an incorrect price!
  • Those who are arguing against need to be a clearer argument about why this situation is fundamentally different from any other
  • Sovereign bond markets are easily some of the most liquid and well-functioning markets ever to exist

(b) Many financial market participants ARE thinking about these issues.

• Asset manager Cathie Wood has AGI timelines of 6-12 years and is betting the house on that (“AGI could accelerate growth in GDP to 30-50% per year”)
• Masayoshi Son raised $100 billion for Softbank’s Vision Fund on the basis that superintelligence will arrive by 2047

The prospect of AGI is not a Thielian secret.

(c) Do make sure to read section X on “Trade risk and foom risk”, where we acknowledge that if you are both (i) extremely skeptical of market efficiency, and (ii) think foom is the likely takeoff scenario, then trading seems less like a good idea.

2. Stocks versus bonds

• Again we refer to detailed discussion in this companion post (appendix 1) on stocks:
  • (1) Stocks cannot capture the risk of unaligned AI
  • (2) Developers of TAI might not actually profit much
  • (3) The developers of TAI might not be publicly traded or even exist yet
  • (4) The development of TAI could even lower stock prices!
• To be clear, though: the economic logic suggests stocks are bad for forecasting timelines (due to the four reasons mentioned in that post)
• BUT stocks still could be good ways to earn money betting on short timelines (if the four sources of noise mentioned in that post don’t turn out to hold)

3. Other empirical evidence on real rates

• Again we refer to detailed discussion in this companion post (appendix 2) and the important discussion of econometric caveats in section V (“Caveats”)
• We emphasize that we would love to have more/better empirical evidence  with respect to asset pricing under existential risk (appendix 3)
• The challenge with using the historical data (e.g. the Bank of England brought up in the comments) is – as discussed in section IV and in the appendix – that these data are infected with (1) poor estimates of expected inflation and (2) poor estimates of credit risk
  • For example, in the Schmelzing paper that has been cited: certain claimed spikes in the risk-free rate, e.g. during the Napoleonic war, look far more like a spike in default risk
  • Or the ex ante real interest rate in World War II in his data is negative, which seems unlikely

Suppose you are one of the 0.1% of macro bonds traders familiar with Yudkowskian foom.  You reason as follows:  "Suppose that in the next 2 years, we get even more alarming news out of GPT-4 and successors.  Suppose it's so incredibly alarming that 10% of macro traders notice, and then 10% of those  hear about Yudkowskian foom scenarios.  Putting myself into the shoes of one of those normie macro traders, I think I reason... that most actual normal people won't change their saving behavior any time soon, even if theoretically they should  decrease their saving,  and that's not likely to have macro effects.  Still as a normie trader who's heard about Yudkowsky foomdoom, I think I reason that if Yudkowsky's right, we're all dead, and if Yudkowsky's not right, I'll get embarrassed about a wrong trade and fired.  So this normie trader won't trade on Yudkowsky foomdoom premises.  Therefore I don't think I can profit over the next two years by shorting a TIPS fund... even leaving aside concerning feelings about whether going hugely short LTPZ would have model risk about LTPZ's actual relation to real interest rates in these scenarios, or whether other traders would expect big AI impacts to hit measured inflation because of AI-driven lower prices or AI-driven unemployment."

And then one week before the end of the world, the 1% of most clueful macro bonds traders... will take vacation days early, and draw down their rainy day funds to spend time with their family.  They still won't make macro trades about that, because the payoff matrix looks like "If you're right, you're dead and not paid, and if you're wrong, you're embarrassed and get fired."  Then haha whoops it turns out that the world didn't end in a week after all, and people go back to work with a nervous laugh and a sick feeling in their stomachs, and everybody actually falls over dead three weeks later.

If Omega tells you today that everyone will be dead in 2030 with probability 1, there's no direct way to make a market profit on that private information over the next 2 years, except insofar as foresightful traders today expect less foresightful future traders to hear about AI and read an economics textbook and decide that interest rates theoretically ought to rise and go short TIPS index funds.  Foresightful traders today don't expect this.

To put it another way:  Yes, savvy market traders don't believe that in 2025 everybody will realize that the world is ending.  The savvy market traders are correct!  Even at the point where the world is ending, everybody will not believe this, and so at no point will the savvy trader have made a profit!  The death of all of humanity induces a market anomaly wherein savvy traders don't expect to be able to profit from everyone else's error because no event occurs where the real thing actually happens and everybody says "Oops" and the savvy trader gets paid off.

There just isn't any mystery here.  You can't make a short-term profit off correcting these market prices even if Omega whispers the truth in your ear with certainty.  That's it, that's the mystery explained, you're done.

EliezerYudkowsky @ 2023-01-12T13:44 (+61)

Trying again:

OP seems to ambiguate between two ideas, one true idea, and one false idea.

The true idea is that if Omega tells you personally that the world will end in 2030 with probability 1, you personally should not bother saving for retirement.  Call this the Personal Idea.

The false idea is that if you believe in foomdoom, you should go long real interest rates and expect a market profit.  Call this the Market Idea.

Intuitively, at least if you're swayed by this essay, the idea in Market probably seems pretty close to the idea in Personal.  If everybody started consuming for today and investing less, real interest rates would go up, right?  So if you don't believe that Market is about as strong as Personal, what invalid reasoning step occurs within the gap between the true premise in Personal to the false conclusion in Market?

Is it invalid that if in 2025 everyone started believing that the world would end in 2030 with probability 1, real interest rates would rise in 2025?  Honestly, I'm not even sure of that in real life.  People are arguing clever-ideas like 'Shouldn't everyone take out big loans due later?' but maybe the lender doesn't want to lend anymore, if everyone knows that.  There's a supply collapse and a demand collapse and yes I see the theoretical argument but real-world monetary stuff is in fact really strange and complicated; I didn't see anybody calling the actual interest-rate trajectory surrounding Covid in advance of it actually playing out.

In real life, what zaps you when you think there's a worldwide pandemic coming and try to trade interest rates, isn't that you didn't know about the pandemic ahead of the oblivious market, it's that you guessed wrong about what the market would really actually do in real life as the pandemic played out and finally ended.

You can sometimes make a profit off an oblivious market, if you guess narrowly enough at reactions that are much more strongly determined.  Wei Dai reports making huge profits on the Covid trade against the oblivious market there, after correctly guessing that people soon noticing Covid would at least increase expected volatility and the price of downside put options would go up.

But I don't think anybody called "there will be a huge market drop followed by an even steeper market ascent, after the Fed's delayed reaction to a huge drop in TIPS-predicted inflation, and people staying home and saving and trading stocks, followed by skyrocketing inflation later."

I don't think the style of reasoning used in the OP is the kind of thing that works reliably.  It's doing the equivalent of trying to predict something harder than 'once the pandemic really hits, people in the short term will notice and expect more volatility and option prices will go up'; it's trying to do the equivalent of predicting the market reaction to the Fed reaction a couple of years later.

The entire OP is written as if we lived in an alternate universe where it is way way easier than history has actually shown, to figure out what happens in broad markets after any sort of complicated or nontrivial event occurs in real life that isn't on the order of "unexpected Fed rate hike" or "company reports much higher-than-expected profits".  And it's written in such a way as to mislead EAs reading it about the general confidence that the field of economics is able to justly put in predictions about broad market reactions to strange things.

If you haven't already looked at OP's recommended investment instrument of (short) LPTZ, which holds inflation-protected Treasuries and is therefore their recommended way of tracking real interest rates, I recommend the following exercise:  First try to figure out what you would have believed a priori, without benefit of hindsight, real interest rates would do over the course of a pandemic.  Then, decide what investment strategy you'd have followed with LTPZ if you thought you knew about a pandemic ahead of the market.  Then, decide what you think happened to real interest rates with benefit of hindsight.   Then, go look at the actual price trajectory of their recommended instrument of LTPZ.

I am not sure I can properly convey this thought that I am trying to convey; I have had trouble actually conveying this thought to EAs before.  The thought is that people often do long careful serious-sounding writeups which EAs then take Very Seriously, because they are so long and so seriously argued, but in fact fail to bind to reality entirely, in a way that doesn't have to do with the details of the complicated arguments.  Very serious arguments about what ought to happen to the price of an ETF that tracks 15-year TIPS, via the intermediate step of arguing about what logically ought to happen to real interest rates, are the sort of thing that, historically, average economists have not really been able to pull off; it's a kind of thought that you should expect fails to achieve basic binding to reality.  What would LTPZ or its post-facto equivalent have been doing around the time of the Cuban Missile Crisis?  My model says 'no prediction'; they'll have done whatever.  Afterwards somebody will make up a story about it in hindsight, but it is not the sort of thing where history says that long complicated analyses are remotely reliably good at doing it in advance.

But there are even weaker links in the argument, so let's accept the LTPZ step arguendo and pass on.

An even bigger problem is that, since everybody is going to die before anything really pays out, the marginal foresightful trader does not have a strong incentive to early-on move the market toward where the market would end up in equilibrium after everyone agreed on the actual facts of the matter and had time to trade about them.

Prediction markets, I sometimes explain to people, are tools for transmitting future observables, or more generally propositions that people expect to publicly agree on at some future point even if they don't agree now, lossily backward in time, manifesting as well-calibrated probability distributions.

To run a prediction market, you first and foremost need a future publicly observable measurement, which is a special case of a place where we expect most people to agree on an extreme probability assignment later, even though they don't agree now or don't make extreme probability assignments now.  You cannot run a prediction market on whether supplementing lots of potassium can produce weight loss; you can only run a prediction market about what an experiment will report in the way of results, or what a particular judge will say the experimental evidence seems to have indicated in five years.  You cannot directly observe "whether potassium causes weight loss" as an underlying fact of biology, so you can't have a prediction market about that; you can only observe what somebody reports as an experimental result, or what a particular person appointed as judge says out loud about the state of evidence later.

The marginal foresightful trader usually has a motive to run ahead of the market and make trades now, based on where the equilibrium ought to settle later; not because they are nobly undertaking a grand altruistic project of transmitting facts backward in time and making the market behave nicely from the standpoint of theoretical economics, but because they expect to get paid after everyone makes the common observation and the market settles into a new equilibrium reflecting that state of knowledge.  And then they expect to have that money, or to get a bonus for earning that money for their proprietary trading firm, and for that money to still hold its value, and for them to be able to spend the money on nice things.

In the unusual case of foomdoom, even if doom proceeds slowly enough that a large-enough group of marginal foresightful traders see the foomdoom coming, even if there is somehow a really definitive fire alarm that says "extremely high probability you are dead within two to four years", it is incredibly unlikely that everyone in the world will agree that yes we'll all be dead in two to four years, and that the markets will settle into the equilibrium that an economist would say corresponds to the belief that we'll all be dead in two to four years; which is what's required for the foresightful proprietary trader to score a huge profit and get a big bonus that year and have time to spend it on some fancy way of passing the remaining time.

People do not usually agree on what will happen in two to four years.  This kind of agreement that reliably reflects a fact, and makes a market pay out in a way that you can trust to correspond to that fact, is usually achieved after that fact is publicly definitively observed, not two to four years ahead of the observation.

In case of foomdoom the world never settles into equilibrium later, the bets never pay out, there is never that moment where everybody says "What a foresightful trader that was!" and agree on the fact that yes we sure are all dead now.  So even if a proprietary trader sees doom coming, they do not have much of an incentive to dutifully transmit that information backward in time in order to make the market behave now in the way that an economist thinks ought to correspond to the equilibrium it would settle into after everybody agreed that they were dead.

That incentive would only exist if you expected everybody to agree that they were going to die in a few years, far enough ahead of everybody actually being dead, for the markets to settle into equilibrium and foresightful traders to collect bonuses on having made the trade before that.  Which is a much stronger and stranger thing to claim, about a planet like Earth, than the usual much weaker claim that a few sharp traders might see a fact coming, and move the markets a few years ahead of time to where they would go after everybody agreed on that fact later.

Though even then, of course, we have cases like the financial crisis of 2006-2008, where some traders did see it coming and turn huge profits, but couldn't move enough marginal money around to actually shift the entire broader market.

To suppose that the market is broken around foomdoom is really not a remotely surprising market behavior to suppose!  Even in a world where nearly all the prices are efficient relative to you and that's why you can't make 10%/day trading Microsoft stock!

What happened in 2006-2008 was much more broken than that!  Marginal traders saw it coming, and some of them won huge even though CDSs were not trivial to short; but they didn't move enough money to shift anything remotely as large as 'real interest rates' ahead of the actual materialization of the disaster.

The market's behavior around Covid also showed much more obliviousness than this; it showed the kind of obliviousness where people I'd previously marked as the strongest EMH challengers reported collecting vast profits over a timespan of a couple of months.  (But on chains of logic much less fraught than OP's, because in real life you can't call LTPZ movements or real interest rate changes in advance, just things on the order of 'buy volatility'.)

We should not believe 'the market', in the sense of that unusually intelligent entity whose opinions we actually pay attention to, driven by the highly incentivized marginal trader, has any opinion on AGI except that "in the next few years, not everyone will have started believing that they are going to die in a few years after that".  The market is showing no actual opinion on foomdoom, only on what most market participants will believe about foomdoom in a couple of years.   The usual incentive mechanism whereby, if a pandemic starts, in a few years most market participants will agree that this pandemic happened and foresightful traders will collect bonuses and spend them - as is responsible for the market sometimes but not always being foresightful, because it is paid to be foresightful - is in this case broken.  We are really always seeing, when the market foretells an observable's value a few years later, that the foresightful marginal trader thinks that a few years later lots of people will hold a certain opinion; it's just that usually, this common opinion is being mundanely driven by a direct observation.

The market says, "It won't be the case that in 2030, everyone agrees that they were killed by AGIs."  The market isn't saying anything about whether that's because everyone agrees they are alive, or because nobody is left to agree anything.

OP reads like somebody has heard that markets sometimes anticipate things ahead of them happening, that markets sometimes transmit information lossily backward in time, and doesn't quite seem to have understood the mechanism behind it; that what everyone will agree on later, unusually foresightful marginal traders can sometimes cause the market to reflect now even though not everyone agrees on it yet.  Instead they are talking about "What if the market believed..." as if this kind of market belief reflected a numerical majority of the people in the market believing that foomdoom would kill us all before 30-year bonds paid out.  But this is not where the market gets its power to say things that we ought to pay special attention to (though even then market isn't always right about those things, or even righter than us, especially if those things are a little strange, eg Covid etc).  The market gets its power from unusually foresightful marginal traders expecting to get a payoff from what everyone believes later after the thing actually happened and therefore most market participants agree about what happened.  And this transmission mechanism is broken w/r/t AGI doom in a way that it wouldn't even be broken for an asteroid strike; with an asteroid strike, you might get weird effects from money losing its value in the future, but at least people could all agree on the asteroid strike coming.  With AGI, I think you'd have to be pretty naive to expect everybody to agree that AGI will kill us in two years, two years before AGI kills us.  So it is inappropriate to skip over a step we can usually skip over, and compress the true proposition "The market doesn't believe that everyone in 2030 will believe that in 2030 everyone is dead", to the false proposition "The market doesn't believe that in 2030 everyone will be dead."

Though again - also just to be clear - AGI ruin is a harder call than Covid and I wouldn't strongly expect the market to get it right, even if transmission weren't broken; and even if I thought the market would get it right after seeing GPT-4 and that transmission wasn't broken, I wouldn't buy "short LTPZ" as a surefire way to profit.

Ben_West @ 2023-01-14T04:49 (+14)

What would LTPZ or its post-facto equivalent have been doing around the time of the Cuban Missile Crisis?  My model says 'no prediction'; they'll have done whatever.  Afterwards somebody will make up a story about it in hindsight, but it is not the sort of thing where history says that long complicated analyses are remotely reliably good at doing it in advance.

In appendix 3 the authors cite a paper which looks at more-or-less this precise thing:

Figure 4.6: Observed Treasury and aggregate-equity movements around President Kennedy’s 22 October 1962 19:00 EDT Cuba address. For the cumulative value-weighted CRSP return, a positive value on a date up to and including that of the address indicates a negative return through the date of the address, and a negative value after the address indicates a negative return from the first trading day after the address.

It seems like government interest rates didn't change much? But I don't think I understand this graph.

EliezerYudkowsky @ 2023-01-14T22:01 (+1)

Those look like nominal rates, not real rates. 

Karthik Tadepalli @ 2023-01-14T22:24 (+8)

Inflation linked bonds are recent, so virtually all historical analyses are going to use nominal rates. I think this is a tradeoff well worth making to say something about asset pricing under existential risk. For the effects to be wrong because of nominality, the Cuban missile crisis would have had to affect market perceptions primarily because of... inflation expectations. I feel comfortable rejecting that as a story.

Peter Chatain @ 2023-01-11T04:05 (+34)

I'm a bit confused by this comment. Is the claim that you can't profit from taking out a loan that you have to pay back in 2040? The fact that you get money now and have to pay it back at a time when hypothetically money no longer matters seems like profit to me. If Omega whispers the truth into my ear that in 2040 there will be foom, then that's guaranteed profit.

lexande @ 2023-01-11T19:09 (+26)

Nobody will give you an unsecured loan to fund consumption or donations with most of the money not due for 15+ years; most people in our society who would borrow on such terms would default.  (You can get close with some types of student loan, so if there's education that you'd experience as intrinsically-valued consumption or be able to rapidly apply to philanthropic ends then this post suggests you should perhaps be more willing to borrow to fund it than you would be otherwise, but your personal upside there is pretty limited.)

zdgroff @ 2023-01-11T22:35 (+9)

It might be challenging to borrow (though I'm not sure), but there seem to be plenty of sophisticated entities that should be selling off their bonds and aren't. The top-level comment does cut into the gains from shorting (as the OP concedes), but I think it's right that there are borrowing-esque things to do.

Peter Favaloro @ 2023-01-12T00:01 (+11)

The reason sophisticated entities like e.g. hedge funds hold bonds isn't so they can collect a cash flow 10 years from now. It's because they think bond prices will go up tomorrow, or next year. 

The big entities that hold bonds for the future cash flows are e.g. pension funds. It would be very surprising and (I think) borderline illegal if the pension funds ever started reasoning, "I guess I don't need to worry about cash flows after 2045, since the world will probably end before then. So I'll just hold shorter-term assets."

I think this adds up to, no big investors can directly profit from the final outcome here. Though as everyone seems to agree, anyone could profit by being short bonds (or underweight bonds) while the market started to price in substantial probability of AGI.

lexande @ 2023-01-12T02:25 (+7)

If you're in charge of investing decisions for a pension fund or sovereign wealth fund or similar, you likely can't personally derive any benefit from having the fund sell off its bonds and other long-term assets now. You might do this in your personal account but the impact will be small.

For government bonds in particular it also seems relevant that I think most are held by entities that are effectively required to hold them for some reason (e.g. bank capital requirements, pension fund regulations) or otherwise oddly insensitive to their low ROI compared to alternatives. See also the "equity premium puzzle".

Marcel D @ 2023-01-11T19:22 (+4)

I don’t know whether I defend Yudkowsky’s view (I only skimmed), but:

1. The “profit” in the scenario you describe doesn’t seem sufficient to move the market (as small investors), because it isn’t “profit” that you can keep growing and reinvesting (snowballing) until your Special Insights (TM) fix the market;

2. If you end up with a scenario where 2040 arrives and it’s Utopia, then there may not be any serious incentive to care, whereas there would be downside risks if it doesn’t occur: if you’re right, cool, you lived slightly better for 20 years out of a 1,000,000,000-year life of happiness; if you’re wrong you might be in so much debt you can’t pay medical bills/whatever, and die before actual AGI date.

3. I’m not confident you could get massive loans with no collateral and no business model or whatnot to repay the loans come 2040.

lexande @ 2023-01-11T21:17 (+14)

Beyond just taking vacation days, if you're a bond trader who believes in a very high chance of xrisk in the next five years it probably might make sense to quit your job and fund your consumption out of your retirement savings. At which point you aren't a bond trader anymore and your beliefs no longer have much impact on bond prices.

RobBensinger @ 2023-01-12T00:52 (+6)

Or leave your job to go try and reduce x-risk?

Insofar as humans care about x-risk and act on this care, you'd expect bond traders to be people who either think x-risk is very low, think it's high and think they have no nontrivial way to affect it, or think it's high and think trading is their best way to affect it (e.g., through earning to give).

Yonatan Cale @ 2023-01-10T22:33 (+13)

People disagreeing, would you say why?

My guess for the pushback: 1 week before the end of the world, you think a sizable part of the population will notice and change their economic behavior drastically. I imagine this scenario contains a slow "attack" by AI that everyone sees coming? 

(agree vote = yeah that is the pushback)

crafkin @ 2023-01-11T02:34 (+32)

If the only scenario for AI were: exist at more or less normal levels of economic growth, then foom, I think Eliezer would be correct. However, I think a likely scenario is that there is accelerated growth, coincident with accelerated risk (via e.g. AI terrorism or wars), before foom. This cluster of outcomes may even be modal. In that case, interest rates would very likely rise and traders would be able to profit, before foom. 

Put another way, we often place substantial weight on non-foom good and bad AI scenarios, even if foom is a risk. The market seems to be ruling out non-foom AI risk as well as non-foom AI growth before foom. Eliezer is correct that the market may not be ruling out the (IMO unlikely) scenario of "no-AI induced growth or risk prior to foom, then foom."

zdgroff @ 2023-01-11T22:26 (+7)

I'm trying to make sure I understand: Is this (a more colorful version) of the same point as the OP makes at the end of "Bet on real rates rising"?

The other risk that could motivate not making this bet is the risk that the market – for some unspecified reason – never has a chance to correct, because (1) transformative AI ends up unaligned and (2) humanity’s conversion into paperclips occurs overnight. This would prevent the market from ever “waking up”.

However, to be clear, expecting this specific scenario requires both: 

1. Buying into specific stories about how takeoff will occur: specifically, Yudkowskian foom-type scenarios with fast takeoff.
2. Having a lot of skepticism about the optimization forces pushing financial markets towards informational efficiency.

You should be sure that your beliefs are actually congruent with these requirements, if you want to refuse to bet that real rates will rise. Additionally, we will see that the second suggestion in this section (“impatient philanthropy”) is not affected by the possibility of foom scenarios.

EliezerYudkowsky @ 2023-01-12T12:03 (+3)

I wouldn't say that I have "a lot of" skepticism about the applicability of the EMH in this case; you only need realism to believe that the bar is above USDT and Covid, for a case where nobody ever says 'oops' and the market never pays out.

John G. Halstead @ 2023-01-13T09:38 (+11)

What do you make of the 'impatient philanthropy' argument? Do you think EAs should be borrowing to spend on AI safety?

EliezerYudkowsky @ 2023-01-13T14:08 (+4)

Not until timelines are even blatantly shorter than present and long-term loans are on offer, and not unless there's something useful to actually do with the money.

Benjamin_Todd @ 2023-01-13T10:55 (+2)

Am I being dumb or do you mean short TIPS? If real interest rates rise, TIPS go down.

EliezerYudkowsky @ 2023-01-13T14:07 (+2)

Corrected, thanks.

Simon_M @ 2023-01-13T12:01 (+1)

Yes - you are correct.

Nice post, but my rough take is there is 
 

• it's relatively common markets are inefficient, but unexploitable; trading on "everyone dies" seems a clear case of hard-to-exploit inefficiency
• markets are not magic; impacts of one-off events with complex consequences are difficult to price in, and what all the magical market aggregation boils down to are a bunch of  human brains doing the trades; e.g. I was able to beat the market and get n-times return at point where markets were insane about covid; later, I talked about it with someone in one of the giant hedge funds, and the simple explanation is, while they were looking into it, at some point I knew more about covid than they were able to assemble
  • example of such hard-to-predict event are e.g. capabilities and impacts of a specific model 
• the dichotomy 30% growth / everyone dies is unrealistic for trading purposes
  • near term, there are various outcomes like "industry X gets disrupted" or  "someone loses job due to automation" or "war"
    • if you anticipate fears of this type to dominate in next 10 years, you should price many people increasing their savings and borrowing less
       

I'm curating this post because I think it raises important points that I'd like more people to engage with, and because the discussion on it has been really interesting (124 comments right now). I also think it'd be very good to have more genuine critical engagement with the case for prioritizing work on AI existential risk. 

However, the post seems wrong or at least heavily disputable in important ways, so I will also point out relevant considerations and comments. Please note that I'm not an expert; in some cases, I'm merely summarizing my understanding of what others have said.

This comment ended up being very long. Before it goes on, I want to direct attention to some other content arguing against the case that risk from AI is high or should be prioritized: 

1. But have they engaged with the arguments? (Phil Trammell)^[1]
2. Counterarguments to the basic AI risk case (Katja Grace)
3. How sure are we about this AI stuff? (Ben Garfinkel)
4. This thread by Matthew Barnett (which is responding to arguments against AGI being near)
5. Wikipedia's section on skepticism about risk from AGI

The overall claim that the post is making is the following (I think): 

• Markets are a good way of finding an outside view on a topic — in this case, on transformative AI. Long-term real rates would be higher than they currently are if markets believed that transformative AI was coming in the next ~30 years. So you should expect longer timelines for transformative AI. Also, if you really believe that transformative AI is coming soon, you can make money by disagreeing with the market's current position (by shorting bonds).  

One thing that seems worth pointing out before we get into the disagreements: the arguments in the post rely on the idea that, if transformative AI is coming at a certain time, we should expect that more and more people will believe this as that time draws near. This doesn't seem unreasonable to me, but I didn't see it outlined as an assumption in the post. 

Moving on: some disagreements (a non-exhaustive list):

1. Interpreting or correcting markets like this is a messy business, especially when trying to predict events that are years out, especially when there's not a lot to profit from in the near-term or continuously. And the crazier things get, the more unusual things markets might do. As Jan points out, 
   1. "the dichotomy 30% growth / everyone dies is unrealistic for trading purposes
      1. near term, there are various outcomes like "industry X gets disrupted" or  "someone loses job due to automation" or "war"
         • if you anticipate fears of this type to dominate in next 10 years, you should price many people increasing their savings and borrowing less"
   2. And Eliezer writes: 
      1. You can sometimes make a profit off an oblivious market, if you guess narrowly enough at reactions that are much more strongly determined.  Wei Dai reports making huge profits on the Covid trade against the oblivious market there, after correctly guessing that people soon noticing Covid would at least increase expected volatility and the price of downside put options would go up.
      2. But I don't think anybody called "there will be a huge market drop followed by an even steeper market ascent, after the Fed's delayed reaction to a huge drop in TIPS-predicted inflation, and people staying home and saving and trading stocks, followed by skyrocketing inflation later."
      3. I don't think the style of reasoning used in the OP is the kind of thing that works reliably.  It's doing the equivalent of trying to predict something harder than 'once the pandemic really hits, people in the short term will notice and expect more volatility and option prices will go up'; it's trying to do the equivalent of predicting the market reaction to the Fed reaction a couple of years later.
   3. And from this comment: Expecting the market to be (decently) efficient here is less reasonable, because 
      1. "Suppose you have someone who has better insights than everyone else about some asset. They may not be rich and for various related reasons they are unable to immediately correct the market (i.e., the market is actually temporarily inefficient). However, if they are [right], they either a) can keep profiting over and over again until they become liquid/rich enough to individually correct the market, and/or b) other people see that this person is profiting over and over again so they jump in and contribute to market correction."
         The problem is that it might be the case that there is only one or two cycles for profit with AGI until the world goes crazy, but it could take many years for this strategy to actually profit, during which time the market will be "temporarily" inefficient. If real interest rates don't rise for 15 years, and only start to rise ~5 years before AGI, the market is inefficient for 15 years because small players can't profit to fix the situation.
2. Markets are really bad at dealing with extreme events
   1. The classic argument I've seen for prediction markets goes like this:^[2] prediction markets work because people can expect to be paid for being right. Say someone sets up a market on whether humanity is extinct by 2030. If we're not extinct, traders who bought "not extinct" shares profit. If we are... no one profits. So you don't really have an incentive to buy "extinct" shares, even if you know for a fact that humanity will go extinct before 2030. (There are cleverer ways to set up this kind of market, but we'll skip them for now.) (Related post/section.) (Also explained in this comment, among others.)
   2. This applies here, to a certain extent, and several commenters point this out. The post tries to get around this by focusing on real rates, but there's only a limited amount of profit possible if the markets are incorrect in this direction; as suggested, you can stop saving past 2030 (or discount according to your timelines) and try to short government debt^[3], but this is not enough to profit massively and not enough to fully correct the markets if they are off in this way. 
   3. Jakob points this out in a comment which also notes, "You can argue that one could take a short position on interest rates (e.g., in the form of a loan) if you believe that they will rise at some point, but that is a different bet from short timelines - what you're betting on then, is when the world will realize that timelines are short, since that's what it will take before many people choose to pull out of the market, and thus drive interest rates up. It is entirely possible to believe both that timelines are short, and that the world won't realize AI is near for a while yet, in which case you wouldn't do this." Or, as explained in other comments, "If [you find out] that everyone will be dead in 2030 with probability 1, there's no direct way to make a market profit on that private information over the next 2 years, except insofar as foresightful traders today expect less foresightful future traders to hear about AI and read an economics textbook and decide that interest rates theoretically ought to rise and go long TIPS index funds.  Foresightful traders today don't expect this." (More on this.)
3. One proposed mechanism for profiting off something like knowing that humanity will go extinct by 2030 is taking a big loan that you don't expect to pay out. However, lenders would generally ask for a big collateral/margin on a loan with such a long timeline (otherwise lots of people would take loans like this and default). Additionally, if the treasury rate went down in the short-term, the lender might close out the trade at a loss for the borrower, even if the borrower is right on a longer time span. (A commenter points out that there might be student loans that work for this.)
4. I think the examples listed in the "Empirical evidence on real rates and mortality risk" section are importantly different from transformative AI because (1) the catastrophes aren't existential (see above, on extreme events) and (2) in those cases there's info that some people have — insider trading, which we don't necessarily have here but which is good for correcting markets. So I think you should expect this approach to work better for the examples than for the AI case.
   1. I don't know what happened in situations that seem slightly more relevant, like the Cold War, and I would be interested in hearing more if someone knows or can look into it. But even those situations seem very different.
5. A number of people in the comments are discussing whether enough people trading/investing/borrowing etc. are aware enough of arguments about transformative AI to be able to form an opinion on this and correct the market if the market is in fact off in this direction. Rohin writes: "If you already knew that belief in AGI soon was a very contrarian position (including amongst the most wealthy, smart, and influential people), I don't think you should update at all on the fact that the market doesn't expect AGI."

1. ^{^}

   "... What's going wrong, I think, is something like this. People encounter uncommonly-believed propositions now and then, like “AI safety research is the most valuable use of philanthropic money and talent in the world” or “Sikhism is true”, and decide whether or not to investigate them further. If they decide to hear out a first round of arguments but don't find them compelling enough, they drop out of the process. (Let's say that how compelling an argument seems is its “true strength” plus some random, mean-zero error.) If they do find the arguments compelling enough, they consider further investigation worth their time. They then tell the evangelist (or search engine or whatever) why they still object to the claim, and the evangelist (or whatever) brings a second round of arguments in reply. The process repeats.

   As should be clear, this process can, after a few iterations, produce a situation in which most of those who have engaged with the arguments for a claim beyond some depth believe in it. But this is just because of the filtering mechanism: the deeper arguments were only ever exposed to people who were already, coincidentally, persuaded by the initial arguments. If people were chosen at random and forced to hear out all the arguments, most would not be persuaded. ..."

   Related, I think: The motivated reasoning critique of effective altruism, Epistemic learned helplessness

2. ^{^}

   I can't find a quick link about it that concisely explains just the argument — would appreciate one!

3. ^{^}

   ChatGPT's explanation of what "shorting government debt" means: 

   In finance, "shorting" (or "short selling") refers to the process of selling an asset that the seller does not own with the expectation that the price of the asset will decrease. The seller will then purchase the asset at the lower price, in order to make a profit. Short selling is typically used as a way to speculate on a market decline, or to hedge against potential losses in an existing long position.

   Shorting government debt is done by borrowing government bonds from a lender, such as a broker or another investor, and then selling them on the open market with the expectation that their value will decrease [so you have to pay back less than you earned by selling the bonds on the open market]. The borrower hopes to be able to repurchase the bonds at a lower price in the future, and return them to the lender, while keeping the difference as profit.

Alternatively, returning to the LTPZ ETF with its duration of 20 years, a 3 percentage point rise in rates would cause its value to fall by 60%. Using the 3x levered TTT with duration of 18 years, a 3 percentage point rise in rates would imply a mouth-watering cumulative return of 162%.

I am confused. This is not a lot of return at 10 year timelines, and calling it "mouth-watering" seems a bit excessive. A cumulative return of 162% over 10 years is equivalent to around 10% annually, which is around as good as putting a money into a normal ETF over the last 20 years (which averaged around 8-10% over the last 20 years), and this is not considering that since I am short the market, in any world where I am wrong, I will likely lose a lot of money. If I take the downside into account, I end up with an annualized return of around 5-6% on this portfolio, which really doesn't seem great to me.

Taking the math here at face value, you are suggesting a trading strategy with a ~5-6% annualized return, which is maybe barely competetive with other investing strategies, and a lot less than the average return of most EA-adjacent investor who have been thinking about this stuff for the last few years. Like, yeah, maybe you can very slightly outperform the market, but this seems hardly like a strong correcting force, and this definitely doesn't seem like a good pitch that I should move my investment away from just holding tech-concentrated ETFs with a bunch of Nvidia exposure.

I might just be misunderstanding the returns you are claiming here, and the exact financial trade. Am I missing anything? What trading strategy do you concretely recommend that will make me money if I believe that timelines are shorter than this that outperforms something simple like buying Nvidia stock (and ideally will make me money at least a year or two before I do expect the world to end, since I don't think I really gain much by having more money just before AI kills everyone)?

(Edit: Adjusted calculations for 10 years instead of 18 years)

I spent about an hour today trying to convince a friend that works in private equity that OpenAI is undervalued at $30B. I pitched him on short AI timelines and transformative growth, and he didn’t disagree with those arguments directly. He mostly questioned whether OpenAI would reap the benefits of short timelines. A few of the points:

• It’s a competitive industry with other players on par or not far behind. Google, Meta, and Anthropic are there already, and startups like Stability and Cohere could quickly close the gap. This is especially true if “scale is all you need”, rather than human capital or privately generated data.
• The main opportunity is B2B, not B2C. Businesses are more cost sensitive and interested in cheaper alternatives than consumers, who gladly accept name brands.
• Profits often lag behind research breakthroughs for years and even decades. There’s no billion dollar app for GPT yet. Investors “don’t care about anything that’s more than 15 years away.”

IMO these are boring economic arguments that don’t refute the core thesis of short timelines or AI risk. OpenAI is getting a similar evaluation to Grammarly, which also sells an LLM product, but with worse tech and better marketing. It’s being evaluated on short term revenue prospects more than considerations about TAI timelines.

Thanks for writing this! I think market data can be a valuable source of information about the probability of various AI scenarios--along with other approaches, like forecasting tournaments, since each has its own strengths and weaknesses. I think it’s a pity that relatively little has yet been written on extracting information about AI timelines from market data, and I’m glad that this post has brought the idea to people’s attention and demonstrated that it’s possible to make at least some progress.

That said, there is one broad limitation to this analysis that hasn’t gotten quite as much attention so far as I think it deserves. (Basil: yes, this is the thing we discussed last summer….) This is that low real, risk-free interest rates are compatible with the belief
1) that there will be no AI-driven growth explosion, 
as you discuss--but also with some AI-growth-explosion-compatible beliefs investors might have, including
2) that future growth could well be very fast or very slow, and
3) that growth will be fast but marginal utility in consumption will nevertheless stay high, because AI will give us such mindblowing new things to spend on (my “new products” hobby-horse).
So it seems impossible to put any upper bound (below 100%) on the probability people are assigning to near-term explosive growth purely by looking at real, risk-free interest rates.

To infer that investors believe (1), one of course has to think hard about all the alternatives (including but not limited to (2) and (3)) and rule them out. But (if I’m not mistaken) all you do along these lines is to partly rule out (2), by exploring the implications of putting a yearly probability on the economy permanently stagnating. I found that helpful. As you observe, merely (though I understand that you don't see it as “merely”!) introducing a 20% chance of stagnation by 2053 is enough to mostly offset the interest rate increases produced by an 80% chance of Cotra AI timelines. You don’t currently incorporate any negative-growth scenarios, but even a small chance of negative growth seems like it should be enough to fully offset said interest rate increase. This is because of the asymmetry produced by diminishing marginal utility: the marginal utility of an extra dollar saved can only fall to zero, if you turn out to be very rich in the future, whereas it can rise arbitrarily high if you turn out to be very poor. (You note this when you say “the real interest rate reflects the expected future economic growth rate, where importantly the expectation is taken over the risk-neutral measure”, but I think the departure from caring about what we would normally call the expected growth rate is important and kind of obscured here.)

This seems especially relevant given that what investors should be expected to care about is the expected growth rate of their own future consumption, rather than of GDP. Even if they’re certain that AI is coming and bound to accelerate GDP growth, they could worry that it stands some chance of making a small handful of people rich and themselves poor. You write that “truly transformative AI leading to 30%+ economy-wide growth… would not be possible without having economy-wide benefits”, but this is not so clear to me. You might think that’s crazy, but given that I don’t, presumably some other investors don’t.

Anyway: this is all to say that I’m skeptical of inferring much from risk-free interest rates alone. This doesn’t mean we can’t draw inferences from market data, though! For one thing, on the hypothesis that investors believe “(2)”, we would probably expect to see the “insurance value” of bonds, and thus the equity premium, rising over time (as we do, albeit weakly). For another thing, one can presumably test how the market reacts to AI news. I’m certainly interested to see any further work people do in this direction.

This post's thesis is that markets don't expect AGI in the next 30 years. I'll make a stronger claim: most people don't expect AGI in the next 30 years; it's a contrarian position. Anyone expecting AGI in that time is disagreeing with a very large swath of humanity.

(It's a stronger claim because "most people don't expect AGI" implies "markets don't expect AGI", but the reverse is not true. (Not literally so -- you can construct scenarios like "only investors expect AGI while others don't" where most people don't expect AGI but the market does expect AGI -- but these seem like edge cases that clearly don't apply to reality.))

Personally I feel okay disagreeing with the rest of humanity on this, because (a) the arguments seem solid to me, while the counterarguments don't, and (b) the AGI community has put in much more serial thought into the question than the rest of humanity.

If you already knew that belief in AGI soon was a very contrarian position (including amongst the most wealthy, smart, and influential people), I don't think you should update at all on the fact that the market doesn't expect AGI.

If you didn't know that, consider this your wake up call to reflect on the fact that most people disagree with you. You don' t need to think about financial markets or real interest rates or what trades would make you rich; all of those effects are downstream of the fact that most people disagree with you.

(Separately, I do expect you can bet on belief in AGI through financial markets, since I do expect that if AGI is coming soon the rest of the world will eventually figure that out. But it's not clear when you can expect to make money; that depends on when exactly the rest of humanity figures it out.)

This is one way of looking at the data (with my overlapping data claim already noted).

Here is another (much longer) way of looking at the data.

Here is 800 years of history courtesy of the BoE

Eyeballing that, I'd say the relationship is strongly negative...

I've been discussing this concept for some time now, so I'm glad to see some people take a more formal stab at it. However, I must say that I'm overall disappointed with this post. I'll just lay out a few summary points, and if people are actually still reading this deep into the comments and want to hear more thoughts, I can oblige later:

1. With the *slight* exception of the "you could be earning alpha" section, it does not really get deep into the causal mechanisms for why you should expect markets to be efficient.
2. I think this post should have done a better job of aggregating and responding to contrary viewpoints; I feel like the post largely bypassed the key arguments (cruxes) of existing critics and went straight to people who were not familiar with the EMH+AGI debate , especially with all the references to empirical evidence (see next point).
   Granted, "better job" implies that the article did this at all, which I don't recall it really doing, aside from occasional references to other viewpoints (IIRC).
3. The empirical sections I thought were decent, but they missed the crux of the debate. 
   1. The fact is, we don't seem to have much of any precedent of this kind of scenario, with some debatable exceptions regarding the Cold War / Cuban Missile Crisis—yet the authors didn't even spend that much time focusing on these examples which seemed to be the most relevant.
   2. Overall, I thought that the empirical sections were not very helpful for the debate, aside from perhaps targeting audiences who are the very early stage of the debate and hastily think "I'll dismiss EMH in general because of X." 
      (I am normally a big proponent of EMH-style reasoning; it's not like I and many other people I know that are part of this EMH+AGI debate are saying "EMH has never worked!"^[1]).
4. One cross-cutting objection off the top of my head is that the people with Special And Justified Knowledge may not be able to profit fast enough to correct the market. [I have read other comments' responses, and respond to one response in the next point]
   This especially applies to two closely related causal mechanisms of the EMH, profit snowballing and dogpiling: "Suppose you have someone who has better insights than everyone else about some asset. They may not be rich and for various related reasons they are unable to immediately correct the market (i.e., the market is actually temporarily inefficient). However, if they are right/superior, they either a) can keep profiting over and over again until they become liquid/rich enough to individually correct the market, and/or b) other people see that this person is profiting over and over again so they jump in and contribute to market correction."
   The problem is that it might be the case that there is only one or two cycles for profit with AGI until the world goes crazy, but it could take many years for this strategy to actually profit, during which time the market will be "temporarily" inefficient. If real interest rates don't rise for 15 years, and only start to rise ~5 years before AGI, the market is inefficient for 15 years because small players can't profit to fix the situation.
5. "But those are just two causal mechanisms," the authors/defenders hypothetically reply, "and sometimes the market still corrects even without those mechanisms; look at the big short! And there are probably enough AI-conscious investors such that they could alter the market..."
   1. First, I think it's worthwhile to highlight my view that the debate can unproductively explode at this point because the original authors didn't (IMO) do a good job of laying out their own causal mechanisms. This forces critics into a game of whack-a-mole filled with delays at the need to comment, wait for responses, address new causal mechanisms, parse out alternate branches of disagreement, etc. 
      (However, I think the following subpoint addresses a fairly large part of the debate)
   2. Second, I don't think that the authors did a good job of differentiating between "sudden surprise takeoffs" (e.g., ~1 year of warning time and ~1/3rd of people believe this) vs. "forecastable takeoffs" (e.g., ~10 years of warning time and >1/10th of people believe this). This seems somewhat cruxy in at least one direction—against the authors' viewpoint. Ultimately, (correct me if I'm wrong) it seems that the authors' proposed strategy for profit relies on the belief that as you get closer to expected AGI date, more/richer people will start to agree with your predictions (and still see benefits from getting in on profit): otherwise, prescient investors could believe "AI is very likely to occur around year X, but  very few people or institutions will recognize this before X-3 years, such that real interest rates probably won't change much at all until it's too late, and when they do change
      1. The counterparty/non-payment risk may be high;
      2. I prefer a 50% chance of being moderately wealthy for 10 years to a 50% chance of being really rich for ~2 years before I die (with a 50% chance of being poor for 10 years if I bet big and am wrong);
      3. The world might experience chaos which undermines my ability to spend money on things I value, etc."
   3. Third, I don't think the claim that "there are probably enough AI-conscious investors..." is supported in this post, and I'm hesitant on this point. I am willing to budge, and this could be a fairly important point if we are in a "forecastable/slow takeoff" scenario, but I would like to see the post focusing on that leaf of the debate, not trying to recreate the trunk of the debate tree. And again, if we are in a "sudden short timeline" scenario, I suspect that this possibility doesn't matter all that much.

1. ^{^}

   Sure, some people may hold this view, but a) I'm skeptical you'll convince them with this article, and b) you can't just focus on empirics and then declare victory when there are still many critics who have objections you haven't directly addressed.

A few years ago I asked around among finance and finance-adjacent friends about whether the interest rates on 30 or 50 year government bonds had implications about what the market or its participants believed regarding xrisk or transformative AI, but eventually became convinced that they do not.

As far as I can tell nobody is even particularly trying to predict 30+ years out. My impression is:

• A typical marginal 30-year bond investor is betting that interest rates will be even lower in 5-10 years, and then they can sell their 30 year bond for a profit since it will have a higher locked-in interest rate than anything being issued then.
• Lots of market actors have a regulatory obligation (e.g. bank capital requirements) to buy government bonds which drives the interest rate on such bonds down a lot, to the point that it can be significantly negative for long periods even when the market generally expects the economy to grow. Corporate bonds have less of this issue but are almost never issued for such long durations.

It's true that the market clearly doesn't believe in extremely short timelines (like real GDP either doubling or going to zero in the next 5-10 years). But I think it mostly doesn't have beliefs about 30+ years out, or if it does their impacts on prices are swamped by its beliefs about nearer-term stuff.

I am confused by some of the logic in this post.

• If TAI arrives soon, either I'll be dead (so I should borrow and spend all my money now -- this part makes sense to me) or I'll be fantastically rich in a post-TAI utopia, so you say I should borrow and spend all my money now, to smooth out my consumption.  Apparently this is the consensus of all mainstream economics and also the results of common-sense.
• But you also say: if TAI arrives soon, that means real interest rates should be higher, so I should engage in a risky investment strategy of shorting the bond market.  This strategy will make me poorer in the short-term, but will pay off by making me richer later, once markets realize the consequences of TAI.

That second idea seems like the opposite of consumption smoothing??  Maybe it's worthwhile because I would become rich enough that the extra volatility is worth it to me?  But what's the point of being rich for just a short time before I die, or of being rich for just a short time before TAI-induced utopia makes everyone fantastically rich anyways?

I also do not find it plausible that a vision of impending TAI-induced utopia, an acceleration of technological progress and human flourishing even more significant than the industrial revolution, would... send stock prices plummeting?  I am not an economist, but if someone told me that humanity's future was totally assured, I feel like my discount rate would go down rather than up, and I would care about the future much more rather than less, and I would consequently want to invest as much as possible now, so that I could have more influence over the long and wondrous future ahead of me.  You could make an analogy here between stable utopia and assured personal longevity (perhaps to hundreds of years of age), similar to the one you make between human extinction and personal death.  The promise of a stable utopian future (or personal longevity) seems like it should lead to the opposite of the short-term behavior seen in the near-term-extinction (or personal death) scenario.  But your post says that these two futures end up in the same place as far as the discount rate is concerned?

To quote a Peter Thiel joke that didn't make it into my post about investing under anthropic shadow, "Certainly if we could just live to all be 100, that would be quite a transformation. There is good news and bad news. The bad news is: If you don’t believe in the good news, you’re not saving enough for retirement."

Id suggest investing in companies correlated with AI progress not shorting bonds. Some of the best investments are private but you can still buy stocks like MSFT, TSM, Samsung and ASML. This seems like a much better way to bet on AI progress. 

Also if you expect aligned AI you should probably consume much less? Inequality might get locked in and returns to capital might skyrocket. Bad time to be a big spender. If we get unaligned AI and the world ends or aligned AI paradise so be it.

I am surprised that critical commenters have focused on the irrationality or inadequacy of financial markets, rather than what feels like the more obvious point:

Unaligned AI need not imply extinction, and aligned AI need not imply 30% growth. Financial markets can be inconsistent with these implications without being inconsistent with Big Deal AI.

On unaligned AI: eyeball the reviews of the Carlsmith report. Looks like average P(xrisk | misalignment) ~= 45% among reviewers.

On aligned AI: 30% growth is crazy high! The authors are unwilling to make their claims for less-crazy growth figures:

Great post! This is a fascinating argument and makes me think quite a bit.

A couple of tests pitting EMH against mortality risk since nuclear war would have similar ramifications (with a similar amount of uncertainty given that it occurs!) 

• What happened to bond prices during the Cuban missile crisis? How complete did the public think nuclear war would annihilate society? Was nuclear winter a thing?  You can make the argument that the public thought P(death) per year rose from ~2% to 20%-50%?  What would this theory suggest happen to bond prices? 
•   What  about following the collapse of Soviet Union? After settling down, this arguably lowered the probability of nuclear war by 1%/year.  That gives us (1- pbinom(0, 30, .01)) = 26% reduction of existential risk over 30 years. Should still be measurable in long-term treasury rates.

Seems relevant (link) :

(agree/disagree with this comment to agree/disagree with the tweets below)

Edit: Yudkowsky commented on the post, consider replying to him directly

If there's no strategy to profitably bet on long-term real interest rates increasing, you can't infer timelines from real interest rates. I think the investment strategies outlined in this post don't work, and I don't know if there's a strategy that works.

I want to caution against the specific trading strategies suggested in this post:

• Shorting LTPZ will cost you 7% per year in borrow fees.
• The short ETFs TBF, TBT, and TTT have 1% fees. In addition, short or levered ETFs rebalance frequently, and are known to have large hidden fees of 1-4% per year. Short ETFs are not suitable for long-term holding unless you're excited about contributing towards Jane Street salaries.

Even without these fees, the investments would not be a slam dunk compared to buying stocks, as pointed out here. But the fees really make this look very unattractive.

I want to suggest a bunch of caution against shorting bonds (or tips).

• The 30yr yield is 3.5%, so you make -3.5% per year from that.
• You earn the cash rate on the capital freed up from the shorts, which is 3.8% in interactive brokers.
• If you're right that the real interest rate will rise 2% over 20 years, and average duration is 20 years, then you make +40% over 20 years – roughly 2% per year.
• If you buy an ETF, maybe you lose 0.4% in fees.

So you end up with a +1.9% expected return per year.

This would have a third of the volatility of stocks, so you could leverage it several times, but then you'd need to pay the margin cost of ~4%.

So it doesn't seem like an amazing trade in terms of expected returns (if I've estimated this correctly.

It gets worse if you consider correlations – if we go into a recession, yields might fall 1-2%, which would mean you lose 20-40%, and you make those losses at the worst possible time – when everything else is going down.

In addition, a neutral portfolio is something like 50% equity, 20% real assets and 30% bonds, so that should be our prior, and then you'd want to make a bayesian update away from there based on your inside view.

In effect, in your portfolio optimizer, you could set the expected returns of long bonds  to be say 1.5% rather than 3.5%.  My guess is that would spit out having say 0-10% bonds rather than 30%, but not actively shorting them. 

Tldr my guess is that most investors (if they believe the thesis) should just underweight bonds rather than actively short them.

I'd be very keen to hear more comments on this.

I think this post contains many errors/issues (especially for a post with >300 karma). Many have been pointed out by others, but I think at least several still remain unmentioned. I only have time/motivation to point out one (chosen for being relatively easy to show concisely):

Using the 3x levered TTT with duration of 18 years, a 3 percentage point rise in rates would imply a mouth-watering cumulative return of 162%.

Levered ETFs exhibit path dependency, or "volatility drag", because they reset their leverage daily, which means you can't calculate the return without knowing what the interest rate does in between the 3% rise. TTT's website acknowledges this with a very prominent disclaimer:

Important Considerations

This short ProShares ETF seeks a return that is -3x the return of its underlying benchmark (target) for a single day, as measured from one NAV calculation to the next.

Due to the compounding of daily returns, holding periods of greater than one day can result in returns that are significantly different than the target return, and ProShares' returns over periods other than one day will likely differ in amount and possibly direction from the target return for the same period. These effects may be more pronounced in funds with larger or inverse multiples and in funds with volatile benchmarks."

You can also compare 1 and 2 and note that from Jan 1, 2019 to Jan 1, 2023, the 20-year treasury rate went up ~1%, but TTT is down ~20% instead of up (ETA: and has paid negligible dividends).

A related point: The US stock market has averaged 10% annual returns over a century. If your style of reasoning worked, we should instead buy a 3x levered S&P 500 ETF, get 30% return per year, compounding to 1278% return over a decade, handily beating out 162%.

Hi All, 

I'm an economist who's been thinking about some related issues. 

I agree with the article that increases in anticipated capital productivity (or decreases in anticipated longevity) should tend to increase global interest rates. However, the blog post ignores a giant secular trend that could be driving recent (last few decade) low interest rates – a global "savings glut” due to the growth of a giant, thrifty, Asian upper and middle class. This group has a huge hunger for investment vehicles, and has been driving down global interest rates. 

In my recent global automation simulation paper, which models in detail demographic and productivity trends across the regions of the world, we find that interest rates will continue to decline over the next thirty years if automation (defined as capital-biased technical change) continues at its historical rate. If automation were to proceed at a rate 5x its historical rate, we calculate that world interest rates is ~still~ projected to decline, and be lower in 2050 than today. That said, if automation were to occur x10 faster than its historical rate moving forward, we would anticipate interest rates to be almost 50% higher than they are today in 2050. 

In the language of the blog post, I’d argue that *rho*, the time discount rate of the representative global saver, is in essence getting smaller as more of world income is going to thrifty middle-aged Asians. A fast enough rate of automation can overcome this headwind, but there is a large headwind to be overcome. 

Paper here: https://www.nber.org/papers/w29220 see section “4.3.3 Faster Rates of Automation”

I haven’t run the simulation to see what the effect would be today of a giant anticipated increase in automation in 30 years, or of everyone dying with certainty in 30 years, but those are certainly possible in our framework, and something we could try if anyone thinks it would be interesting.  

Thank you for the post! I'm very interested to see more work on this topic.

I feel a little bit unsure about the focus on the bonds – would be very curious to hear any reflections on the below.

As you say, if real interest rates rise, that should affect all assets with positive duration.

Perhaps then the net effect of having the view that real interest rates will rise is just that you should reduce overall portfolio duration. A 60:40 portfolio has an effective duration of ~40 years, where most of that duration comes from equities. Perhaps someone who believes this should target, say, a 20 year average duration instead (through whatever means seems least costly, which could mean holding fewer equities).

Perhaps equivalently, if real interest rates are going to rise, then all financial assets are currently overpriced, so  maybe the effect would be holding fewer financial assets in general, and holding more cash / spending more. 

My understanding is that an important part of the reasoning for a focus on avoiding bonds is that an increase in GDP growth driven by AI is clearly negative for bonds, but has an ambiguous effect on equities (plus commodities and real estate), so overall you should hold more equities (/growth assets) and less bonds. Is that right?

That makes sense to me, but then I still feel unsure about, having tilted towards equities, whether your overall exposure should be higher or lower.

(And tilting towards equities will increase the effective duration of your portfolio, making an increase in real interest rates worse for you all else equal.)

If we use the merton's share to estimate optimal exposure, that depends on the difference  between the expected return of the asset and the expected real interest rate over your horizon. Perhaps with equities you might expect both returns and the interest rate to rise by 3%, which would cancel out, and you end up with the same exposure. But with bonds only the interest rate will rise, so you end up with much lower exposure (potentially negative exposure if your expected interest rate is higher than the expected returns). Is that basically the reasoning?

Great post

Regarding the second point about how EAs (or anyone else) might exploit an inefficiency in this space, I think it's tricky just because the amount of other risks that inform the pricing of long-dated bonds. Many of these (climate, demographics, geopolitics, populism etc..) could wipe out any short (or especially leveraged short) position before TAI is realised.

As noted in my other comment I expect for someone with high-conviction views on short TAI timelines there are bets that are:

• Much higher in expected returns
• Less capital intensive
• Less susceptible to other risks

Examples of these bets are broadly discussed elsewhere but often are related to long/short equity bets on  disrupting/disrupted companies and companies part of the supply chain (semiconductors design/fab/tooling, datacentre, data aggregators, communications etc..)

I think perhaps at best short long-dated bonds could form part of a short-timelines TAI bet in order to hedge against long positions elsewhere/maintain neutrality against other factors rather than the core position. It feels likely there are considerably better options for someone taking such a bet (as you allude to in the opportunities for future work)

By the way, someone wrote this Google doc in 2019 on "Stock Market prediction of transformative technology". I haven't taken a look at it in years, and neither has the author, so understandably enough, they're asking to remain nameless to avoid possible embarrassment. But hopefully it's at least somewhat relevant, in case anyone's interested.

This is one of my favourite forum posts ever. A pleasure to read. Congrats to the three of you.

Whenever I see charts like this in a financial context I twitch. We have 30 years of data for UK real rates, less for other issuers. There are ~2 non-overlapping UK data points on your second chart where I can count at least 15(?) data points?

I think the fundamental assumption that aligned AGI would cause dramatic economic growth are simply wrong. Like super duper wrong.

It’s important to differentiate between AGI and both SuperIntelligence and God. Most EAs are thinking about an omnipotent and omniscient being.. not AGI.

Is being a high IQ person in Russia or India that useful? Why then do their smart people migrate to the US? Because the US can use that intelligence (Sundar Pichai) while at home they may become a mid level bureaucrat (Sundar Pichai’s dad).

Say you create an AGI, comes out and tells you, “I’ve solved fusion, let’s build plants” … you go “Cool bro, need He3? It’s on the moon, costs too much to go there, figure out how to get there cheaper first”

So there are physical, logistical, real limits to what can be achieved in the physical world with lots of intelligence. Also economic. Because there won’t just be one AGI, there are likely to be multiple, at the very least latency means one on Earth and one on Mars, and I suspect latency will dictate multiple on Earth.

So we are likely to find other bottlenecks besides intelligence alone limit economic growth, and that these will have to be figured out. And also AGI again not being omniscient, will take time to figure things out. Like tell it to solve aging and it comes back and asks for 100 years of compute time.. and it’s just not feasible.

This is what weirds me out about Yud and other EAs.. it’s clearly a religious belief that we are creating an omnipotent being.. rather than a perfectly ordinary intelligent creature that is still limited by the availability of data, compute, data storage, network latency etc.

Regarding the first point about the extent to which we should update timelines based on the fact the bond market is not pricing in short timelines for TAI; my prior is that in general the fixed income (bonds) markets are fairly efficient and are more sophisticated/efficient than equity markets. This leads me to initially believe we likely should update based on this/consider it more strongly than bullish equity sentiment towards some AI themes.

However on the flipside I think the size of this market does mean it can retain inefficiencies around subtle themes for longer.  I think of this as a form of Expecting Short Inferential Distances - there are a lot of inferential reasoning steps around TAI, scaling, take-off  etc.. which make it slower for conviction to spread when compared to something like demographic shifts which have a much more straightforward causality. This is relevant because to move government bond markets requires people to take this bet with a huge amount of assets as this is a very capital-intensive trade with a lot of exposure to other uncorrelated risks/conflating variable (climate, demographics, geopolitics, populism etc..). The reason I think it may be unlikely that many people are making this bet is related to this:

An analysis of the most capital-efficient way to bet on short AI timelines and the possible expected returns (“the greatest trade of all time”).

I suspect there are far more highly levered bets that market participants with a high-conviction belief in short TAI timelines could take, potentially diluting the impact on lower beta instruments (like bonds). For example I expect even being long fairly broad equity markets might outperform this bet and much more targeted bets (especially if they could be hedged against other risks, bringing them closer to  a 'pure' TAI bet) could be expected to return many multiples of the short-US30Y trade.

If the amount of money being managed by those with high conviction TAI views is 'small' (<<$100bn) then I expect there are many  more favourable inefficiencies/price dislocations for them to exploit and not a sufficient mass of 'smart TAI' money to spill over into long-dated bonds.

Here's another way of putting things, that I'll post here for reference:

Suppose I think Google is undervalued, because it is going to have a $1T dividend in 2030, and the market doesn't realize this.

1. I buy Google today at some cheap price.

2. Possibility 1: before 2030, the market "corrects" and realizes that it was undervaluing Google. The stock price rises, and I receive capital gains.

3. Possibility 2: the market does not "correct" before 2030. I still get the big dividend in 2030, and was able to get it for a cheap price in 2023.

---

The above seems exactly analogous to the case with existential risk.

---

Suppose I think bonds are overvalued, because in 2030 the world is going to blow up.

1. I short real rates today.

2. Possibility 1: before 2030, the market "corrects" and realizes that it was overvaluing bonds. Rates rise, and I receive capital gains.

3. Possibility 2: the market does not "correct" before 2030. I still was able to take out a cheap loan in 2023 (i.e. by selling short bonds), and don't have to pay it off in 2030 when the world ends.

Tyler Cowen on the effect of AGI on real rates:

In standard models, a big dose of AI boosts productivity, which in turn boosts the return on capital, which then raises real interest rates.

I am less convinced.  For one thing, I believe most of the gains from truly fundamental innovations are not captured by capital.  Was Gutenberg a billionaire?  The more fundamental the innovation, the more the import of the core idea can spread to many things and to many sectors.

Furthermore, over the centuries real rates of return seem to be falling, even though there are some high productivity eras, such as the 1920s, during that time.  The long-run secular trend might overwhelm the temporary productivity blips, I simply do not know.

I do think AI is likely to increase the variance of relative prices.  Observers disagree where the major impacts will be felt, but possibly some prices will fall a great deal — tutoring and medical diagnosis? — and other prices will not.  Furthermore, only some individuals will enjoy those relative price declines, as many may remain skittish about AI for quite a few years, possibly an entire generation.

That heterogeneity and lack of stasis will make it harder to infer real interest rates from observed nominal interest rates.  Converting nominal to real variables is easiest under conditions of relative stasis, but that is exactly what AI is likely to disrupt.  Furthermore, real inflation rates, and thus real interest rates, across different individuals, are likely to increase in their variance.

Overall, that blurring of nominal and real will make the Fed’s job harder.  And it will be harder for Treasury to forecast what will be “forthcoming real interest rates.”

What effect do you think an AI boom would have on inflation?

It seems like it would be deflationary, since it would drive down the cost of goods and labour, though it might cause inflation in finite resources like commodities and land, so perhaps the net effect could go either way?

(I partly ask because a common framework in investing for thinking about the what drives asset prices is to break it into growth shocks, inflation shocks, changes in investor risk appetite and changes in interest rate policy. If AI will cause a growth shock and deflation shock, then normally that would be seen as positive for equities,  ambiguous for real assets and nominal bonds, and negative for TIPs.)

Thanks for writing this!  Great to see this written out finally.

Briefly, to reiterate / expand on a point made by a few other comments: I think the title is somewhat misleading, because it conflates expecting aligned AGI with expecting high growth. People could be expecting aligned AGI but (correctly or incorrectly) not expecting it to dramatically raise the growth rate.

This divergence in expectations isn’t just a technical possibility; a survey of economists attending the NBER conference on the economics of AI last year revealed that most of them do not expect AGI, when it arrives, to dramatically raise the growth rate. The survey should be out in a few weeks, and I’ll try to remember to link to it here when it is.

One possible explanation is an expectation of massive deflation (perhaps due to AI-caused decreases in production costs) which the structure of Treasury Inflation Protected Securities (TIPS) and other inflation-linked government bonds — the source of your real interest rate data — doesn't account for.

While TIPS adjust the principal (and corresponding coupons) up and down over time according to changes in the consumer price index, you ALWAYS get at least the initial principal back at maturity. Typical "yield" calculations, however, are based on the assumption that you get your inflation-adjusted principal back (which you do if inflation was positive over its term, as it usually would be historically).

This means that iff there's net deflation over its term, the "yield" underestimates your real rate of return with TIPS by the amount of that deflation.

From an altruistic point of view, your money can probably do a lot more good in worlds with longer timelines. During an explosive growth period humanity will be so rich that they will likely be fine without our help, whereas if there's a long AI winter there will be a lot of people who still need bednets, protection from biological xrisks, and other philanthropic support. Furthermore in the long-timeline worlds there's a much better chance that your money can actually make a difference in solving AI alignment before AGI is eventually developed. So if anything I think the appropriate altruistic investment approach is the opposite of what this post suggests; even if you think that timelines will be short you should bet that they will be long.

From a personal point of view, it's likewise true that marginal dollars are much more useful to you during an AI winter than during an explosive growth period (when everyone will be pretty rich anyway), so you should make trades that move money from short-timeline futures to long-timeline ones. But I do agree with the post that short timelines should increase your propensity to consume today. (The "borrow today" proposal is impractical since nobody will actually lend you significant amounts of money unsecured, but you might want to spend down savings faster than you otherwise would.) (Edit: Though the amount it makes sense to consumption-shift is smaller than you might expect.)

Borrowing money if short timelines seems reasonable but, as others have said, I'm not at all convinced that betting on long-term interest rates is the right move. In part for this reason, I don't think we should read financial markets as asserting much at all about AI timelines. A couple of more specific points:

Remember: if real interest rates are wrong, all financial assets are mispriced. If real interest rates “should” rise three percentage points or more, that is easily hundreds of billions of dollars worth of revaluations. It is unlikely that sharp market participants are leaving billions of dollars on the table.

(a) The trade you're suggesting could take decades to pay off, and in the meantime might incur significant drawdown. It's not at all clear that this would be a prudent use of capital for 'sharp money'.

(b) Even if we suppose that sharps want to bet on this, that bet would be a fraction of their capital, which in turn is a fraction of the total capital in financial markets. If all of the world's financial assets are mispriced, as you say, why should we expect this to make a dent?

There are notable examples of markets seeming to be eerily good at forecasting hard-to-anticipate events:

Setting aside that the examples given are inapposite^[1], surely there are plenty in both directions? To pick just one notable counterexample: The S&P 500 broke new all-time highs in mid-Feb 2020, only to crash 32% the following month, then rise 70% over the following year. So markets did a very poor job of forecasting COVID, as well as the subsequent response, on a time horizon of just a few months!

1. ^{^}

   Both of these were in rapid response to recent major events (albeit ahead of common wisdom), as opposed to an abstract prediction years in the future

this is delightful. Such a good post . Great stuff, fellas.

Very nice post! I'm not sure if you have looked into this but market aside, given that people in EA believe in claims about AI risk and short timelines, are charities in EA spending money in proportion to the seriousness the EA community seems to take short AI timelines and AI x-risk? For example you cited some reports from Open Philanthropy like Bio Anchors where you extracted some of the probabilities used in your calculation. Do you think Open Phil's spending is in line with the expected timelines suggested by Bio Anchors?

I just ran across this post and I apologize that I haven't been through all the comments. I think, however, that the issue you are encountering here is a liquidity trap. 

So, the actual relationship you have is that nominal interest rates tend to track NGDP growth. Now since  real GDP growth is just NGDP growth minus inflation and real interest rates, are nominal interest rates minus inflation it stands to reason that real interest rates tend to track real GDP growth.

However, the problem is "How are you going to actually get 32% (30% real plus a Fed target of 2%) NGDP growth? Is the Fed supposed to commit to printing that much money? It can't possibly. 

So what you are really going to get is 5% nominal rates and 25% deflation. The deflation will act as a forced-savings mechanism because borrowers are liquidity-constrained and for savers the return on cash is high. Both groups will have less cash than they want which will cause them to reduce consumption which will dampen demand. The reduction in demand will then free up resources for capital accumulation.

If you stop and think about this, this is the world many technologists envision. The price of most things collapses, and the investment in compute explodes.  This all happens, though, without a huge upsurge in interest rates. 

There is actually more complexity in the transition to the long-term equilibrium but a basic liquidity trap is enough to explain why expectations of future interest rates do not rise despite the promise of rising GDP.