Updated: 12/22/24

Credences

I believe there is a 66% chance that we experience an AI-related catastrophe with 1M-8.1B (a) deaths or (b) people severely negatively impacted in the next 1-4 years (2025-2028). The 8.1B implies 99.9% of people, not 100%. Most of my probability mass is on a major catastrophe with 10M-5B lives lost or severely negatively impacted in the short-run.

“Severely negatively impacted” = a rating of 7+ on a subjective 1-10 scale. Some examples of what a survey of random respondents might call a 7+ on this scale: AI-augmented pandemic, massive job losses, city/country collapse, wide-scale cyberattack leading to restricted or modified internet use, etc. 

I put roughly equal credence in (a) and (b).

My rough timeline is:

• 2025: 20% cumulative
• 2026: 40% cumulative
• 2027: 75% cumulative
• 2028: 100% cumulative

I believe there is a 75% chance that humanity kills or displaces itself due to AI in the next 1-15 years (2025-2040).

• 2025: 10% cumulative
• 2030: 60% cumulative
• 2035: 80% cumulative
• 2040: 100% cumulative

I’m much more optimistic if we immediately create effective international governance and enact a nearly universal global pause. This would lower my credences by about 50%, so a 33% chance for short-term AI-related catastrophe and 40% chance for short- to mid-term extinction or displacement. 

The Challenge

Our collective challenge as a species is to develop a mechanism that somehow effectively prevents a powerful unaligned ASI from ever being created. This must work (a) 100% of the time in (b) preventing 100% of people from (c) ever succeeding.

Unfortunately, many rogue actors are set on omnicide and will try to use AI or ASI to achieve their goals. Many more altruistically-minded innovators are convinced that the risk of extinction is worth it for the chance of “talking to God” or “solving all the world’s problems” or “creating utopia” (in their image). They are almost equally insane, but the latter also sometimes has billions of dollars to spend on their insanity.

In the near future, anyone with a laptop will be able to cause humanity’s extinction. Rogue actors or reckless altruists will be able to continue frontier AI development with impunity, unless humanity decides to collectively stop them. Outstandingly good governance and a public deeply committed to existential safety will help, but will not likely be enough. 

Therefore, our overriding goal right now ought to be a global pause so today’s and tomorrow’s Ted Kaczynski AI developers operating in off-grid locations cannot be successful. The closer major labs push the frontier, the harder it will be to stop these lone actors from causing catastrophe. Ironically though, it’s more likely the labs themselves cause our extinction rather than a lone actor misusing their work. “The road to hell is paved with good intentions” seems to be true here. We are already far, far past any sensible point for pausing. 

If we can’t enact a pause, then we must somehow solve the alignment problem. But after 20+ years of work on that, leading experts have claimed it’s highly unlikely. I never thought it was the most probable path forward myself. I thought it might be easier to unify humanity around cautious and wise technological progress that benefits all. I still hold some hope for that path forward.

All that said, if we do make it past this incredibly dangerous acute period, I think we or our descendants will likely achieve deep existential security and incredibly beautiful, utopian futures. 

Context

It’s important to note that I’ve been making predictions for decades, but not rigorously. I’ve been more right than wrong when I’ve written them down. But no one can predict the future extraordinarily well, least of all me. 

For artificial superintelligence, in ~1998 I loosely predicted it for ~2030. In 2008, I firmly recorded it as for 2028. Originally I thought it was roughly 60/40% that artificial superintelligence would help humanity evolve versus destroy us. After I discovered instrumental convergence and the orthogonality thesis about a decade ago my prediction shifted to 80/20% in favor of destruction.

I’ve known for ~26 years that AI would be the most important invention of humanity. I have been quietly mystified since then why others didn’t also come to this conclusion. I actually spent tens of thousands of hours trying to teach others to see this and other important truths about reality more clearly. I mostly failed. 

Change Your or My Mind

If anyone working on frontier AI development wants me to help them explore their beliefs around the rational assessment of risks and rewards, please give me a call. Historically I have charged for this service–it’s philosophical counseling/executive coaching–but I will waive fees here. 

Likewise, if anyone with a credible background wanted to share their reasons for optimism, I would love to speak with you. I will pay $1,234 USD to anyone that moves my credences up or down by >20%. Credibility usually means:

• Has evidence of understanding social and technological development (e.g., PhD in related field, startup that accurately anticipated a major change, etc.) 
• Has evidence that they have processed most of their subconscious defense mechanisms (e.g., denial, repression, rationalization, etc.)
• Has evidence of making calibrated and accurate predictions over a 10+ period