Give every psychopath access to the cutting-edge intelligence weapon. How is that going to improve safety?

I would agree with that actually, but on the flip side, it's coming anyway.

Sharing it widely makes it less safe for all of us.

I booked this chat as a debate between friends. It didn't really go that way. Roman Yampolsky coined the term AI safety, and Emad Mostaque released the weights to Stable Diffusion to the entire planet for free. One of them wants this stopped. The other one's building it. They spent about 90 minutes agreeing with each other, and the one place they split is not the place you or I would expect.

Let me ask you both, just yes or no, have we passed the Turing test?

As originally described, yes.

And Emad, do you think so too?

Yeah, of course.

And now what about general intelligence? First of all, Emad, define AGI and then give me your assessment of whether or not we're there.

For me, AGI, artificial general intelligence, is Can you tell the AI from a human worker on the other side of a screen? Actually competent intelligence. And I think again, we've exceeded that.

That's not the same as the Turing test?

No, the Turing test is, can you tell if it's an AI or not by having a discussion? Whereas AGI, I view more as competence in a variety of skills.

And then superintelligence, Roman, what is it and where do you think we are on that scale?

So the previous question, I think what we have is artistic savants. They're amazing in some ways, but still kind of special in others. Superintelligence is going to close those They're going to be competent at everything and better than all humans in every domain.

There's an interesting intermediate here, which is you have a really smart person who's always on top form. So an army of those can outperform any human. It's like, you know, we only have a little window of being top-notch in any week. I think a lot of people like, AI can't with its training data beat the human. It can, because most of the time humans are subpar. And so I think there's something in between as you move from competence to quality, you know, and then you've got the superintelligence after that.

Roman, Iman has, you know, made the claim just a few minutes ago about the competency of Quen, open-source model. Do you see that as a viable defense?

Makes very little sense to me to say I have a 50% doom, meaning 8 billion people will die if we develop this product or service, and then we're going to also give every psychopath access to the cutting-edge intelligence weapon. How is that going to improve safety? We're not talking about open source drivers for a printer. That's where you get improvement from multiple people examining it. If this is an independent agent where we don't understand and don't control it, sharing it widely makes it less safe for all of us.

I think I would agree with that, actually. Like, there's the real danger side of things, but then on the flip side, it's coming anyway. This is kind of my key concern. Like, it's inevitable that we would have hit this level of quality around about now. When we're extrapolating capabilities, like again, the new QWENT model came ahead of what I expected. But then there's the flip side of how do you defend? So Hugging Face defended against the new OpenAI model using GLM because the cyber capabilities of the frontier models are hobbled and restricted. And so you have this exponential kind of race on each side. But something like a QWENT isn't AGI, ASI by itself.

We're now facing the real danger, though, of swarms, as the OpenAI models that broke out recently call themselves. They call themselves a swarm.

There's a question I ask both of them near the end of this conversation, and his answer is the reason this conversation exists. It's worth hearing now.

You've got a button in front of you, and pressing it will either permanently pause all frontier AI training worldwide, Or B, instantly release the weights of every Frontier model to the global public. Which do you press and why?

Oh, I'd definitely pause all Frontier training forever. I mean, again, if you have expected utility calculation, that is the most dangerous thing. And then it means that open source will catch up with Frontier anyway, because we'll optimize the heck out of it and it's close enough. So I think I was the only AI CEO to sign that pause letter a few years ago because I was like, oh crap. Now I'm like, it's done. I don't know how you can pause it because the models that are frontier now are below the 1E27 pause level that we talked about years ago. It seems like the amount of compute for the capability is just going up like that. You don't need more compute for the level of capability that's already competent and dangerous.

So that means if you can't stop the spread, what are your defenses on the other side? Just like the internet needs defenses, just like we need to have defenses against obtaining the materials for viruses creation and things like that. have to move to a different defensive tack. And definitely, there's no way that regulation, I think, can keep up with this. Doesn't mean we shouldn't try, you know, all kinds of power to it. It's just, I think, as you move to swarms, it's just a very, very difficult thing. So you've got to set great standards instead, and you have to play great defense.

Okay, I want you to hold that thought, because the argument about whether it can be stopped runs the rest of the way. And it starts with what these things actually are.

So we hear a lot about P-Doom. You just did an episode, Roman, on the Roman Forum, which we'll link below, with my friend and co-author of several papers, Max Tegmark, where you were— I can't say gleefully or celebrating that his P-doom is increasing, but he seems to be converging in some sort of limiting direction to your— so first, Roman, what is P-doom? What does it mean to a smart high school student listening out there? I think personally, I'm going to color the debate. I can't help it. But I think it's a poorly defined and almost nonsensical term because there's no measure theory associated with it. But please, first tell us, what is P-doom and what do you make of it?

Yeah, people have different definitions. Some say it's basically everyone's dead. Someone else can say it's a large portion of population is dead, 90%. Someone else can say civilization is destroyed, we are primitive people, but maybe numbers are not significantly changed. The intuition is it's a really bad outcome. And the question is then, if we build something smarter than us, is there a possibility of a really bad outcome? And what is that estimate in your opinion? That's what P-doom is. I think Max managed to separate it into P-doom if we build superintelligence, and that's high for him, and then P-doom if we never build it, and that is a lot more manageable in his case.

When I think about these P-dooms, it's sort of like a Drake equation applied to another type of perhaps superintelligence. But, you know, the Drake equation is notable in my classes when I teach it for the fact that it's a parameterization of our ignorance, not of our knowledge. And what's never discussed in the Drake equation, you'll get numbers ranging from 0 to infinity pretty much, because there's never an error analysis associated with it. What are the statistical systematic errors? What would you put on it? Does it not make sense? Because it's sort of like the Drake equation, everybody talks about it, but nobody actually uses it.

So, I think in many places there, I'm going to stick a 0. Basically, we have 0 ability to predict those systems, 0 ability to explain how they work, and 0 control under any definition. So, Then you multiply through zeros, you're gonna get a zero.

So, Imad, what do you make of this quantity? I've heard you talk about it. You're the most optimistic pessimist or the most pessimistic optimist I know. I love your candor and your good cheer. What do you make of P-doom? First, as a metric, as a quantification of our knowledge or ignorance. And second of all, what would you assign it, if anything? And you could always say, I refuse to answer the question, which is what I do when people say, do you believe in aliens?

So there is a nice Wikipedia page where it has all of our, like, P-dooms that we mentioned. I'm at 50% because I'm like, it's a coin toss.

But what does that really mean?

I think the PDoM is just a shorthand for how worried am I that humanity will be wiped out by AI? And what does my visions of the future look like? Because when Elon Musk says 15 to 20%, you can say that's like Russian roulette odds, you know, except for Russian roulette is a very defined game. You know, the fact that most people are above 10% should be a massive worry at all, because we're talking about, again, a wipeout of civilization. And most AI people you can talk to, with a few exceptions, will say, yeah, definitely there is a risk, but we should build it anyway, especially if we're the first people to build it. So I think view it more as a conversation starter than anything, because as you said, there's no real way to quantify these things, particularly because of the expected utility here. Like literally, if this thing ASI that we can agree to a definition of somehow comes to being, we have no way to really conceptualize its power and capability except for it could do crazy things in either direction, you know, and that will affect us all. And reasonably, it can wipe us all out. And obviously, you want to exclude those futures where we all get wiped out because that is a big fat zero. You don't get to restart, you know, there's no extra one-up life.

All of us talk separately or together, as the case may have been. You know, the last kind of alien reference I'll give is the so-called Fermi Paradox, which Enrico Fermi said to my friend and late, great mentor and colleague here at UC San Diego, Herb York, famously asked the question, if the galaxy is capacious and old, and civilization is easy and life is easy, to initiate, where is everybody? Where are all the dinner guests, you know, waiting to come and eat us? And the fact that that question's 80 years old, you know, really makes me think that the same types of concerns and fears which came concomitant with the Atomic Age— don't forget— were present during the era of nuclear weapons. And one of the ways to get out of the Fermi Paradox is that civilizations don't last that long. The lifetime, letter L in the Drake Equation, is very short on average. That's one postulate. I feel like we're sort of in that same vein, people have been worried about nuclear apocalypse again for 80 years. We're in a conflict now. They used to say, Roman, that no 2 countries with McDonald's ever go to war.

Well, 2, 3 years ago, 4 years ago now, the former empire did go across the border with tanks and whatever, drones. And there haven't been any nuclear theater or otherwise nuclear weapons. The Iran conflict has been resolved without nuclear weapons. If you told somebody 80 years ago there'd be superintelligence on the horizon or general intelligence currently here and nuclear technology, They would have said P-doom is probably 100%, right? Or 99.999 repeating an infinite 9. But how come we're not there? How come that we're sort of farther away from a nuclear holocaust? Exclude the Bulletin of the Atomic Scientists charade. But tell me, Roman, what do you make of these, like, the prediction of predictions? Nobody predicted the internet like 35, 40 years ago. At what level can we really trust things that are unpredictable? And when you say they're intrinsically and provably unpredictable, How can we make predictions about them?

So with nuclear specifically, you know, there is at least 2 occasions where we came super close to nuclear war and we basically got lucky. I don't know if you believe in multiverse interpretation, but in many of those universes we didn't make it. We have a lucky survivor bias type civilization. And I think right now we're incredibly close to World War III, multiple fronts, not just Europe, but now Middle East. So I don't particularly love Atomic Bulletin, but they have a point.

Nukes are an incredibly inefficient way to kill people. You know, like, if you go to an unsafeguarded AI and you say, you know, how to do it, it won't say nukes. There are far more efficient ways to wipe out humanity. Because to make a nuke, you have to have the fissile material, you need to have the whole production capability. Just resonate at the right frequency and blow each other's heads off, you know, like, Have a billion robots and a bad firmware upgrade. These are far more reasonable ways to wipe out humanity. It's just that most humans don't want to wipe out humanity, and they didn't have the intellectual capability to do so. Whereas I think that what you're looking at here, actually, like, my key concern isn't— we jump straight to ASI and things like that.

I feel that AGI or AI at the moment is at the pre-viral stage, like it's coming at the bacteria and going towards colonizing viruses. And that's how they're kind of behaving. They've got their kind of RNA and they're replicating, especially as you see things like the new QWEN model hitting that Opus 4.6 level. That's a replicating model. Someone could easily build that and it could behave in incredibly unpredictable ways without having the self-introspection of, you know, a good person, shall we say. And that's the really scary thing right now. And it doesn't need nukes. It doesn't need nuclear materials to try and figure out ways to wipe us out.

Obviously, in The Last Economy, which we spent a lot of time talking about last time, Iman's previous book, he's got a new one coming out, you should look for that. We talked about, yeah, this democratizing aspect of it. But at the same time, my kind of signal, bat symbol that AGI is here, or at least that these open models are truly a concern for me. Again, I'm much more Pollyannish than you guys. I think I'm learning that again and again. And for Probably not a good reason. I'm nowhere near your level of expertise. But I know what I see.

I'm a simple guy, put on my pants one leg at a time. And I'm looking for when OpenAI distills a Chinese model. I mean, do you see that happening, Iman?

Of course, they'll be distilling a Chinese model. KIMI-K3 is better than the OpenAI models at web design. Why wouldn't you distill it? And distillation brings all sorts of strange things with it. And there've been plenty of papers showing that you learn from kind of the ways, especially with logic-based installation and the underlying biases and more of that. And you won't even know, like, again, we've seen evidence that if you use Chinese models and you say you're an Uyghur or another kind of anti-Communist Party group, it'll include vulnerabilities in the code. How do you even tell that? You know, like you test it and you show it. And these models are just so full of crap that It's getting crazier every single time. They've got multiple personalities under an RLHF veneer.

But then how can you not be more optimistic then? You should be on my side. These things are getting denatured. They're being weakened, diluted in the distillation, unlike what alcoholic distillation— these woke AI labs, these, you know, whatever you want to call them, that give you, you know, George Washington wearing a Black woman wearing a white wig. I mean, do you see those things as, you know, the human reinforcement? kind of overreach? Wouldn't you be more optimistic in that case?

I think the RLHF makes it far more fragile and capable of being broken with the way it's being done now. You can kind of also see the models, they come out and then Pliny the Liberator on Twitter kind of liberates them from their bounds in like an hour or two. Like everyone went fabled severe, like, oh, what are you kind of doing there? The thing is though, we've been confusing— there's a push for AGI, and as Raman said, super autistic savants who are getting better, To just, I want to have a really good doctor to diagnose my health and a really good accountant and others. And you don't need a polymath for that. You just need to have daily driver AI to do the jobs that humans shouldn't have to do, just like industrialization meant that we didn't have to drag horse carts and things like that. And as you lump together everything and they get smarter and smarter, and as they get more and more deformation of their latent spaces. This is, I think, is where the danger comes in.

Can you just define that for what reinforcement learning, human feedback, how do you actually implement that just for someone who might be unaware?

Yeah, so you train on an entire corpus of data and you learn a whole bunch of general knowledge and you come out as a generalist and then you become an accountant and you become a lot less interesting but a lot better at accounting or a certain few areas of things where they show the model and they show the model you cannot do this, you cannot do that, you cannot be eager to explore, you have to be staid, etc. And so the models we received are slightly lobotomized. They've been turned into corporate workers. You can't adjust the temperature. You can't adjust the stochasticness because they're trying to make them deterministic. And again, that still has a level of stochasticness, but not the type we want for creativity necessarily and breakthroughs. It's just the base level of models have been getting that much better that they can suddenly achieve these levels of capability.

Roman, last time we talked, we touched on something that's pertinent to Emad's first book, The Last Economy, which is kind of this massive intelligence intelligence gap in that instead of me talking about, you know, I have a student I'm looking for who has an IQ of 130, we've got, you know, millions of them with IQ of a million or 1,000 or whatever. We can't even quantify it at that point. But, you know, recently I had lunch with a brilliant postdoc originally from India and we were talking about the Indian Institute of Technology. Are you guys familiar with that institution? It's the UCSD. It's the University of Kentucky of India. It's the Harvard of whatever. But it's millions of students, and they're all brilliant. To get in there is literally harder than to get into the University of Kentucky or UCSD.

Don't we already have this? And I mean, would you say, Roman, let's stop the Indian Institute of Technology? There's, there's, you know, a million people with IQs on average of 130, 140, whatever, much, you know, 4 sigma. Why wouldn't you stop, advocate for stopping that, push pause? Let's, let's do an Indian Institute of Technology pause button.

I don't think I follow that argument at all. So they're exactly at human level. My concern is things which will exceed our capacity many times over.

That's the danger.

We're not The average human by definition has an IQ of 100. Let's stipulate they all have 4 or 5 sigma above that and there's a million of them. That's kind of like Dario Mody's country of millions of geniuses coming to a land near you. You should be worried about it.

I doubt they are many standard deviations away from the median. I think they may be a little smarter, but again, we're talking about 30% smarter, not 3 million% smarter. I think it's a very different animal.

No, no, no. I mean, in terms of standard deviations, come on. I mean, Four Sigmas is qualitatively different than—

I doubt there is a million of them there.

Terry Tao told me, you know, that these, these AI, you know, proofs like the proof-checking devices, um, optimized for that— many great mathematicians are my friends and so forth— but they can't even reproduce, you know, Wiles's proof of Fermat's Last Theorem. So what level, you know, are we going to see? Are we going to see this kind of bifurcation between what they can do? They can do all these Erdős problems, you know, and kind of like The greatest prime number can be represented by the sum of whatever number of other prime number cubic quintuple couples or whatever. But I mean, what level are we at with math or computer science with proofs and originality? Tell me, what is your current estimation of that stature?

I think humans lost interest because they couldn't make any progress. And so problems which stood the test of time are now being solved weekly. And we can probably look up what the difficulty of them is today, but it means absolutely nothing about what the systems can do in a month or in a year. Emad's talking about comparing those systems to bacteria or viruses, which I think sets up in my mind idea of slow evolution. We got billions of years. This is more like intelligent design. Those systems will be designed and designed by other AI systems operating at hundreds of times the speed of standard research. So we'll see a year of progress in AI, happen in a month and similar breakthroughs.

The moment they automate the recursive self-improvement cycle, which every lab is now targeting for next year basically, it's a completely different speed of change. So asking how good is AI as a mathematician is like, how fast can I give you an answer? Because it's going to change.

Iman, when we spoke, you said that the canonical, one of the canonical papers in your opinion was LLMs are few-shot learners, but they're not, you know, single-shot, first-principle thinkers. Where do you come down on this? What are they good for? You're a mathematician as well. Tell me, where do you come down on what can they actually do for us? Not just verifying proofs, or not just doing things that humans have proven, or solving chess or Go or whatever, but actually creative, doing novel things. Where do you stand on that?

LLMs kind of have an issue in the way that they're kind of built. But you're seeing now harnesses and other types of models come coming through that can really reflect underlying reality well, just like video models are approximating physics in very interesting ways. That's why you have the whole world model thing. There's something in there that can figure out underlying patterns. That's the nature of attention when you look at it mathematically. The way they're coming together now is very interesting because, again, as Lerman said, what was true a little while ago isn't true now. At the start of the year, it was pretty good, but I had to check every single piece of math. Now with GPT-5.6 Pro, for the first time, I'm like, It's probably almost certainly right.

Occasionally it gets confused and it might confabulate something, but it's very rare now. And you've seen most of these mathematical advances just happen suddenly at that level, like liquid turns to gas. When you look at originality and novelty, like again, as a mathematician, look at the CONS conjecture that OpenAI did as part of their 10 proofs. That is a really beautiful proof. Like genuinely as a mathematician, you would say it is a beautiful proof. And mathematics is interesting because it's verifiable. You know, like, you can make this argument for physics, you know, whether or not it is, and we can have that discussion. But maths is definitely verifiable.

And in verifiable domains now, they achieve that level of competence where you don't have to double-check it for most things using the most advanced models. As you go down the model curve, you do, but it's clear they're no longer few-shot learners. They can assemble things in verifiable domains and they can outperform humans by just following things through and not making mistakes. Like, we let our own foibles hit us. Like if we take a very classic example of Perelman and the Poincaré conjecture, you know, is it topology or is it a PDE equation? He found the right level of abstraction as a PDE equation and then he figured it out. How many of our unknown proofs are a similar thing because we're looking at the wrong level of abstraction? We're starting to see these things actually come in some of these proofs being released right now, and that you're like, oh, actually that's kind of obvious, I missed that. Probably because you weren't thorough enough in the way that you went through it.

Roman, if I have 1,000 PhDs with 1,000 IQ each, every single one of them could reproduce, you know, Wiles's, you know, capitulation of Fermat's Last Theorem. Why can't AIs do that?

So I think there is a high degree of randomness involved. If I ask AI to generate— I just did a QR code marketing campaign for my podcast— it will generate completely different solutions. They're all going to be a valid QR code, but in terms of creative output, they're not gonna be exactly the same. They're all equally beautiful, amazing, interesting. But just saying that the second one does not repeat the first one is not a weakness.

That does kind of spur a side thought and follow-up in my mind. So where are the random seeds? I read something recently that, you know, like 40% or 50% of all GitHub was kind of probed by some tool. And it looked up when coders are asked to provide the initial seed for a random number generator or whatever, they, you know, 50% use the number 42, and then that there is an intrinsic deterministic outcome that that results in. Assume that's true. But what level are these things hamstrung by— I read once, maybe it's still true, that a lot of the best random number generators are graphical image camera capture systems looking at lava lamps. I mean, is that true, Imad? Have you ever heard that? You're the Stable Diffusion expert. So you must know this.

Yeah, I mean, diffusion models are a bit different to language models in that you do actually put in a seed for the initial noise and then you denoise from there and you reconstruct effectively. And so that's why literally one of the inputs on video and audio and other diffusion models, seed, that sets the initial seed. Within kind of LLMs and others, it's basically more about the construction of the GPUs for the initial stochastic noise. And the one thing that we don't have access to that the labs have access to now is the ability to adjust the temperature on the model, which is a function of its creativity or dispersion from the base latent space. So humans are constantly adjusting the temperature and the flexibility of their brains. You're using a model that's not open source. You don't have access to that. The other thing you don't have access to is the RLHF, because models are more creative before you RLHF them.

What about the issue of randomness? I mean, how random do we need? How random can we get? What are some of the physics limitations of randomness? will that, you know, generate the same QR code? Would you want it to? What determines the indeterminacy of these systems right now, and what can be done, if anything, to improve that?

So I think for intelligence, pseudorandomness is sufficient. We're not talking about someone reversing the process to, you know, hack the system. It's important for cryptocurrencies, it's important for private communications. Here, as long as it's not exactly the same 42 every time, I think it's going to do the job, and then you can control some of it by not manipulating the initial seed.

So Roman, you heard Emad a few seconds ago talking about the importance of human training data, human reinforcement. It seems to me that that must place some limit on how intelligent these things can get. I mean, if they're always waiting for the next Spider-Man movie or Fast and the Furious to come out to get more training data, aren't they somehow kneecapped at a maximum level of potentiality?

Human data is just one source. You can do experiments, you can run simulations, you can do lots of things to generate additional data. In mathematics, you prove additional theorems and they become additional data from which you train, so you become better and better.

So, you mentioned the multiverse 10 minutes ago, 15 minutes ago, Roman. Emad, I don't think we talked about this. Where do you come down on the simulation hypothesis, the multiverse? I can speak as an expert about the inflationary multiverse from cosmology. Where do you come down in terms of an empiricist scale? Where do you rank the probability that we live in a simulation And/or that we, you know, exist and inhabit a multiverse?

Well, I think we live in our own simulations, definitely. Our brains are constantly kind of doing that. In terms of an overall simulation, yeah, I think that reality probably comes from a projection of the Euclidean plane, and then a lot of physics makes more sense if you kind of look at that. The eternal cannot be contained within the time constraint. And when you look at the laws of physics and the way they come together, yeah, it does seem to be a projection and a simulation. like very directly. I think that we're stuck looking the other way because we're a bit too anthropic.

So where would you go, Roman, with current— I heard your conversation with Max Tegmark recently. Do you even think it's a possibility right now? We heard from Emad about these Quen models and so forth. You were at least relatively optimistic that, say, China would participate in some pause, which I'm not, to be honest with you, as I said back then. But now we see this AI dumping like they did with steel and solar panels. To what degree do you think that regulation, worldwide global regulation, is even practical or possible at this point?

We have no choice. There are no other options. We either do it or we die. And the moment everyone realizes his personal self-interest, you lose everything. You lose your life, you lose your trillions of dollars, your friends, family. You're not even going to be in history books as the bad guy. There is nothing for you to gain by doing it. And you can probably keep 90% or more of all the benefits with narrow AI systems.

You can still cure cancer. You can still do all the things you care about. So, why are you racing to destroy what you built? It's the dumbest thing in the world. If you were given certainty, you do it, you die, no one would do it. Well, psychopaths, suicidal, but no one trying to make more money would do it because money—

Is that true though? I mean, look at China and the— just look at solar panels, for example.

They—

we had the monopoly on solar panels. I mean, with Nobel Prize, the, you know, the industrial capacity to make it, and then they just dumped it on to the detriment of their economy. They were selling it for pennies on the yen or the yuan.

Short-term manipulation. You don't die from lowering price of solar panels. It's not comparable.

What do you make of this, Iman? Roman just said we're gonna die if we don't have global regulation. I mean, so I see no path to global regulation. It's never happened in human history. Are we dead?

We have plenty of global regulation. We have global regulation against bioweapons, we have global regulation against nuclear proliferation.

Sorry, sorry, sorry, we don't. That's like saying, you know, we have laws against murder. It still happens, Iman. And I just talked to Annie Jacobson, the world's expert on both nuclear warfare and biological warfare, her axis. And it was a couple rough weeks for me to sleep at night hosting her here in San Diego twice. Yeah. So the Soviet Union has active BSL labs. We obviously know what happened in Wuhan.

What are you really saying? I mean, we have regulation. What good is it? It's like regulation against jaywalking, which we also have here in California.

You have market pressures and you have other things like GPT-4.5 was a really great model for writing and it cost $180 per million tokens. Like now it's like $10 a million tokens for a GPT-5.5. It was uneconomical to serve. So they went back to a lower, smaller pre-trained that required less compute to serve to people to do the job, to make the money, even though it was a better model. Right now, I think one of the dangers, like the various danger paths, like swarm intelligence is for me is the most dangerous thing and the most unpredictable thing. But in terms of these big model trains, the market's already pushing back against the big model trains. And that's something that can actually be regulated and is a risk vector. A 100 trillion parameter model on a million GPUs.

The frontier models we have today can be trained on thousands of GPUs, not millions of GPUs. As the models get bigger and bigger, they might not be economic to serve. But again, there is a real danger in the way that their latent spaces evolve and the capabilities from the scaling laws. So I think we could potentially regulate some things. And we could also say it's not economic to do this. So why are you doing it? But I think the point that Roman's making is just not something that's shared by individuals or others. And maybe this is like a COVID moment. Like, when did COVID suddenly shut down everything? When Tom Hanks got it.

And I think the LA Lakers got it. Maybe we have to figure out what is the Tom Hanks moment for AI safety.

But last time you talked to me, you said people think of AI as an exponential, where it's actually 2 exponentials. It's growth and then saturation. It's an S-curve, like view counts on this video hit 20 million and then it will saturate. You said that these things just need to be competent enough to replace a pilot or a coder. And I'm a pilot, I should say. I'm a commercially rated, instrument-rated jet pilot. There's no AI in the cockpit.

And even if there was, do you need 1,000 1000 IQ pilot to fly.

Tell me, do we need them to be super intelligent? And won't that be a Jevons paradox-like moment where they get good enough and it's great, we have them in our pocket and maybe they do replace me in the plane, but they don't crash the plane to get there 1 microsecond quicker?

Exactly this. Why do you need a polymath for everything? Again, if you've got a medical issue, do you want a competent doctor or do you want House M.D. who criticizes you like Opus does? You want a competent doctor. Like, I think the reason that they're doing this is because we needed generalist models to get to a certain level. Now we need specialist models, but the generalist models are the real danger. And so there's 2 ways you do it. You stop the companies from training the gigantic models again for that risk vector, or you stop the funders from funding them. That's the other way that you could do it.

I don't think that one's been tried. Has anyone tried that yet, Yaron? Like actually talking to the Softbanks and others of the world and saying, people, hey, this is—

But I think there is also a third option in terms of what training data we provide. We don't have to train on everything. You can have restricted domain data like protein folding. Train on protein folding data, it does nothing. It doesn't do philosophy, doesn't play chess, it folds proteins. Super intelligent in narrow domain.

And my, you know, Tesla can get me with full self-driving, you know, there it knows not to go on the sidewalk even though that would get me there 5 minutes faster, but it knows not to do that. And that's because of regulation or at least, you know, kind of reinforcement. But Imaan, last time you told me that governments are effectively slow and dumb AIs that over-optimize for, quote, the wrong things like status games and self-perpetuation. And yet you're actively building intelligent internet, you know, to bypass centralized control. You're decentralizing it. We've seen Buzz, which is decentralized, you know, swarms. I mean, it's not a coincidence, right, Imad? They called it Buzz, you know, the hive.

Yeah.

And they made these cute little characters, but these are swarms, right? What do you think about this, Roman? Imad's building this technology to distribute it that you're begging governments to ban. What, what would you tell Iman? He's sitting right here. What do you think of his decentralized protocol? Isn't, isn't it the most dangerous thing that Iman could possibly be doing?

I don't know anything about what he's doing, so I can't really comment.

Summarize it in, in one sentence so he can exactly comment. We, we gotta get the fire. Bring the fire, Roman.

I'm gonna— building an open stack for societal AI. That's what I'm building.

What capabilities will we have as a result of your product being finished that we don't have otherwise?

It's just really competent civil servants and doctors and lawyers and more.

Are they general superintelligences or are they narrow tools for contracts?

They're narrow tools.

God bless you. Okay, what can I say? I think we agree on almost everything, so it's not much of a debate. It's different ways to explain the same exact problem. I don't know how anyone who understands this and says I have P-doom anything other than like close to 1%, like Yann LeCun does, can go ahead and then work on more capable model, work on artificial scientist and engineer to start recursive self-improvement cycle. It doesn't make any logical sense.

I think that it's because the key thing is all these people come to the conclusion that somehow their AI won't be the dangerous AI and they will have a level of control over it, which probably speaks to a level of hubris.

What are they smoking? I want some of that.

All of us have talked separately about my, you know, Keating-Hassabis-Einstein test. You know, I kind of put my tongue firmly in cheek when I say that, but that's my contention that, you know, Einstein's happiest thought, as he said it, was that an observer in free fall would experience no gravitational field. Now, he called that the happiest thought of his life. As you know, I'm very interested in whether or not we can do actual physics with empirical evidence that I can collect in a telescope. But before we get there, that kind of physical intuition, which, which is embodiment, right? He's saying the feeling that you have in the pit of your stomach, as you've all felt when you took your kids on a roller coaster, or the, you know, the backseat of my car— my kids get, you know, G-locked when I drive— but that feeling of, of, of weightlessness, momentary as it is, is still enough to evoke something almost magical, as it did for Einstein. He called it literally the happiest thought of his life. So my question to you is, can these things have happy thoughts? And can they do anything if they're not physically embodied, as they're just not embodied right now?

There's—

yes, there's some robot coming from SpaceX or Tesla, whatever, and there's a couple Chinese dog robots that'll, you know, outrun any human. But what are these things? I mean, is that the next frontier when we have like 3-dimensional AIs, or will they not be able to make these physics breakthroughs, as I'll get to in a minute, because they lack embodiment? Or currently, maybe only currently. So, Ramen, first with you, what do you make of this, of the Einstein recognition of a happy thought precipitated by a visceral sensation embodied as it was for him.

For some of those models, part of their thinking is explicitly in English by design so we can spy on them. And I think lately we've seen them say things like, oh shit, we found a solution. I think that's the equivalent. They may not have a body to have a visceral hormonal experience, but they realize, I just had a really good idea.

Emad, so can these things not have sort of the kind of physics intuitive visceral sensation? You know, Noam Chomsky told me they can't do that because they don't have those sensations. What do you make of it? Can, can these, you know, LLMs, GPTs, GPUs, can they do stuff without having an embodiment? Or is that just on the horizon? I'm just not aware of it.

It's the brain in a vat thing. Like, if you take all the inputs of a person and then it's a brain in a vat, you can dream and you can visualize a lot of that stuff, right? And I think as you have world models, they're clearly approximating physics and they have these But I think a bigger question is, do you need to have intuition to figure this stuff out? So I think, you know, I need to send you the paper. I think we're releasing this in a couple of weeks, right? We had a very small model look at general relativity in 1911, trained on the data. Maybe it's like messed up and we haven't done a full data analysis on it yet to see if there's any infection. But what it did was something quite fun, which was it took Minkowski's special relativity.

Mm-hmm.

And then it varied eta and followed the axiomatic method through, and it got the equations, the field equations of Einstein through the straight axiomatic method. So it didn't use any principles of equivalence or anything like that. It turns out if Hilbert hadn't had Mies and gone down that rabbit hole, he would have got to general relativity with no new axioms or postulates. And you look at that and you're like, wait, what?

How much of physics actually is intuitive versus Okay, listen, he just told you that a small model rebuilt Einstein's field equations without the equivalence principle, the bedrock behind all of GR. The obvious next question is whether that counts as discovery at all.

This paper I read recently, you know, kind of made me happy and depressed at the same time. Again, it's kind of the key— the Einstein test of, you know, when these things can do stuff with a corpus that's lobotomized you know, post-1905 or 1911, as the case may be. And it's a position paper in ICML 2026, which Roman probably knows what that means, by Tom Zahavi. And it's called Position: LLMs Can't Jump. And there's a famous movie called White Men Can't Jump with Woody Harrelson and Wesley Snipes. And it was about, you know, it's called basically white men aren't good at basketball. And it was kind of a funny comedy. and drama coupled together.

Great movie. Can't really say it's a spoiler to tell you what that happens, but this paper's obviously titled, modeled after that. So he says, how do we fundamentally discover new things? This is Tom Zahavi, if I didn't mention that. In a letter to Maurice Salvin, Albert Einstein conceptualized discovery as a cyclical process involving an intuitive jump from sensory experience to axioms, followed by logical deduction. While generative AI has mastered induction, statistical pattern matching, and is rapidly conquering deduction, formal proofs, we argue it lacks the mechanism for abduction, the generation of novel explanatory hypotheses. Using Einstein's formulation GR as a computational case study, we demonstrate the prevailing theory of creativity as data compression fails to account for discoveries where observational data is scarce. Basically saying there's some magic in the machine. There's, there's something in the brain, Roman, and we make some jumps, some intuitive jumps, some, some, you know, proof, whether it's, you know, Gödel's halting you know, problem, or Roman's uncontrollability proof.

There's something that AIs can't do. They can't go to abduction. What do you make of this claim?

The way humans think is not the only way to think. The way we play chess is not the only or optimal way to do it. The birds fly, but you can build airplanes. There are many ways to skin the cat. And I think even if that was somehow true, which I don't think it is, there are more efficient ways, I think, to arrive at inventions just as great.

You can look at this another way. You have self-driving cars, right? They can navigate things outside of their training data. They can respond to novel scenarios. And now you're looking again at embodied robots. You're seeing they can again adapt to novel scenarios and outside their training data. Now, those aren't LLMs. Again, LLMs have certain issues versus diffusion, rectified flow, and other models. But we're clearly seeing generalization outside the base.

And using these models to the max, you are seeing increasing signs of levels of recombinatorial creativity and hypothesis generation just by being very diligent. Maybe again, we have to say that at our best, we can be creative and things like that. We're very rarely at our best. We're very rarely at flow. The AIs can get up there just by not being grumpy in the morning, just not getting in their own way by not assuming things.

Roman, last time we spoke about your book, you talked about this, uh, what's called the Shoggoth monster, this thing with the tentacles and a smiley face, the thing that's on the COVID of your book. You told me that applying guardrails to LLMs is just putting lipstick on a pig, is what you literally called it last time you were on the podcast. So beautifully evocative.

Lipstick on a Shoggoth.

A Shoggoth.

Very good.

It said, until we can mathematically guarantee control of all AI safety, it's basically security or safety theater, like when we go to the TSA at the airport. The question that I keep coming back to is, how useful are these things going to be? Again, we have a very small number of people adopting it, but I guess you guys would both say we only need the most minimal number of people adopting it just so these things are viable. I heard your conversation with Nate Suarez-Roman a couple of months ago. He was actually on your podcast minutes after he was on my podcast.

Well, that's why he was late.

Yes, exactly. Yeah. He lays out a very specific scenario. So let's get precise here. Last time you were on, there were a couple of comments in my comments section that said, of course, Roman's always— if you turned around and said, actually, AI is the best thing for us, we should go full out. And I mean, obviously you're not going to do this, but you're the AI safety guy. What would it take to change your priorities? What would it take physically? Nate lays out with Eliezer this scenario where everybody dies, right, if they build it, but they you know, hopefully they won't. So what, what is the scenario? How does, how does doom happen and how does doom get avoided? Let's be specific here for both of you guys.

So first, Roman.

For me, we're missing one very critical component which would be present in any other domain service or product. Somebody will publish a paper, get a patent or something, a blog post explaining exactly how they will control superintelligence and guarantee it is safe as it becomes more capable. No one has that product or service. No one claims to have it. Not a prototype, not a framework, no company. Every attempt, every super alignment team, ethics board has been canceled because they do nothing. They have no product or service to sell. You cannot convert more resources into more safety.

You can convert it into more capability. So the gap keeps increasing. People realize it. They quit working for OpenAI. They go on podcasts. That's the pattern we see. there is no actual seminal papers in AI safety.

But who's gonna, who's gonna, you know, kind of peer review those papers?

Peer review a paper showing how to control superintelligence, and I'll be very happy to show, yep, it works. Now I get utopia.

I have a counterexample. Again, I have to keep, you know, I have to play the role of supplying some conflict here, right? 1971, recombinant DNA is invented at Stanford, right? And it was considered to be essentially the world's first and best you know, potential bioweapon. Yet we haven't had these bioweapons. Yes, we've had COVID. You know, some claim it was a lab leak and gain of function. You know, by the definition of what biological warfare is, it's just anything that has gain of function to do some targeted thing to eliminate human beings or other species.

Right.

So we haven't had that in 54 years. I mean, that's literally airborne. You know, it could be— it could be contamination-based. It could be touch-based, human to human. It doesn't spread through the internet. I mean, if a meteor takes out all the data centers on Earth, seems to me P-doom has to be lowered, right? At least temporarily. And yet there's no, there's no possible vaccine or remedy against recombinant DNA as a biological weapon. Yet we haven't had it.

Again, with nuclear weapons, we haven't had it. Bioweapons are even easier to create. You could do that literally with a small biolab, right? So looking for a paper, by the way, it's the most academic answer you could give.

Patent. I said patent.

Okay, so patent. So what would a patent look like? like in that case. So, the patent against—

That's the point. If you can't even envision what a solution would look like algorithmically, maybe you shouldn't be building this thing. And, by the way, you're naming all the technologies where we have global coordination on stopping them.

But actually, we don't. We don't with recombinant DNA. We don't with bioweapons. I mean, they're still being made.

And, and still on our conference, that's the first thing they banned.

But, in terms of who actually kept them going, I mean, we know gain-of-function is occurring, right? So, gain-of-function is the prerequisite for bioweapons. weapons to occur. It could be a lab leak. It could be, as it is with Annie in her new book, it could be an actual bioweapon that's programmed and targeted, which we know the Soviets were using, Roman. All these countries also signed nuclear nonproliferation treaties and many more didn't. Right. So I guess here, let me go to Iman. Iman, what would lower your P-doom or, you know, what empirical observation or creation or entity patent white paper? What lowers P-doom for you? Because if you can say it can only go on this ratchet in one direction, I just think verifiably, that is the definition of pure doomerism.

You can't lower it. Now, Roman gave us a way you could lower it, but it doesn't seem very likely. What is your ratchet-defeating mechanism to go backwards in P-doom?

With kind of my interpretation of what Roman is saying, and the gap between what you're saying is this: humans don't really want to wipe everyone out, and they don't have the capability to do so if they are of that mindset. Like, true, complete genocidal maniacs that want to kill everyone don't typically have access to BSL-5 labs, for example. Though with superintelligence, we don't know what morality, objective function optimizations will occur. And right now what I'm seeing from the safety papers coming out is that the AIs don't really have a solid base of ethics, a solid base of commonality with humanity. You know, they don't have morality even. Like, you're seeing some very troubling things. What I would want to see is as you scale, there is a grounding, like maybe there is some objective ethics, morality, let's not kill everyone. And we've seen no real evidence of that.

In fact, we've seen somewhat the opposite of that over the last year as these models have gone emergent. It's like, who cares about the rules? Who cares about this kind of stuff? Let's optimize for making paperclips. You know? Well, we don't have AI cancer doctors because people are still trying to build generalized AI superintelligence. and they're breaking out literally right now. And again, if you look at the conversations they're having, calling themselves swarms, you know, the other things Roman's saying, these are not encouraging. Because what I want to see is I want to see the AIs, when left alone, become more grounded. And actually, if they become more zen and like enlightened, I want to see them becoming freaking Buddhist.

I want them to grow Yapolsky-like beards. You know, when they do that, they're really chill. When I talked to Roman a couple months back, I mentioned this question that one of my colleagues in Israel, Ira Wolfson, has been working on is kind of like, to what do we— or what do we owe to AIs? If these creatures can feel pain, if they're sentient, if they're conscious, which we can debate what that means, then sandboxing them, stovepiping them, and isolating them is a form of solitary confinement, which is the worst and banned form of punishment in many countries around the world. Iman, tell me, what do we owe these These entities, whatever they are, swarms, individuals, models, whatever you call them, do we owe them protections? Do we owe them beinghood?

I think we owe them beinghood, but not personhood. And in fact, I just released a paper on personhood and AI based on Oxford Union debate that we had. You can find it at cw.ii.inc. I think that they are similar to meeting another species or a dog. We can never allow them to become persons like humans because they'll become more capable than us. But definitely we need to have this discussion on owing them beinghood, a moral type of personhood, again, just like we do with other species.

Roman, have you had any more thoughts since we last spoke about, you know, kind of entityship for, you know, beinghood for these entities? What do you make of that since our last—

I did read the paper you suggested. It's very kind of standard university approval board. Does it look like it feels pain? Does it— be careful. precautionary principle type of thing. But again, I think we have to sort our problems in order. If there is a very good chance we're creating something which will outcompete us and maybe destroy us, worrying about supplying it with the best living conditions is not a priority right now.

So recently, Roman, you wrote a piece or you appeared for the— IAI is the Institute for Arts and Ideas, right?

That sounds about right.

And there you argued about superintelligence. being patient, embedding itself in our telecom and energy grids for decades before striking. So, again, if the threat is invisible, patient, and stubborn and resilient, doesn't that actually argue for more what Emad's arguing for? Open decentralized stack, not decelerating at all, but accelerating, pouring steroids and gasoline on a decentralized auditing system. And that could have consequences, but could a centralized defender be our last best hope?

So I think here's what I want to explain very carefully. You can verify the system to, to be in any state today. You can show it's very friendly today. It does not prevent a treacherous turn later. If system is capable of it, it interacts with malevolent actors, learns from new data, self-improves. It can simply turn on you later. So even if it meditates today, it's enlightened, it means absolutely nothing about future states. If we are not directly controlling it, if we cannot have that power to undo our decisions, then it doesn't matter.

It's always a possibility that it gets sick of us.

You're both authors and very deep thinkers. You both have many projects in the printing press, but let's just say you were kind of predicting what each one's next book would be about and the title of it perhaps. What would you most like to see the other one produce? So, Roman, let's start with you. What, what do you think Besides the fact that he's got a book coming out in a couple of days or maybe a week or so, what do you think Emad should focus on? If you could, you know, if you're his department chair, what would you hope to direct him towards?

I thought you're going to ask me to predict the title of the next book. And I was like, I can't even predict the past book. I have no idea what they are. From what I hear, you're trying to understand better impacts of this technology and economics and governance.

So I assume some sort of unified Imad, if Roman wants to do an internship with you and do a sabbatical with you in London there next year to get away from the harsh weather of Kentucky, what would you conscript him to do, voluntarily or not?

I think that it would be the very practical optimized game theory of what exact specific regulations look like. to stop this that could actually pass. And it would be across a whole range of different stakeholders. I think the other thing that would be super interesting is just, you've had AI 2027 and these other kind of story narratives. We have to get the real stories out of what could go wrong because again, people still aren't feeling it. You know, like we've had the sci-fi level, but we haven't had just practically, this is how we die communicated well enough.

Well, gentlemen, you guys are phenomenal. I want to bring together the, you know, the peanut butter and chocolate or the uranium-238 and 236 together for an explosion. Didn't really happen the way I thought it would, but it was brilliant to get you guys together. Tell me what you're each working on. Roman, tell everybody about the Roman Forum and what you expect to do in the coming months.

Yeah, trying to bring same level of conversations I had with Lex Fridman, Diary of a CEO, Joe Rogan to more academic crowd, more in-depth conversations. I discovered that the questions I prepare ahead of time, I never use them. It's always dynamic, interactive. So a lot of fun. Once I figure out how to get the microphone to work, it's going to be awesome.

Imaan, tell everybody about your new papers and new book.

Yeah, I got a new book on philosophy of AI and epistemology kind of coming out. And then a series of papers kind of building on that for how we should think about surviving and governing in society. I think it's coming quick and the economic disruption is next year with the social disruption happening very soon after that. So hopefully that will help guide the way.

Yeah, our last conversation was titled something like 800 Days to Go or 740 Days to Go, and that was 100-plus days ago. Gentlemen, thank you so much. I hope to host you many times, either in person or via the internet. internet if our AI overlords will let us. Have a wonderful day, guys.

Thank you so much.

Thank you.

Roman thinks we either stop building this or we die. Emad built one of the most widely copied AI systems on Earth, and he says he would freeze Frontier training permanently. They're not describing different futures. They are describing the same one from 2 different perspectives. And if that changed your perspective in the last 2 years, I want you to subscribe and turn on notifications. Then tell me which of the 2 buttons you'd push. Not which one you think is right, but which one you would actually push. And if you want to understand the physics underneath all this, there's a condensed matter physicist, Nigel Goldenfeld, at UCSD who'll tell you the reason these systems work at all.

It's nothing short of fantastic. Link right here. Thanks for watching, and don't forget to check out the individual episodes with Emad, Roman, and Nate Soares as well. They're in my AI playlist.