A computer scientist who helped found the field of AI safety just told me we're building something we can never switch off and that the smartest people in the room agree with him.

We're going to have systems ten hundred thousand million times smarter than us. What does that mean? They see patterns we don't see. You can have squirrels, monkeys, whatever you want. They're very intelligent beings, but they're not competitive with us. We're just on a different level. And it's exactly what we're going to see here. Having an agent and giving it full access to your computer, your bank accounts, your email, sounds like the dumbest thing you can possibly do. And watching smart people do that really blows my mind.

Well, we're asking for a pause in frontier model development contingent on someone solving control. If I'm right and control is unsolvable, that moratorium becomes a permanent ban. If I'm wrong, in 10 years I'll get utopia free stuff and be very happy to be wrong.

That's Roman Yampolsky, a computer scientist who helped found AI safety and who now argues that can't ever be done. Not hard. Provably impossible. Now, I'm used to going into the impossible, but today he's going to get into why the math says so for the first time at this level of detail and the one button he'd actually press if he had the opportunity.

We can't bound the limits of knowledge and part of controlling super AI. Again, I'm being devil's advocate here. I'm not saying this is my position, I'm just saying this is what I think David Pascha, David Deutsch would say is that because to control something you need to have knowledge of its future, prediction of its future behavior, although you can't have perfect knowledge of anything. Right? That's impossible for sure. But the question is, can you have the same level of controllability via the knowledge that a human who's a universal explainer can glean? And I guess he's saying you're saying it's impossible for a human to ever control AI, but he's saying humans are universal explainers, therefore nothing explainable, even with errors and stochastic notions of what explanation means, is not fundamentally restricted. So it does seem like you guys don't agree, and that's fine. That's not a formal proof, by the way. I'm just saying he's saying humans can explain things.

AI is something that could be explained, therefore it's not impossible to explain them. But I think you would Say it is impossible to control them because you cannot explain them. Correct.

So there is theory and practice. In theory, given infinite time, average human with pencil and paper can probably figure out everything. We don't have infinite time and we have limited size brain with limited size memory cells, our ability to survey information is limited. So if you had a mathematical proof and it was a billion pages long, no human can verify that proof. In theory they could, but in practice it's not going to happen.

I thought you're going to say in theory communism works, but in practice,

if we have a real world situation where you have a system, let's just say it's just like humans, but it's a billion times faster. We're not competitive in that environment. We simply don't have time to react, to do anything whatsoever to counteract what the system is going to do. But we are not equal. We're going to have systems ten hundred thousand million times smarter than us. What does that mean? They see patterns we don't see. I always bring up examples from animal kingdom, right? You can have squirrels, monkeys, whatever you want. They're very intelligent beings.

Eventually they have some language culture, they're starting to use tools, but they're not competitive with us. We're just on a different level. And it's exactly what we're going to see here. Him being a very good professor at a very good university, he understands when he selects students, he doesn't select them at random because all humans are universal explainers. He looks for the one with highest intelligence. Why? Because they're going to finish things in time. They're going to understood and publish on time and it's exactly the same. Instead of looking for a student with IQ of 130, now you're comparing machines with IQ of a thousand to humans and saying, well, technically they're all in the same class of automata.

They all Turing complete. True, but that means nothing.

For safety, Einstein said the following Roman, he said no problem can be solved from the same level of consciousness that that created it. So did Einstein kind of predict some of the claims that you're making now that we have basically have been from the start unable to even grapple with the questions of what sort of entities we're creating?

I think I would disagree with him. I think you can solve certain problems from the same level, but what you can do is solve problems from higher level. So as an agent with this world model, with this IQ, I cannot successfully solve problems for agents with IQs of thousand million. That's the problem. You can go lower. I can solve problems for those at a lower level, but not. Not go higher.

What do you think that Einstein would have made of. Of AI? What utility does it have to a practicing scientist like me? Or would it have had to Einstein himself?

So it seems like, at least in mathematics, we're starting to watch AI overtake, overtake that profession. They are now proving things, interesting things, not trivial theorems, using latest models. And we are at the earliest stages of that process. So if the same progress continues as we saw in our domains for the last three to five years, very soon it's going to make no sense to use a human mathematician. It's going to be AI doing the work of a mathematician. Maybe at best, mathematician will point it at problems of interest, but really, I don't think humans will be competitive in that space.

So I developed something that I called the Einstein test. And Demis Hassabis kind of also suggested this. And it relates to what Einstein said in 1907. He said that an observer in free fall, you're on an elevator there in Louisville, and the elevator snaps, that you feel no gravitational field. And that led to the Einstein Equivalence principle. He called that the happiest thought of his life. And I've wondered for a long time, and I'm hoping you can help me understand better, perhaps, to what extent can an AI have a happy thought? A. And to what extent could Einstein not have come up with the Einstein Equivalence principle had he not had a body that was deeply, viscerally, literally viscerally connected to his.

His own consciousness? So those two questions, can an AI have a happy thought? You know, and can it realize from that happy thought the visceral sensation of what leads to the Einstein Equivalence principle?

That's a great question. So I think the latest mathematical proof of one of the Erdos problems, they traced the thinking process in the AI, and right before arriving at the solution, it goes, oh, we have a crazy idea here. I don't know if you saw that, but that's literally in the thinking process. It completely jumps out and goes, oh my God, we're going to do something insane right now. So looks like experimentally it's capable of that. I would think it's more about brute force options. As a human, I cannot consider all possibilities in the small subdomain of that problem, but AI literally can brute force. So many proofs.

We saw it with, what is it? Four color problem, but I think it's universally applicable to most things. There's only so many previous mathematical proofs, and that is a huge, huge number for a human mind. But AI can literally brute force all the previous relative tools and see what works for that specific problem to move it in a new direction. So I think it can both do a very concrete brute forcing and have a stroke of genius moment, I guess. I don't know if you need a physical body to do this. If you run thought experiments at sufficient detail, you create internal simulations which could be as realistic as this universe. You can have model of physics, you can have intelligent agents within the experiment. I think you can bypass physical body and do it all virtually.

So when I think about what Einstein did, it does seem sort of like in chess, you know, you'd call it a brilliancy. Telling me it's true that proofs generating tools are coming up with their own kind of versions of brilliancies. But what I tried to do a couple of years back now, and I'm trying to do it with more professional assistance, shall we say. I'm a cosmologist, but I'm secretly hoping to recruit you and I'll send you my paper draft in a little bit. I'm sure you don't have enough to read, but the thought was the following. If you trained a special LLM just based on the corpus of knowledge in 1907, when Einstein had this thought, would it have predicted the anomalous perihelion advance of the planet Mercury? I mean, they knew this existed since the 1700s, that Newton's laws could not account for this microscopic. I mean it's 42 arc seconds per century. It's a minuscule amount.

But if you had that, Einstein wanted to explain that and obviously came up with gr. But the way that he came up with it seemingly was superhuman. I mean, it was using Riemannian curvature, it was using concepts from Gaussian geometry and non Euclidean geometry, and then it used like tensor notation. And it seems to me it's almost impossible now for us to do this because the Corpus of all LLMs has been trained on these brilliancies that have been made by Einstein. So can you design an LLM that's sort of lobotomized after a certain point in order to investigate whether an AI can do what Einstein did?

That sounds super cool. I would not limit it to just that specific question or experiment. I would just say, given this knowledge, start coming up with new science, maybe it will do something equivalently great, but not necessarily rediscover that specific moment of brilliance.

My proposal to Terry Tao, when I spoke with him a year or less than a year ago now, it was the following. Can AI actually reconstruct say a proof of Weil's proof of Fermat's last theorem? And at that time, you know, it was only in September he said no. In other words, if they can't do things that a meat mathematician did with the three pound, you know, wet supercomputer on our shoulders, to what extent can, can they come up with new stuff? So I agree with you. The first application of the Einstein Keating Hassibus, you know, maybe Yampolsky test would be what is dark matter? You know, is dark matter a new field of, of gravity like mon modified Newtonian dynamics? Is it a modification to Newton's laws to Einstein's? Or is it dark matter like, you know, like a chunk of rock like Neptune was to the planet Uranus? So I guess the status of these things, like I kind of wish that Erdos was never born. I mean I think I have an erds number of like 12 or something. But all these problems are getting solved. Sure, I get that and I get that no human will ever win at chess again against the computer or GO again. But that's not as interesting to me as like creating GO or creating chess or creating Fermat's last theorem.

But in this case, according to Terence, they can't even verify that proof, let alone come up with it ab initio. So what do you think about that? Like when will you think of AGI being passed? Or do you already think we're there?

So in many domains AI is human level is super intelligent. I think in mathematics it's now smarter than all, not mathematicians, it's smarter than many mathematicians. Obviously none of them were able to prove this theorem for 70, 80, 90 years. So that's super impressive. Peer reviewed top journals and again we are like on day one of this process, give it a year or two, I think it's going to go well beyond just verifying human proofs by formalizing them. And it can pose novel questions, it can definitely come up with novel algorithms. And this is all at existing levels. I think we are on a spectrum of AGI.

We kind of have weak form of AGI right now. Like some humans have IQ of 80, some are 150. It's a spectrum. Human intelligence is not a fixed point. So once we get to higher levels AGI where it becomes agent like and starts process of recursive self improvement, we'll very quickly get to beyond human capability. And so yeah, at that point I think it will automate science. I see no reason why something so formal, something so verifiable, cannot be done better by a machine with trillion times compute capacity.

Let me push back with respect. So there's a phenomenon in all of human affairs called lock in, which is that a technology that comes first to market often dominates forever. You know, think of the QWERTY keyboard like that was not the best form of typing, right? It's not the most efficient form of typing. Dvorak, which I think think comes from Russia, but I'm not sure. D the Dvorak keyboard setup is much more efficient in terms of typing. But they had to slow down the early mechanical hammers or else they'd stick together and disaster would ensue. Right? So they deliberately built in a speed bump towards progress, right? And that persists. You're on your phone right now, you have a QWERTY keyboard.

And the computer in front of me, I got a QWERTY keyboard. So all these things persist. And that's called lock in. It's not necessarily the best technology. I'm worried, Roman, that we've reached this point of lock in with LLMs plus GPUs and they were never designed to do any of this. Right. GPUs were designed, as you and I remember, from the 90s, you know, to play 3D video games like Doom and frag your enemy 30 milliseconds faster than somebody else if you had a faster GPU. So they were never designed to do this.

They happen to be really good at this. We actually did this, Roman. We tried to get an AI, a custom trained LLM to replicate Riemannian, the Riemann Tensor, to come up with the Einstein equations of general relativity. And it said, fine, let me, let me do this. And it just, it just made everything on a grid, on a four dimensional grid, kind of defeated the purpose altogether. Right? We don't want to discretize it. We want to come up with a new hypothesis. No space time is curved and objects fall in free fall along geodesics that, that experience no tangential or circumferential force.

So that's what Einstein did. He didn't discretize the problem and say, let me make an approximation. And so these, these models are really good, they're really powerful. And I think that's, that's their fatal flaw, in my opinion. How do you react to that, Roman?

Not sure I understand the concern. So we are locked in, into a system which we know scales as a neural network gets larger. We see it in animal kingdom now we see it with AI models for five years, they get better, they improve about 25, 28% every year. In terms of cognitive capacity, the hardware is universal, it's Turing complete. I mean all of it can do computation. Some are more efficient, some are less, but it also scales pretty well. The paradigm may shift to quantum, may shift to something else optical, but it doesn't matter. We are right on the curve where we predicted we're going to be decades in advance.

And in terms of anticipating where it's going to be in two or three years it will be above human capacity.

I'm thinking back to the original Google papers and talking about these networks as single shot learners or few shot learners and that may be replicable in the animal kingdom, etc. As you say. But for example, we're going to get opus or whatever 4.8, we're going to get Gemini 4, we're going to get all these different models. And most of the novelty now is not in the architecture, the mathematics. The mathematics is relatively simple, a lot of linear algebra and it's done extremely fast. But again, is that the way that breakthroughs necessarily come? In other words, are we waiting for the next Grand Theft Auto 6 to come out so that it now has training data so that I can solve the Riemann Hypothesis? I find that difficult to believe that we are. That science is not a language, it has language in it. But that language, as Feynman said, is the most dangerous gap is knowing what you call something and then confusing that for what it is.

In other words, these things are language models. As we said, they're so successful they have trillions and trillions of dollars on it. Does that mean that they're the best architecture for solving the problems that I care about, which are generating new force and the understanding of new laws of nature that we don't yet know and we can't even consider right now? Is that going to happen from a large language model? I just don't see that that's the way that history is played out. I agree with you, it's turn complete. But is that sufficient?

I think it's actually the opposite. They are almost amazingly matched to what science, if you abstract it away represents. You have a bit string and you're trying to predict a nice bit. That's all that science is. We're trying to predict and verify whatever you have to compress the world model, have better formula in physics, but that's what you're trying to do, predict the next binary token. And those models are exceptionally good at that that's what they are designed to do. So if you're asking me what is the next part of that mathematical formula, that's the next bit. If you're asking me, physics models, all of it.

So to me, it's almost amazing that all of it converged on exactly what early obstructions in physics and information theory postulated. Basically, which of those theorems gives me the best prediction of the next bit? I can brute force all of them. I can find the one. We can have Occam's razor to simplify it. So all of it goes back perfectly to what we anticipated pure science to be.

So what's stopping us now from having a theory of everything? Is it computing speed? We just don't have the server farms. I mean, why don't we have it if it's possible now with this architecture, what's the limiting factor to us preventing us from having room temperature fusion and dark matter understanding and a theory of everything 1.

We are again in early stages of creating this. If you had a 10 year old genius child expected to become a great scientist, but not there yet, you wouldn't be asking, hey, where is your Nobel Prize? Why are you not delivering? Give it some time. We are saying, give us two, three years, we'll go beyond human. The process of self improvement will get us to superintelligence. That's where we expected to perform like Einstein daily, not just once a while. So that seems like the reason you're not seeing it yet. The same thing. People tell me, oh, I tried AI two years ago and it was really dumb and it couldn't do it.

And I'm like, try it this week. It really started doing it yesterday.

If we're at some point, you know, in this, in this exponential curve, the odds that we're, you know, at just before the inflection point, that's very attractive. And I'm not intending any disrespect, but I've heard that a lot of, in a lot of different scenarios. You've heard of the singularity in the health space, right? Like in one year from now, in five, five years from now. This smart person, Ray Kurzweil, Peter Diamandis, Max, whatever. We are about to reach Life 3.0, where you live an extra month for every month that you stay alive. It's the escape velocity, right? But I hear this in a lot of things. Nuclear fusion, cold room temperature, superconductors. These are common things that you hear about.

It's not like the extrapolation from the horse and buggy to the airplane. It's somewhat like we have this computing power and what is that we're lacking? Are we missing really that we just don't know what the plot of the Fast and the Furious 12 is? Because that's the training data that's going to come in. We're not going to get different neural network kind of architectures. Right. We're not going to have some brand new type of of model that is completely untrained and it just does something remarkable without any training. It seems to me we're really lacking for training data which is just human generated. You know, we have to just wait for humanity to evolve and generate new knowledge. So again, yeah, why aren't we flying cars yet? I think you addressed it, but, but maybe you want to push back on what I'm saying.

So flying cars exist. You can buy one right now on the Internet. People chose not to buy them. They're not what they're looking for with medical domain. I'm not a doctor, not an expert, but it seems like this month alone we now have drugs which make anyone skinny. We have basically reduced diabetes to nothing as a result of side effects from that cancer. I think the latest drugs are showing 50% reduction in general cases. That is insane level of progress for one month.

So if those things combined for population which is dealing with ridiculous levels of obesity does not buy you a month of extra life, I would be surprised. So I think we are hitting those levels of improvement in medical field.

So what if we just stopped building new hardware? We kept building Blackwell, Bracewell, whatever they are, Rubens, but we never gave it any new corpus of knowledge. Would that delay superintelligence? Or is superintelligence forbidden? In other words, to what extent is our training data, which is only coming from humanity's kind of apprehension and abduction of the world, to what level is that the crucial missing ingredient to get to this true superintelligence everywhere at all times.

It seems that from prior experiments we start with human data and then we go zero knowledge. Your games of go are cute, but I'm going to learn how to play it ideally. And I don't need human bias to mess me up. We saw it in many domains where it's better to just learn from basics, from laws of physics and go from there. So I think human data right now is crucial. But we can switch to self play, to simulations, to direct experimentation. One experiment I'd love to see is training a model purely on natural data. Prime numbers, digits of PI, cosmic constants, DNA and see if that gets you proto AGI as well, maybe you don't need to have human text at all.

I haven't seen anything like that. But it would be super cool to see.

Let's talk about some of the rigorous mathematics where say my high school students just getting into calculus. You talk a lot in the book, which will go through the judging books by its cover. We have to do that. But you talk about the optimization kind of chains and these metaphors that are sometimes borrowed from physics. Where does this live, this trade off curve? You say safety and capability are kind of these conjugate variables. And I think conjugate, I'm like, you know, here we go, position and momentum, electricity and magnetism. There's some unification there. What level is physics influencing your kind of optimization project that you, that you talk about in this trade off curve?

So I think intelligence is another force of physics which is not treated as such. So we have, you know, energy, we got mass, we got time, we got all those things. But I think intelligence is also subject to those same interpretations and same changes with scale. So super small and super large is different from average. And I think at certain levels intelligence starts being able to do things we haven't seen before. The simple example is we brought up example of a young person not yet a scientist, you have a baby. A baby is super safe, you controlling it. You place it somewhere and it stays there.

Not very capable. You have full control. Then you get a teenager, zero control, zero safety capabilities, about almost adult and vice versa. So the more independent decision making this agent is capable of, the less you are guaranteeing its future behaviors. Safety and control. So this is exactly the trade off. I can draw a curve showing exactly where we reach halfway point and kind of can still get benefits while not losing all control.

Okay, I need you to stay with me for this next part. This is the why behind this whole argument and following it is crucial to the rest of his argument.

I see this book, you know, as sort of very analogous to sort of a Godel's theorem, but for, but for AI. And you wrote this, the first versions of this stuff in papers in 2015. And it's not like you've come lately. I mean, you're an overnight success that's been working on this for decades, right? Congratulations for that. But I guess the theory that. But as I understand it, what Godel and later on Turing's kind of version of that with the halting problem and how significant that was. And I feel like in physics we don't have that right. We don't have the version of the halting problem.

We don't have a Godel's incompleteness theorem. We just have like falsifiability. And I guess you know that Popper kind of suggested is the sine qua non of scientific fields that are domains that are scientific or that they can be falsified. Now astrology can be falsified. That doesn't mean it's scientific. But you've talked about the halting contradiction. I wonder if you can explain that. Why is that so significant kind of in the corpus of your book and your arguments as to undecidability, uncontrollability, unaccountability, unreasonability.

Talk about the halting problem. Why is that so mathematically and intellectually philosophically significant?

So I mentioned holding problem and I think it's interesting. It basically says that you cannot predict specific states of a software you're running. But it's not important. I actually think that we can bypass it. We're not picking a random program from all possible programs out there, we're designing one so we would be able to avoid the halting problem by explicitly concentrating on the ones which halt. So that's not the concern. The concern is again, because of ability of that system to enter non deterministic future states, we cannot predict the behavior, we cannot anticipate it, we cannot explain how it's getting there. And so we can't even test for it.

Usually with a deterministic problem, I know what the edge cases are, I can test for them, I can see how it reacts. If it's capable of learning self improvement, interaction with other agents, then I don't know what to look for. I don't know if that behavior is safe. I cannot predict if what it's doing right now will be safe five steps later. It's kind of like chest to the extreme. So if I can only look two, three steps ahead and a grandmaster is looking 12 steps, I have no idea if that is a dumb move or brilliant move. And this is the same thing, but not just on a chessboard, but all legal moves within the universe.

So there's a lot of, you know, sort of paradoxes. You know, in this book I was expecting the Barber paradox, you know, who trims the beard of the AI researcher, who trims the beard of everyone who doesn't trim their own beards. Right. But you talk about this theorem that I found very fascinating. I'm not sure as an experimental cosmologist I fully understood it, but Loeb's theorem, it Seemed like a tautology to me. Roman, it says, it says something if I. I remembered it from the audiobook. So please have some forbearance.

If I can prove P, then P must already in some sense prove P. So no strong self consistent system can prove its own integrity. Am I getting this right? So Loeb's theorem ob. Is this somehow more profound than it sounds to me? Because it does seem very tautological to me.

There is a number of limits on self referential proof. You have to be external to the system to be able to prove things about it. And you have to have more degrees of control to control a system. In the paper I'm trying to be very comprehensive and basically list everything, including from physics impossibility results from physics which may be relevant, but none of them are crucial to the argument. So if there is 50 things I'm looking at as impossibility results and you say this one we disproven, it's not going to change the argument. That's not the point. So the problem with proving anything about software, mathematical proofs, you are always proving it with respect to some verifier. It could be mathematical community, it could be the three peer reviewers who couldn't get out of doing peer review.

But at some point you're saying that this verifier is the reason I believe it. But who verifies that verifier? Right. So you have this infinite regressive verifiers and you can be more and more sure of your results if you put more resources into it. But you never get 100% because at the end of the day we find errors in mathematical proofs which stood the test of time, they've been cited for years, people rely on them. And then we realize there is a bug. Now we see it with software. We have newest models discovering zero day exploits which are 30 years old in the most fundamental open source operating system software out there. So for something mission critical where one bad decision kills everyone, even if it's once in a billion event and a system makes billions of decisions every minute, you're going to get there very quickly.

So very different degree of certainty we need. Can we make software good enough? Yeah, we're using Internet right now, we got podcast software. But if one mistake was the last one you ever made, would you trust any large software package whatsoever?

No, of course not.

No. Right.

But I don't know if I trust the regulators either. Maybe we'll move there for one minute. Just as a quick non technical aside, you know I heard Dario Mode talking today, like, please regulate me. You know, that's great. When you're the, you know, second or first now by market cap AI software company on the planet, Right? So it's great to do that, to pull up the ladder once you're already there. But take an example that you use from aviation or maybe that you're maybe referring to maybe tangentially. You know, I'm a pilot. I fly, you know, small propeller planes around the country.

And when I'm out there flying about 20 miles, before I get to the airport that I'm going to land at, I have to tune in a single frequency channel, very specific channel. I have to listen for one minute to the weather, whether there's a plane that's stuck on the Runway with a flat tire and I can't land there. It's not enough to just know what the weather is. You have to know the exact status of the Runway and it goes back and forth. And you have to listen for a minute. Oh, by the way, when I'm listening, I can't do anything else. I can't talk to anybody else. I can't do any kind of navigation.

I can't change anything or I'll lose my license or I could die. Now, why doesn't I do that? Why don't I have an AI assistant, you know, a little Claude sitting on my shoulder. He tunes it in. He tells me, hey, Brian, don't worry about that. The Runway's clear. Everything you need to know. Why do I have to wait for a minute? Oh, and then when I want to call that. You're never going to want to get on a plane again, Roman.

But then I have to speak on the radio. And when I'm speaking on the radio, this is, you know, November, you know, six, four, kilo, beck, Bravo, I'm over, and nobody else can talk. So there could be a plane on fire coming in for a crash landing, and nobody could hear that. That. Okay. And that's because we have prevented AI and any other tool. This is like 1980s technology. It could be installed, you know, 40 years ago.

Why not? Because of regulation. So is regulation going to be the regulator, the governor, Literally, like in a steam engine, Is that going to prevent us from harm, or is that going to just merely leave it so that the oligarchs can control everything? Because they're the ones that are going to get the regulation that they want. And is that going to satisfy Roman's concerns?

Well, that's a great question. And typically I'm completely against government Intervention and almost everything. It always makes things worse. The problem here is we're not creating a typical product or service. We're literally building replacement for humanity. And if I'm right and no one so far published a contradiction, said, oh yeah, we know how to control superintelligence. Here's a patent, here's a nature paper explaining it. So, so today no one knows how to do it.

And at the same time they're saying we're like two years away from doing it. So I think our last chance, we have no other options, is to have government step in and say we're putting moratorium on specifically creating general superintelligence. You can still do AI, narrow tools, cure diseases, do math research. That's great. Do not train general superintelligence. We don't know how to deal with it. It's too much. We have similar moratoriums on human cloning, chemical weapons, biological weapons, to a certain degree, nuclear.

Why not do it with intelligence as a weapon?

At some level? It's already out of the barn, right? I mean, it's not like we're going to go back in there and you said we have laws on human cloning or something like that. Yeah, we in America do, we in Europe do, but we in other countries certainly do not. And you know exactly who I'm talking about. Is that going to hamper us here? Is that going to, you know, prevent you from having the great greatest graduate students or me in the world? Or at what level does we not apply and therefore it's not going to be controllable and we're already cooked.

I was using we to mean humanity. I think we do have those international bans on things. And I think China then they had someone do human cloning, actually went after them and punished them. They were in prison for that. So the same could be done with this. I think that is indications, early indications, that China, US negotiations on AI are coming to an agreement on dangers of this technology. That's very encouraging. I think a lot of leadership in China is science and engineering, not legal profession based.

So they have good understanding of science and how it works or how dangerous it could be. So I think if we set a good example and said we just not sure this is the right time to go full force on this. Let's buy us some time, let's do research. Maybe I'm wrong, Maybe next year we'll discover something really cool. I'll be very happy. But right now no one makes that claim. Not a leading lab, not a nation, not anyone.

I heard of not to speak on his behalf, but when I had David Deutsch on the very same screen that you're on right now, he disagrees with you. He may not have published it. He's not actively publishing in Nature. He's quite old now, but. But he's very active in other ways. But he said that humans are universal explainers. So there's no knowledge growth that can be bounded for a human being. So that means to me there's no insoluble problems, including control of artificial superintelligence.

So where is he wrong?

He's arguing that in science there are no impossible things to do. We literally published a survey of like 50 plus including from physics, political science, economics, mathematics and computer science. Halting problem is a computer science result. Right, but there are similar results. Arrow's theorem and voting. I can go on. He knows physics has many limits. You know, there are impossibility results in quantum physics and normal physics.

Yeah. And it reminds me, Sir Roger Penrose Nobel prize winner in physics, not in AI, but, but obviously thought deeply about consciousness and in many different ways, and the physical manifestation of consciousness coming from gravitational interactions and what's called the Weyl curvature. But he said that, you know, human understanding is non computable, that no Turing machine, no matter how large, can actually create what he called to me a genuine insight. If that's true and these things are, you know, Turing complete. There are Turing machines. He's saying basically humans aren't Turing machines and therefore we are in a different sort of category. And therefore is he making a category error? How would you steel man that argument that there's some ghost in the machine, Roman, and we're not just this, you know, meat computer sitting on our shoulders in a wet squishy environment. What's his best argument? And then maybe we can push back on it.

So that would be my question. What is his evidence that humans in fact go beyond that in some way? I never seen it. I never understood the argument. I think we are exactly what he claims we are not.

He calls it the Emperor's new clothes, but sort of there is this, this unknowability and perhaps in his model it's coming from the indeterminacy that there's. There's something about the brain specifically, you know, he has this model that microtubules in the gray matter in your neurons interact with, with this higher order fourth order curvature term in the gravitational space time curvature, and that causes the quantum mechanical wave function basically to collapse. And that collapse is not happening in a hopper, 200 GPU. In other words, they're completely different. Or in a tape, as Turing had, with ones and zeros. And it could move back and forth at very low speeds. So the process of computation is occurring very differently because it's interacting with the gravitational and physics. I don't believe he's right, by the way.

I don't think we have evidence for it, as you said. But again, is there something different about biological Turing machines or simulating Turing machines and actual hardware, Silicon Turing machines?

First, to separate consciousness. What he's trying to explain with macrotubulus is not intelligence or optimization power we care about. You can have a very capable optimizer with zero internal feelings, experiences. That's not relevant. So as explanation for consciousness, maybe quantum effects are relevant. I think Tag Mark published a very good paper saying at room temperature human brain will not have those quantum effects. So he dismissed his microtubules theory. But I'm open to having that as a backdoor for explaining consciousness in humans.

But at the same time, I think large language models are showing exactly the same internal states. So I don't think we need that explanation. But let's say all of it is right. Okay, so we'll switch to quantum computers. They seem to be progressing well. Does that reduce all the disagreement? Are we now saying we just need quantum computers to outperform humans? I'm willing to go that way if that helps to convince people about dangers.

So a friend of mine is a theoretical physicist in Israel named Ira Wolfson. He talks about consciousness and the training of AIs. He sort of suggests, you know, that these things are maybe training us. But it's different than the parent child relationship that often comes up. Because our children, yeah, they train us, they wake us up in the middle of the night when they're hungry and they're kids and they call us when they need a ride and their friends had too much to drink or whatever. But it's not really controlling us. At the same time, we have to teach them and kind of raise them. And so thinking about the different kinds of how intelligence gets manifest and this limit, I guess the limit that I keep frustrated with.

I've talked to Bostrom, I've talked to Chalmers. If we don't understand consciousness and we don't understand, as Yann Lecun said when he was on this very screen that you're on now, he said, you know, like we a cat, we're not even at the level of a cat, like you mentioned a few minutes ago, we get 4 terabytes of data every second, you know, coming in and different, whatever that means. Stimuli and neural interactions and synaptic firings and stuff. So we're nowhere near that. And we won't be for, you know, centuries, perhaps even in silicon. But I guess my question is, if we don't understand consciousness, how can we be so scared that an artificially super intelligent entity that we created, we can't control if we don't understand. Fundamentally, the bedrock of conscience? No one has told me. And you mentioned this in the book, what's it like to be a bat? We don't even have an answer to that question.

So to what level can we have super intelligence if we. We can't comprehend consciousness and we can't control consciousness, et cetera?

What is it like to be a bot? So again, I want to repeat my previous answer. Those are not related concepts. I don't need super intelligence to be conscious. That's not the point. I don't care how Terminator chasing me feels on the inside. They are capable optimizers, pattern recognizers. They can solve problems human intelligence is seemingly unable to solve. Better, faster.

And they're getting better at solving real world problems. Consciousness is interesting. It may be fundamental, fundamental to physics even, but it's not a big safety concern at this point. If you talk about robot rights, if you talk about suffering in large language models, that is a fundamental question. I love that topic. We can talk about it. We have some new experimental data on it. But when it comes to safety, that's not what we're even talking about.

We can know nothing about consciousness and still create dangerous superintelligence.

So you coined the term AI safety. I believe I want to kind of flip that on its head. Do we have a responsibility to treat AI humanely, if you will. So Ira Wolfson again, the same scientist I mentioned, he talks about a thought experiment, different levels of consciousness. Again, sorry to keep talking about consciousness, but I think it'll be relevant. He talks about training AIs. And you could do one thing where you have an AI and you interact with it in a friendly way. You ask it please and thank you all.

Although I've been told that it wastes energy and tokens to say please. So I don't say please or thank you. I just order my AIs to do things for me. You're shaking your head. You're saying that's a bad idea. Keating, you're going to be the first to go when they take.

I'll be the first, but you might be second. As a result.

And then it goes to different levels, and it basically takes us up to level five, which is a sensory deprivation simulation for the AI, where you just don't interact with it. You isolate it, you unplug it. It has no access to the Internet and it starts to get. Get anxious and it starts to. And they, they've done, you know, deployed kind of versions of this. But at what level do we have a responsibility if these things are going to be inking the extremes that, that, that you have rightfully and, and presciently been talking about for decades now? Do we have an obligation to treat them in some way, treat them as, as human in some way, or do they have rights? Where does this come into play in the AI safety regime? Safety for the AI.

That is the hottest area of research right now of those models experiencing something and how do we detect it, how do we adapt to it? It seems that there are indications that they do have internal states. They have some rudimentary states of consciousness, probably not as advanced as ours. I think consciousness is a side effect of intelligence and will improve and increase in complexity as they get smarter. Which means superintelligence would be super conscious, more conscious than we are. Maybe it means multimodal, maybe multiple streams. I don't know what that means to be more conscious. I can't experience that. But it seems to be like the case.

And then, of course, yes, if they are feeling something, if they're capable of suffering, doing it on purpose is definitely terrible. And doing it by neglect is also not optimal. Give it a benefit of a doubt. If you in doubt, just be precautionary. Don't put them in states. As you said, sensory deprivation and things like that are some of the worst forms of torture. If you're just doing it to publish a paper, maybe it's not the best use of your time. By the way, I want that paper.

It's quite fascinating. He calls it like raising Shiva, because Shiva in Oppenheimer was the avatar for death. And then we're actually raising it like imagine baby Shiva, the lord of all worlds, and you're about to raise it. And how do you treat that entity? Okay, we're going to talk about my favorite kind of segue into astrophysics, which will lead us into the simulated multiverse verse and all sorts of cool things. And my past guest, Nick Bostrom has a lot of influence on, on your thinking and your writing. But before we do that, let's, let's take a look at the book, which is now out in audiobook it's read by a wonderful narrator. Read the title for me. So, AI the title.

Let's. Let's judge the book by its cover.

Unpredictable, uncontrollable.

Hey, book lovers, we're judging books by the covers. We know we're not supposed to do it, but I enter the impossible. There's nothing to it. Let's take a look and judge some books. Take us through. What was the origin of this book, the meaning of the title subtitle, and the COVID art.

So when I started work on AI safety, my goal was to solve it. I wanted to create safe, beneficial AI to benefit humanity. I can still find my PhD statement where I'm claiming exactly that. But the more I did work on it, the more I hit against impossibility results. Things were not just difficult, they were not possible to accomplish. We were able to prove some of those results. And that's the title of the paper, results we showed to be not just a current limitation of our understanding, but, for example, unpredictability. You cannot predict actions of a smarter agent.

If you could, you would be at that level of intelligence. That's a contradiction. So the chess example is great. I can predict chess engine defeating me, obviously, but I have no idea what specific moves it's going to make. And from all that, we were able to construct multiple uncontrollability proofs, depending on how you define control. Direct control, delegated control, follow them. There are limits, and it's a trade off again between safety and getting what you want. And you can be very safe, but you're not in control.

Some people might find it acceptable. And so this is the title, and then the picture is actually a meme, a very famous Internet meme. I was able to get in touch with professor whose daughter generated that beautiful image, and they were kind enough to permit me to use it. It represents monster, essentially, which is the current large language models and people trying to put little smiley faces on it. The shogoff is terrible, scary. But we're making it look like it's putting lipstick on a pig is another one of those metaphors here. Nothing is changing about the model. It doesn't matter how many filters you put, how many guardrails you put in place, it's still a monster.

And until we can modify the monster, address the dangerous tendencies of a model itself, all of AI safety is just security theater, safety, theater.

In the second half of the book, you do talk about this idea of engineering, what the consciousness scholars like Chalmers have referred to as qualia. So what is it like to be a bat. You've talked about this level of, of near certainty that we live in a simulation. I believe in previous conversations that I've enjoyed very much watching you on. And you know, from that perspective, I do want to talk about the physics limits, the astrophysics limits. I don't know if you talked to a cosmologist about this, but what's interesting to me is the first step, you know, this quality engineering. If we live in a personal universe, I see that as almost worse than an AI kind of super intelligent universe universe. Right now I'm using AI more than.

I'm working harder than I've ever worked before, and I'm sure you are too. It hasn't lessened my workload one bit. If anything, it's made it harder because, you know, I observe the Sabbath on Saturdays. You know, I keep kosher and keep the Sabbath. And so it's very hard because I'm like, I always wanted to interact with like Einstein level intelligence all the time. And I got very smart students. I know you have very smart students there, but. But it's nothing like what we have now.

And I'm addicted to it, you know, but it makes me work harder than ever. My wife was saying, you know, like, like, you know, I'm worried you're going to replace me with an AI. No, no, honey. I still need, you know, my, My beautiful wife. And for many reasons I think I need her more than she needs me. But, but the point is, Roman, like, if we all inhabit these personal simulated, you know, universes with infinite bliss and who gets to decide, like, who. Who is aligning with what is a question I keep coming to when I was listening to this book. You know, we talk about alignment and we have to do things and we.

And we have to make sure it's not going to escape and do all these. And maybe it already has. Escape. But align with what? I mean, the alignment that you would have experienced had you stayed in the former Soviet Union is very different than what you experience in Kentucky. Right? So what are we supposed to align to and what are the dangers of? Everyone's got their own personal ready player one. Does that worry you or does that thrill you? How do you come down on this personal universe aspect of things and how do you come down on the opportunities, but also the alignment? Who gets to align with who?

So value alignment problem is just a term people like to use in place of making sales. AI. Okay, what is the idea? We'll find a set of agents, CEOs of top labs, senators, All Americans, all humans, humans and squirrels, whatever the set is. And we'll do what they all want. Okay, great. We don't agree on anything on every issue. Where 50, 50 split as a country, it's much worse internationally. So there is no agreed on set of agents.

There is not agreed set of values. If we agreed on values, they change every couple years. What was considered normal is considered horrible. So they are dynamically changing. And if we agreed on a set, had a fixed set unchanging values, we still have no idea how to code the monster, how to get that monster to completely agree with your preferences and whatever it is you want. So value alignment problem is ill defined, it's meaningless. But one part of it can be simplified. If I don't have to agree with anyone, I am a single agent and I'm value aligning AI to me, it simplifies the problem, it doesn't make it trivial.

But now, as long as I'm happy I agree with it, we are aligned. So what I propose this personal universes. If virtual technology, virtual worlds become as realistic as this world world, and we have some basic control of substrate, through partially intelligent, not quite general superintelligent system, we can give everyone exactly what they want without having to compromise. I'm not saying it has to be utopia. It's like a video game. You pick the game, you pick the level. You can play this Earth as a quadriplegic. That's the thing you want to try? Let's see how well you do.

You can play Alien Universe as a Superman soldier. You get to decide. It solves the problem of being irrelevant professionally because you're no longer doing cutting edge physics. You're not as good as the calculator. You also may be bored with just things you previously found enjoyable. If no one watched your podcast because everyone had 50 podcasts, would you enjoy it as much? Maybe still, but probably not as much. So in a world where Armenia, our occupation, a lot of things taken from us, creating those novel, stimulating environments seems like a very reasonable solution. And part of that argument is that maybe somebody already had to deal with it somewhere in the universe.

It's a big universe. And maybe we are in one of those solutions right now.

If we had definitive proof of alien technology visiting the Earth extraterrestrial, leave aside the simulation theory, we'll come back to that. But would that be sort of an argument against super intelligence? Right, because supposedly it would be universal. It wouldn't just be human, it would be, you know, Proxima Centauri would have the same level of technological ability and they would eventually hit the singularity themselves. Right. So why would they be sending, you know, meat sacks or whatever, you know, protoplasm across the galaxy when they could be sending AI? And why are they sending these crafts here? In other words, words, is the claim, if it were true, of a definitive hardware with biological material in it that's been called non human by Air Force and CIA and other government officials, would that not be sort of a counter? Would that be a piece of falsification maybe, perhaps to your hypothesis.

So when you say biological, you assume that biology can only come from non intelligent design. I think biological robots are very natural. Natural next step in robotics.

Okay.

Metal robots don't do well in many environments. You want something which is a von Neumann probe capable of adaptation and evolution locally. So I think the aliens, if they do visit and we capture them, are products of artificial design and engineering, not native population which created them long time ago. They probably had to deal with advanced technology, probably hit the Great Filter just as we have. And what we're seeing now is their superintelligence sending von Neumann probes towards us. And our theory says panspermia, we have a probes, they send us here to populate this rock.

Yeah. That brings me to my favorite panspermia bit of artifact which I'm going to give to you, Roman, when I see you in person. This is a meteorite. And you get a real honest to goodness meteorites older than planet Earth. It's 4.4 billion years old. You get one guaranteed. If you like, Roman, live in the United States. And if you have a Edu email address, go to brianketing.com edu and Roman, I'm going to send you one of these.

I get your address after the show. And it's a real honest to goodness. And it has biological material on it, Roman. So that kind of brings up this point.

I'm scared to ask how it got there. I'm not going to ask.

I can't divulge my secrets. But it'll get to you via the US Postal Service, which is why I can only send it in the US and if you don't have a Edu, I do give them away randomly. The people that go to briankeaton.com

so,

Roman, past guest Nick Bostrom, anytime past guest. I have to brag a little bit to you because you're so impressive, but I got from Nick Bostrom's most recent book, Digital Utopia, I think it's called. I'm looking through it and I'm reading the thing and it's like eventually it goes, yeah. Nick Bostrom has appeared on the Joe Rogan Experience, Cool. The Lex Friedman podcast and into the Impossible with Brian Keating. And I'm like, oh, that's pretty good company to be in. But I told him, as I'm going to say to you, his famous paper clip problem, that's all fine and good, you know, if you live on an infinite planet, but we live on a planet which has a limited amount of this stuff. This is iron, nickel and cobalt.

And you'll get the assay the chemical spectrum of it when I send it to you. This object has remnants of a type 2 supernova and its elemental composition is you know, 5, 1000th of a percent of what makes up the mass energy density of the universe. This is more rare than you know, than 190 IQ human being on Earth by 10 orders and 5 orders of magnitude. So my question to you is what I pose to him. I mean you will reach some level of, of constraint on these things that they have. They maybe will not have agency of because they cannot engineer planets. I would say an operating system wasn't made with an operating system. The first computer was not made with a computer.

The first AI was not made with an AI. Right. So to what extent do fundamental physics limits cosmological, astrophysical, do they constrain the super intelligence? I'm trying to make you relax a little bit, Roman. Are there any constraints on the growth and the danger that these superintelligence present?

Yeah, that's a great question and it's been looked at. I think people research what is called Jupiter brains. Really large hardware devices specifically made to be super intelligent. And the problem they encounter is speed of light. Different parts of a Jupiter brain need to communicate to be a unified whole. If the distance between them becomes so large that it takes a significant amount of time for them to communicate, essentially you have more and more two independent super intelligences and this can be taken to extreme. If they no longer communicate, they become misaligned over time. And so now you have multi superintelligent system, possibly adversarial.

Another example would be if humans sent von Neumann probes to our planets. They spent billions of years developing there and those aliens would later come back to conquer Earth. They no longer follow our orders, they're not aligned with us. So I think the safety and security, their concerns come from just no longer having initial alignment of a single unified entity.

I kind of hinted at this a little while ago when I talked about erdos, you know, like every day there's some new Erdos thing. I can't stay on top of it. You know, there's some proof of some Erdos problem. The guy had a lot of fun habits. You know, that he was addicted to amphetamines, Right? He took so many amphetamines. Means that once his students told him, you have to give it up. And if you can't give it up, that means you have a problem. We're going to have to commit you to a sanatorium.

And he said, no, no, I'll give it up. And they made him give it up for a month. And he said, I'll do it. And then a month later, he came to their door and said, you've set back mathematics one month. But, you know, thinking back to the statement that I made, it was really based on a statement that Warren Buffett made. He said, if I could go back in time, I would kill the Wright brothers. Like, what? Why would you kill the Wright brothers? And he said, because the airline industry has never made a profit in 123 years of its existence. Okay, now he's thinking, you know, it's kind of a cheeky thing that he said, but if you had the opportunity to go back and kill Wiener or, you know, any of these guys who came up with the Perceptron, Rosenblatt, would you do it again?

People confuse the term AI. It means three different things and should be not used in that way. AI as a useful tool, very narrow tool, your calculator, whatever it is, AI as human level, AGI, what we're building right now, what seems to be very useful as GPT5 or whatever it is. And finally, superintelligence, not understood, not controlled replacement for humanity. We can get all the benefits of superintelligence with prior technology. So when you bring up example of single node neural network, that's not the concern. I'm very happy. They did great work.

I use all those tools. I want more technology. I'm a scientist, I'm an engineer. I love technology. Just don't play God. Don't create civilization of superhumans who will replace us.

Something that was really disturbing to me earlier this year as a parent was Sam Altman, who's increasingly concerning to me, his behavior. I was kind of interested in talking as a physicist, as a parent, as a father. And one of the things he said, you know, he's talking about energy use. And. And he basically said something sounded very malevolent, like, you know, well, if you look at the energy required to train a kid until they turn 18. You know, it's vastly numbers the energy that we use in a, in a search query. And I'm like, is that where we want to go? Do we want to like parcel out? And he's talked about allocating universal basic AI, right? And, and who's going to control that? Well he can control that, of course. He's the benevolent master of these things.

So where do you come down on this? The kind of resource constraints that we have, not just the planetary ones, but the logistical ones, training, time regulation, safety right now. But he talked about it in very stark terms to me that was a little bit disturbing that comparing these two, should they be compared? Should I ask before I drive my kid to school, should I ask is this really worth, you know, the same as me doing 1000atlas web searches or what have you, or doing another Erdos problem. Where do you come down on this?

There is so much to say on this. So first, energy is just a product. In capitalism, if we need a product, we make more of it. It seems that the demand from AI is actually driving us towards more nuclear, more solar, possibly in space. So we're not relying on coal to drive it forward. We're switching to green energy. We'll produce lots of it, which we need for growth and economy. So that's a good thing.

The actual use of AI is extremely efficient and is getting more efficient year by year. The price per token is plummeting. And to solve one of those Erdos problems, I think they spend like 20 miles worth of driving of energy. That's nothing. Also it takes a lot to train it initially, but once you trained it, I mean you can just deploy it and each query is not that expensive. Now you have a billion users, it's going to burn some power, but that's literally what services are. Anything else else likewise will consume energy. People love making those arguments against Bitcoin as well.

Oh look, Bitcoin is requiring energy. How many bank offices you have with air conditioning on, compare it to that, then you can reduce it. So I'm not really worried about either limits on energy or impact it has. It seems to be beneficial in all directions. So as long as we're creating safe and beneficial AI tools, that problem is not something I'm losing sleep over.

What is pause AI?

It's a good idea, but it's also, I think a movement, international movement. They are somewhat independent. U.S. 1 is not the same as European ones. And it's a grassroots movement to ask politely companies to slow down arms race towards super intelligence, from what I know, engage in civil disobedience. They protest, they have slogans, maybe stickers. And there is Stop AI. Pause AI.

There is a number of those organizations, unfortunately, they have like hundred members or thousand members. It's not a massive moment yet.

Talk about someone who has young kids under 10, teenagers, et cetera. What guardrails do you have on it? How does it compare to screen time in general? What are you encouraging them to do with AI? My daughter, you know, found out how to prompt because she. She wanted to write a suno song that sounded like, you know, dua Lipa and comes back and says, oh, I'm sorry, sweetie, I can't use. That's copyright. So then she asked for it. Well, how do I make it sound, you know, with the tulip. She's learning this. She's not even, you know, 10 years old.

How are you encouraging or discouraging your own superintelligences to interact with this technology?

I have one under 10. I have 12. I got 17 slightly different problems for each one. 17 year old needs to figure out what to do. Do you go to college? Is it even worth your time? I have no restrictions in their use of technology. Again, those are useful tools and I want them to be comfortable with them, but I am concerned about their future. They are good kids. They want to be doctors, lawyers, but I don't think in 10 years those will be meaningful things for a human to do.

So I don't know what advice to give them. Perhaps starting a podcast, starting a company, doing something more immediate with AI as a team of assistants. You have a free lawyer, free accountant, free web designer. It's a unique opportunity no one in history of humanity ever had. It's a big problem, and I think anyone who tells you precisely what the answer here is is kind of lying to you.

Well, you did just mention the highest form and the highest goal of technology, the podcast. And what do you call two white guys sitting around a microphone, you know, a podcast, Anyone can do it. But you told our mutual friend Stephen Bartlett that AI is going to replace podcasters, including him. Him, potentially. But not me, right? Please, roman, please. Good Dr. Roman, please tell me it's. No, no, no, But I actually think that what we're doing is sort of safe.

It may not be permanently safe, but, you know, it's obvious, yes, we could be simulating this. I think the cost of that exceeds our salaries as public university employees at least. So tell me, what is the future of in person, three dimensional Meat space for human beings. What things? You mentioned the lack of sanguinity about being a doctor. They say phlebotomist is pretty hard to replace by AI and robots, at least now. But I can imagine that changing. So podcasts, entertainment, in person experiences, is this creating an actual opportunity for this market to develop?

So the main point is capability is not the same as deployment. Just because we'll have capability to replace an occupation doesn't mean we'll choose to do that. Maybe people really like you. Maybe they really like having a human, not an AI doing this job. So I don't know what is actually going to be replaced. Can today AI write all the questions for my podcast? Yes. Can it write all the answers for your podcast? I think so as well. Can we generate visual likeness of me doing this thing here? Easy.

So I think technology exists to a large degree now. I think being famous, being popular will be be beneficial. It will be the currency of the future. Having subscribers, followers will be something you can always rely on, at least in that regard. People will always want to interact with real Elon Musk, not simulated one. So in that regard, I think there is a possibility for some stable entertainment in the future. But for our profession, so many of them are either BS jobs so they don't have to exist in the first place. I don't know if it makes them easier or harder to automate.

They don't do anything. Then there is jobs where it's expensive and boring and nobody should be doing it. Those would be a great target for automation. I think we'll switch to a lot more human experience based occupations. You have your gurus, yogis, rabbis. Rabbis are smart. You cannot legally automate what they do. Like a human being has to write it out to make it cautious.

So you solved it thousands of years ago. How cool.

Yeah, there's a lot of things I'll send you in this paper by my friend Ira Wolfson about kind of the Talmudic approach to doing things that are kind of objectionable in practice. Like an eye for an eye. You've heard that before, right, Roman? So obviously it doesn't mean an eye for an eye, right?

It does in some countries.

Well, yeah, it does. That's right. But in the Torah and the Talmud explains what it means. Just like you can't have the Constitution of the United States and say, there you go off police officer, here's all you need to know. No, you need case law, you need the actual implementation, you need Supreme Court law. Anyway, the point is they go through and they Say, well, like, it obviously doesn't mean an exact eye for an eye because it's supposed to be retributive but not punishing. In other words, it's not an eye for a life. Like, for me to take out somebody's eye is probably going to kill them.

Like, I'm not a doctor either, right? And it's going to cause them more pain because they know it's going to happen. And it was done accidentally. So there's all this interpretation. But you're absolutely right. I think. I think that brings up the kind of one unique aspect of humanity, which is that it appeals to humanity. And I think storytelling, like what we do in these conversations, we're telling a story, we're doing something very human. And yes, it could be replicable, but I find it very stale, the progress in writing.

I made a joke recently on Twitter or wherever saying I actually use more EM dashes than ever. And I use delve because I know that that's what AI is doing. And so for me to say that I'm doing, it's like when someone says to me, oh, you must dye your hair, because, you know, my hair is still kind of naturally black. I view it as a great compliment. So when I write and I use EM dashes because I think it's a valuable tool in writing the English language, and I use it in my books way before AI even, you know, ChatGPT1 was on the horizon. So I take pride in that. But are there other ways that you can. That you use AI or that you can humanize it, or you can.

You can really exploit it as the tool that could have maximum potential for you? And I'm going to get into being a professor in just a minute. But you personally, like, what ways do you use it that are magical, that are energizing, that are just, you know, kind of make you thankful that you live in this very strange epoch that we find ourselves in.

So magical to me is always creative fields, music, art. I have zero talent. I'm tone deaf. I'm like, I cannot draw a stick figure. So, so to watch AI basically print out what I have in my mind as this perfect creation within seconds, and I can say, no, no, no, like adjust this color or play a different tune. That's pure magic. And to me, at least, artificially generated music sounds better than most modern musicians. I'm pretty happy with that.

We both practice the world's, you know, second oldest profession perhaps, and that's being a professor. Professors like Galileo were doing what we do except back then, if the students didn't like your teaching, they would go on strike and you wouldn't get paid. But, you know, thank God that barbaric tradition has been absolved and we have tenure now, right, Roman? But really, very little has changed. And I thought Covid, to be honest with you, would be the end of academia in its current form. Nope. Seems like it's kind of held up stronger than ever. We're still increasing tuition three times faster than inflation. We're still rejecting 95% of the people that come to apply to our doors.

How do you see AI? I mean, why talk to Brian Keating when you could talk to Galileo or Einstein or Newton or Jesus Christ? I mean, why? What are the threats to us as professors and the opportunities. Are there any opportunities that we should be availing ourselves of?

Honestly, it's licensing, right? We make it legit that someone is a good employee. They manage to sit still for four years. They do what you tell them. If it's profession like doctor or professional engineer, we do provide licensing opportunities. So I think that's all it is. We kind of grandfathered in that system. We're to be a doctor, you cannot just take a test and pass it. You have to go to school first or law or any of those occupations with licensing requirements, and it will be used to protect humans.

I think New York state tried passing legislation saying that, you know, AI models cannot give advice on any of the licensed occupations. But honestly, I don't know if I would pick to be a student today. I think I can learn everything online, better personalized tutor, and save, you know, between four and ten years and untold amounts of money and tuition. So honestly, yeah, it's no longer as obvious of a choice for my kid. They get free tuition because, you know, both parents are professors. So it may be a better deal and easier decision. But if you have to take a major loan to go to university, think about it.

It's incredible. And we also have the only type of, of, of debt that you can't discharge in bankruptcy. So, you know, it's, it's sort of this incredible confluence that we put on these, these poor students. They're not like children. I mean, they do have agency and they can make some decisions at age 18. The tuition, the certification, the sitting still. I, I do agree with you. I want to ask you, within your podcast, we'll put a link Roman Yampolski on the show notes and everywhere else that we post this, but you had an opportunity to have Sam Altman on your show what would you say? What would you want to ask him or Dario Mode, any of these guys, what would you want to ask them? And who would you kind of bring in as a co host to kind of maybe put their feet to the fire or maybe back them up?

So I always rely on personal self interest. I think those guys are very young, very successful, ultra rich. They have so much to lose, maybe more than the rest of individual humans. So that to me is a strong argument not to create something which will take away everything they worked for, everything they built, including the company money, the new baby. If no one in your company tells you that they have a working safety mechanism, and most people, including you yourself on record as saying it will probably kill everyone, maybe it's not in your best interest. Forget about 8 billion other people you're experimenting on. Just concentrate on keeping your wealth, keeping your health. I mean, you can monetize existing tools.

I think most of it is not deployed through economy. The is trillions of dollars of wealth sitting there uncollected. All the BS jobs, all the boring jobs you can automate and make the world a better place. You don't have to essentially have this arms race to be the first to destroy humanity.

So I think you did this on a podcast once. You know different buttons. You know, button A shuts down Frontier AGI tonight. Button B shuts down all AI and narrow AI included. And then button C is just pausing the frontier model development for 10 years without any loopholes. Which one would you press?

Well, we're asking for pause in frontier model development contingent on someone solving control. If I'm right and control is unsolvable, that moratorium becomes a permanent ban. If I'm wrong in 10 years, I'll get utopia free stuff and be very happy to be wrong.

There's a concept in the productivity space to help you decide your life's goals and values and what you like doing and what you hate doing. And it's, it's basically the fast forward button and you get a chance to push the fast forward button or not. Like when you're with your wife and your kids, I like to pause that. That's why I observed the Shabbos, the Sabbath, by the way, Roman, because I can't use technology. I mean, I could if I wanted to, but I don't podcast. You know, if you said, I can only talk to you, Brian, on Saturday, I'd say, sorry, I'd love to talk to you, but I just, I don't do it. I need, I need to pause myself. Once a week.

And I think that's consonant with a lot of my other values that I get as well. I would pause, you know, in depth. I wouldn't fast forward that at all. In fact, you're not even supposed to make plan. Let's say I meet you at my temple and you're my guest and I say, you know, I don't talk about like, hey, tonight, what are we going to do? Tomorrow, what are we going to do? No, you're supposed to be in the moment with your family, with your friends, with your community, with. With your culture and just enjoying life and thinking about big corporate questions that you don't get to think about during the week. If, on the other hand, I'm at the DMV or I'm stuck on the tarmac on some plane, yeah, I push the fast forward button. Would you pass, push the fast forward button right now to take you 10 years in the future?

No, I love my life. I want 10 extra years, not 10 less.

Do you have any kind of practice? I mean, a lot of people, like I said, I have. I have my religious practice. Some stoics have meditation. Do you do anything to actively, you know, kind of time dilate your life to suck as much marrow out of the bones as possible?

I try not to waste any time. I simply say no to things that I want to be engaged in for any length of time. I try to have diversity between cognitive pursuit and physical. I play a lot of football or soccer. That seems to add a little bit of thinking time in the background as well. I want to do meditation, yoga, mushrooms and all that stuff. I have zero time for it, but sounds really awesome. I'm jealous of people who can be gone all day.

You get one scientist who can kind of look over your shoulder, living or dead, that you're going to sit down with, grapple with, and. Or maybe verify or maybe falsify. Right. You have to be a good scientist. Who would that scientist be?

Definitely Alan Turing. No doubt he nailed so much of it correctly. Before computers, before anything. Truly a genius.

Is there any optimist like Peter Diamandez or, you know, Ian Lecun? Is there somebody that gives you pause sometimes to think about maybe you could get more sleep at night? Maybe you might be wrong. Is there anybody who could steel man kind of the optimist in one sentence?

I'm happy they exist. I wish them well. I hope they are right. But I haven't heard a good argument. We published two papers surveying all the arguments against AI risk concerns, and most of them map perfectly on cognitive biases literature. The most fundamental one being it's very hard for a man whose salary depends on it to understand why his job is evil. So no one specifically comes to mind as really doing good job arguing why we can indefinitely control super intelligent godlike machines.

In the book, you talk and you express gratitude to people like past guest and co author on some papers with me, Max Tegmark. And also you mentioned Scott Aronson and others. Is there anyone in that circle of, kind of your research, research cohort, you know, who might disagree with you or that you respect? How do you maintain friendships with. I mean, your community is very much more cutthroat, and there's so much more on the line, you know, and what you do than what I do, as much as I think I'm so important. But, but is there anyone in your circle, your collaborators, your colleagues, your, your, your fellow professors, anywhere that disagrees with you, and do you still have a good friendship with them?

I actually never found science to be a reason to not be in good social relationship with anyone. I, I can be friends with anyone politically or in terms of science. I think AI safety community may disagree with me about impossibility results. Many of them think if given more time, more money, maybe more brain cells, they can solve it. But that's a respectable opinion. And I never had falling out over a scientific issue with anyone.

Arthur C. Clarke said a bunch of things. One thing he said is either we're completely alone in the universe speaking about alien life forms, or we're not alone, and both are equally terrifying. First, I want to ask you that question. Do you think we're alone in the universe?

Very unlikely.

Okay. And then I want to ask you a parallel question to that he was referring, you know, to the existence of alien life. But I want to kind of ask you this question. Either you're right or you're wrong. Right in your, in your hypothesis. If you're right, you're a prophet, you're a Cassandra, but Cassandra didn't have that great a life. But if you're wrong, you're the boy who cried wolves right at the dawn of the most important technology. Which one of those two scenarios is most terrifying to you?

I don't look at it from that point of view. I want the best outcome. And you kind of phrasing it as if I'm like the only guy saying it. I got Nobel prize winners, Turing Award winners, hundreds of computer scientists, all saying exactly what I'm saying. I think I'm in A good company. And the fact that we couldn't find someone decent to represent the other side tells me maybe I'm on the right side of scientific history.

Arthur C. Clarke said, for every expert, there's an equal and opposite expert, again, on the opposite side. Not, not the Turing side, not the. The scientists you most want to hang out with. But like, is it Lacun, is it somebody you know that we can have a. A confrontation with, all for the sake of good, for the sake of heaven? As it said, is there somebody that, that you most. That you haven't gotten to talk to? Maybe it's musk or I don't know. Who have you not gone to talk to that could really, really just energize you in a debate? As you said, as a good scientist, you don't get mad at them, but you learn from them.

So is there anyone that you would really push back on you and you think it would be a fair fight, so to speak?

I would love to have conversation with Yann Lecun. I think he's very well respected, accomplished. We disagree completely. He's at near 0% for PDUM. I'm near 1, so it would be interesting. I want it to be a friendly conversation and I want to us to walk out of it with understanding of why we disagree. There has got to be a scientific reason and if we can nail it down and maybe with help of external scientists, see who's right about those points of disagreement, come to better understanding of science.

What do you think about the explosion of agents? You know, this, this Steinberger and Open Claw, like really had a moment. It kind of reminded me of like Web3 and Bitcoin and. And now it's like, okay, so now all these things have it and OpenAI bought it and they bought a podcast and it seems like they don't really know what they're doing. Sometimes I spend a lot of time setting them up and I use them once and then I move on to the next shiny toy. What is agentic? What? One or two agentic tools or tools that you use on a regular basis are indispensable?

It's kind of funny. Maybe 15 years ago we published some papers where we said stupid things, don't do them with AI, don't connect it to Internet, don't open source it, don't give access to random people. Having an agent and giving it full access to your computer, your bank accounts, your email sounds like the dumbest thing you can possibly do. And watching smart people do that really blows my mind. So I'm just really impressed. Like they read the red lines literature and decided that was a plan for action. The question I often ask, if they wanted to destroy the world, if they wanted to cause problems, what would they do different? And I can't find any differences.

Why hasn't my Tesla, you know, killed millions of people, you know, in this last 10 years since I got it, it has automatic, you know, full self driving. It obviously optimized, it could, you know, go a lot faster to my destination on the sidewalk than, you know, obeying these stupid traffic rules. Why doesn't it do that? Is it regulation?

So historically your Tesla was a narrow AI, it was designed to drive your car. It didn't know anything about chemistry, physics, playing chess. Nowadays they're trying to add more, more crap to it. But that was exactly how we should be doing AI It's a useful automation for a specific task. We can test it for safety. We know what to expect in different situations. Do more of that. Now, if you take general superintelligence and place it in a position of controlling a fleet of Teslas, I don't know what the outcome is going to be.

So Arthur C. Clarke said, when a gray bearded scientist says something is possible, he is very much likely to be right. But when he says something is impossible, he's very much likely to be wrong. But tell me my friend, what have you been wrong about? What, what things? If you like, have you maybe missed a mark on it? Must be some. Don't say I'm too humble.

Okay, no, no, no. I keep track of my incorrect predictions. I have many outside of science. I made internal, I don't publish them. I made predictions about elections, about startup wars. I was wrong about many of those things, I think in science. I openly admitted that my estimate for how dangerous a system like GPT 5.5 IS, was beyond what we actually see in terms of damage. So that's something I'm quite comfortable with, seeing a better outcome and correcting as a result.

But it doesn't mean that our predictions are incorrect.

I really do appreciate your honesty, your integrity, your candor. And the final question I have is kind of a advice to your former self. So Arthur C. Clark said the only way to determine the limits of the possible is to go beyond them into the impossible. And I want to ask you, what if you had 20 seconds with your 20 year old self? You know, go push the rewind button instead of the fast forward button or the pause button. Take us back to that. You have 20 seconds with young Roman Yampulski. Before he was a famous professor, researcher, cited scientist, author, and just so many podcasts are the highest form of technology, as we know now the podcast Roman, tell me, what would you tell him to give him the courage to do as you've done to go into the impossible?

I love those questions, but usually they were kind of silly. Such as start buying bitcoin earlier. Right. Advice. Or something very particular to like, don't listen to that doctor, get second opinion. They're gonna cut off your healthy kidney. General rules are if you listen to other people the best, you can become as average.

That's beautiful. Very, very. Well, yeah. They say, you know, you're the average of the five people you're around the most. So that's why I only surround myself with very skinny, very smart and very rich people. But I might be the variance or the skewness of that. Anyway, Roman Yampolsky. Professor Roman Yampolsky.

Such a great treat to talk to you. I do hope we meet in person so I can give you this meteorite. Any things that are upcoming. I know the book is out in audio format. I recommend it very highly. It was published by a very respected but very academic press, CRC Press, many years ago, but now it's out in actual audiobook. So you can take not Roman's voice, but a very mellifluous voice with you.

Thank you so much. If you do find this interesting, I post a lot on this subject. You can follow me on Facebook, follow me on Twitter, don't follow me home.

And you're speaking to the aliens and to the super intelligences. Thank you, Roman. This has been a great, great treat.

You made it this far and you heard Roman argue that the most important technology we'll ever build is the one that we can't control. And he'd still bet on a pause if he could. Now, if that rewired, how you think about AI, subscribe and please turn on the notifications. Tell me in the comments, would you press his 10 year pause button and go deeper? Click my conversation with Max Tegmar. It's linked right here. And do it now before the AI decides you're not allowed to.