Is the Universe Random, Deterministic, or Both? (ft. Andrew Jaffe)

Is the universe intrinsically random? In this conversation, we dive deep into why the universe may be fundamentally intrinsically random. Whether inflation is on life support, the truth behind the Hubble tension, and whether cosmology is approaching the event horizon, the limits beyond which humans can never know. Today, we're joined by one of the architects of modern cosmological inference, Professor Andrew Jaffe, author of a new book called the Random Universe, that. That argues that every observation in science is shaped by the models we bring to it, biases and all. So what's one model, or personal belief that you held for years and years and years and then later only discovered that you were completely wrong?

Oh, gosh, let's see. What did I, what did I think I was wrong at? Well, almost nothing. No, if only that were true. But, I mean, there's. So there's lots of different realms I could go into. I guess many. The ones where I find myself off wrong the most often are probably interpersonal relations. So we all build models not only of just the physical world around us, but the people around us all the time.

And I suppose, you know, you, you're in relationship with people, friends, enemies that become friends. I mean, that's a great, That's a great example. When you thought you understood somebody and you thought that they were diametrically opposed to you in some ways or other, and then you realize that you've just been interpreting everything they said through your lens. And when you, when you build a different model for them, which also is a little bit of a different model for yourself, you, you can interact with these people in a totally different way and realize all the things that you have in common. And, you know, that happens with some other scientists when, but not, you know, not about their science necessarily, but just about the way they work. And you might have thought that, you know, there's somebody that you really couldn't work with. But when you sat down and you're forced to, then you get a much more detailed model, right? There's somebody that you, that you had this sort of very vague understanding of from far away, and you thought, oh, no, I don't like that person. They, they're too gruff.

They're, you know, they, they, they rub me the wrong way. But when you get to know them, you learn that, no, you know, they're just. Their way of tentatively exploring the world is by poking at it a little bit harder than I do. And so when they po. You know, part of that is poking at other people. And when you see that they're not doing it out of malice, but out of, out of curiosity, it it your understanding of their personality.

In the book you talk a lot about observations of children, especially your daughters, who I've had the pleasure to meet here in San Diego. But the kind of overarching thing is your you know, bemusement with their personalities, how they're coming to encounter the world and make models of the world and sort of you observing them observing the world. And in the book you write that all observations are theory laden. Which kind of struck a note with me that reminded me of another character in the book, which is Sir Arthur Eddington, who said never trust an experiment until there's a theory to back it up. I thought he was joking, but it sounds like you're more inclined to take him at his, at face value. What do you make of that statement?

Yeah, and I don't think he was joking or actually no one's ever, no one's sure he actually said those words. But there is a quote that I have in the book which I'm not going to reproduce here, which is a somewhat more nuanced version of that. And, and it's, yeah, this is along the lines of, of what you said, quoting me that that observation is theory laden. You can't just, you don't just sit there and see a little bit of gray over there and a little bit of yellow over there and a red patch and then kind of interpret that. You go and thinking, oh, the world is made of stuff is really made of those objects that are at different distances and made of different things. And in order to really disentangle that, you need to go into your brain and your mind need to go into the little screen of your retina which really, you know, is like a ccd. It's got lots of little, it's got lots of little, little photon receptors and somehow convert that very raw Image into a 3D time dependent picture of the world, which is what you have when you look at the world. And that's true for your visual field and it's true for all the things that you're not looking at right now.

Like I, you know, you are pretty sure what's behind you, you're pretty sure about what's in the other room. I'm in my house. I know that my kids are down there and if I listen really carefully I can hear them, right? And I know that it's not somebody, you know, playing a trick on me, probably it really is them talking. I can go out and probably catch a bus down to somewhere I need to. Need to. Because I have a model for that in my head, right? All of these things, which look like they're just about the data of the world, are really about taking that data and converting it into pieces into. In this calibrated model. Right? That's sort of.

We scientists, like, talk about calibrating things, and that's what we're always doing all the time. The models aren't perfect models that tell you where everything is. They're models that have a lot of free bits that you can move around in your head and figure out what really is there or what's usefully there. Right. It's all from evolution, right? So it's all what is useful for us to understand about the world.

So, you know, I guess if that's true, can we ever really escape the models? Do they become sort of a. A prison of our mind? Or is what we call objectivity itself just another model? Is that something that, you know, we can never really break free of?

I don't think we can break free of the modeling. I don't think we want to break free of the modeling, because I think the alternative is not being receptive. The alternative is not having any idea what's going on. But what we can do, and what we certainly do as scientists, is we keep testing the model, and when the model starts to break, then if we are good scientists, and I think that means if, you know, the book starts saying we're all scientists, if we're just good human beings being good at the. What, you know, billions of years of evolution have. Have done to us, then we get rid of the old model and replace it with a new one. We, you know, we change the map of our city when the road. When the road layout changes right in our heads, just like the actual map changes.

We, you know, when. When you learn something new about people, you change. Like I was saying before, you change the model of them in your head. This is. This is a feature, not a bug, that it's. That it's all models, right?

One kind of overused trope that I've encountered many times and I don't really know how to replace it, is the notion that there is one scientific method and people say, oh, well, you know, scientists apply the scientific method. And I. I kind of feel like that's, you know, like saying, you know, chefs apply the culinary method. You know, there's many ways to be a ch. Right. I like to think of it and my, you know, my minuscule experimentalist brain by, you know, kind of Making an analogy with like a staircase, that there's actually at least two different types of scientific method, and one is deductive and one is inductive. And you talk a lot about this in the book, echoing what David Hume was concerned about, that you could never really justify induction. And I wonder if you could explain how do you approach the scientific method? As you know, I think you might be.

Well, I think you might be maybe the second highest cited scientist on the podcast. I think I've had on, I have on Stephen Strogatz and I've had on Yann Lecun and others, and they may have more citations by total number, but maybe not by H index, although I have had on your friend Dick Bond. But let me ask you this question. How do you, as an eminent scientist, how do you think about the scientific method? I mean, obviously, don't sit down and let me frame my hypothesis before I hire a grad student. So how do you think about it? And what are the roles of induction and deduction, if you wouldn't mind defining those for the audience?

Sure. So deduction is the kind of thing you may have learned when you were in school and you did proofs, right. You start with some small number usually of definitions and axioms, you know, just the definition of what a point is, what a line is, things like that, or just the definition of numbers and what plus means things, you know, very, very simple things. And then you try to figure out what you can just prove logically by using those, those definitions and the way they produce new things. So, you know, if you know plus, then you can sort of define what one plus one equals two is. And, you know, you can learn a lot about, you can define prime numbers and you can learn about that there's an infinite number of primes and all of these things which the ancients did, you know, without even the benefit of the same kind of way of writing algebra down as we have now, without overleaf, certainly without latex and overleaf, what they were able to do is produce a body of knowledge that starts with very simple definitions and gets to things that seem kind of hard to imagine that they actually come from only knowing those small number of things, but they do. So, you know, you can prove from, you know, from just the five postulates of Euclid, you can prove that triangles have 180 degrees and the number of Platonic solids, and, you know, all these things like that, which are, which seem pretty complicated. And you can prove that there's an infinite number of primes and you can prove that the square root of two isn't the ratio of two integers.

And all these things which were kind of monumental, but just follow logically from these definitions. Whereas what we do as scientists. And now I'm kind of giving the game away about what I think, because I'm contrasting that to what we do as scientists, which is we take some of those kinds of things as well, and we use the math and we use the geometry, but we also take observations of the world and we use those observations and we try to see if there are regularities in those observations, and we figure out ways that we can generalize from those regularities. Right. So, you know, and the. All the silly versions are things like, well, it's, you know, the sun has risen every day for millions of days, as far as humans know. So how can we prove just from that fact that the sun will rise tomorrow and from that fact alone, the answer is, you can't prove that the sun will rise tomorrow. With that fact and gravity, which is a whole big set of theories, you can prove that the sun will rise tomorrow or that fact.

And just having a physical model for things moving around each other, even if you don't really know why, even if you haven't invented gravity, just knowing that, you know, if you were Copernicus, so you had this not particularly good model with circles, but it had the sun at the center and the planets going around it. That is enough to say, okay, with this physical model, I can be sure, as long as that's really what's going on, that the sun will rise tomorrow. And, you know, and you could have done the same thing from Ptolemy's model where the Earth is at the center and there's all these epicycles too. So, you know, we, we. But there's no way you can logically prove that. And that. That is what worried Hume. So you mentioned Hume before, David Hume, the.

Yep. The Scottish Enlightenment philosopher. And he pointed out that there was this idea that science was this deductive proof procedure. Right. And. But he wanted to see what, you know, science, which wasn't even really a word at the time, but what natural philosophy could prove about the world. And he worried that it couldn't prove anything because you couldn't be certain of anything. And that was the word that he used.

And that he thought was the gold standard of meaning that if you couldn't be certain of something, then the only alternative is chaos. Essentially, he actually used the only uncertainty, the only alternative to certainty was probability. But he didn't have. He didn't have the meaning of the word, the mathematical meaning of the word that we would attach to it. He just meant, oh, that thing is probably true, which seems very vague. And that wasn't enough for him. And, and I think he was right to be worried about this. Without this probabilistic understanding of how we can become, if not certain, more certain.

And I think he didn't acknowledge the existence of more certainty. You could either be certain or uncertain. And uncertain wasn't good enough.

Something can happen or not happen. There's 50, 50 chance.

Exactly.

Yes.

To coin a phrase.

I mean, do you agree with him? I mean, do scientists simply behave as if induction works and it sort of is evolutionarily rewarded, as you said, if you didn't really have an expectation that the sun would rise every day, I mean, you wouldn't be likely to contribute your gene pool, you know, 30 years hence. So is it an evolutionary reward mechanism that's, that's kind of reified induction, or is there something more to it?

Well, I think, I think yes, but it's more of a virtuous spiral than a vicious circle. So the reason why evolution has taught us to use induction is because it works. And why does it work? It works because the universe does display these regularities. And that's what you need. You need a universe that is explicable. And that seems to be the case. It didn't have to be the case. It could change tomorrow.

Right. All of a sudden, all of the laws of physics that we think we know could be different tomorrow. Is that allowed by logic? It's allowed by logic. It's not allowed by the physics. So, you know, it is logically possible that the universe does not obey these regularities. And, but it has worked so far. It would be very difficult to proceed. Forget as scientists, just as, as beings in the world, as you say, you know, who have, who have undergone this evolutionary process to take advantage of the regularities in the world around us.

Right. It's not just our brains that take advantage of amoeba, take advantage of the same regularities in the world around them to know, not to know in any, in any brain sense, but just to, to react as the little machines that they are to the, to their surroundings in a way such that if they move towards the kinds of things that gave them nutrition in the past, their bodies are built such that, that will work again in the future. And they, and to the extent that works, that's great. And when it, when it fails because they run out of the nutrient in the, in the reservoir or the water dries up or whatever it is. Then of course, the algorithm, their, their algorithm that does that has failed. So, you know, it's not that these are guaranteed to work all the time. It does depend on the regularities of the world maintaining themselves. And in some cases, like I just like that example, they don't maintain themselves.

Hey, everybody. I'm usually the one that asks my guests to judge their books by their covers, but today I'm asking myself to judge my own book by its cover. My newest book, Focus, like a Nobel Prize winner, is chock full of advice. Life tips and focus and productivity tips from nine of the world's greatest minds, Nobel laureates, ranging from economics to peace to physics, of course. I hope you'll check it out. And my publisher's gotten Amazon to run a special, so go to Amazon and get the Kindle copy today. So we'll get to the Simons Observatory in just a bit. But I'd like to connect what we just talked about.

Jim Simons himself, who did some work in what are called minimal surfaces or minimal varieties in Romanian forms. And it's a beautiful kind of proof that if you like that induction fails. He basically, and many others have shown that you have this thing called a minimal surface, which you can think about as you take a coat hanger and you dip it in soap and it makes a bubble, it makes a film. And that film minimizes the area on the suspended between the boundary, right? So you've got this one dimensional boundary and. And the soap film becomes two dimensional. And you can do this in three dimensions, and you can do this in four dimensions. And they won't have cusps, they won't have these kind of singularities. And so it actually works up to dimension seven, so.

Which you can't visualize, right? So what Jim demonstrated is that it fails, or with other mathematicians perhaps too, that it fails at Dimension 8, you know, which is kind of a weird number for it to fail at, right? So here you are, you're just marching. I would have stopped at dimension two. I mean, I couldn't derive it anyway. But the point is that it's a remarkable thing that induction works perfectly well until it doesn't. And my question is, if induction, to use Yogi Berra, is kind of like making predictions about the future is especially difficult. How can induction be useful about describing things that happened billions of years ago in cosmology? How does cosmology work at all if induction is shaky?

Well, because we have this amazing model which Sort of is the Big Bang. But the Big Bang itself has all these components. It takes into account the model of gravity, which is general relativity. It takes into account lots of aspects of the model of particle physics, just atomic physics and chemistry. And when you combine all of those things, actually the Big Bang isn't something you have to invent on top of those things. It kind of pops out from them, right? People had derived from Einstein's equations. The idea that there would be one of the possible solutions to Einstein's equations was an expanding universe that was actually known before the observations. But it was just sort of a curiosity because nobody thought that could possibly be a useful model for anything that was real.

But then it turned out from Hubble's observations, we saw that things were expanding. And Hubble and Lemaitre realized that you could write this down as a nice law and that fit in well with the predictions. I don't think Einstein himself didn't do this calculation, but the predictions from Einstein's laws. So when you. We can do induction on these amazingly large spatial and temporal scales, because we have a model. So it's all about models. And we do induction and we become not absolutely certain, but probabilistically certain. We become more and more sure that the model makes sense.

But of course, as scientists, it's not that we hope that this remains kind of boring. And yes, we fill in the blanks of the model. We hope it's going to break. We want our models to break because that's when you really learn the new stuff.

Do we, though, Andrew? I mean, to be devil's advocate, with love and respect, I mean, we hear all these things from the announcement in the first JWST data release that galaxies are, you know, behaving completely in disarray. There's panic at the disco. There's no way for them to be in accord with enough time to start their spiraling gyrations in just a mere 200 million years. We hear about the Hubble tension where you eggheads on Planck and the boffins on supernova disagree, you know, with a statistical chance of being a fluke of 1 in 30 million. We hear about dark energy being constant, then not being needed, then being variable, and then that's indicated disarray. And then dark matter, we can't detect it, but the dark matter makes up 85% of the universe according to our, you know, beloved colleague Katie Freeze, many times past guest. So tell me, Andrew, are we really so sure we have such a great model? I mean, would you bet Your neighbor's dog on this model.

My neighbor is a very nice dog, so I wouldn't do that. But I think we. So the problem is that the model is stressed in lots of ways, but it hasn't broken. And when you break a model. So one of the differences I think between this probabilistic way of looking thing, looking at things, and some other sort of inside baseball, different ways of doing science, it's Bayesianism versus frequentism and things like that is you don't just break the model and say, oh, the model is dead. You break the model because some other model fits better. Weird thing about all of these ways in which the Big Bang model seems to be stressed is that no one has really successfully come up with an alternative that fixes really any of these tensions that you mentioned before. So there isn't like a good candidate for what the dark matter is that we should have detected.

There's lots of candidates that make sense that we haven't seen them yet, but there aren't any that sort of say, oh well, it could have been this, but, but it's not. There are some ideas for, you know, you specifically mentioned the 1 in 30 million chance of the Hubble tension. We can talk about the 1 in 30 million chance. What I think is not an accurate assessment of the, of the chance. But in any event, people have come up. In fact, our, our mutual friend and collaborator Mark Kamienkowski is one of the people behind sort of some very, some of the most prominent ideas for solving these, this particular tension. But even that doesn't fit the data incredibly well. It kind of ameliorates the problem to some extent, but it's not a very natural outcome from anything.

So the way I come down to sort of there were two classes of things. One is that there are these holes. Not holes, there are these blank spaces in our models that have names, but they don't really have a thing identified with what those names are. So we give them a name, but we don't know what the dark matter is. We don't know what the dark energy is. And those one hopes are things that better applications of experimental methods will give us some ideas for what models we can build around them that are advances on what we have before. The other is kind of internal tensions within the model where we have two different measurements of the same thing or we have the formation of very early objects that seem difficult to do in the model. And I think in some of these cases we can say, well, the observations may not have been done carefully.

Enough. And so I, my egghead friends and me on Planck and my other friends who are also eggheads, but I'm not collaborating with, who are measuring the same things in a different way, get a different answer. But both of these are so hard. Experimental science, like being a theorist, is easy, being an experimenter is hard. And especially when you then have to enlist theorists to help you analyze your data. And, you know, I don't know. I don't know how good we are at that. So, you know, it wouldn't surprise me if, you know, all of the.

If everything is true, if the model is a little bit wrong, but maybe not in a kind of really exciting way, but that the experimenters have also. Or let's. The analysis of the experimental data has also gone awry in various ways that have led us astray in the interpretation of these results. But, you know, because I think the model has worked really well. But, like, you know, like I opened with it, we want these models to fail, but so far we haven't found an obvious way in which we're really sure that they've failed.

So, Andrew, we have now come to the patented segment judging books by their covers, which is something you say you're not supposed to do, but how else are you going to develop Bayesian priors if you don't judge a book by a cover? And you are now seeing this everywhere at Waterstones, I think it's called over there. You posted a gleeful picture. It's got encomia on the back. From my kid's favorite astrophysicist named Brian. Sir Brian May. Andrew Jaffe's fascinating mission here is a profound examination of our fundamental beliefs, but how the universe works. Shockingly, he shows how uncertainty is at the root of every physical law. Of course, the guitarist for Queen, no stranger to astrophysics and as well as being a great popularizer of what we do.

So the random universe. Walk us through the COVID the title, the artwork, and the beguiling subtitle as well, please. Hey, book lovers. We're judging books by the covers. We know we're not supposed to do it, but into the impossible.

There's nothing to it.

Let's take a look and judge some books. Okay, Andrew, back now. Let us judge the book by its cover, please.

Great, right? So I actually, I think kind of unusually, did start with the title of this book. So I had wanted to write a book for a long time. I don't know why, but I thought I had a book in me and I didn't know what it was going to be about. And I kept pestering my friends who had written books. You hadn't yet written a book or I would have probably pestered you about it. But I pestered my friends who had written books and I said, oh, I want to write a book. I've written lots of magazine articles and things, I want to have a blog, but I want to write a whole book. And so in particular, one of the friends that I bothered all the time was a colleague of ours named Pedro Ferreira, it's a professor at Oxford.

And I went to some, actually it was a book related event in London more than 10 years ago now. And I had been bothering him of course, and on the, in our respective Ubers home, he called me up and he said, Andrew, here's the title of your book. And he said the Random Universe. And I said, oh. And actually almost immediately, I know this sounds ridiculous, but the, the whole concept of the book, what it would be about, how I would structure it, came into place not in gory detail, but really, you know, the, the, the set of ideas that I wanted to explore immediately came into place. And from then on it was, well, it was then a very hard road to getting the actual book. But I wrote a book proposal, I shopped it around and then I had kids and so I decided I would let it lay low for a few years while I tried to get them out of their, out of their nappies as, as the Brits hearsay. And then I revisited it in sort of 2018 and shopped it around again.

And luckily enough, my now colleagues and publishers at Yale University Press said, oh, we'd be happy to actually publish this book. And so I, I did. And it took a little longer than usual because in about 2020 the world underwent a big change and I was, I was a bit busy doing things like many of the rest of us were for a few years at home, but I was able to do it and I finally put it together and it finished its first full ish draft in April of 24 and a year and a half later, because that's the timescale for these things, the book finally appeared. So that's the story of the book and its title, the COVID I don't know the way book covers work these days. You know, I gave a bunch of ideas to the publisher and they produced this and I was immediately struck by how good it was. So what does it show? It shows sort of a grid that has been deformed by, you know, by. And it's sort of similar to the way we think of this, the way gravity deforming the underlying space and time. So it's something that is familiar to people who've thought about or studied or even seen things about Einstein's relativity.

They took advantage of a couple of the letters in the random universe to make this little sine wave that is the A in random and the V in universe with a couple of little extra bits of, you know, bits of axes there. So I, yeah, I'm very happy with the way this, the way this came out. It sort of brings out a lot of the, the, not quite the ideas, but the sort of sensibility of, of being both very precise because that's what these, these grids are. But then randomized, right? Being pulled, pulled apart by gravity or whatever is pulling it apart as really does happen in our universe. And then the, the subtitle which we came, we came to rather late really crystallizes the two main or the three main parts of the book model. So it's how models and probability help us make sense of the cosmos. So models I've been talking about a little bit, right, how in order to understand anything we need, in order to make sense of anything, we need models. But especially as scientists, the only way we can use our models is to build a probabilistic framework around them.

Build the thing where we can say, okay, this thing is 68% true. You know, I believe with 68% confidence that this is true. Or more to the point, this Hubble constant, this expansion rate of the universe that we talked about before has a 68% chance of being between. And I'm just going to make the numbers up. This isn't what the current things say, but between 66.2 and 67.9 and a 32% chance of being outside those bounds. And when you put those numbers all together, you end up making one of these famous bell shaped curves. So that's how models and probability help us make sense of. And then of course I'm a cosmologist, so we have to make sense of the cosmos.

I start the book by talking about how we learn about anything as human beings. And then, and then by the end of the book, I hope I've taught people how we use these mental tools to do actual science with cosmology.

So one of the subjects of the book who is near and dear to my heart is this guy, Galileo Galilei. And you say the most well known scientific laws and theories try to describe the mechanism that underlies some sort of phenomenon in the world, as Galileo wrote in the Assayer, The Il Sagiatore 1623, the Grand Book of the universe is written in the language of mathematics. And later he quote the kind of concept that's known as the sort of the. The unreasonable effectiveness of mathematics in the physical sciences. So it was kind of going on the limb that Galileo wrote this about math. I mean, it was before calculus was discovered or invented. And we can argue about that. But are there models that people follow because they're beautiful, in other words, because they appeal to our sensibility? Obviously, string theory is an example of that at some level, not being born out of any empirical evidence at all, as your colleague or nearby colleague, Joe Conlon has written about.

I've had him on the podcast. But tell me, Andrew, what sorts of models do people follow out of inertia? Whether that's due to some notion of beauty, simplicity, parsimony, but not evidence?

Well, I would like to think that we're somewhat immune to that in physics, but I think the string theory example may be a counter example. I mean, I think what we'll. What we'll probably find when we. If. But let's hope when we figure out what supersedes the particular mix of general relativity and quantum mechanics that we use in particular as cosmologists, because we really need both of those ingredients, eventually we are likely to find something that supersedes that. And I suspect we'll go back in retrospect and say, ah, these are the things that they should have been paying attention to. They should have said, oh, yeah, well, you believe this because it makes, because it was nice. But in fact, once you saw this piece of evidence, then you would have realized that it's wrong.

And I think in physics we do that to some extent. I think as jobbing physicists, the way that plays out is a little bit different. What we do is we build simplified models of things, approximations to things that we know are approximations. Right. And we will often use them to calibrate our understanding. So in, in cosmology, very often when we're trying to build models of how galaxies form, for example, we. We use what's called the linear theory of structure formation. And the linear theory of structure formation is great because you can write down the equations really simply and you can solve some of them on a piece of paper and then others with a really simple computer program.

Right. And that's in contrast to, essentially the main alternative is simulate from knowing the really small microphysical laws of how things work, and then try to simulate what A galaxy will look like. And we will often use these linear theories or maybe small corrections to those to build our intuition. But sometimes we will keep using those far too long because we can't run the computer programs fast enough to be able to do the full. The full problem. Or we just really like to do things with pencil and paper and we don't want to use the computational techniques because it feels a bit like cheating or it just feels like we're not learning as much. Right. I think one thing that's coming around now, especially when we're not only doing simulations, which at least are based on the underlying physics, but we're kind of trying to use machine learning and artificial intelligence to not even have to do that, but to use a very small number of those simulations to learn how to go from the physical laws to what the universe might look like without doing all the intermediate physics.

And that might end up working. And in some cases, when you're really careful, it clearly does work really well. But the worry is it's a good tool, but you might not have learned the things that you want to learn. I think I'm getting quite far off the question you asked about using our models when they're no longer good models.

But I think model capture.

Yeah, I think. But that does happen. And I could perhaps cast aspersions on fields other than fundamental physics and cosmology, where to me, as an outsider, it seems. It seems like this happens more often. I talk in the book about some famous philosophers of science, people like Karl Popper, who sort of advocated this very deductive view of science, where you can prove things only because you can prove things are wrong. You can't prove things are right at some level, but you can prove things are wrong by finding some observation that contradicts the evidence or contradicts the model. Sorry. And if you can do that in an ironclad way, then you have done deductively, you've ruled out some model.

Now you have to be very careful because you can't even be absolutely sure of that because all observations have some probabilities associated with them. And so even then you're only probably, but really, really almost certainly at some level disproving things. And there are models that are hard to disprove. And the kind of famous examples, people can decide whether they believe that these are such examples. But are psychoanalysis, where you can go and attach Freud's ideas to almost any situation and say, ah, yeah, this is a proof of that. And then also famously, kind of Marxism, at least as A kind of supposedly scientific theory of history, whether you believe it's a good system or not, probably falls into that same category where you've gone and retrofitted the words of Marx from Das Kapital to fit something that looks very, very different from the kinds of things that he predicted would happen based on the supposed laws of history. And the worry is, of course, that cosmology fits into one of these categories. And with all of these things you mentioned before, all of these potential internal contradictions that we might be papering over, and that the real theory is very far from that, but we've allowed us to add in these epicycles, these things that, that modify the theory in lots of details, but purport to leave the overall idea of the theory the same.

But have you, you know, have you really modified the theory so much that it's unrecognizable? And for sure, some of the people who look at cosmology, and I would say mostly from the outside, think that cosmology is in that state already. That the fact that we have these unknown things like the dark energy and the dark matter, the fact that there are these measurements of the same quantity that give us different answers mean that, but that. But we're sort of still willing to say that, oh, no, the theory is fine, we just need to fix these things and it'll be okay. Kind of imply that we're in this degenerating phase. So this was a, this was a term invented by Lakatos, who's a. One of Popper's students and disciples in the late 20th century philosopher of scientist, that we could be similarly in a state where we're just trying to patch up the theory when it's failed. And I think the big answer I would give to that, the rejoinder I would give to that, is if you find me a theory that's better, then I will happily give this up. But right now, anyone who claims they've done that hasn't yet succeeded at that.

Right, because you really need to. You need to not only explain these places that are in tension, but all of the successes of the model too. And the model has an enormous amount of success at explaining astrophysical and cosmological information that, you know, that our experimental and observational colleagues and theoretical colleagues have been amassing for, you know, 100 years now.

Yeah, sort of the Churchill's quip that, you know, democracy is the worst form of government, except for all the others. And maybe, you know, lambda CDM is the worst model, except for all the others, yeah. So let's get to the random part of the random universe, because we talked a lot about the universe. We'll come back to it towards the end as we go through a rogues gallery of different, you know, kind of rogues turn, random turns and so forth. But let's go through what randomness is. And you talk about a distinction between randomness in nature and randomness in our observations of nature, our perceptions and our models of nature. What is that distinction based on? Is it something to do with our classical brains that think in entropic randomness versus the real quantum universe, which is indeterminate? How do you think about the word random as it applies to nature itself and in our models themselves?

Well, I think you can think of all randomness, even if randomness is fundamental to the way the world works. And we think it might be, and we'll talk about that. I'm sure you can think about randomness as nonetheless being about us. Because one good way to define randomness, and it's not the only way, but one good way to define randomness is unpredictability. And so then you need somebody to be doing the predictions, right? So if I can't write a computer program or an algorithm that can predict things, then it's random. So sometimes that's just because I don't have enough information. And that can be of a very simple kind. So if I run, you know, if you ask a computer for a random number, there's some fixed, not random algorithm that it's using to do that.

But the algorithm might involve, you know, taking something from the clock of the computer. So it depends on exactly when you've done it. And there are more sophisticated versions that aren't even things like that. And then doing some numerical operations on that which give you a sequence of numbers that are reproducible. So if you started with the same, you know, with the same time, or if you give it a number called the seed, then it'll loop around eventually. Although eventually might be in billions and billions and billions of, of random numbers. So that's one way to kind of make things seem random when they're not. Because you can make these algorithms that, at least if you don't know the procedure that has produced them, then it's as good as random.

And for most cases, that not only as good as random, that is random, right? So if I flip a coin and I look at it and I ask you if it's heads or tails for you, it's a random choice, right? Even if I'm completely sure what it is. So what's random to you is not random to me because I have the information and you don't. So lots of the things that are random in the world can be thought of like that. A lot of the times when physicists come up against randomness are in thermodynamics, in the study of essentially systems with enormous numbers of particles, molecules and specifically gases. But it's not only in gases, but gases are really easy to think about because there are all these molecules that are just whizzing around and they're basically not interacting, they're just sort of occasionally bouncing off of things, but that's about it. And if you knew where all of those individual molecules of gas were and where they were moving to, you could predict more or less absolutely what the state of this gas would be. And you could even use that knowledge to do kind of weird things like separate all the hot bits of the gas from the cold bits of the gas and change the temperature from being, you know, 70 degrees Fahrenheit to being a bit. That's 50 degrees Fahrenheit and a bit.

That's 90 degrees Fahrenheit, which never. It is demonic. That's right. So that is Maxwell's demon at work. And it's because that demon has this extra information, but absent that information, you can't do this. So this kind of randomness is, can be purely described as a lack of information. Now, we also think that there are aspects of the universe that are even more fundamentally random than that. And that is, of course, this idea of quantum mechanics, this incredibly well checked theory about a description of the world that only ever gives you probabilities.

Right? The only thing you can ever calculate are things related to probabilities. Now it's a little, you know, it also tells you what the energy levels of the hydrogen atom are. And those are kind of probabilistic. But, but for, you know, for all practical purposes, they are actually telling you something about the world. So, so, but you, but knowing what energy level you're in is a probabilistic statement. So this is still fundamentally about probabilities. Now what's interesting is that when we learned quantum mechanics in physics class, we learned a version where it's sort of a good word you used before, reifies the, the randomness and the probability as being something about the system and about the world. So it's sort of somehow like a random number generator that's automatically in the system.

And actually that's a very puzzling way to look at it when you think of that probabilities are kind of about our knowledge. But other asks, other ways of interpreting the rules of quantum mechanics make more plain that the kinds of probabilities that come up in, in quantum mechanics might be just as knowledge based, as model based, as, as the kinds of the coin flip and of the gas. And it's just about our lack of knowledge. And you know, so it might be that that's all there is, that randomness is only about, you know, beings who care enough to ask about, you know, how often things happen.

What is. Oh, let me rephrase that. Okay, so everybody says that they, you know, can apply quantum mechanics to many different applications, even in cosmology. But obviously people have said from Feynman on down that, you know, if you think you understand quantum mechanics effectively, you don't understand quantum mechanics. It's a tell that you don't understand it. So, you know, the randomness and the characteristic of randomness, both in this book and in physics in general, they're very different. I wonder, you know, is it true that there is a way to link between the classical thermodynamic entropic description where randomness comes into play and the quantum mechanical description of reality, you know, as being subjective, possibly in interpretation based. You know, I always say we don't need an interpretation for Newton's laws, so why do we need it for quantum mechanics? So what, what is the fundamental similarity between quantum randomness and if we can't understand it, how can we apply it to inflation, for example, or quantum gravity?

Well, so, you know, the, the fundamental quantumness of, the fundamental randomness of quantum mechanics is a, you know, is it like the other sorts of randomness? Well, I think it, it might be. It does depend on the interpretation. I think it's not true that Newton's laws don't need an interpretation because I don't know what a force is. So you have to kind of tell me, you know, what does it mean for there to be this force that acts at a distance? Maybe relativity needs less of an interpretation because I can kind of picture this, this, you know, bending sheet of rubber, but that's really not what's going on. So that's also not really the best model. So models need, you know, they're just the math, right, in some level. So you need some interpretation structure on top of it even then. But yes, I think it's true that, that people don't differ too much in their interpretations of those things, but they do differ about their interpretations of quantum mechanics, but it's also true.

And Feynman, as I talk about in the book, was purported to have said, but probably never said, that the kind of interpretation of quantum mechanics that most of us actually use most of the time when we have to use quantum mechanics is called shut up and calculate. And that's that we don't have to worry too much about what's really going on under the hood, because the set of mathematical rules that tell us how to use quantum mechanics to make predictions, albeit probabilistic ones, about the world, are more or less unambiguous. And so as long as we're happy to use those rules and that we can apply them on enormous scales as well as laboratory scales, then we're fine. It is dissatisfying. I'm not saying that we should just shut up and calculate, but I certainly separate the. The parts of Andrew Jaffe, physicists that think about the interpretation of quantum mechanics and the parts that use quantum mechanics relatively frequently to do the physics that I do as a cosmologist, and I have both of those within me. But if I spend too much time worrying about the interpretation when I'm just trying to calculate the wave function and use that to figure out something about whether this atom is going to move from one state to another, I'll stop doing the calculation I need to do. So I just need to shut up and calculate sometimes.

So I actually kind of raised this question with Jim Peebles about 10 years ago, and he kind of echoed it. I think it was Merman, not Feynman, who actually. Exactly. But he said, Keating, what you need to do, Peebles told me, is shut up and measure. And I wonder if I can use that insult, that epithet that I'm still trying to crawl out of my cave about in a way to kind of maybe criticize our own darlings, kill our own darlings, which has to do with inflation. I mean, do you think that inflation can either be proven or falsified in the paparian sense, or subject to the limitations on what that word really means? And if not, what will we be left with? Social proof, kind of authority, Proof by authority. What are some of the most promising avenues to falsify inflation, shall we say?

Well, so to start with the proving rather than falsifying part, I think it could have been, if not proven, then made much more likely if we had. And of course, this is a story you know very well had successfully observed the gravitational radiation from the early universe that that is, that is produced in almost all models of inflation, but is not always produced at a measurable level in models of inflation. And we believed we had observed that in 2013, and sadly that turned out to be incorrect.

2014.

2014, sorry. Yes. And if we had, then I think that would have been extremely strong evidence in favor of inflation probabilistically. Now, absent an alternative model, like I said before, I don't think, you know, the only way to really test a model is to have an alternative to it. And right now, most of the alternatives to inflation have fallen by the wayside because they are, they have predicted things that have not turned out to be true. Now, you know, we have colleagues who believe they have models that are, that are as good as inflation as some of the things they are not as well tested. You know, universes that are sort of cyclic, where the thing, sort of something arises kind of phoenix, like from the, from the previous universe. And it seems to be that if you, if you get those initial conditions just right, and if we understand the way gravity and quantum mechanics work together in this particular circumstance, then it might be the case that you get the same kinds of effects as inflation.

Now, I think the jury is out on whether these models actually do what it says on the tin, like they say around here. But inflation itself is not without its problems, just from a kind of theoretical idea standpoint, right? So it requires the. There's a time before inflation, right? And it requires the universe to have somehow found its way to inflate. And in many scenarios, that actually seems highly unlikely. And so that requires then embedding it usually into ideas called eternal inflation, where you have, where the universe writ large is way bigger than just the bit that we see today, Even if the bit we. And strangely, the bit we see today could still be infinite and this could still be true, which is hard to wrap one's head around. But it is possible that that's true, that we're in sort of a bubble that is inflating, but the universe as a whole somehow is even vaster than this. And then you can kind of make this happen, but then that puts really strict requirements on what the fundamental theory underlying all this must be like for this to be able to be happening.

And it might be that the string theory ideas that we talked about before, for all their problems, for all the sort of problems related to their beauty, not actually giving you some good ideas, it does give you some things that may turn out to be, if not testable, at least things that make predictions right for the way the universe might be. And the problem is they usually make too many predictions. And there are lots and lots and lots of ways the universe might be. So just the fact that we, that we live in one of them might not tell us much. But there are, you know, there are these, there are these ideas that leave inflation and the structure that it's embedded in with some testability. But I agree that there's a chance that we might not ever observe these gravitational waves, that we might not get any evidence of a universe beyond our own. And you know, it's, if you, if the experiments can't happen, then, then we may never know. Now, experimentalists are very clever and an adage that is used in technology, usually in information technology type things, but also applies in the technology that we use is that people kind of overestimate what can happen on the two or three year timescale, but underestimate what can happen on the ten year timescale.

And I think that's true for the kinds of technology that we use. So I started graduate School in 1989 before we'd observe gravitational waves, before we'd observe fluctuations in the cosmic microwave background. And if you had, and everybody thought that sort of those things were really imminent, even, even gravitational waves. But. So we were wrong about that, but we were right about the cmb. And then if you had told me that we'd make a map as good as WMAP or Planck A mere 10 years after the end of those first CMB observations, I would have thought that was nuts. But we did. And so technology really does march on.

And these things are kind of power laws. They become much better than you can really even conceive of over short periods, over medium sized periods of time. I guess even if short periods of time they're kind of going like this, then they always take off. And we don't know what that time scale is always. But for most human things they're, you know, a couple years. And so you wait a few more years than that and things look radically different.

You end part three of the book by talking about the limits of knowledge. And it kind of made me think of the possibility of a fundamental model event horizon, you know, beyond which we cannot know more because we can't, you know, trust the experiment in the absence of a theory because we really will reach an end of the ability to generate new observations and new experimental evidence. That's surprising in the information kind of theory context. So do you see that happening or am I just being a boomer? You know, I do feel like for the first time, when you started graduate school, you know, they were kind of talking about CMB experiments that would measure, you know, anisotropy, but maybe if they don't, that, that, that would be sort of the last CMB experiments after the dipole, maybe the quad, maybe cobe. I remember hearing, you know, WMAP would be the last experiment, but I never really took that seriously. Then we had all these great ground based telescopes, our colleagues in SPT and ACT and eventually our team on Polar Bear, which you're a member of and I'm a member of for 20 years now and now in the Simons Observatory. And then there was always, you know, the future, which had to do with CMB Stage 4. Sounds like a disease, horrible disease, but was actually a pretty brilliant experiment modeled not too differently from Simon's Observatory.

I mean, let's just be clear, it was sort of, you know, Simons Observatory on super steroids, funded by the US taxpayers. But, but tell me, Andrew, I mean, with the cancellation of CMB Stage 4, are we possibly at the experiment horizon with the Simons Observatory? Might this be the final, you know, kind of experiment and then if that's true, will we reach the event horizon in model space not soon thereafter in say 2035 when your daughters are, you know, professors? Summer, what do you make of this model horizon? Am I just being, you know, kind of a fake? Cassandra?

Well, if you're fake, that's, you know, I'll have to see. But no, it's a worry that we will try to do these, we'll do these next set of experiments or maybe, you know, if something like CMBS4 that you mentioned doesn't happen in five years, presumably in 40 years, it'll be easy and cheap. Right? So, so maybe, you know, maybe it will happen then even if it doesn't happen now.

Right.

If we wanted to rebuild Coby now, it would be easy. Right? So, you know, we can. The experiments that seemed impossible a generation ago are straightforward now.

Yeah, we do them as lab experiments in our.

Yeah, exactly. So, you know, so I think the worry of any particular experiments, cancellation, giving us a horizon, are probably overblown. But no, there's a worry that, you know, eventually you get all the information that's available in the microwave sky and you know, there's a WMAP balloon behind you, I can see and a beach ball behind you and you know, when we, when we've gotten the full sky down to very, very small, you know, high resolution and we were able to clean out all of the foreground astrophysical Stuff that's in between the cosmology bits and us, then, you know, then maybe the CMB will be finished as a field, as a field of study, and maybe we will not be able to do any better than that. And maybe, you know, with some of my colleagues who, you know, with whom I'm worrying about things like the overall shape of the universe, we have this idea that one thing that gives you a lot more information is if you could map out all of the structure inside the observable universe out to very, very great distances. So before there were stuff, right, from the. From the very first objects all the way to us on all sides of the sky. If you could do that, then there's an important sense in which you know everything there is to know in terms of the facts of the universe. And then you have to go and see what, what do.

Are there theories that we want to test that make different kinds of predictions? And there's still going to be random statistical predictions, right. We're never going to admit, maybe this is one of the crucial points that we haven't been explicit enough about. Right. We're not thinking about theories that say the Andromeda galaxy is going to be over there, and the Milky Way is going to be right here, and there's going to be galaxies there and a big wall structure over there. None of the theories predict that. What they predict is something about the random distribution of things. And when I say random, I think people have in mind that things are either random or ordered and there's nothing in between. But that's not true, right? Things can be random but have probabilistic structure to them.

So the reason why that ball behind you has the pattern that it does, it doesn't look like a model television screen of noise where things are not related. Unfortunately, people now may not know that a television screen used to have static on it, but it doesn't look like static. There are patches of a particular size, typically, and patches that are nearby each other tend to be more likely in particular ways. And we can describe the probability distribution of those spots on the sky in a particular way, and we can do that in three dimensions in our observable universe as well. And if we can find theories that make those probabilistic predictions that match what we see, then we can discriminate such theories from each other and whether we can, you know, move beyond the current theoretical horizon of the kinds of things we can test. Now, once we have all of this more data and in terms of literal information content, it's a vast more amount of data. But whether it's, whether it's, it's useful information content that helps us distinguish the theories, that's the question. And I agree that there's a real worry that we will, you know, that we may not have a way to figure out the kind of thing that the dark energy that we've, you know, mentioned a few times is the kind of thing that the dark matter is.

Right. We may not be able to identify the kinds of fundamental theories that produce the inflation that we have or whether it was inflation or something else. Right. Right. Now, those theories fit well the idea that it is something like matter, but that gravitates in particular ways, but is otherwise very weakly interacting with the rest of the stuff in the universe that fits the data really, really well. And maybe we'll have to be content with that for a while and maybe I hope not, and I don't think so because people are really clever. But maybe forever. Yeah.

We see sort of a glimpse of this perhaps with the state of affairs in high energy particle physics where it's even worse than the state of cosmology because we are blessed by Jim Simons and recently the UK and Japan and the National Science foundation to have sufficient funding to build instruments that will last for the better part of the next 10 years. But they're kind of quibbling over what would a magical unicorn be able to provide for them in terms of they could get $20 billion in funding. And they always say, well, you should build it, but because you might discover something new, which I think is a horrible reason to build anything. But I won't get into my. This is not about my favorite subject, me. It's about you. And I do want to close with the way I close all my cosmology classes. At the end of each quarter I quote from your again adopted homeland.

Now, your fearless leader, Winston Churchill, who said that this is not the end, I say at the end of class, I say this is not even the beginning of the end. But perhaps, perhaps, Andrew, it may be the end of the beginning. And in the very beginning of the book, which is again, very wonderfully written, very accessible, wasn't what I expected. It had, you know, less cosmology and more Bayesian reasoning and frequentist analysis and really a few equations, but essential ones. That's really beautiful. But you're right at the very beginning of this wonderful book that the book was really your attempt to hold yourself accountable for explaining the universe clearly. And I wonder after finishing it and after maybe today's conversation and the publicity tour and seeing it in Waterstones and elsewhere. What model of yourself have you updated, if any?

Gosh. Well, if you go look on at least some of my social media bios, I now start with the word author, which I guess is a, is a new model for that. Yeah, I thought I could write a book before, and now I know I can. I have actually written a book. So, yeah, it encompasses more whether what that means for the future for models are about predictions. Right. Whether that means that I'll be able to use that new skill for other purposes other than just, you know, appearing on, on, on amazing podcasts with my, with my colleagues and others. I don't know.

But yeah, I've, you know, I've, I've, I always like trying to explain what I do to other people. You know, I had a blog when they were, when they were just new. You know, I've been doing this sort of thing for a while, but at the length of a book, it's a whole different thing. And you, I guess I, I didn't know I had a philosophy. I guess maybe that's, maybe that's what I've learned. I, I, you know, this, this was my attempt not just to expl, like, like you said, not just to explain cosmology. It's not really a book about cosmology. It's a book about how we learned about the universe.

And I guess I hadn't quite realized that I could write down, maybe cogently explain what my view of, you know, of science and how we do science and how particularly we do our kind of science as cosmologists. Because it seems, you know, it's, it's when we, when, I don't know, we know when, when. I'm sure when you give talks and when I give talks about cosmology, people are always flabbergasted that we can conceive of these vast scales, right? This, we're talking about 14 billion years. We're talking about billions of light years, right? And these, you know, and the numbers of galaxies, you know, trillions and trillions of galaxies in the observable universe. These. And each of which contain, you know, each of which is as large as our Milky Way or, you know, many of which are as large as our Milky Way galaxy, which itself is incomprehensibly large. Right? We will probably, as a species, never get to explore any of it. And yet here we are making amazing predictions and statements about, you know, these, these vast scales.

And the fact that I, I think now I understand how we're able to do that and that it isn't. It's hubris in the sense of, you know, the fact that we're able to understand the world at all is kind of amazing. But it's also, you know, just kind of what we're bred to do by billions of years of evolution producing thinking beings that need models of the world to just, you know, find dinner at night. And as a byproduct of that, we can also reason about everything that there is.

Well, Andrew Jaffe, professor at Imperial College, award winning cosmologist and now author of the Random Universe. Andrew, this may not get you to 160,000 citations. Maybe it will. I mean, I can wish that upon you. I asked AI, you know, what books has Brian Keating written? And it said, you know, losing the Nobel Prize, into the Impossible and A Brief History of Time. So may you get at least 10% of Stephen's sales on this wonderful book. Thank you so much for joining us.

Thank you, Brian. It's been fun.

I hope you've enjoyed this conversation with Andrew Jaffe. If you'd like to dive deeper into randomness and cosmology, enjoy this deep dive I did with Dick Bond, one of the most preeminent cosmologists in history. Click here and don't forget to, like, comment and subscribe. It really helps the channel stand out above the noise that we call the YouTube algorithm. Thanks and see you next week on into the Impossible.