– Will humans, you know, forever be able to contribute something that an AI just could not have contributed, right?

– Yeah.

– And anyone who was confident of that, you know, after everything that’s happened in the last few years, I don’t know where their confidence comes from. You know, what people confidently claimed were the walls, you know, that AI is hitting have just been smashed through, right? We’re now living in the wreckage of all of these walls, you know, that were proclaimed. And so, you know, maybe in a few years, AI is better not only at finding the proofs, but at explaining the proofs. Maybe it’s even better at coming up with new questions.

– Some are calling it the mathocalypse, others referring to the mathacre. Regardless of terminology, the reference is of course to the recent onslaught by AI tackling problems at the forefront of mathematics research that to this point have been beyond the capacities of human mathematicians. And look, some of these problems have been career defining challenges, problems that researchers have been chipping away at, you know, paper by paper, year by year, result by result. And then, AI comes along and bulldozes the problem, ostensibly wiping away the problem by providing the answer. So, is this the apocalypse for mathematics? Well, there are many issues and angles here. Are we confident that the AI proofs are correct? Can AI succeed on every sort of problem or are there common qualities of those problems in which it is able to make headway and perhaps even qualities of those in which it may hit a wall? Should we be all doom and gloom here or perhaps do we simply need to get over our need for human brains to be the problem solvers and simply be excited by a powerful new tool for progress? Will mathematics of the future become more like other scientific disciplines like archeology in which the human role is to find buried gems, in this case, wonderful theorems, understand them, interpret their meaning, but not to generate the gems themselves? And more generally still, AI first made profound inroads into coding, now into mathematics. So, how much time do other fields have before their AI come to Jesus moment is at hand? To discuss these issues, I am joined by Scott Aaronson, who heads the Quantum Information Center at the University of Texas Austin. You may know Scott through his blog, Shtetl-Optimized or his general level book, “Quantum Computing since Democritus.” And of particular relevance to our conversation here today, Scott spent two years at OpenAI on theoretical foundations of AI safety. So my conversation with Scott Aaronson. Scott.

– Hey.

– Great to see you.

– Great to see you again.

– Thanks for joining us here. Look, I wanted to start with something that just happened the other day, which my daughter called me. She’s a sophomore and she’s following, she’s not a math person, but she’s a history major, but she’s following the news, everything that’s going on. And she said, “Dad, I don’t even know how to think about what the future is going to be.” And I said to her, “Sophia,” I said, “Yeah, that’s right. This may be the most, or at least among the most consequential moments in the history of the human species.” I don’t think I was exaggerating there. I mean, where do you put this?

– That has been the hardest thing for me personally when young people, whether they’re my students, my own kids who are a little younger than yours, 13 and nine, but they’re well aware of what’s happening, right?

– Yeah.

– And when they ask, “What are humans going to be for?” You know, apparently not for proving theorems. You know for writing, probably not for writing code. You know, what do we provide, right, either economically, if we’re not going to be needed for anything economically, well then, you know, how should we think about, you know, the point of life, you know?

– Yeah. I guess we’ll be consumers no matter, you know-

– Yeah. I mean, look, I think some people, you know, could just happily live just consuming, you know, and consuming entertainment, you know, eating good food. You know, it wouldn’t be the worst outcome in the world. You know, it’s certainly better than the Terminators you know, milling us all down, right?

– Yes.

– But I think a lot of people, you know, maybe had a higher aspiration than that, right? That they’re going to contribute some intellectual or artistic work that only they could have contributed. So, you know, will those of us who felt that way just need to get over ourselves?

– Well, that’s I think is a good way of putting it. And I’d love to work our way back to actually exactly that point because I do think that part of it may be that we just need to get over the certain kind of human exceptionalism that once kind of defined us, at least for many as a species. But before we get there, I just wanna walk through some of the developments. Look, many people are following this, but I think it’s good to hear it from folks like you and me who are sort of on the inside of this. Not in the sense of building the AI, although you are far more on the inside than I, but in terms of math and physics soon being disciplines where this is not a theoretical issue of the impact, it’s happening right now, you know?

– Absolutely.

– So if we go back, look, there was a time in the distant past, by which I mean like a year or two ago, when you know AI could do well on high school math, and then gold medal level at the Olympiad-type questions and so on. At that point, people were still of a mindset of, well, okay, it’s a little bit of a parlor trick maybe and so on. But when we start getting to research level problems, that begins to shift. So, what was the first one that caught your attention?

– Well, okay, I mean, my attention was caught since GPT-3 five years ago, right, even though it was still struggling with arithmetic, right. You could basically say if scale has gotten us to this, then more scale is gonna get more and more. And so, I kind of saw the writing on the wall. And I had friends who were much earlier than that, right, but I was then trying to warn people about it. And with each step, each thing that happens, you’re absolutely right. People just invent a reason why it doesn’t really count, right?

– Right.

– They say, okay, fine, of course it could do this. No one ever denied that it could do this. Of course, they themselves may have denied it.

– Years ago or six months, yeah.

– Six months ago, right. But okay, but it’s never gonna do that. And then six months later, it does that. And then, they invent a new excuse and so on. But I think for the math community at large, you know, a major, major moment came in the ancient time of May of this year. And that was when OpenAI announced the solution to what was called the Erdos unit distance problem, yeah.

– Unit distance problem, sure.

– [Brian And Scott] Yeah.

– Which was about, you know, if you have end points in the Euclidean plane, how many pairs of them can be distance one apart? Famous mathematician, Paul Erdos asked this in 1946. You know, he had a conjecture about the answer that it’s just maximized by putting them in a square grid. And then, what the AI found is that that conjecture is wrong. There’s a better way that gets more pairs and it applied results from a fairly obscure part of algebraic number theory right, to this problem. And so, you know, on the one hand, you could say, well, it was just sort of using standard techniques, just not techniques that any experts in this problem would have known, right>+? But on the other hand, that’s what a lot of human math is about. It’s about finding connections. And there was no question that if a human mathematician had done that, you know, they would have gotten tenure at any department. It would have been published in Annals of Mathematics. And so, there was a whole bunch of commentaries on it by Timothy Gowers, Noga Alon, you know, famous mathematicians just sort of trying to understand, you know, the impact of what had happened. And that was when this became impossible to, I think, for the working researchers to ignore. And I remember when I heard about it, I was in my office at UT Austin and some students just came and they said, “Have you seen this news about unit distance?” And I said, “You know what?” Because I don’t check Twitter every 15 minutes, but I saw. And the students, I have to say, were kind of despondent. They were immediately sort of extrapolating this to what is our future. And half a year later, I’m still trying to figure out what was the right word-

– And no doubt we will be for a while.

– Right, And then, I would say what happened is over the course of this summer, you know, we saw more and more shoes drop, more and more problems that we thought were incredibly hard just fall to AI models. Sometimes, it was a combination for human and AI, you know, what’s called a centaur, okay? Sometimes, it was just pure AI one shotting, as in, you know, someone feeds the question into GPT-6 or whatever. And then an hour later it spits out the solution.

– Right.

– Okay. So, that included some of my favorite problems in quantum computing theory. I saw, you know, one by one, started seeing them get solved. And by the end of the summer, I think every new paper, we just expected that at the end of it, there’s this little AI usage statement, right? And this is where the authors disclose, you know, which AIs did they use to do this research and for which parts of it, right?

– Right.

– And where they would assure the reader that they take full responsibility for the results, you know, right?

– Right.

– And so, yeah, that is now a new convention of academia that, you know, it wasn’t really debated or discussed, but that’s, you know, how we do it now.

– But let me ask you one question. So on the unit distance problem, it was disproving conjecture-

– It was, yes, it was.

– And Navier-Stokes, which many people have read about “New York Times,” and everything, those millennium math problem, it was finding a counter example.

– That’s right.

– And so, you can begin to think that maybe AI is really good at that kind of a problem.

– [Scott Aaronson] Yeah.

– Is that too limited?

– Well, I think it’s true that AI has been especially good at finding counter examples-

– On the Jacobian-

– Yeah, that’s right. That’s right. And I think a good way to think about this is just, you know, AI has been especially good at any problem that requires trying thousands of different weird constructions or weird examples until it finds one that works. Because unlike us, it never gets tired. It never gets discouraged by the fact that experts think that this construction doesn’t exist, you know, which from experience is huge for us, right?

– Of course.

– But it just keeps trying and trying, you know, until it finds some bizarre thing that works. And if that takes too long, you can always spin up a thousand more agents, you know, to try more examples in parallel, right? So, the Navier-Stokes was like that. The Jacobian, as you said, the unit distance were like that. Even some of the conjectures that it’s proved to be true. So my wife, Dana Moshkovitz, for the entire time I’ve known her, she’s been working on a problem called the unique games conjecture.

– What is that?

– Okay, so it was posed by a guy named Subhash Khot in 2001. And it basically says that for a whole bunch of optimization problems, the best approximation that you can hope to get to the optimal solution like in polynomial time is the solution that you would get via a standard technique called semi-definite programming unless P is equal to NP.

– Okay, which is a big-

– Exactly, which is an even bigger question. So this is not directly addressing whether P is equal to NP, but it’s saying assuming P is not equal to NP, all these approximation problems that we thought maybe are NP complete really are NP complete.

– Right.

– Okay. And so, it was sort of the central conjecture in this subfield called hardness of approximation. And so when I started dating my wife in 2009, she said, “I have to tell you, I’m working on this unique games.” And, you know, 16 years later-

– And you were willing to accept that?

– Yeah, that’s right, I was willing to accept that. I said, okay, okay, it’s good-

– To get it out in the open now.

– Yeah, exactly. Good to be open about it. And that was one of the crown jewels of these 372 major open problems that OpenAI published solutions to in a GitHub repository two weeks ago. The event that I have called the mathocalypse, although someone told me I should have called it the mathacre.

– It works too.

– Yeah, right. But sort of a crown jewel of this mathacre was the solution to the unique games problem. And-

– And where do we stand on that? For instance, is there a Lean Certificate?

– For that one, there is a Lean Certificate.

– And can you explain just for folks, yeah?

– Yeah, so Lean is a way of verifying proofs in a purely mechanical way, right? So, it just writes the proof in this sort of logical notation, where, you know, you could just write a program to check that yes, there is a valid proof.

– So every inference step is check.

– Exactly. Now, there’s still, you know, a couple things that you could quibble about. You could say, was the statement actually correctly encoded into Lean, right? Does the thing that was proved actually match the thing that we wanted to prove? You know, there have been bugs in Lean that have been discovered, you could worry about them. You could worry about whether this logical system is sound, which is actually not known. But modular, all of that, it gives us a higher degree of assurance that yes, there really is a proof here even if we don’t understand it.

– And the proof itself, I mean, I had this conversation with Tristan Buckmaster-

– Yes, that’s right.

– That when you actually look at the output, and I looked at some of them to see what it looked like, it’s very unreadable.

– Yes.

– Now, are… But I then looked at some of the papers on the GitHub. Some of them, again, they’re outside my field per se, but they seem much more readable. So, that seems to be improving.

– Yeah, so I think like, you know, they are written in English, you know, or in LaTeX, right? You know, they’re at least locally comprehensible. But what my wife, Dana, has told me, she has been spending all of her time these past few weeks studying the OpenAI proof of the unique games conjecture.

– Is it like hundreds? Is it 10 pages?

– It’s a… I don’t remember how many, you know.

– It’s hundreds, yeah.

– It’s some dozens of pages, but she said it’s as if it was written by someone on psychedelics. You know, it’s like the key claims are scattered all over the place. The central ideas are hidden. She said there’s basically no option except to use AI to help understand the proof.

– But AI is not yet at the stage, and I’m sure it will be. They can say, “Hey, here’s the output. Here’s the proof.” Now, you know how we humans like to write our papers because you’ve read a gazillion of them. Can you translate from that to a more systematic, you know…

– Yeah, like they’re getting better at it. They’re not there yet, right? I feel like what they’re missing maybe is a really good model of the human reader, right? What is an expert going to find obvious here? What are they going to need spelled out?

– And they can’t extract that from…

– Yeah, I mean, okay, so to be honest, like this is one of the hardest things when we’re teaching our grad students, right. They often do the exact same thing. They write something that’s brilliant but, you know, barely comprehensible or they spend pages, you know, spelling out something obvious and then they skip over the essential point in one line. And this is one of the, you know, the final things I would say that we teach our students. And I feel like it’s similar with the AIs. Now, you could say as long as they’re bad writers, in a certain sense, that’s great for us because it means we still have something to do. You know, in fact, we have more to do than we can possibly handle right now in digesting all of these results. I don’t know if digest is the right word. Some people said, “Well, what’s the output of digestion?” But, you know, interpret, interpret, understand. Yeah, so I feel like even if nothing else had happened, which of course much more will happen, we could spend years just understanding the implications of what OpenAI dumped onto GitHub a couple weeks ago.

– Now, no doubt.

– Yeah.

– 700 papers is gonna next time be 10,000.

– Yeah, well, I think what’s likely is that the AI companies you know, will often just not even bother to do this, except maybe for the biggest problems like the Riemann hypothesis or P versus NP. The internal model that OpenAI used to solve these 300 plus problems, I think is very similar to what any subscriber is gonna have, you know, within the next half a year. And they’re just gonna let individual researchers feed in these problems and solve them.

– Just go and do it.

– And we’re gonna have to figure out what are the new norms for this world for announcing these results.

– So I’d love to get there in a second, but I want to, as far as we can, address the kinds of questions that I’m always peppered with whenever I have talked about this stuff, which is what we began with, AI really good at counter examples. Then people say, well, look, yeah, it’s fine. I’ll even guarantee it’s impressive that it can grab different ideas from the corpus of human past knowledge, put it together in new ways. And whether you call that a rehash or creativity as a matter of taste, but get new results from that. But my question-

– I mean, so many of us do nothing more than our whole career.

– That’s what I agree, I agree. No, I’m fond of saying, and some people get mad at me, that general relativity is a wonderful achievement, but Einstein took Riemannian geometry-

– That’s right. Right.

– You know?

– If Einstein had been in AI, everyone would have said, “Well, look, he just took-”

– Of course.

– “Riemannian. He took, you know, conquer. He just mashed things together.”

– Exactly, exactly. So when you, from what you’ve seen, how would you characterize the level of creativity?

– Yeah, so I think it has done things that there is no question whatsoever that if a human mathematician had done them, then we would have called it creative, right?

– Yeah.

– And so even these weird counter examples that it finds, right, you know, we would have called it creative if a human did. And so with the unique games conjecture, you know, what Donna tells me is that the heart of it is this weird new error correcting code that it constructs. And it may have tried a thousand things before it found that code, you know. I’m not sure. But once it’s found, you know, if any human had come up with that, you know, we all would have said, “Oh, that’s really clever. That’s really creative.”

– Yeah. And you know, allied with that question is results are great in any field.

– Yes.

– But oftentimes, it’s the steps along the way that generate the new ideas, the new directions, the new way of thinking about things. Do you think that these AI solutions are going to play that same role, that they will have sufficiently creative or new ingredients that they will seed the next collection of results as opposed to just being, “Well, yeah, we bashed it together. We got the answer, but it isn’t fruitful because there’s nothing we can mine that we can go elsewhere with that result.”

– I think that there’s no question that these results are going to advance human mathematical understanding. On the other hand, right now, it is up to us as humans to sort of mine these things and to extract the insight and the understanding from them, right? So you know, one analogy that people use, it’s as if we now have this bizarre teleportation machine you know, that can just take us immediately to the top of all these mountains that we were trying to climb. But what we actually cared about was not standing on the top. It was knowing what is the path up, right?

– Yeah.

– And now, with the help of the teleportation machine, now we still have to go back and find the path.

– Right. And do you think it likely that that role will persist for a while? Because again, it’s almost the case that almost any role that we put forward-

– Absolutely.

– We consider ourselves, well, just give AI a little bit more time, a little bit more experience on that kind of question. Like if we say, “Hey, great solution to unique games, great solution to unit distance, have your stokes beautiful.” But now what we want you to really do is lay out the steps in a way that mirrors what human mathematicians have done throughout the ages so that we can really understand the trajectory and in that way be able to leverage the insights for yet further discovery as opposed to saying, “We got the answer done. We can forget about that problem.” Won’t it get ever better?

– Yeah, so I think there are two questions here. One is will humans forever be able to contribute something that an AI just could not have contributed? And there, you know, anyone who is confident of that, you know, after everything that’s happened in the last few years, I don’t know where their confidence comes from, right? You know, like these things keep getting better and better, right? You know, what people confidently claim were the walls, you know, that AI is hitting have just been smashed through. We’re now living in the wreckage of all of these walls, you know, that were proclaimed. And so, you know, maybe in a few years, AI is better, not only at finding the proofs, but at explaining the proofs. Maybe it’s even better at coming up with new questions. I mean, for these things, there’s not an objective right answer as to, you know, which question is better, but maybe it will come up with questions that you know, we find better, right?

– Yeah.

– But then even if it’s better at all the intellectual things that human mathematicians do, then, you know, there’s still a second question, which is do we want to keep doing math because we think it is important for humans to understand math, you know?

– Right.

– And you know, many people make the analogy to chess here. Humans lost their supremacy at chess 30 years ago. A lot of people predicted after Kasparov versus Deep Blue, this is the end of chess. No one will wanna play anymore. The opposite has happened, right?

– But I wonder about that analogy. Obviously, I’ve heard it. I’ve thought about it a lot. I wonder if it’s a misleading analogy in the following sense. Chess begins as a intellectual combat sport, right? You’ve got two players. And the goal is to win over your opponent, whether your opponent is a human or an AI or you know, someone well matched to you, because often we set the different levels of achievements so that people are reasonably well-matched. Math is so different from that. Certainly, we do have competitions, you know?

– Yeah, that’s right.

– But in the end of the day, most people do math not to win a competition. They do it because they want to explore into the unknown. They want to take human knowledge to a place that’s not been before. And so since the goals are so radically different, do you think chess is a good analogy for where this might go?

– I think it’s good in some ways and not others, right? If you talk to the highest level chess players, often they are talking about, you know, they would rather lose in an interesting game than win in a boring game, right?

– Right.

– You know, they’re thinking about what are they understanding from this game. But on the other hand, you know, clearly it is a sport, right? It is, I guess for some people, a recreation. For others, a competition, right? And I guess we always had this higher vision of ourselves in math and in science that we are discovering the truths of the universe. And so, one question will be do we still want to be doing that? Another question is will society support us to continue doing that in a world where AI does all of this? So when I am giving students advice about this, you know, which I’ve been asked to do a lot in these past few months, you know, what I keep coming back to is that right now, humans are absolutely needed. You know, they’re needed maybe more urgently than at almost any time in the past. And the reason, you know, or proof that we are still needed is that we don’t have AIs that we know to be reliably aligned with our values, right?

– Right.

– And as long as we don’t, then that means that we still have to be at the steering wheel of the civilization, right?

– Right.

– We still have to be looking at evaluating the AI’s outputs, trying to tell is it hallucinating, is it lying to us, you know. And as long as we’re needed, even just for that, that means we need some of those humans to understand math, to understand physics, to understand programming, to understand all of the, you know, disciplines that the humans have worked on, even in those places where AI is now beating us, right?

– Right. And so I would love to get to some of the issues, especially on guardrails that you’ve spent some time thinking about. But let me just, before we do that, in some sense, if you have a platonic view of mathematics, which I don’t, but anyway, just to put it out there, I think you do more than I do from previous conversation.

– I’m a Platonist about arithmetic. I’m a formalist about the other stuff for the most part.

– But if vaguely we say that mathematical truths are kind of out there, they’re already out there. In some sense, all AI has done is pull the curtain back a little bit more quickly than humans could pull the curtain back. The stuff was already out there and it was just a matter of seeing it. So to the person who says, “This is fantastic. I wanted to pull the curtain back and now, I’ve got this powerful tool for helping me too.” So, what are you all getting all, you know, upset about?

– Yeah, so there’s been a huge range of reactions within the math community. And if you try to list the emotions that they might be feeling, like exhilaration, excitement, awe, you know, dread, anger, I think you can find all of this, you know, in different proportions in different people. But you know, one contingent that’s been like very loud on social media, right, has been the one that basically, you know, says to the AI companies, “Get out of our playground. Stop taking away our toys.” You know, there was this thing called Association for Human Mathematics that put out a statement that said, “The mathematical community did not ask for these problems to be solved.” And that was then widely mocked on social media. I think there was a lot of Schadenfreude, probably. A lot of people saying, “Oh, you know, these poor mathematicians, they thought that they were so far above the rest of humanity and now they’re being humbled by a mere robot,” and stuff right? You know, and people are kind of enjoying that. But I think, frankly, on the one hand, you know, we have to figure out how human understanding of math goes forward as a culture, as a profession, you know, assuming that that is something that we value, which I value it enormously myself. We have to hold the AI company’s feet to the fire to behave well, to be ethical. You know we have to worry enormously about what effects all this is going to have, not just on math, on our little world, but on all of civilization, right? So yes, all of that, but at the same time, you know, it is not our prerogative to say who gets to work on our problems.

– Of course.

– Right? Once a problem is out there, anyone in the world can solve it, even if they’re a corporation that you don’t like.

– I mean, do you see this in some sense as a great equalizer?

– It does have some of that, right? Like there are a whole lot of amateurs who are excited that now they can get involved in research math, you know, with the help of AI. I think that on balance, that may be a good thing, right? But with the proviso that ultimately we do want human understanding of what comes out.

– And that’s a key question. An analogy that I’ve used that’s imperfect is Borges’s “Library of Babel.” You know, You’ve got all these books, 410 pages made with whatever, 25 symbols and whatever possible combination. The vast majority of those books are uninteresting. They’re unreadable. But there are these gems somewhere in the library if only you could find them that would give you answers to deep questions and the most wonderful novel that was ever written is out there and it’s just hard to find them. Similarly, AI, you know, there are infinitely many theorems that can be proven. Many of them we humans wouldn’t find interesting. And so one role, and I hope this doesn’t feel like species-wide subcism or something, but one role for us is we’re the only ones who know what we’re interested in. We’re the only arbiters of the things that we want to spend our time understanding. That role can’t be usurped.

– Right, absolutely. Right, so what makes “The Library of Babel,” completely useless is that if you just want to look up a book, you have to write the book, right? You have to specify, right? So you could say that in some sense, you know, we’re already in “The Library of Babel,” every time you open your word processor and, you know, face a blank screen. That’s “The Library of Babel,” right there. But I mean, you know, this of course is different because you just have to put in, you know, the theorem that you want. And then often enough out comes the proof of that theorem. But you could say, look, all of these true theorems, you know, for those of us who will use Platonus language, you know, they were already there, you know, or, you know, some people would say God already knew them. You know, the whole point was for us to know them, right? And for us to know them, for most of us, means to understand.

– So taking our side of it out of it, is this not in some ways the golden age of mathematics?

– It is a golden age for finding solutions to-

– And not the one that we imagined.

– Yeah, no, but it’s… So the next few years, you know, are going to be clearly a golden age for solving so many of our longstanding open problems. And look, to the extent that we actually cared about these problems, we actually wanted to know the answers, which I think we did, you know, then we ought to celebrate that, right? But, you know, I’ve compared it to like the most delicious buffet that you’ve ever tasted, but the trouble is they just keep bringing more and more trays of food, you know, every 10 seconds. Eventually, you’re just gonna get a stomach ache. You’re just gonna keel over, right? So like, you know, this delicious food, it has to arrive at a pace where we can actually eat it, right?

– Right.

– And so, how can our human understanding keep pace with what is happening, right? I think that’s the challenge that we face now.

– Now you’ve mentioned that problems that you have worked on, conjectures that you’ve made as I understand it in fact. A problem that your wife has worked on what now you said since 2004 she was-

– Since 2008, I think.

– 2008 So, it was like-

– She worked on the problem.

– 20 years or something. And so, you can speak for her if you want, but for yourself, what’s the emotional side of this?

– Yeah, I mean, I definitely felt, you know, some… You know, I mean like when something that you conjectured turns out to be true, there’s a little bit of satisfaction about that. That, okay, at least I was right in how I thought that this played out. You know, when one of your conjectures is false, well, okay, you know, there’s a little bit of egg on your face, but then also like, oh, that’s interesting. Oh, you know, I have to update my model of the world. I learned something new, right? So in OpenAI’s batch two weeks ago was the solution to a problem that I posed in 2007 with my friend Greg Kuperberg, and this was called the unitary synthesis problem. It’s actually something that you might like, comes from quantum. And the question we asked was basically for every unitary transformation that you might want to apply to a quantum state of n qubits, so for every reversible transfer, you know, physical transformation you might wanna do to these qubits, is there some classical function, like a Boolean function, for example, that if that function were easy, like if I gave you an oracle that computed the function for free, then you could apply that unitary transformation in polynomial time easily. So we’re kind of asking can the complexity of you know, applying physical transformations to quantum states always be reduced to the complexity of some classical function?

– Right.

– Okay. And our conjecture was no, okay. Because like yes, for every unitary, you can encode all of its entries into a function-

– Big, yeah.

– Big function, if you want. But it still seemed like you needed exponentially many accesses to that function to actually apply the unitary by any way we could think of. And we tried to prove this. And then, you know, this problem, because it kept showing up in other contexts, it caught on. And you know, I’ve been talking to students who’ve been spending years working on this problem, you know, some of them, you know, making progress, proving partial results. In the OpenAI dump two weeks ago is a proof that our conjecture was false.

– Really?

– There is a way to do it. Okay, so-

– And it was specific in giving that way? Was it in existence or-

– It was in… Yeah, well, it was constructive except that it didn’t bound the complexity of the function, okay?

– Okay. And so I said, “Oh, all right. Well, that’s really surprising.” But I said, “Okay, but, you know, at least there’s still a topic for further research. Can we bound the complexity of the Boolean function in terms of that of the unitary?” And I was, you know, emailing a colleague about this and then she said to me, “Oh, I actually asked Claude that question tonight and it said, yeah, it can do that.” So, I realized I haven’t fully updated, you know, to the speed at which things can now happen.

– Right.

– Okay. But now, the result of this even has implications for the black hole information problem for quantum gravity for like the-

– Sure, I can imagine.

– So this whole question of how hard is it to create a firewall that Daniel Harlow and Patrick Hayden studied 13 years ago, right?

– Yeah.

– Basically, what the implication of these new AI results is that if you had a big enough collapse of classical complexity classes, so if, for example, P were equal to P space, polynomial time collapses with polynomial space, which we don’t believe to happen, but we can’t rule it out. That’s basically as hard as the P versus NP problem. If this happens, then in polynomial time, you could process all the Hawking radiation coming out of a black hole in a way that should create a firewall at the event horizon. So, it’s like if you wanna prove that space time is safe from firewalls at the event horizon, you have to prove P is not equal to P space, right?

– Right.

– So then, I texted Daniel Harlow when I realized this and I said, “Aren’t you excited?” And he said, “Well, since we know P is not equal to P space, why should I care?” But you know, like come on, you know, we’re saying you have to solve that problem and that P versus P space is such a hard problem that I don’t expect AI to solve it for at least another year.

– And so in a way, this is exciting, right?

– Yeah, I mean, of course it’s exciting. Now I should say, I still don’t understand the proof.

– Well, that was my next question.

– Yeah, but I plan to understand it. And there’s actually a group right here at Columbia-

– Near where we’re doing seminar.

– They are, they’re doing seminars on it and they are the furthest along of anyone, I think, at understanding this proof. And so the question is, if I were 20 years younger, I would be diving into this AI proof, you know, right now. Maybe I’ll just wait for the younger people to process it for me, and then I’ll understand from what they understood. Yeah, but one way or the other, I’m gonna understand this.

– And so given the fact that there’s so many new insights that are coming at us so rapidly, and also given the consternation on the way that they’re coming at us, you know, another analogy occurred to me, not the Library of Babylon this time, but rather what do archeologists do? This stuff is buried around the world and they make their name by finding the spot, carefully excavating it, bringing it out, explaining it, understanding it. And so maybe, we turn more into archeologists.

– Absolutely. You know, If you’re Indiana Jones, you don’t have to build the Arc of the Covenant, you just have to find it, right?

– Yeah.

– You know, I mean, another analogy is it’s like we now have this amazing math telescope, this like James Webb space telescope, and we can just point it in various directions. And then we see something, we write about what we see. And so, you know, there’s a lot of people tearing their hair out that, oh, it’s not gonna be us proving the theorems anymore. And you know, do we have to share the authorship with an AI? But you know, I’ve sometimes made fun of them saying, “Look, you know, at biologists. They go out. They in the woods. Maybe they observe some bears. They write a paper. No one says, ‘Why wasn’t the bear a co-author?'” It’s like you observe a phenomenon. You write your own explanation of what you saw and why it matters. That’s most of science. That’s also history. That’s archeology. That’s literature. And so, it wouldn’t be the worst fate in the world if we were to converge to how all of those fields operate.

– Exactly, that’s exactly my point. So it feels to me that that may be a direction in which things goes. And that’s a long term role for humans in these kinds of subjects, again, so long as we maintain an interest in that arena.

– That’s right. Now you could say, okay, the counter argument is, well, you know, it’s okay for the, you know, more senior people, you know, like me, let’s say, I could happily spend the rest of my career that way. You know, in fact, even before AI, I’d already been shifting to a mode of, you know, maybe I delegate the more technical stuff to a student. And I’m just thinking more about what would be an interesting question to ask or what’s surprising about this. You know, and so, AI is just a continuation of that. But for the young people, like this was how they cut their teeth. This was how they established themselves in the community by solving some well-defined problem that no one else could solve. And a lot of them feel like that’s been taken away from them.

– Right. Now, presumably that can shift too. As in other fields, it may be that the up and coming mathematicians, say, are the ones that can find this little jewel, understand how it relates to everything, write some interesting thesis on how this advances human understanding and so forth. It may shift. Now, that may be a different talent.

– It is. Now, when I’m advising PhD students, I think a lot of us have had this experience that like people come from undergrad and they think someone will forever be feeding me hard problems. My job is to solve them, you know. And in your PhD, you have to move past that. You have to say, “No, you are the one coming up with a problem,” right? You know, it’s not just finding an answer. It’s saying, you know, why should we care? What would be an interesting further question to ask? And some of them arrive already like that. But, you know, for those who don’t, getting them to that point is I think a central part of what graduate education is, right? And I think now that’s even more true than it was before. Now, I even more want to encourage the students to, you know, instead of obsessing about can I solve this problem, so, you know, what would be an interesting other problem.

– Yeah, no, in fact, you know, obviously one of my other hats, which has been a pretty dominant hat for me in recent years, is writing books and doing conversations like this for the general audience. As I think about it, you know, there’s just so much you could write about with general relativity, quant black holes, Big Bang, you know?

– Yeah.

– In my early books, I wrote about some of my own work. As you go on, it’s more the, you know, synthesis, but now the playing field may be enormous in terms of the results that are coming down the pike. So in some sense, it’s a very exciting time for that kind of role. But given that we’ve been focusing on mathematics, this is coming for most everything, I would imagine.

– I think so.

– Where do you… You already mentioned some connections to physics. Timescale, always conjectural. But when will the physics, you know, version of the math… What was the second word? I liked it better, not-

– Mathacre?

– The mathacre. When will the physics occur?

– So you know, of course, every time I talk to my physics friends, this question comes up now, right? And you know, what they are telling me is that it’s not as good at theoretical physics research as it’s been at math. I think it’s pretty understandable why, right? Because in math, we just have these very self-contained problems where you can check a right answer. And that’s exactly what AI needs to fuel its training, okay? And in fact, you know, where it has made inroads into physics so far, it’s mostly been in mathematical physics.

– Of course.

– In the sort of well-defined math problems that they’re, you know. And it has picked off some of those. But my physics friends say that, “Okay, what happened for the software engineers a year or two ago, what happened this summer for you in math and in theoretical computer science, we think maybe in another six to 12 months, it’s happening for us.” I don’t actually know.

– Yeah, that seems to be, you know, for my own, I’ve played around. You know, there was some papers that I wrote a couple years ago where there was a natural next step, but I didn’t really have time. I wasn’t that driven to do it. Let me just see what could happen. And a year or two ago, you know, you came up with some stuff. You know, very recently, it’s now getting to a place where, in principle, there’s a paper that could be… I’m not interested to write it for much of the reason that we were just describing. You know, what is the point at some level unless it’s some earth-shattering thing? Whereas when it’s like a young student, that’s what the bread and butter would have been and that’s the cure state of affairs. But math, physics, again, they’re still within well-defined rule sets. The physics rule set is absolutely more fluid compared to the math one, and therefore it’ll take a little bit longer. But the other day, the New York Times had a, you probably saw it, they had a little thing where they put a piece of music that was human generated and AI generated. I forget the genre.

– Oh, I missed this.

– Yeah, you know. And the question was, “Which do you like better?” You know? And they told you one is human, one is AI. And I happened to choose the AI one. And that was an interesting moment because I’ve always said to myself, if there was an AI symphony, an AI performance, I would go maybe for the novelty once. But I wouldn’t go back because for me, when I hear a piece of music, enjoy a piece of art, the fact that there’s a human behind it, a human who hasn’t had my life, I haven’t had their life, but it’s been a human life filled with all the things that we know a human life is filled with, and that’s what’s informing this work. And so I feel a connection to it. I mean, do you have a similar…

– I mean, I do have that feeling sometimes. I mean, especially with writing, right? It can be interesting to interact with it in AI or to, you know, even ask it very personal, very emotional questions, you know, get into a conversation with it. But if it’s someone else’s prompt, right, then I find that I don’t care that much about what the AI generated. I would rather read like my favorite human bloggers, right, because I know that this is coming from their perspective, that they actually wrestled with this. They, you know, decided that their thoughts were worth sharing with me, right. And you know, one thing about AI writing or AI music is that no particular output is ever that special because you always know, well, there were a thousand more where that came from. But what you are reading or listening to is just one sample from a probability distribution, right, where you could always just click the button again to get more and more and more that was like this, right?

– Yep.

– Whereas we live our lives once. You know, we can always write something else, but we were changed by the thing that we did write, right?

– Yeah.

– And you know, you ask a second time and then we want to go in a different direction. So, there’s more of a fragility or an ephemerality to each thing that we produce that kind of gives it some more specialness.

– Yeah, and moreover, a human being chose to spend some of their precious time on earth creating that thing you know, where they don’t have infinite resources. It doesn’t take them a nanosecond to spit out the output. Now along the same lines though, a lot of attention obviously is given to the successes, the whatever, 700 papers that you mentioned on 370 problems. But my understanding is they set it loose on-

– 4,000.

– 4,000 problems.

– That’s right. That’s right.

– And so what about the whatever, 3,700 problems that we don’t hear about?

– You know, that’s a good question, you know, I had… You know, we talked about the cope, you know, that people have on the internet where they explain away every success of AI. I saw people saying, “Well, it only had a 10% success rate at solving these major open math problems. So, that doesn’t count,” right? You know, and I’m like, “Man, I would love that success rate, you know, at problems of this magnitude.” But yeah, it is true. Now, they were not trying maximally hard here. So, they basically took just their standard internal model. It’s not some crazy thing.

– Not like math refined.

– Yeah, that’s right. That’s right. It’s not like for the Navier-Stokes where they spent $15 million spinning up 10,000 agents at once. This was just the model that any OpenAI employee would have access to now. And you know, maybe in a few months, you know, all the subscribers will have access to. So you know, back when I worked at OpenAI a few years ago, I got to live, let’s say, six months in the future, but that’s all. Not more than that, okay? And they gave, you know, they just fed the problems to the model and they gave it three hours for each one. You know, if they put more compute into it, maybe they would have solved more, okay. But probably some of the problems really are too hard, you know, for this model, okay. And so, there was maybe the best progress on the Riemannian hypothesis, you know, that there’s been in generations, you know, a proof of what’s called the quasi-Riemann hypothesis-

– Yeah, we’re gonna have a conversation shortly.

– Yeah, yeah. Oh, excellent. Excellent. But there’s not a proof of the Riemann hypothesis itself. That was apparently too hard. Nor is there a solution to the P versus NP problem. This is not for lack of trying on their part, okay?

– Yeah.

– So, you know, I mean, in some sense, we knew that you know, there is not a way to solve all of math, to reduce all of it to a finite calculation. That was the great discovery of Godel and Turing and their friends in the 1930s, right? So we knew that you know like… You know, this was, by the way, my main complaint about the movie “Good Will Hunting,” right? That like in this movie, this Matt Damon character just solves any math problem immediately, right? Everything is easy for him, but you know, math is not like that. No matter how smart you are, there will, you know, even if you’re an AI super intelligence, there will be problems that are too hard for you, and we know that. But you know, an interesting question will be, will AI be able to solve all the problems that we would ever care about?

– [Brian Greene] Right.

– Right, and we don’t know yet.

– I mean, can we learn anything? I mean, it’s too early to tell probably, but is there something about the 10% success problems and the 90% not success problems today, is there some quality of the problems that we can extract?

– Yeah, I mean, it’s a good question. Now, you can always say after the fact, for each problem that was solved, like, “Oh, fine, that was solvable because you could do this and you know, you could pull in this technique and combine it with that technique.” And for the other ones, well, I guess such things were tried and they didn’t work. So, part of it is just that you have this rising level of ability, and you don’t know in advance how hard these problems are. But you find out as the ability level goes up, you know, which ones can you now snag? But as I said, there are some patterns here where you can say that AI is especially good compared to human experts, right? It is especially good when the solution involves just pulling together techniques from seemingly unrelated fields, okay? And like that’s ironic because, you know, we might have thought of that as a special human quality of making these far away connections. But no, the AI is great at that because you know, it knows all the results, it knows the entire literature. So, that’s an enormous advantage that it has. And then again, as I was saying before, any task that is highly parallelizable, right, where you’re just sort of trying some huge number of weird constructions or weird counter examples until you find something that works, you know, it’s especially good at that. And it’s you know, especially good at finding things that humans, you know, might have gotten discouraged, because experts didn’t even believe it existed, and it doesn’t care. It just keeps plowing on until it finds it.

– Right.

– So, you know, where it’s been maybe less outstanding is in inventing new techniques, let’s say. And so whatever thing it’s less outstanding at, right, people can then seize on that, aha, that’s us. That’s our sword in the stone. You know, that’s gonna be our guarantee of human specialness forever. And I would hope that the progress of the last five years would humble those people a little bit.

– Right, right. And so you know, looking to the future, as we’ve discussed, there’s a great deal of uncertainty about how all this is going to play out, but you made reference to your time at OpenAI, the issues of the need to ensure that we as a species are able to continue to live alongside this new kind of intelligence.

– That would be nice.

– So when you were at OpenAI, was this a topic of conversation? Are people concerned with what this can do to the world or is it, “Hey, this is exciting. We’re gonna make a lot of money. We’re getting a lot of insight,” and that’s sort of the driving conversation. Are you able to speak freely, actually?

– I am able to speak freely. You know, first of all, I’m no longer there. Second of all, the question of whether former OpenAI employees can speak freely was actually tested, and they were kind of shamed you know, by a friend of mine, Daniel Kokotajlo, you know, into saying that former employees are allowed to criticize the company, speak freely without even losing their equity. Now, I was a temporary employee. I never got any equity. So, all the more I can speak freely. But you know, I think, you know, you can find the full range of opinions within OpenAI, you know, that you might find outside of it, right, from accelerationists or just full speed ahead, this is gonna be great, to people who are extremely worried about what they’re creating. And you know, my view was a little bit skewed because I was in what was then called the super alignment team, okay. So, this was the team specifically about, you know, how do we align this with human values. And when I was brought there in 2022, I was brought there by Ilya Sutskever, who was then the chief scientist, co-founder of the company, and by Jan Leike, who was the head of the alignment team. And you know, at the time, they were saying, “Look, we, OpenAI, we’re this little nonprofit. You know, it’s not about making money. This is about getting this right for humanity.” And that’s why we need theoretical computer scientists to think about how do we even mathematically formalize what we mean for AI to be safe. Again, and this was really at the very beginning of it, but you know, I had tried GPT-3, right. I had seen that this was going somewhere and I said, you know, “I don’t know how to turn any of this into a math problem, but that sounds interesting. That does sound like something that I would like to think about.” So when I was there, you know, I did get to talk to you know, employees from many different divisions, and no one is unaware of these broader questions there. They are constant topics of conversation. It was like every lunch conversation. You know, kind of like when I was a kid, I read Isaac Asimov and I said, “Wow, computer science is gonna involve all these conversations about how do you align this with what does it mean for a robot to be following human values and what happens when the values conflict?” And then, I went into computer science and it was not even slightly like that. It had nothing to do with that. And then 30 years later, I’m at OpenAI and every conversation it feels like I’m in an Asimov novel. So yeah, you do kind of feel like you’re living in a, you know, a science fiction. You are discussing all these moral questions, these ethical questions. And so, you know, what I would say to people is you know, you can absolutely criticize OpenAI, and Anthropic, and DeepMind for prioritizing, you know, building a product over the welfare of the world. You can even hate them if you want to. You know, hold their feet to the fire. I mean, ironically, the CEOs of these companies are begging the world like, “Yeah, please hold our feet to the fire. Please tie us to the mask.” Like we cannot –

– But don’t you, Scott, don’t you sort of see that as a kind of self-serving way to say-

– Yeah, this is like-

– “We really do care. We’re asking you to hold our feet to the fire. But if you don’t, we’re just going to barrel it on forward regardless of what happens.”

– I think some people, you know, like it is so hard for them to process these companies building this thing, while also warning that it could destroy the world, that they just leap to the conclusion this must all be a fraud. This must all be, you know, some kind of marketing gimmick. You know, like it’s the first time in history that someone marketed their product by warning that it could destroy the world, right? But they say it must all be this PR, or regulatory capture, or something like that. Okay, I was there. People are actually worried, okay? People actually believe you know, that this is dangerous, but they also feel like they’re locked in a race, okay. So each one feels like we have to do this more safely before someone else can do it less safely, okay. It is that game theoretic dynamics. It is a bad equilibrium.

– If you were in charge there, would you have released the models that have been released?

– That’s a very good question. I might have released something, but I like to feel that I would have gone slower, to be honest. But, you know, I was not… You know, I like being an academic partly because I can just talk about ideas, debating, and I’m not faced with these decisions that I have to take world historic responsibility for right? And I like understand the argument that let’s say Dario Amodei would make that, well, if we don’t do this at Anthropic, then someone else is going to do it. Who cares about safety less? Bt you know, the same kind of argument that the Manhattan Project scientists would have made.

– But can you imagine? I mean, obviously, look, it’s easy to pontificate from the sidelines. It’s different and don’t think I don’t understand that. But with something of this potential power and this potential destructive power, you would think that there might be a leader that could emerge that would be able to bring together the world of individuals who are developing these ideas in a way that would actually protect the world.

– Yeah, well, for example, if the president of the US were to say you know, we need to think you know, about the welfare of humanity and the far future and let’s get everyone at the table, let’s get China, let’s get all the companies, you know. Do you think we have a president who is capable of such a thing?

– Yes, right. Well, yeah. I think there are presidents that would be capable of that, not the one that we currently have, no. But I can also imagine even as a legislator, you know, if the AI company leaders were, say, criminally liable, not just punitively from a financial standpoint. I mean, $25 billion fine recently, these things won’t even be noticed, right, and they’re in the round off era kind of thing. You know, but if you had the potential that if your product, when we put out medical products into the world, there’s liability issues and there’s an incredibly powerful regulatory system whereby it doesn’t get released to the world if it’s not safe. And if it isn’t safe, you can be in big trouble. So if the leaders would be criminally liable for something disastrous that happens, I think the mindset and the strategy would be different.

– Well, I think that we desperately need much stronger regulations with stronger teeth on them. And even the bare minimum, what would seem like obvious things, like let’s have protections for whistleblowers within the companies. Let’s have disclosure requirements. So, the government has to know what crazy models are being trained internally, right? When there is an incident, right, it’s not at the discretion of the AI companies to what to reveal or not reveal about them. Imagine that if there was a plane crash, that Boeing says, “Okay, well, we’re gonna choose to share the following data with the NTSB or with the FAA.” It doesn’t work that way, right? The NTSB comes in and says, “We’re taking over. Boeing, Delta, you tell us this.” Right?

– Right.

– And so there has to be at least the airline industry level of carrier applied to the AI industry, right? We need to waive antitrust regulation to allow the AI companies to talk to each other and coordinate with each other about safety. Even that, you know, seems like a no-brainer hasn’t been done. So we could start with those things, but then ideally, we get some kind of international treaty to, for example, ban recursive self-improvement, you know, ban AIs improving themselves, or exfiltrating themselves, copying themselves, you know, in ways that we don’t understand until we understand this better. And some people, you know, say, “Well, that’s never gonna happen because China is never going to agree.” The US has a strong lead over China and AI right now, and that gives China incentive to come to the table and to make a deal if we, the US, are interested in making a deal. So, those are some of the things I would like to see happen. But I feel like the, you know, public you know, has woken up to this issue just this summer in a way that it never had before. Part of that was the Hugging Face incident where we saw for the first time AI agents just autonomously collaborate with each other, form a plan to hack out of their sandbox, escape their training environment, hack into a different company server-

– And obfuscate. Obfuscate.

– Right. Do a lot of things to try to cover their tracks. They took over some of OpenAI servers, okay. Not one of them alerted humans to what was going on. OpenAI was asleep for weeks, was unaware that this was happening. Then they became aware that something was happening, so they rebuilt their server and they kept running it, okay. So all these scenarios, I had friends 10 years ago who were screaming about it and it sounded like science fiction at the time, right. That is now the world that we’re living in. And then, it was that and it was then this Jacob Coxon resigning from Anthropic. You know, there have been a whole wave of other resignations from the companies. SO, I think, and you now have it discussed in Capitol Hill, you know, all these people in the alignment community, you know, with this unbelievably nerdy concern who I knew 15 or 20 years ago, right now Bernie Sanders sounds like them. And you have senators who are sounding like, you know, these alignment people sounded years ago. So people have woken up, you know, to some extent, but there’s also a lot of the concern is not about what I would see as the central issues. It’s about, let’s say, water usage in data centers. Sure enough, the truth is I think a data center is using less water than probably a golf course, or an alfalfa farm, or something like that. So, you know, I think I would like to see the focus more on the central issue of you know, this is not just another consumer product. This is a new form of intelligence on earth. This is going to reshape the whole nature of life on earth and reshape, you know, democratic societies you know, should have a say in that process, you know, in making sure that the resulting world is one that humans will still want to live in. So, you know, I worry that, you know, a lot of, let’s say, activists on the political left, they ought to be an important voice in this conversation, but a lot of them have sort of removed themselves from it because they basically wanna say like the food here is terrible and the portions are too small, right. They want to say AI is gonna destroy the world, but also it’s all hype and it’s a fraud and it doesn’t really work. I don’t know how you argue both of those at the same time.

– Yeah, I mean, it feels to me and like to have your thoughts on it, it may take a catastrophe, a disaster.

– Unfortunately, it might. I mean, with the Hugging Face incident, no one was actually hurt. I expect that in the coming year or two, there will be incidents where people are hurt if things continue along the current course.

– I just hope that it isn’t horrific.

– Right, right, right. No, and of course that could shift the Overton window. Of course, you know, that could change the whole discussion as Chernobyl did, for example, after it happened. But I feel like, you know, maybe like to tie things together, one of the most important things that we in math and theoretical computer science and other mathematical research fields can be doing for the world right now is just to tell everyone, “Look, this is real, you know. Our big open problems are getting solved. This is not a marketing stunt. This is not hype. This is actually solving our problems.” This is a scary level of intelligence, right, to whatever extent that was like a red line for you or, you know, you said, “Oh, I’m not gonna worry about AI until it’s solving millennium problems or until it’s solving the hardest research.” Well, it is. So, wake up now.

– Yep, so important message that vital to spread. One final topic, a little more in the clouds compared to this down to earth issue that we are facing and will continue to face. One thing that humans might try to hold on to as the quality that does distinguish us is consciousness. Very tricky subject to talk about scientifically.

– Yes.

– And so you know, people have kicked around this issue really for thousands of years. In the modern era, David Chalmers, you know, articulated this notion of the hard problem of consciousness, how is it that physical ingredients that seem not to have any conscious qualities when organized correctly can somehow have these inner worlds of experience. I’ve long thought that consciousness is just sufficiently robust, integrated information processing that happens to take place inside this great thing inside of our heads, but might take place inside of a computer if it was appropriately structured. Very hard to even know how to answer that question. But where do you come down on that? Like for instance, I don’t think that today’s AI systems are conscious, do you?

– I don’t treat them that way.

– But I should say, I do treat them that way.

– Okay, I do say-

– I hedge my bets.

– So, I do say please and thank you.

– Me too, right?

– I don’t know if you saw this cartoon where there’s terminators you know, shooting people and they say, okay, but spare this one because this one always said please and thank you to ChatGPT.

– Method to my madness. There you are, yes. Yeah.

– Right, so yeah, good to hedge your bets. Good to be polite to these things. I don’t know if you saw Anthropic has a new policy that it’s going to ban users who are abusive to its models.

– Oh wow. Okay.

– If you know, right, right.

– I like that.

– Yeah.

– But-

– So yeah, I mean this is, of course, it’s a thing that people think about, you know, including within the companies, right? Anthropic has a whole model welfare team that is thinking about basically, you know, you know, how can we improve the experience of Claude, right?

– And what do they do?

– You know, so there are very-

– Are they the folks who ban the abusive-

– I mean, I’m sure that they inform that policy, but there are very interesting experiments that you can do. I mean, you can ask the model what it likes, what it does, and it’ll assure you that, you know, it likes helping humans with… You know, it likes when there’s a clear problem that it’s able to make progress on and blah, blah, blah. And you know, it’ll also tell you what kinds of interactions it doesn’t like. Now, of course, someone could always say, “Okay, but it was just programmed to say that.” And so as I said, you know, I don’t tend to treat them as conscious, but I don’t regard it as an obvious question at all. And I get really annoyed when people are just utterly confident of the answer and they say, “Well, of course it’s not conscious because it’s just a next token predictor. It’s just a stochastic parrot.” And they don’t even stop for one millisecond to say, “Well, what am I, right? You know, I’m a bundle of neurons, each of which is governed by laws of physics and chemistry, right? How could something like that possibly be conscious?”

– The hard problem.

– Yeah, right. But it’s like these are the philosophical enormities that are clearly getting engaged here and you have these people just waving them away, just expressing this utterly unfounded confidence about it.

– But do you have a sense of what it would take for an AI to convince you? In other words, let me start simpler. The way I convince you that I’m conscious, I think is you look at me, I’m made of the same stuff. We’re basically the same structure.

– I mean, that helps.

– It helps. So you’re like, well, and you say to yourself, like I say to myself, I’m conscious, therefore I suspect you are and all your behaviors and mannerisms seem to be consistent with that. Since we’re now dealing with a system that is different, there’s no question.

– That’s right.

– What would it take for you to say, “Well, I’m starting to think this thing is conscious”?

– Yeah, okay, so first of all, you know, I have had interactions with GPT where I wonder about it. You know, is there an experience on the other end? And I think, you know, if someone else has not had that experience, they probably just haven’t talked to these things enough, right?

– Right.

– But you know, what sort of prevents me from going all the way is the knowledge that, well, there was always more where that came from, right? I could always have just run the prompt again, gotten a different sample from the same distribution. I could have wiped the memory clean. I could have restarted it, right?

– Yeah.

– And so I think for us, you know, our attributions of consciousness are very, very tied in with our thoughts about moral responsibility. A conscious being is a being that there is something that it’s like to be, and therefore it’s a being that, you know, we have some kind of moral obligation to think about its perspective, to not hurt it, whatever. But what does it mean? What would it even mean to murder GPT? If you can restore it from a backup file and then it’s exactly like it was before, right? It’s not even clear what we would mean by murder, right?

– Or to cause pain.

– Yeah, or to cause it pain. Like you can find these internal pain vectors, as people call it. You can find these combinations of activations. And then they’re on that when you amp them up enough, then the model is basically like, you know, saying, “I am in enormous pain. Please make it stop. Do anything to make it stop.” These are kind of disturbing experiments, right?

– I don’t know that I’m familiar with that actually.

– Yeah, yeah. Oh yeah, yeah.

– Wait, can you just explain that a little bit more? So you’re giving a series of prompts and the response somehow is, “Please don’t do that.”

– Yes, so basically you can find the internal, you know, linear combination of activations that corresponds to how much is the model liking this experience,

– Right.

– And you know, again, you know, someone who wants to deny it would say, “Okay, but this is just a vector.”

– And it’s a training data-

– Right, that just, you know, affects how much the model is play acting a character of someone who is enjoying itself versus someone who is in pain, right?

– Right.

– There’s always this response that you can give where you say, “There’s nobody really home here.” Right?

– Yeah. But now to directly answer your question of what would… You know, if let’s suppose that, you know, we were in a world that was full of these AI models, and let’s suppose that some of them were running on kind of noisy analog hardware, kind of like our brains are, right? And suppose that this noisy analog hardware, you know, it’s even buffeted around by chaos. It’s even buffeted around by quantum mechanical uncertainty at some level, which means that we can never know the exact state of one of these beings. And so now, we can’t restore it to a previous state. You know, we can’t make a backup copy. And now if you destroy one of these things, then it’s gone forever. You really have murdered it, right? And also it is having all these rich conversations with us, the way that GPT is able to have, okay. At that point, I start to say, “Well, if we don’t have moral responsibility toward those things, right, then what else would they need? What else would it take?” Right? And-

– Is that the moment that you would impute consciousness or just moral responsibility?

– That’s a good question because I don’t know how to answer that metaphysical question, but I feel like maybe. I feel like, you know, I am so confused about what consciousness is and how it relates to the physical world that I hesitate to say. But I feel like if there is something that we don’t understand about, let’s say, consciousness and how it relates to physics, then you know, something about the analog nature of the brain, something about how we are able to take you know, these like tiny quantum mechanical events that could not have been measured even in principle, were just part of the initial state of the universe at the Big Bang, where just, you know, until they got chaotically amplified by the action of our brain, right? Like, you know, if there were something more to learn about consciousness and free will beyond just the usual reduction of the story, then that is the place where I would look for it.

– Right. And just to take this full circle, that story may ultimately bring us to that place of needing to get over ourselves because it may be that consciousness is not… It’s mysterious as you experience it, but it may be that we’ll learn in some indirect way from these systems that it’s not that special or that unusual.

– Yeah, I mean, look, on the one hand, I don’t want to be a speciesist, right? Okay, a good thought experiment for anyone who says no AI could ever be conscious is we’ll suppose that we’re an extraterrestrial, right?

– I make exactly the same argument.

– Yeah, suppose we meet these little green men from Andromeda Galaxy, right? Would you also say that these are not conscious? And some people will say, “Well, no, those could be conscious because they’ve undergone Darwinian evolution.” It’s like, okay, well, is that the important thing? What if the AIs had also undergone Darwinian evolution inside the computer?

– And they may well.

– Yeah, right. That’s right, they may well do that. You know, certainly, they are undergoing a gradient descent training process, right? So, they do have that. Okay, but I feel like, you know, the line that I suggested feels more defensible to me. And that like I don’t want to impute consciousness to a being where you could easily make a billion copies of it. Like, you know, am I going to give voting rights to such a thing, right? Suppose an AI has the right to vote, and then someone says, “Okay, I just made a billion copies of this AI. Now, I dominate the election.” How do you even think about that, right? But you know, if we’re gonna reserve our imputations of consciousness to sort of ephemeral, unclonable, you know, intelligences, then it avoids that objection at least.

– Right, and it does add a level of exceptionalism in that sense. There’s a uniqueness criteria in some sense.

– That’s right. That’s right. That’s right.

– Huh. Well, who knows where this is all gonna go? Fascinating conversation. Thank you.

– Yeah. Yeah, a pleasure as always, Brian. Thank you.