– Others in the field have said to me, I was blown away by what Chat was able to do.

– I certainly wasn’t blown away. It was always obvious that if you had a richer database, you’d do better, as you increase the data, these models were gonna get better at what they do, and what they do is they build an approximation of how people use words. It was also obvious that that was not enough to get to artificial general intelligence.

– Hey everyone, thanks for joining us. Today’s conversation is in the area of artificial intelligence, the mind, the future of humanity in an ever increasingly digital world. And I’m so pleased that our guest today is Gary Marcus, who is a scientist, entrepreneur, and bestselling author. He’s recognized as a leading voice in artificial intelligence. He is professor emeritus at NYU, I guess that would be in the Departments of Psychology and Neuroscience, I believe. And he’s also the founder of Machine Learning Company’s Geometric Intelligence that was acquired by Uber as well as many other interesting AI ventures that we’ll discuss. So Gary.

– Great to be here.

– Thank you so much for joining us. You know, before coming down here today, I took a look and I think you’ve been in seven previous World Science Festival programs, if I’m not mistaken, over the years.

– Is that right? Some of them very memorable. Like the one that I did with Pat Metheny, where he did the algorithmic composition with his, what they call solenoid robots.

– I don’t even know if I’ve seen that one. There are World Science Festival programs–

– That one’s never been released, but we’ll talk about that later.

– Oh geez.

– It should be.

– So that’s an interesting secret back there. So it must be a rights issue or something like that. Welcome back to New York, I gather from what we were discussing earlier, you’re now based in Vancouver.

– That’s right.

– But no doubt coming back here brings back the memories.

– Always love being back.

– Absolutely. So I wanted to begin with four questions, which is sort of the right number for this time of year. And I want you to just give a single yes or no answer to each, and then we’ll dive in.

– Why is this AI different from all other areas?

– There you go, exactly. But I’m not gonna phrase ’em exactly that way, but that would be more appropriate. Question number one, do you think that the whole idea of scaling, more data, more compute power is kind of reaching an asymptote? And so this may not be the way that future breakthroughs will be found?

– I think we’re already moving away from scaling. People don’t wanna admit that because they make a lot of money selling the scaling hype. But the reality is that most of the progress in the last couple of years has actually been from other stuff. People kind of slough it off, they call it the harness, but that harness is really symbolic AI. You’re starting to use classical AI like loops and conditionals, python interpreters and all this kind of stuff. So if there was hypothesis that literally all you needed was scaling and if you had enough scaling, you would get AGI. And everybody’s now realizing, even though they’re not saying it out loud, that that’s not actually working. And so they’re using all kinds of gadgets outside the LLMs that are not reliable in order to kind of rescue the mission.

– So that would be a no.

– That’s a no.

– Perfect, all right. We’ll go further into that, second question. Evolutionary psychologists have long known that we humans tend to over attribute agency and intelligence to the external world for natural reasons. Do you think that plays a part in the kind of ways that some of us impute a kind of human essence to these LLMs?

– 100%, yes.

– Good, all right. Third question, related to that. Do you think it’s ludicrous when some of your colleagues, people in the actual AI field, not just the general person who may be influenced by, you know, our evolutionary past, when they say that current LLMs might have a degree of self-awareness or consciousness, do you think that’s ludicrous?

– I think it’s absolutely ludicrous.

– Okay, and final question, again, we’ll get into all these. Is it hopeless to take up a musical instrument as an adult?

– It is not.

– That’s very hopeful. And we’ll get there as well. So getting into it, your background, you began studying the brain, neuroscience as a graduate student. Isn’t that what your degree was in?

– My degree was in cognitive science, brain and cognitive sciences technically. I first got interested in computers when I played with a paper computer at age 10. My science explainer, my own science explainer career started the same day. So I learned how to use a paper computer.

– What’s a paper computer?

– You could say it’s a simulation of a computer, or you could say it is a computer, depending on how you make your definitions. But you have registers, you have a tape that moves along. And that night, I explained on television how this thing worked.

– You did?

– This cute kid explaining.

– Where was that?

– This was in Baltimore, Maryland. It was on the campus of Johns Hopkins University run by a program for gifted students in Baltimore City Public Schools. And from that minute on, I became obsessed with computers and artificial intelligence. And I got into college early, mostly because I had written a Latin to English translator, a bit of AI, in 10th grade. I never finished high school. Sorry.

– Did you know Latin?

– I knew some Latin. I did know some Latin and it did like a semester’s worth of Latin or something like that. It wasn’t perfect, but it helped me to understand AI techniques and so forth. And I became disillusioned with the field of AI for the first time. I’ve become disillusioned more than once because I saw that it was just a bunch of hacks. It wasn’t really working that well. It wasn’t very deep. And I decided instead to study cognitive science. I’d taken a cognitive psychology class when I was 15 and decided to study that instead. So I always what I really wanted to study was cognitive science, which is interdisciplinary. It’s cognitive psychology, but it’s also linguistics, it’s philosophy, computer science. And I went to my college, which actually just closed, Hampshire College.

– I read about Hampshire College closing.

– It’s very sad that it closed. But I went there because when I applied or when I sent away for information, they sent me an excerpt of a textbook that was coming out in cognitive science. I was like, oh my god, these people know what cognitive science is. And the lead author became one of my mentors, Neil Stillings. And so I always wanted to take an interdisciplinary approach to things. And AI was a strand of that. But I kind of lost interest in AI for awhile ’cause I just didn’t think it was very good. And then in 2010, I guess, or so, Watson won in Jeopardy. And I was surprised by that. I’m not often surprised. I think I have a good track record of predicting what’s actually gonna work and not work. And I blew that one. And as a scientist, a neuroscientist too, I pay a lot of attention to when I get things wrong. And so I was like, why did I get this wrong?

– I mean, you thought it would lose?

– I thought it would lose. I thought, you know, to win in Jeopardy, you need to do language and inference and like, this is very different from a chess computer. And it won anyway. Later I got to know David Ferrucci, who led the project. It turns out that, I won’t say it was a trick, it was really a fantastic piece of engineering. But the reason that it worked was because 99% of the answers to Jeopardy questions are titles of Wikipedia pages. So you just had to kind of find the nearest Wikipedia page rather than really understand–

– Is that a known fact?

– It’s in one of his papers, in fact.

– And is that what, obviously there must be a reason for that. It’s the way that people who write Wikipedia pages are Jeopardy fans or it’s the reverse?

– The other way around. People who write Jeopardy questions, I mean, they just tend to pick out, you know, well-known concepts.

– Work backwards.

– And then they work backwards. So how can we ask about, you know, King Henry VII or whatever.

– Right. Wait, so people do well in Jeopardy, do they sort of make use of that fact?

– I don’t know actually. Actually, I did a podcast called Humans Versus Machines and we had Ferrucci who built it. And then we had Ken Jennings who lost to it, who was a wonderful guest on the podcast.

– It worked out well for him though in the end.

– But I didn’t actually ask him that question. I wish I had when I had that chance. That would’ve been a good question. Anyway, so that brought me back into the field of AI. And my dissertation had actually been partly on neural networks. It had been comparing neural networks that they’re the ancestors to today’s LLMs with children and how children learned language. Most of my career for a long time was really about that difference.

– And can you say a few words about it? I mean, what did you find there?

– Well, so I worked with Steven Pinker, who I know you know.

– Yeah, sure.

– And he actually wrote a book called Words and Rules. It was partly about stuff that I did.

– I see.

– My dissertation was a few different papers really put together. But the core of it was one on how children learn the past tense of English. And we looked at these things called overregulation errors. So kids would say things like breaked and goed and so forth instead of broke and went. And I did one of the first big data analyses. I wrote Unix shell scripts to go through this compilation of transcripts between parents and kids. And found out a few things. One of the things I found out was people made a theory about how kids did this all the time. And they didn’t actually do it all the time. People would notice the errors but not notice when kids were correct. So the kids are actually correct 96% of the time. But there were these theories about how they couldn’t tell the difference they were wrong. So I learned something valuable about looking at data, but also we compared it with the neural networks. And the neural networks. Someone had shown that neural networks could overgeneralize and said, oh, this is just like kids. But when we looked in the detail, it was really very different from kids, in particular, the neural networks. And this is very relevant to today. Were very much driven by similarity. And kids and adults, there were a whole range of studies that Pinker and I and some other people did together. The adults could generalize the regular verbs regardless of similarity. So Steve had great examples like I won’t quite get it verbatim, but something like in grim notoriety, Gorbachev or sorry, Yeltsin out Gorbacheved Gorbachev. And so there aren’t other verbs like Gorbachev, but we still know this abstract rule, add -ed. And that was problematic for these neural networks. So the thesis showed that even though you could sort of broadly approximate what kids were doing with these neural networks, they didn’t really get the notion of freely generalized into something else. And so I pursued that for a number of years, not really interested in building AI systems for myself, but trying to understand like what would it take cognitively, minimally to make an intelligent agent like a human being? And that led to some work that I think is really, really important in 1998 showing that these systems could not generalize abstractions far beyond where they were. So you could sort of think of roughly a distinction between extrapolation and interpolation. And that was close to what I had done in my graduate work. And I kind of purified it with neural networks, did some experiments on those neural networks. This is a rumor by the way, that I’ve never played with a neural network. It’s like totally false. And in fact, my most important work was these experiments with these neural networks showing that they couldn’t generalize what we nowadays call out of distribution.

– Does that mean like out of the training set?

– Yeah. So it’s slightly more complicated than that. So neural networks can memorize a training set which is relevant to a lot of stuff around copyright law. There are limits on that. We could go into the details, but roughly speaking, they can memorize their training set and then they can generalize. If you imagine like a cloud of points, and this is what the point of that paper was, they can generalize to that cloud of points reasonably well. And I’ll give you an example in a second. And if you go beyond that cloud of points, they start to break down. And that is still true today. And we can talk about some of the, you know, evidence that has confirmed this over and over for the last 30 years. The simple example then I had was just the identity function. F of X equals X, right? And the input is the same as the output. And I did it with binary numbers. And so I would feed in like all the odd numbers. So that’s leaving the right bit always a zero. And I wouldn’t give, or sorry, I wouldn’t do all the odd numbers, I would do many of the odd numbers. The system would get all the training examples right. It would get most of the odd numbers it hadn’t seen right before. But now I would do an even number. So that’s outside the cloud of points. And at that point, it would break down. It broke down over and over again. I gave a bunch of different examples like that and I said, hey guys, if you really want to build something with a cognitive strength of a human, you’re gonna have to deal with this in one way or another. And that led to a book called The Algebraic Mind in 2001. And the point of that book was that humans seem to be able to do a kind of mental algebra, maybe not consciously. I mean we can do calculus in the process of catching a fly ball. That doesn’t mean every person can articulate the calculus. But I said, underlyingly, we have these systems that can do these abstractions. And I showed that babies could do this in a paper I had in 1999 in Science. And I said, look, these kinds of neural networks are bad at this. We’re going to need a different approach. And the subtitle was Integrating Connectionism with Cognitive Science and connection was an older term for neural networks. And so the point was, if we want to build the neural networks, we need the symbols too. And when I came back into the field, just to take this back to the biographical part. And then we can talk about some of the implications. When I came back into the field, neural networks became popular again. They had disappeared between like 2000-2010. A few people like Jeff Hinton kept working on them, but most people stopped working on them. And then GPUs came along and suddenly you could train them on much more data. They started working better.

– Was that a key moment?

– That was absolutely a key moment. But it brought me back into the field ’cause I looked at these things and I’m like, I know what’s wrong with them. In fact, one key moment is John Markoff wrote a piece about deep learning in New York Times in November, I believe it was, of 2012, because of this GPU revolution. And I said in a New Yorker piece, literally the next day, I think it was literally the next day, hey, but there’s also problems here. I said, this stuff is great, it’s gonna be useful, but it’s not gonna be good at abstraction, formal reasoning and so forth. And so we’re still going to need other mechanisms. In a way, the tension between those two articles completely has defined not only this kind of academic discipline, but it’s literally defining the entire world now because the entire world is invested in scaling, which is really trying to take these old neural networks on the theory that it’s gonna get to AGI. And it’s not for the reasons that I pointed out in 2012.

– So why are smart people who are in the field? I mean, look, I’m sitting across from you. I know you know what you’re talking about. So I’ll sort of take in what you’re saying. I’ll compare it to what I’ve heard other people say and so forth. But there are folks in the field who are, have their fingers right in the details. Why does so many think that more data, more compute, build these huge centers that can undertake these tasks, that’s all that we need?

– I think there are a few different answers to that. So there’s different people that have different views. Some of them don’t actually, I think have the sophistication to think about distributions in the right way. And I think as a physicist, you’re sort of grasping it right away. And some people just don’t understand that. You have people like Sam Altman that just don’t really have technical chops, but are good salespeople. They don’t really understand the side of the technical detail. You have economic reasons. So it’s a great story to tell investors, right? You remember that venture capitalists get a cut of what they’re selling. So if you can say, or what they’re investing in, if you can say I have a trillion dollar investment and you have a plausible story and you get 2% of that, you’re doing really, really well. And this whole thing allowed this story of more and more and more. I just need more and it’ll work. And so everybody’s getting the 2% cut along the way is doing great. Even if it fails at the end, they get their 2%, right? They get, you know, a cut of if it actually works too. But if it doesn’t work, I mean 2% of a trillion dollars is a lot of money. So some people just kind of want it to work ’cause this story works. You have people like Altman who just really need more money because the dirty secret about this is almost all the companies are losing lots of money. And Altman’s been playing a kind of double or nothing game of you know, continuing to raise the valuation, raising more money. But it’s not clear he can do that forever. It’s not clear that people will go for a trillion dollar IPO given that the company’s in some ways facing hard times and so forth. But it has been a story that has allowed him to raise a lot of money. Then there are people that I think know the math but don’t really know the cognitive science and don’t really understand intelligence. You know, the old thing to a man with a hammer, everything is a nail. There’s a lot of that that’s gone on in machine learning. I think, you know, in the last 12 months or something, people are starting to realize that the scaling doesn’t really work, but they pushed scaling really, really hard. Some of it is what I call naive extrapolation. My favorite example of this was a tweet on Twitter somebody posted where they said my baby doubled his weight in his first month in life. So I extrapolate that he’s gonna be a trillion pounds by the time he goes to college. Of course the guy was joking. But he was making a point, which I think is really deep, which is most things that look exponential initially don’t continue that way. So, you know, if you double the number of mosquitoes, like it’s not that they’re gonna cover the planet eventually ’cause they’re predators, whatever. There’s lots of reasons why exponentials don’t materialize. But I’ve seen a lot of this, I mean the whole field is basically naive extrapolation. It’s like we fit, you know, these three points to a curve and so we sure we understand everything that’s gonna happen.

– Let me ask you though, so you did say that you got Watson wrong and that was sort of a moment. Did did you get November of 2022 right or wrong?

– I mean, I guess it depends on what, you know, what projection you’re making. So I’ll tell you something I got wrong and something I got right, something I got wrong is I just didn’t think it was gonna become that popular because it was immediately apparent something that I did get right, that it was gonna hallucinate, that it was gonna be unreliable. I wrote about that stuff then. I warned that it was gonna cause misinformation, which it does, et cetera. So like I saw all the drawbacks, but I didn’t see the enthusiasm. Some of the enthusiasm comes to something else you already indicated you wanna talk about, which is a lot of people anthropomorphize that. And I should have seen it. In fact, I wrote a book in 2019 about what some people, well that talked about what we called the gullibility gap in the book Rebooting AI. in which we should have called the ELIZA Effect. I think more people would remember it. And the ELIZA Effect is you can see a very dumb machine and think that it’s much smarter than it is. And I didn’t quite, when I looked at ChatGPT realize how much the ELIZA Effect was gonna change the world and they did a bunch of things to increase the ELIZA Effect. ELIZA was this psychiatrist in 1965 that, you know, was just dumb keyword matching and some people thought was real. So they did things like had ChatGPT type things out word by word. And it just felt human to some people.

– You’re saying rather than just giving you a block, you sort of feel it’s flowing as a conversation does.

– Which was just a gimmick, right? But it it was a gimmick.

– Yeah, it was a powerful one.

– I mean, it’s true that they do prediction word by word. Maybe they didn’t do it deliberately as a gimmick, but it certainly it came across that way as like it’s thinking or whatever. Certainly, they could have put the whole thing in a buffer and then displayed the buffer as many people do in computer systems all the time and they didn’t.

– I guess the more straight forward question is were you at all surprised that sufficient data and sufficient compute in this context of neural networks that you already knew you was limited, were you surprised at what it could do? Because again, I ask only because obviously I was, but let me just give you just a point to compare. I’ve spoken to a lot of folks in the field like yourself, and some of them answer in a way that I think is gonna be close to yours. They’re like, eh, you know, we kind of knew this, or others in the field have said to me, I was blown away by what Chat was able to do.

– I certainly wasn’t blown away. I mean, I had been writing about this particular technology, large language models since 2019 and all of the problems I had been pointing out all along, and it was always obvious that if you had a richer database, you’d do better. You know, there’s this old stuff that I learned from Chomsky, but I don’t know if it was due to him originally. It was certainly pretty sure it was in the paper with George Miller in 1965 showing about different higher order approximations. So if you try to approximate English with what we call bigrams, so just pairs of words, you get a decent approximation of English, you can start tell that it’s English. If you do it with three words, it starts to look pretty good, if you do it with seven words, so you take the probabilities of any string of seven words. It starts to look really good, right? I mean, this is a function of the law of large numbers and having more data, and if you did it with like a hundred words, it’s gonna be great. Like that’s no mystery. We’ve known that since at least 1965. And they may have gotten it from somewhere else, I can’t quite remember. So, you know, we’ve known that for decades and decades. It was obvious that as you increase the data, these models were gonna get better at what they do. And what they do is they build an approximation of how people use words. It was also obvious that that was not enough to get to artificial general intelligence. And so everything that’s followed to me is like, well of course it’s gonna be that way. You could say, you know, you can use it from more things than you thought. And then we should drill down into what you think the use cases are. The most impressive thing to me that I’ve seen so far is Claude code. And that actually is what I would call neurosymbolic. So it does actually have this symbolic harness around the pure LLM that’s inside of it.

– Right, right. But so it does raise the question, which we alluded to early on. Why are human beings who use these systems so ready to imagine that they are much like we are?

– Again, there’s multiple answers. So some of them are motivational, like some people want to be here in the moment of singularity and some of it. And so like they, you know.

– Just get overexcited.

– They just get overexcited about it. Some people have money at stake. You have, you know, great careers.

– The user, the user, like the average user.

– I think the average user, the thing is they don’t have training in cognitive science. Right? And so what they do, I think is a kind of small sample error. So they get a small sample of skilled performance and they attribute that to being a human. They don’t really have a context to think about mechanisms and algorithms and so forth. Here’s another example of the same thing, which is people get used to driverless cars and assume that they drive like people, but they don’t really, and so a famous thing that I saw, I’m trying to think of the guy’s name, there was a guy who worked at Waymo before was even called Waymo. And he showed this video once that really left in impression on me. In the early days of Waymo, they said to Google employees, you can try this out, but you must pay attention at all times because this thing is very experimental. And they had a camera, a dash cam, like facing inward. And this video is all these people like Google Engineers who, you know, you can’t get that job unless you’re a pretty smart person. Google engineers, they drive it for like 20 minutes. This is great, and then they would like reach for their briefcase in the back and not pay attention anymore. Because based on a small sample, they assumed as like a person. But the truth is it does some things like a person and others not, I mean, like Teslas for a while had a serious problem. I don’t know if they’ve solved it, of running into stopped emergency vehicles like 10 times or something like this that happened. They ran into fire trucks and police cars and so forth. No human driver is very likely to do that. Not actually like people, they’re solving problems that people would solve in different ways. Sometimes superior and sometimes not. But the average person doesn’t think, is it the same, is it different? Does this data represent the larger thing? So it turns out, for example, you can get ChatGPT to write, you know, a poem in the style of, I don’t know, a drunken sailor. And it’ll be great, right? And then people are like, wow, the only model they have in their head for how you could do that is you must be a human being. Turns out you can actually make a poem in the style of a drunken sailor pretty easily if you just have enough data and you have an algorithm like an LLM or other ways you could do it. But the average person doesn’t have that training to think about that. A related thing is benchmarks. So anybody who’s worked in the field of psychometrics, which is you know, hundreds of thousands of people or something like that, knows it’s very difficult to make a test actually measure what you want it to do. In psychology, there’s always like some way around your test and you know, you know, it’s very difficult to make valid tests, right? You can make a reliable test. So I take it on two different days, I get a similar score, but a valid test, this is a key distinction in psychology, reliability versus validity is very, very difficult. So people who know about things like psychometrics are not surprised to find out the benchmarks don’t actually tell us that much. But people who don’t know any better are like, look at the benchmarks, these things must be doing really well. And then it turns out like the companies are training towards the benchmarks. The benchmarks weren’t really that good in the first place. They only looked at a small piece. So like I think everybody in the field now realizes that the performance on the benchmarks does not necessarily capture what you are actually going to get at home. So if you have that training, you know that. But if you don’t, it looks good.

– But you know, on the other hand, you know, these systems have done really well on like the International Math Olympiad. You know, that is usually something that you think of as more than just predicting, you know, seven word or 10 word. I mean that feels like it needs something more.

– So there’s a couple things. One of the systems that did better on the International Math Olympiad actually was explicitly a neurosymbolic system. This was about two years ago from DeepMind, and so it had things like theorem provers that are symbolic systems integrated with some generative AI. The second is like OpenAI made a big announcement about having gotten gold. They never released that model as far as I know. And so, like, I mean maybe?

– Do you think that it wasn’t?

– I don’t know how it works. So what I do know, be careful about how well I know it. What I strongly suspect is that there is at least two things going on now with respect to International Math Olympiads that may not generalize to other things you wanna do. One is, I’m pretty sure they’re using theorem proving systems like lean right now. Both to generate data and maybe to assess solutions to validate the solutions. So you’re, again, you’re relying on classical AI techniques basically in order to prop up the neural network. And what’s limited about that is very few things in life are formally provable. And so the systems it would appear from reading the literature are doing the best on math, on coding and so forth where you can create as many test examples as you want, which was like Go and how DeepMind won on Go and you can validate things, right? That doesn’t mean it’s gonna be able to make good inferences about a war or a new product where they don’t have the data and so forth. And so it would appear that we are getting new systems that are actually in a new way, domain specific. So it used to be that all systems were domain specific by design. You build a chess computer, we do nothing else, right? And then the notion was with these chatbots, they’ll be able to do anything. But the reality is they can, if you build these harnesses and stuff around them, they can do these formal things pretty well, but they’re not necessarily generally intelligent.

– And so is everyone, not everyone. Many people speak of AGI as that’s the goal. And once we have that, then we are at the mountaintop. And who knows, we can talk about what life looks like post–

– Whether it’s a good idea or bad.

– But is AGI the right target necessarily? I mean when you think about the human brain and disabuse me of this idea, but every time I hear general intelligence, I think, well we don’t really have general intelligence. We have really good specific intelligences to do the kinds of things that allowed our forebearers to survive on the African Savannah. And so our brains evolved to be able to do those things really well and sometimes not so well. But that is general in a sense. But it’s also highly modular and specific.

– Yeah, it’s very complicated territory here. And you know, I’ve thought about a bunch of these things from the cognitive science perspective, first of all is definitely the case that we have some domain specific modular adaptations, for example, in choosing mates.

– Choosing?

– Mates. You know, there’s a lot of evolved psychology around that. We have that for recognizing moving objects. You know, et cetera. So a lot of our knowledge is modular and domain specific. We do have some, I would say that somewhat general. So, you know, the average human can learn a lot of different things if you give them context, motivation, et cetera. You know, ranging from playing games to dancing to cooking or whatever. And not all of those are evolved modules. Even reading is not an evolved module. It trades on some evolved modules, but there was no reading in the environment of adaptation. And not everybody learns to do it. Most people given the right circumstances can. Chess is not an evolved module. So we can learn new modules better than our current machines can do.

– View those as an overshoot of like the evolutionary capacities. Like when I think about physics, just as an example that I know well, I often think of it as well, you know, our forebearers needed to understand the basic mechanics of their environment. And as we have gotten to a place where survival is not the primary thing that drives what we do, we’ve been given the luxury of developing those modules to understand aspects of the environment that we don’t need to know to survive. So it’s kind of an overshoot. Do you see these others as overshoot too?

– I don’t know exactly what you mean by overshoot.

– Beyond the needs of survival but along the same trajectory of ideas.

– Yeah, okay, I’ll accept that. I think a lot of it is what you just described as an overshoot. It’s kind of extension of existing modules, right? Fodor wrote this classic book called Modularity of Mind that’s very much on these issues. Jerry Fodor, the great philosopher of mind who passed away maybe a decade ago. And the last chapter was what about what he called quinine and isotropic problems. I haven’t read it in a little while. But the basic idea was that there was both these modules, which most of his book was about, and also central processing if that’s the right term for it, that was more flexible and could sort of use any available information. So these modules are often restricted in their domain of application. They only use certain kinds of information. And then there’s other central process where you can kind of use any old information, you know, if it’s relevant at the time. And the human mind’s very good at not perfect, but very good at finding what’s relevant. So like, I don’t know if, if this table were to catch on fire, then we would just think about a whole different set of things and we would summon it quickly and we would be like, maybe we should stop this conversation and move and so we can jump to this other sets of topics and use a lot of information that we might know like about the materials it’s made of, et cetera. And this part of the mystery of human cognition is we don’t really know how that part works. We’re actually better, not perfect, on the modules where some of them we’ve been able to understand in detail like at least how we do what we call low level object recognition. Maybe not complex scene understanding, but at least some of these things we know something about how the brain does, the central cognition, we actually know almost nothing about how it really works. We have some vague metaphors and so forth. But we do know that people are pretty good at acquiring new skills, thinking about new problems. I’ll give you an example is chess, right? So you might call chess an overshoot, I would say in slightly different language that it draws, for example, on geometric reasoning abilities that you know are innate and probably kind of predictive things. Maybe a little theory of mind, et cetera. And you know, it takes hard work. It’s not trivial. But what’s interesting is LLMs don’t actually learn to play chess properly, even with a lot of data.

– In the sense of they make wrong illegal moves?

– You got it. Illegal moves, in the sense they make illegal moves. They also, like you remember the Atari 2600, there’s a chess cartridge in the 1970s video game system. One of their first console systems, can actually beat ChatGPT in chess. So it’s also not that skilled, right? The Atari game thing had a formal domain specific, you know, purpose built algorithm for chess. And if you just feed a lot of games into ChatGPT, right? It does see a lot of games ’cause that’s in the data they scraped from the web and it sees books like Bobby Fisher Teaches Chess is where I learned to play chess. And you might have too, since we’re somewhat similar in age, you know, that’s gonna be in the training database, just like every book. And so they get the explicit rules of chess but they don’t induce them. And here’s what I think is the hardest open problem. A lot of us have been talking for a while about what we call world models. So in chess your world model is just the board, the pieces, the history of the moves ’cause you have to keep track of a draw of the same position, repeats a few things, but that’s kind of it. And then you might have the opening books, you might have this knowledge about particular games. So you have this simple model of the world. And when my daughter learned chess, she had to induce that model, right? When the Atari game programmers did it, they just built it in. We know how to build in world models for all kinds of things. Not everything, but, but especially simple things like chess. The rules haven’t changed in 2000 years. There aren’t that many other rules. Someone’s gonna write in the comments. What about En Passant? Yeah, I know about En Passant, which is a little bit more recent. The rules mostly haven’t changed in 2000 years, right? And there very restricted. So we can build the world model there. But the question is, how do you have a system that can look at a new environment or watch a new movie and induce a world model? So for example, my kids were recently watching and reading Harry Potter and they developed a very complex understanding of that world that was separate from our world. So they knew, for example, that in that world you can fly on a broomstick and you can’t in this world. And then they could make inferences about what would be possible given that you could fly on a broomstick and maybe what kinds of broomsticks and so forth. We don’t really know how to get AI to do that. To be exposed to a new movie, a new book, a new game. Or like every time, you know, a 13-year-old picks up a new video game, they build a model of that world, right? They play Grand Theft Auto. There’s a world there and there’s certain things you can do there and certain things you can’t do in the real world, et cetera. We need to have AI build world models from the data that it observes. And we don’t really know how to do that. Chess should be easy and still the LLMs fail at that.

– Well how do I square that with the following experience? So I’ve mentioned this little story before. It’s not unusual or particularly new, but I was working on a physics project, the details of which don’t really matter so much for the question, but it had to do with universes different from ours, a different world if you will. Not a Harry Potter world, but this is a world in which there are some extra dimensions, a world in which some of those extra dimensions may circle back on themselves, which may or may not be true of our world. But certainly we don’t see that. This is a world in which the extra dimensions could have some weird mathematical properties where there’s no definition of left versus right. A sort of non orientable quality of these objects, exactly. And so working with some colleagues, we were able to get some, you know, modestly interesting results for what would happen in that world and we wrote a paper. Before we put it out, I wondered if I treated Chat like a graduate student, how long would it take it to get to that place? Not giving, you know, when we have graduates, we don’t tell them the answer. We give them a little bit of guidance and try to shape. And so within like a half hour I was able to get Chat pretty much to the answer. So it had to understand all sorts of fairly esoteric mathematics that even most physicists never encounter. It had to kind of imagine, I’m using very anthropomorphic words, which I probably shouldn’t, but it had to kind of put itself in this weird other world and deduce qualities like flying on a broomstick. But here there are mathematical qualities that follow from, you know, the set of rules in that world. When I got to the end of that, again, I could be over, you know, attributing, but I felt like I now have like my best graduate student right here in this box.

– So I mean, results on that are mixed. Certainly some people have reported experiences like that. Some people still, you know, report weird hallucinations. You’re there in the loop, which I think is, you know, fundamentally important. There are interesting new results basically showing that people get better results out of these things if they have the sophistication to guide them. And if they don’t and they just ask for an answer, they don’t learn anything, they don’t get as good results and so forth. So I mean, I think it’s right to think of them as tools. It’s interesting to ask like what mathematics and assumptions they can absorb and how well they can stick to that. I have not seen a lot of good results about actual, you know, new discovery autonomously. I have heard more and more people in physics and fields like that saying I was able to use it as a kind of apprentice to do these things. I question how well it’s actually absorbing all of the, you know, the assumptions of the world. But you know, it’s an empirical question.

– Right, do you have any examples or have you read or heard or whatever of examples where you would say this was a creative output? Would you ever use the word creativity? And again, I ask this ’cause I’ve had this conversation with a variety of people, some in the arts, some in AI, you know, some of the arts are like, yeah, this is creative. Others who I’ve spoken to are like, no, it’s a regurgitation machine. You know, it’s simply an algorithm. You know, there’s nothing creative in that at all. And then what strikes me is I wonder maybe we’re over attributing to us creative powers that maybe feel mysterious, but maybe we are doing the analog of just predicting the next word. And so maybe we’re just aggrandizing creativity.

– It’s a complex thing and it’s a question of, you know, partly a definitional question of how you define creativity. I think that in general, what these systems tend to do other things being equal, is to produce something kind of in the training space to use the terms we were describing before and tend not to go outside the box, tend not to do things that are truly original, you know, that can still feel quote, creative, right?

– Isn’t that what we do?

– I think it’s what many people do most of the time. I think that the most creative people do go outside the box, even if that box itself, or even if that thing itself is kind of recombination, it’s still like nobody thought to do it that way.

– But that recombination isn’t that what these system, like for instance, I give you an example, which is a bad one because people get a little irritated when I say this, but I have the deepest respect for Albert Einstein. I shouldn’t use the word, but, I need another connective word here.

– You dug yourself in the corner there.

– I gotta stop there.

– And still.

– And still when I look at the general, thank you, look at that, predicting the next word. And yet when I look at the general theory of relativity, I see it as him taking in a way that nobody who ever imagined you should do, mathematics from the 1800s on differential geometry, gravitational physics that comes from Newton ultimately. But he was able to blend these two together with his understanding of special relativity that he himself. But it was putting together things in a novel way. There isn’t sort of a thing.

– But that’s the thing, right? It’s the novel part. It’s the novel way it, it’s looking at these things that nobody thought to put together. I think many cases of creativity are like that. I’ll give you a couple examples. Dylan started thinking about Bertolt Brecht’s poetry and that led to his kind of lyrical explosion. You could have done that before, but nobody had thought do it.

– Exactly.

– For my music book, I interviewed Tom Morello and he learned guitar a little bit later than average person. He didn’t really get serious till he was like 15. He got an honors degree in college while playing eight hours a day trying to catch up with his peers. And still he became very technically proficient, but he didn’t really have a signature style. He was doing like studio work. And he says it was when he started listening to DJs doing their little tricks and he said, how can I do something like that on guitar that he found his voice, that was the creative part of Morello’s career and everything else is built around that. And so somebody else could have done that. You can certainly call it recombination of, you know, Djing, music and you know, once you understand how he did it, you might be able to do it too. But the creative part that really gave him his career was saying, hey, let’s take these things that nobody has thought about putting together. That’s what I mostly don’t see from LLMs.

– I’m saying is recombination of existing is not necessarily a bad thing. Look what it can yield.

– It’s not, I think recombination is cool and I think, you know, like I write for a living sort of, and you know, it’s all recombination in some sense.

– Hey, Shakespeare used 12,000 words.

– Well he’s I think a grand total of 12,000 of the whatever, few hundred thousand that were available.

– But the interest comes in, you know, thinking about recombining things in new ways and LLMs tend not to do that. You can even tell them that, right. You can tell them recombine these things that have never been done and they might do it. But the impetus to do that typically comes from a person, I’ve not seen.

– If one’s talking about genius level creativity of say an Albert Einstein or a Dylan, right? Maybe that is beyond reach, maybe it will always be.

– Reach or very creativity depending on how you define it. You know, you could make the argument there, right?

– Which argument though?

– You could make the argument that an LLM is creative in the way that an ordinary non genius person is. It’s hard to make the argument that they kind of reach the outer boundaries.

– Good, yeah. So I think that’s a reasonable way of describing it. And it feels to me that that’s really impressive, if geniuses are like the one in a million or call it the 100,000, but probably the one in a million, if these artificial systems can replicate the level of creativity of the 99.9 whatever percent. That’s amazing, isn’t it?

– I mean, in some way it is amazing. Like ’cause I know how the trick is done. Like, I’m not as impressed with it, but I can understand why somebody would be impressed with it. And there’s certainly utility in it, right? Like if you want to do marketing pitches and you don’t have the budget to hire the best people, you can use this and you’ll get, you know, at least some credible results. Probably you as a human will sort of sort out the wheat from the chaff and that’s using, you know, some of your own creativity. But it will come up with some possibilities. I’ve talked to Bob Mankoff a lot about this. He used to, as you may know, he used to be the cartoon editor at The New Yorker. And he’s very interested in LLMs and humor and you know, if he gives it a prompt, it’ll come back with 20 possible. Like he’s interested in the caption contest.

– Are they any good?

– Which he invented.

– For instance, can it win a caption contest?

– You know, if he does the screening, then yes. Like he has it do 20 examples and I’m gonna get an angry email from Bob and you know, two of them might actually be pretty good and he can pick them out. I mean he has, you know, very good taste on this.

– He doesn’t use them, does he? Well anyway, we’ll put that to aside. Yes, exactly. But so you mentioned that the state-of-the-art systems have begun to use other techniques. Can you drill down a little bit on that? For instance, you know, in some of these systems, I played around a little bit as many of us have, and you can watch it, it kind of in, in ghosted script tells you what it’s doing. Is that real? Because I sit there, I’m like, oh, that’s a good move. Yep, yep. That’s what I, oh, smart graduate student would definitely do that integral. And so is it telling me what I want to hear?

– Physicist Brian Greene, ask Jerry Marcus what’s real.

– Yes.

– I mean, it depends what you mean is real. They’ve built these harnesses that will call LLMs to do certain things. And if the manufacturers of these things wish to, they can give you some trace to use computer science term of what it’s actually doing. You know, how straight they are with this, what they suppress and so forth. Different companies are gonna vary on that. When they call it reasoning, I would not use that word myself. And we could talk about that. It is true that given the harnesses that they will like kind of farm off sub problems, which is a classic symbolic technique by the way. And one of the most classic symbolic techniques is means ends analysis where you break things into pieces. And they’re now using that, they use the LLM to do the individual pieces, but they have some, you know.

– So it’s calling Python

– It’s calling Python and so forth. And if it says calling Python, it probably is, sometimes if it doesn’t say it’s calling Python, it may actually be doing so. So I played around a bunch with these systems maybe 18 months ago. A friend had an example that he texted me, which was something like draw a picture with exactly 17 circles in it. And sometimes it would make explicit that it was calling Python and sometimes it wasn’t. And then like six months later they were not making it explicit, but you could kind of tell when it wasn’t. So, you know, whether the reflection you could call, well I guess I’ll call it the trace, you know, whether the trace is vertical or not and complete or not, is kind of up to the people who build the systems for how much they wanna disclose. There are reasons why they might not wanna disclose all of it because these companies, not only have they stolen every book that’s ever been written, but they steal from each other. And so they don’t necessarily want other people to know the so-called chains of reasoning that they use in their system.

– Is it a marketing ploy? I mean like behind the scenes, they don’t need to provide you that.

– I think it was a marketing ploy and they’ve done it somewhat less because they don’t want people to rip off the steps that they’re following.

– Right, right.

– So anytime a company like OpenAI exposes what it’s doing underneath, other people are gonna try to copy that. And so at first it was a marketplace like, wow, this is cool. And it built on that anthropomorphic effect that we were talking about earlier. But then I don’t know the current state of play exactly, but I think people have drawn back from being complete because they don’t wanna be ripped off.

– Right, now I get it. Now it certainly gives you the impression by design that the system is reasoning. But you are gonna?

– I’m not happy with that. It’s a common term in the literature, but there are lots of studies that show that there’s a lack of abstraction in that. There was a great tweet a year ago, Josh Wolfe, who’s a venture capitalist who hangs out not very far from where we’re sitting right now, wrote about this paper that Apple had called the Illusion of Thinking or Illusion of Reasoning or something like that in these reasoning models. And so they kind of tore apart some of these things. And Josh Wolfe, you see why I like this tweet said that Apple had Gary Marcus-ed LLM reasonability.

– This is now a past tense verb.

– He used my name. Exactly, exactly. You got it. We call this a denominal verb. So he’s taken my name, turned into a verb saying that Apple had basically dissected hype, which is, you know, something I tend to do. And what they did in that paper is they took simple problems like the Tower of Hanoi. So you have three eggs and rings on top. And they showed that a pure LLM, not one that’s writing code, but a pure LLM maybe with this reasoning stuff, but not writing code, could play let’s say Tower of Hanoi with seven pegs and then it would completely break down at eight pegs. Whereas a human child who learns to do it with seven can easily do it with eight. They learn a recursive algorithm and they can generalize it. And so the point was what looks like reasoning is really a lot of memorization and so forth. There’s a level of abstraction that a physicist can do that a logician can do, et cetera, that these systems can’t do. It’s again, definitional, what do you want to count as reasoning? But it’s clear that it’s not very abstract. And I think a lot of the reasoning that we care about really is pretty abstract, at least if done well.

– And so what would it take to get to the place where you’d say, yeah, it’s actually reasoning. Do we need to understand the human brain better and emulate it?

– Not necessarily. So my view is that the human brain is pretty bad at a bunch of things, pretty bad at a bunch of things. I wrote a book called Kluge, you might know the old engineers word for a clumsy solution to problem duct tape, rubber bands. I think there’s lots of limits to humans. I don’t think we wanna replicate everything. If you wanna do math, don’t replicate a person who can’t carry the one, right? So I don’t think it’s a necessary thing for intelligence to be modeled on humans or not closely modeled, but there are things humans are still better at. In particular, these kind of learning about a new world, learning a new skill, and kind of flexible reasoning. People are still better. And so we might want to learn from how humans do that. We might not, I mean, it might turn out that there’s nothing we can salvage from human reasoning that everything can be done better. That’s conceivable to me. Right now, there’s certainly still at least a few things that humans are better at than even the best machines.

– Well, you mentioned before, just to be concrete, the Jerry photo and the reference to coin, sort of different kinds of reasoning, if I can use that. Daniel Kahneman had his, you know, thinking fast, system one, system two. Is that an essential thing? I mean, do the systems invoke that at all now? Is that part of what people are doing?

– So the way I think about it, first of all, I think that that’s an oversimplified distinction. And Danny, if were he still here would not be offended by that. I mean, he also argued that it’s simplified, but it’s a nice way to get started on the problem. And so I’ll adopt it, you know, temporarily, I would say that neural networks as we know them, though there might be other ways to build them, are very good at system one thinking and very poor at system two thinking. So remember, system one is fast, automatic, reflexive, statistical, and system two is more deliberative, more reasoning, more symbolic, and these neural networks are not very good at it. This is another way of making the argument that I often make, that we need neurosymbolic AI, we need the system two is the symbolic stuff, and system one is the neural network stuff. We need to have a marriage between those two.

– Can you tell more about this neurosymbolic stuff? So I know little examples of it, but how would you frame that?

– So I mean, neurosymbolic is an approach. It’s not just one thing. Just like programming is an approach, right? With programming, you can build a video game, you can build a database. You can build a browser, but you know, if you think about classical programming, you have things like variables and operations you can do over those variables. You have conditioning, you have typically, you have iteration and loops and so forth, right? These are classical symbolic things. And the idea that became popular, especially in the 2015 era, was that you didn’t need any of that stuff. You could just do it all with a large neural network. And by the end of that decade, people thought with a large language model in particular, and the notion of neurosymbolic AI is, no, you actually need a bunch, you know, the neural networks are good at pattern recognition, you should use ’em, you shouldn’t throw ’em away. They’re genuinely useful for pattern recognition or at least many kinds of pattern recognition. But sometimes you need this other stuff. You need rules, for example, to do planning. And you need, you know, you probably need iteration to go through loops to compare results and so forth. And like why tie your hands behind your back and do everything with this one architecture that’s popular and imagine that if you poured out data, it’s gonna do everything? Like if you look at the human as a model, humans don’t just do that statistical stuff, they do it pretty well. So if I say the word inextricably, you know that that could be followed by linked or bound, but it’s probably not gonna be followed by too many other things. Like, you know, the kind of statistics of that. But you also do a bunch of abstract reasoning, especially as a physicist. But you know, everybody does to some extent. And so the notion of neurosymbolic AI, which is the through line that I pushed throughout my career, is that you need to have someone both, that they have complimentary strengths. So let’s talk about that for a second. The neural networks are good at kind of absorbing large amounts of data, but they’re not good at abstract reasoning. The symbolic stuff has never been very good at learning, but is very good at abstraction. Oh, and sorry, the neural networks are pretty bad at sticking to facts. So that’s the hallucination stuff, which I wrote about in this 2001 book and the symbolic stuff, they don’t make stuff up that way. My favorite example of this is, do you know the actor Harry Shearer? He was in Spinal Tap, the bass player? So he’s a friend of mine. And I had been doing this running riff about Henrietta the Chicken. Somebody said that I owned a, somebody used ChatGPT and it claimed that I owned a pet chicken named Henrietta, which I don’t, I don’t have a pet chicken, I wouldn’t name it Henrietta. So one day Harry Shearer sends me an email with a subject header, no Henrietta, but, and what it is, is a biography of him written by ChatGPT, I think it was ChatGPT, which says that he’s a British entertainer, comedian, voiceover actor. And in reality, he was born in LA. He’s an American who does these things. But there are many, the point is there are many British–

– So statistically speaking, it’s not a bad guess.

– So statistically speaking, it’s not a bad guess. And that’s what LLMs do is they make not bad guesses statistically speaking, but you could just go to Wikipedia and see where he’s born. And because he’s a pretty well known actor, he could also go to IMDB and Rotten Tomatoes in a million places. He’s done interviews, et cetera. So you should be able to stick to the facts or at least say interesting. My statistical mind is telling me that he might be British, but I have 3000 other examples in my database that he’s American. You might want to consider that. But LLMs don’t do that. They don’t give you a confidence measure.

– But along those lines, just to, again, an example, you know, early on in whatever it was, 2022 or something, fiddling around like everybody else was at that time. And so I asked Chat who I was married to, and it said I was married to Renee Fleming, the opera singer. We had actually done an event at one point together, I presume that’s where the lane came from. Then I asked if we had any children and it listed her kids, not mine, you know? But that doesn’t happen anymore.

– It still does, it still does.

– But my question is, over time, presumably we’ll get to a place where the hallucinations are so far out and would only arise in highly engineered prompts. Can’t we get to a place where we’ve just improved it, improved it, improved it? And we don’t, for all intents and purposes, have to worry about that?

– That’s just not happening. So I mean this goes back to the naive extrapolation thing. You assume that if there’s progress every week, that you’re eventually gonna solve the problem rather than asymptote, and that’s a bad assumption. Empirically, I just saw data on Claude. They said it was 92% honest. I mean, first of all, it’s not really honest or not honest, right? That’s again, anthropomorphic.

– Of course.

– And second of all, that means it’s still an 8% error rate, which for some problems is fine. And it actually leads to the problem that I noticed with the overregulation. But in reverse, when people notice one kind of data and not the others, they notice it. Well, it depends on their personality actually here. So people like me notice the editors and people who want to believe in the system notice the correct thing. Very few people do the math of looking at the numerator and the denominator, but they did. And they said it was, you know, 92% correct. And this is, you know, billions of dollars.

– But where was it like two years ago?

– I don’t know that number offhand. But you’re still seeing this residue. And the residue comes because of, you yourself said it a few minutes ago, because they’re statistical approximators making good guesses. So what happens is when you add more data, if they can mimic that data, they’re good, but sometimes they don’t mimic that data. Sometimes stuff gets inflated, sometimes they have to make inferences. If they have to make inferences, it doesn’t work out so well. And so you continue to see these errors.

– So if you look at the human brain, we got to a place somehow where we can do, to use the simplified language, the system one, system two, however, you know, fast and slow. When I hear some people talk about these artificial intelligence systems, they describe as sort of the next critical juncture, which maybe we’re at already, when these systems can self-improve so that they’re kind of undergoing an evolution by natural selection that they’re creating. Now the timescale for biological evolution, extremely large. Is it the case perhaps that with the small timescale for digital evolution, the systems themselves can get to a place where they can do system one and system two?

– I guess there are a few different variables there. So one is, what do you mean by the system? So if you mean pure LLMs?

– No, I think.

– No. If you mean, could some AI system do some interesting self evolution improvement? Surely yes. I mean, we don’t maybe have the right kinds of systems. There’ve actually been a lot of experiments in evolving AI systems and none of them have gone very far. And I have a theory about why, which is that The interesting stuff that’s happened in our evolution, some of the interesting stuff, the stuff that we tend to think about as self-absorbed humans, happened pretty recently when there was already, I think of it as like a subroutine library that was vast, right? So if you think about how the brain unfolds, my book, The Birth of the Mind is about that, you know, there was an enormous amount of evolution before primates or before humans and so forth. And essentially the genome is like a self assembling computer where each gene, essentially, I’m oversimplifying, but each gene can be thought of as kind of like a subroutine that you call. Right? And so, you know, people talk about like we’re whatever, 2% different from chimps or something like that. If you have this huge subroutine library, you can do a lot once you have it, if you don’t, if you’re just a bacteria and you change two genes, bacterium, you’re not gonna get anything all that exciting out of it, right? It depends what you have in place for how you can leverage those routines. Or another way of thinking about it is like people can do all kinds of crazy stuff with Python now because there are these huge libraries out there. And so, you know, you can, I don’t know if you wanna make a weather app, somebody’s already got the weather data has scraped it from the web for you, whatever you, you don’t have to write that from scratch anymore, right? And the genome is like that. So we have, you know, routines for building whole hemispheres and stuff like that, oversimplifying a little bit, but you get the idea. And when people build these AI evolution systems, they tend to do things like evolve one of the quote neurons. They’re not really even neurons, but whatever, the nodes in the neural network, you know, with this weight or something like that, they’re not really evolving at the algorithmic level or at the high algorithmic level. They’re really kind of like at the low level of the individual neurons. And so that doesn’t really go anywhere in a short amount of time. You know, like the last seven million years of our evolution or whatever the number is, as compared to like the billions of years before that, you know, there’s more acceleration because there’s more of a subroutine library in the genome. And so I think if somebody could figure out what the high level subroutines are and do evolution on that, we might actually get a lot of progress. But it just hasn’t been what people have done. And that’s partly because of a bias in the field. The field has very much been on the empiricist side, which like we learn everything from data and opposed to the nativist side where you start with a lot, there’s a really interesting asterisk on that I’ll come back to in a second. But the bias in the field has been to really like learn everything from data, not build anything in. And I think what we should be doing is building a lot in thinking thoughtfully and then evolving on top of that. And when we do that, we might actually get interesting results. The weird thing is, these people who used to like attack me for being a nativist now build these system prompts that have all kinds of crazy stuff built into the system that it doesn’t learn. Like you remember the river crossing problems, you have like a wolf and a guy. These systems are so bad at variations on that, that at one point Anthropic was innately building in things about river crossing problems, right? It still wasn’t working that well. And so like they’ve done this very weird thing instead of building principled nativism. And what would principled nativism be? You’ve maybe met Liz Spelky over the years.

– No, I have not.

– Harvard developmental psychologist, you should have her on your show. She has done all of these cognitive psychology experiments with little kids and shown, for example, that they seem to know that objects persist in time and space.

– Object permanence.

– Object permanence and so forth much earlier than people recognize. And she has argued that there’s what she calls I think core cognition or something like that. And then, you know, there’s a core and there’s a periphery that you learn. But this core, she has argued I think quite well is innate. And I think we should be like, what is the core that we need to build into our systems? Not learning it from the data. Such that when we get to the data, you can learn more sophisticated things. So for example, she’s talked about and argued that humans have built in the notions, not explicitly, but of object sets, places, events, stuff like that. If you have those, then you can learn what are the objects, what are the sets, what are the places? But if you just have pixels like Sora did, you don’t really assemble that into representation of the world. Today I wrote a piece, maybe that dates this too much, but today I wrote a piece, there’s a new image generation system that OpenAI happens to put out. And you can ask of things like draw a picture of a bicycle and if you move even a little bit from what’s there on the internet already, you find that it does weird things. So I asked draw a picture of a tall tandem bicycle with a rear, how do you say it? Panniers and rear rack in the back, right? And the image was pretty good, not perfect. But then there were weird things like maybe one of them I think had a output, had a derailer in a tire. And it’s like it doesn’t actually understand how these things function. What is the causal relationship? Even though it’s been trained on ungodly amounts of data, far more than a human being would ever be exposed to. Like there’s still something missing in his functional understanding the world, despite having all that data, which is why we should be listening to Liz Spelke or Elizabeth Spelke, as she’s formerly known, what would be the core such that when you get exposed to all this data, you can extrapolate the right things around it?

– And so does this view make you less nervous about the future of humanity in the face of AI?

– It doesn’t, because I think the problem is not so much intelligence is power. So you can have an unintelligent person, very powerful.

– Really?

– Even a person with possible neural degeneration with enormous power and just hypothetically speaking and that person could do great harm. And the analog here is we have these systems now that actually make lots of kinds of mistakes, but we are giving them a lot of power in the world. We’re using them to plan military targets. We’re using them to decide who gets jobs and all of these things throughout society, plus bad actors can exploit them. So for example, large language models, because they don’t really understand facts, can easily, but they are good at style transfer, can easily be induced into writing fake news. That sounds plausible, right? They don’t have the wherewithal to say, I’m not gonna write a story saying such and such. It’s not true. They don’t understand what true is, right? So bad actors can exploit them and on their own they could just, you know, do pretty wacky stuff. And that creates danger. The danger that I am most worried about right now, I think is actually accidental nuclear war, which I think could come in two different ways from current systems that are not that smart. One is that you could get accidental nuclear war from mist targeting and what ensues. So for example, we all know that the US appears to have bombed a school in Iran. And that raises the temperature, in this case, maybe it didn’t raise it too much ’cause it was already high enough. There’s not much delta. But you can imagine incidents like that that get out of hand, right? And that could lead to war and even accidental nuclear war. And then you have the misinformation potential. And so, you know, misinformation has been really rampant in the last few months, including in the current military conflicts around the world. And so you can imagine a route there where, for example, we get mad at the Russians because we think that they turned off our power grid, but really it’s someone else’s doing this or that. And then, you know, we have a skirmish and one thing leads to another. So I mean, that’s just one set of scenarios. But there are a lot of scenarios where AI, I won’t call it stupid, but not reliable AI like we have now, can actually lead to things like accidental conflict.

– So those are existential and obviously deeply important to the future of the species. But what about on the more day-to-day level? I mean, there’s always been a lot of fear when any new technology comes along that there won’t be a place for human beings to still do useful, interesting, important work. And I think this is the first time when people really started to take that idea seriously. So do you think with the way things are going, obviously if there’s a radical change and the world models are included, and we may be in a completely different domain, but where we are right now, will AI, do you see them displacing a great deal of the workforce?

– Not in the short term, I mean, I sort of have good news and bad news and I’ll give them both and then go through them. So the good news is, in the short term, I don’t think that much employment is actually gonna be replaced. And the simplest way to think about it is a distinction I guess I got originally from Erik Brynjolfsson, which is between tasks and jobs, AI is pretty good at many, but not all tasks. It’s not very good at many complete jobs. And so what happens as a function of that is it makes people who do jobs do those jobs better or more, you know, more efficiently. Sometimes it makes them work harder or whatever. Has different permutations, it changes the nature of those jobs. But there are very few things that AI does reliably enough, especially generative AI, which is most of what we’ve been talking about, such that it can actually replace a person. There may be some, but most of the jobs so far that I think have been replaced are really just CEOs looking to do job cutting. And they’re using it as a shield. There aren’t that many studies showing that full jobs can be replaced by AI. You know, Jeff Hinton said in 2016, we might as well stop training radiologists. There’s still been no radiologists replaced as far as I know.

– Is that just a transitory moment?

– It is transitory. I mean there’s a question of duration. Someday radiologists really will be replaced.

– But someday is not like?

– It’s not tomorrow. Similarly, driverless cars, we’ve been hearing about them since 2012 when Sergei Brin said they would be ubiquitous in 2017, and still they’re kind of limited, we call them geofenced to particular cities where they’ve been really extensively mapped and so forth. So I don’t think there are any driverless cars here in New York.

– I think it’s coming here. Waymo is coming here.

– You know, they’re gonna still have a driver in the front and it’s gonna take awhile. They’re gonna have to do incredibly detailed mapping to really make it work. And you know, it’s not clear that they’ll be able to make it work on snowy days and stuff like that. So we’re still gonna have drivers for a while. Like there was a lot of people I think initially including me, who thought, hey, we’re gonna lose millions of jobs at once. And I think that will happen, right? So the thing about AI and automation is once you figure out the algorithm, you can make copies of it for free. And so at some point people will actually figure out a way of doing driverless cars that is fully generalizable, doesn’t depend on these detailed maps. Works more like a human driver who you could more or less drop anywhere where there’s engineered roads and that’s gonna be massive when it happens. It’s not gonna happen next week. Probably not gonna happen this decade. I mean, almost certainly not going to happen this decade, but in the next 50 years it might well happen.

– But as someone who thinks about AI and AI policy, its impact on society. Have you played out or have you seen people play out? Sure. So we’ll have less of those jobs because AI can do this. But as with all technologies, it opens up this, that or the other thing.

– I’ve heard that argument many times. I’m not impressed with it. I don’t know the right answer either. So like, I feel like a lot of things I gave you I’m very confident of. And this one I’m less. So, you know, what I would say is like we had this job a couple years ago, there were all these news stories about prompt engineer, how they were getting $300,000 with an English major, right? But that disappeared, like it lasted for like six months. I think in the end, the net effect is gonna be that employment is crushed and that we’re gonna have to move to a different model of humanity where you find meaning not through your work, but through your art, not through your paid employment. I think that that will happen eventually. I don’t know whether it’ll happen gracefully or through riots and people starting to take shots at CEOs and stuff like that. I don’t know how smoothly we’re gonna get there, but the long term is, you know, brains are just computers at some level. People get mad at me when I say that, but I think that there’s good reason to believe it. We are going to be able to replicate its function at some point and we’re really just arguing about which decade that’s gonna happen. I think it’s gonna happen this century? Could be that it’s in the next century. Certainly like 500 years from now, AI is gonna be able to do the vast majority of things that people can do.

– So just to be clear then, the trajectory of our conversation has been, you’re deeply skeptical about LLMs and large datasets scaling to the place that you just described, but you’re not at all deeply skeptic, you’re quite optimistic, if that’s the right word.

– Exactly.

– That these systems in one way, shape or form will be able to do what the human brain does. When we get there, do you also envision that these systems, to go back to the second or third of the questions I started off with, do you envision that these systems will be conscious at that point?

– I hope not. And I don’t think so. But I wouldn’t rule it out, I guess. So I think we should not mess around with building conscious computers. Unfortunately, some people are gonna try to do it. I don’t think LLMs are at all conscious. I think that’s just a mistake. Like when an an LLM says that it’s anxious, it’s just mimicking a person that said that it’s anxious. There’s no emotional content to it. It’s like saying tht counterfeit money is money. I mean it’s just, you know, there’s something fundamentally missing.

– Which is my view too, again, not in the field. And that’s why I’ve been so intrigued maybe is the right word, that there’ve been a small handful of people that have had these conversations with who are in the field and say, I would not rule out that these systems have a degree of self-awareness.

– Yeah, I think that’s crazy.

– It feels crazy.

– I could even imagine systems that can analyze their own code and so forth. And still I wouldn’t wanna ascribe sentience at least to some of them. Maybe some subset of them might be, that might be a necessary step towards sentience, but maybe not sufficient. I don’t think we should want to have sentient systems. You know, AI is hard enough to control in ways that are human compatible to use Stuart Russell’s term, hard enough to control now without sentience, right? Like the LLMs don’t really follow instructions. And if we had sentient machines, they might decide they don’t like how humans have treated them and like, I don’t think we want to go there.

– Now, there’s two question there. One, how would you ever know if they really are, a deep question.

– It’s deep and very hard question.

– Hard question. But the secondary question is, it could be the case that sentience may be an absolute ironclad quality of the kind of intelligence that we or this AI system in the future might have. You might not have a choice, right? It just may be what happens when systems are able to process things the way that we’re familiar with in our own heads.

– Because the definitions are so lacking here, I can’t say no to that, but I can for example, say there’s no reason a driverless car system has to be sentient.

– I would agree with that. But a driverless car system is very specific. Yeah.

– You could I think reasonably ask the question though, like at what level of generality can you achieve without sentience?

– Exactly.

– I think is the question you’re asking. And I think it’s a good question and I don’t think we know the answer.

– Right, right. Do you think that these systems may actually be the way that we gain insight into what our own sentience is? Because in principle, you can open up the hood, you can, you know, tinker, you can see what’s going in a way that you can’t really ethically do with the brain.

– I always think of the end of Wittgenstein’s Tractatus where he says, whereof we cannot know. We must remain silent. I just think like these questions are interesting to bat around a little bit, but we just don’t have a grounding, like you said, you use slightly different words, but I’ll use my own. We don’t have a consciousness meter. We don’t have some measurement we can take, you know, is this glass conscious? That sounds like a ridiculous idea, but there’s a theory called panpsychism. And we don’t have some independent validation there. And so I always get nervous, like I’m pretty confident saying LLMs are just imitating stuff. There’s nothing more going on there. But I could imagine some boundary cases, it might be hard. I mean, already we have hard boundary cases. Are dogs conscious? You know, most people would say yes, some people might not, depending on the definition they draw.

– But if you get down to an amoeba?

– And if you get down to an amoeba, like, you know, most people have the intuition that dogs are in fact conscious, but some people don’t. And amoeba like most of us think they probably aren’t. But like we don’t–

– We don’t know.

– We don’t know how to draw these lines.

– So just two other quick questions if you indulge me. So Nick Bostrom, who I spoke to not too long ago, wrote a book called I think Deep Utopia or something like that, where he envisioned how the world might be in the good scenario. We all know the dystopian scenario. These things don’t like us, they wipe us out or they cause a nuclear war.

– Or just bad actors.

– Exactly. Or use them in that way, absolutely. But if you go the other direction, the opposite direction, what would the utopia emerging from this look like, do you think?

– I think it would be Peter Diamandis’ view of abundance. You know, I don’t agree with everything that he says, but I do think that’s something that we could hope for, where basically the economics of the world are such that we can generate anything we want. That doesn’t mean there’s not gonna still be money. People are still gonna fight over beachfront real estate and stuff like that.

– Because there’s a limited resource of any sort.

– Certain things like beachfront real estate are by definition essentially limited.

– Create them in a holographic deck system, you know, whatever.

– Yeah, exactly. So there’s always gonna be some of that. But you can imagine a world where basically food is infinite.

– And energy and fusion.

– And energy is, you know, essentially for practical purposes, infinite. And so cost of production for things gets cheaper. And there’s a political side to it, right? Which is how we distribute such wealth. I mean the worst case in my mind is like three people take that infinite production, take all the beachfront real estate and screw the rest of us. And that’s not at all out of the question, given the way I see things evolving right now. But there is a scenario where we basically make sure everybody has pretty much what they need except for the beachfront real estate. But they have, you know, they can, they have all the transport they need. They have all the food they need, any book they need, et cetera, et cetera. That actually seems to me to be plausible depending on the kind of geopolitical forces. And that becomes a world in which people have to find their own meaning. I’ve sometimes used the example of Burning Man, which I’ve been to once.

– I’ve never gone.

– I think everybody should go once and it’s what they call a gift economy. And at least for a week it works, right? Except for what they call center camp where you can buy coffee. There’s no exchange of money. So like when I went, I waited in line for popcorn and I went to like, reach for my wallet and I realized popcorn’s free because everybody’s giving away something. And you know, it’s kind of a magical place. It still has its own politics. There was a murder there last year for reasons I don’t follow up. I don’t know what they’re, but in general, it’s an interesting, like not quite a thought experiment ’cause it’s real, but it’s temporary. So like for a week it’s pretty cool. People do these amazing art projects and then they often burn them. They’re not there to sell those art projects.

– It’s like a mandala. You let the wind just sweep it away.

– They let the fire take them away. I find that kind of amazing. And we might have a world like that. It’s a really open question, like, is that sustainable? And you know, the people who go there like making art, but not everybody does. And what are those people gonna do? And like, you know, you get to dark sides again where people are just in their cell phones all day, even worse than they are now. But I think it is an interesting thought experiment. What if money is not the main thing except on these margins of, you know, the beachfront real estate which just have to be allocated in a certain way. We could land there. Another part of the utopia is I think we can do a lot better with medicine. There’s a lot of ridiculous claims running around. Like Dario Amodei, who’s the CEO of Anthropic said that we’ll double lifespan the next 10 years. That is never going to happen. That’s absolutely insane because you have to do clinical work because biology is so complicated. But if you’re talking like in the longer limit, not 10 years, but a hundred or a thousand or whatever it takes, you know, probably we will be able to deal with most disease. We won’t be able to deal with suicides maybe, but maybe we’ll have better mental health interventions. You know, people may still get hit by a car, you know, there might be some death, but it might be much, much less sure than we have right now. Not in 10 years, you know, that’s probably not plausible. But eventually we might have, you know, much better medicine, much better material science. If we were wise, we would have better political structures. I don’t know if we have the wisdom.

– Yeah, no, it’s kind of scary. And it’s also scary that a handful of companies right now are holding the cards in a lot of what we’re talking about right here. But sticking in the utopian perspective. Final question. So my dad was a composer musician. Life was tough to make money there, so he kind of steered us kids away from music. So I took a piano much later, did it for a while, dropped it. Here’s my question. Is there still hope for me if I want to go back, I don’t need to be Rachmaninoff, but just to get to a place that would feel satisfying.

– Totally. I mean, and so I wrote this book, Guitar Zero about learning to play guitar at 40. And I read the literature, I’ll just tell you the mail that I got, I got lots of emails from people who like took up drumming at the age of 80. And say you think you’re old when you’re 40.

– You’re just a kid.

– You’re just a kid. And here’s, you know, my practice, I’m playing guitar, playing two hours a day. I’m finally playing this kind of jazz that I wanted to play. So yeah, you’re not gonna become Rachmaninoff, you know, starting at whatever that number is.

– But how much effort did you put in starting at 40?

– I was putting in a couple hours a day.

– With a teacher no doubt.

– Partly with a teacher. And the book is partly about teachers. One of my teachers is Terry Roche, who lives here in Manhattan. And I also went around and watched Teachers of Kids doing Suzuki and stuff like that. You know, a teacher is a very good thing. As compared to when I wrote that book in, well it was published in 2012. The software now is amazing. So, you know, chord recognition is one piece of AI that’s really impressive and works pretty well. And so like you can put in any song into certain–

– And it will tell you.

– And it will show you the chords and so forth. And will also detect what chords you’re playing and whether you got it right in your timing and stuff like that. So, you know, there’s lots of software in addition to teachers. I strongly recommend teachers too. And in the book I talk about how to find a good teacher. Not all of them are, you know, you don’t want a teacher who just teaches you to memorize a song. You want to understand some music theory and understand, you know, what your physical technique is and and so forth. But I think that physical injuries that you know, prevent you from, you know, making the chord shapes or whatever. And then maybe you just pick a different instrument. I think anybody can, there’s also a million apps on iPad where you can make really kind of amazing music. And so I think if you wanna make music at any age is entirely possible, you know, don’t set your sights on being the best in the world. Don’t even try to like be better than your neighbors, but just like learn to improvise for example. It’s super satisfying. It’s a wonderful thing.

– And the very kind of thing that in the utopian vision–

– You have lots of time to do it.

– The thing you have time to do. So Gary, thank you so much.

– Thank you.

– Fascinating conversation.

– Fantastic.

– Thanks so much.