00:00Let me bring in Gary Marcus, cognitive scientist, and he's joining me from Vancouver, Canada.
00:07Gary on X says that he is the original OG Gen AI skeptic. So, Dr. Marcus, welcome to this broadcast.
00:20You have said, Dr. Marcus, that it is preposterous that humans will be extinct in five years
00:27and equally preposterous that everything is going to be fine.
00:31The two most dominant narratives are entirely wrong. Those are your words.
00:35If both extremes are wrong, what exactly do you place Jacob, where do you, I should say,
00:43where do you place Jacob Coxon's warning that the anthropic researcher who quit saying that labs are, in fact, gambling
00:53with our lives?
00:56I think the specifics of what he has said are preposterous, and he is kind of like doubled down in
01:02a series of interviews.
01:04So, he is very much on the side of like, we might all die in five years, and I think
01:08that's ridiculous.
01:09But I think that there are serious warnings. People in the field talk about P-Doom, the probability of extinction.
01:16I think that's pretty close to zero. You know, if the robots come for us, we will fight back.
01:23We're geographically diverse. We're genetically diverse.
01:26AI is not going to wipe out the entire population.
01:28But there's also what I call peak catastrophe, the chance that something serious might happen.
01:34If something serious means, for example, electrical grids being knocked out by cybercrime,
01:39maybe by malicious individuals using these technologies, or from disinformation causing accidental wars,
01:47I think that that's very serious.
01:48I think that we probably will have some very serious side effects.
01:53We're not going to literally have extinction.
01:55People don't need to, I think, have nightmares about that.
01:58I mean, eventually, you know, the sun will explode or something in the very long term.
02:01But in the near term, we don't have to worry about those things.
02:05But we do have to worry, excuse me, my voice is giving out because I've done so many interviews.
02:10We do have to worry about the consequences of insecure AI, basically.
02:17Another way I would put it is, like, a lot of AI is fine.
02:20Like, GPS navigation systems are not going to kill anybody.
02:24You know, people basically know how to make that software.
02:27But we have this new kind of software called AI Agents, where you take large language models that are not
02:33reliable,
02:33you hook up a lot of them, and you connect them to the Internet.
02:37And sometimes they do bad things.
02:38The OpenAI Hugging Face Incident was an example of that, where a set of AI agents couldn't follow basic rules,
02:46like don't hack into other people's systems.
02:48And that inability to follow rules is where the danger really comes from.
02:52It's not because AI is too smart.
02:53It's because current AI is not that smart.
02:56It can't follow rules.
02:57That means it can be exploited by bad actors.
03:01And it means that it might just misunderstand instruction and do something bad.
03:06Not extinction level, but serious.
03:09So, like, Jacob Coxon, I think, is way overstating the case.
03:14But there is something to be worried about.
03:16We can't just sit here and have no regulation and hope for the best, as some world leaders may have
03:21suggested.
03:22Okay, Dr. Marcus, I'm going to move away from what Jacob said to BRICS summit here in New Delhi.
03:30Prime Minister Modi told BRICS that weaponization of technology can hinder shared progress, calling for inclusive access.
03:39Is the danger you see the same one Prime Minister Modi is naming nations using tech access as leverage?
03:47Or is the more urgent weaponization happening elsewhere, perhaps in how AI capability itself is being marketed?
03:59I mean, there's a lot of different things going on.
04:01It's hard to say, like, which is most important.
04:04There is, though it's not really what I was talking about, issues about equitable access to these technologies, about how
04:11the countries other than U.S. and China are going to have access and what that's going to look like.
04:17So there's a whole set of issues about economic distribution and things like that.
04:24There's another set of issues, which is really one that I'm talking about, which is that current AI is not
04:29very well controlled, or people use the technical term aligned.
04:32And that can mean it does things we don't want it to do, or that bad actors can exploit it
04:37to do bad things.
04:38And that's a different set of risks than the economic set of risks.
04:42And there's still other risks, like non-consensual deepfake porn is yet a third category of risks that is, you
04:50know, starting to cause a lot of problems in high schools and so forth.
04:54The thing about AI is it's not one thing, but many.
04:57There's many different underlying technologies with different capabilities.
05:01There's all kinds of social and political things that are factoring in.
05:04So we can't just say, like, this is the one thing you should worry about.
05:08The truth is it's like a hydra with many different heads.
05:10There are also many advantages, but there are many disadvantages we need to worry and to rein in.
05:15Then are you worried more about disadvantages than advantages, given the pace of AI and the concerns being raised now
05:23by the big, you know, the tech giants themselves?
05:30I'm pretty worried at this point.
05:32I think I'm long-term positive in the sense that I think that there's a possible path to an AI
05:39that is really good for society.
05:41But short-term, I'm worried.
05:42I think the current technology is unreliable.
05:45And I think that unreliability causes problems.
05:48And I think that the companies that are running it are mostly pretty greedy and not really mindful of the
05:53consequences.
05:54And I think the governments are mostly letting those greedy companies do what they want at the expense of humanity.
05:59So right now, I think AI is probably a net negative for society.
06:03I don't think it has to be that way, but I think that's kind of where we are right now.
06:07And we need to up our regulation game.
06:09And we need to have better technical tools as well.
06:12So what can be these regulation and technical tools?
06:16What's responsible AI then?
06:19Or should I say, how do you make AI responsible?
06:24On the governance and regulatory side, I think the number one thing that we need is that every country needs
06:31to do something like what the U.S. does for drugs with the FDA, which is to say, we require
06:36a cost-benefit analysis.
06:37You have to show that the value of your drug offsets the consequences, the risks.
06:43You know, lots of drugs have side effects.
06:45That doesn't mean we don't put them on the market.
06:47But you have to make the case that the benefits outweigh the risks.
06:50And you have to have independent scientists weigh in on that.
06:54You have to make an argument that what you have is good, and they may send you back to the
06:58drawing board or ask you to do more experiments or come up with a way of dealing with something.
07:03We need that for AI.
07:04That's the number one thing that we need on the regulatory side is some kind of screening procedure.
07:10It needs to be transparent.
07:11It needs to have independent scientists in the loop.
07:15Secondly, on the technical side, we need an AI that can follow instructions.
07:19That might sound trivial, but it's actually pretty hard given the starting point we have now.
07:24So the starting point that we have right now is these large language models that try to predict essentially what
07:31a person would say in a given context.
07:33And it does that by looking at the statistics of how people talk across a whole lot of different kinds
07:38of text and so forth.
07:39That system is not built from the ground up to follow instructions.
07:43It often follows instructions, but it's not guaranteed to do so.
07:47And that's a real problem.
07:49And I think we need to go back to the drawing board for better technology.
07:52So, Professor Marcus, China here is pushing open source AI models to the global south.
07:58And BRICS itself has just unveiled an open source AI ecosystem framed as tech equity.
08:05Is there a genuine or do you see this as genuine democratization or a subtler form of the same weaponization
08:14that Prime Minister Modi has warned against?
08:20It's probably both.
08:22You know, there's some value in having open source AI for the world.
08:25And it's China pushing its particular advantage.
08:29They're not as good on the closed source models.
08:31They're better on the open source models.
08:32And they want, you know, what we call vendor lock-in.
08:35They want people to use their models.
08:37So it's a bit of both.
08:39It's complicated.
08:40Okay.
08:40So the U.S.-China chip war is fought over who controls the physical layer AI runs on.
08:46If scaling is really a dead end, as you argue, has the entire chip war and China's open source counter
08:55push been fought over the wrong resource then?
09:02I think that people have been fighting over the wrong thing.
09:05So right now, what matters maybe is who has more chips.
09:09But that's not going to last.
09:11What's really going to matter in the long run is who builds the better AI.
09:15And I think there's actually much more room to improve AI than people realize.
09:18We still have massive problems with planning, massive problems with these systems hallucinating, with reliability, and so forth.
09:26I think that the real advantage comes from solving those technical problems.
09:30On the chip side, I think that China is going to catch up.
09:33The price of chips is going to go down.
09:36And people are actually going to lose a lot of money on chips when they depreciate.
09:39And you have, you know, more chips than you really need.
09:41The other thing is that people are going to make more and more efficient systems.
09:46They're optimized to use fewer and fewer chips to run locally rather than on the cloud.
09:51And so we're going to wind up with all these data centers we don't really need.
09:55And the advantage is not going to be in the data centers, I think, but in developing a better technology.
10:00But everybody is wedded to large language models right now and maybe not thinking outside the box enough.
10:05Okay, Professor Marcus, can a country then build real autonomy in AI by adopting China's open source models?
10:13Or does that just trade American dependency for Chinese dependency?
10:21This is, again, a complicated question.
10:25I don't know that any country can build autonomy at this moment except by building a better and maybe more
10:34creative technology.
10:36There's going to be some dependency right now.
10:39I think there's a lot of talk about building sovereign AI.
10:42And I think more and more countries are going to do that because they don't want those dependencies.
10:47The positive thing for countries that might want to do that is that the general techniques that people are using
10:53are pretty well known.
10:55And so it is possible for other countries to build their own.
10:59And as efficiencies come in place, I think the advantage of having giant clusters of GPUs will diminish.
11:06You won't need quite as many in order to do most of what you want to do.
11:10I have two more questions here, Dr. Marcus.
11:13After Jacob Coxon was who we saw, Dario Amadei and also Sam Altman both spoke about slow down,
11:21that how there is a need to slow down the pace of growth.
11:25Where is the future then, according to you?
11:28I mean, it's hard to predict the political future.
11:32It's kind of changing literally by the hour.
11:35In one minute, Dario says, we should have a pause.
11:37And Sam says, great.
11:38And then five hours later, Sam posts something on Twitter saying, well, we're not really stopping.
11:43We're just trying to do things a little bit more carefully or something like that.
11:48So there's a lot of back and forth about what these guys mean.
11:51There's back and forth about what the U.S. government wants to do.
11:54One minute they're saying they don't want to regulate anything.
11:57The next minute they say, we've got regulation.
11:59And that regulation is, you know, something secret that we don't want to tell anybody about.
12:03And so the kind of political winds on this are changing constantly.
12:09And I think are pretty unpredictable.
12:12You know, in the United States, there's a lot of pressure against data centers.
12:15A lot of people are now concerned about so-called runaway AI.
12:20You know, Trump could change his mind on all of this on a dime.
12:23And then we'd be in a different place.
12:25So I feel comfortable making technical predictions about what particular technologies can do.
12:30But on the political side right now, I don't think it's ever been more unpredictable than it is right now.
12:34And I don't think it's ever been more consequential either.
12:37Like, for example, whether or not China and the U.S. come to any deal about slowing down would have
12:42profound consequence.
12:43Maybe they'll do that.
12:44They probably won't.
12:45There's a good argument that they should.
12:48These decisions that Xi and Trump are going to make in the next several weeks are super important but very
12:55hard to predict.
12:56Okay, so let's try and sum this up, sum this conversation up, Dr. Gary Marcus.
13:01Because you are saying that it's difficult to predict on the political side.
13:06What would you want the countries to do then?
13:11I think that the best thing that we could do is to stop thinking about AI as a zero-sum
13:16thing and start to think about AI as how we could collaborate and make the world a better place if
13:22we work together.
13:23That's not really how people are thinking about it now.
13:25They're all thinking about it as a race.
13:27But that race has not really helped anybody but NVIDIA.
13:30It hasn't really helped individual people.
13:32It's put economies like the U.S. perhaps in jeopardy.
13:35It's raised the level of tension in the world.
13:37Maybe we need a different paradigm that's much more cooperative.
13:40All right, Dr. Gary Marcus, pleasure having you and getting your thoughts on what is appearing to be a very,
13:48very fast-changing world even in the space of AI.
Comments