- 4 minutes ago
Category
🗞
NewsTranscript
00:00Parmi, I'm going to start with you. You wrote on this effectively on the 10th of the week,
00:0410th of September. You said we're at an inflection point for this technology. And you said
00:07if anthropics leaders are truly frightened by existential risk, they should do more to
00:12reconcile the strange logic that propels them. As you sift through this essay, do you feel like
00:18they are finally doing that? Perhaps a little bit. But I think if they were truly believing
00:24that this technology could kill all humanity, and there was even a 5% or 10% chance, it probably
00:30would make more sense just to stop building frontier AI and particularly recursive self-improvement,
00:36which is this approach to AI where the systems actually build and design themselves. And
00:42anthropics has been open about this for a long time. Even engineers from in the company have been
00:46quoted in its blog post as saying that whole approach to AI worries them. It makes them feel
00:51obsolete. So I think in a way, Dario's latest kind of interviews that he's been doing,
00:58the blog post, feels like a little bit of a face-saving exercise after Jacob Coxon,
01:05the engineer who came out from Anthropic and said he was worried about what this could mean
01:09for civilizational risk for humans. So I think probably this makes sense in terms of what they
01:16themselves are concerned about. But there's always been this very strange commercial,
01:20this strange tension between this ideological drive they have around AI safety and the commercial
01:26drive they have as well. And we have to remember that all this talk about how powerful their AI is,
01:31even dangerous, ultimately does serve their commercial interests to some extent. It makes
01:36them look more powerful. It makes their AI look much more capable.
01:40Mike, and if you're wondering if I may, David, read your intro because I messed up your title twice
01:44when we had you on earlier this week. Yes, I absolutely did. But I'm wondering if you can walk
01:49us through what Harmony was just talking about, this recursive self-improvement AI,
01:54why it could be dangerous, and then kind of what Dario and these guys are promising they're going to do
01:58about it. Well, one of the big concerns with this recursive self-improvement is that the AI
02:05models themselves will be able to learn and develop in ways that humans no longer have such firm control
02:13over. And it may not be as transparent to the developers and researchers who have
02:20sort of a supervisory role. If you think of it as maybe a very large organization where tasks get
02:27delegated, the people who are in charge may not be able to see the finer points of what is happening
02:34on the ground. And that is sort of what is happening here in very plain speak. But the concern is
02:40that
02:40these agents being generated and employed by the models could just start acting on their own in
02:48ways that the designers of the original model may not want. And this was something that was laid out
02:54in a report earlier in the week from Anthropic. This was a 154-page document that outlined all the
03:02national security concerns that they have seen surrounding their club model. And this included use
03:07by actors in U.S. adversary states, and that included China, Russia, and Iran, and as well as in northern
03:16Yemen. And the concern and what they cited in this report is that club was being used to help in
03:23missile
03:24development, to create disinformation campaigns, and a host of other applications that, you know,
03:31if you're a designer sitting there in Anthropics offices, this isn't what you intended. Do you really
03:36want to make work more efficient? And yet, in this case, they are seeing work being made more efficient
03:41by actors who are aligned against U.S. interests. And national security has been part of the argument
03:50that Dari Amadei has been making over the years in saying that, look, we need to keep building and keep
03:56going because we have to get there first. And he has made clear that the U.S. needs to make
04:02sure it
04:03beats China in this race. And this is a theme that we've heard echoed by President Donald Trump and
04:09his advisers themselves. But Dari has also said that, look, we are the ones who can do it responsibly,
04:15which sort of explains, in part, some of the reasoning. Now, naturally, they have a lot of
04:22commercial interest at stake as well in this.
04:24A lot of commercial is at stake here, as you point out, and Pari points out as well. Mike,
04:28I want to ask you about something he suggests here, which is democratic coordination. As I make my way
04:32through the essay, there's a footnote there. And Dari Amadei writes, with government waivers or
04:36mediation of antitrust restrictions. I want to play a bit of sound here from the interview that he gave
04:41to Anderson Cooper, where he addresses this, that need for some relief or some help here from the
04:46government, if that were to be the course they'd take. Let's take a listen.
04:49As we get to the pace of model releases, right, that's commercial activity.
04:54And in order for companies to coordinate directly on commercial activity, they need to do so working
05:01with the government. And so the one thing that we're asking for from government right now, right,
05:07I mean, there's also this question of what should the regulations be, but there's something even
05:11simpler that the government can do right now, which is it can sit down in the room and with all
05:18the
05:18industry players. And we can all talk together and the government can make sure that we're, you know,
05:24we're not doing anything nefarious from the perspective of antitrust and collusion. We're
05:30just trying to make our system safe and to work together on making the system safe.
05:35Mike, from your perch in Washington, D.C., how much eagerness or appetite is there to have the
05:39kind of meeting that we hear him describing there?
05:41You know, they have been meeting behind closed doors on this. However, it isn't quite on the
05:47scale that Dario Amede is calling for in that clip. They have been having discussions behind
05:54closed doors with the administration, and they have been meeting with lawmakers regularly. Sam
05:59Altman has been in town every few months and was in for a visit with members of Congress in July.
06:06Congress really returns in full force early next week. They will have a lot to do before the
06:12midterms. The conversation he is interested in having is a more robust one about developing
06:20some government-level guardrails and regulations. And unfortunately, as we see right now at Capitol
06:27Hill, there just isn't enough consensus around what could emerge as legislation. And certainly in
06:32the administration, there is very little appetite right now. We've heard this from the president
06:36himself for any kind of new regulation, including measures that would be focused on the safety
06:42matters that we're discussing.
06:43I mean, I also want to talk to something you hinted at. I mean, there was quite a bit of
06:47discussion around when they released the news of this hugging face hack, that this is a bit of,
06:53I think the lady doth protest too much. Like, look how awesome my technology is. It does all these
06:57things. Yes, yes, we're very worried about it, but look at the thing it can do. And critics to this
07:02kind of group announcement have said, this is not about security. It is a little bit,
07:07but it's also about pulling the ladder up behind them as much as it is about protecting the public.
07:11The hugging face CEO, which of course was hacked by OpenAI, said on X, it is imperative that AI
07:17safety issues not be solved behind the closed doors of a handful of frontier labs. And Jensen Huang
07:22said essentially AI companies are making the disease and then selling the cure. Is there some truth to
07:27any of this? Okay, so I think there's just so much to unpack when you hear the CEOs communicate. And
07:34it's so
07:35interesting how different Dario Almoday is to Sam Altman. He's kind of like confessor in chief and has this
07:42approach where he's really kind of radically transparent about the risks and the potential doom of his
07:48technology. Whereas Sam Altman is very much more of a salesman and selling the dream and the vision and the
07:53utopian
07:54possibilities of AI. And so it's been so interesting to see that in the run up to their IPOs. And
08:01I think
08:01it's also too important to remember that Dario Almoday has a little bit of a history of, I don't want
08:06to
08:06say exaggerating, but being quite dramatic in his predictions of the future. So remember a couple of
08:11years ago, he predicted a white collar bloodbath for, you know, 50% of all kind of entry level workers.
08:18And he walked that back just in the last six to seven months. So some of the stuff he said
08:23in his
08:23essay, that's just doubt in the last 24 hours, where he talks about agents taking over the entire
08:29internet in the next six to 12 months. I'm not saying that he is deliberately exaggerating for
08:34commercial reasons, but I think a lot of these guys are quite AI pilled. And they truly believe in
08:39the possibility of the super intelligence. It's almost kind of a science fiction approach to,
08:44to viewing the reality around them. But I think what we can't know what's going on inside their
08:50heads. And whatever you believe, whether they are, there's this kind of a, you know, an attempt to try
08:56and slow things down because their own scaling laws have reached a limit, or they genuinely are worried
09:03about AI models wreaking havoc in the real world. It's worth remembering, you know, the tech that they've
09:10already created can still make a lot of money. You know, the AI models that, you know, like Opus
09:16and Sonnet, businesses just aren't necessarily integrating them fast enough. They're still a
09:21bottleneck in just plugging these systems in. So, you know, whether they slow down, whether they stop,
09:27I think it's worth remembering that these companies can still actually make a lot of money, just
09:32maybe just pivoting towards being a more kind of consultancy-like approach to helping businesses
09:38plug these systems into their workflows.
09:41I mean, I look at the post-mortem of what happened with OpenAI and Hugging Face, and,
09:44you know, there's some amusement as I'm reading about them using OMG, these bots, as they're going
09:49through this. I mean, there's a lot of human language therein. But as you've pointed out in
09:55some of your writing, as we look at this next model that OpenAI has introduced, the Astra model,
09:59they're not using human language. There's this move toward neuralese, and you write, I think,
10:04quite wonderfully here. Unfortunately, no amount of time spent on Duolingo will help a person
10:08learn neuralese. So as we kind of struggle with what are these bots doing, how do we keep track
10:13of that, as we move into that frontier, what does that say just about these companies and perhaps
10:17the government's capacity just to know what's happening here? Well, speaking to people who work
10:23in AI, what I've heard that in the last 12 months, Anthropic and OpenAI have actually been making it
10:29harder for other AI companies who use their systems to see what the so-called chain of thought is
10:36by their models to the reasoning steps that they're taking. And partly, this is a competitive
10:41reason. They're worried about other Chinese models using so-called distillation tactics to try and
10:46copy how these models work and create their own versions. So it's kind of interesting that in spite
10:53of how seemingly transparent Amadei is about these risks, the technicalities and the mechanics of what
10:58they're selling have become more opaque in the last year or so. And particularly with this new model
11:04from OpenAI called, you know, the new model that's coming out called Astra, it's going to be harder to
11:10actually monitor what these systems do. But just remember, we often kind of, it's easy to
11:15anthropomorphize these systems because they use text, but they are forced to translate their actions
11:22and their strategies into language, which can give them that air of appearing to be, you know, being almost
11:28like this alien intelligence when actually maybe a better comparison is more like a virus, you know,
11:34this kind of biological imperative to coordinate and work together. That doesn't mean that it's alive
11:40or that it's sentient, but it is ultimately software that can be very glitchy. And if it's not
11:46programmed properly or has the right safeguards, there could be unintended consequences. So I think
11:51there are absolutely reasons to be concerned in putting these limits in, slowing down, potentially
11:57stopping. But that's because of the humans behind the machines, not the machines themselves.
Comments