- il y a 2 mois
Open Science
Catégorie
🤖
TechnologieTranscription
00:00Let me ask you a question. Are you a proponent of closed-source or open-source?
00:10Open-source? What about closed-source?
00:13Nobody? Alright.
00:17So, the debate between open-source and closed-source software
00:21is one of the most significant discussions in the tech world.
00:26On the one hand, open-source software often provides the advantage for people to pitch in collaboration when it comes
00:34to updates.
00:36We see examples like Mozilla Firefox or Brave for internet browsing
00:40and Moodle for online learning management systems.
00:44However, closed-source software often comes with costs and may have longer wait times for updates.
00:52Some include Microsoft Office for productivity, Google Chrome for browsing, and even Adobe Photoshop.
01:01Understanding the nuances between these two approaches allows users to make informed decisions based on their needs and references.
01:10Here to sort the intricacies of open science and push the open-source benefits,
01:17please welcome these extraordinary leaders.
01:43Hello, hello everybody. I have the privilege of sitting next to probably the biggest celebrity here today.
01:50Both a huge result of the amazing French tech ecosystem, but also kind of one of the founding fathers of
01:58AI,
01:58which is the conversation we're here to have today.
02:01So, I know traditionally you have to start simple, right, in a talk like this and start with the easy
02:06stuff
02:06and then build up to the big questions.
02:08But we've only got 35 minutes and I've got one of the foremost scientists on AI in the world,
02:13so I'm not going to start simple, I'm going to start with the big picture.
02:18Jan, I've heard you talking about and saying that one of the most challenging scientific questions of our time
02:25is to understand what is intelligence? What is the nature of intelligence? How can we reproduce it in machines?
02:31And you have this amazing view from the time that you've been working in it.
02:37Can you talk a little bit about what you have learned through the course of your work
02:42and where we are today in terms of understanding really what is intelligence
02:46and is it even possible to create a machine version of it?
02:50So, you can get an idea of that by observing the history of the progress in AI and connected fields.
03:00So, the idea, for example, that intelligence emerges from a large number of very simple elements
03:06that are interconnected with each other and that the function basically results from the connection.
03:14That idea emerged in the 1950s from neuroscience, from system theory and things like this,
03:20and the fact that we could perhaps reproduce this in machine was not obvious.
03:24It was called, at some point in the 50s, it was called self-organization.
03:29That's like a principle that perhaps explains why life emerges, but also intelligence.
03:34And so then how to reproduce this with machine, I think, was a big question.
03:37And for a while, there were people working on this in the 50s and 60s.
03:41And then another current of AI, if you want, that was not at all concerned with this,
03:46was trying to, you know, that sort of, that current gave way to sort of classical computer science, if you
03:52want, right?
03:53So, more algorithms and complexity theory and stuff like that, expert systems, classical AI.
03:59But I think the concept that learning algorithms, the way we use them today, deep learning, you know,
04:06the techniques have existed since the 1980s, but the type of, those kinds of techniques are much better at designing
04:13intelligent systems than human engineers.
04:15And that's very humbling for engineers, that, you know, it's impossible to design a system that can recognize images by
04:23hand.
04:24But if you let a system learn, it works.
04:29And it took a lot of, you know, a change of mind for scientists and engineers to realize that this
04:36was the case.
04:36And that's what allowed the emergence of machine learning and deep learning that we see today.
04:41So, when we look at what we have today, what we kind of hold up as the most advanced AI
04:46today,
04:47which is large language models for the average person, that's how they see, you know, the most advanced AI.
04:53What do you think that is the most advanced form of it that we have today?
04:58And is that a path towards, you know, recreating human intelligence, in your view?
05:03Okay, so my picture of the progress of AI, let's think of this as some sort of highway on the
05:09path towards reproducing,
05:11perhaps human level intelligence or beyond.
05:13And on that path, you know, that AI has followed for the last 60 or 70 years,
05:18there's been a bunch of branches, some of which gave rise to classical computer science,
05:23some of which gave rise to pattern recognition, computer vision, you know, other things, speech recognition, etc.
05:28And all of those things had practical importance at one point in the past,
05:34but were not on the main road to, you know, ultimate intelligence, if you want.
05:39I view LLM as another one of those off-ramps.
05:43It's very useful.
05:45There's a whole industry building itself around it, which is awesome.
05:49We're working on it at Meta, obviously.
05:51But for people like me who are interested in what's the next exit on the highway,
05:56or perhaps not even the next exit, like how do I make progress on this highway?
06:00It's an off-ramp.
06:01So I tell PhD students, young students who are interested in AI research for the next generation AI systems,
06:08I tell them, do not work on LLM.
06:10There's no point working on LLM.
06:12This is in the hands of product divisions in large companies.
06:17There's nothing you can bring to that table.
06:19You should work on the next generation AI system that lifts the limitation of LLMs,
06:24which, you know, all of us have some idea where they are.
06:28But tell us why.
06:29Why do you think that it's an off-ramp?
06:31Why is it definitely not the solution, like some believe, like, say, OpenAI?
06:36Okay, so about a little over a year ago, early in 2023,
06:43Meta essentially created a product division called GenAI, which is focused...
06:48So this is the organization that produces LLAMA,
06:51Meta AI, which is the intelligent assistant built on top of LLAMA,
06:56and then image and video generation systems, right?
07:01And that group, initially, the core engineering and R&D team of that group,
07:06was formed by people from FAIR, which is a fundamental AI research lab.
07:11So 60, 70 people from FAIR were basically transferred to that product group,
07:15and then a lot more people were hired, of course, to do their products.
07:22And what that has done is basically...
07:26It's allowed FAIR to refocus on longer-term research.
07:31What is the next generation AI systems?
07:34How do we build AI systems that understand the physical world,
07:37that have persistent memory, that can reason, and can plan?
07:43These are four characteristics of intelligent behavior that LLMs basically cannot do,
07:48or they can only do them in a very superficial, approximate way.
07:52So these are the challenges of AI for the next few years, perhaps the next decade.
07:56This is what FAIR now is focused on, really the next generation.
08:00And you will see this, some of this research filtering into, you know,
08:04LAMA, four, five, six, maybe.
08:07But ultimately, you know, kind of tracing a path towards human-level AI systems,
08:15and perhaps even superhuman.
08:17You've said famously that LLMs aren't even as intelligent as a cat, right?
08:22If that's the case, why are they useful?
08:25Can they be AI assistants?
08:27Or if not, you know, what's the point of them?
08:29You know, computers are useful, calculators are useful.
08:33A hammer is useful, and your hammer is not as smart as your cat.
08:36So, yeah, I mean, tools are useful.
08:38They don't need to be necessarily smart in every way to be useful.
08:41Now, there is something that is very important to understand,
08:44which is intelligence is not a linear scale on which you can just measure
08:49if something is more intelligent or less intelligent.
08:52It's a collection of skills and a capacity to acquire new skills quickly.
08:58Right?
08:58So, humans, I mean, how is it that any teenager can learn to drive a car
09:04in about 20 hours of practice?
09:06And we still don't have cell-driving cars.
09:08And we certainly don't have cell-driving car systems
09:10that can train themselves to drive in 20 hours without causing any accident.
09:14That's just completely out of reach, right?
09:17So, that tells you that there's a big gap between the type of intelligence
09:21that we can reproduce in machines
09:23and the kind of intelligence we observe in humans and animals.
09:25How is it that we don't have domestic robots
09:28that can, you know, clear up the dinner table and fill up the dishwasher
09:32and, you know, clean the house?
09:35Because that requires an understanding of the physical world
09:39which we are not able to reproduce with machines today.
09:43And so, in that sense, for that kind of understanding,
09:47AI systems are way inferior to what we observe in a cat
09:52or even a parrot or even a rat, actually.
09:57So, you know, the joke I used to say 10 years ago is that
10:00I'll be happy if by the end of my career
10:02we have an AI system that is as smart as a rat or a cat.
10:06It's still true.
10:07Yeah, we're not there.
10:09So, in terms of then what you're betting on right now,
10:13what you're kind of drawn towards as the path, you know,
10:17as you said, the actual highway, not the off-ramp,
10:20what are some of the ideas that you're playing around with
10:23that you think is going to really move us towards true, you know, machine intelligence?
10:29Okay, so there's something I have been advocating for almost 10 years now,
10:34which is self-supervised learning.
10:36And it's basically a set of techniques for machine learning that,
10:40or a way to use machine learning that has been astonishingly successful
10:44in the context of language.
10:46Okay, so what's the basic idea of self-supervised learning?
10:49You take an input, let's say it's a text, you corrupt it or you transform it in some ways,
10:53and then you train a very large neural net to reconstruct the original input, right?
10:59So, for LLMs, the idea is you take a text, you remove some of the words,
11:03and then you train a gigantic neural net to predict the words that are missing.
11:07Depending on the architecture of the neural net, you can now use this neural net
11:10to predict the next word in a text, right?
11:13You show it a word and it predicts the next word.
11:15And then you inject that word in the input and predicts the next, next word, etc.
11:18This is how every LLMs work.
11:21That's called autoregressive LLM.
11:23We should really call them autoregressive LLMs.
11:27And it's astonishing how much knowledge those systems can learn from textual data
11:32by being trained on essentially the totality of all publicly available text on the web, right?
11:37Typically, 10 to 20 trillion tokens.
11:40A token is like a subword unit, you know, it's about a word.
11:43So, it would take you or me maybe 100,000 years to read through this kind of material,
11:48so that looks like an enormous amount of data.
11:51So, the obvious thing is, an obvious idea, which we've been playing with for 10 years
11:56and largely failing until recently, is why don't we use the same idea
12:01to train a system to understand the world?
12:04The way, perhaps, animals or humans learn to understand the world, right?
12:08Babies open their eyes, and within a few months, they really understand that the world is three-dimensional,
12:12that there are animate and inanimate objects. Within a few months, they understand the notion of gravity
12:18and, you know, physical intuition and things like this, right?
12:21So, we understand the world by learning mostly from observation.
12:25How can we reproduce this in the machine?
12:27The obvious idea is, use the same idea that we use for text, take a video, remove some parts of
12:32the video,
12:33and train the gigantic neural net to predict what's missing in the video.
12:37For example, take a video segment, remove the last half of the video segment,
12:44and then train the system to predict the second half of the video from the first half, right?
12:49If the system is capable of doing this, then it will have understood the nature of the world, right?
12:54It will have understood that objects can move independently,
12:56and, you know, perspective changes when you move the camera,
12:59and, you know, all kinds of things about the world, like babies.
13:01And that doesn't work, and we've tried to make this work for 10 years.
13:04It's not a new idea, it's a very old idea.
13:08But there's been a bit of a breakthrough, what I think is a breakthrough, over the last four or five
13:15years,
13:15where we realized the best way to do this is to not have the system actually reconstruct the input,
13:22not have it predict what's going to happen in the video in all details,
13:27which is the idea of a generative model, right?
13:29We talk about generative AI and generative models.
13:31A generative model is something that generates the inputs.
13:35The solution we find to that problem is non-generative.
13:39So it's the technique, which is called JEPA, that's an acronym that means
13:43Joint Embedding Predictive Architecture, it's a mouthful.
13:46And the idea is basically you take the video, and you run it through an encoder
13:51that produces an abstract representation of that video.
13:54Then you transform or corrupt that video, also run it through an encoder,
13:58and then you train a predictor, but the predictor is trained in this representation space,
14:03in the abstract representation of video.
14:05So the system doesn't have to waste resources predicting every detail of what goes on in the video,
14:09which are really completely unpredictable.
14:12So that idea is really kind of a new concept.
14:15I mean, the concept actually is quite old.
14:17The basic idea of this goes back to a paper of mine in the early 90s,
14:20but not for video at the time.
14:22But we kind of rediscovered a little bit this idea, and we are kind of working on,
14:28you know, training systems to understand the world this way.
14:31So a system that you have them observe a piece of video, it extracts a representation
14:37that might be the idea that the system has of the state of the world at a particular time.
14:43And then perhaps you observe an action that either the agent itself takes,
14:48or maybe someone else takes.
14:51And then you train the system to predict what is going to be the state of the world
14:55after I've taken this action.
14:57And if you have a system like this, that's called a world model,
15:00predict the state of the world at time t plus one,
15:03from the state of the world at a time t, and an action you might take,
15:06then you have a way of predicting what the consequence of a sequence of action is going to be,
15:11which is now you have a system that is capable of planning,
15:14because it can figure out by search what sequence of action can I take
15:19so that I fulfill a particular goal or an objective.
15:22I call this objective-driven AI.
15:24It's a very general concept. It's not a new idea either.
15:27But that might be the architecture, future architecture of AI systems,
15:31that might be able to reason and plan and understand the physical world.
15:35I've written a long paper about this two years ago, which is publicly available.
15:39I've given lots of technical talks on this.
15:41And the interesting thing is, it's not a generative architecture.
15:45It doesn't involve LLMs.
15:49It involves world model, and it can plan.
15:51And reasoning is kind of a special case of planning, so it can do all of that.
15:55So that's kind of the hope that we have for the next few years.
15:57There's a lot of details to fill out there.
16:00It's a conceptual idea, but it's going to take five, ten years.
16:03It's something like a five or ten year plan, yeah.
16:06I mean, I think you're going to see effects of this in practical systems
16:09in kind of shorter term, perhaps in a few years.
16:13But my prediction is that perhaps five years from now,
16:16nobody in their right mind would use LLMs, actually.
16:19They'll be out of date, perhaps put out of commission by systems of this type
16:24that are objective-driven.
16:25Okay.
16:27So let's say that you are successful and your approach creates this human-level
16:33or animal-level intelligence.
16:36You obviously feel that that's a safe and a good thing to pursue and to get to,
16:41but your peers and your friends even disagree with you on that, right?
16:47So you and Yoshua Bengio and Jeffrey Hinton, who won the Turing Prize together
16:53and have worked and know each other for many years,
16:56you have very different views on what happens when we get there
17:00and whether it's a good or a bad thing.
17:03Can you lay out why you disagree with people that you clearly respect so much
17:08and what those disagreements look like?
17:11Yeah.
17:12You know, I can't put words in their mouth,
17:16but I can have some interpretation of what I think their thinking is.
17:20I mean, they're good friends, we talk all the time.
17:22I think Jeff, you know, recently retired, basically,
17:26and had a goal in his career,
17:28which was to discover the learning procedure of the brain, of the cortex.
17:33And for the longest time, he thought that the learning procedures
17:36that we use for neural nets, like backpropagation,
17:39were not the answer, that there was another algorithm that the brain was using.
17:42And then, you know, last year he had a bit of an epiphany.
17:45He discovered that, you know, the large LLM, I mean, the LLMs were working so well
17:50that perhaps all you need is backpropagation.
17:53And perhaps the goal of his life disappeared.
17:56That, you know, he contributed to backpropagation.
17:59Maybe that was a win.
18:00There was no need to do research anymore.
18:03And he went a little further.
18:04He said, LLMs basically have subjective experience.
18:10They really feel what they say, okay?
18:13I completely disagree with this.
18:14I think he's wrong.
18:16It's not entirely clear to me why he's claiming this.
18:19Perhaps because, you know, he's kind of retired,
18:21so he doesn't feel like he can contribute to the field anymore.
18:23But that's one point.
18:26The second point is something he agrees with Yoshua about.
18:31It's the fact that the way technology is used depends on institutions in society
18:36to direct those systems toward good use as opposed to bad uses.
18:41And both Yoshua and Jeff do not have a high degree of confidence
18:46in the ability of society to kind of make the best of new technological revolutions.
18:54Yoshua, for example, is very concerned about climate change
18:57and he sees the international community not doing much
19:00or certainly not enough about climate change.
19:02He says, maybe the same thing will happen with AI
19:04and we need to do something different.
19:07I mean, there is something to be said for that.
19:09But also, they are somewhat worried longer term
19:15about the effect of having systems that are smarter than humans.
19:19So I don't think that because a system is intelligent,
19:24it's going to want to take over or it's going to be dangerous necessarily,
19:29particularly if it's on the type of the type that I described earlier,
19:32objective driven.
19:33Those objective driven AI systems have to abide by the objectives
19:37that we give them.
19:38They cannot deviate from it.
19:39And we can do some guardrails so that they cannot deviate from that.
19:44And so I think those systems would be intrinsically safe by design.
19:50It's not going to be easy.
19:51It's going to require years of careful engineering and everything.
19:54But I think we can design them in such a way that they are intrinsically safe
19:59because by construction they have to optimize those objectives and guardrails.
20:02But there are several scientists who disagree with this, right?
20:06And who believe that even having clear objectives,
20:09that super intelligent AI systems might go around that or misunderstand
20:14or ultimately, I don't know, this idea of breaking free from those guardrails
20:20and being conscious.
20:21I mean, can we just put this to rest?
20:23Is that possible?
20:24Is that even a possibility you consider?
20:26Well, you have to put some guardrails and those guardrails are not going to be simple to design
20:31or even to train if they are trained.
20:35The same way we have guardrails in human societies that actually control superhuman entities.
20:42They're called laws.
20:43Laws are basically cost functions, right?
20:47That change the objective function that humans have to abide by by telling people,
20:53you know, if you do these things or telling corporations,
20:56if you do this thing, you know, you're going to have to pay that much, right?
20:59A fine or whatever or maybe go to jail.
21:02So that changes the objective function.
21:04We are objective-driven.
21:05Some of it is hardwired into our mind by evolution.
21:09That's human nature.
21:10And some of it is because of social norms and because of laws.
21:17So it's going to be the same for AI systems.
21:19They're going to have intrinsic objectives that are built into them by us, designed for them to be safe.
21:27They're going to have other types of objectives.
21:31But the way to design this objective is going to be iterative.
21:34It's not, you know, we're not going to have one day we don't have AGI and the next day we
21:40do have AGI.
21:40This is not going to be an event, okay?
21:44All the stories about, you know, AGI achieved internally or whatever.
21:47I mean, this has become a joke.
21:48But it's not going to be an event.
21:50There's not going to be a date before which we don't have AGI and after which we have it.
21:54It's going to be iterative progress towards systems that are smarter and smarter,
21:58can solve more and more problems and have more and more guide rails to prevent them from doing stupid things.
22:03If I design a robot, a domestic robot, and, you know, there's kind of a common joke that runs that
22:08says,
22:09if you tell this robot, go fetch me a coffee as fast as you can,
22:11and there is someone standing in front of the coffee machine,
22:14if you don't put any guardrails, the robot would just kill that person to get access to the machine, right?
22:19And that would be ridiculous.
22:20Of course, you would put guardrails, like, you know, don't hurt anyone and things like that,
22:23which you can put at a very low level or high level.
22:25And the design of those objectives is not trivial,
22:28and we're not going to make a super intelligent system without the guardrails.
22:31This is going to be kind of progressive, the same way that the first time we built a turbojet,
22:36we didn't put 1,000 people, you know, or 200 people on board that plane, right?
22:40We sort of refined progressively and, you know, built planes that had, you know, four jet engines,
22:46so that if one fails, you know, it can still fly and things like this.
22:49It's only today that we can, you know, fly halfway around the world on a two-engine turbojet.
22:54Yeah.
22:55So you can account for bad actors in terms of users
22:59and even the designing of the systems.
23:01What about corporate misbehavior or even, you know, big corporations with good interests?
23:08But we currently have a situation where there's a bit of a race dynamic here going on, right?
23:13There's Google and Microsoft OpenAI, of course, meta in the game.
23:18Maybe Apple comes in.
23:19And are you concerned about that sort of profit-driven race to build things and put them out,
23:27getting in the way of doing good science and building a safe system and being sensible
23:31and believing in humanity and all of that?
23:34Or being in the inside of one of these companies, do you actually trust that this is going to be
23:39well handled?
23:40Right. So I can't speak for the other companies, but meta has taken a particular stance in that respect of
23:47open sourcing its models.
23:49And the reason for open sourcing is that it's easier for a larger community of people to find issues with
23:55a model if it's open source.
23:56It's also easier for the community to fine-tune it, et cetera.
24:00And then you have to ask the question, what do you think are the real dangers of AI systems?
24:05So I do not believe in, certainly not with current technology, there is no danger of extinction or whatever.
24:11There's not even much of a danger of, you know, badly-intentioned people getting a recipe for, I don't know,
24:17a bioweapon or something like that from an LLM as they exist today.
24:21You know, there's been experiments on this where it was tested, you know.
24:25Yeah, there was a Stanford study recently as well.
24:27There was a study from the Rand Corporation a while back.
24:33And you don't get much more information with LLM than you get with a good search engine and access to
24:38a library.
24:39So that's not going to make any difference there.
24:42You know, there are immediate dangers of, you know, misinformation and things like this.
24:46It's certainly, you can use generative AI to generate, you know, fake text, news.
24:53That doesn't seem to have much of an effect.
24:56My colleagues at Meta, who work in integrity and content moderation, tell us that they've not seen an upsurge in
25:05attempts to produce, you know, disseminate this information using LLMs, for example.
25:10Then there is the issue of fake images and videos and audio.
25:18And that's a real danger.
25:20I think the industry broadly is, you know, working on standards of authentication for real content and taking down things
25:30that is obviously generated or at least tagging it.
25:34So I think there's going to be progress over the next months and years.
25:39But again, we don't see like a huge upsurge of disinformation there.
25:43What about cybersecurity as an issue with open source?
25:46I think the UK government has put out something on this along with other Five Eyes nations.
25:50Is there not a concern with open source systems of...?
25:55No, not particularly.
25:56I mean, the systems that are open source or not can equally be used for this kind of purpose in
26:03various ways.
26:05But again, the bottleneck of things like disinformation and, you know, unacceptable content, if you want, is at the dissemination.
26:14It's not at the creation.
26:15You know, groups that, you know, became masters of disinformation a few years ago with the US election like QAnon,
26:22it's just two guys.
26:23You know, they don't need AI.
26:25What made them influential is their dissemination circuit, the sympathizers.
26:29It's not really at the creation level.
26:32There's interesting studies also...
26:34But the dissemination too is done through social media networks.
26:36Yeah, but the social media have been, particularly Facebook and Instagram, have been very familiar with this problem for many
26:44years, you know, for over 10 years, 10, 15 years now.
26:47So there are systems in place to take down inauthentic content, to take down content that is obviously generated by
26:55bots or generated by people who happen to be all in some big warehouse in St. Petersburg, paid by the
27:01Kremlin or something like that.
27:02So, I mean, there are measures against this.
27:04In fact, what's interesting is that the biggest use of most sophisticated AI at Meta is for content moderation.
27:13It's precisely to take down things like that.
27:16You know, hateful content, false information, attempts to corrupt the electoral process or the democratic process.
27:22This makes massive use of sophisticated AI.
27:25It's not a new problem.
27:26It's really not a new problem.
27:27So staying on the theme of openness for a second, and then I'll move on to Meta.
27:32Have you seen a change in terms of how open researchers are being now in terms of publishing their new
27:39work around LLMs or other kinds of AI, particularly coming from companies now?
27:44And do you think that's going to slow down the pace of new discoveries and new developments in AI because
27:51of how closed off, you know, all the companies are becoming with their publishing?
27:56Yeah, that's a very important question.
27:58So what I've seen over the last 10, 12 years is two changes.
28:03The first change was in the late 2000, early 2010, when the research community in AI started publishing really quickly
28:11on Archive, Archive.org, not going through the regular process of publication, submitting to a journal or a conference and
28:20then waiting for the answer.
28:21You just post your paper right away so that other people can see it and build on top of it.
28:28And then you submit it maybe to a journal or a conference, right?
28:31And I adopted this philosophy when I created FAIR in late 2013.
28:37I said, we are going to practice open research.
28:40We're going to publish our research as fast as we can.
28:43We are going to open source our code, almost all of it.
28:49And that way we're going to basically accelerate progress over the entire industry.
28:54And at that time, Google was not nearly as open as we were, but they became more open because of
29:01us, essentially.
29:01Because they had to kind of...
29:03And it's a way to attract scientific talent.
29:05To attract good scientists.
29:06You also get research of better quality.
29:09If you tell people you have to publish, they kind of have higher standards for the type of results they
29:15obtain or the methodology they employ.
29:18It's actually easier then to convince your own product groups to adopt the technology.
29:25If you say, look, you know, this won a big prize at a conference.
29:28Like, this is the best technology ever for the problem you're trying to solve.
29:32Or, look, it's open source and there is hundreds of companies using it and we're not.
29:35That's embarrassing.
29:37So there's a lot of advantages to that.
29:41And so the community became very open.
29:43Open AI was created originally with the idea of doing open research, right?
29:48And this is why the AI progress that we've seen over the last ten years has been so astonishing.
29:54It's because the exchange of scientific and technical information has been so quick and so efficient and so open.
30:02But then what we've seen with all the commercial interest that has started to pop up over the last two
30:08or three years is that a lot of labs are starting to climb up.
30:13So open AI is not open anymore at all.
30:18They don't say a word about what they're working on.
30:23Google has some parts that are open and some parts that are not so open.
30:28not so much open source either.
30:30I think that's really bad for the industry because, you know, the problem is not whether, you know,
30:35Meta, Google, Open AI, Microsoft, whatever, is six months ahead of the other.
30:40It's really, do we have the capability in AI systems that we need?
30:43And the answer is we don't.
30:45LLMs are not the final answer.
30:48We need something much better than this.
30:49How are we going to get it if we all work in secret?
30:52So that's one of the reasons why Meta has been so open.
30:56The other reason, of course, is that, you know, it enables the emergence of an industry, right?
31:01A lot of people who are here in the room and certainly on the floor, particularly startups,
31:06would not have any activity in AI unless they were able to use open source platforms like Llama, which came
31:12out of Meta,
31:13and similar platforms like Mistral, for example, and a few others.
31:17So this really creates a new industry.
31:19It's a very similar phenomenon that occurred in the early days of the internet.
31:23You know, the internet now runs on open source software.
31:26Even the cell phone network runs on open source software.
31:29And it's because that's what the industry demands.
31:32AI is going to be a platform.
31:34And because it's going to be a platform, it needs to be open.
31:38So you talked about Llama and you recently released Llama 3.
31:42So, you know, that seems to be, you know, from a sort of, you know, in terms of capabilities, very
31:49powerful model,
31:50highly popular, being used really widely.
31:53Can you talk a little bit about sort of what you've seen as the most interesting users for it,
31:59both maybe within Meta and outside?
32:01How is Meta most likely to benefit from it and build with it?
32:05And also outside of Meta for startups that are using it would have been some great examples.
32:11So internally, there are several users, right?
32:13Of course, there is something called Meta AI, which is, you know, a dialogue system, LLM.
32:18I'm wearing those smart glasses from Ray-Ban, Ray-Ban Meta right now.
32:22And you can talk to Meta AI through that and ask questions.
32:25You know, I can even take pictures of you.
32:28All right.
32:29You got a picture.
32:30You got a picture.
32:32All right.
32:36I can actually ask it by voice, but the environment here I don't think makes that possible.
32:42So that's external use.
32:44Also, image generation systems, video generation, audio generation.
32:47There is a demo on the Metaboost today.
32:50Then internally, there is a system called MetaMate, which basically has been fine-tuned with all the internal knowledge base
32:59and data at Meta.
33:00So you can ask MetaMate any question about the company, and it will tell you.
33:04It will help you in your daily life.
33:05So it's very useful for employees.
33:07There's, of course, code generation systems to help programmers and software engineers at Meta.
33:13So, you know, all classical use.
33:15A lot of things for content moderation, obviously, and security and things like that.
33:20But the more interesting thing, I think, is what people are doing outside Meta with it.
33:24So, for example, there is a project in India to fine-tune Lama 2, maybe Lama 3 now, to essentially
33:33speak and understand all 22 official languages of India, which you can do with an open source platform.
33:39It's much more difficult to do with proprietary platform.
33:44And for India, that doesn't even begin to solve the problem because there are hundreds of languages and dialects, right,
33:49all across India.
33:50Another very interesting project is a friend of mine, former colleague from FAIR, called Moussa Fassise.
33:59He's from Senegal.
34:00So he went back to Africa a few years ago, and last year he created a startup called Kera Health.
34:06And the startup basically uses open source LLM that they fine-tune to first speak the local languages, French and
34:13Wolof, but there's like half a dozen others, and will provide access to medical information for people.
34:23It's very difficult to see a doctor in Senegal unless you live in one of the big cities.
34:28There's only five doctors for 100,000 inhabitants, so it's, you know, you need AI to help people get access
34:36to information.
34:37It's not just about health, it's also about, you know, agriculture and just everyday life.
34:43And a lot of people in some parts of the world are essentially illiterate or have trouble kind of communicating
34:51through written language.
34:53And so having AI systems that can communicate by voice in their local dialect, I think, would give access to
35:01technology to a lot of people who currently don't have access.
35:04Yeah.
35:05So speaking of kind of outside of the Western countries all over the world using it, I'll end with France,
35:10but before that, I'm curious about China.
35:13So I heard somewhere that about 40% of LLM downloads are in China.
35:19And also a colleague of mine in Beijing says that all Chinese startups are a huge fan of your work
35:23in particular and your philosophy and your openness.
35:28Do you, you know, what are your thoughts about the innovation coming out of there?
35:32Are you still, do you still, are you a proponent of collaborating openly when there is such a tense geopolitical
35:38climate?
35:39And can you tell us about sort of China and AI innovation and where you've talked about that?
35:44Okay.
35:44So the point of open sourcing and openness is the fact that you don't have to collaborate, right?
35:52You can use a piece of technology without talking to the owner of that technology.
35:55You can just download it and use it, right?
35:57And fine tune it, whatever, improve it, perhaps.
36:00I think in the first week that Lama 3 was put online for download, there was 1.2 million downloads.
36:09And I think in four weeks, they went up to 170 million downloads.
36:13I mean, this is an astonishing number.
36:15You know, are there 170 million people around the world that can actually run LLMs on their own machines or
36:22fine tune them or improve on them?
36:24Like the number of derived models from Lama 3 on Hugging Face and various platforms is astonishing.
36:31It's in the tens of thousands.
36:33So I find that really wonderful.
36:36Now, when you ask about China, I mean, we can ask that question about access to technology.
36:44What is the right trade-off between giving access or restricting access?
36:48And is there a way to partially restrict access that does not throw out the baby with the bathwater?
36:55And the answer is no.
36:56The answer is if you want a vibrant ecosystem where that progress is fast,
37:02the progress is going to be faster in regions of the world where the communication of information is fastest, right?
37:10Silicon Valley is Silicon Valley because there is a very, really fast exchange of technical information.
37:16People circulate and people have, you know, conferences and talks and blah, blah, blah.
37:20And so that kind of pulls the entire ecosystem and that diffuses the technology very quickly into the industry.
37:28China is not like that.
37:29The diffusion speed in China is much slower because a lot of it is top down.
37:33And so you don't get this kind of effect.
37:36That's the first thing.
37:36Second thing is the Chinese government is relatively scared of AI because, you know, those are systems that give information
37:47access to citizens,
37:49but it's not controlled in the same way as, you know, blocking Wikipedia or whatever.
37:54And so they actually put more constraints on what AI systems can do.
37:59And that sort of creates a bit of an isolated ecosystem.
38:02Third thing is that scientists in China are really good.
38:06You know, they're not that far behind in terms of technology development.
38:10And, you know, there's a lot of really interesting papers coming out of China and really good ideas that help
38:15the entire world of AI that came out of China.
38:17A good example of this is the single most cited paper in all of science is an AI paper about
38:24neural nets, about convolutional neural nets, about a particular type of convolutional neural net called ResNet.
38:29The main author of this paper is a gentleman called Kai Ming He, who was at the time at the
38:36Microsoft Research Lab in Beijing, published this paper.
38:40It was a revolution.
38:42OK, everybody uses this thing.
38:44The paper has got hundreds of thousands of citations.
38:47Amazing.
38:48Just a year later, Kai Ming He joined Meta, became a scientist at Meta.
38:54He recently left just a few months ago.
38:56He's now a professor at MIT.
38:59A complete win for Western industry and, you know, US education system at MIT.
39:05But the original invention was from China.
39:08Like, do we want to cut ourselves from innovation coming out of China? No.
39:11And a great example of success for open science too, right?
39:15Absolutely.
39:16Moving back to where we are today in Europe, we're here in France and you in particular, you know, alongside
39:23many others have done a lot sort of in terms of concentrating talent here and building up the ecosystem.
39:29I saw just this morning Shaq Khan, who is one of the original investors in Spotify, and he and many
39:36other Europeans talk about how so many of the best brains and AI are European.
39:41If you look at the sort of founders and, you know, godfathers, as you're called, of deep learning, they are
39:48European.
39:49Yet, you know, for whatever reasons, you've moved out to Canada and the US and elsewhere.
39:55So, you know, there's a question of what can we do here in Europe to attract this talent back, to
40:03nurture it here, and to grow AI companies with the clear inherent talent that exists here in Europe.
40:10And to keep, you know, to keep it here and benefit from it rather than it just being really concentrated
40:15in the value, as you talked about.
40:18Right. Right. It's a vast topic. So, the research community, and to some extent the AI industry as well, but
40:30the research community is sort of a world market, right?
40:33People tend to go to the places where they think they can have the biggest impact.
40:38They think they are being given the means to succeed, right?
40:42And certainly in the academic world, the US is very different from most of Europe.
40:51I say most of Europe because Switzerland can actually rival the best American universities in terms of things like, you
41:00know, means for research, resources for research, quality of students, etc., compared to the best American universities and also in
41:10terms of salary.
41:11I think what hurts the academic system in Europe, in the EU in particular, is very low salaries of academics
41:20and public research scientists.
41:23And so, until about 10 years ago or 12 years ago, it really wasn't cool for some of the top
41:29students to go into, to do a PhD and go into public research or academia.
41:34And there were very few opportunities in industry to really do ambitious research in industry, certainly in Europe.
41:42And, you know, when I graduated from my PhD in France in 1987, I moved to the US with the
41:50idea of coming back to France.
41:51I kind of stayed in the US, but the opportunities I was given at Bell Labs, which was kind of
41:57a mythical industry research lab at the time, were incredible.
42:00Like, you know, you could do incredible things because you were given the resources to do it.
42:05And I couldn't find this in Europe at the time.
42:08But now it does exist. Now, there is FAIR in Paris, which I created in 2015.
42:15There are other labs from other companies, Google and, you know, others that have been created in France and in
42:21other European countries.
42:22So this has given opportunities for young, talented students to actually see themselves as doing a career in research.
42:31And that has completely changed the landscape where, in France, talented students would not do a PhD.
42:41They were going to finance. Finance was a cool thing to do in 2010.
42:48But then only five years later, the cool thing to do was not finance anymore.
42:51There was a financial crash, first of all.
42:54But also, AI became the coolest thing you could do.
42:56And then they realized, if I really want to do AI, I actually need to do a PhD.
42:59So you saw a lot of talented young people starting doing a PhD or become entrepreneurs and, you know, plunge
43:08into AI research or development.
43:11That's why you have today, you know, 600 startups in AI in Paris.
43:17And this is, you know, the most sort of ebullient place for startup and AI investment in Europe.
43:26So the work is then to preserve the integrity of organizations like FAIR that can do real fundamental research in
43:33the face of all of the commercial onslaught that we're seeing in terms of developing AI.
43:37Yeah, I think it gave people, you know, young people and talented people the idea that you can have a
43:42career in sort of deep technology and, you know, first become a scientist and then try to apply this, what
43:49you've learned, either, you know, in industry research or by doing a startup.
43:54And it's a change of mentality also that was facilitated by various initiatives that the French government did to kind
44:03of facilitate, you know, raising money and creating companies and stuff like that.
44:07So I think we see sort of a really virtuous cycle that creates a vibrant ecosystem.
44:15the Paris AI ecosystem is on fire today.
44:19Absolutely.
44:19I agree.
44:20And thank you so much for joining us.
44:21We've come full circle to talking about research and scientists and the value of that in this world.
44:26Thank you so much for sharing and teaching all of us today.
44:30Thank you.
44:31Thank you so much.
44:31Thank you.
44:32Thank you so much.
44:34Thank you.
44:35I appreciate it.
44:36Thank you very much.
44:38Thank you very much.
Commentaires