- 1 year ago
Jeff Dean, Chief Scientist, Google DeepMind and Google Research, Google Interviewer: Jeremy Kahn, Fortune
Category
🤖
TechTranscript
00:00Hey, everybody.
00:01Thanks for joining us.
00:02Jeff, thanks for being here.
00:03You're having me.
00:04I'm going to go to the audience for Q&A
00:06pretty early on in the session.
00:07So please have your questions ready.
00:09Wait for Mike Handler to come to you, and then stand up,
00:12and please state your name and state your question.
00:15Yeah, so Jeff, in the introduction,
00:17Andrew talked a little bit about the merger
00:19between Google Brain and Google DeepMind.
00:21Google had these two separate AI fundamental research units.
00:25What was the reason for combining them?
00:27And what benefit have you seen from that merger?
00:30Yeah, I mean, I think both organizations
00:33were doing fantastic work.
00:34And a lot of different AI innovations
00:38have come out of both groups prior to the combination,
00:40things like Transformers, and BERT, and AlphaGo,
00:44and various other alpha-related projects on the DeepMind side,
00:48AlphaFold.
00:50But I think as we realized that we were both striving,
00:54both organizations were striving
00:56to build large-scale, multimodal models that can
00:59do lots of interesting things, it
01:01made sense to kind of combine the people working
01:04on those ideas so that we have a better set of ideas
01:08to build on to pool the compute.
01:10So we sort of focus on training one large-scale effort,
01:14like Gemini, rather than multiple fragmented efforts.
01:17And so that's been really good.
01:19I think it's been really great to see people come together
01:22and really just kind of work together
01:24on building awesome stuff.
01:26Right.
01:27And has there been a change in the tempo with which you're
01:29able to move things from research to product
01:32as a result of that merger, or a change in the culture
01:34at all from that merger?
01:36Yeah, I mean, I think, obviously, we
01:38want to build things like Gemini that
01:41can sit underneath a lot of different products.
01:43And so we have a pretty strong focus
01:45on how do we make great models that
01:47are great for cloud applications,
01:49but also sitting underneath various search features,
01:53or Gmail, or other kinds of existing Google products,
01:56but also can create new kinds of products.
01:59So that sort of large landscape of product focus
02:04is really important.
02:05But we're really focused on the fundamental research
02:07and making the best things possible,
02:09and then rely on product teams to kind of take
02:11those building blocks and figure out
02:13how best to use them in their setting.
02:14Great.
02:15When I talk to companies about using generative AI,
02:19one of the things that always comes up
02:20is the rate of hallucinations from these models.
02:23And it seems to be one of the things that's
02:24created a bit of a bottleneck to actually deploying them
02:26in any kind of real-world use cases.
02:29I know you guys are working very hard on this issue.
02:31There was some hope that when multimodal models came around,
02:34like Gemini, that that would reduce hallucination rates
02:36because the model would understand
02:38both the language and also the visual imagery that
02:41went with that language.
02:42It's made some difference, but that
02:44hasn't completely gone away.
02:46How are we doing on sort of solving the hallucination
02:48problem, and do you think it's a solvable problem
02:52in the current architectures?
02:53Yeah, I mean, I do think we're making progress.
02:56Like, it's a difficult problem, for sure,
02:57because the models are trained to generate probabilistically
03:00plausible sentences, and those are not always true.
03:05But I do think there are some glimmers of hope.
03:08So one of the things in the latest Gemini models
03:11is the fact that it has this very long context capability.
03:14So you can put lots of information
03:16that you're trying to sort of use in answering questions
03:20that you're doing, and that information to the model
03:23is much clearer than sort of the training data
03:27that the model was exposed to in the pre-training phase, which
03:29is sort of like a big jumble of tokens that
03:31are used to generate these probabilistic models,
03:35but is not as clear as information
03:37in the context window.
03:38So that information, I think, is much easier,
03:41and the model tends to hallucinate less
03:43about that information.
03:44So I think that's one promising direction,
03:46and we're obviously pushing on lots
03:48of different other directions.
03:49Right.
03:49At Google I-O this year, I know you guys
03:51announced that you have this thing called Project Astra,
03:54which is an attempt to kind of build
03:56AI models that have agency.
03:58What kinds of actions will Astra be able to perform,
04:02and how soon is this going to be released?
04:04Yeah, I mean, we're hoping to have something out
04:06into the hands of test users by the end of the year.
04:09So it's pretty exciting, I think.
04:12The ability to combine Gemini models with models that
04:16actually have agency and can perceive the world around you
04:18in a multimodal way is going to be quite powerful.
04:23I think we're obviously approaching this responsibly,
04:26so we want to make sure that the technology is ready
04:28and that it doesn't have unforeseen consequences, which
04:30is why we'll roll it out first to a smaller
04:33set of initial test users.
04:35But I think it's going to be pretty interesting,
04:37because all of a sudden, the model
04:40can be perceiving the world around it, around you,
04:43and doing stuff on your behalf.
04:45Right, interesting.
04:47Bill Gates recently said he thought large language models,
04:50that there were sort of two more turns of the crank in terms
04:53of scaling up these models further.
04:55But he also said that he didn't think
04:56that those two turns of the crank
04:58would kind of get us to AGI.
04:59I'm curious what you think.
05:00How many turns of the crank are left on sort of transformer
05:03based LLM models, and where will those turns get us?
05:07Yeah, I mean, I think as you see the models grow
05:11in capability, some of that is due to scaling.
05:13So every time you increase the scale of the model
05:16and the amount of training data that these models are trained
05:18on with the model capacity as well,
05:21you do see new capabilities kind of emerge.
05:24You see the models hallucinate less.
05:26You see them sort of able to do things
05:28that they couldn't do before at smaller scale.
05:30So that is a promising sign.
05:32And a couple more generations of scaling
05:35will get us considerably farther.
05:38I do think we're going to need some additional algorithmic
05:41breakthroughs, which is something
05:42that we've always focused on, is how do we sort of take scaling
05:47but combine it with algorithmic approaches that combine well
05:51with scaling to sort of improve the capabilities of the model.
05:53I think things like factuality and reasoning capability
05:56where the model can sort of imagine plausible outputs
06:01and reason its way through which one makes the most sense
06:04are the kinds of things that are going to be important
06:07to really make these models robust and more reliable
06:10than they already are.
06:12I want to go to questions from the audience.
06:13If you have a question, please raise your hand
06:14and we'll get a mic handler to you.
06:16There's one right down here, if we can please
06:19get a mic to this gentleman.
06:20And if you could please stand up and say who you are.
06:25Thanks, Mark Papermaster.
06:26I'm CTO at AMD.
06:28And Jeff, we're all excited with the advancements
06:31like you all are doing with Gemini.
06:33But we also all are talking about the demands of power.
06:37How do you think about it holistically from your role
06:40in terms of everything from the chip engineering
06:42all the way through the models and the application stack?
06:46Yeah, it's a good question.
06:48I think one of the things that I really like about Google
06:52is we have a fully integrated system.
06:54So we design our own chips for AI,
06:57although we also use other people's chips.
07:00And then we have our own sort of system software
07:01on top of that we can make efficient use of those chips.
07:04When we first introduced the Tensor Processing Unit in 2016,
07:09that was quite a big advance in terms of energy efficiency.
07:12It was about 30 to 80 times more efficient
07:14than CPUs and GPUs of the day.
07:16And we've now been through multiple successive generations
07:19of TPUs that have also made pretty significant improvements
07:22in energy efficiency.
07:24And we've also, for many, many years,
07:26focused on the overall energy efficiency of our data centers.
07:30We have very low PUEs.
07:32We have a lot of data centers of ours
07:35are powered by 90% or more clean energy, which
07:38is super important as these things go up.
07:40But despite this, Jeff, you just reported
07:42that your CO2 emissions have gone up
07:45over the last several years despite these sort
07:47of improvements and despite these optimizations.
07:49So can you talk a little about what's going on there
07:51and sort of what's the path out of this?
07:53I mean, if we have carbon footprint scaling at a rate
07:56that's going up at the same kind of slope as capability,
08:00maybe that's not where we really want to be.
08:02Yeah, I mean, we have a public commitment to,
08:04by the end of 2030, have essentially
08:06100% of our data center operations
08:09powered by clean energy.
08:11And that's not necessarily a linear thing,
08:12because often we work with clean energy providers, solar capacity
08:23providers, or wind providers.
08:24And some of those things come online in 2027, 2028, 2029.
08:28And so those things will provide sort of significant jumps
08:32in the percentage of our energy usage
08:34that is carbon free energy.
08:35But then also, we want to focus on making our systems
08:39as efficient as possible.
08:40Right.
08:41And is the access to renewable power
08:43become a constraint on sort of how quickly you
08:45can scale the technology?
08:50You know, I think there's been a lot of focus
08:52on the increasing energy usage of AI.
08:55And from a very small base, that usage is definitely increasing.
08:58But I think people often conflate that
09:00with overall data center usage, of which
09:03AI is a very small portion right now, but growing fast.
09:06And then sort of attribute the growth rate
09:08of AI-based computing to the overall data center usage.
09:11So I think it's really important to look carefully
09:13at all the data and really understand the true trends that
09:17underlie this.
09:18Cool.
09:19Other questions from the audience,
09:20please raise your hand if you have a question.
09:22If not, I've got plenty more for Jeff.
09:23But if someone else in the audience has a question.
09:25Oh, here in the back.
09:27The gentleman in the colorful shirt first, I think.
09:29And then if you can please stand up and state your name.
09:31Hey, Will Wilson from Antithesis.
09:33Jeff, what is your current take on the debate over AI risk
09:36and alignment and all that stuff?
09:39Yeah, I mean, I think it's a good and healthy debate,
09:42for sure.
09:42I mean, I think there's many different viewpoints
09:45along a spectrum of optimism and pessimism
09:49and how much concern should we have
09:51for disastrous consequences.
09:53I guess I'm somewhat in the middle.
09:55I think there's a huge amount of benefit from AI technologies
09:59to the world.
10:00So we should make sure that we sort of do our best
10:05to make sure that the world gets access
10:07to those things for things like education and health care.
10:10But we also need to make sure that some of the risks of AI
10:14are sort of mitigated or diminished
10:17in rolling these things out for the positive benefits.
10:20So it's obviously hard to predict the future
10:25in a very fast-moving field.
10:27I think probably the most extreme viewpoints
10:29on either side are probably not the most likely outcomes.
10:35Do you have a take on the California Senate Bill 1047?
10:39Do you have a view on whether that's a good approach
10:41to regulation or not?
10:42I don't.
10:43I mean, I would say, in general, it
10:45is really good for governments at all different levels
10:49to be thinking about how they want to regulate AI.
10:51I think there is certainly, you know,
10:53I sort of think of it as a very general technology.
10:56And often, the regulatory process probably
10:58should focus on how is that technology being
11:01used in a particular setting, right?
11:02So if you have autonomous vehicles,
11:04that makes sense to have a certain set of regulations
11:06there.
11:07In health care settings, there's already
11:09sort of a relatively comprehensive framework
11:11around rollout of new technologies and health.
11:14And so I think adapting those for new AI-based things
11:17makes a lot of sense.
11:18Cool.
11:18I know we have at least one other question,
11:19because I saw a hand up before.
11:20So why don't we go to someone else?
11:23Just here, yeah.
11:24Please stand up and state your name.
11:25Yeah, hi.
11:26This is Siddhartha Garol with SAS Software Company Freshworks.
11:29So Jeff, I'm a Zoogler, so I have a soft spot
11:31in my heart for Google.
11:32But having said that, one of the challenges we have
11:35is, as we look at models, we don't
11:37want to be locked into one particular model,
11:39because we don't know which model or which company
11:41is going to drive innovation the fastest.
11:43So what's your advice for companies like us
11:46in terms of how we should think about not locking ourselves out
11:49of future innovation by picking Gemini?
11:52Yeah, I mean, I think I hear those concerns.
11:55I think we think we're going to innovate quite quickly
11:58and give people what they want in terms of models.
12:01But we also recognize there's a whole bunch of choices.
12:04This is partly why we, for example,
12:05have open source models called Gemma
12:08that we've recently introduced.
12:10We just provided Gemma two updates a few weeks ago
12:15of 9 billion and 27 billion parameter models.
12:18And the 27 billion parameter model
12:19on the LMSIS Elo rankings actually
12:24is the top ranked open model at the moment,
12:27or it was at least a week ago.
12:28I haven't looked.
12:29This week, this is a fast moving field.
12:32But open source models can provide people some comfort
12:35that they have a little bit more control.
12:37But I also think a lot of people
12:39want sort of APIs that they don't have to necessarily spin
12:42up a whole bunch of GPUs themselves.
12:44They can just sort of use it as a building
12:46block for getting stuff done.
12:49Cool.
12:49Other questions?
12:51Down here in front, there's a question.
12:52I see if we can get a mic to this gentleman just here,
12:56right here in the front row.
12:57Oh, can we get a mic to him right here?
13:01Sorry.
13:06There we go.
13:07Please state your name.
13:08Hey, my name is Ketan.
13:09I'm CEO and co-founder at Union.
13:12Firstly, I'm a huge fan of your work.
13:1420 years, every paper, everything,
13:17this is what we are built on.
13:19Secondly, I think this is mostly looking
13:21into the future, purely technical question,
13:23not talking about the implications of business.
13:25But what's the one thing that will surprise you
13:30in a good way, in a bad way, in those cranks
13:32that we're spinning?
13:33Let's say we have those two.
13:34The next generation, I'm not even
13:36talking about the second one.
13:37What is the one thing that will surprise you in a positive way?
13:41And what will disappoint you if it doesn't happen?
13:45Yeah.
13:46I mean, I think one of the things that is probably
13:50going to be a surprise to people is the improvement
13:53in multimodal capabilities.
13:55I think the first couple of iterations of Gemini
13:58really sort of introduced multimodality.
14:00But I think it's going to be a much more naturally integrated
14:02thing for many different kinds of modalities, audio,
14:05video, imagery, both on the input and generation side.
14:09So I think that's going to be an exciting new direction.
14:16On the what will surprise me on the downside,
14:19that's always hard to predict, right?
14:21But it's always going to be disappointing when it happens.
14:23Because you're like, oh, I could have
14:24sworn this thing was going to work out.
14:26But it didn't.
14:27And I think this is part of the nature of kind
14:30of a research-driven but still fairly close to product
14:36kind of development framework is you're constantly
14:39trying to bring new research innovations into the thing
14:43you're building at the largest scale
14:45and make sure that those things have been
14:47de-risked at a smaller scale.
14:50And sometimes those interactions you
14:52can really predict really well.
14:53And we do our best to sort of scientifically,
14:56rigorously evaluate how they're going to interact.
14:58But when they scale up, sometimes
15:00they interact in ways that you can't predict.
15:01OK, let's get another question while we still have time.
15:04Let's go right here.
15:05Hey, Healy Seifer, Boom Pop.
15:07Thanks for being here.
15:08So one thing I think about a bunch
15:10is the fact that education in the States
15:11is like we're 60 or 70th out of 194 countries.
15:14And I think with access to AI, it's
15:16wonderful because as internet and mobile phones are around,
15:20everyone can ask questions and get answers.
15:22The trick, I guess, and this is kind
15:23of a very high-level question, which I think is the stuff you
15:25think about probably, is if you're getting the answer all
15:28the time, how do you also balance the responsibility
15:30of giving that person who might not have education
15:32to think critically, to think about the answer?
15:34And how do you caveat?
15:35How do you say, look, AI may be right, may be wrong?
15:38And how do you make sure that, I guess, critical thought
15:40remains center in humanity as AI becomes more prevalent?
15:43That's a good question.
15:44Yeah, it's a very good question.
15:46First, I think there's huge potential for AI
15:48to help in education.
15:50We know educational outcomes when
15:52you have a personalized tutor are two standard deviations
15:55away from those when you have a much more sort of group
15:58focused setting where you're not
16:00being challenged at the level of an individual and someone
16:04who knows what you know and knows what you don't know.
16:07So I think that's the potential.
16:09I think part of deploying these systems in education
16:13is you want to have the system really challenge people
16:17in how they learn best and make sure that they're actually
16:20mastering the material before they sort of go off.
16:22But it does enable them to kind of take
16:25the right educational path for them.
16:27And I think that's hugely, huge potential.
16:30Yeah, I just want to ask quickly.
16:33There's been a lot of concern that the use of generative AI
16:36is churning out a lot of misinformation,
16:37a lot of poor quality content.
16:39Google, obviously, the company, that its entire business
16:41is based on finding good quality information on the internet.
16:44And there's a lot of concern about sort
16:46of the poisoning of the information ecosystem.
16:48How are you looking at that at Google?
16:49And are you concerned that this technology you're helping
16:51create is also kind of ruining the very thing you
16:53built the business on?
16:55Yeah, I mean, I think this gets to our responsibility
16:57AI principles.
16:58Like, we've always focused on how do we roll out
17:00AI technologies in ways that sort of benefit people
17:03and that sort of minimize things like harmful bias
17:06or misinformation or other things like that.
17:09And so it is true these models can create misinformation.
17:13Misinformation, though, is not a new problem.
17:15It's something that's existed for years and years.
17:18But AI also gives us tools to battle misinformation.
17:21Right, so you can do both.
17:22I'm afraid we're out of time, but I
17:24want to thank you all for listening.
17:25It's been great.
17:26Jeff, thank you so much for joining us.
17:28Thank you for having me.
Be the first to comment