The Smartest Machine On Earth

TV Hub

1 hour ago

Can IBM's Watson computer win on Jeopardy!?

Transcript

00:00Major funding for NOVA is provided by the following.

00:06David H. Koch

00:10and

00:13Discovering New Knowledge

00:17HHMI

00:22and by

00:23The Corporation for Public Broadcasting

00:26and by contributions to your PBS station

00:29from viewers like you. Thank you.

00:40Dave Ferrucci is a nervous parent.

00:43He spent the last four years building a revolutionary new computer

00:47and it's about to face its biggest test

00:49in front of an audience of millions.

00:51Hello, my name is Watson.

00:54I hope we will have a good game today, but first I have to test my voice.

01:01When the cameras roll, the computer, called Watson, will make history

01:05as it competes on the popular quiz show, Jeopardy.

01:08Jeopardy!

01:09It's frightening, right? I mean, it's a different experience.

01:12It's a very different experience for a scientist to sit here and, you know, have this happen live.

01:16Six.

01:17Lots.

01:18Four.

01:19Four.

01:21One else.

01:22Five.

01:24One.

01:25Two.

01:26Two.

01:27Two.

01:28Two.

01:29Two.

01:30Two.

01:31One.

01:32famous for winning 74 consecutive games.

01:37There's some contestants' pride.

01:39I want to beat my human competition, but you know,

01:41as a species, I would like mankind to beat the big bad computer.

01:45Let's finish leaders of World War II.

01:48This big bad computer is the culmination of four years' intensive work.

01:54IBM has put Watson through hundreds of practice games

01:57with a stand-in host and real contestants.

02:00After Germany invaded the Netherlands,

02:02this queen, her family, and cabinet fled to London.

02:05Maria?

02:06Who is Beatrix?

02:07No.

02:08Watson?

02:09Who is Wilhelmina?

02:11That is correct.

02:13It's a human standing there with their carbon and water

02:16versus the computer with all of its silicon

02:19and its main memory and its disk.

02:21It seems like it should be easy for the computer to win

02:24with its enormous memory and processing power.

02:27But the human brain makes an intimidating opponent,

02:31especially on Jeopardy.

02:33Jeopardy questions are tricky.

02:34They have puns in them.

02:35They have little jokes in them.

02:37Just understanding the question is a pretty big deal.

02:42This trusted friend was the first non-dairy-powdered creamer.

02:46Watson?

02:47What is milk?

02:48What is milk?

02:49No.

02:50Maria?

02:51What is coffee, mate?

02:54Humans communicate very fluently in, you know, natural language,

02:57and that's where computers struggle dramatically, right?

03:00Now, Watson is on the verge of conquering that challenge.

03:04Here we go.

03:05A garment worn by a child perhaps aboard an operatic ship.

03:09Watson?

03:10What is hena for?

03:11Yes.

03:12How did you get that?

03:13So, we're going to be able to hear about...

03:15If Watson wins on Jeopardy, it will be a major breakthrough

03:19in a quest that's gone on for decades.

03:21The audacious dream to build a machine as smart as a person.

03:25The quest for artificial intelligence.

03:28AI.

03:29I have electrodes.

03:30My brain is bigger than yours.

03:39When we started doing AI, the goal was why can't we build a person?

03:44We all know how to make people.

03:45That's easy.

03:46What if we could build one out of silicon?

03:48Pioneers drew their inspiration from the world of science fiction.

03:52As a child, Isaac Asimov turns up.

03:55So, here I'm an adolescent and the robots, intelligent machines are part of my life.

04:03At your service.

04:05Let's see about getting them built.

04:10In the early days, computers grew rapidly more powerful,

04:13quickly mastering complex equations.

04:16The first programs we wrote at MIT solved problems

04:20that only very educated people could solve,

04:23like problems in calculus and then algebra.

04:28The computer pioneers thought they were on a fast track

04:31to building human-like intelligence.

04:33I confidently expect that within 10 or 15 years,

04:36we will find emerging from the laboratories

04:38something not too far from the robot at science fiction fame.

04:42In the beginning, we thought, well, maybe 10 or 15 years,

04:46and we'll have something that's really smart.

04:48In the beginning, people really were amazed at how much computers could do.

04:53When you see something is improving very fast,

04:56you simply assume it will continue improving that fast indefinitely.

05:00In the 60s, confidence was so high,

05:03it inspired one of the most iconic film characters of all time.

05:07Hello, Hal, do you read me? Do you read me, Hal?

05:10Affirmative, Dave. I read you.

05:13When I was a kid, I saw 2001 A Space Odyssey,

05:17and Hal was just the best thing ever.

05:19Open the pod bay doors, please, Hal.

05:22I'm sorry, Dave. I'm afraid I can't do that.

05:26You know, it was a murdering psychopath.

05:29Hal?

05:32Hal?

05:33But it was intelligent, could talk to people,

05:36could see people, could lip-read, could do all this stuff.

05:39A machine that could do that.

05:41And I'd never even seen a real computer at that time.

05:43I was mesmerized.

05:44I was instantly mesmerized by the character Hal.

05:49There's this one part in the movie

05:51where one of the actors in the movie is sketching quietly,

05:55and Hal asks him...

05:57Have you been doing some more work?

05:59A few sketches.

06:00May I see them?

06:01He says, oh, I'm sketching.

06:02And he says, can you show me the sketch?

06:04And he holds it up.

06:05Sure.

06:06Can you hold it a bit closer?

06:08That's when I got goosebumps,

06:09because that's such a human thing to say.

06:11I think you've improved a great deal.

06:13And I said, this is wonderful, Hal.

06:15What does it take?

06:16What does it take to build something like this?

06:18Bishop takes Knight's Pawn.

06:20More than 40 years after the creation of Hal,

06:23no one has answered that question.

06:25No real computer or robot

06:27has been able to interact with humans

06:29as seamlessly as Hollywood imagines.

06:32Tell me, what is love?

06:36The problem is our own human computer,

06:39the brain, a complex entity

06:41that's defied any attempts at replication.

06:44We just had no idea how sophisticated the brain was.

06:47The computer has always been king

06:49when it comes to calculation

06:51and processing huge amounts of data.

06:53Pen is in the pen.

06:54What's the middle letter?

06:55E.

06:56Excellent.

06:57But simple skills that humans master early in life,

07:00like understanding language

07:02or recognizing objects,

07:04continue to baffle researchers.

07:06You know, people vastly misjudged

07:09how subtle we are when we're intelligent.

07:13People just hugely underestimated that.

07:18But the dream of building a computer

07:20that could talk and match wits with humans

07:22never really died.

07:24And a few years ago,

07:25a new plan was hatched,

07:27sparked by an unlikely event.

07:29This is Jeopardy!

07:32In Wagner's operas,

07:33this eldest Valkyrie is stereotypically dressed

07:36in a horned helmet and breastplate.

07:39Ken.

07:40Who's Bernhilde?

07:41Correct.

07:42In 2004,

07:44Ken Jennings' 74-game winning streak

07:46on Jeopardy! set the country abuzz

07:48and caught the eye of an IBM executive

07:50while out to dinner.

07:52All of a sudden,

07:53the entire restaurant cleared out

07:55to the bar that I'm sitting at

07:56to go see Ken Jennings.

07:58This grand old Opry comedy star

08:00used to wear a straw hat

08:01with the $1.98 price tag still attached.

08:04Ken.

08:05Who's Minnie Pearl?

08:06Minnie Pearl.

08:07Howdy.

08:08Charles Licko wondered

08:10if a computer could ever play

08:11as well as Ken Jennings.

08:13So he pitched the idea

08:14to some of IBM's top scientists.

08:16For the ones that knew Jeopardy!

08:18they said, Charles,

08:19that's just too hard.

08:20I think the prevailing view was,

08:22you know, these questions

08:23were difficult to understand,

08:25difficult to even comprehend

08:26what was being asked.

08:27Yeah, I was like, no way.

08:28I was like, no way.

08:29But one researcher,

08:31Dave Ferrucci, was intrigued.

08:34My view was, maybe this isn't

08:36as completely impossible

08:37as we think it is.

08:39And now...

08:40For over 40 years,

08:41Jeopardy! has been

08:42pop culture's IQ test.

08:44Clues are given as answers,

08:46and contestants have to respond

08:47in the form of a question.

08:49Mother's 1,600.

08:50It's a larger vessel that guards

08:52and supplies smaller ones.

08:54Christina.

08:55Was it Mothership?

08:56Mothership, yes.

08:58The show Central Concede

08:59is a little syntactic reversal

09:00whereby they give you an answer

09:02and you supply a question.

09:03You don't say, George Washington.

09:04You say, who is George Washington?

09:06To win,

09:07contestants need to be

09:08human encyclopedias.

09:10Because it's essentially

09:11everything under the sun.

09:12First in this round.

09:13You know the categories

09:14at the same second Alex

09:15tells the folks at home

09:16the category.

09:17One, you have to have

09:18a broad knowledge

09:20because we have

09:2113 categories on each show.

09:23Kate, start.

09:24PH for 400.

09:26For the record,

09:27Thomas Edison invented

09:28the first practical one

09:29of these in 1877.

09:30Contestants also have to be fast.

09:32Ariel.

09:33What is the phonograph?

09:34Good.

09:35They typically have

09:36three seconds or less

09:37to come up with an answer.

09:38The mortar and pestle

09:40is a symbol of this profession.

09:42Ariel.

09:43What is a pharmacist?

09:44Pharmacist is right.

09:45To compete on Jeopardy,

09:46IBM's computer must have

09:48an enormous knowledge base

09:49because it will not be

09:50connected to the internet.

09:52But the far bigger challenge

09:54for the machine

09:55will be understanding clues

09:57which can be extremely

09:58convoluted or obscure.

10:00You'll find this flower

10:01before Pickle Bottom

10:03in a line of handbags

10:04and bedding.

10:07And that would be Petunia.

10:09Back to you, Ariel.

10:11There will be a lot of puns.

10:12There will be double meanings.

10:14And these were things

10:15that computers historically

10:16are terrible at.

10:17Human language is a minefield

10:19for computers.

10:20Consider this sentence.

10:22How does it go?

10:24I shot an elephant

10:25wearing my pajamas.

10:27Was I wearing the pajamas?

10:29Was the elephant

10:30wearing the pajamas?

10:31Right?

10:32So there's different interpretations,

10:33different ways to parse the sentence.

10:34The word shot, you know,

10:35what's really going on there?

10:37There's already ambiguity in there.

10:39It could actually be shooting.

10:41Sort of a hunting shooting.

10:43If I'm a photographer

10:45and I'm immersed in that context,

10:47I may interpret that

10:48as shooting with a camera.

10:52Which one did I mean?

10:53You have to look at the context.

10:54But a computer has no context.

10:57It's just an electronic brain in a box.

11:00In 2006, Ferrucci tackles this challenge

11:04along with the best and brightest programmers

11:06from IBM

11:07and the country's top AI labs.

11:10The way we solve this is actually...

11:12To start, they run a test.

11:14We had an existing

11:16state-of-the-art system

11:17that people had worked on

11:18for a number of years

11:19and we tried applying that

11:20to the Jeopardy challenge.

11:21They feed one of IBM's

11:22most sophisticated computer programs

11:24hundreds of Jeopardy questions.

11:26Like this one.

11:27In 1698, this comet discoverer

11:30took a ship called

11:31the Paramore Pink

11:32on the first purely scientific sea voyage.

11:34The correct answer is

11:36who is Edmund Haley?

11:37The computer says

11:38who is Peter Sellers.

11:41The computer ran a search

11:42through a million documents

11:43looking for key words from the clue.

11:46It homed in on a description

11:47of one of the Pink Panther films

11:49in which one character

11:50was a paramour or mistress.

11:53The star of the movie?

11:54Peter Sellers.

12:01It's probably the last answer

12:02a human would come up with.

12:04But it's typical for computers.

12:07The team has a long way to go.

12:10Just how far becomes clear

12:12when they compare the computer

12:13to the best human players.

12:15They create a graph

12:16called the cloud.

12:18Each dot represents

12:20a Jeopardy! champion's performance.

12:22Jennings is at the top.

12:24What you see is a cloud

12:26that's around,

12:27they answer around 50%,

12:28the winners do,

12:29and they get around 90% correct.

12:31And where is the computer in this cloud?

12:34If we ask it to answer

12:35all the questions,

12:36it would be getting

12:3710% of the questions right.

12:38You can't go on Jeopardy! like that.

12:39I mean, the best humans

12:40are 90%,

12:4192%.

12:42We weren't even close.

12:43To win at Jeopardy!

12:46the team will need

12:47a whole new way

12:48to tackle human language.

12:50One that takes advantage

12:51of the computer's basic strengths.

12:55At its electronic core,

12:57a computer speaks

12:58a very simple language,

12:59binary code,

13:00on or off.

13:03But with that simple code,

13:05it can follow instructions

13:06and solve complex problems

13:08once reserved

13:09for intellectual giants.

13:11It used to be the case

13:14that intelligence was chess,

13:16right?

13:17If you can play chess,

13:18that's intelligence.

13:19Computers have mastered the game.

13:21Chess is easy for computers

13:23because the rules

13:24are very well-defined

13:25and very clear.

13:26The rules of chess

13:28are relatively simple.

13:29A board of 64 squares.

13:31Each piece,

13:33pawn, knight, queen,

13:35can move a certain way.

13:37And there's a single goal.

13:39Take out your opponent's king.

13:44For humans,

13:45it is the ultimate game

13:46of strategy.

13:49The way computers

13:50play chess

13:51is not at all the way

13:52people play chess.

13:54We humans

13:55look at the board

13:57and have conceptual ideas

13:59like control the center,

14:00attack on the right.

14:02Very different

14:03from the way computers

14:04play chess.

14:05A chess-playing computer

14:08looks at virtually

14:09every possible move

14:10it could make

14:11and every response.

14:13Every way

14:14the game could play out.

14:15Computers play chess

14:16through searching

14:17a tree of moves

14:18down to a very deep level,

14:20looking ahead

14:21on every possible path.

14:22But they do it

14:23by brute force.

14:24They go 20,

14:2530, 40 moves ahead

14:26and seeing all the bad things

14:28that can happen.

14:29A person can't look

14:30that many moves ahead broadly.

14:32This is the power behind

14:35the most famous chess game

14:36in the history of AI.

14:38When in 1997

14:39another IBM computer

14:41named Deep Blue

14:42beat the reigning world champion

14:44Gary Kasparov.

14:45The chess world champion

14:47walked away from the match

14:49never looking back

14:50at the computer

14:51that just beat him.

14:52The victory makes Deep Blue

14:55look pretty smart.

14:56But is it?

14:58Deep Blue.

14:59Deep Blue.

15:01It's only acting

15:02as if it's intelligent.

15:03It's not really intelligent

15:05in the way

15:06that we humans are.

15:07It's good at one thing.

15:08It's playing chess.

15:09It can't do anything else.

15:10There's no other

15:11understanding in the world.

15:12It's just about chess moves.

15:14This lack of understanding

15:16has hampered every computer

15:17program that's tackled

15:18human language.

15:20A perfect example

15:21is a program

15:22from the 1960s

15:24called ELISA.

15:25ELISA was one

15:26of the first programs

15:27that had anything resembling

15:29human conversation.

15:31It was a dialogue.

15:34You type things in

15:35and type things back.

15:36How do you do?

15:39Please tell me your problem.

15:41I'm feeling sad.

15:46Then it types back.

15:49Did you come to me

15:51because you're feeling sad?

15:53ELISA was programmed

15:55to respond like a psychiatrist.

15:57But it had no real insight.

15:59Instead, it followed simple rules

16:02and rearranged key phrases.

16:04So if I say,

16:05I'm...

16:06dead.

16:08It responds,

16:11do you enjoy being dead?

16:13It doesn't have any understanding

16:15that dead is a different

16:16kind of condition.

16:17It really is just doing

16:18this sort of fill-in-the-blanks

16:19kind of pattern magic.

16:20Anyone who tried to solve

16:22the language problem

16:23hit the same brick wall.

16:25The computer's profound ignorance

16:27of what we take for granted

16:28every day.

16:29There's just so much more

16:31that we know

16:32that we don't know we know.

16:33I mean, just we know

16:34all kinds of stuff.

16:35Like, you press the up button

16:36in the elevator,

16:37that means it's going to go up.

16:38Or milk is white.

16:40Or water is wet.

16:41I mean, there's just stuff

16:42that we know

16:43that we don't even realize we know.

16:45That's one of the things

16:46that makes it hard.

16:47All the common sense knowledge

16:50a human brain collects naturally

16:52seems much too complex

16:53to program into a computer.

16:55But that hasn't stopped

16:56one scientist from trying.

16:58So we have actually manually

17:02entered about 6 million rules.

17:04That's about 3% of what it's

17:07going to need to know

17:08in terms of actually spanning

17:10what you and I would call

17:11human common sense.

17:12For the last 25 years,

17:16Doug Linat has been leading

17:17a team trying to create

17:18human-like intelligence

17:20by teaching a computer

17:21common sense, rule by rule.

17:24The program is called Psych.

17:27And at headquarters,

17:28the walls are covered

17:29with logic diagrams.

17:31In a way, the magic of this,

17:34the power of this,

17:35is if you just tell it each rule

17:38one by one by one,

17:39and you give it general

17:41logical reasoning capabilities,

17:43that's all you need to do.

17:45So far, Psych has 6 million rules

17:48and can answer a lot

17:49of common sense questions.

17:51Like this one.

17:52Can a can, can, can.

18:02Psych says no,

18:03and it explains why.

18:05Here, it's essentially saying

18:07the reason why

18:08cans can't can-can

18:10is that cans are inanimate objects,

18:12and it knows that can-can dancing

18:14requires at least partially

18:16having a brain and using it.

18:19It's just enough for Psych

18:21to get the right answer

18:22for the right reason.

18:2625 years ago,

18:27many experts considered

18:28rules and logic

18:29the best hope

18:30for building

18:31artificial intelligence.

18:32But it's become clear

18:33these alone are not enough.

18:36It's not just a matter

18:37of piling in more and more stuff.

18:39There are basic principles

18:40that we didn't understand.

18:42Putting in more and more stuff

18:43doesn't get you basic principles.

18:46At IBM,

18:47as Dave Ferrucci

18:48and his team

18:49tackle the Jeopardy challenge,

18:50they know that facts and rules

18:52are just the beginning.

18:54We couldn't write rules

18:55for every combination of word

18:57and phrases and context.

18:59They need a new system,

19:01more fluid and flexible,

19:02to navigate the twists and turns

19:04of many different kinds

19:05of Jeopardy questions.

19:07They name their system

19:09Watson,

19:10after IBM founder Thomas Watson.

19:13The electronic Watson

19:14consists of 2,800 processors.

19:17That's like 6,000

19:19high-end home computers.

19:21Altogether,

19:22it's the size of 10 refrigerators.

19:24The team starts filling his memory banks

19:27with about 10 million documents,

19:29most downloaded from the internet.

19:31Because when Watson plays Jeopardy,

19:33he must stand alone,

19:34just like his human competitors.

19:37All kinds of content, okay,

19:39encyclopedias, dictionaries,

19:40thesauri, books, plays,

19:42you name it.

19:43The entire World Book Encyclopedia,

19:45Wikipedia,

19:46the Internet Movie Database,

19:48much of the New York Times Archive,

19:50and the Bible

19:51are just some of Watson's resources.

19:53And to build on Watson's foundation

19:57of data and rules,

19:59the team turns to a powerful tool

20:01in the computing world.

20:04It's called machine learning.

20:08Machine learning is just like

20:10human learning from examples.

20:12Before, people would just write rules,

20:15write rules by hand.

20:17Nowadays,

20:18it's all based on examples.

20:21To understand how machine learning works,

20:24consider for a moment

20:25the letter A.

20:27What if you had to describe it

20:28to a computer?

20:29It's a real problem

20:31faced by the U.S. Postal Service,

20:32whose computers must decipher

20:34all kinds of addresses,

20:35printed and handwritten.

20:37We all know what an A looks like.

20:39I know when I see it,

20:40but there's just way too many

20:42different types of A's.

20:44There are fonts where the A is just

20:46a triangle pointing up.

20:48That's an A.

20:50Pretty quickly you realize,

20:51there is no simple set of rules

20:53that you can write down currently

20:55for a program to determine

20:57whether a letter is an A or not.

20:59Humans might not be able to come up

21:01with the rules that reliably

21:02identify all kinds of A's,

21:04but it turns out a computer

21:06can do it for itself

21:08if you give it enough examples.

21:10The way you do it is you just

21:12get an A, send it to the program

21:14and say, that's an A.

21:16Here's another A, different one.

21:17That's an A.

21:18Here's another A.

21:19It's a different one.

21:20That's an A.

21:21Then you would give it another example

21:22and you would give it another example

21:23and you would do that a million times.

21:25The computer hunts for patterns

21:27among all those examples

21:29and it finds them.

21:31So the next time it meets a letter A,

21:33even one it hasn't seen before,

21:35it will recognize it.

21:37This is machine learning

21:40and it's a crucial element

21:41of Watson's programming.

21:43The team trains Watson.

21:45But here, instead of letters,

21:47the examples are tens of thousands

21:49of old Jeopardy! questions,

21:50along with a cheat sheet

21:52of all the correct answers.

21:54Using machine learning,

21:56Watson will hunt for patterns

21:57between the type of question,

21:59the correct answer,

22:01and the kinds of evidence

22:02that support that answer.

22:04Now we do this over thousands of questions.

22:06So we come up with some way

22:07to weigh the evidence on average

22:09so that we come up with the right answer.

22:11Now, when he's faced

22:12with a brand new question,

22:13Watson uses what he'd learned

22:15from these patterns

22:16and declares his confidence

22:17in each possible answer.

22:19In the end, we get a list

22:20that says here's the top answer

22:22and we're 75% sure it's right.

22:25Watson has now become

22:26a complex architecture of rules,

22:29raw data, and machine learning

22:31that enables him to use statistics

22:33to choose the right answer.

22:35To test out this system,

22:37the team scours the halls

22:38for IBM employees

22:40who can play Jeopardy!

22:43And everyone squeezes in

22:45to a conference room.

22:46In 1978, New Jersey Monthly reporter

22:50Steven Levy famously found

22:52this man's brain.

22:54Watson.

22:55What is Einstein's?

22:56Albert Einstein's, yes.

22:58The Fifth Amendment says

23:01that private property

23:03shall not be taken for public use

23:05without this.

23:07Watson.

23:08What is just compensation?

23:10Yes.

23:11With this new system,

23:13Watson surges into the winner's cloud.

23:16We took a huge jump

23:17with machine learning.

23:18Watson with a commanding lead,

23:1924,863.

23:22We saw a huge jump in performance

23:23and we were like,

23:24whoo!

23:25Up to now,

23:26appearing on the TV show

23:27has only been a dream.

23:28But Watson is performing so well,

23:30Dave Ferrucci decides,

23:32it's time to call Jeopardy!

23:34call Jeopardy!

23:40In December 2009,

23:42Jeopardy! producers arrive at IBM

23:44to size up Dave Ferrucci's new creation.

23:47Like any human contestant,

23:49Watson must audition

23:50to earn his spot on the show.

23:52You spent all this time, you know,

23:54developing this system

23:55and pushing its capabilities.

23:57And then here you are,

23:58sitting here,

23:59all the executives are there.

24:00You hear computer,

24:04you think,

24:05well of course the computer

24:06should have all the answers.

24:08You hear about Q&A technology,

24:10well isn't this just a big search engine?

24:13And they're waiting to see,

24:15you know,

24:16what really is going to happen

24:17and you just don't know.

24:18You don't know.

24:20To impress the executives,

24:22IBM builds a makeshift studio,

24:24hires comedian Todd Crane

24:26to act as game show host,

24:28and brings in former TV contestants.

24:30That was one of the tensest days

24:32I've ever had.

24:34Because we had never seen it

24:35play against Jeopardy! players.

24:37Select again, Dave.

24:38And I remember like the day before,

24:40you know, we're tuning everything.

24:41You know, I was putting in

24:42the best strategy that we had.

24:43I was putting in the best stuff

24:44that we had.

24:45And I thought,

24:46well this is just going to kill them.

24:47Miranda.

24:48What is the Cats in the Cradle?

24:50That's correct.

24:51Is this I Am the Walrus?

24:52Yes.

24:53What is Crocodile Rock?

24:54Yes.

24:55They were just like professional athletes.

24:57It was a really tough few games for us.

25:00In the first round,

25:01it seems that Watson is auditioning

25:03not for a game show,

25:04but a sitcom.

25:06Where do we go next?

25:07L underscore underscore underscore underscore

25:10for one pound.

25:11There are suddenly unexpected bugs

25:14that need fixing.

25:15We weren't dealing with Roman numerals well.

25:17So it'll say like Henry V,

25:18we would say Henry V.

25:20In 1682,

25:21he came to the throne at the age of 10,

25:23along with his weak-minded half-brother,

25:26Ivan V.

25:27Watson?

25:29What is Peter?

25:31More specific?

25:32What is Peter I?

25:35No.

25:37Hmm.

25:38Carrie or David?

25:39Carrie.

25:40Is Peter the Great?

25:41That is correct.

25:42The final Jeopardy,

25:43where contestants must place bets

25:44and write down the answer.

25:46Things only get worse.

25:47No.

25:48Under the category flags,

25:50the clue is,

25:51in a policy begun in 2002

25:53as a symbol of the war on terrorism,

25:55U.S. Navy ships fly the 18th century flag

25:58with this four-word motto.

26:00You know a little bit something about 18th century flags.

26:05David, let's see if you did.

26:07What is the four-word motto we're looking for, David?

26:10What is,

26:11don't tread on me?

26:12That is correct.

26:13Let's see if Watson got it right.

26:16What is September 11th?

26:19Watson didn't recognize the word motto,

26:21and after scanning through millions of documents,

26:23he found the word terrorism

26:25associated with September 11th so frequently.

26:28That seemed like the best answer.

26:29By the time they break for lunch,

26:32it's humans two,

26:34Watson zero.

26:35And it's not clear

26:36if Watson will ever be ready for primetime.

26:39This was taking a risk for me

26:41in the sense that you're sitting here and saying,

26:43you know what, I think this is possible,

26:44and then you fall flat on your face,

26:46and people say,

26:47well, we're never going to believe Ferrucci again.

26:49Did I expect to get fired?

26:50No, but maybe.

26:53But after lunch,

26:54the producers are treated

26:56to a different side of Watson.

26:58Watson.

26:59We came back,

27:00and the third game was neck and neck,

27:02incredibly competitive.

27:04In act three of an 1846 Verdi opera,

27:07this scourge of God is stabbed to death

27:10by his lover,

27:11Ota Bella.

27:12Watson?

27:13What is Attila?

27:15Be more specific?

27:16What is Attila the Hunt?

27:18Thank you very much.

27:19Attila the Hunt.

27:20I'll take that.

27:21That afternoon,

27:22Watson climbs back in the game.

27:23Wordsworth said they soar,

27:25but never roam.

27:26This Brit...

27:27Watson?

27:28What is Skylark?

27:29That is correct.

27:30It's a device clamped to the wheel of a parked car

27:33with overdue tickets.

27:34That is correct.

27:35Watson?

27:36What is Boot?

27:37Be more specific?

27:39What is Denver Boot?

27:41That is correct.

27:42This African-American folklore laborer.

27:45Before I let that steam drill beat me down,

27:47I'll die with my hammer in my hand.

27:49Watson?

27:50What is John Henry?

27:51That is correct.

27:52Select again, Watson.

27:53It may appear that Watson has redeemed himself,

27:57but the producers are troubled by his erratic performance.

28:01Their verdict?

28:02Watson isn't strong enough for Jeopardy!

28:04At least, not yet.

28:08Why is Watson so erratic?

28:11To understand his weaknesses,

28:13you have to appreciate the complexity of the task.

28:19Consider this clue.

28:21Keanu Reeves had a Nokia phone,

28:24but it took a landline to slip in and out of this,

28:27the title of a 1999 sci-fi flick.

28:30The correct response is,

28:32what is the Matrix?

28:34But how can Watson figure that out?

28:36First, he breaks down the clue into grammatical parts,

28:40identifying key words and phrases.

28:42Then, Watson's powerful search engines

28:45churn through millions of documents,

28:47including the Internet Movie Database.

28:50What we do next is we take these documents

28:52and we pull out candidate and answers.

28:54And we'll pull out, okay, Keanu Reeves.

28:56That could be a candidate.

28:57We'll pull out Nokia.

28:59We'll pull out The Matrix.

29:00Other movies starring Keanu Reeves

29:02also become possible answers.

29:04We'll pull out The Matrix 2.

29:06We'll pull out Speed,

29:07Bill and Ted's Excellent Adventure,

29:08all this stuff.

29:09Whoa!

29:10And Watson pulls out other famous sci-fi flicks,

29:13like Blade Runner,

29:15and it generates hundreds of possible answers.

29:17With hundreds of choices,

29:19how can Watson pick the one answer that's correct?

29:22The next thing that Watson is going to do

29:24is going to take those answers and say,

29:25well, let's assume all of them might be right.

29:28So these are its competing hypotheses.

29:31Watson starts considering evidence

29:33for and against each candidate,

29:35using rules like a movie is sometimes called a flick.

29:39And we'll look at things like,

29:41well, it's looking for a flick.

29:43Is this candidate answer a flick?

29:45Is The Matrix a flick?

29:46Yes.

29:47Is Speed a flick?

29:48Yes.

29:49Is Keanu Reeves a flick?

29:51No.

29:52So we're starting to learn something.

29:53Within a matter of milliseconds,

29:55Watson analyzes every possible answer

29:58in hundreds of different ways

30:00and scores each piece of evidence

30:02behind every answer in the list.

30:04That's a lot of scores.

30:07The problem is you have all these different scorers

30:09and they don't agree.

30:10You know, some of the scorers are going to say

30:11The Matrix is the right answer.

30:12Some of the scorers are going to say

30:13Keanu Reeves is the right answer.

30:14Some are going to think

30:15Matrix 2 is the right answer.

30:16And a lot of scorers think

30:17Blade Runner is the right answer

30:19because it shows up so often as a sci-fi flick.

30:22So you need someone at the end

30:24to listen to all these different votes

30:26and decide what's going to be the best answer.

30:28This is where Watson's machine learning kicks in.

30:32Having studied thousands of other Jeopardy! questions

30:35and their correct answers,

30:36Watson has learned what evidence is important

30:39and what's not.

30:40What machine learning will start to do

30:42is learn how to weigh them differently

30:44and say, hey, questions like this,

30:45calling on a phone, not calling on a phone,

30:47not so important.

30:48This other stuff, do I have a sci-fi movie?

30:51Is the person named, the character in that movie?

30:53Very, very important for questions like this.

30:55In this case, he successfully weighs the evidence

30:58and identifies sci-fi flicks from 1999

31:01starring Keanu Reeves.

31:03So he picks the one answer

31:04matching all those elements.

31:06The Matrix.

31:15Watson's elaborate system doesn't always work.

31:18But without machine learning,

31:19he wouldn't stand a chance.

31:23Machine learning isn't just important.

31:25for Watson.

31:26It's driving a revolution in computing.

31:28It plays a major role in computer models

31:31that predict the weather days in advance.

31:34And all those recommendations you get

31:36from Amazon or Netflix?

31:38No human is writing up rules

31:40about your likes and dislikes.

31:42Instead, computers are comparing your preferences

31:45to millions of other customers

31:47and finding patterns and learning about you.

31:50Today, machine learning is conquering many problems

31:55once thought too complex for computers,

31:58like speech recognition.

32:00In the earlier days, people decided

32:03that they would try to program computers

32:05to recognize speech.

32:07Which word am I saying now?

32:09Pick up the big block at the right side.

32:11In the 60s, this voice-directed block world

32:15was the height of technology.

32:17Computers could be programmed

32:19to recognize the audio signals

32:21of specific words and phrases.

32:23Pick up every small block.

32:25But they had to be reprogrammed

32:27for every new speaker,

32:29because everyone's speech is slightly different.

32:31Even though it's very easy for you and I

32:33to recognize the word ice cube...

32:35Ice cube.

32:37It's very difficult for us

32:38to write down the rules

32:39that would allow a computer

32:40to look at the microphone signal

32:42and see that it's ice cube.

32:44But now, computers are trained

32:46with millions of examples of human speech.

32:49Here's a microphone signal,

32:50and this is the word ice cube.

32:52Ice cube.

32:53Here's another one.

32:54Ice cube.

32:55You end up with much more successful

32:57speech recognition systems.

32:59Today, speech recognition software,

33:02though not perfect,

33:03is remarkably accurate

33:05and getting better all the time.

33:08All the ones we have today

33:10are based on machine learning

33:12simply because it works the best.

33:14And some programs

33:15are taking it a step further.

33:17Where do you come from?

33:18Not only transcribing your speech,

33:20but translating it

33:22into a foreign language as well.

33:24Shanghai.

33:25It's very nice to meet you.

33:27Very nice to meet you.

33:29Every language has so much ambiguity,

33:32double meanings and metaphors.

33:34What are your specials today?

33:36Accurate computer translation

33:37seemed impossible

33:38with the old rules-based approach.

33:40Today's special is roast beef fried rice.

33:48All of these interactions are so complex

33:50that you couldn't in your lifetime

33:52write all these rules.

33:53It's just too enormous,

33:55too daunting an effort.

33:57My wife asked me to buy some crackers.

34:00What kind of crackers?

34:03What kind of crackers?

34:05Rice crackers.

34:08Alex Wavell fed a computer

34:10millions of examples of English texts,

34:12together with their translations,

34:14into about a dozen different languages.

34:17Now, he's got a program

34:19that can run on your phone or iPad.

34:21Do you often go shopping here?

34:23And translates on the go.

34:29Machine learning has been so successful,

34:31mastering more and more tasks

34:33once only done well by humans.

34:35The computer program translates

34:37between languages.

34:39Some researchers believe it may be

34:41a crucial building block

34:43for making real artificial intelligence.

34:46There are two ways of building intelligence.

34:49You either know how to write down the recipe

34:52or you let it grow itself.

34:56And it's pretty clear that we don't know

34:58how to write down the recipe.

35:00Machine learning is all about

35:02giving it the capability to grow itself.

35:04Some people find the idea of a machine

35:07that can learn threatening.

35:10But when Watson's having a bad day,

35:12it's hard to imagine him threatening anyone.

35:19Did you see the Daily Double?

35:20Yeah.

35:21So we were way ahead.

35:22We were almost locking out.

35:23Another Daily Double!

35:25We get the Daily Double.

35:26There's like two or three clues left.

35:28And so what do we do?

35:29We bet big so we can lock them out.

35:30I'll wager $5,200.

35:33Well...

35:35The team is nervous.

35:36They know Watson will never make the cut on Jeopardy!

35:39Unless he can stop making dumb mistakes.

35:41Here's your clue.

35:42It was about letters and it was...

35:44A woman wrote to this artist.

35:47In the late 40s,

35:49a mother wrote to this artist

35:51that his picture, number nine,

35:53looked like some of her son's finger painting.

35:55We answer with Rembrandt.

35:56Who is Rembrandt?

35:58Ah!

35:59Really?

36:00Although Watson recognizes most dates,

36:03he doesn't know that the 40s refer to the 1940s.

36:06And Michael...

36:07He's a 40s artist.

36:08We answer with Rembrandt.

36:09It was a...

36:10It was a time...

36:11It was a time problem.

36:12We all know that it was Jackson Pollock.

36:14Watson loses that game.

36:16We got the Double wrong and found the difference.

36:19And his opponents show him no mercy.

36:21Both beat him.

36:22Good for you!

36:23Humans!

36:24Woo!

36:25Sometimes the reason something is the right answer

36:28is very obvious to a human.

36:30Like, for example,

36:31it may be asking for a she or a he.

36:33This first lady was born Thelma Catherine Ryan

36:36on March 16, 1912 in Nevada.

36:39Watson?

36:40Who is Richard Nixon?

36:42Oh, here you go.

36:45Right, Patricia?

36:46Who is Pat Nixon?

36:48That is correct.

36:49Richard Nixon was never a first lady.

36:52I want to understand what...

36:54Our new gender stuff is not in the system.

36:57It's not in the system.

36:58To my knowledge,

37:00people take offense at being called the wrong gender.

37:02Watson doesn't care about stuff like that.

37:04It's making statistical judgments

37:06based on how different pieces of evidence

37:08have gone together in questions and answers

37:11that we've given it.

37:13The two famous comedians' noses that made impressions.

37:17What is Jimmy Durante?

37:22More specific?

37:24Sorry.

37:25All I know is what is.

37:27Jimmy Durante.

37:29Saying it slower doesn't make it right.

37:32Now, you hear how Todd Crane makes fun of the computer?

37:35And, you know, I had my kids.

37:37I had them sign the confidentiality agreement

37:39and come in and see a couple of these games.

37:41And, you know, their comment was,

37:43why does the host make fun of Watson, Daddy?

37:46This what-are-you-doing website's name

37:49also refers to a type of nervous laugh.

37:51Watson?

37:52What is evil laugh?

37:54Oh, no.

37:56In terms of comedy duos,

37:59he is the best straight man in the business.

38:01You're gonna kick yourself Twitter.

38:03Because he doesn't, he doesn't get it.

38:05He doesn't get why his inappropriate answer is funny.

38:09And that's, there's, you can't,

38:10you can't ask for better writing than that.

38:13Once in a while, okay, we could all take a joke,

38:15but over and over again.

38:16And Watson's defenseless, right?

38:19So he's, he's making fun of and criticizing

38:22a defenseless computer that represents people

38:24with real feelings, real families.

38:26Or maybe if I don't have any feelings,

38:28my kids have feelings.

38:30And feelings are intensified

38:32by the fact there's just one more month

38:34before the Jeopardy producers will return

38:36to make a decision.

38:37Watson, you have control.

38:39The pressure is on to boost Watson's strengths

38:42and eliminate his weaknesses.

38:44His strengths are obvious.

38:46He dominates when it comes to purely factual questions.

38:48Watson usually does very well at these factoid questions,

38:52looking up facts, you know, history, geography, entertainment.

38:57Clark Gable was happy to see him come in

38:59and finish directing Gone with the Wind.

39:01Watson?

39:02Who is Victor Fleming?

39:04That is correct.

39:05Watson, you choose again.

39:06For 1600, please.

39:08His Westerns include Wagon Master and Ford Apache.

39:12Watson?

39:13Who is John Ford?

39:14Good for 1600.

39:15Who is Mike Nichols?

39:17Good for 12.

39:18Last clue on the board for $800.

39:20Here it is.

39:21Revenge of, in 1978, was the fifth in this series of comedies

39:26directed by Blake Edwards.

39:28Watson?

39:29What is the Pink Panther?

39:31I just want to check.

39:32Your buzzers are working.

39:33Just wanted to check in and wake you from your nap.

39:36Watson has made it into the clown,

39:38but nowhere near championship level.

39:41Ferrucci must come up with a plan to somehow step up his performance

39:46before the Jeopardy producers return.

39:49The team has spent almost four years,

39:53enormous intellectual and emotional energy,

39:56and tens of millions of dollars on developing Watson.

39:59But all this effort isn't just for a machine that can play Jeopardy.

40:03IBM has much bigger goals.

40:06I'm already looking sort of beyond Jeopardy.

40:09I'm thinking, where can we go from here?

40:11Ferrucci imagines a day when Watson might perform much like the computer in Star Trek.

40:16The captain just starts talking to the computer and they say, computer.

40:20Computer.

40:21Ready.

40:22Could a storm of such magnitude cause a power surge in the transporter circuits?

40:27It's an information-seeking tool that's capable of understanding your question.

40:31Creating a momentary, interdimensional contact with a parallel universe.

40:36And dialoguing with you to make sure that you get what you want.

40:41Do you have one of those?

40:43I don't have one of those, right?

40:46You don't need to be a starship commander to find this helpful.

40:51In a world overflowing with data,

40:53intelligent expert systems that can answer vital questions have been the holy grail.

40:58Now, we could be close to a Watson MD.

41:02Think of it this way.

41:03There's a bunch of information, new diagnosis, new treatment options, new discoveries.

41:08Can anybody keep that all in their head all at once?

41:11A machine that could access and organize all that information could help doctors analyze symptoms and wade through piles of medical journals.

41:19It changes the paradigm in which we work with computers. That's the vision.

41:24How am I doing? I hope we will have a good game today, but first I have to test my voice to make...

41:32Before that vision can be fulfilled, before Watson can even compete on Jeopardy!

41:36The team needs to get more bugs out of his system.

41:39One of Watson's most embarrassing weaknesses is he cannot hear.

41:44Instead, Watson receives each Jeopardy! clue as an electronic text message at the same moment his competitors see it on the board.

41:52As a result, he doesn't know the other contestants' answers.

41:56Only the female of this equine pest of the family, Tabinidae, feeds on blood. The male feeds on nectar.

42:04Bill?

42:05What's a mosquito?

42:06No.

42:07Watson?

42:08What is mosquito?

42:10No.

42:11Harvey?

42:12What's a horsefly?

42:13Yes, thank you for not saying mosquito. Good job. Good for $2,000, Harvey.

42:17Ferrucci and the team have been working furiously to boost Watson's performance. As part of their plan, Watson will now receive correct answers electronically after their reveal.

42:28If the fix works, it will be in the nick of time.

42:32The Jeopardy! producers are back, and they're about to determine Watson's fate.

42:38You know, this was a measure of our progress, and we wanted to hear, yeah, you're there, you've made it.

42:43The new and improved Watson gets his first big test with a category called Celebrations of the Month.

42:49Administrative Professionals Day and National CPA's Goof-Off Day. Watson?

42:55What is holiday?

42:56No, that's not even close, really.

42:58Watson fails because he doesn't get that in this category the answer must be the name of a month, something his human competition quickly figures out.

43:07Arthur?

43:08What is April?

43:09What is April? April 18th is national.

43:11I don't understand why we're, we don't understand the question, I don't understand what, we don't understand the category, basically.

43:16D-Day anniversary and Magna Carta Day.

43:18But now, he electronically receives these correct responses.

43:22Arthur?

43:23What is June?

43:24Good for 200.

43:25Can he learn from the answers?

43:27National...

43:28What Watson does here is it sees that, well, all the answers I've seen have been the month in which this thing in the clue occurs.

43:33Matt?

43:34What is November?

43:35Good for four.

43:36So then it knows in the next clue to look for, um, what month does this thing occur in?

43:42Uh, celebrations for six.

43:44Celebrations for six.

43:45National Teacher Day and Kentucky Derby Day.

43:49Watson?

43:50What is May?

43:51Yes, we got it.

43:52Very nicely done, Watson.

43:54He figured it out.

43:55He took us four.

43:56There you go.

43:57Thanks to the team's efforts, Watson is soaring higher in the cloud, and now approaching the level of champions like Ken Jennings.

44:08Watson is on a roll.

44:10Choose again.

44:11You're tripping.

44:12You're 16th.

44:13Jacques Cartier found this largest island of the Hoca Laga archipelago while searching for gold.

44:19Watson?

44:20What is Montreal?

44:21Yes.

44:22Who is Zebulon Pike?

44:23Good.

44:24What is Providence?

44:25Aquarius?

44:26Texas?

44:27Flo Grommel.

44:28Watson?

44:29What is Chao?

44:30That was right and acute all at the same time.

44:33The Jeopardy executives have seen enough.

44:36I think we've gone from impressed to blown away.

44:40Very nicely done, Watson.

44:42Finally, Watson will get his chance on Jeopardy.

44:47A computer playing against human champions.

44:50They say it can think.

44:52In a game that is a very symbol of intelligence.

44:55But can it think like a Jeopardy champion?

44:58It will be a contest the world has never seen.

45:00Good luck, Watson.

45:01But does this mean that the dream of artificial intelligence is coming true?

45:06So artificial intelligence, to me, is trying to get computers to do stuff that if people

45:13did them, you'd say, oh, they're demonstrating their people-ness.

45:17That's what makes humans humans.

45:19That's stuff they're doing.

45:20But without experience or emotion, can a computer like Watson ever learn and understand the world

45:26the way we humans do, from early childhood on?

45:30Right now, no machine can understand the meaning of a play,

45:35what it means to be King Lear or Macbeth or Hamlet.

45:40To be or not to be, that is the question.

45:50No machine can understand the parables in the Judeo-Christian Bible.

45:55All they can do is grubble through data and find regularities.

45:59But for Dave Ferrucci, that kind of understanding was never the goal.

46:04It's not going to emerge as a human.

46:06Because it doesn't connect the information to human experience, to human cognition.

46:10When you think about a great symphony, and when a human sits down with that,

46:14that music is affecting that human at an emotional level.

46:20The computer doesn't have that human experience.

46:23It doesn't have that human emotion.

46:24It's not human.

46:25It's a computer.

46:29Watson may never experience the world the way we do.

46:33But with his enormous knowledge base, his skill at interpreting language,

46:37and his ability to learn,

46:43could he actually be considered intelligent?

46:46Oh my God.

46:48It is more intelligent than the average Jeopardy player in answering Jeopardy questions.

46:54That's impressively intelligent.

46:56The time has come for Watson to take the stage, where his intelligence will be put to the ultimate test,

47:05in front of millions of Jeopardy viewers.

47:07Am I having fun?

47:08It's nerve-wracking.

47:11For the big match, Watson now has a physical presence, an avatar.

47:17My name is Watson. How now, brown cow.

47:20The team has been working right up till the end.

47:23I think I dream about Jeopardy questions now.

47:25I have nightmares about Jeopardy questions.

47:26I talk to people in the form of a question.

47:28Have they done enough?

47:30They're about to find out, as Watson meets the world's two best Jeopardy players, Brad Rudder and Ken Jennings.

47:38We've never had this caliber of a player.

47:41And there's a reason why Ken won 74 games in a row.

47:43There's a reason why Brad has never been beat by a human.

47:48The whole team has been waiting four years for this moment.

47:52I knew this was going to happen, but I never imagined quite like this.

47:55What do you say we play Jeopardy?

47:57With another stand-in host, Watson meets his opponents for an exhibition round.

48:02Kathleen Kenyon's excavation of this city mentioned in Joshua showed the walls had been repaired 17 times.

48:09Watson.

48:10What is Jericho?

48:12Correct.

48:13400, same category.

48:15This mystery author and her archaeologist hubby dug in hopes of finding the lost Syrian city of Urkesh.

48:23Watson.

48:24Who is Agatha Christie?

48:26Correct.

48:27Watching from the wings is Jeopardy's host, Alex Trebek.

48:30He doesn't get everything right, but he doesn't miss very much.

48:34Watson.

48:35Let's finish.

48:36Chicks dig me.

48:40At Mount Carmel in Israel, Dorothy Garrod was the first to find this prehistoric human skeleton outside of Europe.

48:48Ken.

48:49What is Neanderthal?

48:50You're right.

48:51By the end of this 10-minute game, man and machine are neck and neck, and no one dares predict what will happen when they face off in the looming showdown.