Skip to playerSkip to main content
JHolme
  • 20 hours ago
AI.Confidential.S01E02
Transcript
00:129-1-1, what is your emergency?
00:15I hit a bicycle that was in the road.
00:17Can you tell us they're injured at all?
00:18They are injured, they need help.
00:24A car crashes in the middle of the night.
00:31A woman is run down.
00:33What's her name?
00:38It's the first death of its kind in the world.
00:41Are you the driver?
00:44You're...
00:45A pedestrian killed by a driverless car.
00:49The car was an auto-drive.
00:51I can see it, and all of a sudden, it was just terrible.
00:54Although the vehicle was driving itself using artificial intelligence,
00:58there was a human operator behind the wheel.
01:02Overheader?
01:03Yeah.
01:03Okay.
01:04Do I need a lawyer then?
01:05I mean, I can't give legal advice.
01:08For the first time ever, the decision had to be made.
01:12Who was to blame for this crash?
01:15Was it the human or the AI?
01:21Artificial intelligence.
01:23A machine beyond the mind of man.
01:25For decades, scientists have dreamed of creating incredible machines
01:30that could talk like us, learn like us, think like us.
01:37But what we didn't imagine is the impact they would have on us.
01:43In this series, I'm exploring what happens when AI collides with human lives,
01:49unearthing stories far stranger than we could ever have imagined.
02:19I'm Professor Hannah Fry.
02:21Hi. I'm a mathematician, and I've spent my career examining the ways technology can transform our future.
02:28Here, in Phoenix, a multi-billion dollar industry has the potential to save millions of lives.
02:35That is weird.
02:39We're on the cusp of a self-driving revolution,
02:42and driverless taxis are now a reality in several countries around the world.
02:48All right, request it.
02:50You hail them on an app.
02:53Ooh, you'll have my initials on top.
02:55This is my first time in one of these.
02:58Worked with Google for years.
03:00Never been in a driverless car.
03:02Imagine.
03:03Here it is.
03:06Where's it going to stop?
03:10Where's it going to stop?
03:12There?
03:14Ooh.
03:16Ooh, that was quite well done.
03:18Yeah, look.
03:19Big Jeff on the top.
03:23No one in the car.
03:26Let's tell the ride.
03:28Go.
03:30Oh!
03:32It's quite nerve-wracking.
03:35We'll do all the driving, so please don't touch the steering wheel or pedals during your ride.
03:41It's green, but it's slowed down.
03:43Oh, it's turning right, that's why.
03:47Oh, yeah, I sort of don't trust it.
03:49I don't trust it.
03:51All the changing lanes.
03:58It's not timid, is it?
04:01The technology needed for a car to drive itself is nothing short of miraculous.
04:06The car uses a combination of sensors to understand what's going on around it.
04:14First, cameras.
04:17Great at identifying road signs, traffic lights, pedestrians and other cars.
04:22But they struggle in bad weather.
04:26So, many driverless cars also have radar.
04:30Often built into the front bumper,
04:32which sends out radio waves that bounce off objects
04:35and measure what comes back.
04:38It works well over long distances,
04:41but not so well up close.
04:44That's why some driverless cars also use LiDAR,
04:48the spinning cylinders you sometimes see on the roof and side of the car.
04:52It's like radar, but with lasers,
04:54and is able to build up a detailed 3D picture of nearby objects.
05:02Precise, but easily confused by reflective surfaces
05:06like windows and shiny buildings.
05:10This is sort of inside the computer's brain, as it were.
05:15So you're seeing a view of what it thinks is around you.
05:19The AI takes these three imperfect systems,
05:22pieces together what's actually out there,
05:25tries to predict what will happen next,
05:27and decides how the car should steer, brake and accelerate.
05:32I mean, I was writing about these things 10 years ago.
05:36They were a research project.
05:38They were understanding the environment,
05:39advancing the engineering, kind of tweaking the software.
05:43It was only very, very recently
05:44that they'd become commercially available,
05:47that just anybody could hail one.
05:53And soon they're even coming to the UK,
05:56starting with the busy streets of London.
06:01Who's good at sticking to the speed limit, I'll tell you that?
06:07I think there is actually quite a lot to look forward to
06:10from this future with driverless cars.
06:13Like, humans make terrible drivers.
06:16And these things, they're not falling asleep at the wheel,
06:19they're not drink-driving, they're not getting road rage.
06:22Like, fine, maybe they're not going to be perfect,
06:25but there is a lot of scope for roads
06:28to be much safer than they currently are.
06:30And I think that is a goal that is worth pursuing.
06:35But the story of how we got to this point
06:38is marked by tragedy and loss.
06:44There was a pedestrian walking a bicycle.
06:47Once the pedestrian got into the lane of traffic,
06:49the vehicle struck the pedestrian.
06:51It was a self-driving vehicle.
06:54It was in the autonomous mode at the time.
06:57In Arizona in 2018, for the first time,
07:01someone was struck and fatally injured
07:04by a driverless car.
07:07Rafaela Vasquez was the human backup driver
07:09in the self-driving Uber vehicle
07:11that hit and killed Elaine Herzberg.
07:15After years of avoiding the limelight,
07:17the woman at the center of the story,
07:20Rafaela Vasquez, had agreed to meet me.
07:24Hey.
07:25Hey, how are you?
07:26Good, how are you doing?
07:26Do you want to come in?
07:27Yeah, thank you.
07:28You haven't spoken on camera about this before, have you?
07:30No.
07:31How are you feeling about it?
07:32I don't know.
07:33I have a whole plethora of emotions going through me,
07:35but my biggest issue going through this
07:38was not being able to rebut anything
07:41or even defend myself.
07:43It's like getting a chance to speak for yourself.
07:44Yeah, because I've been dealing with it now
07:46for, what, seven years.
07:48So this is you?
07:50Yes, and that's to get you in and out of the buildings.
07:53Rafaela got a job at Uber in 2017,
07:56not long after the company had decided
07:58to develop their own self-driving cars.
08:01I like a lot of technology stuff.
08:04Self-driving vehicles were starting to emerge.
08:07I knew that Phoenix was becoming a hotbed for it.
08:11So I would see autonomous vehicles roaming around.
08:15I'm interested in that.
08:16So I went and applied.
08:18I passed everything instantly.
08:20So I was excited.
08:23Her job was to ride in the autonomous cars
08:26as Uber began testing their new technology
08:28on Arizona's public roads.
08:31There was a lot of engineers, coders,
08:33but everybody was super nice.
08:35And you have this chance to see the new technology
08:37from the inside?
08:38Not just see it, I'm testing it before it even comes out.
08:40I loved my job.
08:42I absolutely loved my job.
08:44When you were in the cars in those early days,
08:47how good did you think they were at driving?
08:50They were better than what I thought they were going to be.
08:52And it also depends.
08:53Like, Uber vehicles, for the longest time,
08:55had problems with overreacting
08:58with things on the side of the road.
08:59And some builds were great,
09:01but then they'd do another update
09:02to try to fix something else,
09:03and it makes something else go haywire,
09:05and then you just don't know.
09:08So much has happened to you since that, right?
09:10Yeah.
09:19During its first year of testing in Phoenix,
09:21Uber assigned two operators to every self-driving car.
09:25All right.
09:26And we're engaged.
09:28One rode in the passenger seat
09:30to track the car's performance.
09:32On the laptop, I can monitor a lot of the prediction software
09:35so I can see, like, where the car is going.
09:37Any unexpected behaviour had to be logged on a computer.
09:42The second operator sat behind the wheel
09:45and kept their eyes on the road.
09:47They were expected to take control
09:49if the AI in the car malfunctioned.
10:01But a year into testing, Uber changed the set-up.
10:05Now one operator had to watch the road
10:08and monitor the car's actions.
10:11OK, come on then.
10:14This decision to switch to a single human in the car
10:17came just a few months before Raffaella's crash.
10:21Even riding in the car, I still get nervous.
10:25Yeah.
10:25That's why I'm hesitating,
10:27because I'm just trying to not have a panic attack.
10:29I understand.
10:31OK.
10:34Oh, sorry.
10:35No, don't apologize, please.
10:37There is no pressure.
10:40Like, we do...
10:41No, I want to do this.
10:41Are you sure?
10:42Yeah.
10:43OK, here we go.
10:46I don't want my driving to unnerve you.
10:49No, I'm just worried you're going to get pulled over.
10:52Because you're not driving like a typical person.
10:55Driving like a granny?
10:57Well, you're driving like an autonomous vehicle, actually.
10:59Oh, really?
10:59Right at the speed limit.
11:02Each operator was allocated a route
11:05to which the car would drive again and again.
11:07It could be monotonous work.
11:10Like, the most boring one I hated
11:12was this one through this little neighbourhood.
11:13You come out, you go here,
11:16then this is all, like, 20 miles an hour down here,
11:19down here, and then boom.
11:20Back, and that's all it is.
11:23Round, round.
11:24Three hours.
11:25With the car driving itself?
11:26Mm-hm.
11:29Humans are not good at paying attention
11:31when things get boring.
11:33And with only one operator monitoring a repetitive route,
11:36there was a danger of getting distracted.
11:41But when it wasn't quiet on the streets,
11:43there could be a lot for one person to do.
11:48God, there's lots of students out, isn't there?
11:50Yes, and this is what we're testing.
11:53Yeah.
11:53And you can't predict what somebody's gonna do.
11:55So I would always take it out of autonomous mode
11:57during these areas.
12:00Right.
12:00That's why two people was important,
12:02but then all of a sudden they changed it.
12:06But there are several screens
12:07that you're continually having to look at.
12:10Yes.
12:10We were supposed to push buttons and enter codes
12:13any time something happened.
12:14I'm sort of trying to put something into the iPad
12:16while you're supposed to be monitoring the road.
12:19Yeah.
12:26On the night of March 18th, 2018,
12:30Raffaella was on her usual test route.
12:35The car was in autonomous mode
12:37and had been running for 19 minutes without incident.
12:48At 9.58 p.m., it turned right onto Mill Avenue.
12:57At the same time,
12:59a pedestrian began walking across Mill Avenue,
13:03pushing a bicycle by her side.
13:15Raffaella was looking down
13:16as the vehicle approached the woman.
13:20The car should have detected her.
13:24But it didn't.
13:33By the time Raffaella looked up
13:35and slammed on the brakes,
13:36it was too late.
13:38The vehicle struck the woman
13:40at 39 miles per hour.
13:43The pedestrian killed
13:45was Elaine Herzberg.
13:47She was often seen on her bike in the area.
13:52She never gave up.
13:54She always helped people.
13:56She was funny, funny, funny, funny.
13:58If you were her friend,
13:59you felt true love from her.
14:01She didn't deserve what happened to her.
14:13So this is the park?
14:14That's the park.
14:14Right.
14:15I did that venue, that theater.
14:17Everybody crossed there,
14:19including homeless people,
14:20because homeless people would come up here to the park.
14:22That's where she was going.
14:27This was the first time Raffaella
14:29had returned to the site of the crash.
14:36Do you see that sign on that post, that light?
14:39Yeah.
14:40Yeah.
14:40She was over there.
14:41Oh, gosh.
14:44And she's just screaming the whole time.
14:54The screaming, it was terrible to hear it.
15:01And then, but then what was worse is when it stopped.
15:05And then the ambulance showed up.
15:08And then they said she's passed away.
15:10And then I lost it.
15:14Somebody died.
15:23Following the collision,
15:25the police launched a criminal investigation
15:27into the artificial intelligence car
15:42building a system that can drive a car involves much more
15:46than turning a wheel and pressing pedals.
15:49You also need to teach it to do things that humans do instinctively,
15:53like spotting pedestrians and recognising road signs.
15:58We do that without even thinking.
16:00But training a machine to do it has proved incredibly difficult.
16:05So, back in the early days of AI,
16:08the only real form of intelligence that we knew about
16:10was human intelligence.
16:12And so, people look to the human brain
16:15for some inspiration about how to build an electronic brain.
16:21And the thing about the human brain
16:22is that it is made up by these billions and billions of neurons
16:27that are connected together.
16:29And as you think,
16:30you are essentially sending these little electrical impulses,
16:33these little bursts that can be big or small
16:36through this network in your brain.
16:40In the 1950s,
16:42people started trying to construct
16:43a much simpler computer version,
16:46where a network of artificial neurons
16:48would pass signals between each other.
16:50This concept became what we now call a neural network.
16:56But rather than all of your neurons
16:59being kind of intermeshed together,
17:01they appear in layers,
17:03like a sort of hierarchy.
17:08Decades later,
17:09people realised you could use these neural networks
17:12to recognise images,
17:14like a stop sign.
17:16Here's a simplified version of how it works.
17:20Each neuron in the hierarchy has its own job.
17:24At the bottom,
17:25they're just looking at a single pixel each.
17:27And as you go further up the network,
17:29things get more sophisticated.
17:32Maybe there'll be a neuron up there
17:34that is checking to see if there's some red.
17:38Maybe there's another one up there
17:39that's checking to see if there's an octagonal shape
17:42that stands out from the background behind it.
17:45And all of this information,
17:46all of these signals get sent up.
17:48So you eventually get right to the very top
17:51to the big boss
17:54who makes a decision
17:55based on all of the information
17:57that has flowed through the network
17:58to finally decide
17:59whether it thinks it's a stop sign,
18:01yes or no.
18:05The extraordinary thing about these networks
18:07is that if you only show it one picture,
18:10you'll just be guessing
18:12whether it's a stop sign or not.
18:13But show it thousands
18:15and tell it when it's right or wrong,
18:18and the AI learns to recognize the sign itself,
18:22adjusting its network every time it makes a mistake.
18:27In the neural networks you'll find in a car,
18:30they'll be classifying not just stop signs,
18:32but pedestrians, vehicles,
18:35lampposts, road markings.
18:36They are gigantic and gigantically complex,
18:41but the principle is still the same.
18:44This is a machine built through trial and error.
18:55But if errors happen in the real world,
18:59on our roads,
19:00the consequences can be fatal.
19:22In April 2019,
19:25a Tesla Model S failed to stop at a stop sign
19:28and plowed through a T-junction.
19:31Who else is involved?
19:33For what I understand,
19:34he was driving the car.
19:36All right, sir.
19:37I'm driving.
19:38I dropped my phone and looked down,
19:39and I ran the stop sign and hit the guy's car.
19:41So, which car are you driving?
19:43This car, Tesla, right here.
19:44The driver of the Tesla,
19:4642-year-old George McGee,
19:48had been on his phone when it fell out of his hand.
19:52He bent down to pick it up,
19:54leaving Tesla's autopilot to drive the car.
19:57Can I get back?
19:58Can I get back?
19:59Let's get back.
19:59But something went wrong.
20:02Boss?
20:03The Tesla hit 26-year-old Dylan Angulo's truck
20:07at 62 miles an hour.
20:10Did you stop at the stop sign?
20:11No, I didn't, sir.
20:12I don't think.
20:12I honestly don't know.
20:13I looked down.
20:14I didn't know how close I was to the intersection.
20:16And I was driving on a cruise,
20:18going through it,
20:18and then I looked down,
20:19and to get the phone,
20:20I dropped and I reached down,
20:22and I didn't see it.
20:24What do you do?
20:25I manage a private equity fund out of Boca.
20:28I'll explain the Tesla.
20:29Yes, sir.
20:32Remarkably,
20:33Dylan survived the accident.
20:39He got ejected.
20:41Shit.
20:43But he wasn't alone that night.
20:46What do you think?
20:47He's a little serious there.
20:49OK, wait a minute, though.
20:50There's ladies' flip-flops.
20:52Yeah, but...
20:53There was a pair of ladies' flip-flops.
20:56Please tell me this.
20:58Get back.
20:58Shit, I'm sorry, sir.
21:00Baiters, flip-flops.
21:11This picture was actually the day of the crash.
21:18And, uh...
21:21We were going fishing,
21:22and we stopped to get bait at the bait store.
21:27At the time of the accident,
21:28Dylan had been with his new girlfriend,
21:3022-year-old Nybele Benavidez.
21:34Is he her?
21:37No.
21:38The impact from the Tesla killed Nybele instantly.
21:43She's so gorgeous.
21:45Nybele always had this peace and happiness to her.
21:49And just being around her would just...
21:52It would rub off on you, you know?
21:55I was going to meet her mom the next day, you know?
21:57We were going to catch the fish,
21:59and then...
22:00I was going to cook lunch for her mom the next day.
22:03And, uh...
22:04Was that the first time you were going to meet her mom?
22:05Yeah, it was going to be the first time.
22:07Mm-hmm.
22:08And, unfortunately, you know,
22:10the first time that I have to meet her mom
22:12were under these circumstances.
22:19Yeah.
22:22Oh, my gosh.
22:24I'm so sorry this happened to you.
22:31I'm so sorry this happened to you.
22:35And I finally get my hands on the police body cam video.
22:42And in that police body cam video,
22:44the driver, yeah, he's like,
22:45I was driving, I was on the phone,
22:47I had the car on autopilot cruise.
22:51I start to do research,
22:53and I had no idea this existed, you know?
22:58This thing called autopilot,
23:00where the cars can drive themselves, you know?
23:03And, um...
23:06And right then and there, I was like,
23:09this guy was relying on this car to drive itself.
23:14This is why this happened to us.
23:28Autopilot is Tesla's advanced driver assist function,
23:32highlighted in their slick promotional videos
23:35that sell a vision of technological sophistication,
23:38safety, and convenience,
23:40showing that it can steer, brake, and change lanes
23:43on real roads in the real world.
23:47Tesla car next year will probably be 90% capable of autopilot.
23:53Like, so 90% of your miles could be on auto.
23:55Other car companies will follow.
23:57Elon Musk has even tweeted
23:59that his cars can completely drive themselves.
24:03But they can't.
24:05The driver's manual says a human
24:07still has to be in full control of the car.
24:10This thing is psychotic.
24:12Oh, my God.
24:14In our city, yeah, it's turning.
24:15I'm not involved in this.
24:15Watch the road.
24:16What do you think, Autopilot?
24:17Am I on the roadside?
24:18Oh, Jesus, that was scary.
24:19Wait, it's a double yellow line.
24:21Am I on the right side of the road?
24:22No, you are not.
24:23Get the fuck over, guys.
24:25Yes.
24:26Look, don't fucking trust this thing.
24:29And there's serious safety concerns
24:31over the autopilot feature.
24:35This Tesla crashed into a highway divider in California,
24:39killing the driver.
24:40Another slammed into a parked fire truck in Utah.
24:43Both had the autopilot feature on.
24:51This is the bend right before the intersection
24:54where the accident happened.
24:56And at night, from right here,
25:00you could already see the red light blinking from right here.
25:03Oh, yeah.
25:04I see it.
25:13This is where her body ended up laying.
25:18You know, we pulled over to look at the stars.
25:22It's crazy how far her body flew, you know.
25:27That's how fast the car was going straight through this intersection.
25:32Six months after the fatal collision,
25:34Dylan, the man behind the wheel of the Tesla,
25:36George Magee, was charged with careless driving,
25:40which he didn't contest.
25:44But Dylan wanted Tesla to also be held accountable.
25:49These car manufacturers,
25:50they need to do a better job with designing these cars.
25:55We want justice.
25:56My Bell's family and I, we want justice.
26:00These cars were allowed on the road
26:01before they were ready to be on the road.
26:03But they were not safe.
26:05And they were advertised as these cars
26:07that can drive themselves.
26:15Tesla offered Dylan and Nybel's family
26:17an undisclosed sum to draw a line under the case.
26:26Tesla would now face a jury trial over its autopilot system.
26:46First at five, the video showing the moments before an Uber self-driving car crashes.
26:51Backup driver, Raffaella Vasquez looks down for approximately four seconds.
26:56Back in 2018, the fallout from Raffaella's crash had left the whole self-drive industry
27:02hanging in the balance.
27:03If there's no human driver, who bears the responsibility when a car makes a misstep?
27:09Is there enough safety built in before we deploy these vehicles?
27:12Is there any suggestion at the moment that Uber has done something wrong?
27:17We don't know.
27:20Under intense media and political scrutiny,
27:24Tempe's police investigation left no stone unturned.
27:28So the police have sent over all of their digital evidence.
27:35Um, and I have not yet watched this.
27:38Let's have a look.
27:40The actual car in the police compound being analysed.
27:45Oh my gosh, look at that.
27:47And then there's also one that says, seize phones.
27:53They're knocking on someone's door.
27:54It's the police department.
27:55What was the...
27:56Blurred everyone's faces.
28:01Hey.
28:02Hey, Raffaella.
28:04We're coming up to do follow-up regarding the accident.
28:07Is we have a seizure warrant for your cell phone.
28:10They've got a warrant for her phone.
28:13Which one, which number did you want?
28:15How many phones do you have?
28:17What?
28:18How many phones do you have?
28:19I have my work phone, and then my personal phone,
28:21but my work phone is the one I had to work.
28:24So the war is saying we need both phones.
28:25Well, we need all the phones.
28:26How many phones?
28:27All your phones.
28:30Why are they taking our phones?
28:36You're welcome.
28:44Hi.
28:45Good morning.
28:46Casey Marsland was the lead detective on the case.
28:49He agreed to show me the footage he'd obtained from Uber.
28:52So this is the footage with all three of the camera views.
28:57When I first hit play, we can see the driver looking down multiple times.
29:02Hmm.
29:05And there didn't seem to be a logical explanation as to why she was looking down.
29:10So we're coming up to the crash now, are we?
29:12Yes.
29:14And there.
29:17Jeff, can you go back a few frames in this?
29:19I just want to see what was happening immediately before.
29:21Sure.
29:24So looking down, looking down, looking down, looking down, looking down, looking down, looking
29:27down, looking up.
29:28Oh, oh, wow.
29:29Yes.
29:30What did she say she was looking at?
29:33Initially, there was a statement that it had to do with the iPad in the center of the vehicle.
29:40And does that broadly stack up?
29:42Were Uber requiring people to look at screens while they were driving?
29:46So the, yes, that was, one of the purposes of her job was to monitor the vehicle.
29:51And if there was anything abnormal, she would need to interact with that screen.
29:55I mean, the conflicting instructions, right?
29:57That like.
29:58Yes.
29:59They're supposed to be looking at screens, but simultaneously monitoring the road.
30:02It definitely presents a bit of a challenge.
30:04Yeah.
30:05However, when we, of course, looked at what the screen was actually showing at the time,
30:09it didn't show that there was any sort of alerts or any sort of interaction with the
30:13screen.
30:14We looked into the phone.
30:15We saw the apps that were installed on her personal phone.
30:19We noticed Hulu was an active app on her phone.
30:23And when we wrote a search warrant to the company, they responded with a printout of the activity
30:28on the account.
30:28And that's what gave us probably the most important information at the time.
30:33Then what did it say?
30:35So this was the information that was provided from Hulu.
30:38And it shows that the show, the voice was being streamed to her personal phone.
30:44So what was your take on seeing all of this?
30:47Essentially, this goes to the beginning of her shift.
30:49She started her shift by driving the vehicle out of the garage.
30:53And it was at that time before she even got onto the roadway that she had set up and started
30:57streaming the Hulu.
30:58And so I think that the conscious decision to provide yourself with a likely distraction
31:05before even getting onto the roadway is an absolutely reckless decision to be made.
31:22So I found, I found a couple of clips around the time and lots of them are not on your
31:30side.
31:30Some of them were terrible, especially at first.
31:32Take a look at this.
31:33This is Hulu Records.
31:35It turns out she was streaming that scene competition, The Voice, at the time when she should have been
31:41watching the road.
31:42Rafaela Vasquez was riding in autonomous mode at the time, but she was distracted and looking down for more than
31:4830% of the nearly 22 minutes before the crash.
31:53Right there.
31:54They're right.
31:55They're right.
31:55I was, I wasn't distracted.
31:57I was doing my job.
31:58I had other things to do other than operate the vehicle because we went to one person.
32:02So I had duties assigned.
32:03Like in the video, you see me look away.
32:05I am looking away.
32:06But if you time them, and even the police did, I never looked away for more than five seconds.
32:11Why?
32:11Because we were trained.
32:12We were trained to look.
32:13Boom.
32:13I've had five seconds, then look back.
32:15Boom.
32:16Over here, five seconds, look back.
32:18Boom.
32:18Our cell phones, we have to Bluetooth them into the vehicle.
32:23It wasn't watching TV.
32:24I was listening to it.
32:25But the police said I was watching.
32:27That means that's a definitive statement.
32:29You just told the public that you have evidence that I was watching something.
32:33That's bullshit because you don't.
32:35You have evidence that I was Bluetooth streaming something, but hello, streaming Bluetooth.
32:40You can listen to stuff.
32:41People do it all the time with music.
32:45Everyone automatically assumed because the police said it that I was watching Hulu.
32:51And because of that, I was negligent.
32:54Therefore, I was unable to prevent the accident.
32:56Therefore, the charge.
32:58The investigation lasted for two and a half years before Raffaella was finally charged.
33:05Where were you when the indictment came through?
33:07I was at home.
33:08My attorneys informed me.
33:10They just told me that I was going to be indicted for negligent homicide.
33:12I said, what?
33:13And they said, negligent homicide.
33:15Negligent homicide?
33:16I said, homicide?
33:18I said, so murder?
33:20That's just me how I interpret that.
33:22So now I'm like, I didn't know what to say or do.
33:28Now I'm in shock.
33:30If found guilty, Raffaella faced up to eight years in prison.
33:42With conflicting accounts from both sides, I decided to track down the official accident report.
33:52The report concluded that the crash was probably caused by Raffaella's failure to monitor the driving environment.
34:10This table is particularly interesting because this is like seeing inside the brain of what the driverless car was thinking
34:17at the time.
34:18The car actually notices that there's something there 5.6 seconds before the collision, which is plenty of time to
34:28start braking or turning.
34:30And initially it's radar that detects there's something in the distance.
34:34It predicts it's a vehicle though.
34:36And then 0.4 seconds later, the LIDAR kicks in and picks up that there's something there.
34:44The LIDAR doesn't know what it is though.
34:46It's just classified it as other, something unknown.
34:49Now the LIDAR keeps changing its mind.
34:53So a full second later, changes its mind to side to vehicle.
34:57Then 0.3 of a second later, changes its mind again.
35:01And then it keeps switching.
35:04Vehicle, other, vehicle, other.
35:06The computer within this driverless car is not connecting the dots.
35:12It's not seeing this as one object whose classification keeps changing, whose path is slowly moving over time.
35:21Instead, it is seeing this as brand new objects every single time.
35:30In fact, it's only 1.2 seconds before impact that it finally decides to settle that it's a bicycle.
35:41Obviously way too late to actually do anything about it.
35:46Terrifying.
35:47You design a system that fails to track the path of an object until it's decided what it is.
35:54If you've got something that you're going to collide with, I don't care what it is.
35:58I care about that it's going to collide.
36:05That's insane.
36:07Although the system sensed the pedestrian nearly six seconds before the impact, the system never classified her as a pedestrian
36:14or predicted correctly her goal.
36:18The system design did not include a consideration for jaywalking pedestrians.
36:26It wasn't designed to recognise people unless they were on a crosswalk.
36:36Oh, I think this is pretty damning actually.
36:41What was this thing doing on the roads?
36:51The report made me want to know more about what was happening inside Uber around the time of Raffaella's crash
36:58in 2018.
37:00After some digging, I found Robbie Miller.
37:04Robbie, hi.
37:06An operations manager who'd been at Uber at the same time as Raffaella.
37:10He had quit on safety grounds just a few days before the fatal collision.
37:16I'd been working in the self-driving space for four or five years at this point.
37:21I was running the self-driving truck fleet for Uber.
37:25Eventually, there was a move to combine the operations of the testing for self-driving cars and self-driving trucks.
37:36How did you feel about that?
37:37I was not at all comfortable with it.
37:40The self-driving cars were having significant issues.
37:42There's a lot of really scary incidences that are occurring near misses, near collisions.
37:50We would have, you know, an incident where the car was driving on the sidewalk in broad daylight.
37:58And you realise, like, they are headed on a path where someone is going to get seriously injured or worse.
38:07And so I gave notice.
38:11There's this overarching fear, I would say, at Uber that Waymo is about to release their self-driving cars.
38:21And it's, I think it's very scary for Uber's leadership to not have a response.
38:27So it's turned into a race?
38:28It is absolutely a race.
38:31Waymo, owned by Google's parent company, Alphabet, was Uber's arch rival.
38:38Just five months after Waymo launched its first public trial, Uber moved from two operators in the car to one.
38:48You need to show to your investors, hey, we're making this progress.
38:52An easy way to do that is just take someone out of the car.
38:55And this is something I mentioned.
38:57You need that second person in the vehicle.
38:59You're not ready to take that person out of the vehicle.
39:02Just a few days after I left, the crash occurred.
39:08Where did you first hear about it?
39:11I was driving and I received a phone call from one of my former co-workers at Uber.
39:22And I sat in the parking lot and cried for 15 minutes.
39:31Wow.
39:33I'm very passionate about the technology.
39:36I believe in the technology.
39:37I want the technology to succeed.
39:42But there's a way to do it.
39:53A few years after Raffaella's fatal crash, Uber sold off its self-drive division, leaving the path clear for Waymo.
40:05Here you go.
40:06Waymo's just crashed.
40:09AI technology not working.
40:13Why is this happening to me on a Monday?
40:15I'm in a Waymo car.
40:17This car may be recorded for quality assurance.
40:20This car is just going in circles.
40:22Hi, there, Mike.
40:23A scab.
40:23I'm calling from Waymo support.
40:25Yeah, I got a flight to catch.
40:27Why is this thing going in a circle?
40:28I'm getting dizzy.
40:30Waymo are now the dominant force in driverless taxis in the U.S., with two and a half thousand of
40:36them on the road.
40:42Experts have praised their safety record.
40:47But some see these cars as symbols of the powerful tech elite and their disconnection from the lives of ordinary
40:55people.
40:56Hey.
40:57Hello.
40:58And there's one group who've decided to take action.
41:02Hello there.
41:02Nice to meet you.
41:03Nice to meet you.
41:04How are you all doing?
41:05Doing well.
41:06Doing well.
41:06They call themselves the Safe Street Rebels and carry out their operations incognito.
41:12What is it about the autonomous driving that you dislike, though?
41:18They are heavily promoting themselves as the future of public transit.
41:23And we just fundamentally don't think they're the future of public transit.
41:26They're just a taxi where you don't talk to someone.
41:29And when that car is not parked, it's still driving around waiting.
41:3350% of the miles they drive, there's nobody in the vehicle.
41:36In addition, they cannot be ticketed for any kind of moving violation in the city.
41:41We have videos of them driving 40 miles an hour on the wrong side of the road.
41:44And nobody can do anything about it.
41:46They're completely immune.
41:47They're above the kind of rules that would stand for an Uber driver or a Lyft driver.
41:51Yes.
41:51And if they can't do something about it, we can.
41:55To express their frustration, the group go around San Francisco disabling Waymos.
42:03Let me understand the strategy, then.
42:05So how easy are they to override?
42:08Pretty easy.
42:09Lovely, pretty easy.
42:11We're not allowed to show you how they disable these cars.
42:14Although it's not exactly high tech.
42:17So you're not putting them out of action permanently?
42:20No, and we're not causing any damage to them either.
42:21It's painter's tape, so it doesn't leave any residue when you remove it.
42:24There's one coming.
42:26All right.
42:27There's one.
42:28There's one.
42:28Check this one.
42:30Someone go, Frank.
42:40Did you hear that?
42:40There was people on the cyborg who were, like, cheering them on.
42:45The disabled car was now blocking the road.
42:53So the one behind hasn't been taped but is stuck.
42:57There's another one coming.
42:58Oh, my gosh.
43:01Three of them.
43:04And they're just stuck.
43:06And then now there's another car behind flashing.
43:09But it has a human driver so they can go around the problem.
43:15Is it actually illegal, what you're doing?
43:17We don't think it's illegal.
43:19We can't find any law that'll break.
43:20It's not vandalism.
43:21So it's hard to point to any law we're breaking.
43:25Are there not other ways that you could do this?
43:27I mean, can you...
43:28I don't know.
43:28Is there, like, a not more sort of democratic way?
43:31It would be nice if we could vote, if we had the opportunity to vote on them,
43:34but we have not been presented with that opportunity yet.
43:37Is this...
43:37Does it feel a bit like this is something that has happened to you rather than with you?
43:41I mean, absolutely.
43:42Hey, there's one coming on the other side.
43:44Hey, other side.
43:44Other side.
43:45Other side.
43:45It's coming.
43:46Yeah.
43:49Yeah, you want to get it back?
43:56They are surprisingly easy to bully, aren't they?
44:01The other thing that I think was really surprising was just how many of them were going past, right?
44:08And maybe it's the time of day, but, I mean, hardly any of them have people inside.
44:17Yeah, it's empty.
44:23They did say something that I hadn't considered before, which is about how they are continually circling when they don't
44:29have people inside them.
44:31And that I hadn't considered, because, of course, there's a congestion aspect, but there's an environmental aspect, too, right?
44:38I mean, they've got to be powered.
44:41Hmm.
44:42Interesting.
44:44Interesting.
45:08It happened fast.
45:17In July 2023, after five years of legal wrangling, Rafaela accepted a plea deal to avoid going to jail.
45:25The agreement indicates that she wished to plead guilty to the crime of endangerment.
45:29The event is non-dangerous, non-repentant.
45:31Is that the crime that she wished to plead guilty to?
45:34Yes.
45:35She pled guilty to a reduced charge, endangerment, with the guarantee it would be lowered to the least serious category
45:43of offence following three years of probation.
45:48Hi.
45:49Hey, how are you?
45:50I wanted to meet Rafaela's lawyers, Al Morrison.
45:54How are you?
45:55And Marcy Cratter.
45:56So good to see you.
45:59Why did you take this case on?
46:01Our job is to fight for the little guy.
46:02No offence.
46:04But she was the little guy.
46:07And the more we got into this case, the more we realized just how one-sided it was.
46:12Where do you think this came from, then?
46:14I mean, is this the police, or is this from Uber?
46:18Uber.
46:19Really?
46:20They did whatever they could to make it not their fault.
46:24And it's David and Goliath.
46:27We have this single person up against a multimillion-dollar company with unlimited resources.
46:34The problem was that there were so many aspects of the way these cars were programmed that didn't account for
46:40real-world situations.
46:41The most obvious one is the thing wasn't programmed to deal with jaywalkers.
46:46To save yourself in a campus, college campus, to not program vehicles to deal with jaywalking, I think, is the
46:53height of negligence.
46:54I was scared to go to trial.
46:57If I lose, it's prison time, no fans or buts.
47:01And the media already destroyed me out there.
47:05That's always in the back of our minds as lawyers and certainly our clients' minds, that it's the exposure.
47:10What's going to happen to me if I go to trial and lose?
47:14But Marcy and I felt like we had a great case.
47:16But we also understood the risk was all hers.
47:19Me not going to trial has no reflection on them.
47:22These two saved my life.
47:24They absolutely saved my life.
47:31No criminal charges were ever brought against Uber.
47:45Rafaelio got dealt a really rough hand and this didn't play out fairly.
47:53Maybe she was watching her phone.
47:55Maybe she wasn't.
47:56Like, I...
47:58Honestly, I don't know.
47:59But I also...
48:00I don't think that that's the point of this.
48:03The scandal here is that you have got this massive, corporate, multi-billion dollar project,
48:12which is not prioritising the safety of the people who are in the cars
48:18or, like, the members of the public who haven't even agreed to participate in the experiment.
48:23It is not enough to say that all of the responsibility lies with the person who was in the car.
48:32That's not enough.
48:34That's not enough for me.
48:47Back in Miami, Dylan and Nibelle's family were taking Tesla to court
48:51over the fatal crash that had killed Nibelle.
48:55Nice to meet you, Anna.
48:56Adam, great to see you.
48:57Great to meet you.
48:59What's up, buddy?
49:03Dylan's lawyer, Adam Bumell, brought me up to speed on the case.
49:07This is a sort of situation where there's no precedent at all.
49:11Does that make it quite difficult to build a legal case when you're, I don't know, forging a new path?
49:17Extremely.
49:17I mean, we had to do it from the start.
49:19This is creating the law.
49:22Obviously, from day one, Tesla's position was this is 100% driver's fault.
49:28The thing is that it's difficult to untangle the responsibility in this case
49:32because you're not saying that the driver had no responsibility at all.
49:36Of course not.
49:37We 100% acknowledge that the driver had responsibility.
49:41And our position from day one has been this is a case of shared responsibility.
49:46Yes, the driver was at fault.
49:48He was distracted.
49:49He was disengaged from the driving task.
49:52But then you have to ask yourself, why was he disengaged from the driving task?
49:56And you realize that Tesla fostered this belief in him, this trust of the system that was unwarranted.
50:05He thought that the car would stop or swerve or do something before plowing into a parked vehicle.
50:12Do you have the data from inside the car?
50:14Do you know what the car was seeing?
50:16So we were able to finally get the data from inside the car showing what the autopilot computer was detecting
50:23and processing.
50:26The computer understood that the car was speeding towards the end of the road.
50:31It appreciated the stop sign.
50:34It appreciated the blinking red light.
50:37It appreciated Dylan's parked SUV.
50:43So it identified a lot of things and it did nothing.
50:55The car knew what was going on in front of it and didn't do nothing to warn the driver to
51:02stop the car.
51:03So this isn't a case of misinformation inside the car's system.
51:08The car has all of the information it needed in order to avoid the collision.
51:12But it was programmed not to behave in the way that the driver thought it was programmed to behave.
51:20Did the car behave as people expected and believed it should based off of the Tesla's marketing and Elon Musk's
51:28statements
51:29and kind of hyping up the capabilities of this car.
51:33When you take that backdrop and you put it against this case, the car did nothing that anybody thought it
51:41would or should.
51:42So this is almost like become a case of like misadvertising or like misrepresentation then?
51:49They make the consumers and drivers believe that the car is more capable, creating false expectations in their drivers.
51:59In the Uber crash, the blame had been laid firmly on Raffaella's shoulders.
52:07With Tesla, would this now be the first time the car itself would be considered at fault?
52:33Tonight, a Florida jury forcing Tesla to pay $243 million to victims in a deadly 2019 crash.
52:41The jury fighting flaws in Tesla's self-driving software were partly to blame.
52:46We can be proud that we stood up and that we did everything in our power to help shine light
52:53on what's going on.
52:56Against all odds, Dylan emerged victorious in court, forcing Tesla to take some responsibility
53:03for the first time ever for a crash involving its autopilot system.
53:12It was a shocking landmark verdict that could well reshape the future of self-driving cars.
53:20When they gave the verdict, how was it?
53:23It was very emotional, you know, I mean, like me and my dad, we started hugging each other
53:29and, you know, praying that we would get justice.
53:34You know, from the beginning, we knew this was a joint liability case.
53:38And the jury decided to hold Tesla accountable.
53:42And I'm grateful for that.
53:44And I'm grateful that people heard all the evidence and saw that Tesla made a mistake.
53:51I really get the impression that this has never been about money for you.
53:55It was more important for us to shine light and to show the world that this technology is not safe.
54:25Tesla planned to appeal the ruling.
54:43Meanwhile, in the UK, British firm Wave will launch their self-driving cars in London later this year.
54:54This feels like so much more high stakes than it did when I was in Phoenix.
55:00I mean, this is slightly wild for me, right?
55:02Like, I've lived in London for 20 years and we're driving in a driverless car down Camden High Street.
55:08This is honestly something I thought was much further in the future.
55:12It never gets old.
55:14Wave CEO Alex Kendall points out that the AI in driverless cars has come on leaps and bounds in the
55:21past few years.
55:22And this is interesting.
55:23This person's body language was sort of turning backwards and forwards away from the crossing.
55:27So that was noticeable, actually.
55:28The car sort of changed its mind a couple of times.
55:30Yeah, we saw that person turn their body back and forth.
55:33The AI has got quite good at learning that kind of intent.
55:36So it's like super, super fancy cruise control.
55:39It's a completely different experience from cruise control.
55:43Have I just undercut your product balance?
55:45You're comparing a floppy disk with a quantum computer.
55:51Think of AI as the next evolution of tooling.
55:55When you think about the wheel, the calculator, the computer, different tools that as humanity we've invented that push forward
56:02society.
56:03I think intelligent machines are going to be the next evolution of this.
56:13Whether you like it or not, driverless cars are coming.
56:16I mean, they're here already.
56:19And I still think there's lots of good to come from that.
56:21But to get to this point, we had to go through that difficult phase.
56:26We had to go through the learning, which, I mean, by definition, learning involves making mistakes.
56:34It's just that these mistakes, you know, these deaths, these lives that were ruined, I don't think that they were
56:41inevitable.
56:41I don't think that they were an acceptable price to pay for a new technology.
56:48I think different choices could have been made.
56:51I think there were different ways to balance the rollout of the new product with the safety of the public.
56:58And I just hope now that these are lessons from the past, you know, I hope that the steepest part
57:04of this learning curve is now behind us.
57:54The chief executive of one of the United States is long.
57:57One of the largest health insurance companies has been shot and killed in New York.
58:01The alleged killer has been identified as Luigi Mangione.
58:05The suspect had something to say about the insurance industry.
58:09The insurance companies are relying on algorithms to make decisions to deny patient care.
58:15Stop denials with AI!
58:17How many people have to die?
58:19The anger and the upset is real.
58:22But there is a human right!
58:25To discover more about AI and how it can shape our future,
58:30go to connect.open.ac.uk forward slash AI with Hannah Frye
58:36or scan the QR code on the screen now.
58:39There is a carro PC.
58:42Anyway, it's very good.
58:43Thanks, everybody.
58:43We'll see you again next time.
58:43Good day.
58:55Think about it.
59:08111Bu1.2
59:08On in the screen now.
Comments