Brainstorm AI 2023: Economic Impacts of AI and ML on the Workforce
Erik Brynjolfsson, Jerry Yang and Akiko Yamazaki Professor and Senior Fellow, Stanford Unstitute for Human-Centered AI (HAI) Shane Luke, Vice President, Product and Engineering, Head of AI and Machine Learning, Workday Atif Rafiq, Founder and CEO, Ritual.work; Author, Decision Sprint Moderator: Jeff John Roberts, Crypto Editor, FORTUNE
Category
🤖
TechTranscript
00:00 So we've spent the greater part of this afternoon
00:02 discussing the business opportunities of AI.
00:04 But obviously, with great advancement
00:06 comes disruption and concern for the future,
00:10 as well as the economic impact of this powerful technology
00:14 in the long term.
00:15 There are a whole range of questions
00:17 to answer on this subject, some of which we just
00:19 got into at the end there.
00:21 What will jobs look like in a decade?
00:23 How do we upskill and reskill the workforce
00:26 so that it's still relevant?
00:28 And how do we measure productivity?
00:31 So we're lucky to be joined by a great panel
00:33 now to discuss some of the possible answers
00:36 to these questions.
00:37 So please welcome for our next session
00:40 Eric Brynjolfsson, professor and senior fellow
00:43 at the Stanford Institute for Human-Centered AI.
00:46 And he'll be joined by Shane Luke, the vice president
00:49 of product and engineering and the head of AI and machine
00:52 learning at Workday, and Atif Rafiq, author of Decision
00:57 Sprint, a book that implores its readers to constantly
01:00 be ready for the unknown.
01:02 And Atif is also the founder and CEO
01:05 of Ritual, an application that leverages
01:07 AI to help teams collaborate on innovation and problem
01:11 solving.
01:12 And they will be interviewed by Fortune's crypto editor, Jeff
01:15 John Roberts.
01:16 [MUSIC PLAYING]
01:19 [APPLAUSE]
01:23 Hello, everyone.
01:29 Welcome to the workplace panel.
01:31 I'm going to get right into it with something
01:33 I heard Vinod Khosla, one of our keynote speakers tomorrow,
01:36 say recently.
01:38 And that was 80% of 80% of all jobs
01:42 will be replaced or seriously disrupted
01:44 by AI in the next 10 years.
01:46 And that's sort of like, oh my goodness.
01:48 So I'd just like to turn to our panel.
01:49 I sort of study this stuff closely.
01:51 Is that an exaggeration, Eric?
01:53 You know, I didn't know you were going to ask that question.
01:56 It was funny.
01:56 I had Vinod over for dinner a couple of weeks ago,
01:58 and he made that statement.
01:59 And everyone at dinner was sort of like, whoa,
02:01 is that for real?
02:02 But he made a really good case that it is going
02:05 to be something on that order.
02:08 And I agree that this is a tremendously
02:11 disruptive technology.
02:12 People have compared it on the scale of the internet,
02:15 or even electricity or beyond.
02:17 The technology is advancing very fast.
02:19 What we've seen already is impressive,
02:21 but there are things in the pipeline that are even
02:23 going to blow that away.
02:25 The bottleneck is in transferring that
02:28 into changes in business organization
02:30 and capturing the value.
02:31 And that's where we still need a lot of work.
02:34 And Atif, what do you think of that?
02:35 Is that overstated or not?
02:37 I mean, at some level, you could say only 80%, right?
02:41 But all joking aside, I mean, I think
02:43 the thing we're not talking about
02:44 is really true knowledge work in organizations.
02:47 So when you think about typical white collar
02:50 work of strategic problem solving, creativity,
02:54 and things of that nature, I think
02:57 we're woefully unprepared.
02:58 Organizations are not preparing their teams
03:01 to be able to tap into AI to do it better and smarter.
03:05 But on top of that, it's a vulnerability for the workforce
03:08 because AI, I think, is the ultimate bar raiser when
03:11 it comes to knowledge work.
03:13 For sure.
03:14 And Shane, just go to you for a sec.
03:15 I think with Workday, you have a window
03:17 into all sorts of companies.
03:19 And what do they want most?
03:21 Or what are they coming to you for?
03:22 What sort of AI guidance are you providing them?
03:24 Yeah, so you talk about the disruption.
03:27 And whatever the numbers end up being,
03:28 there's going to be changes in the workforce.
03:30 And one of the things we see with Workday
03:31 is that a big desire for companies, what they want,
03:33 is to enable their workforce to be
03:34 able to upskill and reskill.
03:36 And I think one of the important and powerful things we can do
03:39 is while AI will be disrupting many jobs,
03:42 it's also an enabler to help you upskill and reskill
03:44 to look at what the future is going to look like.
03:46 And we do a lot of that.
03:47 And we hear a lot of that from companies
03:49 who are thinking about, hey, in 10 years,
03:51 I don't even know what my workforce is
03:52 going to need to be able to do.
03:53 How do we get ahead of that problem?
03:54 And we can use AI to do it while AI is also changing
03:56 the way people work today.
03:58 Yeah, and I want to go back to what COs should do
04:00 operationally.
04:01 But I just do want to go to the future a little bit, too,
04:04 in terms of--
04:05 I think a lot of us in this room might be wondering,
04:07 how do we future-proof ourself?
04:09 And also, I think some of us here
04:11 have children in high school or going into college.
04:14 And what on earth major should they take?
04:16 You don't want to send them in to get a degree that's
04:18 going to be obsolete in two years.
04:20 So Eric, what's your advice for surviving the coming--
04:25 Well, Shane's exactly right.
04:27 These technologies are incredibly powerful.
04:29 But they can be used to augment, not necessarily replace people.
04:32 I think not enough focus is on this capability
04:35 of allowing people to do jobs better and differently.
04:38 So I don't really see any mass unemployment or mass
04:41 replacement of whole occupations.
04:44 What I see is specific tasks will be
04:47 enabled by this technology.
04:49 And the people who can look at the large-scale problem
04:52 solving, broad creativity, how to apply these technologies
04:56 in new ways and have their skills augmented
04:58 are going to do really well.
05:00 But this ultimately, I think, for most people
05:02 is going to be something that allows
05:03 them to do more and better rather than replacing
05:06 and downscaling.
05:08 So what should my kids study in college, you guys?
05:11 I think critical thinking, problem solving,
05:14 and deductive reasoning.
05:15 Because AI is actually already pretty good at that.
05:19 If you feed it a problem statement,
05:21 you're Netflix, you're concerned about password sharing.
05:24 You're Uber, you want a loyalty program.
05:25 You're McDonald's, you're thinking
05:27 about coffee subscriptions.
05:30 Today, you gather a six-person team
05:32 and you say, come back to me in a month with our strategy
05:35 and a set of recommendations and be able to defend it.
05:38 You can do a pretty good job of breaking that down
05:40 and distilling it into the key questions, the unknowns,
05:43 some key answers, and some conclusions
05:45 you can draw from that detective work.
05:47 You can do that today, probably in two days
05:50 with today's technology.
05:52 So the question, therefore, is what are the people doing?
05:55 And so we need to bring the people together
05:58 to actually maybe tap the AI as the first draft
06:01 and then make it better.
06:03 That's not what we're doing in companies today.
06:05 So I think it could be a little bit bad in the short term,
06:08 but I'm long on human contribution
06:11 to disambiguating things for companies
06:13 so that they can achieve their objectives.
06:16 Yeah, I'm still a big believer in the STEM fields.
06:19 I don't think that goes away.
06:20 You hear a lot of people talking about disruption where, hey,
06:23 because you can generate code with models,
06:24 maybe software engineering goes away.
06:26 I don't think so at all.
06:27 Software engineering creates the models.
06:28 You have to data engineer them.
06:29 There's humans touching them at every step of the process.
06:32 So it might change how much code you're writing.
06:34 That's an assist.
06:35 That's what Eric was talking about.
06:37 It's going to enable you to do more.
06:38 It doesn't go away.
06:39 You're going to be writing different kinds of code
06:41 and less of it, but you're still building the systems.
06:43 And so I think that I'm still a big believer in those areas
06:45 being really important.
06:46 Yeah, let's put in our CEO hats for a second.
06:49 I mean, think operationally.
06:50 How do you deploy resources?
06:53 Because I remember we used to have these conversations 10
06:55 years ago about starting a digital unit
06:58 to handle the internet.
06:59 And now it's just so pervasive.
07:01 It's just part of everyday operations.
07:03 And it sort of seems AI, every company's
07:05 kind of got like an AI division.
07:07 And I mean, that's what you do, Shane.
07:10 Is that going to stay that way for a while?
07:12 Or a few of the CEOs should be trying to sort of infuse it
07:15 into all aspects?
07:17 Anyone want to take that one?
07:18 Well, it's definitely affecting almost every aspect,
07:20 especially the knowledge work.
07:21 As Satish was saying, I mean, I get all these CEOs coming to me
07:24 and they're overwhelmed by all these opportunities.
07:27 And I think you need to do it in a very structured way.
07:31 In my research, we've shown how you can take any company,
07:34 break it down to its occupations that are being done
07:36 in the company.
07:37 And then each of those occupations,
07:38 you can break them down into individual tasks.
07:41 A typical occupation has about 20 to 30 distinct tasks.
07:45 And then at that level, you can start
07:47 seeing whether or not the tool can augment or help out
07:49 with things.
07:50 So when you do that, what you find
07:52 is that a typical company, there are about 18,000
07:54 distinct tasks.
07:55 And you can just run right through them
07:57 and analyze which ones have tools that will help out
08:00 with that task.
08:01 With that, a CEO can have a game plan
08:04 of where to prioritize things, what parts
08:06 are going to have the biggest effect on their overall value
08:10 add, where to prioritize the first step and then
08:13 the follow-on step.
08:14 But it makes it a much more manageable and structured
08:17 approach rather than just being overwhelmed by all
08:19 these different opportunities.
08:21 What I see over the next three to five years
08:23 is companies will be taking these technologies,
08:26 applying this task-based approach,
08:29 and getting some really enormous productivity benefits.
08:33 We saw in one call center, there was about a 35%
08:35 productivity gain.
08:36 In some of the software engineering,
08:38 they've gotten up to 100% doubling of performance.
08:42 There was just a paper about some management consultants.
08:44 They got about a 40% productivity gain
08:47 across a set of different tasks.
08:49 In each of them, there was improvement quality.
08:53 There was also an effect that the less skilled workers often
08:57 benefited even more than the more skilled workers.
08:59 Yeah.
09:00 If anyone has questions I want to go to in a second,
09:02 but I just want to follow up what you're saying here, Eric.
09:03 I mean, I think that makes sense.
09:05 Systemically approach it that way.
09:06 But where do you start?
09:07 It's like, what's the prompt?
09:09 OK, AI, I have 18,000 tasks.
09:11 Please tell me what to do next.
09:13 Or you guys want some insight into what's step one?
09:17 I like what Eric's saying.
09:18 I think you can also pair that.
09:19 I would counsel leaders to give AI or task teams
09:24 to work with AI to solve some of your most meaningful problems.
09:27 So pick one or two of the hard things--
09:29 new products, a business model change, something really huge,
09:33 and ask them to write down the problem statement.
09:37 Then ask AI to make it better, or suggest three alternatives.
09:41 And I think you'll see that AI finds tuned things,
09:44 makes things a lot more precise.
09:46 We all know that if a team spends weeks or months
09:49 or even quarters on a different understanding of the problem
09:52 than the rest of the people in the company, that's an issue.
09:55 Because it's going to lead to things getting stuck.
09:58 And it's what holds companies like Google,
10:00 despite having great talent, back because it's big
10:03 and it's become bureaucratic.
10:04 So where I would start is use AI all the way upstream
10:08 to find the problems, come up with the right questions that
10:10 should be explored.
10:11 It's very doable today.
10:13 Yeah, and I think-- so you're talking
10:15 about a top-down approach about thinking about the problems.
10:18 I think that's something you need to be doing.
10:20 There's also the bottoms-up approach
10:21 which models can help with, which
10:22 is understanding your data.
10:23 So if you rewound five or 10 years ago,
10:26 executives of companies probably weren't
10:28 understanding their data in depth
10:29 because you didn't need to.
10:30 That was somebody else's department.
10:32 These days, if your AI is really going
10:33 to be powering your business, you're
10:34 going to want to get under the hood of that.
10:36 And the nice thing is you don't need
10:37 to be super sophisticated technically anymore
10:39 because you have language models that can help you interpret
10:41 and understand your data.
10:43 So really being able to wrap your arms around that
10:44 and understand, well, I have this information.
10:47 What can I do with it?
10:48 And map that to problems that you find at the top level
10:50 is important.
10:50 And often, it's not just the structured data
10:52 that you've traditionally put in data sets,
10:53 but now we're getting to use language
10:55 and other unstructured data.
10:57 So it's unlocking a lot of tacit knowledge
10:59 that companies have.
11:00 And that call center analysis that we did,
11:03 there was just a ton of knowledge
11:04 in the transcripts and all the interactions
11:06 that people were having.
11:07 Previously, there's no way of studying that.
11:09 But with the large language models,
11:11 it was revealing the best kinds of answers, the best way
11:13 to solve problems, and making that accessible
11:16 to the whole workforce.
11:18 I want to go just more granular for one sec.
11:20 Chatting backstage, Shane, about Workday.
11:22 Apparently, there's a lot of demand
11:23 on the recruitment front, parsing resumes.
11:26 I mean, it's just so much out there,
11:28 both internal and external.
11:30 And to what degree is AI going to transform recruitment
11:34 or maybe make recruiters' jobs obsolete?
11:36 Yeah, so I don't think it makes their jobs obsolete.
11:38 I think it's going to assist them a lot.
11:40 You post jobs.
11:41 If you're a big company and an attractive employer,
11:43 you might have hundreds of applicants for jobs,
11:45 sometimes thousands, depending on what the type of role is.
11:48 And usually, that's going to mean--
11:50 without AI and without tools, that's
11:51 going to mean that a subset of the applicants get looked at.
11:54 With AI, you can immediately look at all the applicants.
11:57 You can look at everything that they have.
11:58 You can focus on the skills, both skills
12:00 that are explicitly described in a resume or a CV,
12:03 as well as those that you can infer using AI.
12:06 And so you can take a skills-first approach
12:08 of mapping between jobs and candidates using AI.
12:11 And that, to me, is an enabler for everyone.
12:14 You're not looking at things like, hey,
12:15 what school did you go to first?
12:17 What does your name sound like?
12:18 Anything like that.
12:19 What is any of your demographics?
12:21 You're really focusing on skills,
12:22 and you're doing it at scale very quickly.
12:24 So I think AI is a big enabler there.
12:25 And the recruiter and the hiring manager
12:27 are still going to be making decisions about the people,
12:29 but they're going to be able to do that over the course
12:31 of hundreds of candidates where they might have only
12:33 been able to look at a few dozen in the past.
12:35 Yeah, and this one's for anyone, but is that possibly fraught,
12:38 too?
12:38 It just seems like it could be a legal minefield.
12:40 You go ask the machine to go get you the best candidate based
12:43 on data, and it pattern matches the last--
12:45 three people at this job are white guys.
12:47 So without you knowing, it goes and zeros in on that.
12:50 How do you put guardrails to promote diversity and not
12:54 strip over and break the law?
12:55 Well, you described a classic challenge
12:57 with a lot of machine learning is
12:58 that it'll often spit back some of the same biases
13:01 that you already have in your hiring decisions.
13:03 There was a paper by a couple of my former colleagues
13:05 at MIT, Daniel Lee and Lindsay Raymond,
13:08 that looked at two different kind of AI-powered systems,
13:10 one that used more traditional machine learning,
13:12 if you can call it that.
13:14 And it did tend to just amplify some of the biases
13:16 that the company already had and hire more
13:18 of the same kind of people that had been hired in the past.
13:22 And then they also looked at a separate-- a different kind
13:24 of system that was based on reinforcement learning that
13:26 did a lot of exploring of the different kinds of people
13:29 to hire.
13:30 And that second system was able to not only
13:32 get applicants that perform better, but also
13:35 a more diverse set of people.
13:37 So it was a win by having it explore the space more
13:41 thoroughly rather than just kind of amplifying
13:43 what you were already doing.
13:45 Yeah.
13:45 And so a big factor here is how you design the system.
13:48 So when we think about doing candidate matching,
13:50 we're not looking at profile data about people
13:53 and things about them because they
13:54 are going to be highly correlated to demographics.
13:56 We're instead looking at skills from documents that they have.
13:58 And that's an important thing.
13:59 It's a design decision not to use things
14:01 like the school you went to or your professional associations
14:04 because they have high correlation to demographics.
14:06 And they could learn those patterns in a model.
14:08 And so with a brute force machine learning approach
14:11 where you just gave it everything,
14:12 you could overfit to demographic correlations.
14:14 But with a thoughtful design approach,
14:16 you can avoid doing that and actually, again,
14:17 take a skills first approach so that you're
14:19 focusing more on what skills people have,
14:21 not on who they are.
14:22 I'm not in the HR space.
14:23 But it strikes me that a lot of what's
14:26 going to happen with knowledge work
14:27 follows the pattern of software development.
14:30 So when you hire for software developers, of course,
14:32 you give them exercises, right?
14:33 Tests.
14:34 You don't look at the resume.
14:35 You don't really give a damn about the resume.
14:37 You look at the code and the quality of the work.
14:40 So why not every job having exercises
14:43 that you can interact with generated by these models
14:46 to say, what's your ability and your capability
14:49 around doing the work?
14:50 That's probably the best signal of fit for the role
14:53 than any other things.
14:55 And I think we're on the cusp of some things like that right
14:57 now.
14:57 It's a good future idea.
14:58 We might put that on the web.
14:59 Yeah, take it.
15:00 Yeah.
15:01 I know I have another question or two,
15:02 but we only have three minutes left.
15:03 Does anyone have a question out here?
15:09 Hi.
15:10 I know Mike's coming in.
15:11 Please identify yourself.
15:14 Yeah.
15:15 Hi, I'm Anita Vadavatha.
15:16 I'm head of innovation labs at Iona Ventures.
15:19 My question is, how are you incorporating synthetic data?
15:23 Because really, one way to bypass bias
15:25 is don't touch proprietary data, which
15:28 is all kinds of loopholes, particularly with the EU
15:30 Regulation Act that's kind of kicking in.
15:32 But we can almost simulate everything
15:34 through a synthetic data environment.
15:36 So how are you taking that into consideration already
15:38 at the enterprise level right now?
15:40 Synthetic data is an important area.
15:43 It's also a dicey area.
15:43 You have to make sure that when you generate the data,
15:46 it's representative of the real data
15:47 that you're actually going to be acting on.
15:49 And so it's kind of an open research area.
15:51 It's true that you can generate large data sets at scale.
15:53 It is something that's really useful,
15:55 especially if you want to generate
15:56 particular distributions or balance out distributions.
15:59 So we do it.
16:01 It's a research area.
16:02 I think it's going to end up being
16:03 one of the most prominent ones in the LLM space,
16:06 simply because the way that LLMs are being trained today
16:08 on these large corpuses that are largely
16:10 human-generated on the internet has issues with it
16:12 that you need to be able to get around or balance out.
16:14 A lot of that is done today with our LHF, which
16:17 is having humans kind of supervise
16:18 the models in the background.
16:20 But that doesn't scale very well over time.
16:21 And so I think that synthetic data
16:23 is going to be a big part of the LLM world going forward.
16:25 And it's an area that we're invested in.
16:27 Yeah.
16:30 And just to wrap up, Fortune COO Ellen Murray
16:33 flagged an interesting news item today.
16:35 One of our other partners, Microsoft,
16:36 announced they had signed an agreement with the AFL,
16:40 America's biggest labor union, to collaborate
16:43 on these things going forward.
16:44 It's just sort of encouraging.
16:46 I wonder if it's a bit kumbaya, because this thing just
16:48 feels a little disruptive.
16:50 But do you think that's a model worth pursuing,
16:52 like leaning out and trying to bring your workforce in?
16:55 Or are you just asking for trouble, to be honest here?
16:58 Oh, I think 100%, because in the end,
17:02 the good work in an organization happens at the team level.
17:05 It's not the C-suite.
17:06 It's not the CEO.
17:07 You're very distant from the customer
17:09 and the specific issues.
17:11 So what is sustainable work for teams?
17:14 I think we need to hear from a first-person standpoint,
17:17 kind of like in a product mindset of what actually makes
17:20 work sustainable for all the stakeholders,
17:23 not only the organization, which is
17:25 going to get a boon here in productivity,
17:27 but also for sustainable employment.
17:29 It comes back to what you said at the beginning.
17:31 What we heard in the previous panel with Natalie
17:33 about using the technology to augment people
17:35 rather than replace them.
17:36 And if you bring labor in and say, OK,
17:38 how can we help you do your job better?
17:40 We're not here necessarily to replace.
17:42 Or maybe there's parts of your job
17:44 that we can make easier or less efficient.
17:46 But keep the human in the loop.
17:47 Ultimately, as impressive as these systems are--
17:49 and they are very impressive--
17:51 they're still not replacements for humans in most cases.
17:54 They hallucinate.
17:55 They make mistakes.
17:56 You want to have a human not just in the loop,
17:58 but in charge.
17:59 And going into that up front and saying, look,
18:01 we want to work with professionals, with laborers,
18:05 in all different categories, and use the technology
18:07 to allow us to do new things better than we could before,
18:10 that makes it a win-win.
18:11 In the call center example that I mentioned earlier,
18:14 the productivity went way up.
18:16 Worker wages went up.
18:17 Customer satisfaction went up.
18:19 Worker turnover went down.
18:21 So it was not one group benefiting
18:23 at the expense of the others.
18:24 And it really doesn't have to be.
18:25 OK, well, this seems a very optimistic take.
18:27 And I hope you're right.
18:28 And I'll just end with another thing your friend Vinod
18:30 Khosla said, which was, if all these jobs are disrupted,
18:32 it won't really matter, because it's
18:34 going to create so much deficiency in productivity.
18:36 We'll only work if we want to.
18:38 So that sounds nice, too.
18:40 So there's the optimistic look of the future.
18:42 I'm sure people have other takes.
18:43 But thank you, everyone.
18:44 Please thank our panelists.
18:46 [BLANK_AUDIO]