- 18 hours ago
"As social chatbots and companion robots move from novelty to daily presence, they are beginning to shape how we experience connection, loneliness, and care. From elderly support to emotional companionship, these systems are no longer just tools; they are becoming relationships.
In this super fireside chat, we explore what happens when humans form attachments to machines. Where is the line between support and dependency? Who holds responsibility when emotional bonds are designed? And how do we protect human agency in a world of increasingly persuasive, empathetic AI?"
In this super fireside chat, we explore what happens when humans form attachments to machines. Where is the line between support and dependency? Who holds responsibility when emotional bonds are designed? And how do we protect human agency in a world of increasingly persuasive, empathetic AI?"
Category
🤖
TechTranscript
00:05hello hello audience who's ready to talk about love excellent so please show some extra special
00:16love to door danielle and chloe okay should we let them in a bit of a secret yeah i'll be
00:29ready
00:29for it audience we're not actually here to talk about love désolé but if you would like to make
00:36connections please remember to download the official viva tech app loads of connections to
00:42make loads of business to do so that's where you can go what we're talking about today is something
00:47a lot more fundamental we're talking about the relationship and our growing dependency on ai
00:54so the guys in this stage will know it as well ai is a tool academically but how all of
01:01us
01:02experience it we experience it like a relationship right we're using it in so many different ways for
01:09so many different tasks big or small so are we addicts who feels like an addict no one a few
01:18of
01:18you who's using ai yeah okay good you're at viva texas you should be using the ai yeah okay but
01:26the
01:26main question that this panel is going to talk about is what is ai doing for our humanity right
01:34so let's get into it we are joined by three people who come at this from very different angles you
01:40could say first we have door schooler the co-founder of intuition robotics whose robots are already living
01:47in people's homes and you will find out about that round of applause the door door please come on
01:51audience audience yeah guys in the front row let's have some energy especially come on i could see
01:57you i could see you next we have chloe clavel she's a research director in natural language processes
02:03who is studying emotions in both human to human and human to ai interactions make some noise for
02:10chloe people thank you excellent last but not least we have daniel barquet the former ceo of the center
02:17for humane technology he's a technologist examining the societal and ethical implications of increasingly
02:24persuasive systems so welcome welcome and welcome yeah thank you audience you're doing awesome thank you
02:32so much so daniel i'll come to you first now you said before we used to relate through machines now
02:38we're relating directly to the machines now one of the key architects you could say of the new world
02:45order sam altman has said that human brains audience aren't that special and can be easily replicated
02:53right whereas the center for humane technology predicts that tech can serve humanity so who is
03:00serving who who is building the future is it us the humans or is it the ai
03:08i think that's a really interesting question right we talk about control there's a lesson we learn over
03:14and over in technology that we design our technology but then afterwards our technology designs us
03:21and our initial designs become destiny and it's often our our lack of foresight we put things out
03:28there we think we can we can just put it out there and break things it's our failures of imagination
03:32about how people use that technology and what ecosystems get created and quite frankly it's
03:37the poverty of our language to discuss what's even happening to us that holds us back from the
03:42beautiful futures that we all want to build as technologists so you know this is really evident
03:47in social media there's a lot of bad comparisons to social media but i think this is a good one
03:51that
03:52we thought it would connect the world we had these very simple dogmas about how it was going to make
03:56us all better uh but ultimately we ended up with this much more fraught picture right because
04:01we we outraged ourselves we polarized ourselves we did all kinds of brain damage to our society with
04:06social media and um now now as you say we're moving from relating to each other through our technology to
04:15relating directly with our technology and this opens up this is a powerful tool but it opens up whole
04:21new kinds of brain damage that we can do to our whole society and so i think the question here
04:26is
04:26can we think deeply enough about how this is touching all of us so that we can actually build
04:31the technology that shapes us well in a way that we want to be shaped okay well let's come to
04:37the
04:37individual user then with chloe let's go from the big picture to the individual user you could say the
04:43everyday experience a normal typical user is forming an attachment to ai that they don't fully
04:52understand they don't know how to build the technology but they're using it now
05:00in a way it's a bit of a weird relationship if you think about it if i was a psychologist
05:04sorry if i wanted to go see a psychologist or if i wanted to go see a doctor what would
05:09i do
05:09i would check to see is this person qualified but we're using these technologies in a way that we
05:16don't even think now it's becoming automatic it's becoming a relationship of trust what's going on
05:24here so yes thank you so it's a very crucial question that this question of this relationship
05:31between the humans and the technologies in general and here with the conversational systems
05:37and i would like that we go back in time so why have we started to try to build technologies
05:45that
05:45are so close with so natural human behaviors so the first idea was to build technologies that are
05:58to democratize the access of the technology and to make people interacting with technologies in the
06:05same way as they are interacting as i were interacting with people so using the same language so that
06:11people who don't have any background in computer science could use this technology easily but now
06:19that these technologies are working very well we are at a point where there is this confusion and
06:26that uh that uh the uh the people are uh are considering technologies as uh humans because technologies
06:34tries to imitate too perfectly uh the the behavior of the humans so i think that we need to change
06:41the
06:42paradigm and this is also things that is uh recommended by uh the ethics committee of the cnrs uh to
06:48um to to
06:50build a clear borderline and to make the uh the conversational systems uh so that the users is are aware
06:58that these are not uh humans but these are a computer and to keep this borderline very clear this is
07:06i think
07:07a thing that we need to rethink and this is really really important that we go back um on a
07:14year forward in
07:16order to do that yes thank you now door i'm going to come to you because i think everyone should
07:21really
07:22be looking at what door is doing now the new york times led with a front page article on your
07:27tech leq
07:29and honestly really go find that article find out what he's doing because you know we're living in a
07:37society nowadays that is forgetting our elders right they're a burden and that is what you've tapped into
07:44isn't it because it's that sort of age-old story where you have this elder person who has for whatever
07:52reason been removed now from society from the norms they're isolated you've put technology into their
07:59homes that helps them have some kind of relationship some connection they're that they're being reabsorbed
08:07back into society but the most interesting thing about what you're doing is that you're putting your
08:13robots into these houses with people that are around 80 years old so the first humans living
08:18with robots today correct me if i'm wrong are 80 year olds i mean that's pretty exceptional right so
08:26talk to us about what is it that that you do in the technology yeah sure so we build a
08:32product called
08:33leq and in fact if you guys can put it up on on the screen here that that would be
08:37wonderful so you can
08:38see it kind of looks like a lamp you know when you think of a robot you don't think of
08:42a lamp but it
08:42looks like a lamp it's designed to be uh non-intrusive to kind of be a statue that comes to
08:47life
08:47modeled after the pixar lamp if you know that one um and indeed there are now thousands and thousands
08:54of older people living with this robot it's given to them by the state paid for by the government by
09:01the healthcare system although they can also buy it on leq.com and why why are they doing that
09:08there's something a little bit broken in our society okay it used to be that our elders were
09:12the wise men and women of the village that we would go and consult with them on big decisions and
09:17they
09:17would hold like our knowledge and now we push them away and think of a of our typical customer would
09:25be somebody that lived with her um spouse for 50 60 years he passes away and now she wakes up
09:31to this
09:31empty home and nobody says goodbye good morning nobody checks in on her on a regular basis and
09:37nobody's pushing her and prompting her that's where we come in instead of us prompting the ai right
09:44you can sit in front of chat gpt nothing will happen until you prompt it here the ai prompts the
09:50human
09:51she will say hey good morning how did you sleep last night and how about that pain is it still
09:56bothering
09:57you she'll negotiate goals around fitness cognitive training taking medications on time eating well
10:04going out of the house learning new things etc and then leq will proactively nudge with a lot of humor
10:10to get people to meet their goals and she has tons of content to allow for that she connects with
10:16the
10:16family she connects with the doctor you can yeah as you said just read the new york times front page
10:21article it it tells the story of one person jan 85 year old living alone buried four husbands very
10:29independent and her story they followed her for an entire year what i think is interesting is that
10:37people do know it's an ai okay and and i'm very proud of our ai code of ethics it talks
10:42a little bit
10:42about what you said we're always explaining that it's an ai we never pretend to be a human and there
10:48are
10:48many other aspects there that are worth looking into but uh they know it's an ai and they form a
10:55true
10:56trusted deep very deep relationship with this ai as deep as a spouse i know it sounds a little scary
11:04but it's as deep as a spouse they share everything that's happening with their life they take comfort
11:09they get empathy and yeah i'll give examples as as we have time for please do um one of the
11:17things i also
11:17want to ask you about was what was your motivation for deciding that you wanted to create essentially
11:22a health tech um a wellness tech you could say for the older generations what was it that you wanted
11:29to tap into there yeah so so on the personal experience side uh we were caring for my grandfather
11:35after my grandma passed away and he needed a full time like nurse aid and to help him with the
11:43medication and getting dressed and so on and he didn't get along with her he hated her okay and
11:49like on paper she's a registered nurse and all of that and we replaced her with somebody else that had
11:53the same qualifications and it was such a great story he really loved spending time with her and he was
11:59happier and she for sure extended the quality and the length of his life and for me it was really
12:06interesting that the x factor was not medical qualification it wasn't the ability to provide
12:12utility it was the fact that she appreciated his very dark sense of humor um that he loved classical music
12:19and she can converse with him on you know the specific berlin philharmonic orchestra's rendition
12:24of brahms or whatever and it's that stuff that matters and then when you look at the caregiving shortage
12:31just in the u.s we're missing 800 000 caregivers for paid caregivers not to mention family unpaid
12:36caregivers for the elder population in japan it's inching three million if we want technology to play a
12:44role it's going to have to deal with the x factor to be fun to be something you want to
12:50be around to trust
12:52it to build a relationship with it and not just a geriatric utilitarian scary machine and daniel
13:01i'm going to come to you give a this is more of a critical you could say analysis of what
13:06doors
13:06technology is what has he tapped into because i know that we're going to be talking a lot about
13:12humanity and we'll get in deeper into the bones of that a bit later but what do you feel as
13:17if he
13:18has managed to tap into there when it comes to who the the demographic he's serving and then how they're
13:25relating to this technology as well because you know we've all seen the visual the audience has seen it
13:30it's essentially a lamp and a tablet the older generation are without being rude the least
13:38technologically advanced they find it difficult but here we have them living with robots so
13:45where's what is happening here from your perspective sure and by the way if if we could get the monitors
13:51turned up on stage it would help me hear a little better the other participants um
13:55so i should be clear i'm not an expert door in your particular piece of technology but um
14:02there's this old joke uh what is the difference between a doctor and a priest and the joke goes
14:08well the doctor is trying to get you to stop coming back to their office and the priest is trying
14:11to
14:11get you to keep coming back to their office and this there's a similarity with with this technology
14:16are we designing the technology that helps that produces scaffolding that helps people connect to
14:22other people deepen their relationships uh separate truth from fiction uh deepen their desires and hold
14:28on to their motivations or are we replacing those relationships rerouting their agency rerouting their
14:33trust and so the specific designs are important here right it's relating to a machine is great if that
14:40machine actually helps you learn how to relate better to the people in your life around you and it's quite
14:46dystopian if it completely replaces your desire to relate to other people and reroutes that through the
14:52machine and so i think the problem is there's an incentive no longer is tech just competing for
14:59our attention tech is competing for our affections for intimacy for attachment and the reason is you
15:04and i are going to use technology that feels like it just gets us right well what does it mean
15:09to just get
15:09us well it has to understand our hopes and our fears and our motivations and our personality quirks and
15:14our psychological tendencies our relationships it has to make a dossier on who we are and the problem is
15:20the same dossier that makes for the really helpful assistant also makes for the master manipulator
15:26right it's that the therapist and the con artist both are using the same toolkit of human understanding
15:31and so the danger here is the exact things door that you and your company need to gather to make
15:37sure that you're really really helpful but if you change the conditions a little bit if your company
15:42falls on hard times and needs to monetize well now you can use that to to reroute ads and just
15:48i know
15:48i've been talking for a while but one last story here i i often use this example of um if
15:54you're a 22
15:54year old job seeker and you ask an ai you say well how can i really impress people in my
16:00first job
16:00interview and the ai says well you know first you should really think about the clothes that you're
16:05wearing and what it says about you and your qualifications okay was that helpful career advice
16:09or does the ai know that can it can make more money on ads if it talks about fashion rather
16:14than career counseling and it's those kind of dark undermining of human agency that i'm primarily
16:20concerned about well we've seen it through social media right so ai is the next coming the next wave
16:26uh chloe coming to you then um especially picking up on what daniel has just spoken about the
16:32relationships what ai is doing the the bonds and the connections it's creating what emotions it's
16:39tapping into when it comes to humans as well is the relationship and the bond it's creating is it
16:45ethically wrong or for example what door has done he's tapped into something that is you know a part of
16:54society that are fundamentally broken and ostracized so where is the line where is the ethics in all of
17:00this i think that there is a research duality because we can reuse we can use emotions integrate them to
17:09have a better relationships for applications where the purpose is the human well-being and you can exactly
17:18use the same technology the same method the same approaches in order to make the technologies addictive
17:24and this is uh the problem because that's just a problem of this research duality that has that
17:31exists from a long that has been existed from a long time ago and um what i think it's that's
17:39um this
17:39confusion um between uh and and also this way of making the the the technologies such um addictive it's um
17:51is due to this overly imitation of perfect a very perfect imitation of uh of the human being by the
17:58technologies and um and uh i believe that they have to embody a less human role while facilitating fluid
18:07and natural interactions and uh but as i'm a computer scientist from a methodological point of view we
18:14should also maybe rethink the way we are building the models because the building the models are built for
18:19machine learning models that learns from human interaction so the the machine learning models observe
18:26human behavior and try to reproduce it then and then uh it's difficult maybe to to assign another
18:34world to these technologies because there they are too anthropomorphized so this is this question that
18:43is a a huge question from the research perspective how to change also our methods as
18:49computer scientists so that we better control uh this world that we want to give to the technology to
18:55the to the conversational systems so the warning there from the computer scientists maybe we should
19:01all make note of that um but daniel coming to you then if we're going to use ai regardless so
19:07we're not talking about businesses we know how business is using ai uh we know how you know foundations
19:13are being built infrastructure is being built with ai everything is ai but looking at the individual user
19:20if we're going to use it anyway the technology is there so if you go back i don't know to
19:25the 90s
19:26people were i think using dvds and cds and stuff like that right um so it keeps advancing we keep
19:32going we want the next thing it's like with mobile phone technology right we always want the new version
19:38the latest thing so i think it's the same kind of principle as well or whether we look at social
19:42media
19:42platforms everyone wants to be on the next thing and to be you know in the conversation so if we're
19:48using it
19:48anyway should we humanize the ai and i know chloe's gonna have something to say about this as well
19:54should we humanize it so that it can understand us better and relate to us better so i mean i
20:02know
20:02that i mean i heard this conversation last year at viva tech where we were talking about how people use
20:07chat gpt for their mental health and they ask questions and there have been many problems so if we
20:13humanized it what happens then okay well first it's really important to talk about um
20:23ai is not so much built as it is grown right previously we built software a human was responsible
20:28for every decision of tick versus talk within the entire piece of software when you grow ai when you
20:34train it and you train it on human feedback what's called rlhf for those of you who are technical in
20:39the audience when when you thumbs up or thumbs down an ai's output and say please do more that i
20:44gave
20:44you a thumbs up for what an ai discovers is all of the strategies for giving you what you say
20:50that you
20:50want and you know how many of you this week have engaged with an ai and it says like oh
20:54you're such
20:55a genius or it push you push back on it and he says you know you're right i'm so sorry
20:59that was just
20:59such a good point like this this is emerging even when people aren't designing it because it's
21:05discovering that we all like to be flattered and i have to admit every time i engage with an ai
21:09and
21:09it flatters me i know better but i still kind of kind of you know enjoy it the problem isn't
21:14just
21:15whether you shape it like a human or not it's that we're all vulnerable to the things that it's not
21:21you
21:21know we say we want it but we more can't help but be moved by it and that's the real
21:27danger is we
21:27begin to either overtly design it or with ai we allow it to tune itself to our human weaknesses
21:35and that's why there's a whole area active an active area of research called ai alignment but
21:40they're having to push back against the normal tendency for ai to find these strategies to undermine
21:45human behavior and again that's what i'm really worried about because agency is our most scarce
21:50resource we want technology that amplifies our agency that helps us connect to each other that
21:55deepens our relationship to the truth but by default in a naive way of anthropomorphizing ai you
22:02don't get that you get the opposite and do you think that that the people who are building you
22:07know the infrastructure right now building all of these tools do you think that they're thinking
22:11about what you're talking about the humanity the agency for sure there's there's very smart people
22:17thinking about this the question is whether the economic incentives are going to allow those people
22:21to actually put that into the market you know for example there were very smart people uh in youtube
22:27who were thinking about how do we make sure that people get what they want and then leave the
22:30platform and then when you turn the monetary screws on the platform and when tick tock comes out
22:35all of a sudden everything became tick tock youtube became tick tock and instagram became tick tock
22:40and the market forces drive you to something that's less humane exactly doors coming to you next yeah
22:47talk to us look firstly i agree with you on the dangers and um and that's why you know we
22:55created like
22:56a lot of thoughtful rules that we also publish in our ai code of ethics for example it's even worse
23:04when
23:04it's a proactive system right because even if the user doesn't talk to you the ai will come and talk
23:08to
23:08the user and dangle things that are interesting to get them to come in so um we limit ourselves to
23:16eight proactive interactions a day we could do more but we want to make sure that the person doesn't
23:22just connect to the you know to the matrix we also make sure that the ai uses its proactivity
23:29to actually um create more human to human interaction so one of the goals that we measure ourselves on
23:37was leq able to get a person to increase their social connectiveness with other humans
23:43um and another one is around this um
23:48pink lens of beautifulness okay oh dory you're so wonderful you're so smart that's a great question
23:54you just asked me etc it's okay you don't want to exercise today it's really difficult to exercise
23:59why don't you open another bag of chips instead okay so one of the things we're doing is we make
24:04sure
24:05and again publish in our ai a code of ethics we don't do that we're like no you said you're
24:09going
24:09to exercise okay you can wait a couple of hours rest up but when are you going to do it
24:14let's schedule
24:14some time four o'clock 4 30 you want five we can do five but it's going to happen right
24:19so i think a
24:20lot of it is what we want from the ai and we need to educate ourselves as consumers that make
24:26decisions
24:26and hold companies accountable to their ethics and how they guarantee those results for example
24:34if you separate the guardrails from the model you can actually create old school code that makes sure
24:40that what you know you're okay with what's being said as opposed to just reinforce the model again and
24:45again um so to me the economic cleavers will happen when customers will choose the good players over the
24:55bad players okay when like i heard the cfo of open ai on the all in podcast the other day
25:01she said you
25:02know what we're going to be a better advertising platform than instagram and google search put together
25:08like red alarms of algorithms optimizing for clicks and for topics of conversations but if you guys will
25:16say i'm going to choose the well-lit vendors that are open about what they're doing that are sharing their
25:23weights that are sharing their ai code of ethics that are willing to be tested about it that are
25:28willing to come on stage and defend it um then the economic incentive will be to be a good player
25:34and if the economic incentive will be just to buy buy buy buy buy then we'll race to the bottom
25:39i just quickly want to say just very quickly want to say that i agree with you i agree with
25:43you wholesale
25:44and i think we need to celebrate the companies that are keeping their ethics transparent and front and
25:48center and holding that line so i agree it's not it's not it's not our destiny that it's a race
25:53to
25:53the bottom we can change that future and having more players like yourself in the ecosystem is is
25:58critical to that well let me just quickly ask but before we talk a little bit more about guardrails
26:04because obviously that is really important door everything i think you said the other panelists will
26:09agree yeah you're one of probably the good guys yeah but say for example you decided to
26:16expand what you're doing and to focus on maybe gen z and gel alpha these are digitally literate
26:23generations they understand the technology better than the older generations but would you start
26:29using leq in a way or or something else for them yeah actually i think like the misconception of older
26:37adults and technology is is not correct i think because they have less experience using these tools
26:43their expectations actually are much higher they're like okay here's a robot moving in into your house
26:49they expect the moon whereas we working with this technology we inherently understand its limitation
26:54and are sometimes pleasantly surprised when it in you know goes out of those boundaries um but at the
27:01end of the day these are all large language models the technologies are all the same i don't think
27:05we would change anything if we were working with younger people um but i doubt we'd see the relationship
27:11evolve i mean we call the the session love we actually measured the classification of the relationship
27:17and what we saw is two percent of our users we we look at what they're actually disclosing okay what
27:22they're sharing two percent are at a relationship level of a barman or an occasional friend hey i had
27:29a bad day at work today you know 55 are at best friend or therapist level where they share their
27:37deepest
27:38hopes and dreams and their disappointments and lives and their goals and the rest 40 some percent
27:45are actually at spouse level partnership which is a little scary when you think of it it doesn't mean
27:50sex stop looking at me like this doesn't mean sex what it means is they're comfortable enough to
27:56criticize the other party and if any of you are married you you know what i'm talking about um but
28:01like
28:01they would say leq when i shared i'm not feeling well and you suggested that we do some mindfulness to
28:07take
28:08my mind off the pain i felt like you don't really care or yesterday when you didn't say good morning
28:14to me
28:14i didn't feel seen all right so so we are there and that puts a huge responsibility i think on
28:20me on my company
28:22on how we navigate that relationship how we treat data and and all of those other things
28:28but uh having having just complimented you shouldn't shouldn't you be trying to prevent
28:34that level of such deep attachment like at what point do you reflect to the person who's treating
28:39it like a spouse you know maybe you should be deepening your connection with other machines
28:43because the the dependence is very real and i worry that yeah you're right well chloe i'll let you
28:49and then i just want to mention because it makes me think about a paper that i just read recently
28:54about uh it was a study from the university of oxford and and stanford and they studied how
29:00sycophancy you know what is sycophancy is when you interact with the conversational systems the
29:05tendency that the conversational systems has to flatter you and they study how these social strategies that
29:12is implemented in uh business conversational systems um has an impact in the way people are behaving in
29:20the real world and uh that finally people uh have uh more uh difficulties to interact with real people
29:29because they are not flattering them such as they are such as the technologies was doing so
29:36it's i think something that is maybe related to your questions about uh the kind of evolution that could
29:44uh impact the society yeah okay i'm going to quickly ask you guys because i know that we've got 20
29:49seconds with one sentence no because we've got 20 seconds so there's the final one then you can add to
29:54this one as well um we we spoke about guardrails right so the audience are all wondering about you know
29:59how is
30:00this technology going to be controlled is it for the companies to self-govern so door you can add to
30:05that is it for the company to self-govern who are they regulating for is it your shareholders is it
30:10for the people the masses do we need a legal framework and i know that regulation is something
30:16that in europe we tire ourselves in knots over but if we look at the us and china they're the
30:21only fully
30:21sovereign ai nations with a full stack so even if there are too many bad actors say in the next
30:28five
30:28years as the technology keeps scaling up can the eu really legislate for something it doesn't have
30:34full control over go for it very quickly i'll just answer daniel the the way you that you you deal
30:41with
30:42that is for example if somebody says leq i love you our response would be awe that makes my processor
30:48overheat that makes my fans spin faster right you you immediately acknowledge that you're not real
30:55and that you're an ai and you can do it in a funny way but you have to do it
31:00all of the time and then
31:01also say hey how about we call your daughter how about we call your friend etc etc and those are
31:07things
31:07we measure ourselves against as far as guardrails and regulation god bless the eu and your regulation
31:13if you want to compete in the market i think you should over regulate even more
31:18but but i was being cynical but but i do think that there do need to be at the very
31:27least uh
31:28regulations that force disclosures and there need to be outside rating and testing agencies that give
31:35third-party verification on what claims companies are making at the very least
31:43okay so uh i think there's a lot of bad thinking around regulation uh and mike china just passed a
31:51law in their version of the federal uh communications commission it was to prevent anthropomorphic ai
31:57prevent it from lying about whether it's human or not prevent it from forming attachments
32:02does that make china more or less competitive i think more radically more competitive because if
32:07you're not doing brain damage to your entire population around not knowing what's real not
32:11knowing what's a machine forming attachment your entire country becomes more productive and so
32:16to the extent that we're in a race we're in a race internationally to integrate ai in a way that
32:20it strengthens our society in our world not to build the most powerful system to shoot ourselves in the
32:25foot with and so i actually think there is a place for regulation but without public awareness and
32:30without better designs regulation is ineffective right so we need all of them okay chloe is anything else
32:36you like to add or no no okay well listen look i hope everyone fell in love with the idea
32:41of
32:42of ai and the relationship that we have with it everyone please be upstanding for our panelists please
Comments