00:00I recently wrote a story entitled, ChatGPT may be eroding critical thinking skills, according to a new MIT study. Natalia, you are the main author of this study. Thank you so much for joining me to talk about your research.
00:13So let me ask sort of the biggest question that is on a lot of people's minds was, does your study provide evidence that ChatGPT is making us dumber?
00:25No, no, no, no. I'm so, so happy you asked and you can see my excitement, I guess. We have like a huge FAQ on the website right now. The words like dumb, brain rot. So all of those words are pure clickbait because we, in our study, never use this vocabulary and we do not show this. And we actually cannot show this with even this setup. It's just impossible.
00:51What we have shown is specifically what happens, you know, neural connectivity, essay outputs, scoring, you know, interviews. We put it all into perspective and there are different, there are a lot of interesting things happening.
01:04Can you briefly describe the study?
01:06We invited 54 participants who were all students from Greater Boston University. And we invited them to come in person to the lab. And we asked them to do one very specific task, which is essay writing. And we divided them in three groups.
01:25So first group was tasked to only use ChatGPT. So they couldn't use anything else, just ChatGPT to write their essay. Second, a search engine without AI functionality.
01:39So basically, you can do like a classic OG search engine. And then the third one, we call it, you know, tools free, free tools, you know, which is no tools. And we also call it in the paper brain only because of that. So you only could have used your brain to write your essay.
01:58The research began with three essay writing sessions where students were able to pick writing prompts they were interested in.
02:04So what we have observed during the first three sessions is that their essays were pretty homogenous for their LLM group. So basically what it means in layman terms is they were very similar, right?
02:20Wording that was used, for example, a lot of specific words that were used, like names and persons, different entities. So they were, we would have definitely, you know, observed this.
02:31The researchers studied neural activity using a non-invasive brain imaging technique called EEG.
02:37And then, of course, when we looked at neural connectivity, we also saw that, you know, there is actually a change when you use tools like search engine or LLM versus when you don't use any tools at all, which is in itself actually should not be very surprising, right?
02:54And a lot of studies before us actually explored this phenomena even earlier, not with LLMs, but with the tools we already had, like GPS or Google, etc.
03:05So what we saw is that there is definitely some scaling down happening when you are moving from no tools, meaning like you're only using your brain, up to all the way to the LLM.
03:15So, you know, you have pretty high neural connectivity happening, brain only, intermediate for search, and then definitely lower for the LLM.
03:26After the third session, researchers re-invited participants back for a fourth time.
03:31About a third of the original 54 participated.
03:34So it was a smaller sample size.
03:35But the students were reassigned to different groups.
03:38For instance, the students that wrote with their brain only were given ChatGPT, and the students that had used ChatGPT were now not allowed to use it.
03:48And additionally, we actually let the subjects pick up from the essay that were not new to them, but actually the essay they had already written about.
03:57So something you have already written from those previous three sessions.
04:01So that was an experiment.
04:03The results of this switch showed something surprising.
04:05The people who had been in the brain only sessions for the first three essays performed even higher with ChatGPT.
04:12So we actually in the end, being again very careful with any speculations, do say the timing of introduction might be extremely interesting and potentially important to explore in any future studies.
04:27Can you tell me a little bit about that and why that might be actually potentially optimistic for, you know, LLM usage?
04:33What happens, and it might be actually sounding also very natural for everyone, you used your brain to execute on the task once, two times, three times.
04:45And the fourth time, right, you now have additionally two to use on that very same task.
04:52So basically, hey, now that you have the tool, you have potentially more opportunities to ask the right questions.
05:02Because guess what?
05:03You spent your time thinking about it without a tool.
05:08I honestly feel like that in our society, we are losing a bit this, you know, opportunity, like, think before you speak or like think before you repost or think before you post, right?
05:21So that might be just about it.
05:23Hey, do it yourself first, right?
05:27And then use the tool.
05:29And the tool can potentially maybe augment your output, your product, your use case, whatever you are going for, right?
05:40Does your study show evidence that ChatGPT may erode critical thinking?
05:44It doesn't show this directly, right?
05:49We don't, we really do not want to claim strong, oh, critical thinking is gone here.
05:56Because again, limitations of the study.
05:58It's a preprint and where we have very few participants in a rather specific task, you know, essay writing.
06:07It's literally in the title.
06:09However, what I do want to mention, right, that we, of course, looked what other researchers before us, right, you know, reported on.
06:17We needed to put it in some perspective and connect some dots, right?
06:22And again, this is what we found with our study.
06:26But again, there are other papers that talk that in some use cases, it might actually support some of that.
06:32Like, even in our case, we say, hey, timing might be important, might be important, right?
06:39It is really, you know, to be seen and to be tested.
06:43More data is needed.
06:45Yeah, so given the limitations of your study, why did you feel it was so important to put it out before, you know, preprint?
06:53With this unprecedented speed of release of these tools in all aspects of our lives, education, workspace, etc.
07:03In 4 to 12 months, right, where reviews usually take place, some actually couldn't take up to two years, it might be just late.
07:12We will end up with some, you know, GPT kindergarten, some other things without proper data.
07:17So why do we need to move very fast and break things so fast in our own backyard when we, let's just actually explore this, right?
07:27Let's have more data.
07:29And you know, super secret, it doesn't even need to be brain data, right?
07:33Let's just test it.
07:35Hey, with, without, does it make a difference?
07:39Does it make a change?
07:40And more importantly, include all, all opinions that would matter.
07:44In this case, teachers, you know, educators, caretakers, right?
07:50If you're talking about shielding, where I think it would be so, so critical because we all know those brains are developing.
07:58So why do we need to rush these deployments, right?
08:03Let's maybe pause, evaluate, and there might be excellent use cases.
08:09This technology is not going anywhere.
Comments