00:00In 2011, a team of physicists reported a startling discovery.
00:11Neutrinos traveled faster than the speed of light,
00:14by 60 billionths of a second,
00:16in their 730-kilometer trip from Geneva to a detector in Italy.
00:20Despite six months of double-checking,
00:23the bizarre discovery refused to yield.
00:25But rather than celebrating a physics revolution,
00:28the researchers published a cautious paper arguing for continued research
00:32in an effort to explain the observed anomaly.
00:35In time, the error was tracked to a single incorrectly connected fiber-optic cable.
00:41This example reminds us that real science is more than static textbooks.
00:45Instead, researchers around the world are continuously publishing their latest discoveries,
00:51with each paper adding to the scientific conversation.
00:54Published studies can motivate future research,
00:57inspire new products,
00:59and inform government policy.
01:01So, it's important that we have confidence in the published results.
01:04If their conclusions are wrong,
01:06we risk time, resources, and even our health in the pursuit of false leads.
01:11When findings are significant,
01:13they are frequently double-checked by other researchers,
01:16either by reanalyzing the data or by redoing the entire experiment.
01:21For example, it took repeated investigation of the CERN data before the timing error was tracked down.
01:28Unfortunately, there are currently neither the resources nor professional incentives
01:33to double-check the more than one million scientific papers published annually.
01:38Even when papers are challenged, the results are not reassuring.
01:42Recent studies that examined dozens of published pharmaceutical papers
01:46managed to replicate the results of less than 25% of them,
01:50and similar results have been found in other scientific disciplines.
01:54There are a variety of sources for irreproducible results.
01:57Errors could hide in the original design, execution, or analysis of the data.
02:03Unknown factors such as patient's undisclosed condition in a medical study
02:07can produce results that are not repeatable in new test subjects.
02:11And sometimes, the second research group can't reproduce the original results
02:15simply because they don't know exactly what the original group did.
02:20However, some problems might stem from systematic decisions in how we do science.
02:25Researchers, the institutions that employ them,
02:28and the scientific journals that publish findings
02:31are expected to produce big results frequently.
02:34Important papers can advance careers, generate media interest,
02:39and secure essential funding.
02:41So there's slim motivation for researchers to challenge their own exciting results.
02:45In addition, little incentive exists to publish results unsupportive of the expected hypothesis.
02:51That results in a deluge of agreement between what was expected and what was found.
02:56In rare occasions, this can even lead to deliberate fabrication,
03:00such as in 2013 when a researcher spiked rabbit blood with human blood
03:05to give false evidence that his HIV vaccine was working.
03:09The publish or perish mindset can also compromise academic journals' traditional peer review processes,
03:15which are safety checks where experts examine submitted papers for potential shortcomings.
03:20The current system, which might involve only one or two reviewers, can be woefully ineffective.
03:25That was demonstrated in a 1998 study where eight weaknesses were deliberately inserted into papers,
03:32but only around 25% were caught upon review.
03:35Many scientists are working toward improving reproducibility in their fields.
03:40There's a push to make researchers' raw data, experimental procedures,
03:44and analytical techniques more openly available in order to ease replication efforts.
03:50The peer review process can also be strengthened to more efficiently weed out weak papers prior to publication.
03:57And we could temper the pressure to find big results
04:00by publishing more papers that failed to confirm the original hypothesis,
04:04an event that happens far more than current scientific literature suggests.
04:08Science always has and always will encounter some false starts
04:12as part of the collective acquisition of new knowledge.
04:15Finding ways to improve the reproducibility of our results
04:18can help us weed out those false starts more effectively,
04:21keeping us moving steadily toward exciting new discoveries.
04:25An example of the
Comments