A large-scale international project has found that roughly half of tested scientific claims across the social and behavioral sciences hold up when independent researchers attempt to repeat the work. The research, published in Nature, indicates that the challenges of verifying published findings are not confined to a single field, but rather extend across disciplines including psychology, economics, sociology, education, business, and political science. On average, the effects observed in the replication attempts were less than half as large as those originally reported.
In scientific research, a replication attempt involves testing the exact same question as a previous study using independent data to see if the original conclusion holds true. If a scientific discovery represents a genuine regularity in human behavior, independent researchers following a similar procedure or analyzing a fresh dataset should be able to observe a similar result. Repeating past studies helps verify whether an initial finding is reliable enough to inform public policy, education, or clinical practice.
Over the past decade, large-scale verification efforts have raised questions about how often published findings hold up under scrutiny. A 2015 replication project tested 100 psychology studies and found that only about one-third produced the original results, sparking widespread discussion about research reliability. Follow-up investigations soon tested other areas of study to see if the issue was unique to psychology.
Other fields showed a similar pattern. A 2016 study of laboratory experiments in economics found that 11 out of 18 studies produced a statistically significant effect in the same direction as the original. Similarly, a 2018 investigation of social science experiments published in top-tier general science journals found that about 62 percent repeated successfully, though the strength of the findings was cut roughly in half. Together with other replication efforts, these projects suggested that across hundreds of attempts, roughly half of well-powered replications produced a statistically significant result in the same direction as the original.
Researchers have proposed several systemic explanations for why many published findings fail to repeat. Academic journals have historically favored novel, positive discoveries over inconclusive or negative results, creating a publication filter where studies that happen to overestimate an effect by chance are the ones most likely to be printed. Small sample sizes, flawed measurement tools, and flexible data analysis practices can also produce false positives. At the same time, academic culture has often discouraged verification work; for example, a 2017 study found that only about 3 percent of psychology journals explicitly welcomed replication studies, while a third implicitly discouraged them.
Most prior verification projects focused on a single academic discipline or examined relatively small groups of papers. To determine whether the patterns seen in earlier efforts apply across the broader landscape of human behavior research, a massive collaborative team led by Andrew H. Tyner of the Center for Open Science, alongside senior colleague Timothy M. Errington and corresponding author Brian A. Nosek, executive director of the Center for Open Science and a professor at the University of Virginia, launched a systematic cross-disciplinary investigation.
Nosek also coordinated the 2015 psychology project and co-authored the 2018 analysis of studies in top general science journals. The new work was conducted as part of the Defense Advanced Research Projects Agency (DARPA) Systematizing Confidence in Open Research and Evidence (SCORE) program, bringing together hundreds of researchers from 31 countries.
“This project was part of a larger research program called SCORE that involved thousands of researchers around the world and began in 2019,” Nosek told PsyPost. “We were investigating whether algorithms could predict the replicability of findings to support decision-making about what research is needed next to advance knowledge.”
The researchers began by gathering a random sample of 3,900 quantitative papers published between 2009 and 2018 across 62 journals representing six major disciplines: business, economics, education, political science, psychology, and sociology. To be eligible, a paper had to use real human data, ranging from individual participants to families, businesses, or political entities, and report a positive statistical result. From a pool of 1,500 eligible papers, the team matched specific scientific claims with independent research groups who had the expertise and resources to test them.
In total, the teams completed 296 replication attempts covering 274 unique scientific claims drawn from 164 published papers across 54 journals. For 98 of the papers, researchers collected brand-new data from participants, while for 66 papers, teams analyzed existing secondary datasets that had not been used in the original research. Across the subset of studies where standard correlation metrics could be calculated, the median sample size was 206 in the original papers and 545 in the replication studies.
To ensure the tests were fair and rigorous, the replication studies were designed to have high statistical power, meaning they used large enough samples to have a median 99.6 percent chance of detecting an effect as large as the one originally reported. Whenever possible, teams used the original study materials. Before any data were collected or analyzed, each study’s design and analysis plan went through an internal peer review process, which invited feedback from the original authors, and was publicly registered in advance on the Open Science Framework to prevent researchers from altering their methods after seeing the results.
The researchers first evaluated the outcomes using the most common yardstick in social science: whether the replication study produced a statistically significant result in the same direction as the original paper. In science, statistical significance means that an observed pattern would be unlikely if there were no real effect, typically using a threshold of less than a 5 percent probability. Because some papers had multiple claims tested, the authors calculated results both by weighting claims so each paper counted equally and by counting each claim individually.
When weighted by paper, 80.8 of the 164 papers, or 49.3 percent, produced results in the same pattern that met the threshold for statistical significance. Another 40.4 percent of the papers showed a null effect, meaning the result did not reach statistical significance, while 9.7 percent actually produced a statistically significant result in the opposite direction of the original claim. When looking at the 274 unweighted claims individually, 151 claims, or 55.1 percent, succeeded by this measure, while 35.8 percent showed a null effect and 8.8 percent went in the opposite direction.
Beyond yes-or-no significance thresholds, the researchers examined the magnitude of the findings, known as the effect size. To compare studies that used different statistical models, the team converted the results of 157 papers (representing 249 claims) into a standardized correlation measure called Pearson’s r, which runs from -1 to 1, where 0 means no relationship, 1 means a perfect relationship, and negative values indicate a relationship in the opposite direction. Across the original papers, the median effect size was 0.25, whereas in the replication studies, the median effect size dropped to 0.10.
This drop from 0.25 to 0.10 represents a 58.1 percent relative reduction in the correlation and an 82.4 percent reduction in shared variance, a measure of how much one variable explains another. Overall, 80.3 percent of the replicated papers (126 of 157) and 70.3 percent of the individual claims (175 of 249) yielded smaller effect sizes than the original studies. Some of this shrinkage is expected due to a statistical phenomenon called regression to the mean: because the researchers only tested positive published results, any original studies that happened to overestimate the true effect by luck would naturally be followed by smaller, more realistic estimates upon retesting.
“The basic finding is that establishing the replicability of published results cannot be taken for granted,” Nosek said. “The main practical implication is that this research exposes greater uncertainty in the published literature than we might recognize. Establishing credibility and verifying research claims is an effortful process and our assumptions about the strength and nature of a finding are sometimes incorrect.”
When the researchers broke the results down across the six academic disciplines, they found only modest variation, and the differences between fields were not statistically significant. Weighted by paper, the success rates were 42.5 percent in economics (10.2 of 24 papers), 47.2 percent in business (17.0 of 36), 49.0 percent in psychology (28.4 of 58), 51.1 percent in sociology (9.2 of 18), 52.0 percent in political science (7.8 of 15), and 63.1 percent in education (8.2 of 13). However, some fields contributed only a handful of papers, such as 13 in education, so their individual estimates are uncertain. Similarly, the year a paper was published between 2009 and 2018 did not have a statistically significant relation to whether it held up.
The team also compared studies that gathered brand-new data against those that analyzed existing secondary datasets. Both approaches succeeded at similar rates on the basic significance metric: 47.8 percent of papers using new data held up, compared to 51.4 percent of papers using secondary data. However, when looking at effect sizes, studies using new data saw their median effect size drop from 0.28 in the original work to 0.11 in the replication, whereas secondary data studies started with smaller original effects (a median of 0.13) and dropped only slightly to 0.10.
Because scientists disagree on the best mathematical way to define a successful replication, the authors also evaluated the data using 12 additional criteria, such as checking whether the replication result fell within the original study’s margin of error, or combining both studies in a meta-analysis. Across all 13 methods, the estimated success rate ranged from 28.6 percent to 74.8 percent, with a median of 49.3 percent. The highest success rate (74.8 percent) came from combining the original and new data together in a meta-analysis, a method that leans toward success because it bakes in the original positive data, while methods that penalized replication studies for finding either smaller or larger effects than the original study produced the lowest rates.
“These findings confirm and extend existing work on replicability, such as the ‘Many Labs’ studies,” Stephanie Zellers, a postdoctoral researcher at the University of Helsinki who holds a PhD in psychology and was not involved in the study, told PsyPost. “Many Labs projects aim to test the replicability of specific research findings using a multisite collaborative approach. This work extends beyond this approach by comprehensively testing a wider range of research findings across behavioral and social science disciplines. The authors used a systematic, representative, and random approach to identify claims for testing, though there was some non-random selection introduced by teams choosing to start a specific replication project.”
“I am quite confident in these results and that is due in large part to the transparent and nuanced reporting,” Zellers said. “I appreciated the detailed discussion of replication metrics – these methods of comparing an original result to the replication result in order to decide if the original result replicated or not. Our choice of metric influences what the answer means, when we ask the question ‘did this study replicate?’ They all have their assumptions, strengths and limitations, and I am more confident in these results because the authors used different metrics when possible.”
There are a few things to keep in mind when interpreting these results. A single successful replication does not mean the original finding is beyond doubt, and no two studies in the social sciences are ever identical. Shifting a study to a new year, a new group of participants, or a different cultural setting can alter the outcome for legitimate reasons that scientists do not yet fully understand.
“Failing to replicate a finding does not mean that the original finding is wrong,” Nosek emphasized. “One possibility is that the original finding is a false positive or exaggerated. Other possibilities are that we made an error in the replication study, or that unrecognized factors influenced the finding differently between the original and replication studies.”
“These findings do not mean that social and behavioral sciences are only ‘correct’ about half of the time and should not be interpreted as such,” Zellers added. “Replication is not inherently a positive thing and lack of replication is not inherently a problem. Rather we should focus on the total body of evidence for a claim and critically evaluate uncertainty in the evidence alongside the context and boundaries of the effect.” She noted that “meta-research is not a scientific ‘witch hunt’ focused on debunking some specific claim via replication, it is an essential part of science that helps us understand and evaluate a whole body of evidence for or against some claim.”
In addition, the studies tested in this project were not a purely random slice of all social science. Although the initial pool of papers was chosen randomly, the final 164 papers were those that proved practical to repeat within the project’s budget, timeline, and researcher expertise. Resource-intensive studies were less likely to be attempted, and qualitative research and papers reporting null results were excluded from the start. The mix of fields also shifted: psychology made up about 24 percent of the original sample but 35.4 percent of the completed papers, while education and political science shrank in relative share.
“I would be interested to see a project that tests replicability in a broader group of journals, broader research results like null and qualitative findings, and in different time periods of published results,” Zellers noted. “These types of studies could help us understand drivers of replicability and unreplicability.”
The authors suggest that the core issue facing the social and behavioral sciences is not necessarily a lack of replicability itself, as cutting-edge research naturally involves taking risks on uncertain ideas that may not pan out. In their view, the challenge is overconfidence: treating a single published paper as established fact rather than preliminary evidence.
“I think the authors do a reasonable job in discussing the findings without any sensationalist language about disciplines in crisis or fields being ‘built on noise,’ however I would also caution against any misinterpretation that suggests we should not be concerned about these relatively low rates of replicability,” Zellers explained. “Replication does not prove an original claim true and unreplicability does not prove an original claim false, and so we should not interpret any single replication result as good or bad without more information. That said, the overall low rate of replicability in social sciences fields is concerning because it may stem from systemic issues in scientific publishing, like publication bias toward positive results and overconfidence in published claims.”
“I was surprised that most of the research community readily accepted these findings,” Nosek remarked. “Our prior systematic replication studies generated substantial, heated debate. There are still a variety of perspectives on the implications of these results, but there was much less heated rhetoric. I think that is a sign of progress?”
Future research could use the publicly available dataset to examine what predicts whether a study holds up. Earlier work along these lines, including research covered by PsyPost in 2022, found that the political slant of psychology findings was not associated with whether they replicated.
Nosek added, “Now that we have published several systematic replication studies, many research efforts are using the outcomes to advance various research questions such as: How do you decide whether a finding has replicated successfully? (Not a trivial question!) What factors cause increases or decreases in replicability? How prevalent are each of those causes? Is there an optimal replicability rate for different research circumstances?”
“Discovery science is often high-risk high-reward, resulting both in powerful new findings and many false leads,” Zellers said of investigating optimal replicability rates. “Replication science can help us identify which leads may be dead ends, as well as understand moderators and generalizability. We need both to create robust and impactful science, especially when translating our research claims into care, policy, and practice. What is the optimal balance between discovery and replication science to make the best use of increasingly limited financial resources in research?”
The study, “Investigating the replicability of the social and behavioural sciences,” was authored by Andrew H. Tyner, Anna Lou Abatayo, Mason Daley, Samuel Field, Nicholas Fox, Noah A. Haber, Krystal M. Hahn, Melissa Kline Struhl, Brinna Mawhinney, Olivia Miske, Priya Silverstein, Courtney K. Soderberg, Theresa Stankov, Ahmed Abbasi, Christopher L. Aberson, Balazs Aczel, Matúš Adamkovič, Nihan Albayrak, Peter J. Allen, Michael Andreychik, Eli Awtrey, Erick Axxe, Flavio Azevedo, Miles D. Bader, Bence Bago, James Bailey, Marjan Bakker, Gabriel Banik, George C. Banks, Ernest Baskin, Anatolia Batruch, Annika Beatteay, Sophie M. Behr, Nicholas Berente, Zachariah Berry, Jędrzej Białkowski, Bojana Bodroža, Laura Boeschoten, Miklos Bognar, Christian Bokhove, Diane Bonfiglio, Robin Bouwman, Timothy F. Brady, Scott R. Braithwaite, Gabriel Briceño Jiménez, Cameron Brick, Traci Bricka, Roman Briker, Annette N. Brown, Gordon D. A. Brown, Robbie C. M. van Aert, Kathryn Caldwell, Sara Capitan, Tabaré Capitán, Jesse Chandler, Tessa Charles, Christopher R. Chartier, Rahul Chawdhary, Kent Jason Cheng, William J. Chopik, Bruce Clark, Victoria E. Colvin, C. Cozette Comer, Giulio Costantini, Tom Coupé, Jamie Cummins, Aneta Czernatowicz-Kukuczka, Joshua de Leeuw, David Dobolyi, James N. Druckman, Jianhua Duan, Marin Dujmović, Daniel J. Dunleavy, Patrick K. Durkee, Cécile Emery, Kevin M. Esterling, Thomas R. Evans, Anna Fedor, Belén Fernández-Castilla, Nathan Fiala, James G. Field, Nathan Fong, Miguel A. Fonseca, Alexandra L. J. Freeman, Jeremy Freese, Sandra J. Geiger, Jing Geng, Laura M. Getz, Linda Marjoleine Geven, Ilka Helene Gleibs, Donna Pamella Gonzales, Janaki Gooty, Amélie Gourdon-Kanhukamwe, Cristina Greculescu, Siobhán M. Griffin, Lusine Grigoryan, Martina Grunow, Nicholas Gunby, Braeden Hall, Paul H. P. Hanel, Erin E. Hannon, Sam Harper, Marco Jürgen Held, Louis Hickman, Nathan C. Higgins, Svenja Hippel, Sven Hoeppner, Sanghyun Hong, Thomas J. Hostler, Michael Inzlicht, Kamil Izydorczak, Bastian Jaeger, Kristin Jankowsky, Johannes Jarke-Neuert, Matthew Jensen, Biljana Jokić, Daniel Jolles, Phillip Jolly, Angela M. Jones, Marie Juanchich, Pavol Kačmár, Hansika Kapoor, Andjela Keljanovic, Samjhana Koirala, Marta Kołczyńska, Dimitra Kouroupaki, Ulrich Kühnen, Michelangelo Landgrave, Michael J. Larson, Lyonel Laulié, Alice C. E. Lawrence, Joel M. Le Forestier, Katelin E. Leahy, Sungmok Lee, Jared Leslie, Savannah C. Lewis, Christopher Limnios, Hause Lin, An-Chiao Liu, John Wills Lloyd, Elliot A. Ludvig, Dermot Lynott, Jordan MacDonald, Peter Mallik, Daniel J. Mallinson, Daniele Marinazzo, Corinna S. Martarelli, Joshua Matacotta, Andrew McBride, Cillian McHugh, Gail McMillan, Esteban Méndez, Mitchell Metzger, Michalis P. Michaelides, Johannes Michalak, Leticia Micheli, Jeremy K. Miller, Marina Milyavskaya, Daniel C. Molden, Ambar G. Monjaras, David Moreau, Audrey Morrow, Cristóbal Moya, Liad Mudrik, Laetitia B. Mulder, Katie A. Munt, Arijit Nandi, Kathryn Nason, Carolin Nast, Gideon Nave, Heinrich H. Nax, Florian Neubauer, Phuong Linh L. Nguyen, Austin Lee Nichols, Gustav Nilsonne, Ernest O’Boyle, Jule Oettinghaus, Jeewon Oh, Adoril Oshana, Thomas Ostermann, Rachel P. Ostrowski, Abiola Oyebanjo, Radoslaw Panczak, Jamie Patrianakos, Ignacio Pavez, Yuri G. Pavlov, Sofia Persson, Marco Perugini, Kim Peters, Constant Pieters, Vladimir Ponizovskiy, Nathaniel D. Porter, Jason M. Prenoveau, Danka Purić, Mariah F. Purol, Arathy Puthillam, Kimberly A. Quinn, Marco Ramljak, W. Robert Reed, Michaela Ritchie, Margaret Ritzau, Sean Patrick Roche, Romina Rodela, Jan Philipp Röer, Ivan Ropovik, Jacob Rothschild, Justine Saal, Hani Safadi, Jason Samaha, Mary Sanchez, Soorya Sankaran, David Santos, Amanda C. Sargent, Marian Sauter, Kathleen Schmidt, Landon Schnabel, Amber N. Schroeder, Sebastian W. Schuetz, Brendan A. Schuetze, Michael Schulte-Mecklenbeck, Astrid Schütz, Eric L. Sevigny, Ellie Shackleton, Richard M. Shafranek, Samuel Shaki, Shishir Shakya, Miroslav Sirota, Matthew Ryan Sisco, Maksim M. Sitnikov, L. Robert Slevc, Laura Smalarz, Colin Tucker Smith, Joel S. Snyder, Nicolas Sommet, Fatih Sonmez, Barbara A. Spellman, Natalia Stanulewicz-Buckley, George Stock, Chris N. H. Street, Eirik Strømland, Tina Sundelin, Moin Syed, Anna Szabelska, Barnabas Szaszi, Ewa Szumowska, Anirudh Tagat, Susanne Täuber, Louis Tay, Stuti Thapa, Jason Thatcher, Domna Tsaklakidou, Lars Tummers, Elise Turkovich, Melba Verra Tutor, Karolina Urbanska, Anna Elisabeth van ’t Veer, Marcel van Assen, Niels van de Ven, Ruben van den Goorbergh, Elisabeth Julie Vargo, Leigh Ann Vaughn, Simine Vazire, Jentien M. Vermeulen, Diem Thi Hong Vo, Victor Volkman, Eric-Jan Wagenmakers, Deliah Wagner, Lukasz Walasek, Frank Walter, Lara Warmelink, Liuqing Wei, Marie Isabelle Weißflog, Nicholas Weller, Aaron L. Wichman, Jonathan Wilbiks, Jamal R. Williams, Kelly Wolfe, Finnian Wort, Ryan Wright, Jesper N. Wulff, Xindong Xue, Veronica X. Yan, Yuzhi Yang, Sangsuk Yoon, Iris Žeželj, Yinxian Zhang, Ignazio Ziano, Cristina Zogmaister, Zorana Zupan, Rolf A. Zwaan, Brian A. Nosek, and Timothy M. Errington.