An international study published in the journal Nature shows that different, methodologically justified analyses of the same dataset can lead to different results. The study highlights why transparency regarding methodological decisions is an important part of empirical research. The project involved 457 researchers, including members of the Center for Advanced Internet Studies (CAIS) and the University of Duisburg-Essen.
For the study “Investigating the Analytic Robustness of the Social and Behavioural Sciences”, an international team of 457 independent scientists conducted 504 re-analyses of data from 100 previously published studies in the social and behavioral sciences. Different teams received the same dataset and research question. However, it was left open how they analyzed the data, for example in the selection of statistical models, the definition of variables, or data preparation.
The majority of the re-analyses confirmed the central claims of the original studies. In 74% of cases, the analysts arrived at the same conclusion. At the same time, the analyses often differed in the magnitude of the effect found. Additionally, the average effect size of the re-analyses was smaller than that of the original studies.
Such differences occur particularly frequently in research using survey data, which is often used in the social sciences. These data are often complex and require individual analysis decisions within relatively large degrees of freedom in evaluation. “The study does not show that scientific results are unreliable. Rather, it makes visible how scientific knowledge is generated and validated. Individual studies are rarely the final word on a research question. Reliable knowledge usually emerges from many studies that check and complement each other,” said Prof. Conrad Ziller, political scientist at the University of Duisburg-Essen (UDE).
Prof. Johannes Breuer, Head of the Research Data & Methods Department at the Center for Advanced Internet Studies (CAIS) and Professor of Digital Social Science at UDE, added: “The new study shows above all how important transparency regarding analysis decisions is so that results can be better interpreted. Empirical research often allows several methodologically defensible analysis paths. Researchers necessarily have to make various decisions in the evaluation process, for example in data cleaning, model selection, or the interpretation of statistical results. These degrees of freedom can influence the size of measured effects or statistical uncertainty. Therefore, precise documentation and justification of methodological decisions as well as testing their robustness against possible alternatives are important for reliable research results.”
The study also involved researchers Dr. Amelia Zein (CAIS, formerly LMU Munich), Dr. Teresa Hummler (UDE), and Dr. Paul Vierus (UDE). Like Professors Breuer and Ziller, they were co-authors and contributed re-analyses of datasets.
The large-scale collaboration was coordinated by Hungarian professors Balázs Aczél and Barnabás Szászi (Eötvös Loránd University and Corvinus University) within the program “Systematizing Confidence in Open Research and Evidence” (SCORE). The aim of the project is to develop new approaches to make the reliability of scientific results more transparent.
The study published in Nature is available here.
About the Center for Advanced Internet Studies
The state of North Rhine-Westphalia has been providing long-term funding to the Center for Advanced Internet Studies (CAIS) in Bochum since April 2021 as a central institute for digitalization research. Through evidence-based policy and practice recommendations, CAIS contributes to shaping digital transformation in the interest of society. Founded as a research college in early 2017, CAIS has since awarded fellowships to national and international visiting scholars in the field of digitalization research. Within its research programs on Digital Democratic Innovations, Education Technologies and Artificial Intelligence, Design of Trustworthy Artificial Intelligence, and the Transformation of Digital Democratic Discourses, interdisciplinary teams investigate the challenges of digital transformation. At CAIS, disciplines ranging from the social sciences and humanities to computer science are closely interconnected, and research results are tested in practical applications.