Pre-Print: Observing many students using difference-in-differences designs on the same data and hypothesis reveals a universe of uncertainty
We have fundamentally revised the manuscript of a study that reports on a many-analysts project that we conducted in courses on research design at LMU Munich with our LMU political science students as co-authors. The paper shows how many-analysts studies can be effectively integrated into undergraduate methods courses.
Abstract
The rise of many-analysts studies has highlighted substantial variability in research outcomes when different teams independently analyze the same datasets and hypotheses. This paper makes two key contributions to this emerging field. First, we demonstrate how the many-analysts framework can be integrated into research methods education, by having students act as independent analysts addressing the same research question, followed by a meta-analysis of their results. Enriching the pedagogical toolkit, this didactic approach teaches concepts like researcher degrees of freedom and the importance epistemic humility. Second, this study applies the many-analysts framework to causal inference using observational data to identify outcome variability in this area of research. Specifically, we engage students in independent difference-in-differences (DiD) analyses focused on local governance. The study reveals a wide range of effect sizes and divergent conclusions from a typical DiD design, emphasizing the benefits of robustness checks involving multiple independent analysts in research and teaching.