Observing many students using difference-in-differences designs on the same data and hypothesis reveals a hidden universe of uncertainty

Methodology

Wuttke, Freitag, Kiemes, Biester, Binder, Buitkamp, Dyk, Ehrlich, Lesiv, Poliandri, Schneider, Brenner, Samarskiy [revise and resubmit]: „Observing many students using difference-in-differences designs on the same data and hypothesis reveals a hidden universe of uncertainty”, Plos One.

Authors
Affiliation

Ludwig-Maximilians-Universität München

Ludwig-Maximilians-Universität München

Laura Kiemes

Ludwig-Maximilians-Universität München

Linda Biester

Paul Binder

Bastian Buitkamp

Larissa Dyk

Louisa Ehlich

Mariia Lesiv

Yannick Poliandri

Published

June 2024

Doi

Abstract

The rise of many-analysts studies has underscored substantial variability in research outcomes when different teams independently analyze identical data sets and hypotheses. This paper demonstrates how the many-analysts framework can be effectively integrated into research methods education. We propose a pedagogical approach in which multiple students independently analyze the same research question using identical datasets in their term papers, followed by a meta-analysis of their collective results. This instructional method highlights critical methodological concepts, such as researcher degrees of freedom and epistemic humility. Moreover, this study applies the many-analysts approach specifically to causal inference using observational data with difference-in-differences analyses. The analyses conducted by students revealed a broad range of effect sizes and led to divergent conclusions, thus emphasizing the educational and methodological value of robustness checks involving multiple independent analysts. These findings illustrate how integrating a many-analysts framework in teaching can enhance students’ methodological rigor and appreciation of empirical uncertainty.