Many Labs 3: Evaluating Participant Pool Quality across the Academic Semester via Replication


Charles R. Ebersole, University of Virginia
Olivia E. Atherton, University of California, Davis
Aimee L. Belanger, Miami University
Hayley M. Skulborstad, Miami University
Jill M. Allen, Montana State University
Jonathan B. Banks, Nova Southeastern University
Erica Baranski, University of California, Riverside
Michael J. Bernstein, Pennsylvania State University
Diane B.V. Bonfiglio, Ashland University
Leanne Boucher, Nova Southeastern University
Elizabeth R. Brown, University of North Florida
Nancy I. Budiman, University of California, Davis
Athena H. Cairo, Virginia Commonwealth University
Colin A. Capaldi, Carleton University
Christopher R. Chartier, Ashland University
Joanne M. Chung, University of California, Davis
David C. Cicero, University of Hawaii, Manoa
Jennifer A. Coleman, Virginia Commonwealth University
John G. Conway, University of Florida
William E. Davis, Texas A&M University
Thierry Devos, San Diego State University
Melody M. Fletcher, University of Southern Mississippi
Komi German, University of California, Davis
Jon E. Grahe, Pacific Lutheran University
Anthony D. Hermann, Bradley University
Joshua A. Hicks, Texas A&M University
Nathan Honeycutt, San Diego State University
Brandon Humphrey, Miami University
Matthew Janus, University of Virginia
David J. Johnson, Michigan State University
Jennifer A. Joy-Gaba, Virginia Commonwealth University
Hannah Juzeler, Pacific Lutheran University
Ashley Keres, Ashland University
Diana Kinney, University of Virginia
Jacqueline Kirshenbaum, Bradley University
Richard A. Klein, University of Florida
Richard E. Lucas, Michigan State University
Christopher J.N. Lustgraaf, University of Southern Mississippi
Daniel Martin, University of Virginia
Madhavi Menon, Nova Southeastern University
Mitchell Metzger, Ashland University
Jaclyn M. Moloney, Virginia Commonwealth University
Patrick J. Morse, University of California, Riverside
Radmila Prislin, San Diego State University
Timothy Razza, Nova Southeastern University
Daniel E. Re, University of Toronto
Nicholas O. Rule, University of Toronto
Donald F. Sacco, University of Southern MississippiFollow
Kyle Sauerberger, University of California, Riverside
Emily Shrider, Ashland University
Megan Shultz, Pacific Lutheran University
Courtney Siemsen, Bradley University
Karin Sobocko, Carleton University
R. Weylin Sternglanz, Nova Southeastern University
Amy Summerville, Miami University
Konstantin O. Tskhay, University of Toronto
Zack van Allen, Carleton University
Leigh Ann Vaughn, Ithaca College
Ryan J. Walker, Miami University
Ashley Weinberg, San Diego State University
John Paul Wilson, University of Toronto
James H. Wirth, Ohio State University - Newark Campus
Jessica Wortman, Michigan State University
Brian A. Nosek, University of Virginia

Document Type


Publication Date





The university participant pool is a key resource for behavioral research, and data quality is believed to vary over the course of the academic semester. This crowdsourced project examined time of semester variation in 10 known effects, 10 individual differences, and 3 data quality indicators over the course of the academic semester in 20 participant pools (N = 2696) and with an online sample (N = 737). Weak time of semester effects were observed on data quality indicators, participant sex, and a few individual differences conscientiousness, mood, and stress. However, there was little evidence for time of semester qualifying experimental or correlational effects. The generality of this evidence is unknown because only a subset of the tested effects demonstrated evidence for the original result in the whole sample. Mean characteristics of pool samples change slightly during the semester, but these data suggest that those changes are mostly irrelevant for detecting effects. (C) 2015 Elsevier Inc. All rights reserved.

Publication Title

Journal of Experimental Social Psychology



First Page


Last Page


Find in your library