Assessing Theoretical Conclusions With Blinded Inference to Investigate a Potential Inference Crisis

Authors

Jeffrey J. Starns, University of Massachusetts Amherst
Andrea M. Cataldo, University of Massachusetts Amherst
Caren M. Rotello, University of Massachusetts Amherst
Jeffrey Annis, Vanderbilt University
Andrew Aschenbrenner, Washington University in St. Louis
Arndt Bröder, University of Mannheim
Gregory Cox, Vanderbilt University
Amy Criss, Syracuse University
Ryan A. Curl, Syracuse University
Ian G. Dobbins, Washington University in St. Louis
John Dunn, University of Western Australia
Tasnuva Enam, University of Alabama
Nathan J. Evans, University of Amsterdam
Simon Farrell, University of Western Australia
Scott H. Fraundorf, University of Pittsburgh
Scott D. Gronlund, University of Oklahoma
Andrew Heathcote, University of Tasmania
Daniel W. Heck, University of Mannheim
Jason L. Hicks, Louisiana Sate University
Mark J. Huff, University of Southern MississippiFollow
David Kellen, Syracuse University
Kylie N. Key, University of Oklahoma
Asli Kilic, Middle East Technical University
Karl Christoph Klauer, Albert-Ludwigs-Universität-Freiburg
Kyle R. Kraemer, University of Alabama
Fábio P. Leite, Ohio State University at Lima
Marianne E. Lloyd, Seton Hall University
Simone Malejka, University of Mannheim
Alice Mason, University of Western Australia
Ryan M. McAdoo, University of Oklahoma
Ian M. McDonough, University of Alabama
Robert B. Michael, University of Louisiana at Lafayette
Laura Mickes, University of London
Eda Mizrak, University of Zurich
David P. Morgan, University of London
Shane T. Mueller, Michigan Technological University
Adam Osth, University of Melbourne
Angus Reynolds, University of Tasmania
Travis M. Seale-Carlisle, University of London
Henrik Singmann, University of Warwick
Jennifer F. Sloane, Syracuse University
Andrew M. Smith, Iowa State University
Gabriel Tillman, Australian College of Applied Psychology
Don van Ravenzwaaij, University of Groningen
Christoph T. Weidemann, Swansea University
Gary L. Wells, Iowa State University
Corey N. White, Missouri Western State University

Document Type

Article

Publication Date

12-1-2019

School

Psychology

Abstract

Scientific advances across a range of disciplines hinge on the ability to make inferences about unobservable theoretical entities on the basis of empirical data patterns. Accurate inferences rely on both discovering valid, replicable data patterns and accurately interpreting those patterns in terms of their implications for theoretical constructs. The replication crisis in science has led to widespread efforts to improve the reliability of research findings, but comparatively little attention has been devoted to the validity of inferences based on those findings. Using an example from cognitive psychology, we demonstrate a blinded-inference paradigm for assessing the quality of theoretical inferences from data. Our results reveal substantial variability in experts’ judgments on the very same data, hinting at a possible inference crisis.

Publication Title

Advances in Methods and Practices in Psychological Science

Volume

2

Issue

4

First Page

335

Last Page

349

Recommended Citation

Starns, J. J., Cataldo, A. M., Rotello, C. M., Annis, J., Aschenbrenner, A., Bröder, A., Cox, G., Criss, A., Curl, R. A., Dobbins, I. G., Dunn, J., Enam, T., Evans, N. J., Farrell, S., Fraundorf, S. H., Gronlund, S. D., Heathcote, A., Heck, D. W., Hicks, J. L., Huff, M. J., Kellen, D., Key, K. N., Kilic, A., Klauer, K. C., Kraemer, K. R., Leite, F. P., Lloyd, M. E., Malejka, S., Mason, A., McAdoo, R. M., McDonough, I. M., Michael, R. B., Mickes, L., Mizrak, E., Morgan, D. P., Mueller, S. T., Osth, A., Reynolds, A., Seale-Carlisle, T. M., Singmann, H., Sloane, J. F., Smith, A. M., Tillman, G., van Ravenzwaaij, D., Weidemann, C. T., Wells, G. L., White, C. N. (2019). Assessing Theoretical Conclusions With Blinded Inference to Investigate a Potential Inference Crisis. Advances in Methods and Practices in Psychological Science, 2(4), 335-349.
Available at: https://aquila.usm.edu/fac_pubs/16876