PSY6316 Current Issues in Cognitive Neuroscience
Semester 2, Academic Year 2017-18. Topic:
“How reliable is cognitive neuroscience?”
What has been called The Replication Crisis has sparked widespread introspection about the standards and protocols of science, particularly within the behavioural sciences. This course, though reading a series of landmark papers and class discussion, will consider to extent to which doubts about the reliability of empirical work affect cognitive neuroscience. Can we trust the published papers in this field? Are the effects which we investigate reliable? If not, how can work in cognitive neuroscience be made more trustworthy?
This is a discussion led module so your contribution to the classes is required. You must do the reading in advance if you expect to get anything from the classes. There is also considerable flexibility in the timetable, so if there is an area you think would be useful to discuss in class, please suggest this
Chambers, C. (2017). The seven deadly sins of psychology: A manifesto for reforming the culture of scientific practice. Princeton University Press.
Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7, 531-536.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
Etz, A., & Vandekerckhove, J. (2016). A Bayesian perspective on the reproducibility project: Psychology. PLoS One, 11(2), e0149794.
Crandall, C. S., & Sherman, J. W. (2016). On the scientific superiority of conceptual replications for scientific progress. Journal of Experimental Social Psychology, 66, 93-99.
2. replicability of fMRI
Wager, T. D., Lindquist, M. A., Nichols, T. E., Kober, H., & Van Snellenberg, J. X. (2009). Evaluating the consistency and specificity of neuroimaging data using meta-analysis. Neuroimage, 45(1), S210-S221.
Bennett, C. M., & Miller, M. B. (2010). How reliable are the results from functional magnetic resonance imaging?. Annals of the New York Academy of Sciences, 1191(1), 133-155.
Poldrack, R. A., Baker, C. I., Durnez, J., Gorgolewski, K. J., Matthews, P. M., Munafò, M. R., ... & Yarkoni, T. (2017). Scanning the horizon: towards transparent and reproducible neuroimaging research. Nature Reviews Neuroscience, 18(2), 115-126.
3. analytic flexibility: the dead salmon in the scanner
Bennett, C. M., Miller, M. B., & Wolford, G. L. (2009). Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: An argument for multiple comparisons correction. Neuroimage, 47(Suppl 1), S125.
Carp, J. (2012). The secret lives of experiments: methods reporting in the fMRI literature. Neuroimage, 63(1), 289-300.
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science, 22(11), 1359-1366.
4. Statistical inference & multiple comparisons
Luck, S. J., & Gaspelin, N. (2017). How to get statistically significant effects in any ERP experiment (and why you shouldn't). Psychophysiology, 54(1), 146-157.
Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304-1312.
Rouder, J. N., Morey, R. D., Verhagen, J., Province, J. M., & Wagenmakers, E. J. (2016). Is there a free lunch in inference? Topics in Cognitive Science, 8, 520-547.
Brain Voyager tutorial pages
Lindquist, M. A., & Mejia, A. (2015). Zen and the art of multiple comparisons. Psychosomatic medicine, 77(2), 114.
Han, H., & Glenn, A. L. (2017). Evaluating methods of correcting for multiple comparisons implemented in SPM12 in social neuroscience fMRI studies: an example from moral psychology. Social neuroscience, 1-11.
5. Four classics
Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Med, 2, e124.
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on psychological science, 4(3), 274-290.
Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013)
Gelman, Andrew, and Eric Loken. "The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time." Department of Statistics, Columbia University (2013).
6. Case study: Ego-depletion
Baumeister, Roy F., et al. "Ego depletion: Is the active self a limited resource?." Journal of personality and social psychology 74.5 (1998): 1252.
Gailliot, M. T., Baumeister, R. F., DeWall, C. N., Maner, J. K., Plant, E. A., Tice, D. M., Brewer, L. E., & Schmeichel, B. J. (2007). Self-control relies on glucose as a limited energy source: Willpower is more than a metaphor. Journal of Personality and Social Psychology, 92, 325–336.
Hagger, M. S., Wood, C., Stiff, C., & Chatzisarantis, N. L. (2010). Ego depletion and the strength model of self-control: a meta-analysis. Psychological bulletin, 136(4), 495.
Hagger, M. S., Chatzisarantis, N. L., Alberts, H., Anggono, C. O., Batailler, C., Birt, A. R., ... & Calvillo, D. P. (2016). A multilab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science, 11(4), 546-573.
Sjåstad, H. and Baumeister, R.F. (2018). The Future and the Will: Planning requires self-control, and ego depletion leads to planning aversion. Journal of Experimental Social Psychology, 76, 127-141. doi: 10.1016/j.jesp.2018.01.005
7. Case study: the case of precognition
Bem, D. J. (2011). Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. Journal of personality and social psychology, 100(3), 407.
Why the Journal of Personality and Social Psychology Should Retract Article DOI: 10.1037/a0021524 “Feeling the Future: Experimental evidence for anomalous retroactive influences on cognition and affect” by Daryl J. Bem
8. Case study: Amy cuddy & power posing
Carney, D. R., Cuddy, A. J., & Yap, A. J. (2010). Power posing: Brief nonverbal displays affect neuroendocrine levels and risk tolerance. Psychological science, 21(10), 1363-1368.
When the Revolution Came for Amy Cuddy
9. Reproducibility & validity
Georgescu, C., & Wren, J. D. (2017). Algorithmic identification of discrepancies between published ratios and their reported confidence intervals and p-values. Bioinformatics, btx811.
Computational science: ...Error
A guide to reproducible code
Computational practices for reproducible science
Borsboom, D., Cramer, A. O., Kievit, R. A., Scholten, A. Z., & Franić, S. (2009). The end of construct validity. In The Concept of Validity: Revisions, New Directions and Applications, Oct, 2008.
10. Solutions I: Psychology Theory
Meehl, P. E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195-244.
Weisberg, D. S., Keil, F. C., Goodstein, J., Rawson, E., & Gray, J. R. (2008). The seductive allure of neuroscience explanations. Journal of cognitive neuroscience, 20(3), 470-477.
Bullock, J. G., Green, D. P., & Ha, S. E. (2010). Yes, but what’s the mechanism?(don’t expect an easy answer). Journal of Personality and Social Psychology, 98, 550-558.
Coltheart, M. (2013). How can functional neuroimaging inform cognitive theories?. Perspectives on Psychological Science, 8(1), 98-103.
Jonas, E., & Kording, K. P. (2017). Could a neuroscientist understand a microprocessor?. PLoS computational biology, 13(1), e1005268.
Yarkoni, T., & Westfall, J. (2017). Choosing prediction over explanation in psychology: Lessons from machine learning. Perspectives on Psychological Science, 12(6), 1100-1122.
10. Solutions II: Evaluating a body of literature
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: a key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534.
See also : http://www.p-curve.com/, http://willgervais.com/blog/2014/7/20/my-p-curve
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. Bmj, 315(7109), 629-634.
Lau, J., Ioannidis, J., Terrin, N., Schmid, C., Olkin, I., 2006. The case of the misleading
funnel plot. Br. Med. J. 333, 597–600.
GRIM and SPRITE
Brown, N. J., & Heathers, J. A. (2017). The GRIM test: A simple technique detects numerous anomalies in the reporting of results in psychology. Social Psychological and Personality Science, 8(4), 363-369. http://journals.sagepub.com/doi/10.1177/1948550616673876
SPRITE: “Introducing SPRITE (and the Case of the Carthorse Child)”
The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses, September 2016, John P.A. Ioannidis https://www.milbank.org/quarterly/articles/mass-production-redundant-misleading-conflicted-systematic-reviews-meta-analyses/
Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G. (2018). Meta-analysis and the science of research synthesis. Nature, 555(7695), 175.
Solutions III: Open Science
Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., du Sert, N. P., ... & Ioannidis, J. P. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021.
Transparency and Openness Promotion (TOP) Guidelines.
Wicherts, J. M., Borsboom, D., Kats, J. & Molenaar, D. The poor availability of psychological research data for reanalysis. Am. Psychol. 61, 726–728 (2006).
Counterpoints: “crisis? What crisis?”
Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016). Comment on “Estimating the reproducibility of psychological science”. Science, 351(6277), 1037-1037. http://science.sciencemag.org/content/351/6277/1037.2
Fanelli, D. (2018). Opinion: Is science really facing a reproducibility crisis, and do we need it to?. Proceedings of the National Academy of Sciences, 115(11), 2628-2631.
Psychology Is Not in Crisis
Wellek, S. (2017). A critical evaluation of the current “p‐value controversy”. Biometrical journal. https://onlinelibrary.wiley.com/doi/full/10.1002/bimj.201700001
Voelkl, B., & Würbel, H. (2016). Reproducibility crisis: are we ignoring reaction norms?. Trends in pharmacological sciences, 37(7), 509-510. http://www.cell.com/trends/pharmacological-sciences/abstract/S0165-6147(16)30034-7
Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?. American Psychologist, 70(6), 487. http://psycnet.apa.org/buy/2015-39598-001
Stroebe, Wolfgang, and Fritz Strack. "The alleged crisis and the illusion of exact replication." Perspectives on Psychological Science 9, no. 1 (2014): 59-71. http://journals.sagepub.com/doi/abs/10.1177/1745691613514450
On the evidentiary emptiness of failed replications
Jason Mitchell, Harvard University, 1 July, 2014
What Jason Mitchell's 'On the emptiness of failed replications' gets right
Fiske's "Mod rule or wisdom or crowds"
Inside Psychology’s ‘Methodological Terrorism’ Debate
Annex I: sources of unreliability
Vul, E., & Kanwisher, N. (2010). Begging the question: The non-independence error in fMRI data analysis. Foundational issues for human brain mapping, 71-91.
Szucs, D., & Ioannidis, J. P. (2017). Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLoS biology, 15(3), e2000797.
Geuter, S., Qi, G., Welsh, R. C., Wager, T. D., & Lindquist, M. A. (2018). Effect Size and Power in fMRI Group Analysis. bioRxiv, 295048.
publication (bias) / quirks of peer review, file draw effects
Higginson, A. D., & Munafò, M. R. (2016). Current incentives for scientists lead to underpowered studies with erroneous conclusions. PLoS biology, 14(11), e2000995.
Lee, C. J., Sugimoto, C. R., Zhang, G. and Cronin, B. (2013), Bias in peer review. J Am Soc Inf Sci Tec, 64: 2–17. doi:10.1002/asi.22784
John, L. K., Loewenstein, G. & Prelec, D. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23, 524–532 (2012).
Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196-217.
Culpepper, S. A., & Aguinis, H. (2011). Using analysis of covariance (ANCOVA) with fallible covariates. Psychological Methods, 16, 166-178.
Eklund, A., Nichols, T. E., & Knutsson, H. (2016). Cluster failure: why fMRI inferences for spatial extent have inflated false-positive rates. Proceedings of the National Academy of Sciences, 201602413.
-- see also replies and responses to this
Lieberman, M. D., & Cunningham, W. A. (2009). Type I and Type II error concerns in fMRI research: re-balancing the scale. Social cognitive and affective neuroscience, 4(4), 423-428.
Annex II: extra readings on statistics and theory choice
More on statistics
Anderson, S. F., Kelley, K., & Maxwell, S. E. (2017). Sample-Size Planning for More Accurate Statistical Power: A Method Adjusting Sample Effect Sizes for Publication Bias and Uncertainty. Psychological Science, 28(11), 1547–1562. https://doi.org/10.1177/0956797617723724
Westfall, J., & Yarkoni, T. (2016). Statistically controlling for confounding constructs is harder than you think. PloS one, 11, e0152719.
More on theory choice
Roberts, S., & Pashler, H. (2000). How persuasive is a good fit? A comment on theory testing. Psychological review, 107(2), 358.
Newell, A. (1973). You can't play 20 questions with nature and win: Projective comments on the papers of this symposium.
Mather, M., Cacioppo, J. T., & Kanwisher, N. (2013). How fMRI can inform cognitive theories. Perspectives on Psychological Science, 8(1), 108-113.
Thagard, P. (2009). Why cognitive science needs philosophy and vice versa. Topics in Cognitive Science, 1(2), 237-254.