PSY6316 Current Issues in Cognitive Neuroscience

Semester 2, Academic Year 2017-18. Topic:

“How reliable is cognitive neuroscience?”

What has been called The Replication Crisis has sparked widespread introspection about the standards and protocols of science, particularly within the behavioural sciences. This course, though reading a series of landmark papers and class discussion, will consider to extent to which doubts about the reliability of empirical work affect cognitive neuroscience. Can we trust the published papers in this field? Are the effects which we investigate reliable? If not, how can work in cognitive neuroscience be made more trustworthy?

This is a discussion led module so your contribution to the classes is required. You must do the reading in advance if you expect to get anything from the classes. There is also considerable flexibility in the timetable, so if there is an area you think would be useful to discuss in class, please suggest this


0. Background

Chambers, C. (2017). The seven deadly sins of psychology: A manifesto for reforming the culture of scientific practice. Princeton University Press.

1. Replication

Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7, 531-536.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.

Etz, A., & Vandekerckhove, J. (2016). A Bayesian perspective on the reproducibility project: Psychology. PLoS One, 11(2), e0149794.

Crandall, C. S., & Sherman, J. W. (2016). On the scientific superiority of conceptual replications for scientific progress. Journal of Experimental Social Psychology, 66, 93-99.

2. replicability of fMRI

Wager, T. D., Lindquist, M. A., Nichols, T. E., Kober, H., & Van Snellenberg, J. X. (2009). Evaluating the consistency and specificity of neuroimaging data using meta-analysis. Neuroimage, 45(1), S210-S221.

Bennett, C. M., & Miller, M. B. (2010). How reliable are the results from functional magnetic resonance imaging?. Annals of the New York Academy of Sciences, 1191(1), 133-155.

Poldrack, R. A., Baker, C. I., Durnez, J., Gorgolewski, K. J., Matthews, P. M., Munafò, M. R., ... & Yarkoni, T. (2017). Scanning the horizon: towards transparent and reproducible neuroimaging research. Nature Reviews Neuroscience, 18(2), 115-126.

3. analytic flexibility: the dead salmon in the scanner

Bennett, C. M., Miller, M. B., & Wolford, G. L. (2009). Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: An argument for multiple comparisons correction. Neuroimage, 47(Suppl 1), S125.

Carp, J. (2012). The secret lives of experiments: methods reporting in the fMRI literature. Neuroimage, 63(1), 289-300.

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science, 22(11), 1359-1366.

4. Statistical inference & multiple comparisons

Luck, S. J., & Gaspelin, N. (2017). How to get statistically significant effects in any ERP experiment (and why you shouldn't). Psychophysiology, 54(1), 146-157.

Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304-1312.

Rouder, J. N., Morey, R. D., Verhagen, J., Province, J. M., & Wagenmakers, E. J. (2016). Is there a free lunch in inference? Topics in Cognitive Science, 8, 520-547.

Brain Voyager tutorial pages

Lindquist, M. A., & Mejia, A. (2015). Zen and the art of multiple comparisons. Psychosomatic medicine, 77(2), 114.

Han, H., & Glenn, A. L. (2017). Evaluating methods of correcting for multiple comparisons implemented in SPM12 in social neuroscience fMRI studies: an example from moral psychology. Social neuroscience, 1-11.

5. Four classics

Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Med, 2, e124.

Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on psychological science, 4(3), 274-290.

Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013)

Gelman, Andrew, and Eric Loken. "The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time." Department of Statistics, Columbia University (2013).

6. Case study:  Ego-depletion

Baumeister, Roy F., et al. "Ego depletion: Is the active self a limited resource?." Journal of personality and social psychology 74.5 (1998): 1252.

Gailliot, M. T., Baumeister, R. F., DeWall, C. N., Maner, J. K., Plant, E. A., Tice, D. M., Brewer, L. E., & Schmeichel, B. J. (2007). Self-control relies on glucose as a limited energy source: Willpower is more than a metaphor. Journal of Personality and Social Psychology, 92, 325–336.

Hagger, M. S., Wood, C., Stiff, C., & Chatzisarantis, N. L. (2010). Ego depletion and the strength model of self-control: a meta-analysis. Psychological bulletin, 136(4), 495.

Hagger, M. S., Chatzisarantis, N. L., Alberts, H., Anggono, C. O., Batailler, C., Birt, A. R., ... & Calvillo, D. P. (2016). A multilab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science, 11(4), 546-573.

Sjåstad, H. and Baumeister, R.F. (2018). The Future and the Will: Planning requires self-control, and ego depletion leads to planning aversion. Journal of Experimental Social Psychology, 76, 127-141. doi: 10.1016/j.jesp.2018.01.005

7. Case study: the case of precognition

Bem, D. J. (2011). Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. Journal of personality and social psychology, 100(3), 407.

Why the Journal of Personality and Social Psychology Should Retract Article DOI: 10.1037/a0021524 “Feeling the Future: Experimental evidence for anomalous retroactive influences on cognition and affect” by Daryl J. Bem

8. Case study: Amy cuddy & power posing

Carney, D. R., Cuddy, A. J., & Yap, A. J. (2010). Power posing: Brief nonverbal displays affect neuroendocrine levels and risk tolerance. Psychological science, 21(10), 1363-1368.

When the Revolution Came for Amy Cuddy

9. Reproducibility & validity

Georgescu, C., & Wren, J. D. (2017). Algorithmic identification of discrepancies between published ratios and their reported confidence intervals and p-values. Bioinformatics, btx811.

Computational science: ...Error

A guide to reproducible code

Computational practices for reproducible science

Borsboom, D., Cramer, A. O., Kievit, R. A., Scholten, A. Z., & Franić, S. (2009). The end of construct validity. In The Concept of Validity: Revisions, New Directions and Applications, Oct, 2008.

10. Solutions I: Psychology Theory

Meehl, P. E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195-244.

Weisberg, D. S., Keil, F. C., Goodstein, J., Rawson, E., & Gray, J. R. (2008). The seductive allure of neuroscience explanations. Journal of cognitive neuroscience, 20(3), 470-477.

Bullock, J. G., Green, D. P., & Ha, S. E. (2010). Yes, but what’s the mechanism?(don’t expect an easy answer). Journal of Personality and Social Psychology, 98, 550-558.

Coltheart, M. (2013). How can functional neuroimaging inform cognitive theories?. Perspectives on Psychological Science, 8(1), 98-103.

Jonas, E., & Kording, K. P. (2017). Could a neuroscientist understand a microprocessor?. PLoS computational biology, 13(1), e1005268.

Yarkoni, T., & Westfall, J. (2017). Choosing prediction over explanation in psychology: Lessons from machine learning. Perspectives on Psychological Science, 12(6), 1100-1122.

10. Solutions II: Evaluating a body of literature


Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: a key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534.

See also :,

funnel plots:

Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. Bmj, 315(7109), 629-634.

Lau, J., Ioannidis, J., Terrin, N., Schmid, C., Olkin, I., 2006. The case of the misleading

funnel plot. Br. Med. J. 333, 597–600.


Brown, N. J., & Heathers, J. A. (2017). The GRIM test: A simple technique detects numerous anomalies in the reporting of results in psychology. Social Psychological and Personality Science, 8(4), 363-369.

SPRITE: “Introducing SPRITE (and the Case of the Carthorse Child)”


The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses, September 2016, John P.A. Ioannidis

Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G. (2018). Meta-analysis and the science of research synthesis. Nature, 555(7695), 175.

Solutions III: Open Science

Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., du Sert, N. P., ... & Ioannidis, J. P. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021.

Transparency and Openness Promotion (TOP) Guidelines.

Open Data

Wicherts, J. M., Borsboom, D., Kats, J. & Molenaar, D. The poor availability of psychological research data for reanalysis. Am. Psychol. 61, 726–728 (2006).

Counterpoints: “crisis? What crisis?”

Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016). Comment on “Estimating the reproducibility of psychological science”. Science, 351(6277), 1037-1037.

Fanelli, D. (2018). Opinion: Is science really facing a reproducibility crisis, and do we need it to?. Proceedings of the National Academy of Sciences, 115(11), 2628-2631.

Psychology Is Not in Crisis

Wellek, S. (2017). A critical evaluation of the current “p‐value controversy”. Biometrical journal.

Voelkl, B., & Würbel, H. (2016). Reproducibility crisis: are we ignoring reaction norms?. Trends in pharmacological sciences, 37(7), 509-510.

Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?. American Psychologist, 70(6), 487.

Stroebe, Wolfgang, and Fritz Strack. "The alleged crisis and the illusion of exact replication." Perspectives on Psychological Science 9, no. 1 (2014): 59-71.

On the evidentiary emptiness of failed replications

Jason Mitchell, Harvard University, 1 July, 2014

What Jason Mitchell's 'On the emptiness of failed replications' gets right

Fiske's "Mod rule or wisdom or crowds"

Inside Psychology’s ‘Methodological Terrorism’ Debate

Annex I: sources of unreliability

Non-independence error

Vul, E., & Kanwisher, N. (2010). Begging the question: The non-independence error in fMRI data analysis. Foundational issues for human brain mapping, 71-91.

(low) power

Szucs, D., & Ioannidis, J. P. (2017). Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLoS biology, 15(3), e2000797.

Geuter, S., Qi, G., Welsh, R. C., Wager, T. D., & Lindquist, M. A. (2018). Effect Size and Power in fMRI Group Analysis. bioRxiv, 295048.

publication (bias) / quirks of peer review, file draw effects

Higginson, A. D., & Munafò, M. R. (2016). Current incentives for scientists lead to underpowered studies with erroneous conclusions. PLoS biology, 14(11), e2000995.

Lee, C. J., Sugimoto, C. R., Zhang, G. and Cronin, B. (2013), Bias in peer review. J Am Soc Inf Sci Tec, 64: 2–17. doi:10.1002/asi.22784


John, L. K., Loewenstein, G. & Prelec, D. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23, 524–532 (2012).

Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196-217.


Culpepper, S. A., & Aguinis, H. (2011). Using analysis of covariance (ANCOVA) with fallible covariates. Psychological Methods, 16, 166-178.

False positives

Eklund, A., Nichols, T. E., & Knutsson, H. (2016). Cluster failure: why fMRI inferences for spatial extent have inflated false-positive rates. Proceedings of the National Academy of Sciences, 201602413.

-- see also replies and responses to this

Lieberman, M. D., & Cunningham, W. A. (2009). Type I and Type II error concerns in fMRI research: re-balancing the scale. Social cognitive and affective neuroscience, 4(4), 423-428.

Annex II: extra readings on statistics and theory choice

More on statistics

Anderson, S. F., Kelley, K., & Maxwell, S. E. (2017). Sample-Size Planning for More Accurate Statistical Power: A Method Adjusting Sample Effect Sizes for Publication Bias and Uncertainty. Psychological Science, 28(11), 1547–1562.

Leek, J., McShane, B. B., Gelman, A., Colquhoun, D., Nuijten, M. B., & Goodman, S. N. (2017). Five ways to fix statistics. Nature, 551(7682), 557-

Westfall, J., & Yarkoni, T. (2016). Statistically controlling for confounding constructs is harder than you think. PloS one, 11, e0152719.

More on theory choice

Roberts, S., & Pashler, H. (2000). How persuasive is a good fit? A comment on theory testing. Psychological review, 107(2), 358.

Newell, A. (1973). You can't play 20 questions with nature and win: Projective comments on the papers of this symposium.

Mather, M., Cacioppo, J. T., & Kanwisher, N. (2013). How fMRI can inform cognitive theories. Perspectives on Psychological Science, 8(1), 108-113.

Thagard, P. (2009). Why cognitive science needs philosophy and vice versa. Topics in Cognitive Science, 1(2), 237-254.