Advanced Notice: Research On Research Institute seeks postdocs

research on research institute logo

RoRi is looking for postdoctoral researchers to work on empirical research on research & metascience projects for the next five years. We anticipate opening applications for these positions in late October 2022.

RoRI brings together scholars, research funders and publishers to coordinate research on all aspects of research funding, evaluation, quality and publication. RoRI is now funded for a second phase (2022-2027), and is looking for a postdoctoral research associate with a background in experiment design, quantitative analysis, and/or metascience (research on research) to drive forward RoRI research. This post would be based at the University of Sheffield, working across all RoRI projects but based in the Department of Psychology.

This post would suit someone who wants to make a contribution to this exciting and growing area of research, helping to better understand a number of important questions, such as how research projects are selected, how robust research findings are produced, and how they can be effectively disseminated. We want to know how to improve the research ecosystem, managing the various biases that affect research evaluation and conduct. The appointed researcher will join a diverse and friendly team. In joining the University of Sheffield community, various resources will be available to the researcher, including those relating to additional training, mentoring, and support throughout the duration of the post. Our ambition will be to support the postdoctoral researcher to build skills, experience, and collaborations to launch a successful career in either metaresearch or research policy. The postdoc will have opportunity to creatively input to projects, and develop independent funding applications such as fellowship proposals, as well as lead authorship on project outputs including journal publications. There will be opportunities for travel and research visits with our international partners, as well as the chance to present research to academic audiences at conferences as well as other meetings involving RoRI stakeholders.

We expect to be able to advertise this position in late October 2022, aiming for a January 2023 start (ending 2027). The University supports hybrid working, with an expectation that you would be on campus 2 days a week, with flexibility around this. We welcome applicants from diverse intellectual and demographic backgrounds. Our team is strongly committed to work-life balance and we will support you to manage your time and not expect you to work outside of your paid hours. 

The University of Sheffield is a member of the Russell Group of research intensive universities, has one of the largest researcher communities in the UK and has been recognised as one of the top employers in the UK for LGBT+ workplace equality. Read more here

Informal enquiries are very welcome at this stage, please contact Tom Stafford on

Read more about RoRI here

Engaging Dialogue Generated From Argument Maps


Jan 2021 update: Project is go! Pages here: Opening Up Minds: engaging dialogue generated from argument maps. In Sheffield, we are lucky to be joined by Dr Lotty Brand

Starting January 2021, a two year post-doctoral research associate position. Skills required: experiment design, measure validation, online recruitment and testing, coding and/or statistical computing skills, background in psychology, linguistics or NLP generally, in reason, argument or dialogue specifically. Applications open later this year. Informal enquiries welcome at any point.

The EPSRC has funded our project ‘Opening Up Minds: Engaging Dialogue Generated From Argument Maps’, led by Paul Piwek (Computer Science, Open University), with myself, Andreas Vlachos (Computer Science, University of Cambridge) and Svetlana Stonyanchev (Toshiba Research Europe).

The idea is to design a “dialogue system” interface to existing databases of the arguments surrounding controversial topics such as “Should the United Kingdom remain a member of the European Union?” or “Should all humans be vegan?”. In particular, a user can have a “Moral Maze” style chat with the dialogue system.

“Moral Maze” is a longrunning popular BBC 4 Radio programme in which a panel discusses a controversial topic with the help of witnesses and a host who chairs the conversation. The dialogue system consists of a panel of Argumentation Bots (ArguBots) who present arguments for or against the topic under discussion (the pro and con ArguBots), a host ArguBot and a witness ArguBot (that can provide detailed evidence). The user is invited to join the panel and voice their views on the topic under discussion. Thus the user can explore what they thought and what others thought about the controversial topic.

An important part of the projects will be to evaluate the effects on people’s appreciation of the complexity of debate and attendant ability to comprehend the world from other people’s point of view or perspective.

The computer science research will focus on developing the dialogue agents (‘bots’) to allow users to explore controversial topics through natural language conversations. Our hope is such conversation can be engaging, and also free of the polarisation that we see in human-human interactions over social media around controversial topics.

My job will be to lead on WP3, which will look at evaluating how people experience the dialogues, and how their attitudes and beliefs are affected.

Here’s what we said about that in the proposal:

Work Package 3 – Evaluation (lead: Sheffield) Work on this WP will begin immediately with validation of the measures of open-mindedness, attitude strength and perception of argument coherence, and with the establishment of procedures for participant recruitment and testing. Importantly we need to develop an appropriate control condition which will act as a baseline against which any benefits of engaging with the argument-map via a ArguBots will be gauged. Development of the measures and control condition will also allow statistical power analysis to ensure that subsequent testing recruits enough participants to measure the effects of interest with sufficient accuracy. These can proceed before the full dialogue system development is finalised, using a Wizard-of-Oz (WoZ) protocol. As the dialogue system is developed, this WP will support continuous testing and feedback, allowing user behaviour to be integrated into development. The collected user utterances will be used as additional data to train and evaluate the components of the dialogue system. At fixed points, experiments will be conducted which test the impact of the dialogues on the participants at the three levels of 1) perception of coherence, 2) engagement and 3) impact on attitudes and beliefs. Because of the common cognitive bias to overestimate the extent of our insight into argument structure (see background), testing of the impact on attitudes and beliefs will use direct surveys, as well as before-after testing and novel implicit measures developed for the project and designed to test participants’ comprehension of opposing arguments (i.e. ability to pass the Ideological Turing Test). Ethics approval will be obtained for all data collection and evaluation with human subjects, and this will include necessary steps to mitigate ethical risks (e.g. including procedures for data deletion in the event that participants reveal personal information during the decision making task).

So we’ll be recruiting a post-doc for the project, to work with me in Sheffield and collaborate with the project partners at the OU and in Cambridge. The project starts mid January 2021 and applications will open later in year. I’ll have a better idea of the job specification then, but I expect the ideal candidate will have a background in experimental research with online platforms, be interested and/or informed about the psychology of reason, argument and dialogue and comfortable with interdiscipinary approaches (in particular working with NLP/Computer Science communities)

Informal enquiries are welcome at any time. Hit me up by email or on twitter.

Understanding online political advertising: perceptions, uses and regulation

The Leverhulme Trust has funded this project, led by Dr Kate Dommett (Department of Politics), and involving myself and Dr Nikos Aletras (Department of Computer Science).

Here’s the project abstract:

Microtargeted advertising is revolutionising political campaigning. Despite widespread adoption, and strong claims of efficacy, there is no systematic account of the rationale behind targeted ad campaigns, nor their perception by citizens. This lack impedes the design and implementation of an appropriate response from government or industry. Using voter surveys, in depth interviews with campaigners and analysis of online ad archives augmented by machine learning, this grant will explore the logic and practice of political advertising. It will place the regulation of political advertising within a broader framework of human rationality and the legitimate role of persuasion in politics.

It is due to start in January 2021, and we’ll be hiring two post-docs (each three year posts). One post-doc with a background in political science will conduct research interviews with advertisers, campaigners, policy makers and stakeholders (but also help me with experimental survey design, aimed at gauging public perceptions of targeted ads – the “folk theories” of how they work and should be regulated); the other post-doc will have a background in natural language process/machine learning and work on automated text and network analyses of ad archives and social media data. If you have a PhD, or will have a PhD by January, and could fill one of these positions, please get in touch now to discuss.


Open science essentials in 2 minutes, part 4

Before a research article is published in a journal you can make it freely available for anyone to read. You could do this on your own website, but you can also do it on a preprint server, such as, where other researchers also share their preprints. Psyarxiv allows others to find your research easily, and is supported by the OSF, which means it has the support to hold your papers for the long term.

Preprint servers have been used for decades in physics, but are now becoming more common across academia. Preprints allow rapid dissemination (and citation) of your research, which is especially important for early career researchers. Preprints can be cited and indexing services like Google Scholar will join your preprint citations with the record of your eventual journal publication.

Preprints also mean that work can be reviewed (and errors-caught) before final publication.

What happens when my paper is published?

Your work is still available in preprint form, which means that there is a non-paywalled version and so more people will read and cite it. If you upload a version of the manuscript after it has been accepted for publication that is called a post-print.

What about copyright?

Mostly journals own the formatted, typeset version of your published manuscript. This is why you often aren’t allowed to upload the PDF of this to your own website or a preprint server, but there’s nothing stopping you uploading a version with the same text (so the formatting will be different, but the information is the same).

Will journals refuse my paper if it is already “published” via a preprint?

Most journals allow, or even encourage preprints. A diminishing minority don’t. If you’re interested you can search for specific journal policies here.

Will I get scooped?

Preprints allow you to timestamp your work before publication, so they can act to establish priority on a findings which is protection against being scooped. Of course, if you have a project where you don’t want to let anyone know you are working in that area until you’re published, preprints may not be suitable.

When should I upload a preprint?

Upload a preprint at the point of submission to a journal, and for each further submission and upon acceptance (making it a postprint).

What’s to stop people uploading rubbish to a preprint server?

There’s nothing to stop this, but since your reputation for doing quality work is one of the most important things a scholar has I don’t recommend it.

Useful advice:

Put clear headers in your pre-print, noting the version and/or data, and the status of the pre-print (e.g. under review or published). When you are published you can upload a version with a header saying “Please cite as [xxxxxxx]”).

Useful links:

Part of a series:

  1. Pre-registration
  2. The Open Science Framework
  3. Reproducibility

Cross-posted at

Quit while you’re ahead: a surprising interaction between game performance and motivation

Over the last nine months I’ve been lucky enough to work with Dagmar Adamcová, who has been at the University of Sheffield on an Erasmus scheme internship during her MSc studies at Masaryk University in the Czech Republic.

Dagmar’s project focused on investigating an oddity in player behaviour from a simple online game I have data for: in this game the players tended to quit on a high score. The game is Axon, which I’ve published papers on previously, showing patterns in how people get better at the game with practice. Normally I average over players who play different numbers of games, and in the average of performance against play attempt we see the typical learning curve (people get better quickly at first, then the rate of increase slows down).

The odd pattern of behaviour which Dagmar looked at can be seen clearly if we divide players into subgroups who play exactly the same number of times, and plot their average performance:

(graph from Dagmar’s report)

As you can see, this isn’t a typical smooth learning curve. Players’ average performance *leaps* on their last game. What’s going on? Well Dagmar set out to investigate, and has published her analysis as a Jupyter notebook showing the analysis code, the results and the explanation of what she did.

What she found was evidence that unusually high scores let you predict games on which players will quit. Further, she found that predicting when players will quit is enhanced if you include a psychological definition of ‘high score’. Specifically, the ratio of any particular players latest score to their previous best allows better predictions of when they will quit.

The result is surprising because we normally assume that players of games like to win (and indeed, if success is rewarding we would normally predict that failure, not success, would lead to quitting). My theory is that players are “managing their hedonic experience”, or – as you might say in plain English – quitting while they are ahead.

We’d be interested to hear from anyone who has data which shows a similar interaction between performance and motivation. If you’ve seen a similar thing, please get in touch.

Read Dagmar’s full analysis in her notebook: Quit while you’re ahead: a surprising interaction between game performance and motivation.

Cyberselves: How Immersive Technologies Will Impact Our Future Selves

robodogWe’re happy to announce the re-launch of our project ‘Cyberselves: How Immersive Technologies Will Impact Our Future Selves’. Straight out of Sheffield Robotics, the project aims to explore the effects of technology like robot avatars, virtual reality, AI servants and other tech which alters your perception or ability to act. We’re interested in work, play and how our sense of ourselves and our bodies is going to change as this technology becomes more and more widespread.

We’re funded by the AHRC to run workshops and bring our roadshow of hands on cyber-experiences to places across the UK in the coming year. From the website:

Cyberselves will examine the transforming impact of immersive technologies on our societies and cultures. Our project will bring an immersive, entertaining experience to people in unconventional locations, a Cyberselves Roadshow, that will give participants the chance to transport themselves into the body of a humanoid robot, and to experience the world from that mechanical body. Visitors to the Roadshow will also get a chance to have hands-on experiences with other social robots, coding and virtual/augmented reality demonstrations, while chatting to Sheffield Robotics’ knowledgeable researchers.

The project is a follow-up to our earlier AHRC project, ‘Cyberselves in Immersive Technologies‘, which brought together robotics engineers, philosophers, psychologists, scholars of literature, and neuroscientists.

We’re running a workshop on the effects of teleoperation and telepresence, in Oxford in February (Link).

Call for papers: symposium on AI, robots and public engagement at 2018 AISB Convention (April 2018).

Project updates on twitter, via Dreaming Robots (‘Looking at robots in the news, films, literature and the popular imagination’).

Cross-posted at

Using Candy Crush to study perceptual learning

This is a guest post by Gabriela Raleva, who did a summer project with me in between her first and second years of the undergraduate degree.

gabrielaVisual learning refers to the enhanced sensitivity to visually relevant stimuli. Affective value of stimuli (e.g. reward, punishment), has been proposed to enhance action selection via instrumental learning (Hickey et al., 2010, Wilbertz et al., 2014).  The majority of studies in perceptual learning adopt an artificial approach of training participants in the lab for many sessions before testing them (although, see (Bavelier et al., 2012)). We used Candy Crush game sets as it represents an ideal platform for natural visual learning and assessed the performance of experienced players that have willingly engaged in a lot of practice hours as well as non-players.

In a between-subject design participants completed a visual search task, searching for a uniquely-shaped Candy Crush target among a number of nonhomogeneous Candy Crush distractors. Targets were divided into 4 conditions: neutral value, reward-value, punishment value and control condition (Figure 1). Reaction time for detection of targets was assessed and compared for each condition.

Figure 1. Target-present sets in all 4 conditions: (a) Neutral condition showing a single green candy (circled) which is a target of neutral consequences in the game, (b) Reward-associated containing a single multi-coloured bomb candy (circled) which is of positive consequences in the game, (c) Punishment-associated condition containing a blue bomb (circled) of negative consequences in the game, and (d) Control condition
Figure 1. Target-present sets in all 4 conditions: (a) Neutral condition showing a single green candy (circled) which is a target of neutral consequences in the game, (b) Reward-associated containing a single multi-coloured bomb candy (circled) which is of positive consequences in the game, (c) Punishment-associated condition containing a blue bomb (circled) of negative consequences in the game, and (d) Control condition

The results suggest that players and non-players revealed largely comparable responses in their detection of control, neutral, negative and positive stimuli (Figure 2). This indicates that the results are not due to self-selection bias. However, players were 35% faster at detecting rewarding targets than neutral targets stimuli. In contrast, non-players were on average 5% slower in detecting rewarding compared to neutral targets. Our analyses indicate that players reveal a consistent pattern of greater rewarding/neutral reaction time ratios than those of non-players consistent with the idea that features of affective-associated stimuli facilitate their perception in visual processing. Furthermore, the only Candy Crush condition in which players showed significantly slower reaction times than non-players is the neutral condition. One possible explanation is that players have developed better visual templates in regards to the game (Bejjanki et al., 2014) and therefore exhibit a visually holistic mode of performance (Green & Bavelier, 2003). Players may have learnt to quickly recognize patterns beneficial for the game such as the rewarding bomb. The green neutral candies compose a beneficial pattern only when they can be combined (at least 3) so it is possible that players learnt to recognize a single neutral candy as a distractor and thus suppress it more effectively. Such holistic expert performance is characterized by “chunking” – a process during which individual constituents are processed as a single perceptual or cognitive entity.

Figure 2. Mean reaction time of players and non-players in all 4 conditions as obtained by the target detection task.
Figure 2. Mean reaction time of players and non-players in all 4 conditions as obtained by the target detection task.

Bavelier, D., Green, C. S., Pouget, A., & Schrater, P. (2012). Brain plasticity through the life span: learning to learn and action video games. Annual Reviews of Neuroscience, 35, 391–416.
Bejjanki, V. R., Zhang, R., Li, R., Pouget, A., Green, C. S., Lu, Z.-L., & Bavelier, D. (2014). Action video game play facilitates the development of better perceptual templates. Proceedings of the National Academy of Sciences, 111(47), 16961–6.
Green, C. S., & Bavelier, D. (2003). Action video game modifies visual selective attention. Nature, 423(6939), 534-537
Hickey, C., Chelazzi, L., & Theeuwes, J. (2010). Reward guides vision when it’s your thing: Trait reward-seeking in reward-mediated visual priming. PLoS ONE, 5(11), 1–5.
Wilbertz, G., Van Slooten, J., & Sterzer, P. (2014). Reinforcement of perceptual inference: Reward and punishment alter conscious visual perception during binocular rivalry. Frontiers in Psychology, 5, 1–9.


Postscript from Tom:

One of the great difficulties of studying learning is that true expertise only comes after many many hours of practice. Psychologists often study perceptual learning in the lab with participants engaging in training with specific stimuli that last a few hours. The results of this project demonstrate the potential for using people who have already given themselves hundreds of hours of training with the specific stimuli as a side effect of a game they’ve played. To illustrate how large the effect is I plotted each participant’s Candy Crush level (so 0 for non-players) against the ratio rewarding / punishing stimulus RT : neutral stimulus RT. Even with the small number of participants the effect is clear – Candy Crush players are faster for the value-relevant stimuli relative to neutral stimuli, non-players aren’t.

The black line shows where participants’ reaction times should be if there are equally fast on the valuable Candy Crush stimuli as with neutral Candy Crush stimuli


New grant: ‘Neuroimaging as a marker of Attention Deficit Hyperactivity Disorder (ADHD)’

We have been awarded ~£11k by the White Rose Collaboration Fund. This will allow us to carry out a small neuroimaging study investigating brain activity associated with higher levels of ADHD traits. The collaboration combines expertise and facilities across the Universities of Sheffield, Leeds and York. Paul Overton has previously proposed that the subcortical area known as the superior colliculus may be crucial in ADHD. This is the focus of Maria’s PhD thesis (co-supervised by Paul and me). Jaclyn Billington from Leeds has experience imaging the colliculus, and Tony Morland is the deputy director of York’s neuroimaging facility (as well as having a wealth of experience imaging the areas associated with visual function). Alex Wade and Jeff Delvenne provide additional expertise in visual attention. I lead the project.

Here is the blurb:

We will create a unique network of expertise, personnel and facilities from across the WR network in order to establish a novel biomarker of Attention Deficit Hyperactivity Disorder (ADHD).

Despite a high prevalence (up to 10% of children by some estimates), ADHD remains controversial in terms diagnosis and treatment. Using brain scanning, this network aims to establish a biological marker common to all ADHD suffers. Such a biomarker could revolutionise our response to ADHD, allowing us to better understand the condition, diagnose earlier, manage the symptoms and target pharmacological interventions. This could potentially alleviate suffering and improve function for millions.

Theoretical direction for this proposal arises from Overton’s recent proposal that a core dysfunction in ADHD is hypersensitivity of the Superior Colliculus (SC), a key subcortical brain region known to play a critical role in attention, spatial orientation and saccadic eye movements. The development of this ‘collicular hypersensitivity’ hypothesis was possible because of the tradition of research into the fundamental neuroscience of subcortical structures at Sheffield.

This hypothesis has been taken forward by Stafford (Sheffield) who, with Panagiotidi, has been developing behavioural tests of collicular sensitivity. Early results show that healthy adults who are high and low on ADHD traits differ in these behavioural measures. However, behavioural tests are limited in that they cannot provide definitive insight into the neural basis of function. Teams in York and Leeds provide expertise in functional brain imaging and the neural basis of attention which would allow the direct translation of the Sheffield research programme into a test of a biomarker for ADHD.

Our primary objective will be to test two groups, high and low in ADHD traits for collicular responsiveness, using fMRI brain imaging. This testing will use behavioural measures which have been shown to discriminate the two groups, and analytic and imaging expertise from the Leeds and York based applicants in order to determine collicular responsiveness

New grant: Reduced habitual intrusions : an early marker for Parkinson’s Disease?

SurprisalDensityPlotFor4CharacterWindowI have very pleased to announce that the Michael J Fox Foundation have funded a project I lead titled ‘Reduced habitual intrusions : an early marker for Parkinson’s Disease?’. The project is for 1 year, and is a collaboration between a psychologist (myself), a neuroscientist (Pete Redgrave), a clinician specialising in Parkinson’s (Jose Obeso, in Spain) and a computational linguist (Colin Bannard, in Liverpool). Mariana Leriche will be joining us a post-doc.

The idea of the project stems from hypothesis that Parkinson’s Disease will be specifically characterised by a loss of habitual control in the motor system. This was proposed by Pete, Jose and others in 2010. Since my PhD I’ve been interested automatic processes in behaviour. One phenomenon which seems to offer particular promise for exploring the interaction between habits and deliberate control is the ‘action slip’. This is an error where a habit intrudes into the normal stream of intentional action – for example, such as when you put the cereal in to the fridge, or when someone greets you by asking “Isn’t it a nice day?” and you say “I’m fine thank you”. An interesting prediction of the Redgrave et al theory is people with Parkinson’s should make fewer action slips (in contrast to all other types of movement errors, which you would expect to increase as the disease progresses).

The domain we’re going to look at this in is typing, which I’ve worked with before, and which – I’ve argued – is a great domain for looking at how skill, intention and habit combine in an everyday task which generates lots of easily coded data.

I feel the project reflects exactly the kind of work I aspire to do – cognitive science which uses precise behavioural measurement, informed by both neuroscientific and computational perspectives, and in the service of am ambitious but valuable goal. Now, of course, we actually have to get on and do it.

Update 2019: Our paper from this project now published Bannard, C., Leriche, M., Bandmann, O., Brown, C. H., Ferracane, E., Sánchez-Ferro, A., … & Stafford, T. (2019). Reduced habit-driven errors in Parkinson’s Disease. Scientific Reports, 9(1), 3423.

New project: “Bias and Blame: Do Moral Interactions Modulate the Expression of Implicit Bias?”

The Leverhulme Trust has awarded a 36 month grant to the University of Nottingham, for a project led by my collaborator Dr Jules Holroyd, with support from myself. The project title is “Bias and Blame: Do Moral Interactions Modulate the Expression of Implicit Bias?” (abstract below). The aim is to conduct experiments to advance our understanding of how implicit biases are regulated by ‘moral interactions’ (these are things such as being blamed, or being held responsible). The grant will pay for a post-doc (Robin Scaife) in Sheffield and a PhD student (as yet unknown, let us know if you’re interested!) in Nottingham.

Obviously, this is something of a departure for myself, at least as far as the topic goes (which is why Jules leads). I’m hoping my background in decision making and training in experimental design will help me navigate the new conceptual waters of implicit bias. Some credit for inspiring the project should go to Jenny Saul and her Bias Project, and before that, Alec Patton and his faith in interdisciplinary dialogue that helped get Jules and myself talking about how experiments and philosophical analysis could help each other out.

Project Abstract:

This project will investigate whether moral interactions are useful tool for regulating implicit bias. Studies have shown that implicit biases – automatic associations which operate without reflective control – can lead to unintentionally differential or unfair treatment of stigmatised individuals. Such biases are widespread, resistant to deliberate moderation, and have a significant role in influencing judgement and action. Strategies for regulating implicit bias have been developed, tested and evaluated by psychologists and philosophers. But neither have explored whether holding individuals responsible for implicit biases may help or hinder their regulation. This is what we propose to do.