A personal highlight from 2020 - ReproducibiliTea

I have been doing a bit of reflection on the year, and one of the most rewarding things I did in 2020 was founding and organizing the ReproducibiliTea Journal Club at the University of Chicago! ReproducibiliTea started as a grassroots initiative in 2018 and has since become a global community centered on meta-science, discussing and improving research practices in response to the reproducibility crisis in science.

What's the most favourite thing you've done this year ECRs? I think mine is starting a ReproducibiliTea Journal Club and growing a community that is dedicated to improving their research practices and to open science! #ECRchat

— William Ngiam | 严祥全 (@will_ngiam) December 15, 2020

With repeated failures of large-scale projects to replicate scientific findings that had been published in high-ranking journals, there has been significant concern about whether the current process of science is flawed. Some have even posited that the majority of published findings are false. It is my understanding that science has had various waves of concerns about reproducibility come and go in its history, but this particular iteration of a “credibility revolution” appears to be the one that has stuck and been the most impactful, aided by the advent of the internet.

My fortunate beginnings

I was extremely fortunate to have Patrick Goodbourn and Alex Holcombe as my supervisors during my Honours and PhD for a multitude of reasons. One of them was learning of the reproducibility crisis when I started postgraduate research and inheriting their good research practices, such as posting preprints and ensuring experimental code and data open. Even prior to starting my PhD, Pat had me help collect data on a replication of a repetition blindness experiment. I had naively thought replications to be routine in science but I learnt later that it was actually part of the Reproducibility Project: Psychology, a project that conducted replications of 100 published findings but only managed to replicate 39. It would only be after a couple more years that I would truly appreciate the significance of that project and what it meant for the field of psychology and science.

Alex has been a vocal advocate and innovator for open and transparent science for a long time. It occurred to me that I might have been completely unaware of the reproducibility crisis had I not been placed in Alex’s lab – it was never mentioned in any statistics or research methods classes I had taken, there wasn’t any community around open science or reproducibility, nor were graduate students trained or audited to ensure their research practices were robust and open. With research practices often passed down within lab, graduate students could unwittingly perpetuate questionable research practices.

Doing something about it!

I was concerned that many of my fellow graduates could move on in science without being aware of the multitude of problems in science (publication bias, failed replications, questionable research practices, lack of reproducibility – just to list a few) and the recent reforms aimed at fixing those problems (preregistration, registered reports, contributorship models, open science – again, just to list a few). Seeing a similar lack of directive about reproducible science at my new institution (outside of my current lab), I was motivated to start something to empower early-career researchers.

ReproducibiliTea – a low-key weekly meeting for any like-minded early-career researcher to come along and learn about the reproducibilty crisis or anything regarding metascience – seemed like a good solution for my concerns. (The ReproducibiliTea founders had similar thoughts as well!) A journal club has a relatively low startup effort cost and thankfully I was not alone in my thinking – a few other ECRs wanted to help and became co-organizers with me. Only with a month or so of planning, we figured out a schedule of relevant research papers, and settled on a format where we would have someone lead with a brief presentation of the paper before opening it up for discussion. (I even uploaded my presentations to YouTube!). We’ve even had watch-parties of talks (like this one by Dr Julia Rohrer for the RIOT Science Club), co-working sessions dedicated to improving reproducibility in our own research, and workshops on learning Github!

Why has it been so rewarding?

Sharing the work of keeping up with relevant publications and current discussions on metascience is definitely helpful and a primary aim of the journal club. I have read more papers and covered more content about the credibility revolution than in any previous year, and learnt so much. There have been so many changes to the process of science (that are still changing and developing) and new tools and statistical analyses to keep up with and utilize like pre-registration, Registered Reports, preprints, new open-access journals, contributorship models, open peer review, Bayesian statistics, multiverse analysis and so many more – and they all can greatly impact how early-career researchers do science. Having a community of early-career researchers means sharing experiences with any of those processes or tools and reaching consensus about them, especially if we can share our expertise with each process and collaborate!

But by far the most rewarding part of organizing the journal club is growing the community of early-career researchers that spans across different subdisciplines of psychology and neuroscience. Even with the global pandemic (we didn’t get the chance to have an in-person meeting!), our virtual meetings brought together ECRs across labs and disciplines and is still growing! I had underestimated the importance of community when starting out but having my opinions and experiences validated by other early-career researchers, and being empowered to improve the reproducibility of my research practices was so valuable. With the many problems and obstacles to reproducible and rigorous research in the current academic system, it can be easy to become despondent about the state of science – so having the community of next-generation researchers helped me believe in this credibility revolution actually affecting change in how science is done.

Inspired? Do it!

If you had always thought about starting a ReproducibiliTea journal club, do it! There’s not much to it, it’s relatively straightforward and easy to get started – and I bet that you’re not alone about wanting to spread rigorous and reproducible science across your department! This link directs you to some materials and papers to start!