In 2017, Lenny Teytelman, CEO of protocols.io, organized a panel to discuss reproducibility issues in research. But he realized that it wasn’t enough to discuss the age-old problem of irreproducibility in science or even to discuss potential solutions. Despite all the talk, not much was being done to address the issue head on. It was at this pivotal moment that Teytelman realized that running interactive workshops to train researchers on tools and best practices could be an actionable way to tackle widespread irreproducibility. Luckily, there were other academics and like-minded organizations with similar ideas who were also thinking about reproducibility and shared similar desires to take action. Thanks to Teytelman’s vast network and ability to identify the right opportunities, he was able to bring together talented and motivated groups and individuals with similar ideas to actually do something about it.
Building a diverse team with different talents and expertise
Fortunately, the eLife ambassadors group was already thinking about ways to tackle irreproducibility. Benjamin Schwessinger, a postdoc at Berkeley at the time and an eLife ambassador, had previous experience organizing open access events and grassroots efforts. After thoughtful conversations about reproducibility with Teytelman, Schwessinger joined this cause and helped by expanding this team, leading monthly calls, and planning next steps.
April Clyburne-Sherin had already run several reproducibility workshops as Director of Scientific Outreach at Code Ocean and through her previous work at Center for Open Science and Sense about Science. However, she realized that she couldn’t run enough workshops by herself to make the biggest impact. When she spoke with Teytelman about an opportunity to work with a collaborative group, she also joined early on and contributed her expertise in developing reproducibility workshops and organizational skills.
Meanwhile, Addgene was also thinking about ways to address irreproducibility and thanks to Teytelman’s connection with Joanne Kamens, Addgene also had the opportunity to contribute to this open science collaboration.
The first iteration of these workshops took place at Caltech in Jan 2018 and involved presentations and a handout on protocol sharing, material sharing, computational reproducibility, and tools to enable scientific reproducibility. This workshop was well received, and then when the team started brainstorming ideas for the next workshop, they realized that holding the next one at a scientific conference could potentially enable reproducibility practices being adopted by a wider group of researchers.
Generating material to empower others to practice and teach reproducibility
An early milestone for the group was the “reproducibility for everyone” workshop at the July 2018 ASPB meeting in Montreal. This 90-minute workshop covered many tools and best practices for reproducibility: data management plans, electronic lab notebooks, wet lab protocols, reagent sharing, bioinformatic tools, data and code sharing, and data analysis and visualization. Not everyone in the group was able to make it physically to instruct the workshop at the conference, but because each contributor made the materials accessible, anyone could present the material.
For example, Addgene’s contributions by Tyler Ford included reagent sharing and were presented at the ASPB meeting by the collaborators who could participate. Despite the group not having a public website, all the slides, handouts, and resources were hosted on the Plantae website blog post written by Ben Schwessinger, which resulted in greater access and usage of the materials. To provide greater visibility of these materials, Sonali Roy, a postdoctoral researcher at the Noble Research Institute, co-instructed the ASPB workshop and authored a three-part blog post on the materials that were covered in the workshop.
These workshops illustrate the open and modular nature of the workshop content: different parts could be mixed and matched to serve a specific audience. For example, I joined this effort later in the year and in the fall of 2018, I helped organize a workshop for community scientists at the Global Community Bio Summit at the MIT Media Lab. As a co-founder of a community lab, I realized that not all the materials would be relevant to everyone at this specific conference, and I was able to include and exclude different modules for this workshop.
Empowering others to have the materials and resources to run their own local workshops was a major goal for the team last year. The materials were all out there, allowing others to have the resources for running their own workshops: April Clyburne-Sherin’s and Tyler Ford’s How-to-guide covers everything that you would want to know before, during, and after organizing such a workshop. Finally, the group put a lot of effort into ensuring that the content of the materials and the workshops themselves were assessed and could improve over time. They achieved this by administering pre- and post-workshop surveys, recording results, and tracking material downloads. This was particularly helpful for securing funding to support this effort.
The future of reproducibility workshops and lessons learned:
Since early 2018 the team has grown to include experts who could help provide new materials and reproducibility tools. This team was also recently awarded a Mozilla Science mini-grant, which will fund a project to support a dedicated website to host all the materials in one place and better serve researchers at all levels of the scientific community.
Lessons learned
-
Instead of continuously talking about a problem and potential solutions, take actionable steps - In the case of this collaboration, the “Reproducibility for Everyone” team was assembled when it was realized that interactive workshops that train and provide researchers with tools and best practices could be an actionable way to address the widespread issue of irreproducibility.
-
Use your network well to seek out valuable and impactful opportunities - There are many brilliant minds out there, but how do you get them to connect? Through networking of course! Thanks to individuals in the group who were well connected, this diverse and international team assembled the right people at the right time.
-
Be inclusive and seek solutions by working with communities, not for them - Another strength of this collaboration is that it is driven by academics, with support of industry partners, not the other way around. Sometimes, due to misconceptions and stereotypes, solving a community’s issue without involving them can do more harm than good. Check out this webinar for more on this.
Be on the lookout for upcoming workshops at future conferences this year. Workshops are confirmed for the Evolution meeting, ASM meeting, and IS-MPMI. If you’re interested in having a workshop organized at your upcoming conference or you’d like to join or contribute to this collaboration, please check out the official website, follow the group on Twitter (@repro4everyone), or email the team.
Acknowledgements
One of the highlights of 2018 for me was to collaborate with this incredibly dedicated team. Many thanks to Tyler Ford, Caroline LaManna, Joanne Kamens, Lenny Teytelman, April Clyburne-Sherin, Benjamin Schwessinger for helpful conversations on the history of the group, lessons learned, and to Sonali Roy, Magdalena Julkowska, and Jennifer Tsang for editing.
Additional resources on the Addgene blog:
- Check out this guide on Transferable Skills: External Collaborations
- Learn about the 2018 Minisymposium on Scientific Reproducibility
- Read a blog post on Unique and persistent IDs for improved reproducibility
Other resources:
Topics: Scientific Sharing, Reproducibility
Leave a Comment