This page is a growing list of potentially useful material if you want to become familiar with the topic of reproducible computational research. Start with the basic topics at the top and find more detailed, special, and advanced content further below. If you're looking for guidelines or surveys for your discipline/area of research, see Reproducible research in ..., and if you're looking for help with a specific piece of software or programming languages, visit Reproducible research with ....
🤔 What to know? The basics of reproducible research.
The concept of a "research compendium". Start with this Turing Way chapter, skim the original paper coining the term, and then check out more information at https://research-compendium.science/, e.g., something from your discipline.
📖 What to read, watch, or listen to?
⏱️ I have only 20 minutes!
- Becoming a better scientist with open and reproducible research (2) by Laurent Gatto: https://lgatto.github.io/open-and-rr-2/
- naming things by Jenny Bryan: http://www2.stat.duke.edu/~rcs46/lectures_2015/01-markdown-git/slides/naming-slides/naming-slides.pdf
- "The Golden Rule of Naming Files and Other Things: Thou shalt get only as creative with names as thy own skill with regular expressions." (twitter.com/JennyBryan)
- Write an email to three colleagues with a link to this page and challenge them to do the one hour reading list together with you.
⏳ I have only one hour!
- Read the Overview of The Turing Way's Guide for Reproducible Research and skim the table of contents so you get an overview of available resources. This handbook is a great resource to bookmark!
One of the classic papers: Roger D. Peng, 2011. Reproducible Research in Computational Science. Science. https://doi.org/10.1126/science.1213847
- Markowetz, F. Five selfish reasons to work reproducibly. Genome Biol 16, 274 (2015). https://doi.org/10.1186/s13059-015-0850-7
(If you want to save on time, put on this video while you're doing the laundry or cooking:) - Print it out and put in on your desk: Helmholtz Open Science Factsheet No. 2: Practical Steps Towards Open and Reproducible Research. https://doi.org/10.48440/os.helmholtz.025
📅 I have half a day!
- All of the above, then:
- Top Tips to Make Your Research Irreproducible: https://arxiv.org/abs/1504.00062
- Yasemin Turkyilmaz-van der Velden, Nicolas Dintzner, Marta Teperek. 2020. Reproducibility Starts from You Today. Patterns,Volume 1, Issue 6. https://doi.org/10.1016/j.patter.2020.100099
- Jeffrey M. Perkel | Challenge to scientists: does your ten-year-old code still run? https://doi.org/10.1038/d41586-020-02462-7
- Includes a REPRODUCIBILITY CHECKLIST
- Ten Simple Rules for Reproducible Computational Research: https://doi.org/10.1371/journal.pcbi.1003285
- Know that Reproducibility for Everyone and ReproHack exist, sign up for one of their events
📆 I have one full day!
- All of the above, then:
- Ten simple rules for quick and dirty scientific programming. https://doi.org/10.1371/journal.pcbi.1008549
- Reproducibility: let’s get it right from the start. Nat Commun 9, 3716 (2018). https://doi.org/10.1038/s41467-018-06012-8
- Good enough practices in scientific computing: https://doi.org/10.1371/journal.pcbi.1005510
- Somers, J. 2018, The Scientific Paper Is Obsolete. Here’s what’s next. The Atlantic. https://www.theatlantic.com/science/archive/2018/04/the-scientific-paper-is-obsolete/556676/
- Point of View: How open science helps researchers succeed: https://doi.org/10.7554/eLife.16800.001
♾️ I have all the time I need!
- 👉 The Turing Way's Guide for Reproducible Research 💯 from top to bottom
- https://repro4everyone.org/pages/explore-topics/ and a preprint describing the project:
Auer, S., Haelterman, N., Weissgerber, T. L., Erlich, J. C., Susilaradeya, D., Julkowska, M., … Jadavji, N. M. (2021). A community-led initiative for training in reproducible research. eLife 2021;10:e64719. https://doi.org/10.7554/eLife.64719 Benureau, F. C. Y., & Rougier, N. P. (2018). Re-run, Repeat, Reproduce, Reuse, Replicate: Transforming Code into Scientific Contributions. Frontiers in Neuroinformatics, 11. https://doi.org/10.3389/fninf.2017.00069
- Stodden, V. and Miguez, S., 2014. Best Practices for Computational Science: Software Infrastructure and Environments for Reproducible and Extensible Research. Journal of Open Research Software, 2(1), p.e21. DOI: http://doi.org/10.5334/jors.ay
- David L. Donoho, An invitation to reproducible computational research, Biostatistics, Volume 11, Issue 3, July 2010, Pages 385–388, https://doi.org/10.1093/biostatistics/kxq028
- Katherine Button | Reboot undergraduate courses for reproducibility: https://doi.org/10.1038/d41586-018-06692-8
- Peer, L., Orr, L., & Coppock, A. (2021). "Active Maintenance: A Proposal for the Long-Term Computational Reproducibility of Scientific Results." PS: Political Science & Politics, 1-5. https://doi.org/10.1017/S1049096521000366
- Gute (digitale) wissenschaftliche Praxis und Open Science (Helmholtz Open Science Briefing): https://doi.org/10.2312/os.helmholtz.012
- If you like watching videos, the RIOT Science YouTube channel has a lot of interesting material: https://www.youtube.com/c/RIOTScienceClub/videos
- Selected articles from the Ten Simple Rules collection, for example:
- Ten simple rules for responsible big data research: https://doi.org/10.1371/journal.pcbi.1005399
- Ten Simple Rules for Taking Advantage of Git and GitHub: https://doi.org/10.1371/journal.pcbi.1004947
- Principles for data analysis workflows: https://doi.org/10.1371/journal.pcbi.1008770
- Look for a study from your field on the Awesome Reproducible Research list and try to learn from it
- Sharing data and code during peer review: https://www.cambridge.org/core/blog/2019/08/19/how-to-make-the-data-and-code-for-your-manuscript-available-to-peer-reviewers-before-making-it-public/
🎧 I have no time but a long commute
- Everything Hertz podcast: https://everythinghertz.com/ (also citable and shownotes via https://osf.io/zj7y3/)
📃 Journals to know
- ReScience C
- Software Journals
- JOSS
- JORS
- Open Research Europe has a Software Tool Article type: https://open-research-europe.ec.europa.eu/for-authors/article-guidelines
- List of journals that accept publications primarily about software: https://www.software.ac.uk/which-journals-should-i-publish-my-software
- CiSE (Computing in Science & Engineering) has a special track for reproducible original research, case studies of reproductions, and tools/techniques/methods: https://www.computer.org/digital-library/magazines/cs/track-calls-for-papers
🧰 Platforms to know (at WWU Münster)
- ZIVGitLab: https://zivgitlab.uni-muenster.de/
- JupyterHub: https://jupyterhub.wwu.de/
- NWZ Morfeus: https://www.uni-muenster.de/NWZ/Angebot/ScientificComputing/Morfeus/
☑️ Other knowledge bases, lists, and courses
- 🚀 👍 A curated list of reproducible research case studies, projects, tutorials, and media: https://github.com/leipzig/awesome-reproducible-research
- 👍👍 "Reproducibility for Data Science" course that is part of the UW Master of Science in Data Science: https://canvas.uw.edu/courses/1354201
- Open Science MOOC: https://opensciencemooc.eu/
- Open Scholarship Knowledge Base: https://www.oercommons.org/hubs/OSKB
- FOSTER Open Science Toolkit: https://www.fosteropenscience.eu/toolkit
- https://research-compendium.science
📇 Who to follow?
- https://twitter.com/repro4everyone
- https://twitter.com/ReproHack/
- https://twitter.com/hashtag/ReproducibleResearch
- J. Perkel is a science journalist worth following, see https://www.nature.com/search?order=date_desc&q=%22Jeffrey+M.%2BPerkel%22
🤷🏘️ What to do at my group/lab/institute/department?
- Lowndes, J., Best, B., Scarborough, C. et al. Our path to better science in less time using open data science tools. Nat Ecol Evol 1, 0160 (2017). https://doi.org/10.1038/s41559-017-0160
- Organise a ReproHack (also works remotely!)
⚙️ Research Software
- DFG Handreichung Forschungssoftware
Muster-Richtlinie Nachhaltige Forschungssoftware an den Helmholtz-Zentren
- https://choosealicense.com/
- Open Science MOOC: Open Research Software and Open Source
You think something is missing on this page? Get in touch with o2r.support@uni-muenster.de. Thanks!