The reproducibility crisis and open science

Overview

Teaching: 25 min
Exercises: 0 min

Questions

• What is the reproducility crisi and how can be solved?

Objectives

• Understand the Open Science movement

The Reproducibility Crisis

More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments. Those are some of the telling figures that emerged from Nature’s survey of 1,576 researchers who took a brief online questionnaire on reproducibility in research.

The data reveal sometimes-contradictory attitudes towards reproducibility. Although 52% of those surveyed agree that there is a significant ‘crisis’ of reproducibility, less than 31% think that failure to reproduce published results means that the result is probably wrong, and most say that they still trust the published literature.

Failing to reproduce results is a rite of passage, says Marcus Munafo, a biological psychologist at the University of Bristol, UK, who has a long-standing interest in scientific reproducibility. When he was a student, he says, “I tried to replicate what looked simple from the literature, and wasn’t able to. Then I had a crisis of confidence, and then I learned that my experience wasn’t uncommon.”

The challenge is not to eliminate problems with reproducibility in published work. Being at the cutting edge of science means that sometimes results will not be robust, says Munafo. “We want to be discovering new things but not generating too many false leads.”

The complete section is taken directly without any modifications from this aricle which is cited below:

Baker M. 1,500 scientists lift the lid on reproducibility. Nature. 2016 May 26;533(7604):452-4. doi: 10.1038/533452a. PMID: 27225100.

Open Science

Open research aims to transform research by making it more reproducible, transparent, reusable, collaborative, accountable, and accessible to society. It pushes for change in the way that research is carried out and disseminated by digital tools. One definition of open research, as given by the Organisation for Economic Co-operation and Development (OECD), is the practice of making “the primary outputs of publicly funded research results – publications and the research data – publicly accessible in a digital format with no or minimal restriction.” To achieve this openness in research, each element of the research process should:

• Be publicly available: It is difficult to use and benefit from knowledge hidden behind barriers such as passwords and paywalls.
• Be reusable: Research outputs need to be licensed appropriately, so that prospective users know any limitations on re-use.
• Be transparent: With appropriate metadata to provide clear statements of how research output was produced and what it contains.

The research process typically has the following form: data is collected and then analysed (usually using software). This process may involve the use of specialist hardware. The results of the research are then published. Throughout the process, it is good practice for researchers to document their working in notebooks. Open research aims to make each of these elements open:

• Open Data: Documenting and sharing research data openly for re-use.
• Open Source Software: Documenting research code and routines, and making them freely accessible and available.
• Open Hardware: Documenting designs, materials, and other relevant information related to hardware, and making them freely accessible and available.
• Open Access: Making all published outputs freely accessible for maximum use and impact.
• Open Notebooks: An emerging practice, documenting and sharing the experimental process of trial and error.

Open Science is a concept that extends open research further. It relates to making other aspects of scientific research open to the public, for example:

• Open educational resources: Making educational resources publicly available to be re-used and modified.
• Equity, diversity, inclusion: Ensuring scholarship is open to anyone without barriers based on factors such as race, background, gender, and sexual orientation.
• Citizen science: The inclusion of members of the public in scientific research.

These elements are also discussed in detail in this chapter.

Motivation and Background

There are five main schools of thought motivating open practices to benefit research:

Another benefit of openness is that while research collaborations are essential to advancing knowledge, identifying and connecting with appropriate collaborators is not trivial. Open practices can make it easier for researchers to connect by increasing the discoverability and visibility of one’s work, facilitating rapid access to novel data and software resources, and creating new opportunities to interact with and contribute to ongoing communal projects.

The last two sections were taken directly without any modifications from _the turing way_book which is cited below:

The Turing Way Community, Becky Arnold, Louise Bowler, Sarah Gibson, Patricia Herterich, Rosie Higman, … Kirstie Whitaker. (2019, March 25). The Turing Way: A Handbook for Reproducible Data Science (Version v0.0.4). Zenodo. http://doi.org/10.5281/zenodo.3233986

Key Points

• Open Science practices aims to transform research by making it more reproducible, transparent, reusable, collaborative, accountable, and accessible to society