Suite 4.12, Ashworth House, 18 Kingsley Road, London EC2A 4NE, United KingdomLondon, United Kingdom
  1. Home
  2. Blog
  3. Reproducibility: The Keystone of Credible Research

Reproducibility: The Keystone of Credible Research

A finding that cannot be reproduced is a claim, not knowledge. Here is why reproducibility matters, why it fails, and six habits that make your own work easier to verify.

Reproducibility: The Keystone of Credible Research

In an arch, the keystone is the last stone set at the top. Until it is in place the arch cannot stand on its own; once it is there, every other stone is held firmly. Reproducibility plays the same part in research. Without it a study is a collection of interesting claims. With it, other people can build on the work with confidence.

Reproducibility means that someone else, given your data and your methods, can arrive at the same results. Replication goes a step further: a new team collects new data and checks whether the effect appears again. Both are central to how science corrects itself, and both have been under strain for more than a decade.

A problem that is hard to ignore

In 2015 the Open Science Collaboration attempted to replicate 100 studies published in three leading psychology journals. Fewer than half of the replications produced statistically significant results, and the average effect size was roughly half that of the originals. A year later, a survey of more than 1,500 scientists by Nature found that over 70% had tried and failed to reproduce another scientist's experiment, and more than half had failed to reproduce one of their own.

These numbers do not mean that most research is wrong. They do show that published results are often less certain than they look, and that the problem is spread across disciplines, from cancer biology to economics.

Reproducibility is not an extra task added at the end of a project. It is a way of working from the first day.

Why results fail to reproduce

Fraud gets the headlines, but it explains only a small share of the problem. The usual causes are more ordinary:

  • Small samples. Underpowered studies produce unstable estimates, and the ones that cross the significance threshold tend to overstate the true effect.
  • Flexible analysis. Choosing among many reasonable analysis paths after seeing the data, sometimes called the "garden of forking paths", inflates false positives without anyone intending to mislead.
  • Publication bias. Journals and authors favour positive, novel results, so null findings stay in file drawers and the published record becomes lopsided.
  • Missing detail. Methods sections often leave out the small decisions (exclusion rules, software versions, random seeds) that change the outcome.
  • Unavailable data and code. When the materials are not shared, nobody can check the analysis, however well it was done.

Six habits that make your work reproducible

None of the following requires a large budget. Most of it is planning and discipline.

1. Preregister your plan

Write down your hypotheses, sample size and main analysis before you collect or look at the data, and register them on a platform such as OSF Registries or ClinicalTrials.gov. Preregistration does not forbid exploratory work. It simply makes clear which analyses were planned and which were discovered along the way.

2. Justify your sample size

Run a power analysis, or explain the practical limits on your sample. Reviewers increasingly ask for this, and it protects you from reporting effects that are too noisy to trust.

3. Script everything

Point-and-click analysis is hard to repeat. A script in R, Python or Stata records every step from raw data to final table. Keep raw data read-only and let the script produce all cleaned files.

4. Use version control

Git, through GitHub or GitLab, keeps a history of every change to your code and documents. It ends confusion over files named final_v3_REAL.docx and makes collaboration far easier.

5. Share data and code, with a licence

Deposit your materials in a repository such as Zenodo, OSF or a discipline-specific archive, and cite the DOI in your paper. Where data are sensitive, share a de-identified version or a clear description of how others can apply for access.

6. Report fully

Follow the reporting guideline for your design: CONSORT for trials, STROBE for observational studies, PRISMA for systematic reviews. These checklists exist because the same omissions appear again and again.

Quick test: could a capable colleague, with only your paper and your shared files, reproduce your main table in an afternoon? If not, something is missing.

What journals can do

Authors cannot fix this alone. Journals shape incentives through what they ask for and what they reward. Across the KEYSTONE Research Press journals we are building in data-availability statements, encouragement of preregistered and registered-report submissions, and a willingness to publish well-designed studies with null results. A sound method should count for more than an exciting outcome.

The payoff

Reproducible work takes more effort at the start, and it pays back over time. Your own future self can rerun an analysis months later. Reviewers raise fewer questions. Other researchers cite and reuse your materials. Above all, findings built this way are likely to hold up when others test them.

References

  1. Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
  2. Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature, 533, 452–454.
  3. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.
  4. Munafò, M. R. et al. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021.
KSR Press Editorial Team

Articles from the editors and tutors at KEYSTONE Research Press on research methods, publishing and AI.