Michelle Barker (Research Software Alliance) 
Neil Chue Hong (University of Edinburgh)

Mars 2024

Executive summary

Reproducibility is recognised as a critical aim of modern research and is a part of major research reform agendas such as open science. It is a critical factor in both ensuring the quality and integrity of research, and accelerating discovery. However, the diversity of digital research processes and outputs, including research data and software, create challenges to the creation of reproducible research outcomes. There are also a range of stakeholders in the international research ecosystem who have different roles to play in supporting and incentivising reproducibility, including researchers and research-adjacent support staff, funders, policy makers, publishers, and institutional managers.

This report identifies how reproducibility can be scaled up at research organisations. It was commissioned by Knowledge Exchange, following previous work on open science, to investigate how the practice of conducting research in a reproducible way can be scaled up from pioneers to the majority of researchers and researchadjacent support staff. The report focuses on mesolevel factors, such as groups, organisations and communities, rather than micro- or macro-level factors (as defined in the Knowledge Exchange Open Science Framework), to understand the role of research organisations and the people within them (who both undertake and support research). The findings in this report are the result of a mixed methods approach study, which combined a literature review, survey, interviews and focus groups.

The major output of this work is a framework for understanding approaches to scaling up reproducibility in research institutions. The framework can be used by a range of internal stakeholders with differing goals, such as institutional leaders seeking to align organisational strategies, or managers wishing to provide the support that staff in their part of the organisation may be seeking. The intention is also to enable dialogue between managers and researchers to create collaborative and sustainable solutions for a wider uptake of reproducible research practices. It should be noted that this framework is focused on how well organised an organisation is at scaling up reproducibility practices, not the maturity of reproducibility practices and how well they adhere to what is commonly understood to be best practice.

The framework consists of three parts:

1. Organisational levels: These are levels that an organisation may progress through in its scaling up of reproducibility, and are focused on internalaspects of the organisation.

2. Enablers of scaling up reproducibility: Seven major types of enablers (based on the taxonomy by Davidson et al. (2022) support or catalyse the transition from one level to another, through a variety of interventions:

• Tools
• Education and training in research reproducibility
• Incentives to enhance awareness, accessibility and understanding
• Modelling and mentoring to encourage research reproducibility
• Review and feedback 
• Expert involvement and advice 
• Policies and procedures

3. Assessment worksheet: This allows an organisation to assess its capability to support reproducibility practices, and act as a starting point for discussions around maintaining or improving this capability. It is complemented by guidelines for usage.

This report does not provide direct recommendations to the reader, as there is no single set of interventions that work at all types of organisation. Instead, it should be used to enable research organisations to engage with those involved in reproducibility to help share and extend good practice. An infographic aimed at key stakeholders has also been produced to make it easier to disseminate the outcomes of this study. It will be important to ensure that the majority of researchers are provided with appropriate enablers and interventions, if culture change around reproducibility is to be achieved.

A shorter version of the framework, including guidance and worksheet, is also available as a separate document (10.5281/zenodo.10664660) for use by stakeholders. A useful next stage would now be for the community to engage with the framework, to enable testing and evaluation to increase its value. This could also provide better understanding of the importance of community in transitioning between levels; however, the reality of the status of reproducibility is highly varied not only across the research ecosystem internationally, but also across research organisations, and even within organisations, faculties and teams. The ongoing work of both national reproducibility networks and coordination across these provide one avenue for possibly supporting this, with university consortia providing another.