Toma Susi, Monica Heintz, Eva Hnatkova, Wolfram Koch, Maria Leptin, Martin Andler, Marco Masi, Michele Garfinkel

Open Science is a broad approach to improve the reproducibility, transparency, and robustness of research. By enabling broader access to data, code, methods and publications, it has the potential to increase the efficiency and impact of public funding of research, and societal engagement. Although some aspects are being implemented by specific funders and organisations as well as many individual researchers, Open Science overall is still far from being fully embraced by the research community. At the same time, the ways in which research assessment is carried out at present have become an acute issue both for researchers and organisations. A key factor for overcoming these systemic challenges is to reform academic evaluation and reward systems to include Open Science practices.

This report explores how and at which levels change can happen, and which routes can be taken to reach a comprehensive change that could be applied across the research system while respecting valid disciplinary or other relevant sectoral differences. Several policy options for each stakeholder are proposed. Overall, the pressure by the European Commission and other actors should be welcomed to drive much-needed changes, but this approach may risk neglecting the quality of research in favour of how it is performed. This is why we feel it is crucial for researchers to drive the transition to Open Science and is the main motivation for this work.

We believe that all stakeholders must recognize four essential principles for the successful reform of research assessment, and these require coordination both within and between stakeholder groups:

• Engage researchers in all decisions regarding changes to research assessment: all stakeholders should liaise more with researchers and researcher organisations and include them from the beginning in their decision-making processes.
• End the use of inappropriate metrics: all stakeholders should abide by the principles previously outlined in the San Francisco Declaration on Research Assessment and the Leiden Manifesto.
• Agree on appropriate ways of assessing research and researchers: identify suitable discipline-specific means of evaluation; establish an appropriate balance between qualitative and quantitative evaluation; evaluate which metrics or indicators, if any, are suitable.
• Recognise that reforming assessment requires resources: to facilitate Open Science practices, funders, governments, and universities should provide additional targeted funding.

For research communities, it is urgent and vital to concretely consider how they wish evaluation systems to be adapted to eliminate pernicious incentives and to reward pertinent Open Science practices in their diverse circumstances. There is a serious risk that if they cannot make concrete proposals on how to replace currently prevalent prestige indicators such as journal impact factors and quartile ranks, these will either continue to be (mis)used or new indicators will be imposed without the communities’participation.

Similarly important and critical is the engagement of all concerned parties in the larger discussion of how to move towards new forms of research assessment, with an appropriate balance of qualitative and quantitative evaluation. This engagement requires the involvement of researchers themselves and their community representatives, such as learned societies. By positively articulating what the goals of research assessment should be, research communities can help build better systems of assessment and credit reflecting those goals.

Summary of options

All options are in the context of four framing questions to be kept in mind:

• Which practices should be rewarded?
• What should evaluation be based on?
• How can a change in evaluation culture be achieved?
• Who should be responsible for driving change?

The options are in service of key policy goals for an improved research assessment and reward system:

• Evaluate and reward people and institutions based on their achievements and competencies rather than based on prestige or inappropriate indicators.
• Identify suitable discipline-specific means of evaluation, emphasising transparency, reproducibility and robustness of the research outcomes and of the research process.
• Incentivise Open Science practices at each career stage in a coordinated reform of academic evaluation and reward systems.

The options are in the context of four essential principles:

• Engage researchers in all decisions regarding changes to research assessment.
• End the use of inappropriate metrics.
• Agree on appropriate ways of assessing research and researchers.
• Recognise that reforming assessment requires resources.

Initiative for Science in Europe
1 quai Lezay-Marnesia
67000 Strasbourg
France

https://initiative-se.eu/

contact@initiative-se.eu

This report can be downloaded at: https://initiative-se.eu/paper-research-assessment/

Graphics and report design by Sandra Krahl