Foreword

The evaluation and review of research is a central activity of research performing organisations, scientific institutions, and research funding organisations.

The purposes of such assessments range from research career progression and the selection of project proposals for funding to the monitoring of ongoing research projects and the evaluation of finished ones. To that end, the method of peer review – also known as merit review – was developed and is now firmly established in the science system: peer researchers review projects and proposals to assess them according to pre-defined quality criteria. The merit review system is a key ingredient of science as a self-organising system, as quality assurance lies in the hands of researchers themselves.

In 2012, the Global Research Council established core high-level principles for a rigorous and transparent scientific merit review system. These principles were revised and updated in 2018.

While these high-level principles provide a worldwide agreement at a fundamental level, they give little guidance as to how a merit review system should be organised and implemented. Over decades, research organisations have fine-tuned their assessment methods and peer-review processes. The recommendations presented in this paper are based on a major fact-finding study, supplemented by an extensive consultation process. They represent current best practices implemented in Science Europe member organisations and can be considered the current gold standard in research assessment methods. They provide a framework upon which all research organisations can further develop and optimise their own processes.

At the same time, there is a need to consider how assessment processes should evolve in the future to ensure that they remain effective. Our recommendations provide a starting point from which broader reforms to such processes can be considered. In particular, it will be important in the future to ground assessment processes on a firmer evidence base, for instance by conducting empirical hypothesis-testing studies.

It is important to recognise that these recommend–ations are primarily about assessment processes and methods, and not so much about criteria. In recent times, the appropriateness of various criteria (such as journal metrics) has been questioned through initiatives such as the San Francisco Declaration on Research Assessment and the Leiden Manifesto on Research Metrics. This is a highly relevant debate, where a careful separation of assessment processes and criteria needs to be made.

Science Europe calls on research organisations to build on the momentum now established to consider how they can collectively drive and direct the more profound evolution of research assessment processes that is underway. It is through re-appraisal and knowledge sharing that a clear direction for further reforms to assessment processes can be resolved.

Marc Schiltz, President of Science Europe

Executive Summary

Research performing and research funding organisations dedicate substantial efforts and resources to assessments of research quality and researcher performance. The effectiveness, efficiency, and fairness of such processes needs to be regularly evaluated and monitored – particularly at a time when the research system is changing rapidly.

In 2019, Science Europe launched a study into the research assessment processes at research organisations. It concluded that assessment processes are well-developed, and that changes are made mostly in a minor and incremental fashion. Numerous common challenges to the system were identified and are further explored in this Position Statement and Recommendations (see Chapter 2). These common challenges include the need to continually address biases in assessment processes, considerations of the cost and efficiency of assessments in view of funding limitations, and how to address and recognise the burden placed on reviewers. These challenges indicate that concerted action from Science Europe Member Organisations and other research stakeholders is needed for the research system to continue to evolve effectively.

This Position Statement and Recommendations build on the results of the study, complemented by an extensive consultation among Science Europe Member Organisations and stakeholders from the research community, and presents the following recommendations in more detail:

• Research assessment processes must be clear and transparent at all stages, for all involved.
• Research organisations should continually monitor and regularly evaluate the robustness of their assessment processes, and share best practices to foster mutual learning.
• Research organisations should publicly demonstrate and continually evaluate how they address bias, discrimination, and unfair treatment in assessment processes. Updated guidelines and training should be available to all involved.
• Research organisations should streamline assessment processes to reduce the burden on reviewers and applicants. Standardisation and interoperability of such processes within and between organisations should be considered.
• Broader criteria for the selection of appropriate reviewers should be considered, emphasising the importance of international reviewers. Review activity should be appropriately recognised.
• Research assessments should focus on the substance and content of applications. Processes should aid reviewers in conducting qualitative assessments that consider a broad spectrum of research outputs and activities.
• Research organisations should consider implementing novel assessment techniques. Methodologies and outcomes of these should be shared to promote mutual learning.

The recommendations presented in this Position Statement provide a framework upon which research performing and research funding organisations can adapt their assessment processes and collaborate to reduce the increasing strains on the system and tackle the challenges faced. The recommendations also contribute to the task of future-proofing assessment processes for the broad changes to the research system that are underway. Potential improvements of assessment processes continually arise, and are increasingly facilitated by technological advances that can support the further adaptation of the system.

Science Europe supports its Member Organisations by enabling mutual learning and knowledge exchange. Coordination with other research stakeholders in important areas of research policy is equally important, for a common understanding of the challenges the research system is facing and to foster alignment of policies.

© Copyright Science Europe 2020