Horizon Europe: a constructive reflection on the evaluation process – ZABALA
Horizon Europe is the most ambitious and relevant programme of the European Union to support research and innovation. Its importance for the European R&I ecosystem is unquestionable. Based on the experience of Zabala Innovation, after more than 35 years of working on the preparation of European proposals, we believe it is necessary to open a constructive reflection on its evaluation process.
First of all, it is important to highlight that the evaluation process is well designed, and from a methodological point of view, it works correctly. Proposals are evaluated using common criteria, with the participation of at least three independent evaluators, a consensus process, and a rapporteur who consolidates the comments into a single evaluation report. This system ensures coherence, fairness and comparability between proposals.
The new challenge: maintaining homogeneity with an extremely high number of proposals per topic
In recent years, a growing challenge has appeared: the number of proposals submitted per topic has increased significantly. While in the past dozens of proposals were evaluated, today it is common to see 50, 60 or even more than 100 proposals, competing within a single topic for only 3, 4 or 5 funded projects. To manage this volume, the European Commission is forced to create multiple evaluation panels, which introduces a key challenge: maintaining homogeneity when the evaluation is spread across many evaluators and different panels.
In addition, as the number of evaluators needed for a specific topic increases, it becomes more difficult to ensure that all evaluators have a similar level of experience and aligned criteria on technological aspects, impact and implementation. Even with highly qualified evaluators, ensuring a homogeneous evaluation in topics with more than 50 proposals is extremely complex. As a result, many high-quality proposals are grouped around very similar scores, and small differences in interpretation become decisive.
This risk to the quality of the evaluation should be considered in the evolution of the Work Programmes. The trend towards broader and more open topics increases the number of proposals per topic and, consequently, the pressure on the evaluation system. When only a very limited number of projects are funded, and these are expected to represent the best available solutions, ensuring consistency and homogeneity in the evaluation process becomes absolutely critical.
A necessary reflection
Horizon Europe is a solid programme, but, due to its own success, there is a need to reflect on the design of the topics and to protect the quality and coherence of the evaluation process to maintain the trust of the innovation ecosystem. Assuming that there are no simple solutions and that the system already operates with a very high number of evaluators, any possible improvement should focus on preserving evaluation quality and homogeneity, especially at the highest score levels.
In this context, one possible improvement could be the creation of an additional harmonisation panel, composed of senior evaluators, to review in a transversal way the proposals with the highest scores, especially those close to the funding threshold. This approach, focused on the final stage of the evaluation, could help reinforce trust in the system without creating a disproportionate additional burden or requiring a significant increase in resources.
In addition, as proposed in the G!E position paper on the Future of Horizon Europe, introducing an initial calibration phase, where evaluators align scoring benchmarks before starting the assessment, could further improve the consistency of evaluation results, as well as a final clarification or negotiation phase, allowing applicants to respond to evaluators’ main concerns. Together, these measures could contribute to a more robust, transparent and trusted evaluation process, while respecting the constraints and scale of Horizon Europe.
HE at the crossroads: the next Framework Programme and the European Competitiveness Fund
At the same time, it is important to consider that the European Commission is currently working on the next Horizon Europe programme, as well as on new instruments such as the European Competitiveness Fund. These initiatives include concepts like the Innovation Journey, aiming to support projects across their different stages of development.
These are very relevant and positive developments. However, they can only be successful if they are built on a robust, consistent and trusted evaluation model. Supporting projects along their full innovation journey requires evaluation processes that are perceived as fair, homogeneous and reliable at every stage.
For this reason, the reflections outlined above should be taken into account in the design of future programmes, ensuring that the evolution of funding instruments goes hand in hand with strong and credible evaluation mechanisms.
Written by Camino Correia, ZABALA