Helmholtz Quality Indicators for Data and Software Publications
Helmholtz develops multidimensional quality frameworks for research data and software publications to systematically promote their quality, reusability, and visibility.
Why Helmholtz Quality Indicators?
The evaluation of scientific performance is undergoing a transformation: In line with global efforts to reform research evaluation and the open science movement—which share the overarching goal of improving the quality, impact, and trustworthiness of research—we at Helmholtz are developing new approaches to appropriately account for research results beyond traditional (text-based) scholarly publications. Research data and software are central components of an open research ecosystem and form the foundation for transparency, traceability, and reproducibility of scientific results.
To give these research outputs greater consideration, the Helmholtz Association has developed its own quality indicators for them. These indicators pursue the following goals:
- Enhancing visibility: They promote the visibility and appreciation of the diversity of research outputs, such as research data and software, both within the Association and externally.
- Modernizing research evaluation: They consistently align the evaluation of research and research practices with the realities of digitalization and the openness of science.
- Creating incentives: They motivate researchers to further develop their data and software publications based on specific quality criteria and improve the structural framework conditions at the centers.
An illustrative representation of a radar chart that visualizes the results of the maturity model.
Development of the Helmholtz Quality Indicators
The concept for the indicators is based on an intensive, iterative discussion process within the Helmholtz Quality Indicators for Data and Software Products Task Group (TG Indicators for short). The group is an interdisciplinary team of experts in the fields of research data management, research software, information infrastructures, and library science from all Helmholtz Centers. This active involvement of all Helmholtz Centers ensures that the quality indicators take into account the diversity of scientific disciplines and research practices within the Helmholtz Association.
The quality indicators describe various quality dimensions of a publication. They are based on the FAIR principles (Findable, Accessible, Interoperable, Reusable), supplemented by additional criteria that are specifically relevant to research data or research software.
Criteria are defined for each quality dimension. Evaluation is carried out using a corresponding maturity model that systematically assesses the implementation of individual quality characteristics.
Implementation
The quality indicator will be implemented in three phases to gradually integrate the requirements into the institutional reporting system:
Phase 1:Initial Assessment (2022–2025) During this period, the Helmholtz Centers began an initial systematic, quantitative survey of citable research data and software publications. This decentralized data collection served to gather experience in identifying and documenting research outputs.
Phase 2: Pilot Phase (2026–2028) Starting in 2026, qualitative assessment based on the quality indicator will be introduced step by step. A key focus of this phase is the development and testing of (semi-)automated processes for recording research publications. The goal is to establish the use of the indicators across the board at all centers in preparation for the new reporting year.
It is expected that adjustments to the developed criteria and their specifications will still take place during the testing phase in order to further refine them based on practical experience and with a view to ensuring automation proceeds as smoothly as possible.
Phase 3:Mandatory Implementation (starting in 2029) With the fifth phase of program-oriented funding (PoF V), the collection of quality indicators for research data and software will become an integral and mandatory component of institutional reporting starting with the 2029 reporting year (for publications from 2028).
Current Status (July 2026)
The implementation of the quality indicators requires active involvement at the level of the individual Helmholtz Centers. The choice of the specific workflow for data collection and reporting is deliberately left to the discretion of the centers so that existing processes and local requirements can be appropriately taken into account.
Given the current variety of implementation strategies regarding research software and research data, a continuous exchange of information at the center level is a key component of the implementation. It helps to ensure transparency in the evaluation process, establish individual data collection mechanisms, and inform researchers about the options for registering their research outputs.
Research Software Publications
A set of criteria was developed for research software, and a standardized workflow was created for the Helmholtz Research Software Directory (RSD) (HIFIS) to enable the evaluation of software publications registered in the RSD. The goal is for software developers to evaluate their software publications listed in the RSD based on the criteria developed. This provides them with direct information on how their software publication is rated across the various quality dimensions and where there is potential for improvement.
Although the Research Software Directory (RSD) is provided as a supportive solution, it is not a mandatory requirement, provided that a center has alternative, suitable workflows for assessing the quality indicators for software products.
Milestones and Future Planning
(The milestones are intended as a guide and will be flexibly adjusted to the project’s progress as needed.)
- 2025: Development of quality criteria (initially) completed
- 2026-Q1: Start of the testing phase
- 2026-Q2: Information event for researchers and software developers
- 2026: RSD online manual available and continuously updated
- 2026: FAQs on the Helmholtz Quality Indicator for research software publications available and continuously updated (currently requires a Helmholtz ID account)
- 2026-Q3: Release of the Quality Indicator feature in the RSD (Reporter View)
- 2026-Q4: Information session for reporters and final implementation in the RSD
- 2027–2028: Further development of the quality criteria in the RSD
- 2029: Introduction of Helmholtz-wide reporting using the Research Software Quality Indicator.
Research Data Publications
Milestones and Future Planning
(The milestones are intended as a guide and will be flexibly adjusted to the project’s progress as needed.)
- 2025: Development of quality criteria (initially) completed
- 2026-Q3: Development of the tool architecture for the (semi-)automated evaluation process in collaboration with HMC
- 2026-Q4: Information session on the quality indicator for data publications
- 2027-Q1: Service will be launched: HMC will collect research data publications from 2026, including a completeness check and supplementation by the Helmholtz Centers (at the same time, the entry indicator will be collected by the centers, as in the previous year)
- 2027-Q1: Software development begins for the tool to automatically evaluate the quality indicator
- 2027-Q3: Initial testing phase of the tool using collected research data publications from 2026
- 2028: Further testing phase of the tool with feedback loops and informational sessions
- 2029: Introduction of Helmholtz-wide reporting using the research data quality indicator.