From assistance system to autonomous agent

In recent years, large language models (LLMs) and other forms of generative AI (GenAI) have become increasingly important in the healthcare sector. Due to their medical purpose, they are often considered medical devices and are therefore subject to specific regulatory requirements. Previous decisions show that systems with clearly defined tasks are in principle eligible for approval.

However, the next generation of AI systems - autonomous agents - go far beyond assistance functions. They are designed to carry out complex, targeted workflows independently. Consisting of several networked components such as databases, image analysis tools, clinical guidelines and patient data management, LLMs in these systems perform tasks such as decision-making, error handling and monitoring completed processes.

"We are experiencing a fundamental change in how AI tools can be used in medicine," says Jakob N. Kather, Professor of Clinical Artificial Intelligence at the TUD's EKFZ for Digital Health and oncologist at Dresden University Hospital. "In contrast to earlier systems, AI agents are able to carry out complex clinical workflows independently. This opens up great opportunities for medicine - but also raises completely new questions about safety, responsibility and regulation that we need to address," he adds.

New technologies - old regulations?

The existing regulatory requirements for medical devices are based on static technologies that remain unchanged after approval and operate under human supervision. Autonomous AI agents, on the other hand, are adaptable, versatile and highly autonomous - characteristics that make them difficult to fit into existing approval mechanisms.

"To enable the safe and effective introduction of autonomous AI agents in healthcare, regulatory frameworks need to evolve away from static paradigms. We need adaptable regulatory oversight and flexible, alternative approval pathways," says Oscar Freyer, first author of the publication and research associate in the team of Professor Stephen Gilbert, head of the Medical Device Regulatory Science group at the EKFZ.

Regulatory innovations: Solutions from Dresden

To overcome the existing hurdles, the Dresden researchers propose short-, medium- and long-term solutions: In the short term, the expansion of enforcement discretion regulations could provide a remedy - whereby authorities recognize a product as a medical device but waive certain requirements. An explicit non-medical device classification for certain systems would also be conceivable.

In the medium term, the authors see great potential in Voluntary Alternative Pathways (VAPs) and adaptive regulatory frameworks. These would enable more dynamic monitoring based on real performance data, supplemented by continuous adaptation in collaboration with relevant stakeholders. Conspicuous systems could be returned to conventional approval procedures at any time.

In the long term, the researchers are even considering a qualification similar to medical training: AI agents would undergo structured "training" and have their autonomy proven in stages - depending on their proven safety and effectiveness.

Real-world laboratories (regulatory sandboxes) are also mentioned as a test environment, but are not considered a scalable solution due to their high resource requirements.

The authors emphasize that without far-reaching regulatory reforms, the widespread introduction of autonomous AI agents will hardly be possible. VAPs and adaptive frameworks appear to them to be the most promising ways to maintain the balance between innovation and patient safety.

Cooperation is the key

"Bold and forward-looking reforms are needed to realize the full potential of autonomous AI agents in healthcare," says Stephen Gilbert, Professor of Medical Device Regulatory Science at the ECFZ and last author of the publication. "Regulators need to start preparing now to ensure patient safety in the future and create clear conditions that enable safe innovation," he adds.

About the EKFZ for Digital Health

The Else Kröner Fresenius Center for Digital Health was founded in September 2019 at TU Dresden and the University Hospital Carl Gustav Carus. Funded with 40 million euros over ten years by the Else Kröner-Fresenius Foundation, the EKFZ aims to develop digital technologies at the interface between research, clinics and patients. The focus is on the sustainable improvement of healthcare, medical research and clinical practice through digitalization.

Original publication:
Freyer, O., Jayabalan, S., Kather, J. N., & Gilbert, S. (2025). Overcoming regulatory barriers to the implementation of AI agents in healthcare. Nature Medicine. DOI: 10.1038/s41591-025-03841-1

Source: Dresden University of Technology