Progressing oversight of AI

The use of AI in Irish healthcare is growing, raising important questions about regulation and oversight. David Lynch reports

The coming months will see important developments in the use of artificial intelligence (AI) in Irish healthcare.

The HSE is developing an “implementation framework” to ensure that AI systems are used “ethically, legally, safely, and effectively”.

In addition, HIQA is designing guidelines for AI’s use in health and social care, while the Department of Health and HSE have commenced work on an AI strategy.

Much of this activity has been prompted by the new European Union (EU) AI Act, which has compliance implications for organisations such as the HSE. But there are also broader discussions around the issue of AI beyond this new legislation.

Citizens’ jury

At the end of last month, an Irish ‘citizens’ jury’ published 25 recommendations for health policymakers on the use of AI in the Irish healthcare system.

Organised by the Irish Platform for Patient Organisations, Science, and Industry (IPPOSI), the jury of 24 individuals set out to “offer the public’s perspective” on AI in healthcare.

The jury concluded there was a need for an independent regulator and commissioner to oversee AI and a national strategy to chart the course of AI in healthcare over the next five years. The jury also called for a separate independent commissioner to serve as a public interest watchdog and to protect patient rights.

The jury endorsed the “early, low-risk deployment of high-quality, human-monitored AI tools” to help ease pressures on the healthcare system, drive advancements in treatment and care, and empower individuals to take a more “active role” in managing their own health.

However, the jury highlighted the importance of “strong regulation, transparent oversight, and robust” data security.

Other recommendations included the need to keep “humans in the loop” as the jury emphasised the importance of human oversight and control over AI-enabled care.

Human decision-making

Prof Richard Greene, who was a member of the jury’s oversight panel, is the HSE Chief Clinical Information Officer and Professor of Clinical Obstetrics, University College Cork.

He told the Medical Independent (MI) that discussions on AI among clinicians generally focus on the topics of guidance, ensuring evidence for safe use, and AI evaluation approaches.

A recurring theme of these discussions is the necessity of protecting human decision-making. In medicine, this means ensuring that the final decision continues to lie with a clinician in collaboration with the patient.

Despite the risks, there is a lot of “guarded optimism” about the potential for AI use in healthcare, said Prof Greene.

“For those in the vanguard and innovating in this space, there is a hunger for discussion and guidance. Many of these practitioners are involved in the process of developing this guidance from health science literature and real-world evidence.”

He noted that radiology is currently the most common area for AI use in the Irish health service. A number of radiologists are working with the HSE on developing a framework for the introduction of AI in healthcare, including “clear guidance for clinicians on the introduction of AI into practice”. 

This work will contribute to the HSE’s AI framework, which is expected to be completed by June. 

“We are also working with the Department of Health to evolve a number of priority areas for AI introduction,” added Prof Greene.

Some of these will be in radiology, such as assisting image assessment and diagnosis, enhancing fracture diagnosis on bone x-rays, assisting diagnosis of stroke, and mammography screening. 

Beyond radiology, other areas for review are “ambient listening” to support clinical consultations and developing a note of the clinician-patient interaction. Additionally, AI is being explored for administrative support, such as automating certain tasks through robotic processes and using generative AI for document management.

“The overwhelming principle in all of this will be the ‘human in the loop’ [element]. This was a specific recommendation from the IPPOSI citizens’ jury to have ‘human-monitored AI’.”

Guidance and legislation

Prof Greene told MI there is a need for specific guidance and legislation for AI in healthcare.

Noting the ongoing work by the Department, the HSE, and HIQA, he said the citizens’ jury recommendations “will provide valuable input” into these areas.

The EU AI Act establishes a harmonised regulatory framework for AI systems developed in the EU. It is designed to provide a high level of protection for people’s health, safety, and fundamental rights.

The Government plans to introduce legislation in the coming months to support the implementation and enforcement of the EU AI Act at national level, with a regulator “under consideration” within the planned legislation, according to Prof Greene.

Reflecting on other recommendations from the jury, Prof Greene said the area of “digitisation” was important. 

He said the Government should “urgently and fully implement” the Digital for Care Framework. According to Prof Greene, a national electronic health record system was particularly vital to ensure that foundational AI systems are built in existing well-resourced, digitally mature systems.

“All stakeholders should ensure that the provider-patient relationship remains the primary focus of healthcare. The use of AI in healthcare should be used to augment and support clinicians, not replace them,” he stated. “If supported by AI, care should have human oversight at each stage. The goal is human-centred care, supported by AI. AI should be used responsibly and effectively to improve patient outcomes.”

Prof Greene stressed that equitable access to AI-enabled care was crucial, stating that the Government should ensure the use of AI in healthcare is accessible to all. He said there “should be no barriers” for those accessing care via the public health system compared to those accessing care privately.

The IPPOSI citizens’ jury report is available at www.ipposi.ie