As medical devices that use artificial intelligence and machine learning appear in more hospitals and imaging labs across the U.S., new data from the Food and Drug Administration show the agency has been fielding more submissions.

In 2022 alone, the FDA authorized 91 AI- or machine-learning-enabled medical devices, according to data released on Oct. 5. This broad category of devices can include anything from a basic algorithm to more complex machine learning tools, Michaela Miller, U.S. medtech technology and analytics practice leader for IQVIA, a North Carolina-based analytics and clinical research firm, said by email. 

Devices that received FDA clearance this year include an atrial fibrillation history feature for the Apple Watch, and Tel Aviv-based startup Aidoc’s feature to analyze X-ray exams to detect and triage collapsed lungs. 

MedTech Dive analyzed FDA data on all of the AI- and machine-learning-enabled devices the agency has authorized to date. Here are five takeaways on the rise of these devices.

The number of AI- and machine-learning-enabled devices reviewed by the agency more than doubled from 2017 to 2018, and has continued to grow each year since. In 2021, the FDA authorized a record 115 submissions, an 83% increase from 2018. 

Advances in sensor technology, imaging and data analytics have helped drive the trend, Miller wrote. Still, “the need to regulate and control these technology applications as drivers behind treatment decisions has increased to ensure patient safety,” she added.

Just two years ago, the FDA created its Digital Health Center of Excellence, adding staff to provide more expertise on software devices and to modernize the agency’s regulatory approach. Bakul Patel, who led that initiative, has since left the agency, but while there Patel built significant capacity to evaluate AI- and machine-learning-enabled devices, said Nicholson Price, a professor of law at the University of Michigan and who focuses on life sciences innovation.

“There have been questions about capacity constraints on FDA, whether they have the staff and relevant expertise,” Price said. “They had a plan to increase hiring in this space, and they have in fact hired a bunch more people in the digital health space.”

Of the 521 submissions the FDA has authorized to date, three-quarters have been in radiology and 11% have been in cardiology. Part of the reason for the concentration of devices in these specialties is because there is ample data for device developers to draw on from imaging and electrocardiograms.

“Radiology went digital very early relative to other areas of medical practice,” Price said. “You're not working with as messy of [data], different sources of text analysis, differences in how things are measured. … You’ve got bigger, more comparable, cleaner datasets in [radiology and cardiology] than you have in a lot of other areas.” 

Because data in other areas often isn’t configured the same way across most hospitals and healthcare systems, an algorithm that uses data pulled from patients’ health records to make a prediction at one hospital may not work in another health system, Price said. Even if both hospitals use the same electronic health record system, differences in how that system is set up can affect how well the algorithm works.

 

Of the devices that have gone through FDA review, the majority have received 510(k) clearance, which does not require clinical trials as long as developers can prove their device is “substantially equivalent” to one already on the market. 

To date, 96% of authorized AI- and machine-learning-enabled medical devices have 510(k) clearance, while only three devices have gone through the FDA’s more rigorous premarket approval process. Another 18 devices have gone through the regulator’s de novo clearance process, which is for devices that are not deemed high risk but do not have a predicate.

While most of the devices on the market don’t provide a diagnosis, they instead offer suggestions or alerts for clinicians or patients. 

For example, detecting atrial fibrillation from ECG data may be a relatively simple task from an analytical perspective, but it won’t have the scope of information that a clinician uses when they make a diagnosis, said Giorgio Quer, assistant professor of digital medicine and director of artificial intelligence at the Scripps Research Translational Institute. 

"Devices are focused on a specific task. They are fantastic at that task," he said. “My device, for example, is telling me, 'you’re probably ok' or 'you may have an arrhythmia, let’s get it checked.' It cannot tell you, 'you have exactly this type of arrhythmia.' This is the task of a clinician to interpret the data."

FDA-authorized devices likely are just a fraction of the AI- and machine-learning-enabled tools that exist in healthcare as most applications of automated learning tools don’t require regulatory review, according to Price. Academic medical systems generally haven’t been required to seek FDA authorization for algorithms they develop in-house, Price added. Predictive tools based on electronic health record systems were exempted from regulatory review, though that may change under a new guidance.

“My strong impression is that somewhere between the majority and vast majority of ML and AI systems being used in healthcare today have not seen FDA review,” Price said. “It sure seems like medical devices that are going to be impacting health and safety should go through some sort of regulatory review. Exactly what that should look like is fairly complicated.”

So far, GE Healthcare has received FDA clearance for 42 AI- and machine-learning-enabled medical devices, most of which are in radiology. They include an image-reconstruction algorithm to improve the quality of MRI images and an algorithm to help clinicians detect collapsed lung cases.

“The way we approach this is we try to meet our clinicians’ pain points,” said Vignesh Shetty, general manager of GE Healthcare’s Edison AI and Platform. 

“Radiology is what a lot of our customers and partners have wanted to work with us [on] so far,” Shetty said in an interview. “For a growing number of these clearances, you will notice that we've got use cases in other care pathways as well, including but not limited to oncology, cardiology, and even neurology.” 

Meanwhile, Berlin-based Siemens Healthineers had 29 cleared devices, including a feature to help with quick quantification and accurate visualization of calcified coronary lesions from CT scans. 

Canon Medical Systems, a subsidiary of the Tokyo-based photography company; Tel Aviv-based Aidoc, which makes computer-aided triage systems; and Amsterdam-based conglomerate Philips also had several cleared AI devices.

5. AI/ML devices will get increasingly complex with algorithms that can "learn" and regulators are looking to adapt

Currently, any major changes to a medical device must be cleared by the FDA, so most algorithms remain static after they’re introduced to the market. Still, the agency is preparing for a future when the software integrated in these devices can adapt. 

AI- and machine-learning-enabled devices will become increasingly complex as companies move toward algorithms that “learn” as opposed to algorithms that are “locked and deployed,” wrote IQVIA’s Miller.

“One thing that is really peculiar for AI devices, the core of the device is an AI algorithm, a piece of software. The piece of software may evolve over time, so there needs to be periodic surveillance of how the software inside of the device is changing," said Scripps’ Quer. 

The FDA has taken some steps to modernize how it regulates these types of devices. It has been working on a digital health pre-certification program that would allow the agency to pre-clear trusted manufacturers to update their software products, though this hasn’t been without hurdles. In a recent report, the FDA acknowledged that it would need Congressional approval to move forward with the program.

The FDA also has discussed a predetermined change control plan, where companies could outline anticipated modifications to software-based devices, allowing them to make changes within those boundaries without making another time-consuming submission to the agency. 

Ensuring patients benefit from these devices will depend not only on regulators, but also on developers involving clinicians in the earliest steps of design to ensure software is effective when integrated into their practice. 

“We need to have a deep collaboration between the AI world — the computer scientists or engineers, like myself, who work with the ​data and the algorithms —  and on the other side the clinicians​, who are facing the patient and can provide a clinical perspective,” Quer said. "The clinician always needs to be in the loop and provide that ​clinical input, be the one that judges if this is useful or not.”

Shan Jegatheeswaran joined Johnson & Johnson last year as the company’s global head of digital solutions for its medtech business, after a long career at GE. In this new role, Jegatheeswaran is building tools to improve patient care and help surgeons operate with greater precision and efficiency.

In an interview with MedTech Dive, he talked about transitioning from a career in energy to working in healthcare, his first internship, and the tasks that people don't often think about that go into building successful AI tools.

This interview has been edited for length and clarity. 

MEDTECH DIVE: When did you join J&J? 

JEGATHEESWARAN: I started in July [2022]. Prior to that, I was chief digital officer for oilfield services, a business unit in Baker Hughes [which] is effectively the spinoff of GE Oil and Gas. I started at GE, moved through multiple businesses at GE, spent the last nine years in oil and gas, but prior to that: capital, healthcare, aviation, corporate. It was a good experience.

How did you get back into working in healthcare? 

I liked the opportunity set that we have in healthcare. You hear a lot about the industry when you're outside it, and it's very personal to most people. 

Little known fact: J&J was actually my first internship. So my goal was always to be in pharma/healthcare. I actually have my employee badge from 20 years ago. 

What kinds of projects are you working on for J&J medtech?

My scope is essentially software that either works very closely with a medtech device, or enables an experience with a medtech device. One degree away from the OR, so to speak. It's pre-operating, inter-op and post-op. My team's scope touches all elements of that. As you can imagine, our goal is to make sure that there's a frictionless, seamless experience across the board for the patient, the healthcare team and the supporting team.

What technologies are you most excited about when it comes to software within the medtech industry? 

What I'm most excited about is the people element of the digital story. That often gets overlooked … but I think what's actually very interesting and what is a driver of success is how you drive that intersection between [the] human and the digital experience. There's a little bit of art, but there's also a lot of science.

And then, AI is becoming more and more prevalent, right? My mom talks about it at dinner now. It's everywhere. So how do I infuse AI into an everyday setting so it isn't AI, it's just part of how we do work?

What needs to happen behind the scenes for AI to work well?

When people talk about AI, it's almost like there are three boxes: data on the left side of the screen, AI in the middle and the value on the right side. And that's not the case.

Before you get to the analytics, there’s data aggregation, data tagging, data cataloging, data audit, data policy, and even the procedure around managing data. That's a ton of sweat equity that needs to be done. A lot of it, initially at least, is manual.

We've gotten spoiled with the Netflixes of the world and the Googles of the world, because they have data sets that are tagged already. 

If you take something like a surgical video post procedure, you can have a four-hour cut where the doctor wants the highlight reel, so to speak. That's one element we've spent some time on from an AI perspective. 

It's the blocking and tackling around cataloging [and] aggregation of data. And that's why we work closely with partners like Microsoft and internally to build these solutions. 

How do you prevent bias when you’re developing an algorithm? 

I think first of all, it's the discipline of the control infrastructure on how you run the design and the testing of the algorithm itself. 

We have in place, across the pharma sector, guidelines on how to approach that. We're leveraging some of [those guidelines] and also making that available, sort of transitioning it over to a medtech use case.

From our perspective, we have to be careful that we're not taking homogeneous data sets from our devices, and not considering patient profiles that are not being serviced by our devices. Luckily, J&J has a large breadth from a market perspective — we do possess pretty diverse data sets — but we spend a lot of time with academia plus government to make sure that some of that is kept in check. 

I think the second thing is, before you go to market with an analytic that's driving some level of guidance or insight, we've tested it already.

Speed doesn't necessarily have to be a leading indicator in all cases. Quality should be, and so I'm lucky that I get to build software that's backed by competent, best talent in the world from a regulatory, privacy and government affairs perspective.

Sponsored content

By Nuance

Technology has transformed healthcare, particularly in the areas of medical documentation and common clinical tasks. With conversational AI, ambient AI and generative AI, the industry has been propelled forward, enabling new functionalities and infusing healthcare experiences with intelligence. These advancements are contributing to improved patient care and enhanced clinician-patient experiences, making it possible to move away from inefficient processes.

Healthcare workflow inefficiencies waste time, increase cognitive load and burnout and lead to job dissatisfaction. And their impact will only become more apparent as physician shortages grow, demand for healthcare services rises with the average age of the population and clinicians are expected to accomplish more with limited resources.

It’s therefore crucial to address these underlying inefficiencies now, and optimize clinician workflows to improve the effectiveness of healthcare delivery.

AI can help. But before we explore how, let’s unpack the problem.

How workflow inefficiencies can impact the healthcare journey

Inefficient healthcare workflows can occur in outpatient management, billing, appointment management, approvals, consultations, or any other part of the healthcare journey. In every instance, they have a negative impact on clinicians, patients and healthcare organizations.

Patients experience long wait times for appointments, and their care is impacted by inaccurate or missing information in electronic health records (EHRs). Clinicians become frustrated, burned out and leave their jobs, while insurance payment denials and delays create cash flow problems.

In a recent Sermo survey, increasing workflow efficiency was cited as the top goal for physicians, closely followed by improving work/life balance.1 This isn’t surprising, given that Medscape’s burnout and depression report found bureaucratic tasks (60%), long working hours (34%), and a lack of control/autonomy over life (32%) are the top contributors to burnout.

And, it’s no surprise that “too many bureaucratic tasks” top Medscape’s list. Especially when you consider that for every hour of patient care, two hours are spent documenting it.

But it doesn’t have to be this way. Technology is transforming healthcare, particularly in the areas of medical documentation and common clinical tasks.

Solving workflow challenges with AI

Conversational AI, ambient AI and generative AI have propelled the healthcare industry forward, enabling new functionalities and infusing healthcare experiences with intelligence. These advancements are contributing to improved patient care and enhanced clinician-patient experiences, making it possible to move away from inefficient processes.

Nuance and Microsoft have already put these technologies into practice, with clinical workflow efficiency solutions designed to aid clinicians and care teams.

Want to capture the full patient story automatically to reduce cognitive load and focus fully on patients? Want to have documentation created for you and save significant time so you can catch up on other tasks with time to spare after work? That’s exactly what Dragon Ambient eXperience (DAX) does. It securely records the clinician-patient conversation, automatically converts it into medical documentation and delivers it into the electronic health record (EHR).

Want to place orders, navigate the EHR and sign a note or make a referral—all with just your voice? That’s what Dragon Medical One does. It automates common tasks to save you time and help you operate more efficiently.

Industry-leading solutions from Nuance and Microsoft

There’s a reason why these proven AI solutions are consistently top ranking in their category. For three consecutive years, Dragon Medical One has ranked #1 Best in KLAS, while DAX has ranked #1 for improving clinician experience in KLAS’s top 20 emerging solutions.

These solutions deliver outcomes by addressing challenges in real-world clinical environments and, among other benefits, save clinicians on average over 50% of their time documenting care.

• Capture real-world medical conversations accurately and unobtrusively
• Understand accents, dialects and different speaking styles
• Capture multi-party conversations such as a parent in the room with their child and the physician
• Know what belongs in a clinical note and what does not (e.g., chit chat)
• Correctly capture overlapping dialog
• Understand general and specialty-specific medical terminology
• Know the correct placement of content in the clinical note
• Utilize existing templates

DAX has been in the market for more than three years and is used by thousands of clinicians in hundreds of healthcare organizations of all sizes nationwide in ambulatory practices. Dragon Medical One is used by over 550,000 clinicians in over 12,000 healthcare organizations worldwide, in both inpatient and outpatient settings.

But crucially, whichever Dragon solution clinicians use, they’ll find opportunities to reclaim time in their day to spend elsewhere—whether it’s with patients or in their personal life.

By a recent count, GE HealthCare has obtained Food and Drug Administration clearance for 42 AI- and machine learning-enabled medical devices, the most of any company in the market. The Chicago-based medtech firm has been building out its Edison digital health platform, a suite of tools to bring together data from disparate sources, and apply algorithms to flag health conditions and help surgeons prepare for procedures. 

The majority of GE’s AI devices are in radiology, including a feature to reduce the time it takes for an MRI scan while also improving image quality. The company also has a suite of algorithms embedded in its mobile X-ray systems for critical care, including features to identify critical care conditions and analyze endotracheal tube placement. GE has also expanded into cardiology and oncology, including a feature to simulate injections to shrink liver tumors.  

Vignesh Shetty, a senior vice president responsible for the GE HealthCare’s Edison artificial intelligence platform, talked about the company’s approach to AI and how it ensures the tools it builds are accurate. 

This interview has been edited for length and clarity. 

MEDTECH DIVE: How do you make sure these tools fit into hospitals’ workflow?

VIGNESH SHETTY: We work closely and collaborate on both data and expertise between two broad worlds, one of practitioners and then one of developers. As you can imagine, we can't do this alone. So we try to be as inclusive as possible to the ecosystem of developers. 

Our goal is to bring the two worlds together because what we see is [that] while both of them are passionately striving to solve the same problems, they aren't always talking to each other, or talking early enough. As a result, we see that some of these offerings do not address the right clinical or operational need, or they're not suitably integrated into the workflow.  

One of the things that we invest a lot of time in as a company is to avoid some of these potential pitfalls by continuing to deeply understand the needs of the clinicians and the hospital systems. We also engage and train staff, with the philosophy being that these tools should always supplement — not replace —  the need for human interaction.

When you talk about the platform being vendor agnostic, what does that mean? 

[We ask] healthcare systems what the challenges and pain points [are that] they would like us to help address. They've got all of these different systems, whether that's an EMR, a pathology information system, a lab information system, where different aspects of the same patient data are stored.

These are unfortunately siloed systems, sometimes [from] incompatible vendors, and there are other data collection and collation issues. That leads to cognitive overload on the part of the caregiver, which ultimately causes clinician burnout and suboptimal outcomes for the patient. That's a key recurring theme here.

While there are a lot of honest attempts, there's a tendency unfortunately to build piecemeal solutions which only adds [to] that complexity. So what we're really trying to do [as we] build the Edison digital health platform is to enable outcomes that our customers care about by creating an ecosystem of applications, both GE and non-GE. So even if you don't have all of your CTs or MRI or ultrasounds [from] GE, that's perfectly fine. 

How are you approaching regulators?

At FDA and the European Commission, we are in trade association dialogues and the regulatory review process to help promote the use of good machine learning practices. 

As an example, we've been working with them on the algorithm change protocol. This is a concept that was introduced by the FDA, that was intended not to authorize only the first version of an AI algorithm but allow you to define an acceptable set of modifications that would not require either GE or any [company] to go back and require additional review by the regulator. Since its introduction, the FDA has applied this concept to a limited number of devices, due to some legislative challenges. We are in favor of extending this concept to most of the devices because that allows us to truly deliver on this promise of AI, which includes retraining and improving the performance [of an algorithm] over time, without the delay of a systematic review by the regulator, as long as it doesn't compromise the quality. 

The expectation is that as we get into more clinically significant AI, it becomes not just better monitored, but more explainable. You can explain the cause-and-effect relationships, as well as ensure that it's free of bias by allowing for more frequent iteration and automation in the model retraining. 

Are there any other areas on which you are focused?  

The general emphasis on cybersecurity — because I do think that's another potential barrier for large scale. 

I think while cybersecurity gets a lot of attention in the traditional software world, it doesn't get as much attention as it should in the AI space. Increasingly, hackers are starting to think about what does it take to reverse engineer the AI model by figuring side-channel attacks. Or, they're looking at potential phishing: How do I contaminate and poison the data, then I can influence how that model is trained? And that's by definition how the model behaves in the field. These are relatively new areas, which haven't been given as much thought as cyber in the traditional software space. So [we are] just raising awareness around that and trying to figure out what’s the best solution. 

What’s the big-picture opportunity here?

If we do our job right, for the patient what this fundamentally means is they're going to see a more personalized experience than they have been historically used to. Because today, most of us when we go to the clinician's office, end up in a situation where the doctor has no option but to sort of multitask, trying to listen to you and typing a bunch of things into the computer. 

We generally believe that AI has the potential to take a lot of the mundane but important tasks away from the clinician, so that frees them back to deliver actual care

Medtech companies will be able to more easily update devices with an artificial intelligence or machine learning component under a new regulatory framework being developed by the Food and Drug Administration.

In March, the agency shared a draft guidance for “predetermined change control plans,” essentially a set of modifications that device developers can include as part of a marketing submission.

Typically, medical algorithms are “locked” and do not change after they are cleared by the FDA. The guidance would allow for some modifications, within a predetermined scope reviewed by regulators.

As the number of AI-enabled medical devices increases, the FDA has been considering how to best regulate these software components. The idea of a “predetermined change control plan,” a way to enable software to “learn” and change within set boundaries, dates back to a discussion paper in 2019.

For example, a manufacturer that makes software used in an intensive care unit to alert clinicians when a patient’s condition is deteriorating may want to make changes to reduce the false alarm rate, while maintaining the software’s efficacy.

Or, a company that makes software used to recommend ideal ventilator settings for a patient might want to change the machine learning model to optimize its performance for a subset of patients with a particular condition.

In its guidance, the FDA said AI- or machine learning-enabled devices can have a “significant positive impact on healthcare.” Its goal is to create a “least burdensome approach to support iterative improvement through modifications to an AI/ML-enabled device, while continuing to provide a reasonable assurance of device safety and effectiveness.”

The agency outlined what elements device-makers should submit as part of these plans, including a description of the planned modifications, how the company plans to validate and implement the changes, and an impact assessment. 

A spokesperson for AdvaMed, the Advanced Medical Technology Association, said the group had no immediate comment on the proposal, but would soon be filing its comments to the FDA.

Medical device companies had until July to comment on the draft.

Alison Dennis is a London-based partner with law firm Taylor Wessing, where she co-heads the international life sciences team specializing in regulatory and transactional representation. She works with pharmaceutical and medical device manufacturers to ensure product offerings meet regulatory requirements, and negotiates regularly with regulators. Opinions in this article are the author’s own. 

If you're like me, when you fall ill, you will often resort to at least one of two options: consulting a doctor or checking your symptoms online. The former can be expensive or time-consuming, the latter inaccurate and unreliable. What if there were a middle road, where you could report your symptoms to an AI chatbot that could give you a diagnosis? Would you trust its answer? If so, would you trust it more or less than you would trust a doctor? These tools already exist and are rapidly developing in sophistication, know-how and accuracy. 

This is just one example of the many ways in which AI as a medical device will become ingrained in our lives. One of the key non-technical barriers to adoption of AI medical devices is user trust. As the adage goes, trust is hard to earn but easy to lose. Most people will trust their doctor, yet be aware that healthcare professionals are human and so will inevitably make mistakes. Ironically, it is this humanity which generates trust: These are individuals you can look in the eye and build relations with. 

Not so with AI (or not yet, in any case). Beyond its programming, AI will display no empathy, morality or concern for outcomes. The consequence is that although AI as a medical device has the potential to be far more accurate than a human physician, the public will be less forgiving of any errors that do occur.  

Laying the basis for safety and trust

The foundation blocks for ensuring a high degree of safety, leading to a trusting public, will have to be laid by regulators. Regulation will have to span and classify the spectrum of medical devices, from basic AI-enabled smart wearables with a clinical purpose – such as checking for an irregular heartbeat – to autonomous surgical robots. The detail of the regulation should clearly set thresholds that demarcate increasingly rigorous standards proportionate to the device's risks. 

The healthcare industry will need to educate consumers and physicians on the benefits of AI devices and how to use them optimally to obtain the claimed benefits. 

Manufacturers will need to educate through explanatory notes, FAQs and video demonstrations to give sufficient assurances that their products are acceptably safe, and function as intended. They should also take responsibility for their products, offering customers clear routes to recourse in case of issues. For most, trust will require proof, and seeing is believing. 

Separately, manufacturers will need to demonstrate to regulators the safety, effectiveness and quality of the device before it can be placed on the market. When providing their technical files to the regulator, manufacturers will likely need to build in a technical means for understanding mechanisms, validation and verification, while also noting changes made via AI to the operation of the device.

Will patients and users trust technology once they have received support to understand it? Returning to our diagnostic chatbot example, patients might feel reassured and more comfortable using a device if it includes an explanation of its sources, the degree to which a patient’s symptoms match the diagnosis, and the percentage likelihood that an alternative diagnosis may be correct. 

Don’t be a Magic 8 Ball

To build patient trust, the device might also explain, for example, how the AI was trained on real-world data, whether the data was reviewed by physicians, and which healthcare authorities have approved the device's use. A 'black box AI' chatbot that produces a response without explaining its operations may resemble the digital equivalent of a Magic 8 Ball. 

Manufacturers may wish to go beyond the mandatory requirements and seek additional certification of compliance with international standards or self-certify compliance with good practice principles, such as the ten guiding principles in the Good Machine Learning Practice, agreed between the U.S. Food and Drug Administration, Health Canada and the U.K.'s Medicines and Healthcare products Regulatory Agency. 

One of the commonly cited issues with AI in healthcare is the risk of inbuilt bias, which can be caused when non-representative data sets are used to train AI systems. Any evidence of bias will quickly erode hard-won trust. 

For example, AI medical devices trained to diagnose disease based on data that does not reflect the ethnic or other diversity of the population could alter the effectiveness of the device on the unrepresented or under-represented segments of the population. In response, regulators will require proof that manufacturers have appropriately identified, measured, managed and mitigated risks arising from any bias. 

Supplemental information, such as adherence to stringent cybersecurity standards, maintenance of medical confidentiality and compliance with data processing principles will further reassure patients and users of the manufacturer's commitment to regulatory compliance and high standards, which may further engender trust in the products.

For regulators and manufacturers alike, encouraging trust among healthcare professionals should be an essential target, both in their role as an end-user and as an intermediary with patients. Healthcare professionals will not risk their patients' health, their reputation and possibly their career on untrustworthy technology and so require confidence in the knowledge that they can use and recommend AI medical devices. Gaining the trust of the healthcare community will be a critical stepping stone to achieving a trusting patient population. 

As with all new technologies, it is likely that the younger, more 'plugged-in' generation will initially be the early adopters, leading attitudinal change to the acceptability of AI as a medical device. Ultimately, whether an individual decides to trust AI as a medical device, and the extent of that trust, will be decided by personal preference and access to alternatives. 

It's clear that AI has the technological potential for huge advances, but its success within healthcare will be determined by the ability of the regulatory environment to meet the challenges of a constantly self-developing product, in order to ensure that public trust advances at the same speed. 

The World Health Organization on May 16 warned that the “meteoric” rise of artificial intelligence (AI) tools in healthcare threatens the safety of patients, if caution is not exercised.

Because of the excitement that rapidly expanding platforms such as ChatGPT, Bard, Bert and others are generating over the potential to improve patient health, device developers and others are tossing aside the caution that normally would be applied toward new technologies, the organization said in a statement.

“Precipitous adoption of untested systems could lead to errors by health-care workers, cause harm to patients, erode trust in AI and thereby undermine (or delay) the potential long-term benefits and uses of such technologies around the world,” the WHO warned.

Many of the AI systems being introduced in health care use so-called large language model tools (LLMs) to imitate human understanding, processing and communication. The WHO said the “meteoric” rise of the technologies, many still in the experimental stage, warrant a close look at the risks they pose to key values in healthcare and scientific research. Those values include transparency, inclusion, public engagement, expert supervision, and rigorous evaluation.

“It is imperative that the risks be examined carefully when using LLMs to improve access to health information, as a decision-support tool, or even to enhance diagnostic capacity in under-resourced settings to protect people’s health and reduce inequity,” the WHO said.

Those risks include concerns that the data used to train AI may be biased, leading the tools to generate misleading or inaccurate information, including responses that appear authoritative or plausible when they are wrong. Rigorous oversight is needed for the technologies to be used in safe, effective, and ethical ways, the WHO said.

“WHO proposes that these concerns be addressed, and clear evidence of benefit be measured before their widespread use in routine health care and medicine,” the organization said.

The organization’s warning echoes remarks last week from Food and Drug Administration Commissioner Robert Califf, who said in a speech that nimble regulation of large language models is needed to avoid the healthcare system being “swept up quickly by something that we hardly understand.”

Sam Altman, CEO of OpenAI, agreed with the sentiment that AI must be regulated in testimony he gave last week before a Senate subcommittee. “I think if this technology goes wrong, it can go quite wrong. And we want to be vocal about that,” he was quoted as saying. “We want to work with the government to prevent that from happening.”

Recent examples of new AI applications in medtech include Smith and Nephew’s planning and data visualization software for robotic surgery and BD’s new software to look for Methicillin-resistant Staphylococcus aureus (MRSA).

Dive Brief

• Some types of risk-scoring tools would be regulated as medical devices, according to a final guidance issued by the Food and Drug Administration on Sept. 28. 
• Many of these software tools have been exempted from FDA regulation under the 21st Century Cures Act, as long as the healthcare provider can independently review the basis of the recommendations and doesn’t rely on it to make a diagnostic or treatment decision.
• Clarity on the rules was welcomed by some device makers, even as they cautioned that products may take longer to come to market. Some medical device companies supported the changes, saying they provided more clarity on the FDA’s thinking, while others saw the guidance as overstepping Congress’s intent. 

Dive Insight

The guidance applies to clinical decision support software, a broad category of software tools that can help physicians and other healthcare providers detect illnesses and generate alerts when a patient’s condition changes. For example, software that uses electrocardiogram (ECG) data to detect arrhythmias, like that used by Apple’s and Fitbit’s smartwatches, is regulated as a medical device. The same goes for software that analyzes images to differentiate between ischemic and hemorrhagic stroke, or that is used by radiologists to “triage” patients and review potential cases of pulmonary embolism. 

“The 21st Century Cures Act was very clear that anything that analyzes a medical image directly would be a medical device,” said Christina Silcox, research director for digital health at Duke-Margolis Center for Health Policy. 

The biggest policy change, the FDA noted in the guidance, is that some types of predictive tools, which previously fell into a gray area, should now be regulated as medical devices. The agency gave an example of software that analyzes patient information to detect stroke or sepsis, and generates an alarm to notify a healthcare provider. 

Sepsis Detection

Several such risk-scoring tools are already in use: Electronic health record companies Epic Systems and Cerner have both made sepsis surveillance tools to detect the deadly condition before patients deteriorate. Neither are FDA cleared, and a study last year by Michigan Medicine found that Epic’s tool performed worse than advertised.

“I think that there is a distinction in people's minds around prediction and diagnosis. And you saw this with sepsis, where we’re not saying they have sepsis now, we're predicting they're going to have sepsis,” Silcox said. “In some ways, I think FDA said a risk score, the prediction, it's still a device. That was a space where people weren't quite sure.” 

Bradley Merrill Thompson, an attorney with Epstein Becker Green in Washington, D.C., who advises clients on FDA regulations, wrote that part of the FDA’s argument is around automation bias, or the concern that healthcare providers might rely too much on an automated suggestion. 

“FDA is desperately trying to draw a distinction between software that is too self-confident in its recommendations (Dr., the diagnosis is X), versus software that is more tentative in its recommendation, (Dr., the diagnosis is probably X, but might also be something else.) But that whole line of argument is a new addition from the draft guidance,” he wrote in an email. “Apparently, FDA has a very low view of healthcare professionals if the agency thinks that mere software can ‘direct’ doctors what to do, as if the doctor then has no choice.”

For patients, these tools are often used without their knowledge, and there’s nothing in the guidance that suggests they would have to be notified, Silcox said. But they could still benefit from the protection that someone has looked over the software tools to ensure they’re going to work, she said. 

 “There are always trade-offs, but in my view this guidance could provide a lot of benefit to patients,” Kellie Owens, an assistant professor of medical ethics at NYU’s Grossman School of Medicine, wrote in an email. “It could help protect patients from living under the jurisdiction of clinical decision support software that doesn't work or that systematically prevents some groups from receiving the care they deserve.” 

The guidance comes amid a broader discussion about how bias in medical devices can affect patient care. Multiple recent studies have pointed to evidence that pulse oximeters overestimate blood oxygen levels in patients with darker skin. Researchers at Sutter Health recently found the inaccuracy led to hours of delay in Black patients with COVID-19 receiving supplemental oxygen treatment. 

What it means for software developers

Epic Systems declined to comment on the guidance. Suchi Saria, an associate professor at Johns Hopkins and CEO of Bayesian Health, a startup with a sepsis alert platform, supported the changes.

The existing lack of clarity on how to assess the quality of software solutions, makes it hard for health systems or provider groups to confidently adopt these kinds of tools, Saria said. They’ve had to turn to published and peer-reviewed research, which isn’t available for all of the sepsis alert tools on the market.  

“Traditionally, when it comes to drugs and devices, organizations have relied on groups like the FDA to validate, evaluate and endorse certain products. In that sense, I think it's very exciting that there will be more clarity now through organizations like the FDA in evaluating these kinds of products,” Saria said.

She cautioned that FDA review of these software products could slow down the ability to bring them to market, given the many processes involved in evaluating machine learning solutions for safety and quality.

Despite these concerns, Saria expects a benefit to her company overall. 

“[The] FDA coming in and wanting to play a bigger role in oversight actually only helps Bayesian's case because it allows us to accelerate that agenda,” she said. “We think there's so much noise in the market and it's hurting credible solutions from getting implemented faster.”

Mayo Clinic, which is developing a suite of ECG algorithms with spinout company Anumana, said the examples in the guidance were helpful in evaluating how the Act applies to software products. 

“Overall, the FDA regulatory policy for clinical decision support software uses is much the same as its draft guidance, and is consistent with the FDA’s risk-based approach for medical device regulations, more broadly,” wrote Colin Rom, who leads science and innovation policy for Mayo Clinic’s government engagement team. “Mayo Clinic believes that these policies and an approach based on risks will enable the development and deployment of safe, effective and ethical digital health technologies for our patients.”

Enforcement

There are still plenty of details to be hammered out, including how the changes would be enforced for software tools that are now considered a device.

“I think the guidance provides helpful clarification of the agency's intentions to regulate some clinical decision support tools, but also leaves a lot of questions about what that regulation will ultimately entail,” NYU’s Owens wrote.

An FDA spokesperson wrote in an email that the agency has “discretion on when to take appropriate risk-based actions, as facilitated by the publicly-available guidance. Guidances describe the agency’s current thinking on a topic and should be viewed as recommendations, unless specific regulatory or statutory requirements are cited.”

The agency added that developers are encouraged to work with the FDA as early as possible in the process to get answers to their questions, through the Center for Devices and Radiological Health’s pre-submission program. 

“I absolutely believe that we should take [the FDA] literally, and that they would apply this guidance to regulate a lot of software which is presently in the marketplace but for which the developers have not sought to comply with FDA requirements,” Thompson wrote.

Whether the agency would allow risk-scoring tools developed and used by hospitals internally wasn’t addressed in the guidance, but the FDA has a long-standing policy of allowing for the practice of medicine, Thompson added. Vendors that develop products for sale to multiple institutions would need to comply, he said.