Many people hear “FDA-cleared algorithms” and think all of the work has been done. The extensive research that goes on behind the scenes to vet medical devices at the FDA is no secret, and concluding that FDA clearance comes after a rigorous review is simple enough. When it comes to FDA-cleared AI algorithms related to medical imaging, however, here’s why it pays to do your homework.

Let’s start with the basics. Not everyone understands the difference between the terms “FDA approved” and “FDA cleared” when it comes to medical devices. Do they mean the same thing? Are the terms interchangeable? Not at all.

AI algorithms for medical imaging are not FDA approved, despite various articles you might have read that misclassify them this way. According to the FDA, devices (including software as a medical device) are classified according to risk. For a device to be FDA approved, it must be approved via a premarket approval application as a Class III device, “demonstrating with sufficient, valid scientific evidence that…the devices are safe and effective for their intended uses.” FDA-cleared algorithms, on the other hand, need only to demonstrate “substantial equivalence” to a predicate device.

So, there is far more than semantics involved in the two terms. The FDA processes are very different. FDA approval is much harder to achieve than FDA clearance.

Where does that leave us?

In 2020, the ACR Data Science Institute (DSI) created a catalog of AI algorithms “cleared” for use in radiology to help simplify research by radiologists and developers. As of February 2021, the regularly updated catalog had 111 models, ranging from detecting pneumothorax in chest X-rays to highlighting segments of the brain on MRI.

Each model in the catalog includes a summary with the model manufacturer, FDA product code, body area, modality, predicate device, product testing and evaluation related to product performance, and clinical validation. Many of the models match the ACR DSI’s Define-AI use cases and are linked under related use cases. Clicking on individual models takes users directly to the FDA summary letter for more details. All of these features make it easier to vet the algorithms — an important step for any practice considering incorporating a new algorithm.

Some interesting factoids about the current ACR DSI catalog of FDA-cleared algorithms:

1. The 111 clearances include 94 products, meaning that so far only 17 in the current catalog have undergone updates or revisions that required re-evaluation and clearance. This might be surprising given that many people think of AI and the concept of continuous learning as synonymous. But according to current FDA guidelines, algorithms are locked and cannot be modified in any significant manner without necessitating a (likely costly and time-consuming) repeat evaluation by the FDA. So, except for these 17, all of the algorithms remain exactly as they were initially approved — despite any improvements or learnings the vendors might have had since release.

2. Ninety-four products represent 65 companies operating in this space. So contrary to what you might have thought, the major tech companies are not dominating this space yet. The vast majority of the companies are startups with only one cleared product.

3. As of February 2021, none of the products in the catalog had undergone the more rigorous PMA approval process (Class III); 108 underwent the 510K process and three have undergone the De Novo process.

Why should you care about the catalog?

For starters, if you’re thinking about purchasing an AI product, this catalog is a great place to see what’s available. If you know the type of algorithm you’re looking for (for example, which organ system, subspecialty, or specific use case is most applicable), you can also see how many companies offer products in that space. It’s also a great way to understand more about the types of algorithms that are available — which ones are triage/notification only, post-processing algorithms, detection algorithms, or actual diagnostic algorithms.

A recent survey by the ACR DSI found that 95% of current AI users find FDA-cleared algorithms are inconsistently accurate when tested on their own data.

What does this mean for you? Unfortunately, there’s no way of knowing if an algorithm will work at your institution just by looking through the FDA clearance details — but it may give you some hints. For example, studies have shown that algorithms work best on the images acquired using the same manufacturer of the scanner that the algorithm was trained on. Typically, this information is included in the FDA summary, and you can compare it to the type of scanners you use. More generally, you can also get a sense for the types of studies that were conducted to gain clearance, including how many images were included in their trials or if clinical validation studies were performed.

Looking ahead

In the future, we look forward to a time when algorithm manufacturers provide more robust information about their products’ validation process. A better understanding of the training and validation parameters will help users understand potential biases and pitfalls that can arise in clinical use, leading to a better user experience overall. The ACR DSI hopes to be able to provide validation and training information through the catalog, when available, to assist the medical imaging community in better understanding which algorithms provide the greatest benefit to their patients.

Learn more

The ACR DSI webinar, FDA-Cleared AI Algorithms for Medical Imaging: Explore the Latest AI Tools, shares best practices for vetting commercial algorithms using the catalog. Watch to learn more, and hear how you can find AI solutions that could be a good match for your clinical practice. Watch on-demand>>

Sheela Agarwal, MD, MBA | ACR Senior Scientist | Digital Medical Advisor, Bayer HealthCare