Each year, the ACR Data Science Institute® (DSI) holds a summit to update members and industry members on ACR DSI projects and initiatives. The most recent ACR DSI Summit highlighted that AI is best used like any other imaging technology: grounded in understanding its capabilities and limitations, coupled with an appreciation of the practical deployment challenges, and always with the patient at the center.

To that end, the summit held on June 16, 2021 focused on the three primary objectives of the ACR DSI:
• Facilitating member understanding of AI
• Creating tools for AI adoption
• Keeping the patient at the center of AI-enabled care

Status of AI in diagnostic imaging

One of the consequences of the hype surrounding AI has been that it inspires us to see this developing technology as capable of amazing feats derived from our imaginations rather than what science is truly capable of at this point. The best way to demystify AI is to foster a deeper understanding of AI capabilities, the underlying supporting data, and limitations. Surveying the AI landscape reveals the most common modality for current FDA-regulated pixel-ML products is CT, and the most common anatomy area is the brain with over 100 FDA-regulated AI products. However, less than 10 of the over 100 FDA-regulated AI products in the radiology space contained published randomized controlled trials. The majority were marketed after only non-randomized trials, and only a few were either prospective or tested in real-life clinical settings.

Aside from the scientific merit of AI, it is also worth noting that economic incentives have slowly moved towards the use of AI products in radiology, with some usage examples such as the detection, triage, and communication of large vessel occlusions in the brain now being reimbursed through CMS's new technology add-on payment (NTAP) program.

Where AI products have been adopted in practice, ACR members shared in an ACR DSI survey that performance is often inconsistent. For example, only 30% of the responding radiologists are currently using an AI product in practice. Nevertheless, the survey showed that most radiologists expect continued growth in the use of AI in radiology, and the vast majority of respondents agree that AI provided some value to them and their patients. You can read the results of the survey in this DSI blog post.

Evaluating AI in your practice

In response to the heterogeneity of commercially available products, evolving economics of reimbursement, and the emerging new trends in radiology AI, the ACR DSI unveiled tools to help radiologists address these opportunities. The FDA-cleared AI models web page has become one of the most commonly utilized resources for radiologists seeking to understand the current AI landscape. The ACR DSI Summit unveiled the ACR AI Central as an upgrade to the AI models page with attention towards usability and transparency. The maturing ACR AI-LAB™ allows imaging practices to build and evaluate AI models using their internal data and custom AI projects. Because most radiology practices cannot hire data engineers or otherwise develop experience in AI themselves, having a tool to facilitate these tasks can be beneficial to the typical ACR member practice.

But tools are exactly that — ways for physicians to assess AI models before applying them towards patient care. It remains essential for radiologists to remember that AI models can behave ideally in training scenarios, but fail when applied to patient data the algorithms have not yet seen, a phenomenon called overfitting. AI can pick up and incorporate implicit biases from the training data — biases that can elude even the data scientists building the model. And algorithms can be brittle, with a propensity to fail when the data contains unexpected noise, such as motion or overlying external objects.

Even with an AI product perfectly created to address overfitting, bias, and brittleness, changes in data, people, and disease can cause drifts and degradation in AI performance over time. For instance, an AI model that perfectly detected bacterial and viral pneumonia in 2018 might find itself making many false predictions in 2021 because it was not built using images containing signs of COVID-19 pneumonia. Other sources of drift for AI algorithms include newly marketed scanners, new diagnostic guidelines, and changes in patient populations — such as a contract with a new hospital — and necessitate a robust method for continuous monitoring.

Patient-centered care with AI

The ACR has a long track record of focusing on value-based, patient-centered care, ranging from its Imaging 3.0 initiative to the efforts of the Commission on Quality and Safety. Likewise, the adoption of imaging AI cannot exist in a vacuum. At present, many patients are wary of autonomous AI, both in medical devices and in imaging such as screening mammography. What’s more, surveys also suggest that more patients trust their physicians than AI to make the correct diagnosis and treatment recommendations. In this way, patient opinions, ACR membership surveys, and ACR DSI's experiences are all in agreement: AI does not replace the radiologist or the clinical physician. Instead, AI is a second set of "eyes" and can help enhance both the clinical work and the quality of the images acquired, as well as aid in training the next generation of physicians.

To properly regulate and understand radiology AI and its impact on patient care, it is also vital to hold automated tools to a higher standard than manual processes. Researchers and vendors need incentives to iterate their work and compete towards better transparency, better performance, and better generalizability, with attention towards bias, brittleness, and patient outcomes. This way, data scientists, quality improvement professionals, and patients will work shoulder-to-shoulder to create an environment to foster the next-generation imaging tools that radiologists and patients can trust.

Marching into the next era

The 2021 ACR Data Science Summit delivered a diverse range of technical, financial, and patient-focused insights into today's imaging AI. So much progress has been made on that front that I, for one, cannot wait to learn about ACR DSI's progress at next year's summit.
For those who would like to learn more about these topics, several videos are available in the e-Learning Hub.

Po-Hao (Howard) Chen, MD, MBA is the Chief Informatics Officer at Cleveland Clinic’s Imaging Institute, a practicing musculoskeletal radiologist, and the co-chair for the 2021 ACR Data Science Summit.