ACR DSI Use Cases in Artificial Intelligence are detailed specifications that allow developers to create AI algorithms that assist radiology professionals in disease detection, characterization, and treatment. They are built around a framework that brings radiology professionals, industry partners, and other stakeholders together to develop, validate, deploy, and monitor artificial intelligence (AI) algorithms in medical imaging and the radiological sciences. The following details the ACR DSI Use Case development process.

Define Cases

Define Use Cases:

ACR DSI works with radiology professionals, including diagnostic and interventional radiologists, radiation oncologists, and medical physicists, to define and prioritize use cases around highly relevant clinical scenarios in which AI may help improve patient care. Designed to guide effective algorithm development, ACR DSI Use Cases include narrative descriptions and flowcharts that specify the goals the algorithm should meet, the required clinical inputs, how it should integrate into the clinical workflow, and how it should interface with both human end-users and an array of electronic resources, such as reporting software, PACS, and electronic health records.

In the interest of considering a broad range of possible use cases, the DSI solicits use case proposals from the radiology community. There are radiologists, medical physicists, developers, and others who propose ideas that become DSI use cases and eventually XML modules (within PACS or VRS) for seamless algorithm integration into the clinical environment. Use case panels choose the highest value proposals for collaboration, sharing their clinical expertise while ACR provides administrative support to create the use case.

Open Frameworks:

ACR DSI leverages two ACR open frameworks, Technology-Oriented Use Cases for Healthcare AI (TOUCH-AI) and Computer Assisted Reporting with Data Science (CARDS). It uses these frameworks to deploy an array of tools and common data elements that radiologists can use to create annotated data sets for training, testing, and validating algorithms, integrating algorithm outputs into the clinical workflow, and monitoring algorithms’ performance in clinical practice.

Train and Test Algorithm

Algorithm Training and Testing:

ACR DSI makes tools available to help radiology departments create training and testing data sets around its use cases. ACR DSI maintains a public directory of institutions that have created such data sets and assists in connecting these institutions with developers who want to use the data sets to build algorithms around its use cases.

Validation:

ACR DSI Use Cases include data elements that use statistical tools to analyze algorithm performance across multiple sites prior to FDA approval. ACR DSI’s validation service, known as ACR Certify-AI, offers developers a secure method to statistically validate their algorithms and gather performance data ahead of the FDA pre-market approval process, going far beyond the demographic scope of the original training and testing data sets.

Deploy Algorithm

Algorithm Deployment:

ACR has longstanding relationships with numerous health care IT vendors and has worked with them to deploy tools, such as ACR Select and ACR Assist, into the workflow of radiology professionals and other physicians. ACR DSI Use Cases build on these interfaces to seamlessly integrate AI algorithms into clinical practice.

Built for each use case, ACR Assist–AI modules leverage these pathways to create standard application programming interfaces (APIs) that integrate algorithm outputs into existing reporting software, PACS, electronic health records, and other electronic resources. ACR DSI creates unique identifiers for each AI algorithm developed around its use cases and maintains a public directory of these algorithms. It also manages a directory that tracks which reporting vendors have incorporated which models into their reporting platforms.

Post Deployment Algorithm

Post-Deployment Monitoring:

ACR DSI uses these data elements to collect specific information about the algorithm’s effectiveness from both radiologists at the point of care and metadata about the exam, including the equipment manufacturer, the field strength or number of detectors, and other relevant examination parameters. ACR DSI collects these data in the ACR National Radiology Data Registry, and the ACR DSI algorithm monitoring service, ACR Assess-AI, sends effectiveness reports to developers for FDA post-market surveillance and for adaptive learning to improve algorithm performance in the field.