A paper was published in JACR today summarizing a November 2022 ACR-RSNA workshop on safety, effectiveness, reliability and transparency (SERT) in artificial intelligence (AI). Participants predicted that establishing systems that boost these SERT concepts will be the key to full adoption of AI in radiology.
Although there has been a steep uptick in the number of FDA-cleared imaging AI products on the market, the adoption of these products into clinical practice has increased at a much slower pace. The authors, including ACR DSI Chief Medical Officer Bibb Allen, M.D., FACR, and ACR Informatics Commission Chair Christoph Wald, M.D., Ph.D., MBA, FACR, said workshop participants emphasized that implementing AI in radiology will continue to be limited until the safety, effectiveness, reliability and transparency of these AI products be more fully established and understood.
Participants in the one-day workshop shared their experiences and problems with implementing AI into their own practices and found that the current needs and potential solutions related to the safety, effectiveness, reliability, and transparency of AI algorithms generally fell into two broad categories: AI product development and implementation of these products into clinical practice.
The group of diverse radiologists at the workshop found that safe, effective, reliable and transparent AI product development should include clear clinical task definitions; well-curated data from diverse geographic, economical, and healthcare settings; standards and mechanisms to monitor model reliability; and transparency regarding model performance, both in controlled and real-world settings, according to the paper.
When implementing these AI products into clinical practice, the participants stressed the need for strong institutional governance; systematic selection and evaluation of AI performance (acceptance testing) on local data overseen by local teams; seamless integration into the clinical workflow; performance monitoring and support by local teams; performance monitoring by external entities; and alignment of incentives through credentialing and reimbursement, the authors said.
Dr. Wald shared that the ACR is already building on the insights: “We are very pleased to have received funding from the Gordon and Betty Moore Foundation for a pilot study on key aspects of the AILifecycle across real world sites. Importantly, the pilot will serve to expand the use of the existing ACR AIRegistry to support some of these critical tasks for the benefit of member sites, specifically acceptance testing and real-world performance monitoring.”