Shared Goals, Improved Outcomes: Key Takeaways from the 2024 QSI Conference

Home Blog

Shared Goals, Improved Outcomes: Key Takeaways from the 2024 QSI Conference

October 1, 2024 | Kevin Haines, D.O.

Kevin Haine DO

Kevin Haines, D.O.

PGY-4, University of Connecticut Health Center

Farmington, CT

The rapid rise of artificial intelligence (AI) once promised to address radiology's growing workforce shortages and the increasing demand for imaging, all without compromising quality or safety. Instead, much like the Boeing StarLiner capsule, it has returned to Earth with more questions than answers.

Thankfully, there is a dynamic and dedicated cohort of radiologists, operational managers, technologists, and vendors who are eager to engage with all stakeholders in Radiology: private practices, academic institutions, public regulatory committees, and educators. The American College of Radiology’s 2024 Quality & Safety + Informatics (QS+I) Conference held in Washington D.C. served as an open forum to dissect complex issues and reimagine solutions for Radiology institutions and enterprises. In their opening remarks, Drs. Schlomit Goldberg-Stein and Gloria Hwang, Chair and Vice Chair of the Quality and Safety portion of the meeting, emphasized peer learning as the cultural glue holding the QS+I community together. The weekend’s agenda addressed Radiology’s challenges, AI’s potential, and the future of AI safety, security, and regulation.

Day 1: Addressing Radiology Hurdles

As the saying goes, “To understand where you are going, you must first understand where you’ve been.” Dr. Ashwini Davison, an internal medicine physician now working as the Chief Medical Information Officer of Imaging Data Strategy at Amazon Web Services, gave an insightful presentation on how Quality, Safety, and Informatics have evolved through various political and regulatory landscapes. She highlighted how regulatory pressures have shaped radiology’s current landscape. Dr. Davison highlighted the significant upstream data challenges, particularly those related to governance and image management. During the subsequent discussion, Dr. David Larson aptly cautioned that we must avoid the hype of AI overshadowing Radiology’s real needs.

Keynote Ashwini Davidson at QSI 2024 Other leaders in the QS+I field shared their diverse experiences. Dr. Lane Donnelley, Chief of Quality & Safety at UNC Children’s Hospital, emphasized that radiology quality leaders with operational and process expertise can influence health systems broadly—even in areas where they may not have direct experience, such as laboratory medicine. Dr. Sanjay Shetty, President of Centerwell at Humana stressed the need for Radiology Quality Leaders to be involved at multiple levels of the institution, warning that “if you aren’t at the table – you are on the menu.” Dr. Jay Pahade, Vice Chair of Quality and Safety at Yale Radiology & Biomedical Imaging added that radiologists must be prepared to lead. During his talk, Dr. Pahade stressed that radiologists may be uniquely positioned to understand healthcare delivery throughout the enterprise. This expanded upon a concept introduced by Dr. Shetty who described radiologists as an all-around athlete, different from leaders in other areas. Both Drs. Shetty and Pahade argued that Radiologists are uniquely trained to break down complex problems into manageable pieces and lead across diverse organizations.

During the next session, the speakers identified different ways to address the Radiologist workforce shortage. Dr. Olga Brook from Beth Israel Deaconess Medical Center noted the significant gap between graduating residents and current ACR job postings. She proposed sponsoring international medical graduates. However, she highlighted that the alternate ABR pathway is fraught with challenges. Notably, the H1B visa process is time- and labor-intensive. Dr. Brook showed that the investment in international graduates is worthwhile as many radiologists have successfully navigated the alternate pathway and are now leaders in the specialty.

Dr. Dana Smetherman, ACR CEO, highlighted how reducing distractions can increase radiologist efficiency and improve imaging service quality. During her talk, Dr. Smetherman proposed the concept of a resource or consultant radiologist. Interruptions are focused on radiologists working in this role while their colleagues work relatively distraction-free.

Dr. Myles Taffel, Director of Workflow Analytics at NYU Langone Health, discussed automated study distribution via advanced Radiologist worklists. In his vision, these worklists help to distribute worklists to radiologists assigned to multiple roles across multiple imaging centers. He described efforts to balance RVUs across radiologists. Dr. Taffel also described a new metric, the “relative effort unit” so that more complex examinations can also be distributed equitably.

Finally, Dr. Julia Jacob, Cardiothoracic Division Quality and Safety Officer at the University of Pennsylvania Perelman School of Medicine discussed radiologist extenders. In her model, advanced certified radiologic technologists assist radiologists in their interpretation of radiographs. Dr. Jacob highlighted the careful guardrails placed on extenders. For example, they provide draft reports without impressions and the attending radiologist must overread and finalize each report. Dr. Jacob showed that radiology extenders improved turnaround time within the department and saved 1.8 hours of work per day for attending chest Radiologists. In the later discussion Dr. Manjil Chatterji, Chief Quality Officer from Weill Cornell Medicine, stated that radiologists must better assert their unique role as physicians—identifying work that cannot be easily replaced or encroached upon by other members of the clinical team.

The afternoon breakout session ‘Learning From EPIC Failures: Redesigning the Electronic Health Record (EHR) to Drive QI’ lead by Drs. Jay Pahade, Daniel Glazer, Gowthaman Gunabushanam, and Ms. Anne Gormley, MA gave participants an opportunity to openly discuss the challenges that they face with the electronic health record and its integration with quality improvement efforts. Dr. Glazer highlighted the importance of clearly defining project scope and identifying a project’s impact. This aligned with Dr. Gunabushanam’s four-box model for evaluating processes. This model simplifies projects into four categories: Implement, Possible, Challenging, and Kill.

The breakout session highlighted that effective EHR interventions require predefined outcomes, along with continuous monitoring to ensure that software changes do not disrupt established workflows. Ms. Gormley provided a sobering perspective as an institutional EHR analyst, sharing the complexity of the EHR enhancement request hierarchy. A key takeaway was the idea of establishing an optimization committee to realistically prioritize goals and projects.

Peer Learning goes beyond the traditional RADPEER score-based peer review by incorporating shared experiences, case discussions and constructive feedback as part of a just culture rather than focusing solely on errors. This approach has improved interpersonal relationships and enhanced both individual and organizational performance. There are many practice management resources for those interested in exploring and implementing a non-punitive, peer-learning environment that fosters a culture of engagement and quality.

Drs. Dania Day and Glazer showcased the versatility of peer learning, even in specialties such as Interventional Radiology, where it can boost adverse events reporting. Dr. Mara Kunst discussed how the peer learning framework could be applied to the implementation and ongoing monitoring of AI algorithms. She shared a vision where vendors were engaged in the peer learning process to streamline case submission and prioritize educational content.

Hye Sun Na, AI Evaluation Director at Stanford, presented an abstract describing how large language models (LLMs) can assess the quality of clinical histories for Radiologists. She stated that this

model could help radiology departments audit the information they receive, helping them to work with ordering providers to drive improvement. Ms. Na shared her vision for the future where an LLM helps radiologists by compiling a complete clinical history from relevant notes.

In the day’s final session, Addition by Subtraction, Dr. David Rosman, Mass General Brigham’s Deputy Chair emphasized the importance of information technology integration within institutions. He noted that quality and patient experience need to be central to decision-making. The day’s final speaker, Dr. Juliana Bueno, Professor of Radiology at the University of Virginia, reflected on the human tendency towards additive practices when faced with challenges. She highlighted the value of simplifying training modules and improving communication practices.

Day 2: Altering how we think AI should be implemented, regulated, and perceived

In the day’s opening remarks, Drs. Susan Sotardi and Alexander Towbin, Chair and Vice Chair of the Informatics portion of the meeting, re-emphasized how Quality and Informatics leaders must work together within the specialty to lead change. Dr. Towbin, Associate Chief of Radiology at Cincinnati Children’s Hospital, showcased an example of RadElement an undertaking by RSNA and ACR to drive standardized language and guide quality reporting via common data elements. He shared his enthusiasm for a live demo at the 2024 RSNA Annual Meeting in the Radiology Reimagined Showcase.

“Meh” was the word of the day. Dr. Rick Abramson, Chief Medical Officer of Annalise_AI, introduced it, saying that it captured the current state of diagnostic AI. He recommended that the industry recast AI as a safety device rather than a productivity tool. He emphasized that framing AI around patient safety could help mitigate disparities in underserved populations. Dr. Abramson called for a shift in FDA-clearance standards, suggesting that clinical trials should focus on patient safety impact rather than vague device performance metrics. The challenge, however, is securing reimbursement for AI as a safety tool. He stressed the need for strong advocacy in the approach from the ACR community.

Mike Tilkin, ACR Executive Vice President and Chief Information Officer for the ACR discussed how date is at the core of informatics and AI. He highlighted the ACR’s leadership in radiologic data collection and processing, supported by a range of secure platforms like the Medical Imaging and Data Resource Center (MIDRC), ACR Transfer of Images and Data (TRIAD), ACR AI-LAB, and the ACR Data analysis & research toolkit (ACRdart). These platforms offer datasets and tools for analysis, enabling institutions to access trusted, compliant, and accurate information.

Dr. Ryan Lee, Chair of Radiology at Einstein Healthcare Network, stressed that data governance is a critical issue. He underscored the importance of data lineage–understanding where data comes from, its quality, and how it’s monitored. Dr. Lee also touched on the risks of “adversarial perturbations,” where bad actors manipulate AI to produce misleading results. He reinforced his message with the iconic lines from 2001: A Space Odyssey: “Open the pod bay doors, HAL… I’m sorry Dave, I’m afraid I can’t do that.”

Dr. Tessa Cook emphasized the importance of controlling the data that AI algorithms use. At Penn Medicine, her team rigorously tests AI devices outside clinical workflows to simulate real-world conditions. In her work she stressed the need to clearly define success, saying that the tool has to be right for the department’s needs and patients.

In the next session speakers shared tips on clinical implementation of AI. Dr. Larson humorously likened the state of Radiology AI to a “reverse Lake Wobegon effect’ -- where everyone feels below average. Dr. Matthew Barish, Vice Chair of Informatics at the Zucker School of Medicine, identified two critical factors in deploying AI: generating return on investment and understanding the true cost in implementation. Dr. Jason Poff, Director of Innovation Deployment at Radiology Partners shared the practice’s impressive ability to monitor their clinical algorithms. Dr. Poff showed that routine monitoring can help practices identify model outliers. He then stressed that humans-in-the-loop are needed to differentiate between AI drift and radiologist false negatives. Hye Sun Na shared how she reviewed numerous 510k premarket AI/ML summaries for multiple vendors who offer intracranial hemorrhage protocols. She suggested standardizing these summaries through regulatory guidance could improve comparability. Finally, Ms. Kandice Garcia Tomkins, Quality Improvement Director of the ACR Learning Network advised developing a team-based approach to select AI vendors.

Day 3: What is real and who says so?

Kicking the last day off with a blast of 80’s synth, Dr. Woojin Kim, RadAI Chief Medical Information Officer, ACR Data Science Institute Chief Medical Officer demonstrated the brittle and fragile nature of Generative AI (Gen AI). Dr. Kim warns of generating synthetic data or images, as it may alter minute aspects that we cannot evaluate causing us to unwittingly perpetuate bias. When LLM algorithms attempt to generate reports from radiology images without trained Radiologists as the clinical domain expertise, Dr. Kim showed that there will always be quality deficiencies. To ensure that the Radiology community can define how AI is clinically useful, we will need to routinely reach out across departments, institutions, and cultures to prevent the specialty from suffering our own “model collapse”.

Dr. Laura Coombs, Vice Chair of Data Science Institute (DSI) at ACR provided an insider’s look to the nation’s first AI monitoring registry Assess-AI. This service provides a structured approach to allow the flow of data to the ACRdart portal where concordance performance reports can be generated on real world data to ensure both developers and clinical sites the algorithms are performing as expected.

The ACR Recognized Center for Healthcare-AI, ARCH-AI, presents another first – the first national artificial intelligence quality assurance and accreditation program. Mr. Chris Treml, ACR DSI Director of Operations, described the forward-thinking practice parameters set for the Radiology community. The best practices to adhere to, Mr. Treml shares, include governance, selection, testing, and monitoring.

David Larson asking a question at QSI 2024

Dr. Keith Dreyer, ACR DSI Chief Science Officer, questioned if AI adoption has just plagued us with more data noise and mental effort leading to a larger time burden than originally anticipated. So, to break down how we need to move forward, Dr. Dreyer mapped the Healthcare AI and Medical Device FDA regulatory components. AI software regulation has been defined by what the device claims to do and then its level of ‘risk’ and thus its performance test are established. However, in the new era of Gen AI, defining and evaluating algorithms poses a greater challenge. Dr. Dreyer considers just how the AI application approval process will need to adapt at an institutional, state and federal level. One major take home point explained is that the ACR needs to remain positioned as a pivotal player in establishing medical imaging monitoring, certification, and registration standards.

Dr. Po-Hao Chen, Vice Chair of AI in Radiology at Cleveland Clinic kicked off the information security section with demonstrating that, although Gen AI reconstruction algorithms can produce an image which is 96% accurate from only 12% of provided data, the synthetic 6% could introduce a faux pathology or even render the image useless. Along those same lines, synthetic data produced by Gen AI has potential to outpace originally sourced data leading to increased emphasis on data integrity verification.

Both Dr. Chen and Dr. Safwan Halabi, Vice Chair of Informatics at Lurie Children’s Hospital of Chicago denoted the increasing phishing attacks, data breaches, and cyber-attacks on the health care infrastructures. Growth of an institution’s internet of things through blossoming healthcare AI infrastructure and institutional personnel needs of accessing data remotely increases the potential vulnerabilities which exist in a security net. After sharing his own institutions recent cyber-attack, Dr. Halabi urged training blackout scenarios to expose possible risks. A nimble institution, Dr. Halabi emphasizes utilizes a readymade, easily accessible, and regularly updated disaster recovery plan with a clear chain of command. Looking forward, Dr. Halabi notes the trend of proactive risk identification including utilizing blockchain technology for improved data integrity and security. One interesting ethical discussion question posed was whether practices or institutions should retroactively implement AI algorithms if and/or when AI platforms have been compromised.

Mr. Axel Wirth, CPHIMS, Chief Security Strategist, Medcrypt, shared that cybersecurity events disrupt both temporal and regional mortality rates. Mr. Wirth highlighted a cybersecurity life cycle recommending a clear and widely distributed chain of command. This follows the Manufacturer Disclosure Statement for Medical Device Security (MDS2) which spans general cybersecurity “hygiene” as well as the analysis of risk and license transfer at “end of life” discussions.

As section 1557 of the Patient Protection and Affordable Care Act in enacted on May 1st2025, Dr. Walter Wiggins, AI Innovation Team, Radiology Partners, reiterates how this will transition policy regulation from the AI creator to any entity receiving federal assistance (Medicare/Medicaid). One “insurance policy” that Dr. Wiggins describes is AI post-deployment monitoring to manage AI drift, ensure ROI, and meet evolving regulatory demands. This will enable a multifaceted approach to the general risk in applying new algorithms.

Another consideration of AI implementation workflow provided by Dr. Melissa Davis, Vice Chair of Medical Informatics Yale University, is how the adoption of new algorithms needs to be adaptable to an ever-evolving process. Dr. Davis gave an institutional breakdown of how important it is to establish close ties between IT, Radiology, and Hospital operations to allow both local and system governance to work cohesively. Dr. Davis echoed the difficulties of post implementation monitoring and how to move forward in house as opposed to being reliant on a vendor’s evaluation.

To close out the conference, Dr. Bibb Allen Jr., Senior Advisor of the ACR Data Science Institute, discussed how current regulatory clearance cannot ensure models will work as expected in real-world clinical practice. There is no whole government strategy for oversight of Healthcare AI. Dr. Allen indicates that only 3 government agencies, the FDA, Office of the National Coordinator for Health Information Technology, and the Office for Civil Rights, provide any regulatory oversight. The establishment of AI professional expertise, quality standards, practice parameters and accreditation tracks will carry weight with Congress and other agencies allowing for Radiology to help shape the future of AI regulation.

The Wrap

The shifting sands of AI and Radiology are difficult to contain, define, study, and implement. However, as a first-time attendee to the ACR QS+I conference series, I am humbled at how dedicated and passionate fellow colleagues are with reigning in and focusing the vast amounts of data, personnel, and technology towards patient care. I think this beautifully coincides with the 2024 axiom “Shared Goals, Improved Outcomes”. Many other speakers ended their presentations with a ‘call to action’ or an ‘ask’, so I have given myself the same liberty. I would like to challenge fellow residents and trainees to join in the Quality & Safety + Informatics milieu of difficult questions and tough answers. My favorite phrase shared by experts was “I don’t know” – we have so much to learn and hearing that felt like an opportunity to contribute. So, start local or join the next ACR 2025 QS+I conference in San Diego, because there is a lot of work to do.

Kevin Haines, D.O. | PGY-4 | University of Connecticut Health Center