How AI and ML are Impacting Continuous Certification, Health Care, and Exam Development: Lessons learned from the ABFM

Description: The rapid emergence of artificial intelligence (AI), machine learning (ML), and Large Language Models (LLM) are already impacting certifying boards and their diplomates in many ways. In health care, AI and ML have revolutionized the way clinical notes can be generated and analyzed. In the realm of exam development, these technologies have facilitated hierarchical item classification in medicine examinations, form assembly and detection of bias in items, and enhanced the reliability and fairness of assessments.

Furthermore, AI and ML have played a pivotal role in advancing continuous learning, providing personalized behavioral data collection and adaptive content that cater to the unique needs of individuals. Together, these advancements underscore the fundamental impact of AI and ML, reshaping how professionals approach health care delivery, educational assessment, and lifelong learning, thereby setting a new standard for excellence in these fields by innovation.

The American Board of Family Medicine (ABFM) presented updates on its early experience with AI, ML and LLM at ABMS Conference 2023, many of which have advanced significantly since then. In this session, ABFM will briefly highlight experiences from the use of AI, ML, and LLM applications across three areas of its work:

1) A machine learning analysis conducted on clinical notes related to post-COVID conditions in primary care settings
2) The application of AI in item classification, form assembly, and bias detection
3) Leveraging ML to facilitate more personalized learning experiences in longitudinal assessment platforms

This session will include time for attendees to react and suggest opportunities for further work in these areas that could be broadly applicable across the ABMS community.