Adventures in AI-Assisted Item Writing—Psychometric and Legal Perspectives

Description: This session explores the use of Large Language Models (LLMs) in generative artificial intelligence (AI) platforms to create valid exam content more efficiently. Featuring insights from senior psychometricians at the American Board of Surgery and the American Board of Family Medicine, alongside legal perspectives from an attorney with vast experience in the testing industry, the panel will examine and discuss the outcomes of a pilot program to use LLMs to develop content for in-training exams and generation of cloned items within a longitudinal assessment framework to evaluate learning outcomes.

Discussions will include strategies to use LLMs to generate valid, blueprint-aligned items, experiences with item writers, and comparisons of content created by LLMs and humans. Legal considerations, including copyright infringement risks and alternative legal strategies to protect AI-generated content, will be thoroughly addressed. This interdisciplinary session aims to outline best practices, opportunities, and challenges, setting the stage for future innovation in exam creation through the use of LLMs.

We also plan to have a moderator for the session: Ray O'Meara, Director of Assessment, American Board of Surgery.