Can AI Evaluate Assessment? A Study of Large Language Model Meta-Assessment Performance

Kyle Green, Yu Bao, Stephanie LeRoy, Megan Good   |    Volume 21 Issue 2  |   

Meta-assessment—the practice of evaluating the quality of assessment reports—summarizes institutional measurement efforts and, when used well, offers formative feedback to improve future processes. This study examines whether artificial intelligence (AI), specifically the large language model-based AI systems ChatGPT-5 and Microsoft Copilot Pro, can assist in this process. Using a binary checklist and a multi-criteria rubric, we compared AI-generated ratings to those of a human expert across three versions of a fictional assessment report (strong, moderate, weak). Each condition was replicated five times to assess response variability. AI aligned with human ratings in 59% of individual cases across all report versions, evaluation formats, assessment elements, AI platforms, and replications, but performance significantly varied depending on these factors, with stronger agreement when assessing high quality reports and when using the checklist format. Both platforms tended to struggle with evaluating measurement. Analysis of AI-generated rationale revealed inconsistencies and a tendency to overlook methodological flaws in weaker reports. While AI is not a replacement for human expertise, it may serve as a supplemental tool for reviewing high-quality reports or specific assessment elements. We discuss practical implications and future directions for AI-supported meta-assessment.

Click here to download the full article

« Back to Archive