Construct-Aligned Rating Scales Improve the Reliability of Program Evaluation Data

Abstract

In workplace-based assessment, research has suggested that aligning rating scales with how clinical supervisors naturally conceptualize trainee performance improves reliability and makes assessment more efficient. This study examined the generalizability of those findings for program evaluation by determining if construct alignment improves the reliability with which competencies are ranked as having been achieved in a medical education program. These results extend previous research into the benefits of construct-aligned scales by suggesting that aggregating students’ judgments of their abilities can be used to evaluate the relative successes of a program more efficiently when the scales used are aligned with the constructs of independence and sophistication rather than being phrased in terms of students’ performance expectations.