Objective: SP scoring has been shown to be accurate, but that reliability decreases quickly with increasing checklist length.  A separate observer may be able to reliably score a greater number of items, but adds to the cost of the program.   The goal of this study was to assess the reliability of observers vs. SP scoring from memory of long checklists on an extended SP history and physical examination (H&P). 

Methods: For three years, all second year medical students (MS-2) were evaluated on their performance of a complete (60 minute), head-to-toe H&P, after their Clinical Skills Course (CSC).   A faculty member observed the entire exam and used a comprehensive 130-item checklist to score the students' performance.  SPs also scored the students using a shortened checklist made up of a subset of 50 of the same items.   Responses for each item were compared between the faculty and SP checklists.  Data were also analyzed by faculty member, item number, and total score, to identify characteristics that affected the reliability.

Results: Data are available for 574 exams and 27,849 of 28,700 possible items (97%).  Overall, 91.0% of available responses were in agreement.  There were five faculty members who averaged less than one disagreement per exam, and two who averaged over ten disagreements.  There were 11 items which had fewer than two disagreements, and 10 items with 15 or more, with a high of 33. Increasing student score on the exam was associated with a decreasing number of disagreements. 

Conclusion: There was very good agreement between Faculty members using a 130-item checklist, and SPs using a 50-item checklist.  Comparing equivalent item scores yielded a useful technique for identifying faculty members who may need more training, items that may be poorly written or unclear, and identifies a systematic way of identifying which taped exams to observe for quality control.