Medical trainees are assessed performing clinical tasks but the examiners’ ratings can be highly variable. It is assumed that examiners assessing the same performance should form similar judgments and provide similar ratings. As such, the psychometric models currently used to analyze the ratings assume there is a single point of consensus. This research, however, found multiple clusters of consensus within the variable assessments provided by examiners for a single performance. This finding was consistent across two samples of participants and two different methodologies. Finding more than one point of consensus challenges the use of psychometric models to analyze examiners’ ratings.