The short answer
Inter-rater reliability is the degree to which different interviewers, scoring the same candidate's answers, give the same ratings. In hiring, it is usually measured as percent agreement or with Cohen's kappa, a statistic that removes the agreement you would expect by chance. It matters because when inter-rater reliability in interviews is low, a candidate's outcome depends on who interviewed them, not on how they answered.
Inter-rater reliability in interviews, explained
To measure it, two or more interviewers score the same candidates independently. That can be a panel interview or a set of recorded interviews. Then you compare their ratings.
The simplest measure is percent agreement = number of candidates both raters scored the same / total candidates. Its weakness is that some agreement happens by luck. Cohen's kappa corrects for that: kappa = (observed agreement - chance agreement) / (1 - chance agreement). A kappa of 1 means perfect agreement, and 0 means no better than chance. For ratings on a 1 to 5 scale, where a one-point miss matters less than a three-point miss, analysts use weighted kappa or an intraclass correlation instead.
OPM's structured interview guide defines reliability as rating consistency among interviewers, and notes that unstructured interviews typically show low levels of it, while structured interviews have shown a high degree of reliability.
Example: two interviewers, ten candidates
Two interviewers each gave ten candidates a hire or no hire decision.
- Observed agreement: (4 + 4) / 10 = 0.80, or 80%.
- Chance agreement: (0.6 x 0.4) + (0.4 x 0.6) = 0.48. A said hire 60% of the time and B said hire 40% of the time.
- Cohen's kappa: (0.80 - 0.48) / (1 - 0.48) = 0.62.
Eighty percent agreement sounds high, but kappa shows much of it could happen by chance. The two disagreements also run one way: A is more lenient than B. That is a calibration conversation, not a coin flip.
How to raise it
- Ask every candidate the same questions, and score with an interview rubric that has written anchors.
- Rate independently before any discussion, as OPM advises for panels.
- Train interviewers on recorded interviews. OPM's sample training plan includes critiqued practice using a videotaped interview.
- Check each interviewer's average for leniency or strictness, two rating errors OPM names.
High agreement isn't the same as accuracy. Two interviewers can agree and both be wrong. That second question is validity, which OPM describes as the extent to which an assessment measures what it is meant to measure, such as job performance. Our guide to how AI interviews are scored covers both questions for automated scoring.
Related terms
- Interview calibration: the routine that raises agreement between interviewers.
- Interview rubric: written anchors for each rating level.
- Structured interview: the format with the strongest consistency record.
- Unstructured interview: free-form conversation with no common scale.
- Interview scorecard: where each interviewer records independent ratings.
Questions
What is a good inter-rater reliability score for interviews?
There is no single cutoff that applies everywhere, and conventions vary by field. Use kappa rather than raw agreement, track it over time for each role, and investigate any pair of interviewers whose ratings often differ by two or more points.
What is the difference between inter-rater reliability and validity?
Reliability asks whether different interviewers give the same rating. Validity asks whether the rating predicts what you care about, such as job performance. An interview needs both, and reliability comes first: a score that changes with the interviewer can't predict much.
How many candidates do you need to measure inter-rater reliability?
More is better, because a handful of candidates can swing the result. Start with whatever your panel interviews give you, or have interviewers score the same set of recorded sample interviews, and add to it each hiring cycle.
Want to see interview scores you can check against recorded evidence? Book a 20-minute demo or try a sample interview.
