How to evaluate candidates consistently: a worked example

Your 3 interviewers met the same candidate and scored her 2, 4 and 5 out of 5. You need to evaluate candidates consistently, and a 3-point spread on 1 candidate says the process is not there yet.

That spread is normal, not a team failing. Separate interviewers agree at only 0.44 on a 0 to 1 scale, against 0.74 for a panel. So consistent candidate evaluation is something you check, not assume.

Skip the check and 1 careless score can hand the offer to the weaker finalist. In section 8 it does, 3.67 to 3.53, until the evidence check moves the stronger one to 3.80.

This guide adds the missing step: checking, with 1 subtraction, that your interviewers agree before anyone averages. Our interview scorecard template covers the form, and our candidate interview feedback tool guide covers getting usable write-ups.

Key takeaways

• My interviewers scored the same candidate 2, 4 and 5. Is that normal? Yes. Separate interviewers agree far less than a panel does, so a 3-point gap is common. Check it before you average.
• What makes interviewers score the same way? 4 or 5 criteria from the job, a written answer for every score, and a 45-minute calibration first.
• What goes up before anyone in the debrief speaks? The spread table: every interviewer's locked scores, 1 quote each. The talk then covers only gaps of 2 or more.
• How do I check agreement without statistics? Subtract the lowest score from the highest per criterion, and send a gap of 2 or more (on 1 to 5) back to the evidence.
• Can the weights change who gets the offer? Yes. In the worked example, Candidate A wins 3.80 to 3.67 weighted but loses 3.58 to 4.00 on a plain average.

1. My 3 interviewers rated the same candidate 2, 4 and 5. How do I evaluate candidates consistently?

Treat the 2, 4 and 5 as a signal to check before you average them. To evaluate candidates consistently, fix 6 things: criteria, anchors, calibration, quotes, scoring alone and a spread check.

The spread itself is normal. Huffcutt and colleagues' foundational 2013 meta-analysis pooled 125 reliability estimates from 32,428 people. Separate interviews by different people agreed at only 0.44 on a 0 to 1 scale, against 0.74 for panels.

This happens because each interviewer scores from a private idea of what good looks like, which nobody wrote down. US OPM says structure raises agreement by limiting "the amount of discretion an interviewer is allowed" (OPM).

So shared criteria and anchors are worth the effort. Structured interviews ranked first among the selection methods reviewed in Sackett and colleagues' foundational 2022 re-analysis, even after stricter corrections.

Sections 2 to 7 take the 6 parts in order, and section 8 runs them all on 1 illustrative role, not customer data. There, checking each score against its quote moves Interviewer 1's 2 to a 4, so the 2, 4 and 5 become 4, 4 and 5.

Before and after the evidence check: Interviewers 1, 2 and 3 scored 2, 4 and 5, then 4, 4 and 5 (illustrative).

2. How do I turn the job description into 4 or 5 criteria everyone scores the same way?

Pull the 4 or 5 results the role must deliver in its first 6 months, and write each as something a candidate can show. Weight them before the first interview, then use them at every stage.

Criteria fail when they describe a personality instead of a result, because nobody can hear a "commercial mindset". "Keeps renewals on track" points at work a candidate can describe.

If a hiring manager will "know it when they see it", ask what the best person in the job did by month 3 that an average one did not. That answer becomes a criterion in your candidate evaluation framework.

More structure has limits. In a foundational 1994 meta-analysis of 114 validity estimates, Huffcutt and Arthur found that past a point, extra structure added essentially nothing.

Here is an illustrative Customer Success Manager role, weighted 4 for a must have down to 1.

The last column fixes where you check each criterion, so no stage judges something new. Any must-have below 3 means no hire, the floor from our scorecard template. With several interviewers, apply it to that criterion's average.

How Flowmingo helps

Paste your job and the AI Interviewer drafts the questions and weighted criteria. You can edit the questions before you invite anyone. A Must Have counts 4 times as much as a Good To Have, and it never rejects anyone automatically (How we score). CV screening uses the same criteria as the interview (FAQ).

3. What should a 1, a 3 and a 5 actually look like for each criterion?

Write what a 1, a 3 and a 5 answer contain: what the candidate did and showed, not how they came across. Add 1 example line per score, so every interviewer listens for the same thing.

Anchors work because they move the judgment from taste to evidence. "Great energy" describes a feeling, while "named the action and the result" describes an answer.

US OPM says the most structured interviews use "benchmarks of proficiency" (OPM). Levashina and colleagues' foundational 2014 review lists rating scales as 1 of 8 core topics in the research.

Write the 1, 3 and 5 first, then the 2 and 4. Here is the illustrative must-have, for "Tell me about a renewal you nearly lost."

A 2 names an action and a 3 adds the result, so 2 listeners land on the same number. The University of Washington's HR guide asks raters to agree beforehand what separates a 2 from a 3, which this anchor writes down.

On 1 to 5, a 3 is real credit, so average only the interviewers who heard evidence. If nobody did, ask again before the debrief.

4. How do I run a calibration session so my interviewers evaluate candidates consistently?

Book 45 minutes before the first interview, and have everyone score the same 2 sample answers alone. Then reveal the scores and rewrite any anchor that produced a gap of 2 or more.

Researchers call this frame-of-reference training. In a foundational 2012 meta-analysis with over 4 times the earlier review's studies, Roch and colleagues found it made ratings more accurate. Our agenda is our own version, not the paper's protocol.

Our structured interview guide calibrates on 1 answer, and the longer session in our interview bias guide uses 3. This version uses 2, the fewest that show whether a gap repeats.

Read the anchors for 5 minutes, watch 2 sample answers for 10, and score alone with 1 quote each for 10. Then reveal and check the spread for 10, and fix the anchors for 10.

Calibration session: 5 steps, 45 minutes, with the reveal and spread check as step 4.

Step 4 matters most, because everyone heard identical words. So a 2 against a 4 on the same recording is the anchor's fault, and you fix the wording, not the candidate.

Use a recorded answer from a past hire who agreed to it, or write 2. Rerun the session when an interviewer joins, the role changes, or over 1 in 4 scores spread 2 or more.

Hiring alone, calibrate against yourself: score 2 recordings, then rescore them blind a week later. A gap of 2 or more means the anchor needs work.

5. How do I get interviewers to write feedback with evidence instead of 'good vibes'?

Make 1 quote or concrete example the price of every score: no quote, no score. Anyone can check a 4 beside "I renewed 38 of my 40 accounts", and nobody can check "good vibes".

Busy interviewers write what they remember, which is usually a feeling. A post on r/humanresources says interviewers write essays or "good vibes" instead of scores.

In 3 foundational 2013 studies, Dana, Dawes and Peterson found people formed impressions just as confidently from random answers as from real ones. The participants were students predicting grades, so the hiring link is an analogy.

So ask for a 3-line note per criterion: the score, the candidate's words, and what the next score needed. For example, Interviewer 2's note on Candidate A gives a 4, quotes "I renewed 38 of my 40 accounts", and adds "no change after the 2 losses".

Collect notes within 24 hours, while the answers are fresh. Yale's HR guide asks for evaluations within 1 day.

Write each note as if the candidate will read it, because in the UK they can ask for it (section 9). Our candidate rejection email templates turn the notes into 1 honest strength and 1 gap.

How Flowmingo helps

In a Flowmingo AI interview report, every criterion rating comes with written reasons, strengths and gaps, plus the recording and full transcript (FAQ). A highlight must be an exact transcript phrase, and a paraphrase does not count as a quote (How we score).

6. Should my interviewers score on their own before the debrief, or talk it through first?

Score alone first, every time. Each interviewer submits scores and quotes before seeing anyone else's, then the debrief discusses only the gaps and a formula combines the rest.

Talking first lets 1 opinion set the bar. The foundational 2021 book Noise, by Kahneman, Sibony and Sunstein, names independence as 1 of 3 principles of structured judgment (quoted in reader highlights). US OPM has panel members rate "applicant responses individually" before the group discussion (OPM).

In Conway and colleagues' foundational 1995 meta-analysis, several ratings helped only when a fixed rule combined them, with no evidence of help from discussion.

In Kuncel and colleagues' foundational 2013 meta-analysis, a fixed formula predicted job performance more than 50% better than people weighing it all in their heads. It held even for experts who knew the jobs, and the data pooled many kinds of selection information.

Run the debrief in this order: each person reads their quote, a score moves only against the anchor, and the most senior person speaks last.

Debrief order: score alone, lock and show the spread table, discuss gaps of 2+, then average by weights.

6.1 Should my panel interview together or separately, and how many interviewers do I need?

Either works if everyone scores alone. Panels agreed far more in the Huffcutt meta-analysis (section 1), likely in part because panelists all see the same performance, our reading.

Separate interviews give more independent looks. Singapore's TAFEP handbook suggests more than 1 interviewer where possible, and a diverse panel to help minimise bias.

So use up to 4 interviewers in separate rounds, 1 of them a specialist. Each asks the same core question per criterion and scores every criterion, and the specialist also probes 1 must-have deeper.

Never split the criteria: 1 score per criterion leaves the spread check nothing to compare. A 5th adds disagreement more than evidence, our rule of thumb.

7. How can I check whether my interviewers actually agree, without a statistics degree?

Subtract the lowest score from the highest for each candidate and criterion. On a 1 to 5 scale, a gap of 2 or more goes back to the evidence.

Check 1, the spread: the 2-point flag is our rule of thumb. OPM tells panels to rate alone and then resolve "significant scoring discrepancies", and 2 points is how we define significant.

In the figure, A's renewals scores of 2, 4 and 5 spread 3 points, while the other 3 criteria agree within 1.

Spread check, illustrative: renewals spreads 3 points and goes back to the evidence, while the other 3 criteria agree within 1.

Check 2, interviewer lean: over 10+ candidates, subtract the team average from each interviewer's average. In the figure, Interviewers 1, 2 and 3 average 3.1, 3.6 and 4.4, against a team average of 3.7.

Their leans are -0.6, -0.1 and +0.7. A lean beyond 0.5 either way, our rule of thumb, earns a 1-to-1 calibration, not blame, so Interviewers 1 and 3 each get one.

Interviewer lean, illustrative: Interviewers 1 and 3 average 3.1 and 4.4 against a team average of 3.7, leans of -0.6 and +0.7.

Check 3, the flag rate: if over 1 in 4 scores in a round sit in a flagged gap, recalibrate, our rule of thumb.

For example, 3 interviewers scoring 4 criteria for 5 candidates give 60 scores, and 18 in flagged gaps is 30%. That points to vague anchors, so fix the anchors, not the people.

Pass rates by group are a separate check, in section 9 and our interview bias guide.

8. Can I see a worked example of how to evaluate candidates consistently, from job to decision?

Yes: 1 illustrative Customer Success Manager role, not customer data, with 4 weighted criteria, 3 interviewers and 2 finalists. Before the check, 1 unchecked score of 2 hands the job to B, 3.67 to A's 3.53 weighted.

The spread check flags A's renewals scores of 2, 4 and 5. Interviewer 1 gave the 2 because A "seemed unsure", yet the same note quotes "I renewed 38 of my 40 accounts".

That matches the 4 anchor, so the score moves to 4. Interviewer 3 keeps the 5, because A added "after losing 2, I started reviews 90 days before renewal" in that round. Interviewer 2 never heard it, so the final 4, 4 and 5 has a spread of 1.

The sum for A: (4.33 x 4 + 3.67 x 3 + 3.33 x 2 + 3.00 x 1) / 10 = 3.80.

Unweighted, B's 2 minor criteria would win it, 4.00 to 3.58.

Both clear the must-have floor of 3, B exactly, so A gets the offer on the higher total. That is how you evaluate candidates consistently: the evidence feeds the formula, and the formula decides.

Weighted totals, illustrative: Candidate A goes from 3.53 before the check to 3.80 after it, past Candidate B at 3.67. Unweighted, B would lead 4.00 to 3.58.

8.1 How do I pick between 2 finalists when their scores are close?

Treat 1-to-5 weighted totals within 0.1 as a tie, our rule of thumb. A and B sit 0.13 apart, so no tie, but a real tie breaks in this order:

1. The must-have average, where A has 4.33 against 3.00 for B.
2. Resolve the widest spread still open, then compare again.
3. A short work sample on the weakest must-have.

Never break a tie on gut feel or "culture fit", as our interview bias guide explains, and write the reason down.

9. What should I keep so a hiring decision holds up if a rejected candidate challenges it?

Keep the criteria and weights, each interviewer's scores with quotes, the spread check and the reason for the call. Keep them at least 12 months after the decision, and longer if a claim is open.

This is general information, not legal advice.

In the US, federal agencies have treated a pass rate under 4/5 (80%) of the top group's as evidence of adverse impact (29 CFR 1607.4). So 20 of 100 passing against 30 of 100 gives 67%, worth a closer look under this rule of thumb.

As of 2 November 2026 this rule, from the foundational 1978 Uniform Guidelines, is still in the eCFR. A 9 June 2026 Justice Department opinion and a July 2026 OPM rule move federal agencies away from it.

The EEOC's agenda lists rescinding them in November 2026, but Title VII and private lawsuits remain. So keep pass rates by group, where lawful, as your own monitor.

Separately, 29 CFR 1602.14 asks covered employers to keep hiring records 1 year from the record or decision, whichever is later. After a charge, keep them until the case ends.

In the UK, the Equality Act bans discrimination in deciding "to whom to offer employment" (s39). Once facts point to discrimination, the tribunal must find it unless the employer shows otherwise (s136).

The EHRC says to keep interview notes 12 months, because they can show you did not discriminate. For claims arising from 1 October 2026, most must be brought within 6 months minus 1 day (Acas).

The ICO's draft guidance says candidates can request interview notes. So record only what recruitment needs, and keep records past the claim period only for a clear business reason.

In Singapore, the TAFEP handbook asks for merit-based criteria, a checklist scored per criterion, and records kept at least 1 year. Singapore's Workplace Fairness Act, passed in January 2025 (MOM), will ban hiring decisions based on 5 groups of traits from end-2027, MOM's target (MOM).

Records by place: keep notes, scores and the reason, 12 months in the UK (EHRC) and at least 1 year in the US and Singapore.

Write every note for a tribunal or the candidate. "Renewals 2: said 'I would reach out early', named no result, so not a 3" passes. "Not a fit" fails, because it points at nothing the job needs.

9.1 Is it legal to ask candidates different questions?

None of these US, UK or Singapore rules requires word-for-word identical questions, but different ones are harder to compare.

Under Title VII, a question asked only of 1 sex, race or other protected group is itself evidence of discrimination. The EEOC calls family questions "clearly discriminatory" if asked only of women, and suspect if asked of all (EEOC). The Uniform Guidelines, still in the eCFR (section 9), count even an informal interview as a selection procedure (29 CFR 1607.16).

In practice, ask everyone the same core question, such as section 3's renewal question. Follow-ups can differ when an answer misses, if they stay on the criterion.

When you must change a question, say for a disability adjustment, note what changed and why.

10. Can an AI interview score help me evaluate candidates consistently, and how do I check it?

It helps, though we have not measured how much, because everyone in a role answers the same questions against the same written criteria. Treat it as a fourth interviewer, not the judge.

An AI interview removes 2 sources of drift: different interviewers and different days. It does not make bias impossible, and its follow-ups still vary, so check it with our 10-candidate spot-check.

If the AI and your interviewers keep disagreeing on 1 criterion, fix its wording once instead of overriding scores 1 at a time.

Flowmingo data · 44,665 candidates with an AI interview score, in 700 interview sets of 10+ scored candidates, at 317 companies, interviews completed 9 Jul to 31 Oct 2026 (current scoring)

• About 2.8 points out of 10 span the middle 80% of candidates in a typical set, so scores still separate people.
• 6.1 out of 10 is the median set's average score, but 1 set in 10 averages 4.8 or less and 1 in 10 averages 7.1 or more.
• About 1.2 points separate a company's highest- and lowest-averaging sets at the median, across 127 companies.
• 6,391 of these candidates, at 277 companies, scored 7.0 to 7.4, and the typical one scored higher than 70% of the others in their set.
• 1 in 10 of them scored higher than at most 46% of the others in their set. For another 1 in 10, it was about 89% or more.

These are Flowmingo AI interview scores, not job performance. Sets differ in role, criteria and applicant pool, so different averages do not show inconsistent scoring. Spread is not proof of the right order, and some interviews ended early.

Same score, different rank: candidates scoring 7.0 to 7.4 scored higher than 70% of the others in their set at the median. At the low and high ends, 1 in 10 each, it was 46% and 89%.

Picture 3 candidates on the same 7.2. It sits well above the middle of a typical set, about the middle of a strong one and near the top of a weak one. So rank candidates inside their own set, and never use 1 fixed cut-off across roles.

The foundational 2024 EU AI Act also lists AI used to evaluate candidates as high-risk (Annex III), so checking it this way is sensible.

In every role, the spread check makes an average safer to use, because it sends the scores that disagree back to their quotes.

How Flowmingo helps

In the AI Interviewer, every applicant for a role gets the same question set. Each requirement gets a 0 to 10 rating, weighted by your settings (How we score). Your team can override any score (FAQ). Run the 10-candidate check free on Flowmingo and evaluate candidates consistently.

11. Sources

Every study, law and guide cited above, listed alphabetically by author. Flowmingo figures come from Flowmingo's own platform data.

• Acas (2026, updated 1 October). Employment tribunal time limits. Acas.
• Conway, J. M., Jako, R. A., & Goodman, D. F. (1995). A meta-analysis of interrater and internal consistency reliability of selection interviews. Journal of Applied Psychology, 80(5), 565-579.
• Dana, J., Dawes, R., & Peterson, N. (2013). Belief in the unstructured interview: The persistence of an illusion. Judgment and Decision Making, 8(5), 512-520.
• Equal Employment Opportunity Commission and others (1978). Uniform Guidelines on Employee Selection Procedures, 29 CFR Part 1607. eCFR, checked 2 November 2026.
• Equal Employment Opportunity Commission (n.d., read 2 November 2026). Pre-employment inquiries and marital status or number of children. EEOC.
• Equal Employment Opportunity Commission (2026). Rescission of Uniform Guidelines on Employee Selection Procedures, RIN 3046-AB43. Unified Agenda, via the Wayback Machine (copy of 19 October 2026).
• Equality and Human Rights Commission (n.d.). What equality law means for you as an employer: When you recruit someone to work for you. EHRC.
• European Union (2024). Artificial Intelligence Act, Annex III: High-risk AI systems. AI Act Explorer.
• Huffcutt, A. I., & Arthur, W., Jr. (1994). Hunter and Hunter (1984) revisited: Interview validity for entry-level jobs. Journal of Applied Psychology, 79(2), 184-190.
• Huffcutt, A. I., Culbertson, S. S., & Weyhrauch, W. S. (2013). Employment interview reliability: New meta-analytic estimates by structure and format. International Journal of Selection and Assessment, 21(3), 264-276.
• Information Commissioner's Office (n.d., read 2026). Shortlisting, testing, and interviewing candidates and Keeping recruitment records [Draft guidance, under review]. ICO.
• Kahneman, D., Sibony, O., & Sunstein, C. R. (2021; paperback 2022). Noise: A flaw in human judgment. Little, Brown Spark. Passage on the 3 principles quoted in reader highlights, Goodreads.
• Kuncel, N. R., Klieger, D. M., Connelly, B. S., & Ones, D. S. (2013). Mechanical versus clinical data combination in selection and admissions decisions: A meta-analysis. Journal of Applied Psychology, 98(6), 1060-1072.
• Legal Information Institute (read 2 November 2026). 42 USC 2000e-2, unlawful employment practices. Cornell Law School.
• Legal Information Institute (read 2 November 2026). 29 CFR 1602.14, preservation of records made or kept, 29 CFR 1607.4, information on impact and 29 CFR 1607.16, definitions. Cornell Law School.
• Levashina, J., Hartwell, C. J., Morgeson, F. P., & Campion, M. A. (2014). The structured employment interview: Narrative and quantitative review of the research literature. Personnel Psychology, 67(1), 241-293.
• Ministry of Manpower, Singapore (2025, January 8). Passing of Workplace Fairness Bill marks next step in building fair and harmonious workplaces. MOM.
• Ministry of Manpower, Singapore (2025, November 4). Workplace Fairness (Dispute Resolution) Bill [Press release]. MOM.
• r/humanresources (2026, July 26). Interviewers never fill out scorecards. What finally fixed it for your team? [Online forum post]. Reddit.
• Roch, S. G., Woehr, D. J., Mishra, V., & Kieszczynska, U. (2012). Rater training revisited: An updated meta-analytic review of frame-of-reference training. Journal of Occupational and Organizational Psychology, 85(2), 370-395.
• Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107(11), 2040-2068.
• Tripartite Alliance for Fair and Progressive Employment Practices (2025, November). Fair recruitment and selection handbook. TAFEP.
• UK Parliament (2010). Equality Act 2010, section 39 and section 136. legislation.gov.uk, via the Wayback Machine.
• U.S. Department of Justice, Office of Legal Counsel (2026, June 9). Constitutionality of disparate-impact liability under Title VII. DOJ.
• U.S. Office of Personnel Management (2026, July 31). Removal of references to the Uniform Guidelines on Employee Selection Procedures in Federal personnel regulations. Federal Register, 91 FR 48234.
• U.S. Office of Personnel Management (n.d.). Structured interviews. OPM.
• University of Washington (n.d.). Candidate evaluation tips and guidelines. UW Human Resources.
• Yale University (n.d.). How to effectively evaluate candidates and make a final selection (FL-HR-069). Yale Human Resources.