Performance review bias checker

The complete guide to spotting and reducing performance review bias

The checker above gives you a bias risk level and three fixes; the ratings spotter flags patterns in a handful of scores. This guide explains why rater bias matters so much, how to use both parts of the tool, how to read the results, a worked example, the most common mistakes, and what to change first. It is written for People leaders and managers who suspect their ratings say more about who did the rating than about who was rated.

Why rater bias is the problem to solve first

A performance rating is supposed to describe the employee. Research suggests it often describes the rater. In a well-known study published in the Journal of Applied Psychology, Scullen, Mount and Goff analysed ratings of 4,492 managers, each rated by two bosses, two peers and two subordinates (doi:10.1037/0021-9010.85.6.956). As Buckingham and Goodall summarise it in Harvard Business Review, 62% of the variance in those ratings could be accounted for by individual raters’ peculiarities of perception. Deloitte, they write, found the same “idiosyncratic rater effect” in its own data.

That matters because ratings drive decisions: who gets promoted, who gets a pay rise, who goes onto an improvement plan. If a large part of a rating reflects the manager, then two equally strong people in different teams can get different outcomes, and the company may lose people it would have wanted to keep.

Bias is also about the quality of feedback, not only the number. Correll and Simard’s HBR article “Research: Vague Feedback Is Holding Women Back” describes how vague, non-specific feedback can disadvantage some groups more than others, which is one reason this checker asks whether ratings are tied to goals and documented examples.

Finally, there is plain inconsistency. Daniel Kahneman and colleagues call it “noise” in HBR: people following the same guidelines make different judgements on the same case, and organisations are usually unaware of how much. Their remedy, a noise audit where several people judge the same cases, is essentially what a good calibration meeting does for performance ratings.

The biases, in plain terms

The performance review bias guide goes deeper on each type with examples.

How to use the checker and the ratings spotter

The process checker asks eight questions: whether managers use evidence from the whole period, whether each competency is rated separately, whether you check distributions by manager, whether raters are trained on similarity bias, whether managers calibrate together, whether ratings are tied to goals and examples, how many people give input, and how often you hold performance conversations. Answer for what happened in your last cycle, across the typical team.

The ratings spotter takes 4 to 8 ratings on a 1 to 5 scale from one manager or one team. Enter numbers only, never names. It flags three patterns that numbers can reveal: central tendency (most ratings at 3, or very little spread), leniency (most ratings at 4 or 5, or an average of 4 or more) and severity (most at 1 or 2, or an average of 2 or less).

Use the checker for your process and the spotter for a specific team you are worried about. Run the spotter on two or three managers’ ratings side by side before calibration to see where to focus the discussion.

How to interpret your result

Each checker answer adds 0 to 2 risk points, for a maximum of 16. 0 to 5 is Low, 6 to 10 is Medium and 11 to 16 is High. Those are the tool’s own thresholds, a checklist score rather than a statistical benchmark. The three fixes shown are the areas where your answers add the most risk, so start there.

• Low: your process has the main safeguards. Keep checking distributions each cycle; bias returns when calibration gets rushed.
• Medium: some safeguards are missing. Usually it is calibration or evidence. One cycle with structured calibration typically moves you down a band.
• High: ratings are probably not safe to use for pay or promotion decisions without a second look. Add calibration and written standards before the next cycle.

For the ratings spotter, a flag is a reason to look closer, not proof of bias. A team of genuinely strong performers can produce high ratings. The test is whether each high rating can be backed by evidence against the written standard for that role and level. If it can, the flag is a false alarm; if it cannot, you have found leniency.

Worked example: one team’s ratings

An illustrative engineering manager rates seven people: 4, 5, 4, 3, 5, 4, 5. The spotter reports an average of 4.3 and that 86% of ratings are 4 or above, and flags possible leniency.

The People partner then runs the process checker for the company. Reviews draw on memory plus some notes (1 point), competencies are rated separately (0), distributions are not checked by manager (2), there is no similarity training (2), calibration is informal (1), ratings are partly tied to goals (1), input comes from the manager only (2), and conversations are twice a year (1). Total: 10 points, Medium, close to High. The top fixes are distribution checks, adding a second input, and similarity awareness.

In calibration, the manager is asked to name the two strongest performers and show the evidence for each of the three 5s. Two hold up against the bar for their level; the third reflects a strong final month after a slow start (recency). That rating moves to a 4, with the reason logged. Two of the 4s are compared with a peer team where similar output was rated 3; after discussion, one moves. The final distribution is still positive, but now every rating has a written justification, and the manager knows what “5” means next time.

Sharing the result with your leadership team

A bias risk score is easier to act on when leaders see it alongside a concrete pattern. Bring three things to the conversation: the risk level and the three fixes from the checker, one anonymised set of ratings from the spotter that shows the pattern (for example, a team where every rating is a 4 or 5), and a proposal for the single change you want to make before the next cycle. Keep the discussion on the process, not on individual managers. The aim is a review system that produces fair outcomes whoever does the rating, which is easier to agree to than an accusation that a particular manager is biased.

Common mistakes

• Forcing a curve to “fix” bias. A forced distribution swaps one distortion for another. McKinsey’s 2024 review describes a shift among companies from forced distribution to distribution guidance. Use the distribution as a prompt for questions, not a quota.
• Training without process change. Unconscious bias training alone rarely changes ratings if the process still allows a single rater working from memory.
• Calibrating opinions. A calibration meeting without evidence becomes a debate won by the most confident manager. Require a one-line evidence note per rating.
• Entering names. The spotter only needs numbers. Keep personal data out of browser tools.
• Checking once. Bias drifts as teams and managers change. Re-run the checker each cycle.

What to change first

Most companies get the biggest reduction in risk from three changes: a written talent bar so everyone rates against the same standard, evidence collected through the period so reviews do not depend on memory, and a structured calibration meeting with the three checks in the calibration guide. For a step-by-step plan, read how to reduce manager rating bias in performance reviews, and for a ready-made agenda, the calibration meeting template. To see how much time calibration currently costs, use the review cost calculator.

Sources: Scullen, Mount & Goff (2000), Journal of Applied Psychology, doi:10.1037/0021-9010.85.6.956; Buckingham & Goodall, Reinventing Performance Management, HBR (2015); Kahneman et al., Noise, HBR (2016); Correll & Simard, Vague Feedback Is Holding Women Back, HBR (2016); McKinsey (2024).