A logistics company runs its annual review. Two depot managers each have a team of eight. Anders rates six of his eight as “exceeds expectations”. He likes his people and sees no reason to demotivate them. Birgit, at a depot of similar size and output, rates one as “exceeds”, six as “meets” and one as “below”. She believes a high rating should mean something.
At the pay review, the bonus pool is distributed by rating. A solid performer in Anders’s depot receives twice the bonus of a stronger performer in Birgit’s. When HR compares the two depots’ operational results, they are nearly identical. Nothing in the system was broken. The form was completed correctly in both places. The company had assumed that a “3” from one manager and a “3” from another referred to the same thing, and it had never checked.
Performance ratings are used to decide pay, promotion and, sometimes, who loses their job. They are treated as measurements of the employee. A long line of research suggests they are at least as much measurements of the person holding the pen.
This guide explains what that research found, describes the biases that distort reviews and sets out what organisations can do. The aim is not to abandon judgment about performance. It is to make that judgment more accurate and fairer.
Ratings reflect raters.
In a major study, about 62% of rating variance came from the individual rater and about 21% from actual performance.
Leniency and recency are the everyday culprits.
Managers avoid discomfort and remember recent events.
Evidence, anchors, more than one view.
Keep notes all year, define levels in behaviours, pool several raters and show managers their own patterns.
How much of a rating is about the person being rated?
Less than half, on the best available evidence.
In a study of nearly 4,500 managers, each rated by two bosses, two peers and two subordinates, the largest source of variance in ratings was the individual rater’s idiosyncratic tendencies, which accounted for about 62% of the variance. Actual performance of the person being rated accounted for roughly 21%.
Researchers call this the idiosyncratic rater effect. Each rater has a personal pattern: how tough they are, what they value, how they interpret the scale. That pattern follows them from employee to employee. Two managers rating the same person often disagree substantially. The correlation between two supervisors’ ratings of the same employee is typically only moderate.
The consultants Marcus Buckingham and Ashley Goodall drew attention to this work when describing Deloitte’s redesign of its appraisal system, and the finding has unsettled many HR functions since. If most of the variation in ratings comes from raters, then ranking employees across teams by their ratings is in large part ranking their managers’ habits.
What are the main rating biases?
The rater’s fingerprint is made up of several distinct tendencies.
| Bias | What it looks like | How to spot it in the data |
|---|---|---|
| Leniency | Most of the team rated above average | A manager’s distribution sits high every year |
| Severity | Few or no high ratings | A manager’s distribution sits low every year |
| Central tendency | Everyone gets the middle rating | Almost no variation within the team |
| Recency | The last six weeks outweigh the previous ten months | Ratings track year-end events |
| Halo and horns | One strength or weakness colours every dimension | Scores across dimensions nearly identical |
| Similar-to-me | Higher ratings for people like the manager | Patterns by background, style or alma mater |
| Contrast | Rated against teammates, not against the standard | A good performer in a strong team rated low |
| First impression | Early view never updated | Ratings unchanged despite changed performance |
Leniency is the most common in practice, and the least mysterious. A manager who gives low ratings has to justify them, deliver them face to face and then work with a disappointed team. A manager who gives high ratings is thanked. Unless the organisation pushes back, ratings drift upward until the scale stops discriminating. Anders is a rational man responding to the incentives in front of him.
Memory favours what happened lately. A manager writing a review in December remembers the November project vividly and the February one hardly at all. Employees know this, which is why visible effort so often peaks just before review season.
Do biases fall unevenly across groups?
Yes. Beyond the general noise, there are patterned differences in how groups are described and rated.
Studies of the language of written reviews have found that women receive more comments about personality and communication style and fewer tied to specific business results. An often-quoted informal analysis of reviews in technology firms by the linguist Kieran Snyder reported that critical comments on personal style, with words such as “abrasive”, appeared far more often in women’s reviews than in men’s. More systematic academic work has reported similar patterns for vagueness of feedback.
Ratings of future promise are particularly exposed. In one large retailer, women had higher performance ratings and lower potential ratings than men, a result discussed in our guide to promotion bias. Visibility matters as well: employees who are physically present tend to be rated as more committed regardless of output.
What the employee writes about themselves is a further source of distortion. Self-assessments differ systematically between people and groups, and when managers see them before rating, their ratings can be pulled toward them. See our guide to self-assessment psychology.
Why does the rating scale itself cause trouble?
Because abstract scales invite interpretation. “Exceeds expectations” depends entirely on what the manager expected. A five-point scale with labels such as “outstanding”, “strong” and “developing” asks the rater to convert a year of work into a single figure using a private exchange rate.
Two design choices help.
Behavioural anchors. For each level of each dimension, describe what it looks like in observable terms. “Delivers projects on time” at level 3 might read “meets agreed deadlines on most projects; flags delays in advance”, and at level 5 “meets deadlines consistently and helps others recover slipping work”. Anchors give raters a common reference.
Questions about the rater’s own intentions. Deloitte’s redesign asked team leaders not to rate qualities but to respond to statements about what they themselves would do, such as whether they would always want this person on their team or would award them the highest possible compensation increase. The reasoning is that people are more reliable reporters of their own intentions than of someone else’s abstract attributes.
How do you reduce bias when writing reviews?
With evidence gathered through the year and a little discipline at the end.
Keep a running record. Ten minutes a month noting specific achievements and problems for each team member transforms the year-end review. It defeats recency bias and supplies the examples that make feedback useful.
Collect more than one view. Averaging across several raters cancels some of each rater’s idiosyncrasy. Input from peers, project leads and internal customers also covers work the manager never saw.
Rate the parts before the whole. Score each dimension separately against its anchor, then form the overall view. Starting from a global impression is how halo spreads.
Write the evidence, then the rating. If you cannot cite specific examples for a rating, treat it as a guess.
Check your own pattern. Look at your ratings over three years. If everyone is above average, or everyone is in the middle, the scale is not doing any work.
Does calibration fix it?
Partly. Calibration meetings, where managers compare and justify their ratings to peers, address differences in leniency between managers. Anders would have had to explain six “exceeds” ratings alongside Birgit’s one.
Calibration has its own dynamics. Outcomes can depend on which manager argues most forcefully. Managers who know their ratings will be marked down inflate them in advance. And employees are discussed by people who have never worked with them. Good practice is to require written evidence in advance, have each participant record a view before discussion, appoint someone to challenge unsupported claims, and review outcomes by group afterwards. Where calibration is combined with a fixed quota for each rating, the problems multiply; that is the subject of our guide to forced ranking.
Changing an employee’s rating in calibration without the manager being able to explain the reason to the employee is a reliable way to destroy trust in the whole process. If a rating moves, the manager must leave the room with the evidence that moved it.
Should you abolish ratings?
A number of large companies have tried. Adobe replaced annual reviews with regular “check-in” conversations in 2012 and reported saving a very large number of manager hours and seeing lower voluntary turnover. Deloitte, Accenture, Microsoft and others made changes in the following years.
The results are mixed. Research by the advisory firm CEB, now part of Gartner, reported that in organisations that removed ratings, employee performance and the quality of manager conversations often declined, apparently because managers without a rating to explain had fewer structured conversations and employees could not tell how pay decisions were made. Some companies later reintroduced a simplified rating.
The lesson seems to be that the number is not the main problem. The problems are infrequent conversations, vague standards and unexamined bias. Removing the number without fixing those leaves pay decisions just as subjective and less visible.
Whatever system you use, separate two conversations that are usually combined: one about development (“how can you get better?”) and one about reward (“what will you be paid?”). People cannot listen to development advice while waiting to hear their number.
What should employees be told?
How the system works, including its limits. Tell people what the standards are, who contributes to their rating, how calibration operates and how to raise a concern. A process that is explained and consistently applied is accepted far more readily than one that is opaque, even when the result disappoints; the reasons are set out in our guide to organisational justice.
And give the feedback in a form the person can use. The rating is the least informative part of a review. The examples behind it, discussed in the way described in feedback psychology, are where the value lies.
The logistics company introduced three changes. Managers now keep monthly notes in a shared template. Ratings use behavioural anchors written by a group of depot managers. And each year HR shows every manager their own rating distribution next to the anonymised distribution of their peers. Anders’s ratings moved toward the middle within one cycle. Nobody told him to change them. He had simply never seen his own pattern before.
Frequently asked questions
What is the idiosyncratic rater effect?
It is the finding that a large part of the variation in performance ratings reflects the individual rater’s personal tendencies, such as how lenient they are and what they value, and not differences in the performance of the people rated.
What are the most common performance review biases?
Leniency, central tendency, recency, halo, similar-to-me, contrast and first-impression effects. Leniency and recency are the most widespread.
How can managers reduce bias in performance reviews?
Keep dated notes through the year, rate specific dimensions against behavioural anchors before forming an overall view, gather input from other people, support every rating with examples and compare your rating pattern with peers.
Should companies get rid of performance ratings?
Not necessarily. Some organisations that removed ratings saw manager conversations and perceived fairness decline. Improving standards, evidence and the frequency of conversations matters more than whether a number is attached.