Was It Skill or Was It Luck? The Statistics Behind Exceptional Performance

↓

​Learn more about the People Analytics course →​

Hi Friends,

Been a busy week here in Toronto.

I spent a morning the other day thinking about how my client is looking at performance. And, in particular, measure performance.

Of course, they use the OKR methodology.

But if you know me, I am somewhat method agnostic, trying to focus on value and concepts above any specific method.

Let's take a look at performance case together:

An employee, a manager, or a sales team posts an extreme, record-breaking result. Leadership immediately attributes that outcome to pure, unadulterated skill.

And we love these extreme examples.

When a rep hits 300% of their quota, or a manager gets a 98% engagement score, corporate systems rush to reward them. We crown them as “top talent.” We promote them to management. We tell everyone else to study their habits.

And then, over the next three quarters, something incredibly frustrating happens.

Their performance drifts right back down to about the same where other people were.

Leadership usually assumes the person lost their drive. Or they blame the market. They assume the employee got lazy.

But more often than not? The organization is just witnessing a fundamental statistical reality.

Grab a coffee. We are going deeper.

The Illusion of Pure Skill When an outcome is driven heavily by random variation, an extreme result is almost always followed by a result closer to the average.

When HR and leadership mistake variance for talent, we end up promoting people for getting lucky. And we penalize people for experiencing bad "weather."

To fix this, we have to revisit Michael Mauboussin’s The Success Equation.

Mauboussin teaches us that every outcome is a combination of two variables: Skill and Luck.

If a game is pure skill (like chess), outcomes are highly repeatable. Regression to the mean is minimal.

If a game is pure luck (like a slot machine), outcomes are completely random. Reversion to the mean is immediate.

Corporate performance lives in the messy middle. It is a noisy hybrid zone where skill and environment constantly interact.

Regression to the mean does NOT imply that every individual eventually becomes average. It simply means that extreme observations are frequently accompanied by temporary luck or measurement error.

When those temporary conditions disappear, the next measurement naturally moves back toward the individual’s true underlying baseline.

So do Praise and Punishment even Matter?

This statistical reality creates one of the most dangerous management problems in business.

In Thinking, Fast and Slow, Daniel Kahneman highlighted how flight instructors evaluated pilot cadets. The instructors noticed a distinct pattern:

When a cadet executed a perfect landing and was praised, their next landing was usually worse. When a cadet executed a terrible landing and was screamed at, their next landing was usually better.

The instructors concluded that punishment works and praise fails.

They were completely wrong. They were just witnessing pure statistical reversion.

An exceptionally bad landing is naturally followed by a better one. An exceptionally good landing is naturally followed by a worse one. The verbal feedback had absolutely nothing to do with it.

Corporate managers fall into this exact same thinking every single day.

They intervene during a dip, or they celebrate a massive spike.

But notice that in both cases, we are sitting on the extremes.

When performance naturally drifts back to the baseline, they mistakenly attribute that mathematical movement to their own brilliant management style.

Diagnosing Real Capability

How do People Analytics teams use mean reversion to actually evaluate talent?

We have to stop looking at static numbers and start treating the speed and direction of performance changes as diagnostic signals. So, thinking in trends.

1. Fast Regression = Pure Luck: If an extreme result disappears instantly in the next quarter, that original spike was driven by temporary environmental factors. A salesperson hitting 300% of quota because of a single, unearned enterprise windfall who then drops to 80% the rest of the year did not demonstrate skill. They demonstrated luck.
   ​
2. Slow Regression = Persistent Skill: If an employee posts an exceptional result, and their subsequent performance declines only slightly while anchoring well above the population average, you are looking at persistent skill. Think of a world-class poker player. They experience short-term variance, but they maintain a winning rate over thousands of hands.
   ​
3. Moving the Mean = Capability Building: If an individual’s performance continually shifts away from their historical baseline in a positive direction, you are seeing a genuine structural change. This is not regression to the mean; this is a shift in the mean itself. This is what actual coaching, role fit, and skill acquisition look like.

The CHRO Playbook:

In public markets, investors do not pay performance fees for “beta”—the general momentum of the market. They only pay for “alpha”—the true, repeatable outperformance after controlling for market exposure.

Human capital management requires the exact same financial discipline.

Before you promote someone, award a massive bonus, or scale a team model, use this playbook:

1. Audit the Underlying Metric. Before you evaluate persistence, look at the noise inside the metric itself. Did a recruiter fill double their requisitions because of superior sourcing skills, or because they were handed low-friction, high-paying roles in a favorable market with an abundance of candidate supply?
   ​
2. Expand the Time Horizon. Never base a promotion or a massive compensation increase on a single quarter of data. Require a minimum of three to four quarters of sustained output to prove the new baseline is real. Bad measurement repeated over time does not become good measurement. It just becomes reliable noise.
   ​
3. Isolate the Tailwinds. Force your managers to defend their top performers. Make them explicitly separate the individual’s pure skill from the environmental tailwinds (territory assignments, legacy accounts, budget surpluses).

One exceptional quarter is just an observation. Four persistent quarters across changing market conditions is a signal.

Talent is not an extreme result. Talent is the part of the extreme result that survives repetition.

K

Whenever you’re ready, there are 2 ways I can help you:

#1

If you’re still looking to get started in People Analytics, I recommend starting with my affordable course:

​Practical People Analytics: Build data-driven HR programs to 10x your professional effectiveness, business impact, and career. This comprehensive course will teach you everything from building an HR dashboard for business results to driving growth through more advanced analytics (i.e., regression). Join your peers today! ​
​

#2

If you are looking for support in your human capital programs, such as engagement, retention, and compensation & benefits, and want to take a more data-driven approach, contact me at Tskhay & Associates for consulting services. Or simply reply to this email!