Coming towards the end of my Coursera course on model thinking, I was introduced to the concepts of random walks and skill-luck model. Especially the skill-luck model appealed to me as it can be used to explain success on a personal and organizational level and predict, if success is due to skill or luck and if success is lasting. The following is based on the Coursera course Model Thinking by Scott E. Page and the course readings, mainly on How to Entangle Skill and Luck from Michael Mauboussin at Legg Mason Capital management from July 15th 2010. I have split my text into two parts to make the individual posts shorter.
The skill-luck model
The skill-luck model states that an outcome is based, with varying degree, on both skill and luck. Here skill denotes the knowledge and ability of an individual or organization to steer the future events in to the desired direction. Luck, on the other hand, incorporates the noise, random effects, that affect the outcome and cannot be controlled by the individuals or the organization.
To make the definitions more concrete we could say that the one hundred meter dash is mostly skill based, as the winner is based on the speed of the individual, and the speed is mostly dependent on strength, running technique, reaction time and other things that are either inherited in the DNA or a result of hard training and therefore dependent on the individual.
An example of a pure luck event is lottery. One cannot practice or have since birth any advantage that could help in choosing a winning number (clairvoyance excluded). The winning number is solely based on chance, or luck, as people often say.
A general formulation for the skill-luck model, presented also at the Coursera course by Mr. Page, is
Outcome = A*Luck + (1-A)*Skill
where A is constant between zero and one. The formula shows that, depending on the event, luck or skill may dominate, or they may be of roughly equal importance.
Why use the skill-luck model?
The skill-luck model has relevance on both personal and organizational level. In the society, most people arguably want to be successful, and so do organizations, composed of those very people. Some end up being successful, some not. Sometimes success is lasting; sometimes it barely greets us before already taking part.
Success is often looked up to and merited, while failure may be frowned upon or even punishable. Yet, if success or failure is based more on luck than skill, we should be careful in giving praise and delivering punishment. If success and failure take place randomly, it would imply that the accompanied fortunes and fame are provided without, or due to little, personal merit.
Identifying whether something is luck or skill dominated serves at least five purposes.
- Ability to assess outcomes.
- Anticipating outcomes and regression to the mean.
- Giving guidance where we are most likely to be misled.
- Fair evaluation and appraisal, which also promote learning.
- More optimal resource distribution.
With the skill-luck model we can assess, whether something is based on skill or luck. A good rule of thumb is, that you cannot lose on purpose in an activity that is dominated by luck, or at least losing on purpose is very difficult. An indicator of a skill dominated activity are lasting streaks. If a person consistently beats others and provides well above average performance consistently, providing that the observed sample is statistically representative, the event is likely to be skill dominated. Not all skilled people have streaks of success, but most streaks are done by skillful people, since the probability of streaks by chance is so low. A good example of a long-lasting streak is the already mentioned hundred meter dash, which has been dominated by Usain Bolt since 2008.
Another method for evaluating the role of skill and luck is to establish the base rate and compare it against actual observations. For example, when examining the role of skill in winning sport matches, the base rate would be the equivalent of a coin toss experiment, when the events consist of teams or individuals going one-on-one. If the group’s variance in the observed performance (for example percentage of games won) is much larger than the variance of the coin toss alternative (winning a game has a 50/50 chance), some contestants do consistently better than odds would have it, and so skill plays a relatively large role in the event. Yet, streaks and clusters can also be random. For example, if we throw a coin long enough, we would expect to get 20 heads in a row. The difference between luck and skill is also in the frequency of such streaks, which are achieved more often by skilled individuals, although they might be helped by some luck to keep the streak going on.
Here, in an MS Excel file, is an example of a 6-game league between four teams (the method is described by Mauboussin in his paper). Each team plays 2 games against every other team. 1 in the table in the Excel file means that a team won the specific game, 0 indicates the losing team. As we can see from the calculations, skill would seem to play a large role in this specific game, where team 1 has been unbeatable. To ascertain such results in real life, large enough samples (also temporally) have to be ensured, which is not the case with this example, but it illustrates the point.
Anticipating outcomes and regression to the mean
After having identified the role of luck and skill in a specific event, we are more ready to anticipate outcomes of events. If an event is skill dominated, the most skillful are going to man the top positions. If the event is dominated by luck, anyone could win and predicting the top performers, not to mention the winner, becomes difficult. In a skill dominant event it is reasonable to expect that the most skillful will be the best performers, while the consistency of top performance is an indicator of a skill dominant event. This notion is not circular logic, rather it states that first we have to observe, if an event is skill dominant, as we have done in the previous paragraph. After enough observations have been done and the role of skill determined, we can predict, that in skill dominant events current success should be followed by future success, even if we have only few observations from individual performance.
The dominance of luck or skill has interesting consequences for anticipating outcomes. If an event is dominated by skill, it is quite certain that the few best will always be at the top, since the effect of random events, or luck, is so small. However, it is not certain that the absolutely best will win. Since the best can be very close to another in their skillfulness, the outcome is dominated by luck, by the small variations during the event and their effect on personal performance. Therefore, the top three or top five may consist of the same individuals from one event to another, yet the winner may never be the one with the most skill. This is called the paradox of skill. A further implication is that it is not always possible to find the best or most skillful, only the winner of a single event. If we cannot remove the effect of luck completely, small random effects make it difficult to distinguish the differences between individuals, when they are nearly equally skilled. The table below illustrates the paradox of skill. Player 1 is the most skilled, but due to small differences in skill, the least skilled Player 3 wins the game.
If an event is dominated by luck, it expresses the statistical property known as mean reversion: after an extreme event the next event is likely to be closer to the mean. This implies that extremely high or low values are often followed by mediocre results, which are in general more likely to take place. This also implies that if well above average performance does not regress to the mean or does so very slowly, an event is skill dominant.
To understand how regression to the mean can take place within groups while the variance of results in the whole population (consisting of those groups) stays the same, I encourage you to read pages 16 – 19 in Mr. Mauboussin’s paper.
Guidance to prevent us from being mislead
Not seldom are we inclined to draw conclusions from too small samples. For example, if an event is dominated by luck, we might nevertheless attribute the success of individuals to higher than average skill, although the outcome was dominated by luck. Therefore it is important to distinguish between skill and luck dominant events and also ascertain, what kind of tests are required to reach a solid conclusion.
Fair appraisal and evaluation and promoting learning
The roles that skill and luck play determine fair assessment and appraisal. If we are praised or punished for events that occur due to luck, the consequences are unlikely to be seen fair, eventually even morally wrong. An interesting point is that in some cases people are conditioned to being rewarded for giving bad feedback and going unrewarded for giving good feedback. This leads to restraining from praising a good performance, be it due to skills or luck, and calling errors in to attention, be they due to skill or luck. See Daniel Kahneman’s observation in Wikipedia for more information.
Mauboussin states in his paper, quite right in my opinion, that when luck is dominating an event, we should praise people for sticking to a prescribed process and becoming more proficient in the process and its execution. Only this way do we learn, by deliberately sticking to a process, repeating it and honing it. That is a time consuming process and not always pleasant or easy. However, this repetition improves our skills and reduces the role of luck in the considered event, making it more skill based. Therefore we should, as the cliché often goes, praise and give prize for hard work and effort, not necessarily for the good outcome.
Optimal resource allocation
If we know an event to be dominated by skill, we can improve the outcome by allocating more to the disposal of the most skillful ones, assuming the they still have excess capacity to use those resources. For example, two factories produce similar goods, but the other one has a more advanced technology that leads to lower production costs. However, this factory does not have enough capital to invest into more production capacity that is needed in any case to produce all the demanded goods. We might reduce the funding (bank loans for investments into new equipment) to the other factory and use it to increase the capacity in the more efficient one, thus reducing total average production costs while increasing capacity. If both factories would be efficient one day and completely inefficient the other, production thus being a luck dominated effect, such re-allocation of capital would not make sense.
Like in the case of randomly efficient factories, we should not take a bank loan and ask the last week’s lottery winner to play with the whole sum. As lottery is a pure game of luck, investing in a specific individual does not increase the chance of success.
Edit 10.9.2016: Corrected typos.