Talent vs Luck: A Critical Response Part 1

1 Abstract
2 Introduction
3 The Talent vs Luck (TvL) Model
- 3.1 Our python version of the TvL model
- 3.2 Note about fitting power-laws
4 What is being modelled?
- 4.1 95th Percentile vs 5th Percentile
5 Expected Values
- 5.1 God-like talent cannot save you
6 Concluding thoughts
7 Reproduce figures from paper
8 Estimate p_event

prob-per-timestep

Discussed in Section 4.

A Critical Response: Talent vs Luck¶

Abstract¶

The Talent vs Luck: the role of randomness in success and failure (by A. Pluchino. A. E. Biondo, A. Rapisarda) has been written about in MIT's Technology Review and Scientific American (and possibly more). The authors create a simulation and create a model that they call the Talent vs Luck model (TvL). The authors conclude:

We can conclude that, if there is not an exceptional talent behind the enormous success of some people, another factor is probably at work. Our simulation clearly shows that such a factor is just pure luck. -- A. Pluchino. A. E. Biondo, A. Rapisarda, https://arxiv.org/abs/1802.07068

We contend that this conclusion is baked into the structure of their TvL model from the very beginning. The TvL model that they use in this paper has 80 time steps. Everyone starts with the same capital (10 units), and over these 80 timesteps their capital doubles, halves, or stays the same with some probability and some dependence on their underlying talent.

Within their model, we estimate that a person in the 95th percentile of talent has a ~6.1% chance of doubling their money each timestep, a person in the 5th percentile has a 3.4%. This means that in the TvL model the probability difference per timestep between a person in the 95th quantile vs the 5th quantile is ~2.6%!! Starting from this incredibly small difference in probabilities even between outliers essentially guarantees the conclusion before it even begins. Further, even granting a person God-like talent in the TvL model yields an expected value lower than what she started with! In light of these arguments it is clear that talent does not matter much to their model. Conclusions outside of the TvL model are based on a tilted foundation.

Obviously, we have to back up our claims and explain how we derive these values. We do that in the following Jupyter Notebook. Further, we are releasing our code in the hope that others would check, improve our analysis, and perhaps collaborate. Finally, we will limit our response to the first two sections of the TvL paper in this post, and will turn to the rest of the paper in later posts.

Introduction¶

The Talent vs Luck: the role of randomness in success and failure has found quite a response in the wider public's mind and written about in (at least) the following writeups:

Scientific American: The Role of Luck
MIT Technology Review: If you’re so smart, why aren’t you rich? Turns out it’s just chance.

Various quotes:

Their simulations accurately reproduce the wealth distribution in the real world. But the wealthiest individuals are not the most talented (although they must have a certain level of talent). They are the luckiest. And this has significant implications for the way societies can optimize the returns they get for investments in everything from business to science. -- https://www.technologyreview.com/s/610395/if-youre-so-smart-why-arent-you-rich-turns-out-its-just-chance/

and

The team shows this by ranking individuals according to the number of lucky and unlucky events they experience throughout their 40-year careers. “It is evident that the most successful individuals are also the luckiest ones,” they say. “And the less successful individuals are also the unluckiest ones.”

That has significant implications for society. What is the most effective strategy for exploiting the role luck plays in success? -- https://www.technologyreview.com/s/610395/if-youre-so-smart-why-arent-you-rich-turns-out-its-just-chance/ (emphasis added)

Discussions on various commentary websites:

In [1]:

%matplotlib inline
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
import scipy.stats as ss
import powerlaw
sns.set_context('poster', font_scale=1.3)
plt.rcParams['figure.figsize'] = 10, 6

# Reproducibility
# seed = pd.Timestamp('now').strftime("%Y%m%d%H%M%S%f")
seed = '20180312010828882548'
print("Random seed:", seed)
# Call it with anything you'd normally call starting with np.random. 
random_state = np.random.RandomState(np.array([int(x) for x in seed]))

Random seed: 20180312010828882548

In [2]:

# Paper's values
n_people = 1000
n_timesteps = 80
starting_capital = 10.0

talent_mean = 0.6
talent_sd = 0.1

# 
p_lucky = 0.5
# p_event = not given

# Our estimates
p_event_low = 0.11
p_event_estimate = 0.16
p_event_high = 0.21

The Talent vs Luck (TvL) Model¶

A high level description of the TvL model from their paper:

N_people are placed uniformly randomly in a square environment and maintain a fixed position.
N_events are also uniformly randomly placed in the same environment.
The events are classified as either Lucky or Unlucky (with a 50/50 distribution).
The events then randomly walk around the environment.
At discrete timesteps, their motion is frozen and everyone's capital gets updated with the following rules.
1. If the person is not overlapping an event, their capital is unchanged.
2. If the person is overlapping an event, then if the event is unlucky, their capital is halved.
3. If the person is overlapping an event, and the event is luck, then, if a random number [0, 1) is less than that person's talent, then their capital doubles. Otherwise, their capital is unchanged.

Our python version of the TvL model¶

We claim that the following python function, TvL_population_capital_update accurately captures the structure of the TvL model. It is relatively straightforward python code, with the only added note that random_state.rand() returns a uniform random variable in [0, 1).

Every parameter is given in the TvL paper with the exception of the value of p_event, which we address in the Appendix.

In [3]:

def TvL_population_capital_update(row, p_event=p_event_estimate, p_lucky=p_lucky):
    """Update the capital for a given row in a Pandas Dataframe for a single timestep."""

    capital_last_timestep = row.iloc[-1]
    talent = row['talent']

    if random_state.rand() < p_event:
        if random_state.rand() < p_lucky:
            if random_state.rand() < talent:
                return 2.0 * capital_last_timestep
            else:
                return capital_last_timestep
        else:
            return 0.5 * capital_last_timestep
    else:
        return capital_last_timestep


def create_population_df(n_people=n_people,
                         talent_sd=talent_sd,
                         talent_mean=talent_mean,
                         starting_capital=10.0):
    """Creates a population of people in the starting state and returns a Pandas Dataframe."""
    talent_distribution = np.random.randn(n_people) * talent_sd + talent_mean
    df = pd.DataFrame.from_dict(
        data={'talent': talent_distribution}, orient='columns')
    df['starting_capital'] = starting_capital
    return df

def run_model(dataframe=pd.DataFrame(),
              capital_update_function=TvL_population_capital_update,
              n_timesteps=n_timesteps,
              p_event=p_event_estimate,
              p_lucky=p_lucky):
    for x in range(1, n_timesteps + 1):
        dataframe[x] = dataframe.apply(
            capital_update_function, axis='columns', args=(p_event, p_lucky))
    return dataframe

The following code creates a Pandas DataFrame (df). Each row represents a person, who has a fixed talent (drawn from the appropriate distribution), and a starting amount of capital (10). Each subsequent column is another timestep in the simulation, and the first 10 rows (people) of the DataFrame are displayed after the following cell. The column labeled 80 is the final capital amount for the simulation.

In [4]:

df = create_population_df(
    n_people=n_people,
    talent_mean=talent_mean,
    talent_sd=talent_sd,
    starting_capital=starting_capital)
df = run_model(
    df,
    capital_update_function=TvL_population_capital_update,
    p_event=p_event_estimate,
    n_timesteps=n_timesteps,
    p_lucky=p_lucky, )
df.head(10)

Out[4]:

	talent	starting_capital	1	2	3	4	5	6	7	8	...	71	72	73	74	75	76	77	78	79	80
0	0.549187	10.0	10.0	10.0	10.0	20.0	20.0	20.0	20.0	20.0	...	2.5000	2.5000	2.5000	2.5000	2.5000	2.5000	2.5000	2.5000	2.5000	2.5000
1	0.705182	10.0	10.0	10.0	20.0	40.0	40.0	40.0	20.0	40.0	...	10.0000	10.0000	10.0000	20.0000	20.0000	20.0000	20.0000	20.0000	20.0000	20.0000
2	0.468874	10.0	10.0	10.0	10.0	10.0	10.0	10.0	10.0	10.0	...	1.2500	0.6250	0.6250	0.6250	0.6250	0.6250	0.6250	0.6250	0.6250	0.6250
3	0.638264	10.0	5.0	5.0	10.0	10.0	10.0	10.0	10.0	10.0	...	5.0000	5.0000	5.0000	2.5000	2.5000	2.5000	2.5000	2.5000	1.2500	1.2500
4	0.626318	10.0	5.0	5.0	5.0	5.0	5.0	2.5	2.5	2.5	...	5.0000	5.0000	2.5000	2.5000	2.5000	2.5000	2.5000	2.5000	2.5000	2.5000
5	0.498284	10.0	10.0	5.0	5.0	5.0	10.0	10.0	10.0	5.0	...	0.3125	0.3125	0.3125	0.3125	0.6250	0.6250	0.6250	0.6250	1.2500	1.2500
6	0.520444	10.0	10.0	10.0	10.0	10.0	5.0	5.0	5.0	5.0	...	80.0000	80.0000	160.0000	80.0000	80.0000	80.0000	80.0000	80.0000	80.0000	80.0000
7	0.442721	10.0	10.0	10.0	5.0	5.0	2.5	2.5	2.5	2.5	...	2.5000	2.5000	2.5000	2.5000	2.5000	2.5000	2.5000	2.5000	1.2500	1.2500
8	0.529327	10.0	10.0	10.0	10.0	10.0	10.0	10.0	5.0	2.5	...	0.3125	0.3125	0.3125	0.3125	0.3125	0.3125	0.3125	0.3125	0.3125	0.3125
9	0.731404	10.0	10.0	10.0	10.0	10.0	10.0	10.0	10.0	5.0	...	10.0000	10.0000	10.0000	5.0000	10.0000	10.0000	10.0000	20.0000	20.0000	10.0000

10 rows × 82 columns

Note about fitting power-laws¶

The paper attempts to fit a power-law to their distribution. No details are given about the process, nor is there a reported error bar on any of the fit parameters! We will make a general recommendation that fitting power-laws is tricky, and suggest to the general reader to https://github.com/jeffalstott/powerlaw to find a nice python based package. We do not further explore this as a line of criticism.

What is being modelled?¶

We have reproduced the TvL model using straightforward python code. Now we discuss some of the problems with this model. We can now directly compute the relative probabilities and the direct impact that the varying levels of talent have on a final outcome.

The following function, TvL_probabilities, calculates the probability of each possible outcome for each timestep in the TvL model. The extra input is the talent quantile to be calculated. Putting in the 0.5 quantile returns the probability that a person who is exactly average (talent of 0.6) has to double, stay the same, or halve his/her money in the next timestep.

Quantile: 0.5
  p_halve: 0.08
  p_same: 0.872
  p_double: 0.048

This is using the p_event estimate that we calculate in the appendix. Because we have analytic probabilities, we can also calculate the expected value of a person's capital over time -- and as a function of quantile.

In [5]:

def TvL_probabilities(quantile=0.05,
                      loc=talent_mean,
                      scale=talent_sd,
                      p_event=p_event_estimate,
                      p_lucky=p_lucky,
                      verbose=False,
                     ):
    """The probabilities for each outcome for a person at a specific talent quantile."""
    p_halve = p_event * (1.0 - p_lucky)
    # Because the TvL model uses a Normal (which has support over all reals)
    # inside of a probability (which is constrained to 0, 1) we have to truncate at 1.0
    # to keep things valid. This only starts to be a problem at 0.99999th quantile.
    p_double = p_event * p_lucky * np.min(
        [1.0, ss.norm.ppf(quantile, loc=loc, scale=scale)])
    p_same = 1.0 - p_halve - p_double
    if verbose:
        print("Quantile:", quantile)
        print("  p_halve:", p_halve)
        print("  p_same:", p_same)
        print("  p_double:", p_double)
    return (p_halve, p_same, p_double)

In [6]:

quantile = 0.5
TvL_probabilities(quantile=quantile, verbose=True)

Quantile: 0.5
  p_halve: 0.08
  p_same: 0.872
  p_double: 0.048

Out[6]:

(0.08, 0.872, 0.048)

95th Percentile vs 5th Percentile¶

The probability of halving one's capital is constant for all talent levels. There's a difference, however, if the event is lucky. In this case, the talent and luck combination makes it so that the more talented person has a higher probability of taking advantage of the lucky situation.

How big is the advantage? We can calculate it directly now, so let's compare a person in the 95th percentile to a person in the 5th percentile.

In [7]:

quantile_95 = 0.95
quantile_05 = 0.05

TvL_probabilities(quantile_95, verbose=True)
print()
TvL_probabilities(quantile_05, verbose=True);

Quantile: 0.95
  p_halve: 0.08
  p_same: 0.8588411709843883
  p_double: 0.06115882901561178

Quantile: 0.05
  p_halve: 0.08
  p_same: 0.8851588290156118
  p_double: 0.034841170984388214

In [8]:

double_index = 2
TvL_probabilities(quantile_95)[double_index] - TvL_probabilities(quantile_05)[double_index]

Out[8]:

0.026317658031223566

We see something incredible! For a person in the 95th percentile, they have a ~6.1% chance of doubling their money each timestep, a person in the 5th percentile has a 3.4%. Which means that in the TvL model the probability difference per timestep between a person in the 95th quantile vs the 5th quantile is ~2.6%!!

Let's see what this looks like visually.

In [9]:

quantiles = np.linspace(0.001, 0.999, num=300)

y0 = []
y1 = []
y2 = []
for quantile in quantiles:
    a, b, c = TvL_probabilities(quantile=quantile)
    y0.append(a)
    y1.append(b)
    y2.append(c)
    
yz = np.zeros_like(quantiles)
y0 = np.array(y0)
y1 = np.array(y1)
y2 = np.array(y2)
y3 = np.ones_like(quantiles)

fig, ax = plt.subplots(figsize=(6, 6))

ax.fill_between(
    quantiles,
    y0 + y1,
    y3,
    color=sns.color_palette()[0],
    label='Doubles',
    alpha=0.75)

ax.fill_between(
    quantiles,
    y0,
    y0 + y1,
    color='grey',
    label='Stays the same',
    alpha=0.35)

ax.fill_between(
    quantiles,
    yz,
    y0,
    color=sns.color_palette()[3],
    label='Halves',
    alpha=0.75)

ax.hlines(1.0 - TvL_probabilities(quantile_95)[double_index], 0, 1, lw=1.0, linestyles=':')
ax.hlines(1.0 - TvL_probabilities(quantile_05)[double_index], 0, 1, lw=1.0, linestyles=':')
ax.vlines(quantile_05, 0, 1,  lw=1.0, linestyles='-')
ax.vlines(quantile_95, 0, 1,  lw=1.0, linestyles='-')

box = ax.get_position()
ax.set_position([box.x0, box.y0, box.width * 0.8, box.height])
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
ax.set_xlim(0, 1)
ax.set_ylim(0, 1)
ax.set_xlabel("Talent Quantile")
ax.set_ylabel("Probability per Timestep")
fig.tight_layout()

This is an important plot to our argument. Our recommended way to read this plot is to go along the x-axis to find a talent quantile that you are interested in, and then imagine a vertical line from that point. The 5th and 95th percentiles are marked with thin solid lines. Following the 95th percentile line from the x-axis toward the top you see, first the probability that this person's capital halves. You can also see here directly that this probability is independent of talent and is therefore a constant for all talents (it is the flat rectangular reddish region at the bottom). Continuing up, we find that the vast majority of the time the capital stays the same. Finally, at the top we have the section that depends on talent.

It is clear that there is some sensitivity to talent, because there is a longer blue vertical extent the farther to the right. However, the scale of this is incredibly small. Further, this is directly calculated from the inputs to the model. Concluding that talent does not have a large impact based on this model appears to be reading a true statement about the inputs to the TvL model. The model has almost no sensitivity to talent, so concluding that talent does not have (much) of an effect on the outcome is trivially true.

Expected Values¶

Because we have the probabilities of all outcomes for any quantile in the TvL model, we can calulate the expected value of the simulation as a function of quantile and timestep.

In [10]:

def expected_value(quantile=0.5, starting_capital=starting_capital, n_timesteps=n_timesteps):
    low, same, high = TvL_probabilities(quantile)
    return starting_capital * (-low * 0.5 + high * 2.0 + same * 1.0) ** n_timesteps

In [11]:

# Asymptoticly perfectly talented
quantile_100 = 1.0
expected_value(quantile=quantile, starting_capital=starting_capital, n_timesteps=1)

Out[11]:

9.527218584493426

In [12]:

# asymptotically perfectly talented person
quantile_100 = 1.0
expected_value(quantile=quantile, starting_capital=starting_capital, n_timesteps=n_timesteps)

Out[12]:

0.20763011932496273

God-like talent cannot save you¶

If you were impossibly and perfectly talented, having an impossible infinity of talent, starting with 10 units of capital and playing 80 rounds, your final capital is expected to be 0.38, or losing over 9 units from your starting capital. Even playing one round, your expected value is 9.6 or losing 0.4 from your starting 10.

A few people make it out in the positives, so what gives? Well, there's a huge variance in returns (again, partly why the dependence on talent is so weak).

In [13]:

quantiles = np.linspace(0.001, 0.999, num=300)
starting_capital = 10.0

fig, ax = plt.subplots(figsize=(6, 6))
for timestep in [1, 80]:
    evs = np.array([
        expected_value(quantile, starting_capital=10.0, n_timesteps=timestep)
        for quantile in quantiles
    ])

    ax.plot(quantiles, evs, label=str(timestep) + " timesteps", lw=1.5)
ax.set_title("EV after Different timesteps\n")
ax.set_ylim(0, 10)
ax.set_xlim(0, 1)
box = ax.get_position()
ax.set_position([box.x0, box.y0, box.width * 0.8, box.height])
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
ax.set_xlabel("Talent Quantile")
ax.set_ylabel("Capital")
fig.tight_layout()

Concluding thoughts¶

We were going to put a model that we thought was better in this section, but we found that it distracted from the larger point of the analysis above. We are planning to continue this analysis and welcome feedback, criticism, improvements and corrections.

Appendix¶

Reproduce figures from paper¶

We reproduce Figure 3 from the paper with our simulated data below.

In [14]:

fig, axes = plt.subplots(figsize=(8, 8), nrows=2)

hist_results = np.histogram(df[80].values, bins=150)
x = np.array([np.average(i) for i in zip(hist_results[1][:-1], hist_results[1][1:])])
y = np.array(hist_results[0])
sns.distplot(df[80], hist=True, kde=False, ax=axes[0], bins=150)
axes[1].scatter(x, y)

axes[0].set_xlabel("Capital")
axes[0].set_ylabel("Num Individuals")
axes[0].set_yscale('log')
axes[0].yaxis.set_major_formatter(mpl.ticker.FormatStrFormatter('%d'))
axes[1].set_xscale('log')
axes[1].set_yscale('log')
axes[1].set_xlim(3e-1, 9e3)
axes[1].set_ylim(3e-1, 3e3)
axes[1].set_xlabel('Capital')
axes[1].set_ylabel('Num Individuals')
axes[1].xaxis.set_major_formatter(mpl.ticker.FormatStrFormatter('%d'))
axes[1].yaxis.set_major_formatter(mpl.ticker.FormatStrFormatter('%d'))
fig.tight_layout()

We reproduce Figure 4 from the paper with our simulated data below.

In [15]:

fig, axes = plt.subplots(figsize=(8, 8), nrows=2)
axes[0].scatter(df[80], df['talent'], s=10)
axes[0].set_xscale('log', )
axes[0].set_xlim(1e-4, 5e4)
axes[0].set_ylim(0, 1)
axes[0].set_xlabel('Capital')
axes[0].set_ylabel('Talent')
axes[0].xaxis.set_major_formatter(mpl.ticker.FormatStrFormatter('%d'))
axes[1].scatter(df['talent'], df[80], s=10, c=sns.color_palette()[0])
axes[1].vlines(df['talent'], np.zeros_like(df[80]), df[80], color=sns.color_palette()[0], lw=0.5)
axes[1].set_xlabel('Talent')
axes[1].set_ylabel('Capital')
axes[1].set_xlim(0.3, 0.9)
axes[1].yaxis.set_major_formatter(mpl.ticker.FormatStrFormatter('%d'))
fig.tight_layout()

Estimate `p_event`¶

Figure 5 in the TvL paper gives us the clue for how to estimate p_event. It plots the number of lucky events and unlucky events by the population of 1000 people. Remember that once an event happens, it is 50/50 Lucky/Unlucky. Because there are 1000 people, rolling the dice through 80 timesteps, each roll having some unknown p_event of that timesteps having an event, they have a 50/50 shot of being Unlucky. From this, we can put some reasonable bounds on the value of p_event.

The following figure sweeps through a range of possible p_event values, from 0 through 0.5, along with the maximum number of unlucky events in a population of 1000 with 80 timesteps -- realized a number of times to get a rough estimate.

The horizontal line is the maximum number of unlucky events for a single realization in their paper (15 events -- See their Figure 5b). Where this horizontal line intersects the simulated lines gives the likely equivalent p_event values. The vertical lines are my estimate that bound the likely p_event values.

In [16]:

# Figure 5b has 15 unlucky events
fig, ax = plt.subplots(figsize=(6, 4))

p_events = np.linspace(0.0, .50, num=250)

for trial in range(50):
    max_n_unlucky = []
    for p_event in p_events:
        n_events = np.sum(
            random_state.rand(n_people, n_timesteps) < p_event, axis=1)
        n_lucky = np.array(
            [np.sum(random_state.rand(x) < p_lucky) for x in n_events])
        n_unlucky = n_events - n_lucky
        max_n_unlucky.append(np.max(n_unlucky))

    ax.plot(p_events, max_n_unlucky, lw=0.5, c=sns.color_palette()[0], zorder=1)

ax.plot(
    p_events, max_n_unlucky, lw=0.5, c=sns.color_palette()[0], label="Trial", zorder=1, )
ax.hlines(15, 0, .5, lw=2.0, label="Num observed unlucky events", color='red', zorder=2)
ax.vlines(
    p_event_low,
    0,
    40,
    lw=1.75,
    linestyles=":", zorder=3,
    label="Bounds of our estimate")
ax.vlines(
    p_event_estimate,
    0,
    40,
    lw=3.0,
    linestyles="--", zorder=3,
    label="Our best estimate")
ax.vlines(
    p_event_high,
    0,
    40,
    lw=1.75,
    linestyles=":", zorder=3,)

ax.set_ylim(0, 40)
ax.set_ylabel("N events")
ax.set_xlabel("Probability of Event (p_event)")
box = ax.get_position()
ax.set_position([box.x0, box.y0, box.width * 0.8, box.height])
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
fig.tight_layout()

There are comments.

links

social