Potatoes: A Critical Review

By Pablo Villalobos, Jaime Sevilla @ 2022-05-10T15:27 (+120)

This is a linkpost to https://docs.google.com/document/d/1nd29cmnmHQ9cFEwrjNjE_bXEEr5aESYvyb0kFcoTfPY

Executive summary

• Nunn and Qian study the effect of the introduction of potatoes in the Old World on population growth between 1700 and 1900.
• We think the paper credibly establishes that between one-sixth and one-quarter of the growth is a consequence of the introduction of potatoes.
• The main reason for doubt is the possibility of spurious correlation due to spatiotemporal autocorrelation and the fact that potatoes were mainly grown in Europe, which at the time was experiencing growth due to unrelated factors.
• After performing several tests to account for these concerns, we conclude they are not strong enough to reject the conclusion of the paper.

Introduction

During the 18th and 19th centuries we saw rapid urbanisation and a drastic increase of population throughout the world. Some attribute this to industrialization; Nunn and Qian argue that we owe that to potatoes. 

They are not coy about it either. “According to our most conservative estimates, the introduction of the potato accounts for approximately one-quarter of the growth in Old World population and urbanisation between 1700 and 1900”. 

They back their hypothesis with data from several natural experiments, studying variability of city populations and adult heights between and within countries, and relate it to the timing when potatoes were introduced in different places.

In this article we will summarise, replicate and critically review Nunn and Qian’s paper. We follow the methods established in (Sevilla, 2021) to study whether their findings are robust and whether they successfully establish a causal relation.

Paper Summary

The study purports to evaluate the causal impact of the introduction of potatoes in the Old World in population growth and urbanisation rates.

The theory is that potatoes are more nutritious and provide 3x more calories per acre, so when they started being cultivated in the Old World they produced a positive shock in agricultural productivity, enabling higher populations and wealth per capita.

A previous study by (Mokyr, 1981) estimated the causal effect of potatoes on population growth in Irish counties in 1845^[1], finding an effect size of 0.7. This implies counties with high potato cultivation grew an extra 0.15% that year, compared to counties with low cultivation^[2].  In comparison, Nunn and Qian’s study uses data of the whole Old World, from 1300 to 1900, instead of just Ireland in a single time slice, and looks at urbanisation rates in addition to population.

To estimate the causal impact of potatoes, the authors exploit two sources of variation^[3] on each country’s ability to grow potatoes: the time of introduction and the suitability of the land for potato cultivation. In their baseline analysis they find that around a quarter of the growth in population and urbanisation rates in that time period can be attributed to potatoes.

The main pitfall of this strategy is that it relies on there being no other shocks during that time which are correlated with suitability for cultivation. Unfortunately, Europe is much more suitable for potato cultivation than other Old World regions, and the potato was introduced at the same time when Europe was diverging from other countries due to several unrelated factors.

To check that the effect is indeed causal, the authors use several strategies:

• Adding several controls for alternative drivers of population and economic growth.
• Comparing only countries within the same continent
• Comparing cities within the same country
• Comparing the heights of soldiers in France, using their town of birth to estimate the importance of potatoes in their childhood diet.

In all of those cases, they find a significant effect of potatoes on the outcome. For the within-continent and within-country analysis, the effect size is in the same order of magnitude as in the baseline regression.^[4]

They also perform some analyses to determine which cutoff date to use for the introduction of potatoes:

• A regression interacting potato suitability with the time period, which shows an increasing effect for periods after 1750, consistent with that being the right cutoff.
• A series of regressions with a ‘rolling window’ of 400 years, taking the first 200 hundred years as the pre-adoption period and the last 200 years as the post-adoption period. Again the earlier windows show no effect of potatoes, while the latter ones (1600-1900) do show an effect.

I think the authors convincingly show that potatoes had a robust positive effect^[5]. Moreover, the effect size is within what would be expected from theory and consistent with previous findings^[6].

Conclusion

We tentatively conclude the main claim in the paper is broadly correct. That is, around a quarter of Old World population growth from 1700 to 1900 was caused by the introduction of potatoes.

While the significance and effect size is diminished when taking into account multiple hypotheses and within-continent variation only, this is not enough to make the effect non-significant or much smaller. Spatial autocorrelation does not seem to be an issue due to the characteristics of the analysis, as found by (Kelly, 2020)^[7].

This analysis could be improved by a) finding other sources to estimate expected effect size, and b) verifying the correlated standard errors computed in (Kelly, 2020)^[8].

Acknowledgements

This review has been commissioned by the Forethought Foundation.

Bibliography

Gelman, A. (2014). Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. https://journals.sagepub.com/doi/10.1177/1745691614551642

Kelly, M. (2020). Understanding Persistence. https://economics.yale.edu/sites/default/files/understanding_persistence_ada-ns.pdf

Mokyr, J. (1981). Irish History with the Potato https://journals.sagepub.com/doi/abs/10.1177/033248938100800102

Nunn, N., Qian, N. (2011).  The Potato's Contribution to Population and Urbanization: Evidence from a Historical Experiment https://scholar.harvard.edu/nunn/publications/potatos-contribution-population-and-urbanization-evidence-historical-experiment

Sevilla, J. (2021). Persistence: A Critical Review https://docs.google.com/document/d/14ULAaTofWiQbTCP1ekuaenQJ6saXEzjgiKMznIBrXvQ

Appendix: Table summary

Appendix: Reproduction Details and Calculations

Our procedure is similar to the one in (Sevilla, 2021): In addition to reproducing the main results, we test for spatial autocorrelation^[12], multiple hypotheses testing^[13], and the possibility that the analysis is underpowered^[14].

We reproduce the regressions on the first three columns of tables IV and VII. In the summary table we report the result of column (3) of table IV. The estimate of the effect size is 0.032, which when standardised is 0.051.

To give an intuitive understanding, this means that increasing the amount of suitable land for potato cultivation by 1 percent increases the population by 0.032 percent. To estimate the total impact of population in the whole 1700-1900 period, we compute the total population growth and the growth under the counterfactual where potatoes are not present (the potato variable is 0).

Total population growth in that period was 247%, which in log form is 0.9. The counterfactual growth is 195%, which in log form is 0.67. So the difference in logs is 0.23, or approximately 0.9/4, which is the total impact of potatoes.

The estimate of the effect size on column (3) of table VII, which takes into account only within-continent effects, is 0.020. This similarly translates to a total impact of 1/6 of the total growth.

We compute Moran’s statistic following the procedure in (Kelly, 2020): taking the average of its value for each time period, calculated with a spatial kernel of 1 for the 5 closest points, and 0 otherwise^[15]. We did not compute the correlated standard errors from (Kelly, 2020), instead we take his word that they are smaller than the clustered errors^[16].

We use design analysis as in Gelman (2014) to compute the power of the study and its type S and M error rates. To find an estimate of the true effect size, we look at the effect found by (Mokyr, 1981). He found an increase of 0.15% in the annual population growth rate, which over 200 years constitutes a total growth of 36%, for a logarithmic change of about 0.3. We also use a simple theoretical model which predicts a similar effect^[6].

To account for multiple hypotheses, we make the same corrections as Sevilla (2021)^[17]. First we compute the critical number of hypotheses for the given significance threshold, which in this case is 10, and then we compare it with the actual number of hypotheses tested. In this case the authors test 2 hypotheses (effect of potatoes on population and urbanisation rate), which results in a small increase in the p-value of the study. Note that in this case the result of the analysis was positive for both hypotheses, which gives us further confidence that there are no statistical artefacts

All the code used for the replication can be found here.

1. ^{^}

   Using a 2SLS analysis.

2. ^{^}

   The regression coefficient is 0.7, dependent variable is yearly pop. growth rate and independent variable is percentage of total land devoted to potato cultivation. The counties with highest potato cultivation had 29% of the land devoted to potatoes, whereas for the lowest ones it was 7%. 0.007 * (0.29 - 0.07) = 0.00154 ≈ 0.15%. Over 200 years, this corresponds to an extra growth of 36%, 0.3 in log scale.

3. ^{^}

   They use a variant of the differences-in-differences (DD) method, where the difference in population growth between countries with varying potato suitability is compared before and after the introduction of potatoes. What makes this analysis slightly different from standard DD is that instead of collapsing all data into ‘pre’ and ‘post’ adoption periods, the study keeps the temporal structure of the data.

4. ^{^}

   The effect size is 0.02 for within-continent comparisons and between 0.029 and 0.05 for within country comparisons.

5. ^{^}

   Note: the empirical effects found in this study could be due to the introduction of the potato having a permanent effect on the growth rates of the countries, but it could also be due to it having a one-time effect on the long term population level. In this last case, since potato adoption was gradual, the level change would be spread over centuries. This means we can’t distinguish these two hypotheses with the existing data.

6. ^{^}

   Based on the ~0.3 effect over 200 years obtained by (Mokyr, 1981) and on theoretical calculations: if ⅕ of the land is suitable for potatoes, their caloric yield is 3x that of Old World crops, and all the extra yield is absorbed by population growth, then we should expect a log increase of log(⅘ + 3*⅕) ≈ 0.3, broadly consistent with the effect found by the paper.

7. ^{^}

   See the appendix for the details on analysing multiple hypothesis testing, spatial autocorrelation, and the power of the analysis.

8. ^{^}

   Note: Kelly expressed in correspondence that he no longer trusts the approach in his paper. For now we stand by our conclusion, and await his forthcoming publication with improved methodology.

9. ^{^}

   They test for population growth and urbanisation rate.

10. ^{^}

    Even if the expected effect size was 10x lower, the type S error rate would still be ~13%, and the effect would still explain around 3% of the total growth over that period.

11. ^{^}

    Kelly (2020) finds a Z value of 6.44 for the regression in column (1) of table IV, which we were able to replicate. The same methodology yields this value for the baseline regression.

12. ^{^}

    The phenomenon where spatially close places tend to be more similar to each other than would be expected if they were truly independent. The presence of spatial autocorrelation distorts estimates of standard errors and inflates t-statistics.

13. ^{^}

    If the authors test the effect of potatoes on several different outcomes (multiple hypotheses), this increases the probability of spurious results.

14. ^{^}

    The power is the probability of correctly rejecting the null hypotheses. If the true effect size is real but too small compared to the standard error, then the analysis will be underpowered and won’t be able to detect an effect. This is mitigated by reducing the standard error, which usually means increasing the sample size.

15. ^{^}

    Other kernels, like inverse distance and inverse exponential distance, found lower levels of spatial autocorrelation.

16. ^{^}

    Kelly finds an effect size of 4.11 with a clustered standard error of 1.05, which is reduced to 0.44 after taking spatial correlation into account. These numbers are very different from the ones in the original paper and we are not sure how they are being computed. This is why we chose not to use them. In any case if Kelly is right, using his correction would only strengthen the conclusion.

17. ^{^}

    That is, we adjust the p-value of the main regression using the Šidák correction p_adj = 1-(1-p)ⁿ, where n is the number of hypotheses tested. In addition, we have to adjust the standard error σ so that it still represents a confidence interval of confidence level ɑ ≃ erf(1/√2). To do this, we set σ_adj = σ√2erf^-1(ɑ^1/n).

Ramiro @ 2022-05-11T14:30 (+10)

Take that, people who say "muzak and potatoes" as if it was a bad thing.

Jaime Sevilla @ 2022-05-11T19:09 (+8)

Michael_Wiebe @ 2022-05-27T17:06 (+5)

Did you run additional robustness checks? I like to see a multiverse analysis, aka specification curve, (see here). This involves running all combinations of control variables, since the order in which the controls are added matters, and the authors could have selected only the significant ones. (See also.)

Thomas Kwa @ 2022-05-27T18:21 (+3)

(epistemic status: possibly dumb question by someone learning causal inference)
shouldn't you test only combinations of controls which are good conditioning strategies for a plausible causal DAG?

Charles He @ 2022-05-27T18:56 (+5)

Showing these various specifications, with, as you point out, wonky or strange non-final specifications is sort of a "sanity check". 

(Or alternatively, the authors are "selling" their work by trying to show a consistent story by incremental showing increasingly complex specifications, carefully embedded in a performative narrative).

 

So say, if you're measuring the effect of "education on wages", you might start with your raw, nearly simple regression of wage on years of schooling. Then, as the beginning step of many, you might proceed by showing a specification that includes controls for gender or parental SES (even though this isn't the actual favored specification). 

This gives a sanity check. 

For example, you expect that the effect of education to go down once you add parental social economic status. So if it goes up, something is probably very wrong. At the same time, the parameter values for controls have content in themselves which is useful.

In the end, most empirical economists are skeptical of any single specification, or entire paper really (in some cases, comically, ruthlessly skeptical, exceeding the most negative comment made by this account), so showing these specifications are good in this culture.

 

I like to see a multiverse analysis, aka specification curve, (see here). This involves running all combinations of control variables, since the order in which the controls are added matters, and the authors could have selected only the significant ones.

The parent comment here is recommending all "combinations of control variables". That's different than the normal aesthetic in most empirical econ papers. That might not be ideal in some situations. I haven't heard of this before^[1]. Also, no one I know has ever used the term "multiverse analysis", that really gives the data, specification and model too much credit.

Also, in OLS and most variants, order doesn't matter. I think maybe he's talking about something else, but it's not clear from the paper. I find this confusing and not useful to most readers.

1. ^{^}

   However, my PhD is from Nipissing Technical College of Agriculture, so if some Harvard dude comes in, they are probably right.

Michael_Wiebe @ 2022-05-27T23:57 (+2)

Also, in OLS and most variants, order doesn't matter

See the Gelbach paper linked above.

Michael_Wiebe @ 2022-05-27T23:49 (+2)

Yes, but there are often many plausible sets of control variables that (hopefully) get you conditional independence. I find it easier to plot everything, with the understanding that some specifications are better than others.