Evan Soltas
May 30, 2013

On Wang-Kimball

There's something economists do that the outside world might find silly but really is not: replicate each other's results.

This happens accidentally when two economists come to the same finding independently. One famous example of this is Christophe Chamley and Kenneth Judd, who both found that it is impossible for capital taxation to be welfare-enhancing after redistribution.

It also happens intentionally when economists double-check important findings. This sort of replication is very rare. In both cases, replication gives us marginally more confidence in a finding, particularly when it takes a different approach. If replication fails, then the finding is put in jeopardy.

Carmen Reinhart and Kenneth Rogoff's study, "Growth in a Time of Debt," took months to replicate. The first effort was by Thomas Herndon, who disputed Reinhart and Rogoff's conclusions rather convincingly, particularly the claim of a 90-percent threshold.

Then came Andrajit Dube, who disputed the causal mechanism of high debt to slow growth, rather than slow growth to high debt. The toolbox of econometrics to prove causation is quite limited. My interpretation of the Dube findings was that the causal mechanism is probably stronger in the reverse of the direction suggested by Reinhart and Rogoff.

Then came Yichuan Wang and Miles Kimball, who I should note are both friends of this blog. Wang and Kimball suggest that there is no statistically significant evidence for the causal mechanism running in the debt-to-growth direction. Their findings are slightly stronger in this respect.

Their finding, so far as I can tell from the information they provide at present, comes from the same data as Reinhart-Rogoff and Dube. Their only change from Dube is to shift the period of observation of growth. Dube's method was to estimate the mean level of real GDP growth, looking at the next and preceding three years, based upon the level of debt. Wang and Kimball use the periods 5-to-10 years out and prior.

These intervals are disjoint. That means the Wang-Kimball and Dube findings are not exactly replications of each other. However, the difference between them is technical, and both are seeking to illustrate the same statement: "There is no significant evidence that increases in debt have a causal effect on the long-run rate of real GDP growth." To that extent, they are replications of the same finding. (Kimball has an interesting follow-up post which discusses why this claim might be true, even when there are strong theoretical arguments that it should not be.)

I have a few other comments on how we should read Wang-Kimball in relation to Dube.

First, it's not clear what the optimal interval of observation is. How long, in other words, is the long run? The point here is that it would be possible to choose some absurdly far-off date -- suppose, an interval of 50 to 100 years out -- in which we can be almost certain we would get results that look more like Wang-Kimball in terms of strength, and less like Dube. Economies change significantly over the long run, even if there are debt overhangs, so there should be a downward bias in the likelihood of statistically-significant findings as the interval runs outwards. This is why I'm not particularly convinced by Kimball's claim that his finding is "stronger" than Dube.

Second, what's wrong with Year 4? Both Dube and Wang-Kimball exclude it. If I was to redo both of their analyses, I would effectively redraw Wang-Kimball's left graph differently. I would break the data into buckets: say, 1-3 years out, 3-6 out, 6-9 out, and so on, out to perhaps 20 years. Then I would re-plot all of their regression lines onto the same graph. Then I would change the bucket size: 1-5 years, 6-10 years, etc. and put that into several new graphs. Then we can evaluate the more robust statistical claim: "There exists, over no reasonably long-run interval, statistically significant evidence of a debt-to-growth causal mechanism." (If someone will email me the data, I will happily do this work.)

Third, I have some pedantic concerns about data quality. These are not comments against Wang-Kimball or Dube, exactly, but I think they should consider them more seriously than Reinhart-Rogoff because they are both dividing the data more granularly. The R-R sample comes from 1800 to present. First, it would be useful to see a histogram of how their data is distributed chronologically and in terms of countries. Second, it would be useful to see a histogram that takes the subset of high-debt episodes. I am worried that our conclusions are less robust than we think they are, and that they are driven by only a few debt events that happened at nearly the same time, but in different nations. Paul Krugman's recent replication, which showed that Japan data was doing a lot of the work, set off these concerns.