deaton, 2013 on weights and coding errors
TRANSCRIPT
-
7/24/2019 Deaton, 2013 on Weights and Coding Errors
1/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and
EconbrowserAnalysis of current economic conditions and policy
On weights and coding errors: odd coincidence ordress rehearsal?
Today Econbrowser is pleased to host this guest contribution from Professor Angus Deatonof Princeton
University describing some of his experience with political attacks on academic research.
On weights and coding errors: odd coincidence or dress rehearsal?
by Angus Deaton
The recent criticism of Carmen Reinhart and Ken Rogoffs work on debt and growth has resembled nothing
so much as a public pillorying. According to their critics, a team at the University of Massachusetts, their
results are vitiated by a coding error and by their choice of weights. According to Reinhart and Rogoff, the
critics results are identical to their own preferred results, but which were ignored by their critics. Yet none of
this has deflected a firestorm of public criticism and the public dismissal of Reinhart and Rogoffs work.
I do not comment on the substance of this debate here, leaving it in Reinhart and Rogoffs capable hands.
Instead, I note an earlier and less-known incident in which my own (joint) work came under criticism by
economists at UMass one of whom, Michael Ash was involved in both cases and where there are a number
of almost uncanny similarities to the more important current debate.
The substantive question
In 2003, Darren Lubotsky and I published a paper in the journal Social Science and Medicine. Our topic was
whether income inequality is a health risk to people who live in particularly unequal cities or states. The idea
is that income inequality is like a toxic pollution, harming everyone who lives with it. My own view, then and
now, is that there is no such effect, though I also believe that the extreme income inequality that we see in
the US today is a threat to public health, but through quite different (essentially political) mechanisms.
Our analysis showed that the correlation between higher mortality and income inequality arises from a
failure to control for the racial composition of the population in each city or state. In cities (states) with a
http://www.princeton.edu/~deaton/http://econbrowser.com/http://www.princeton.edu/~deaton/http://econbrowser.com/ -
7/24/2019 Deaton, 2013 on Weights and Coding Errors
2/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 2
larger fraction of blacks, the difference between the average incomes of blacks and whites is larger, perhaps
because employers do not regard blacks and whites as fully substitutable in production. This drives a link
from fraction black to income inequality. But blacks also have worse health than whites in part because of
an apartheid healthcare system that treats blacks less well than whites so that the fraction black is also
linked to overall mortality. Those two links induce a strong positive correlation between income inequality
and mortality. That this correlation is spurious is documented by its vanishing when we control for fraction
black, by the fact that mortality rates of blacks and whites separately are uncorrelated with income
inequality, and by the fact that income inequality and mortality are uncorrelated across space in other
settings where race is not a salient factor.
The topic has acquired some political baggage. The inequality as pollution story is often favored on the left,
particularly in Britain, and those who argue against it have sometimes been accused of doing so on political
grounds. Of course, this is but a pale shadow of the political importance of the debt questions in Reinhart
and Rogoffs work.
A challenge
In December 2005, I received a letter from Michael Ash and Dean Robinson at UMass asking questions about
the data in our paper and saying that they could not replicate our results. All of the data that we had used
were (and are) publicly available on government websites, but they require processing and organizing to be
useful. Such replication queries from other scholars are routine, and while we believed that our procedures
were clearly enough specified in the paper to permit replication, we were happy to help guide others who are
less familiar with this kind of calculation. Darren Lubotsky, who had done the original data assembly,
corresponded with Professor Ash over a period of time, and provided explanations, code, and data to allow
him to replicate what we had done. Correspondence then stopped, and we assumed that the matter had
been resolved.
We were not so fortunate. Almost three years later, Ash wrote to us to say that Social Science and Medicine
was going to publish their paper criticizing our work, as was later confirmed by the editor. Different journals
may have different policies, but most journals seek the opinion of the authors being criticized prior to
making the publication decision. Of course, authors cannot be given a veto, but they will often be able to
persuade an editor that the comment is worthless as was the case here and thus prevent unnecessary
public controversy.
Ash and Robinsons note claims that Lubotsky and I had made a coding error in specifying the weights in our
regressions, and that without the error, inequality retains its significant positive effect on mortality. There
was no coding error, but we had chosen weights that downplayed the larger cities and states compared with
the weights that Ash and Robinson had chosen. And it turns out that except for one case (in one
specification in one data period and with only one of their alternative weighting schemes) all of our results
are unaffected by the change in weights. In spite of this, Ash and Robinson make the extraordinarily
misleading statement in the abstract that correcting the error changes the basic results of the paper with
-
7/24/2019 Deaton, 2013 on Weights and Coding Errors
3/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 3
respect to inequality and mortality in a relevant and substantive way, Ash and Robinsons preferred
specification has exactly the same result as our paper, that income inequality is not a health hazard.
Our response, which was published together with Ash and Robinsons note, makes all of this clear. To our
knowledge, there has been no subsequent controversy.
General implications
It is hard to imagine any applied paper that would be immune to this sort of attack. Weighting can always be
argued over: in standard regression analysis, it is not supposed to matter, and when it does, it is usually
unclear what is being estimated. So if you want to debunk a paper, working through it equation by equation,
trying out a range of weights, you will eventually find something that changes. You can then cry coding
error and hope that the rhetoric shifts the burden of proof back to the original authors.
In our case, as in Reinhart and Rogoff, neither the coding error (in our case there was none) nor the choice of
weights has any effect on the main results. In our case, Ash and Robinson simply ignored the results that did
not support their charges, and claimed that their results were different from ours in a relevant and
substantive way. With Reinhart and Rogoff, they referred only to an early paper, ignoring updated results.
But the effect is the same, to magnify a tiny or non-existent problem and claim that it threatens the whole
enterprise whereas, in fact, nothing of the sort is true.
There is also the question of publication. In our case, I believe that Social Science and Medicine should have
shown us the paper prior to the publication decision, but they did allow us to publish a response alongside
the critique. In the Reinhart and Rogoff case, Ash et al did not submit their paper to a journal where theremight have been a chance of an appropriate professional response, but sent it directly to the world press,
copying it to Reinhart and Rogoff on the same day.
Such smear methods appear to work, and provide a handy template for others on how to disguise political
attacks as legitimate scientific commentary. While it is naive to think that science can ever be insulated from
politics, if these methods of attack are widely replicated, and if journals and newspapers are prepared to abet
them, it will make it much more difficult for serious policy-relevant researchers to do their job. Scholars will
also be much less willing to share data than is currently the case; doing so allows anyone who is
unscrupulous enough to turn your cooperation against you.
This post was written by Angus Deaton
Tweet 0 Share 20Like
http://econbrowser.com/http://econbrowser.com/http://econbrowser.com/http://twitter.com/search?q=http%3A%2F%2Feconbrowser.com%2Farchives%2F2013%2F10%2Fon_weights_andhttps://twitter.com/intent/tweet?original_referer=http%3A%2F%2Feconbrowser.com%2Farchives%2F2013%2F10%2Fon_weights_and&related=AddToAny%2Cmicropat&text=On%20weights%20and%20coding%20errors%3A%20odd%20coincidence%20or%20dress%20rehearsal%3F&tw_p=tweetbutton&url=http%3A%2F%2Feconbrowser.com%2Farchives%2F2013%2F10%2Fon_weights_and -
7/24/2019 Deaton, 2013 on Weights and Coding Errors
4/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 4
This entry was posted on October 9, 2013 [http://econbrowser.com/archives/2013/10/on_weights_and] by
James_Hamilton.
48 thoughts on On weights and coding errors: odd coincidence or dress rehearsal?
Wow. Thanks for publicizing this story. Its a shame that UMass Econ feels itself exempt from common
courtesy. If they try this trick more often, theyre likely to find that turnabout is fair play.
SF
October 9, 2013 at 1:32 pm
There are three ideologies. You are a classical liberal. Ash et alare egalitarians. The notion of credibility andreasoned discussion are themselves liberal concepts. Thus, if you are a liberal, you will tend to believe that
your actions make rational sense and that your arguments should be judged on their own merits. There is a
certain naivet and optimism in all this. You have the confidence that either i) your reasoning is sound or ii)
that you can personally survive if your reasoning is wrong. Thats a pretty high level of confidence.
Egalitarians by contrast operate under the assumption of weakness. Thus, they are always under threat and
over-powered by the intelligent and well-educated, from their point of view. Therefore, there is an attempt to
avoid the engaging in good faith dialoguebecause it involves granting of property rights. I recall Nixon
saying something to the effect that, You cant trust those North Vietnamese. They lie and never keep their
word. (Alas, I am old enough to remember this.) I always found this curious. Why wouldnt the North
Vietnamese keep their word?
Because granting ones word is equivalent to signing a contract: it acknowledges property rights. If you agree
to something, you have granted the counterparty property rights. If property is theft, then credibility is also
theft.
So the presumed means of argumentation is not symmetrical. You are assuming that the argument has its
own logic, to be argued in good faith on its own terms. Egalitarians assume that they cant win in open
combat, that they are the underdogs to your overlord. Therefore, they have to play the cards they have,
which includes misinformation and propaganda. Really, what is egalitarianism? It is taking away things that
Steven Kopits
October 9, 2013 at 2:34 pm
http://econbrowser.com/archives/author/james_hamiltonhttp://econbrowser.com/archives/2013/10/on_weights_and -
7/24/2019 Deaton, 2013 on Weights and Coding Errors
5/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 5
other people own through political force. What did they try to do to R&R? What did they try to do to you?
But dont take it personally. Its the way the system operates. It derives from a political ideology. From your
perspective, its a moral failing. From their perspective, its an essential tool. Deal with it on its own terms.
But keep in mind, they dont think like you do.
Except, Kopits, exactly the same kind of issues pop up in the global warming arena with the political sides
reversed.
RBOctober 9, 2013 at 4:05 pm
Professor DeatonTo quote the old Wendys commercial, Wheres the beef? By your own account it seems
that academia eventually came down on your side. At least thats how I interpret your statement: Our
response, which was published together with Ash and Robinsons note, makes all of this clear. To our knowledge,
there has been no subsequent controversy. Sounds like victory was yours and Prof. Ash was left with egg on
his face. No?
In all fairness you should note that Prof. Ash didnt actually do the research work or write the R-R critique in
any meaningful way. The credit (or blametake you pick) belongs to the grad student Thomas Herndon
and I guess we should include his girlfriend. The accounts Ive read suggest that Prof. Ashs involvement in
the paper amounted to putting his name alongside his graduate students, which is pretty much standard
operating procedure at a lot of universities.
Also, notice that R-R did not handle the situation in the way that you did. The original R-R paper said one
thing and got a lot of attention. A later (error corrected) paper said something a little different. I think most
people would expect some kind of explanation and not just a revised paper with different numbers as
though the first paper never happened. It sounds like you at least acknowledged a small error in your
original paper and successfully explained why it did not upset your original results. About a year ago I
presented a draft paper to a symposium. Further research caused me to revise (in some cases significantly)
some of the conclusions as I refined my methodology and yes, caught a few dumbass errors. But I also felt it
was important to point out and explain the differences between the draft version and the final published
version. I dont think we got that from R-R.
I would also disagree with your claim that the revised R-R numbers are close to the HAP numbers. This is
2slugbaits
October 9, 2013 at 5:43 pm
-
7/24/2019 Deaton, 2013 on Weights and Coding Errors
6/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 6
comparing apples and oranges. The R-R preferred result uses the median. The HAP paper did not replicate
the R-R median but the R-R mean, which is the figure that got all of the attention. I just dont think you can
say that results are similar just because the mean and the median happen to be close. And this is especially
true when the whole argument was over a comparison of the means.
I do agree that one can have legitimate disagreements over weighting techniques used in regressions. It
sounds like the academic community eventually agreed with your weighting scheme rather than Prof. Ashs.
But that does not mean the R-R weighting technique should be off limits to critics. It was afterall a rather
insane weighting technique. Giving a fixed effect based on 19 observations the same weight as a fixed effect
based on one observation is just silly. And even in their revised work the weighting scheme is crazy.
Apparently the US of 1800 is more like the US of today than is any of todays OECD countries. Thats nuts.
Steven Kopits,
Im a rational egalitarian who is now going to obliterate your argument by pointing out that many Nobel
Prizing-Winning economists are (were) also rational egalitarians, including Gunnar Myrdal, Amartya Sen,
James Heckman, Joseph E. Stiglitz, Paul Krugman, and Peter A. Diamond. One could arguably add K. Arrow, J.
Tinbergen, P. Samuelson, R. Solow, T. Koopmans, and several others.
I doubt that any of these economists assume that they cant win in open combat, that they are the
underdogs to your overlord. Your own violation of the principles of rational argument is to generalize from
one ill-mannered article to all egalitarians.
Greg Hill
October 9, 2013 at 6:00 pm
One last comment. Sometimes ugly fights have redeeming but unintended consequences. And the R-R fight
is one of those cases. The basic R-R claim that debt hurts growth was largely unchallenged for 3 years. Then
along came the HAP paper and within a month the scales fell from the eyes of many economists and we
were subsequently treated to a plethora of excellent papers going far beyond the HAP critique. So even if
you thought the HAP paper was an unfair hack job (I dont), then you still have to admit that it spurred
economists to really put the R-R paper under the microscope. And I think it was these further critiques that
really buried the R-R thesis and not the HAP paper. So maybe academic advances resemble sausage
makingnot always pretty.
2slugbaits
October 9, 2013 at 6:05 pm
http://www.the-human-predicament.com/ -
7/24/2019 Deaton, 2013 on Weights and Coding Errors
7/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 7
Professor Deaton,
Thanks very much for sharing your own story. Not surprising I suppose that the same rascals were involved
in a similar drive by.
You raise some important points about the effects of these politicized attacks. Its necessary for economists
to share data and methods if the field is to progress. Yet, as you point out, if your own data might be used
against you to unfairly damage your reputation, youll think twice about sharing it. But its even more
important that economists contribute to the policy debate. Economics already has a bad reputation in the
public mind as a field in which two economists can get the Nobel Prize for making opposite statements.
Thats a misunderstanding of course but it doesnt help when economists unfairly attack one another in a
public forum. Its not just the reputation of the economist being attacked that gets sullied but really the
whole field. People are less inclined to take anyone seriously.
But I think there are some other important lessons to take away from the Reinhart-Rogoff episode. What
you and R&R have in common is that you were both attacked because you had politically important results
that were backed up by real evidence. When thats the case, you have to expect people to attack, often
unfairly.
I dont think its realistic to expect newspapers or political partisans to refrain from this behavior. If you are
saying something important and have evidence to back it up, you have to expect to be assaulted. And when
that happens, you have to defend yourself aggressively.Unfortunately, I think R&R failed to defend themselves effectively. They responded in a very measured way
that was respectful of the UMASS economists. They pointed out the academic reasons why the researchers
were wrong. Its a tribute to their character and objectivity as researchers that they responded this way. It
pains me to say this, since I have only the highest respect for R&R. But this only helped to twist the knife.
When 3 hitmen from Massachusetts mug you on the pages of a international newspaper and on a comedy
tv show, you are no longer in a polite academic dispute. No one who is watching the spectacle understands
the nuances of estimators or macroeconomics. You have to respond in kind. The message people need to
hear in no uncertain terms is that the muggers are WRONG, WRONG, WRONG as well as being foolish and
silly. Point out that they are not experts in the field and have therefore made basic mistakes. Point out that
they have missed perfectly obvious results they should have seen if they had read all the literature. Point out
that their weighting argument reveals an undergraduate understanding of statistics. Make sure people
understand that they have no credibility. If you dont do that, if you treat them like equals, people will
conclude that there might be something to it and will see you as being on the defensive, especially when
theres a full-blown witch hunt on.
People who engage in smear tactics need to pay the price in terms of their own damaged reputations. So do
newspapers who publish the smear tactics. And so do faculty members at your own university who aided
Rick Stryker
October 9, 2013 at 6:27 pm
-
7/24/2019 Deaton, 2013 on Weights and Coding Errors
8/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 8
and abetted the smear.
If they do pay the price, these smear tactics wont work. Thus, Id ask you to qualify your assertion that these
smear campaigns apparently work. Yes, they doif you dont respond to them properly. But its not
inevitable that they work.
When one shows the results are driven by one data point, one indeed has shown the entire enterprise to be
rotten to the core.
LordOctober 9, 2013 at 8:48 pm
I am not sure what you are complaining about
It seems to me that you are complaining about the scientific method.
Another scientist wants to verify your conclusions: he tries to replicate the analysis and highlights the
assumptions (because thats what they are) he is unhappy about. He makes different assumptions and he
gets a different result. Whats wrong with that? Other people will then be able to judge which assumptions
are more reasonable much more easily
Testing other people conclusion for errors is not a disservice in any way. It is a very useful (if somewhat less
creative) effort. All theories are only true until falsified. Sometimes the error might not be materialNot
testing is not an option
acarraro
October 10, 2013 at 2:01 am
While it is naive to think that science can ever be insulated from politics,.. says Deaton.
Deaton is wrong, that can be done easily. Well, Krugman cant tell you the how to, but Bill McBride could tell
you.
Question remains if economics should be treated as a science profession, I question that.
JohannesOctober 10, 2013 at 2:05 am
-
7/24/2019 Deaton, 2013 on Weights and Coding Errors
9/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 9
Rich Stryker:
Yet, as you point out, if your own data might be used against you to unfairly damage your reputation, youll think
twice about sharing it.
Take a look around to see how global warming dissidents reacted to this statement made by Phil Jones of
UEA climate science:
Why should I make the data available to you, when your aim is to try and find something wrong with it?
RBOctober 10, 2013 at 6:27 am
Slightly off topic, but it wouldnt be a complete surprise if Angus got a call from the Nobel committee
sometime soon.
BrianOctober 10, 2013 at 6:49 am
Steven Kopits,
Thank you for setting the term liberal in its proper context when it comes to academia. Liberals do in fact
seek wisdom and knowledge rather than power. They assume that wisdom and knowledge will lead to beinggiven the power they deserve.
As you point out, using the term egalitarian, there are those who seek equality rather than wisdom and
knowledge, and they intuitively understand that this means that they must use power to overwhelm
wisdom and knowledge.
I tend to use the term Progressive because of Woodrow Wilsons appeal to the idea, but Wilsons concept of
Progressive fits perfectly with your egalitarianism which is also illustrated by the contrast of the American
Revolution (based on liberty and freedom) with the French Revolution (based on equality).
Your post is a very thoughtful addition to the post of Professor Deaton.
Ricardo
October 10, 2013 at 9:05 am
-
7/24/2019 Deaton, 2013 on Weights and Coding Errors
10/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 10
It might be important to point out that MIT does have a history of using criticism of the hard work and
research of others to generate academic papers pretending to be rigorous (Paul Krugman).
Greg -
Egalitarianism is based on the declining marginal utility of wealth and income, right?
I dont believe Ive ever seen it argued that way. And theres a reason for that. But, hey, Id be pleased to do
so.
By the way, whats a rational egalitarian?
Steven Kopits
October 10, 2013 at 9:52 am
I think theres another interpretation of my comment. I am reminding Prof. Deaton that his work is also
ideological, but in a different way.
Unspoken but ever-present is the primacy of the value of scarce resources and a certain indifference about
who owns these resources. Thus, efficiency is more important than equity, or put another way, there is the
implicit belief that efficiency more or less is best for everyone. And, further, the professor believes that the
act of arguing itself is neutral. The argument can be divorced from its advocate and the case argued on its
own merits. He, as an individual, does not perceive himself to be at risk as a consequence of debating a
point. Nor does he perceive that any rational outcome of a debate is likely to be personally threatening. The
conclusions are dissociated from the participants.
But viewed from lower down the deck, such an argument may be viewed as having greater stakes. Liberals
were historically criticized for their devil take the hindmost philosophyand if youre the hindmost, you
may have a different view of the merits of the argument. Cheating, stealing and lying dont look so bad if
thats what it takes to keep you from the clutches of the devil.
So, the point I was making was that liberals operate under a specific ideological regime buttressed by certain
assumptions. Egalitarians have their own regime and assumptions. These are not the same.
And, of course, I have not said anything about social conservatives yet.
Steven Kopits
October 10, 2013 at 10:15 am
-
7/24/2019 Deaton, 2013 on Weights and Coding Errors
11/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 1
Regarding sharing of data. Professor Deaton writes: Scholars will also be much less willing to share data than
is currently the case; doing so allows anyone who is unscrupulous enough to turn your cooperation against
you.
Live by the sword, die by sword. If you want open, rational argument, I think data sharing will be essential.
But you need to realize that others (egalitarians, in this case) will have access to types of argumentation that
you do not. But so be it. Warfare is not always symmetrical.
This topic was, in fact, widely discussed at a conference I attended in Sicily in August. Although I was formally
on the energy panel, I found the climate panel more interesting, and spent much of the week with McKitrick,
McIntyre, Monckton, Lindzen and the elusive Svensmark. (Yes, it was fun.)
One of the recurring themes was access to data used by the AGW crowd. (Another was the threats that
leading AGW academics had made to young researchers in the climate discipline. It was quite eye-opening,
and well beyond what I would have expected.)Based on these discussions, I think well see increasing standards of access and documentation related to
data and methodology. I personally think its unavoidable, and academics should upgrade their expectations
and practices accordingly.
Steven Kopits
October 10, 2013 at 11:07 am
Steven Kopits,
You write, Egalitarianism is based on the declining marginal utility of wealth and income, right? I dont
believe Ive ever seen it argued that way. And theres a reason for that. But, hey, Id be pleased to do so.
There are several forms of egalitarianism: a utilitarian view that depends on the diminishing marginal utility
of income; John Rawls theory of justice as fairness; the luck egalitarianism of Ronald Dworkin and others;
the capabilities egalitarianism of A. K. Sen and Martha Nussbaum; and several other strands. So, your first
task is familiarize yourself with some of this literature.
You also write, So, the point I was making was that liberals operate under a specific ideological regime
buttressed by certain assumptions. Egalitarians have their own regime and assumptions. These are not the
same . . . By the way, whats a rational egalitarian?
This paragraph also suffers from a lack of familiarity with the relevant literature. John Rawls is a liberal and an
egalitarian, so is Ronald Dworkin, and both build on some premises shared by non-egalitarian, or less
egalitarian, liberals. In your original post, you claimed that, since egalitarians cant win arguments, they have
to play the cards they have, which includes misinformation and propaganda. All of the egalitarians Ive
mentioned, economists and otherwise, offer reasons for their views; they are rational egalitarians.
Greg Hill
October 10, 2013 at 1:57 pm
http://www.the-human-predicament.com/ -
7/24/2019 Deaton, 2013 on Weights and Coding Errors
12/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 12
Economics is a field where I frequently find that the theory drives the data. Climate science suffers from the
same problem. Peoples careers and beliefs are tied to a particular theory, and the goal of their research is to
justify that theory, not follow the data to a conclusion. Rather than Steven Kopitss liberal vs egalitarian
distinction, I find a tendency to a near religious belief in a particular theory in any field (e.g. social sciences)
which is not subject to exact repeatable observation and experiment. In macro economics Ive found that by
adjusting assumptions and presenting a model correctly its possible to prove any hypothesis, and Ive seen
examples of economists of all schools interpreting data to fit their preferred theory.
In the case of the R+R paper, I noticed two points after the initial controversy came out. The first was a graph
of the raw data, which (if accurate) looks at first glance like a fairly uniform scatter plot of points. It should be
obvious from that distribution that growth is not strongly associated with debt and any trend analysis
should be taken with a large grain of salt.
The other point confirms my suggestion that theory drives data. As soon as the initial error came out, which
said that growth doesnt decline as strongly with high debt (but still confirmed that growth declines with
debt) I started seeing claims that this now proves that the opposite is true high debt actually encourages
growth (with some elaborate analysis which I didnt have time to look at, but seemed to be cherry picking
the data to prove the point).
ThomasWOctober 10, 2013 at 3:45 pm
RB,
I think you are missing the point. This isnt about politics or whose ox is being gored. I believe that all
researchers should turn over their data and methods. Research should be open source. Results need to be
checked and stress tested. I have no problem with people vigorously challenging a researcher with his own
data. Thats how knowledge progresses.
But just as researchers have an obligation to make their data and methodology public, people who receive
the data have obligations too. They have an obligation to present their findings to the original researcher
and give him a chance to explain or rebut. They should fairly characterize any disagreement. They should not
accuse the original researcher of fraud or incompetence without tremendously strong evidence.
I think Professor Deatons point is that Ash did not live up to his obligations. Ash and his coauthor did not
Rick Stryker
October 10, 2013 at 4:14 pm
-
7/24/2019 Deaton, 2013 on Weights and Coding Errors
13/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 13
give Deaton a chance to review and rebut their findings but instead surprised him with an article that was
about to come out. They mischaracterized their disagreement as a coding error, implying a mistake on
Deatons part. And they did not fairly acknowledge that their criticisms, even if true, did not matter for the
final result.
What Ash and company did to R&R was much more egregious. They surprised R&R with a hit piece in the
Financial Times that implied that R&R were dishonest and incompetent. But when you look at the substance
of Ash et als charges, they were ridiculous.
I think Professor Deaton was just bemoaning the fact that when researchers behave the way Ash and
company did, it poisons the atmosphere, causing researchers to be wary about giving out data, which is bad
for progress.
All science should be open source. Climate skeptics should ask for data and climate scientists should hand it
over. Climate skeptics should vigorously challenge the scientists. I have no problem with that as long as the
climate skeptics live up to their ethical obligations too.
Rick,
it poisons the atmosphere, causing researchers to be wary about giving out data,
I agree with this.
On the other hand ..as long as the climate skeptics live up to their ethical obligations, assumption of fraud
and incompetence is often the case with the dissidents, not many of whom can truly be called skeptics.
Interestingly, McIntyre saw parallels between R&R and his bete noire Michael Mann. More here .
RB
October 10, 2013 at 6:34 pm
Dean Baker has a post up discussing this guest post.
http://www.cepr.net/index.php/blogs/beat-the-press/the-inconvenienced-economist
benamery21
October 10, 2013 at 9:04 pm
RF
October 11, 2013 at 3:40 am
http://www.cepr.net/index.php/blogs/beat-the-press/the-inconvenienced-economisthttp://climateaudit.org/2013/04/18/the-hockey-team-and-reinhart-rogoff/ -
7/24/2019 Deaton, 2013 on Weights and Coding Errors
14/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 14
The description of the R & R controversy in this narrative is simply untrue. Five minutes of random searching
on academic blogs will give you a better view of the controversy than this story does.
Greg,
Please, educate me on how any of your thinkers contradict the notion of making transfers from the rich to
the poor based on declining marginal utility of wealth and income. Did any of these thinkers endorse
transferring money from the poor to the middle class or rich? Please cite an example and explain.
And, yes, you do argue like an egalitarian:
All of the egalitarians Ive mentioned, economists and otherwise, offer reasons for their views; they are
rational egalitarians.
So if you provide a reason for your thinking, that qualifies as rational? I dont have my homework because
the dog ate it? That constitutes rational in the context of this discussion?
Rational, in this sense, I would think means that transferring funds from the rich to the poor makes social
sense because it increases aggregate utility. Now thats rational. But thats exactly what I contend: that
egalitarianism (indeed, civil society) is built on the declining marginal utility of wealth and income. Thats why
such transfers make sense.
And let me tell you whats wrong with that from an egalitarians point of view. Declining marginal utility is a
three-edged sword. First, in a static analysis, its absolutely true that transferring money from the rich to the
poor increases aggregate social utility. It has to (if its in cash).
But its also true that utility increases with income. So I can trade off growth against redistribution. And I can
do it quantitatively.
And finally, declining marginal utility also means that inequality at higher income levels is less important than
at lower income levels. And I can quantify that, too.
So if we use declining marginal utility of wealth and income (OK, lets give a nod to Scott Sumner: income) as
our model for equality, then I can offset any incentive scheme for equality against one based on GDP growth
and debt sustainability (the FAA).
In other words, I can take two of my three objective functions for government and put them on an apples-to-apples basis. And thats why egalitarians dont use the declining marginal utility of income in their
argumentation. Because they then have to argue for equality in the open, on equal terms. And they just
dont feel confident they can win that debate.
Steven Kopits
October 11, 2013 at 5:27 am
Steven Kopits
-
7/24/2019 Deaton, 2013 on Weights and Coding Errors
15/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 15
Rick -
Ash has no responsibility to Deaton. Thats what I am arguing.
You state:
people who receive the data have obligations too. They have an obligation to present their findings to the
original researcher and give him a chance to explain or rebut. They should fairly characterize any
disagreement. They should not accuse the original researcher of fraud or incompetence without
tremendously strong evidence.
I think Professor Deatons point is that Ash did not live up to his obligations.
You are making a socially conservative case, (our third objective function, dealing with the allocation of rights
and responsibilities within the group).
You are arguing that Deaton and Ash are part of the same groupacademicsand they therefore should
share common cultural norms. Deaton is upset specifically because he feels these norms have been violated.
I am arguing that they are, in fact, not in the same cultural group. Deaton is a liberal, Ash is an egalitarian.This is not an intra-cultural disagreement, its an inter-cultural one.
In an inter-cultural agreement, you may use tools which you would not in an intra-cultural one. For example,
you might write a critical post on a widely read economics blog.
October 11, 2013 at 5:37 am
Steven Kopits,
You write, Please, educate me on how any of your thinkers contradict the notion of making transfers from
the rich to the poor based on declining marginal utility of wealth and income.
Sorry, but youve got to take responsibility for your own education. I mentioned lots of egalitarian thinkers
whose work you could Google, or even look up on Wikipedia. Until you do that, youll remain stuck in your
echo chamber, arguing against a simple-minded caricature of egalitarianism that youve created out of whole
cloth.
Ill probably regret this, but you might take a look at http://works.bepress.com/greg_hill/3/. Good luck.
Greg Hill
October 11, 2013 at 10:07 am
Antiderivative
October 11, 2013 at 12:14 pm
http://www.the-human-predicament.com/http://works.bepress.com/greg_hill/3/ -
7/24/2019 Deaton, 2013 on Weights and Coding Errors
16/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 16
In our case, as in Reinhart and Rogoff, neither the coding error (in our case there was none) nor the choice
of weights has any effect on the main result
Actually, it did. Once corrected, the 90% debt-to-gdp ratio cliff became insignificant and causality was shown
to go the other way that poor economies lead to high debt-to-gdp rather than high debt hindering
economic growth.
I am baffled that economists are still implying that R&R results validate fiscal consolidation in a time of severe
AD deficiency.
as a PhD in molecular biology, I find it literally unbelievable that you economists, when you publish a paper,
dont have an excel file (excel is the most portable, and can easily handle a few million data points) with all of
the data, going from whatever the primary was to your endpoint, and when you get a request, you just email
or dropbox the excelto say that the data is on a govt website is just BS and is not acceptable.
For instance, sometimes people say that http://www.bls.gov is a reference.
Have you ever been to bls.gov ?
or, just as bad, people will say some page xxx.gov is a reference, and when you go to that page, there are
links to 20 diff data sets, and you dont know,explicitly which dataset has been used (much less, has the data
set changed over time)
what is WRONG with you people !!!
how on earth do you people call yourselves scientists when you dont deposit the RAW data on acceptance
of your MS ????
ezra abrams
October 11, 2013 at 3:48 pm
Steven,
I guess Im talking about what ought to be obligations rather than what people perceive their obligations to
be. Maybe Ash doesnt think he has those obligations. But I bet if you asked him hed say he does and would
mean it sincerely.
I think there is something to what you are saying in general. But in this particular case, I have a more
pedestrian view of what happened. I think HAP, motivated by ideological bias, believed that they found
evidence of errors and dishonesty on the part of R&R and rushed to tell the world. But what they thought
Rick Stryker
October 11, 2013 at 9:33 pm
http://www.bls.gov/ -
7/24/2019 Deaton, 2013 on Weights and Coding Errors
17/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 17
they found was mostly a reflection of their own ignorance. I think we can see that if we look in detail at the 3
claims they made.
First, they found a spreadsheet error in the data. That really did happen and R&R acknowledged that. These
kinds of errors happen to everyone. They are not evidence of chicanery and the error didnt change the
results materially.
Second, they found that some countries had not been included that would have changed the results. That
was true but was a mistake on HAPs part. They just didnt realize that the data wasnt available at the time to
be included at the R&R paper was written. Had they been more careful and checked with the authors, HAP
could have avoided making that mistake.
Third, they claimed that the R&R weighting method was unconventional and non-standard. Here they
reveal their ignorance of econometrics. Youll have to bear with me as I explain this, as it is technical.
The issue that HAP didnt realize they were talking about is a problem in panel data econometrics, in which
you have both time series and cross sectional data. The HAP way of weighting is ordinary least squares in
which country specific differences are assumed away and only differences across all countries matter. R&R
assumed whats known as fixed effects which goes to the other extreme, assuming that country specific
differences are fixed constants. As JDH has already pointed out, these are the two extremes of the moregeneral random effects model. A random effects estimate will lie somewhere between R&R and HAP.
How do we know where the random effects estimate will end up? Basically, it depends on this quantity:
Z = v(u(i,t))/(v(u(i,t) + Tv(a(i))
where v(u(i,t)) is the variance of a random error that depends on country i and time t, v(a(i)) is the variance of
a random variable that depends only country i, and T is the number of time observations.
If Z = 1, then you get the HAP estimator. If Z = 0, you get R&R. In practice, Z will lie somewhere in the middle.
Lets look at the extremes.
If the number of observations T is small, then Z goes to 1 and you get HAP. That makes sense intuitively since
if you dont have enough data in time, you cant isolate the country specific differences. On the other hand, if
T is large, Z goes to 0 and you get R&R. Intuitively, if T is large, the across country differences wash out.
If V(a(i)) is small, then Z goes to 1 and you get HAP. Makes sense because then there is no country specific
variation, just what HAP assumes.
If V(u(i,t)) is small, then Z goes to 0 and you get R&R. Makes sense since R&R are assuming that only country
specific differences matter.
So, ultimately where you end up depends on how much data you have relative to the size of the variation of
country-specific and across-country differences.
Steven, I apologize for writing down this technical detail but I wanted it be clear that this is really a well-
understood technical question.
If you look at the HAP analysis on the R&R weighting in their paper, you will find no discussion like this. They
make hand waving arguments and pronouncements that R&R is unconventional. They also make an
irrelevant point about serial correlation. But they never even come close to showing that they understand
the real issue.
The reality is that HAP had no clue what the real issue is when they wrote their critique of the R&R weighting.
They just a simple estimate that gave them an answer they liked better and then they ran with it.
Whats worse, they assumed that the mistake they made and the ignorance they exhibited were evidence of
-
7/24/2019 Deaton, 2013 on Weights and Coding Errors
18/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 18
mistakes and ignorance on R&Rs part. And then they told the world on the pages of the Financial Times and
on the Colbert Report. Meanwhile Krugman at the NY times repeated their analysis uncritically, giving HAP a
big credibility boost.
I dont think what HAP did had had much to do with feeling over powered by the well-educated R&R. I think
they really believed they had discovered major flaws in R&R and were in the right. HAPs egalitarianism was
undoubtedly a factor in that it provided the ideological motivation for them to jump to the conclusion that
R&R were mistaken and dishonest. But in the end it was HAP who were mistaken and dishonest.
Rick StrykerIf each of the countries had 19 episodes of debt > 90%, then that would have been one thing.
But thats not what happened. A lot of the countries had only one or two episodes of debt > 90%. Attempting
a fixed effect model with only one or two observations is an exercise in self-deception. A mean based on one
observation is next to meaningless when you know that other countries with many observations showed a
lot of variation around the country mean.
Also note that they subjectively grouped countries into emerging and developed countries. I dont have a
problem with that, but presumably they did so because of different fixed effects. But if they believed in their
fixed effect approach, then why did they feel they had to first group countries into two somewhat arbitrary
larger categories? And dont you find it a little strange that the US in 2009 is treated as being more like the US
of 1946 than Germany, France or the UK of 2009? Ill get back to this point a little later.
If you will read HAPs reply to JDHs post, you will see that they are not saying their approach is without its
problems. In fact, they specifically said that a better estimate would lie somewhere between their approach
and R-Rs approach. Go read their reply. In fact, their whole point about serial correlation was that this was
just one example of why a researcher might not want to give each country/year observation equal weight. In
other words, they were offering up a reason not to weight things they way they did.
Theres also this problem about what R-R did versus what they said they did. Heres how R-R described their
weighting procedure:
The annual observations are grouped into four categories, according to the ratio of debt to GDP
during that particular year as follows: years whendebt to GDP levels were below 30 percent (low
debt); years where debt/GDP was 30 to 60 percent
(medium debt); 60 to 90 percent (h igh); and
above 90 percent (very high).
Anyone familiar with the English language would interpret R-R as saying they were giving equal weight to
each country/year observation. R-R did not say that they were taking an average of each countrysannual
observation, they said that they took an average of the annual observationsand grouped all of those
annual observations into four categories. Even if you dont think their weighting procedure was
2slugbaits
October 12, 2013 at 7:20 am
-
7/24/2019 Deaton, 2013 on Weights and Coding Errors
19/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 19
unconventional, you would have to agree that their misuse of the English language was most
unconventional.
HAP also pointed to another curious omission in the R-R paper. As R-R noted, the decade in which a
countrys debt-to-GDP ratio exceeds 90% is significant. Their word. Strangely, even though they say the
decade is significant, apparently its not significant enough to bother controlling for it in their analysis.
Didnt this bother you when you read their paper? Afterall, in applied econometrics its common to include
some time variable. For example, when measuring production or cost efficiencies in stochastic frontier
models it is usually assumed that firms become more efficient over time, so those models will usually try and
account for that fact. Why didnt R-R since they found the temporal factor significant? Well, perhaps they
didnt include it in their paper because doing so would have severely undermined their results. One of the
things that the HAP results highlighted was how irrelevant the debt-to-GDP ratio becomes over time. If you
look at the last two decades there is no difference in the growth rates across all four categories.
Just a technical note about random effects models. In general they are preferred over fixed effects models;
however, random effects models are not always feasible. The data have to cooperate. The composite error
cannot be correlated with the explanatory variable.
Rick StrykerI dont think the R-R paper fell into the dustbin of economic analysis solely because of the Lefts
embrace of the HAP paper. The HAP paper really prompted a lot of economists who had previously accepted
R-Rs finds (e.g., Miles Kimball) to take a second look. And it was those follow-on papers that really crippled
the R-R thesis. Way back in college I took a course on the Faust tradition in Western literature. One of the
recurring Faust themes was that of Mephistopheles as a pike in a carp pond. There was an old belief that
activity and movement improved the taste of European Carp (considered a delicacy), so it was common to
throw in a pike (considered a garbage fish) into the pond to keep the otherwise slothful carp active. I see the
HAP paper as a pike that wakened the economics community from intellectual laziness. So I guess in that
sense R-R are right to complain about carping critics.
2slugbaits
October 12, 2013 at 7:57 am
Steven Kopits wrote:
First, in a static analysis, its absolutely true that transferring money from the rich to the poor increases aggregate
AnonymousOctober 12, 2013 at 8:13 am
-
7/24/2019 Deaton, 2013 on Weights and Coding Errors
20/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 20
social utility.
Steven,
There was a time when I accepted this but not any more. To make this true it requires more than static
analysis. It requires a suspension of reality and a fixed point in time. If such transfers are seen over time not
only is there a decline in the marginal utility of wealth and income there is a decline in social utility.
Removing resources from the productive in society reduces both finished goods and factors of production.
That means that there is a general or aggregate reduction in wealth not just the utility of wealth but an
actual reduction in wealth. This means that the poor actually see a social decline because they have less
access to goods.
An obvious example of this is the poor of the United States compared to the poor in most of the rest of the
world.
If by static you mean frozen in time perhaps in theory it could be correct but such a freeze must be
theoretical because transfers imply time. In any flow of time such forced transfers reduce both the marginal
utility of wealth and the actual supply of wealth, leading to a reduction in social utility.
Rawls thinking is only meaningful in an alternative universe of his own making. I would refer Greg to Robert
Nozick.
Ezra Abrams,
Unfortunately, its not as simple as that. R&R did make their data available on their website. However, HAP
(Henrndon, Ash, and Pollin) were not able to reproduce the calculations. So they requested that R&R send
the spreadsheet that implemented the calculations. Thats what led to the controversy.
Replication isnt just about access to data. I have my own replication horror story to illustrate. Years ago I
came across a multivariate GARCH model on financial data in a paper. I thought the model would be useful
for something I was doing and I decided to start by replicating it.
The data were clearly defined and publicly available so I didnt have that problem. Also the model was clear.
So I coded it up in matlab, using matlabs optimization toolbox.
And I didnt get the authors results.Even if I started the maximization at the published estimated values, I didnt get the authors results using
the exact data set. I thought it might be a problem with the optimization toolbox. I looked at that code and
tweaked it. Nothing.
At that point, I thought that maybe I need to go to better optimization code. So I re-wrote the estimation in
c++ and used the IMSL c-library for optimization. No luck. I then tried the NAG c-library. Still cant do it.
At this point, I was determined to replicate the GARCH results. Since I didnt have the source code to NAG or
IMSL, I decided to write my own optimizers. I implemented quasi-Newton and trust region algorithms with
numerical derivatives, but that didnt work any better than anything else.
Rick Stryker
October 12, 2013 at 9:27 am
-
7/24/2019 Deaton, 2013 on Weights and Coding Errors
21/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 2
To avoid derivatives, I coded up the Nelder-Mead simplex. Nopenot any better. Maybe I better switch back
to a derivative-based method? Since I thought the problem might be rounding error on the derivatives, I
implemented a c++ class that did arithmetic with arbitrary precision to get around numerical derivative
round off error. Didnt work.
Then I implemented automatic differentiation in c++ to compute exact derivatives of the likelihood function.
That worked no better than anything else.
I gave up finally.
A couple of years after that I ran in to the one of the authors of the paper and told my story. He said the
optimization was a beast and they couldnt do it either. So they implemented some judgmental overides in
the optimization code. Unfortunately, that wasnt described in the paper though.
These sort of problems are endemic and occur in other fields such as biology and medicine. The book Wrong:
Why experts* keep failing usand how to know when not to trust them *Scientists, finance wizards, doctors,
relationship gurus, celebrity CEOs,consultants, health officials and moreis an interesting laymans
account.
These sorts of problems occur even in the hardest of sciences. People know the story about how Eddington
led an expedition to Africa to photograph stars near the sun during an eclipse in order to confirm theGeneral Theory of Relativity. When Eddington confirmed the theory, Einstein became world famous. Whats
not commonly known is simultaneously Eddington also sent an expedition to Brazil to do the same
experiment. That experiment confirmed Newtons theory of gravity. Eddington believed that the General
Theory was correct and so suppressed the Brazil results.
Kopits: Really, what is egalitarianism? It is taking away things that other people own through political force.
You talking about the Republican hostage taking????
anon2
October 12, 2013 at 10:26 am
There is such a lot of sophisticated name calling going on here. I am learning great stuff! Thank you all.
If I had gotten a doctorate in economics, would I have learned all these great name calling techniques? Or
would I have learned economics? Whatever the hell that is.
Did Ash and his colleagues criticize R&R and Deaton? Apparently, so.
anon2
October 12, 2013 at 10:37 am
http://www.amazon.com/Wrong-us---Scientists-relationship-consultants/dp/B005DI6QAM/ref=sr_1_4?ie=UTF8&qid=1381594299&sr=8-4&keywords=wrong -
7/24/2019 Deaton, 2013 on Weights and Coding Errors
22/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 22
Did R&R update their conclusions? Apparently, so.
Did Deaton? Apparently, not.
Are any of these questions settled? Not as far as I can see.
So, bottom line, Deaton does not like how Ash criticizes him or R&R.
BFD.
AnonymousRawls thinking is only meaningful in an alternative universe of his own making.
Just to be clear, Rawls never said that justice as fairness required absolute equality. Its entirely possible that
a radical redistribution of income from the wealthy to the poor might make the poor worse off; and it is for
that reason that Rawls very early on introduces the difference principle in A Theory of Justice. Under the
difference principle unequal income distribution is allowed only if it makes the worst off better off. So
inequality is conditionally allowed. Of course, Rawls doesnt deal with the problem of what happens if making
those at the very bottom better off also makes those that are next to the bottom worse off. Rawls assumes
that the difference principle operates monotonically, which may not always be the case.
2slugbaitsOctober 12, 2013 at 12:10 pm
Rick StrykerInteresting story. Off topic, but since you brought up multivariate GARCH models, Im curious if
youve ever found any practical real world use for them. Anything beyond the most simple MGARCH model is
computationally intractable. And I say this as someone who has dedicated a fair amount of his professional
reputation defending the practical value of plain old vanilla GARCH models. A few years ago I had a deskside
with the Assistant SECDEF explaining why GARCH models should be incorporated in multi-echelon logistics
models. The savings were modest (~$45M/year), but very real. My colleague and I showed that including
GARCH terms dampened the dreaded bullwhip effect in multi-echelon models, but we were never able to
find much value in extending that approach to an MGARCH model.
2slugbaitsOctober 12, 2013 at 1:01 pm
Rick Stryker
October 12, 2013 at 2:21 pm
-
7/24/2019 Deaton, 2013 on Weights and Coding Errors
23/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 23
2slugbaits,
Just where do Ash and Pollin acknowledge in their reply to JDHthat a better estimate would lie somewhere
between their approach and R-Rs approach. They didnt do that because they didnt understand the
relationship between their proposed estimator and R&R when they wrote their paper and they didnt
understand it when they replied to JDH.
If they had understood this, they would have never proposed to do the equal weighting they did. In panel
data, the question is whether to use fixed or random effects. If T is large, it doesnt make any difference
because they are essentially the same. But in the R&R case, when T is not large, its a more difficult question. I
tend to agree with JDH that random effects would be betterbut you really have to try it to see.
HAP called what R&R did an unsupportable statistical technique in their FT article. They called it
unconventional and non-standard in their paper. And they called it unusual in their reply to JDH. But
what HAP did is non-standard. You dont start with ordinary least squares (OLS) as HAP did in a panel data
context. If they had understood this issue better, they might have argued that R&Rs estimate should be
closer to OLS by appealing to the sort of arguments I raised above, rather than making their garbled serial
correlation point. But even if they did that, they cant claim that their OLS estimate is really better than R&Rs
without having done some detailed empirical work using a random effects estimator. They never did that.
2slugbaits,
I never did find a practical use for multivariate GARCH models either. Univariate GARCH models are very
useful though.
I dont know anything about the application you mention but Im sure youre right that MGARCH doesnt
work for what you are doing.
Rick Stryker
October 12, 2013 at 3:42 pm
Rick StrykerHere is where they say that:
We do state in our paper that due, for example, to issues of serial correlation, one might not want to give fully 19
times the weight to the UK experience relative to the one New Zealand year. Just to make sure this is clear, here is
what we say on pp. 7-8 of our working paper:
RR does not indicate or discuss the decision to weight equally by country rather than by country-year. In fact,
2slugbaits
October 12, 2013 at 4:19 pm
http://www.econbrowser.com/archives/2013/04/reply_to_prof_h.html -
7/24/2019 Deaton, 2013 on Weights and Coding Errors
24/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 24
possible within-country serially correlated relationships could support an argument that not every additional
country-year contributes proportionally additional information.
How else do you interpret the words one might not want to give fully 19 times the weight to the UK
experience relative to the one New Zealand year? In other words, they fully recognize that their approach is
not without its problems and that some approach mid-way between what they did and what R-R did was
something to consider. So I dont see where they are disagreeing with JDHs view at all. JDH did not see their
example of serial correlation as convincing, but thats a secondary issue. The main point is that they agreed
with his larger point that some in-between approach might be better than either the R-R or HAP approach.
HAP never claimed to be using a panel data approach. They were clearly using something akin to a cross-
sectional approach. And in a second analysis they included decade variables to capture the time effects. It
turns out that this was important because ALL of the effects of debt on growth were attributable to the
decade in which that debt occurred. Perhaps thats why R-R found the decade significant but for some
reason decided against including it in their analysis. What was unconventional and downright weird about
the R-R panel data was to pretend you could get precise estimators with a fixed effect sample of one.
I dont know which econometrics textbooks you follow, but back in the day I was always taught that the
decision whether to use a fixed effect model or a random effects model is really driven by the followingconsiderations:
(1) If youre primarily interested in differences in the mean, then you will probably want to use a fixed effect
approach. If you are mainly interested figuring out whether or not an observed effect is the result of a
random draw from some probability distribution that includes that effect, then use a random effects
approach. Obviously, in the R-R case a random effects approach was infeasible because you cannot establish
confidence intervals when you have one observation in many of the groupings.
(2) If you are estimating a fixed effect OLS model by adding dummy variables, then you are likely to quickly
chew up degrees of freedom. You can get around this problem by subtracting a population mean, but this
means you are also subtracting away a lot of information. A random effects model can get around this
problem and is one reason why it is oftentimes preferred.
(3) Increasing the size of T does not solve the problem of composite error terms being correlated with the
explanatory variables. Thats why every econometric study using random effects will include a Hausman test.
You can only use a random effects OLS model if the composite error term is uncorrelated with the
explanatory variables.
Going back to Prof. Deatons piece about Prof. Ash not adhering to a certain academic code of conduct, I
think Prof. Deaton likely has a point in his particular experience. Afterall, that was literally an academic fight.
But I dont think that carries over to the R-R/HAP debate. Remember, the R-R paper was not peer-reviewed.
The only reason it even got published in the AER is because of their sterling reputation. Then R-R moved out
of the ivy walls and ivory towers and entered the realm of public intellectual. They testified before Congress,
knowing full well how their paper was being (ab)used. They wrote op-ed pieces. The went on the talk show
circuit. Once they entered the world of public intellectual they forfeited the right to expect the kind of kid
glove treatment they enjoyed in academia. When the topic becomes central to a lively public policy debate
you are no longer entitled to quiet behind-the-ivy discussions between academics: Abandon all hope ye who
enter here.
-
7/24/2019 Deaton, 2013 on Weights and Coding Errors
25/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 25
@JDH Thanks for publishing this.
Tom
October 12, 2013 at 7:19 pm
Anonymous,
You write, Rawls thinking is only meaningful in an alternative universe of his own making. I would refer Greg
to Robert Nozick
When Rawls says, inequalities are just insofar as they work to everyones advantage, do you think this isnt
meaningful, or do you just disagree with it?.
And, as for Nozick, if you find his Lockean story of how individuals acquire a right to parts of nature they mix
their labor with, then Im torn between replying, really? or does it follow that labor deserves the full right
to its product? You might be interested in Nagel and Murphys book, The Myth of Ownership: Taxes and
Justice.
Greg HillOctober 13, 2013 at 12:06 am
2slugbaits,
No, Ash and Pollin did not acknowledge JDHs point in their reply. They quoted themselves, repeating their
serial correlation argument, and then concluded in their reply that In short, doing simple country-year
weighting strikes us as more reliable in this case than taking country averages.
As usual, you have not responded to any of my points but rather with tangents and irrelevancies. My points
were, to repeat:
1) There is nothing unusual, non-standard, or uncoventional about starting with fixed effects as R&R do.
In fact, starting with OLS in panel data as HAP do is whats unconventional. If you think that using fixed
effects in panel data is unusual as HAP apparently do, then make an argument.
2) I wrote down the conditions under which the estimator would converge to HAPs OLS estimator. If you
Rick Stryker
October 13, 2013 at 9:26 am
http://www.the-human-predicament.com/ -
7/24/2019 Deaton, 2013 on Weights and Coding Errors
26/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 26
want to defends HAPs use of OLS, you need to show how their nebulous serial correlation argument results
in a combination of small T, small variance of the country-specific effects, and large variance of the across
country error, such that OLS would be justified.
3) Its difficult to see how this justification could be done a priori. You would need to do empirical work with
the random effects model, which neither you nor HAP have done.
I think you are ignoring Professor Deatons point. Hes saying that if we tolerate smears, how can we expect
serious policy-oriented scholars to inform the public debate? I agree with his point.
Greg,
I apologize. I was the Anonymous who responded to your Rawls commnets.
First, thank you for the link to your paper. I found it to be a very good analysis of Rawls thinking.
I agree that the utilitarian argument fails on the grounds Rawls writes about. Nozick also is weak when he
makes the argument from a Lockean point of view, but Nozick is closer than Rawls.
Rawls makes a static assumption. He assumes that income inequality always means MWO become more
wealthy than LWO, but in a growing economy this is not valid. If at time 1 MWO has a TV but LWO does not
and then at time 2 MWO has a wide screen TV but LWO now has a smaller TV, though MWO is more well off,
LWO is not really also more well off. Where Rawls makes his error is at a similar point to where Keynes makes
his error. Monetary income does not equal wealth. If one acquires more goods at a lower cost, monetary
income or monetary measurement of value could decline, but obviously more goods means would mean
greater wealth. The poorest today live significantly better than the richest of 300 or even 200 years ago.
Related to this, Rawls does not recognize the inter-relationship between the MWO and the LWO. In a free
market world the MWO is a greater producer than LWO that is what created the income differential. When
resources are taken from MWO and given to LWO, but the production of goods declines the whole society
including LWO becomes less well off. This is not a utilitarian analysis of the society as a whole but of each
individual having fewer goods.
For Rawls analysis to be valid he would have to prove that LWO is always harmed with increased prosperity.
There is actually evidence concerning this. The poorest in the United States compared to the poorest in therest of the world.
Rawls reasons from faulty assumptions.
Ricardo
October 14, 2013 at 11:21 am
Michael Ash
October 15, 2013 at 2:57 pm
-
7/24/2019 Deaton, 2013 on Weights and Coding Errors
27/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 27
I have posted a response to Angus Deatons October 9, 2013, Econbrowser post, On weights and coding
errors: odd coincidence or dress rehearsal? which critiques my 2009 paper with Dean Robinson, at
http://people.umass.edu/maash/deaton-2013/response-to-deaton.html and
http://people.umass.edu/maash/deaton-2013/response-to-deaton.pdf.
Professor Ash,
Ive never used stata, but it seems to me you are misinterpreting the stata manual. I believe the manual is
saying that if you want to weight a regression by n, that thats mathematically equivalent to running OLS in
which you multiply the data by sqrt(n).
Look at this stata referencefor example. The technical note below example 7 shows that a regression by
weighted by n is equivalent to multiplying the data by the square root of n. Also, look at example 7. The
example shows that if you want to weight the regression by population, you run the command
. regress drate medage i.region [w=pop]
where they note that w is equivalent to aweight in this context. Thus, if you want to weight the regression by
the square root of the population, it would seem to me that this reference is saying that you would set
aweight = sqrt(pop), just as Professor Deaton did.
Like I said, Ive never used stata but it looks to me like Professor Deaton did the right thing and there is no
coding error. Can you explain further if you think there is?
Rick StrykerOctober 15, 2013 at 6:02 pm
Professor Ash,
To put the point more formally, a weighted least squares regression, with weights w(i) is the solution to the
problem:
minimize with respect to b0, b1, b2,bk
sum(i = 1 to n) w(i)(Y(i) b0 b1*x1(i) b2*x2(i) bk*xk(i))^2
If we define the matrix W to be a diagonal matrix with weight w(i) on the diagonal, X to the be matrix of
independent variables, Y to be the be the matrix of dependent variables, and b the vector of coefficients, we
have the normal equations for the weighted regression:
(XWX)b = XWY (where X is the matrix transpose)
Rick StrykerOctober 15, 2013 at 9:24 pm
http://www.stata-press.com/manuals/r_regress.pdfhttp://people.umass.edu/maash/deaton-2013/response-to-deaton.pdfhttp://people.umass.edu/maash/deaton-2013/response-to-deaton.html -
7/24/2019 Deaton, 2013 on Weights and Coding Errors
28/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
http://econbrowser.com/archives/2013/10/on_weights_and 28
So, the estimate b for the weighted regression is
b = inv(XWX)XWY
Now suppose we define Z = W^(1/2) to be the square root of W, where the weights on the diagonal are the
square roots of w(i). Obviously, Z*Z = W. Lets multiply the regression through by Z, which means that we
multiply the data in the regression by the square root of the weights. Our estimate of b is
b = inv((ZX)(ZX))(ZX)(ZY)
= inv(XZZX)(XZZY)
= inv(XWX)(XWY)
This is the same as the weighted regression.
This shows that a weighted regression is equivalent to doing OLS where you multiply the data by the square
root of the weights. Thats what the stata manual meant.
It seems to me that you have misinterpreted the stata manual. If Professor Deaton wanted to do a weighted
regression where the weights are the square root of the population, he would be correct to set the weight to
the square root of the population, not to the level of the population as you asserted.
Moreover, it would seem that if you have misinterpreted what the stata command is actually doing, you
have also reported incorrect results in your replication article.
Slug,
I never thought I would say this but your October 12, 2013 12:10 PM post on Rawls is excellent especially
your comment, Rawls assumes that the difference principle operates monotonically, which may not
always be the case.
My comment: Rawls thinking is only meaningful in an alternative universe of his own making.If LWO never
benefit from redistribution then his entire theory fails. He must first prove his assumption. He asks a
question then answers it without any proof that his answer is valid. And as you note he analysis is
incomplete as it relates to the entire population.
Ricardo
October 17, 2013 at 10:06 am
The key claim being made by Professor Ash in his response is that when Deaton and Lubotsky (D&L) claimed
that they were weighting the regression by the square root of the population, that meant that they were
Rick StrykerOctober 17, 2013 at 11:32 am
-
7/24/2019 Deaton, 2013 on Weights and Coding Errors
29/29
5/31/2014 On weights and coding errors: odd coincidence or dress rehearsal? | Econbrowser
Comments are closed.
multiplying the data by the square root of the population. As Ive suggested in my comment above, that
view would be a misinterpretation of what weighted least square is. Still, since I dont know anything about
this literature, I thought I would check to see if D&L were somehow implying that they were doing what Ash
and Robinson (A&R) think they were doing.
So, I looked at D&Ls (2009) reply to A&R. There is no ambiguity.
In equation 1, D&L assert that their estimator is
b = inv[sum(i=1 to n)w(i)x(i)x'(i)][sum(i = 1 to n)w(i)x(i)y(i)]
where w(i) is a scalar weight for the ith observation, x(i) is a column vector of explanatory variables for the ith
observation, and y(i) is the ith observation of the dependent variable. x(i) is the transpose of a column
vector to a row vector.
It is easily verified that this estimator is equivalent to what I wrote in my comment above in which the
weights are on the squared innovations, not on the raw data. It is standard weighted least squares. Thus
D&L are correct to set the weight to the square root of the population in the stata code. There is no coding
error.
There are some interesting similarities and differences in the R&R and D&L cases. Whats similar is that in
both cases the critiques were based on a misunderstanding of econometrics. In the R&R case, it was basedon HAP not understanding panel data econometrics. And in D&Ls case, its apparently based on A&R not
understanding how weighted regressions work.
Neither case need have happened. If HAP and A&R had come to R&R and D&L with their criticisms privately
and discussed them, I would hope that we would have not ended up with these false claims.
But the difference is instructive too. The R&R discussion was conducted in newspaper editorials and on a
comedy show. But in D&Ls case, there was a journal involved that was supposed to be an independent
arbiter. In a dispute like this, I would think that A&R should have sent their code and data to D&L for
discussion. If A&R couldnt agree with D&L about whether there really was a coding error, then the editor
should have gotten involved along with the referees to make a judgment. Amazingly enough, it doesnt
sound like that happened here.
So, the moral of these stories is:
Quis custodiet ipsos custodes?