Theory vs. Empirical Results

Theory is belief. It’s overarching values and beliefs that drive a researcher to choose one path of investigation over another. There is no one ‘smoking gun’ that validates (or invalidates) the theory. There is no easy understanding of the theory. It takes many years of hands-on experience to come to truly believe the theory; to internalize it to the extent necessary to be able to further contribute and enhance the theory: to be able to speak the implicit language of the believers.

Lousy Studies

Is my belief against the minimum wage really based on evidence or just the logic of the case? I was forced to confront this in a different setting. I went to the University of Vermont and debated Bill McKibben on buying local, and I didn’t do a very good job in the debate part. I was unprepared for the arguments that he made which were mainly empirical. I thought the logical arguments I made were pretty decisive and compelling using economics to explain why buying local is a bad idea …financially. It might be good for other reasons, but it’s not going to make your community rich, to quote ‘keep all the money in the community.’ But in response to [my logical arguments] one of the things McKibben did was he quoted all these studies and he gave a few specific examples that purport to show that buying local’s a good idea. Now those studies ... are often sophisticated empirical studies that are often done by consultants, they’re often done by localities trying to make a case for buying local… And I found that my argument against his invoking those studies was not very effective. I think it’s right, but it’s not effective. My argument is: are you kidding? Those studies are horrible. Give me a half an hour and I can show you why they’re horrible. And I gave some examples of why they’re flawed, what they miss out, the hidden things they ignore, etc. But it was fascinating to me that unless I had my own lousy study, badly done, purporting to prove that buying local is actually a bad thing, which I’m sure I could cook up or find one, I was disarmed. What I should have said perhaps was, well I can’t argue with your studies, how about you argue with my logic. It’s just interesting that the logic argument is often, I think, inferior in the ear of the listener, to the ‘Oh, I’ve got a study, I’ve got a big appendix with all these data analyses, and calculations that I’ve made.’

...similar study came up arguing whether Walmart lowers wages when it comes to a town ... When I confronted one of the authors of this study with the point that, well gee, wouldn't you think that when Walmart comes to town it would increase the demand for labor and drive up wages? He had found in his study that Walmart lowered wages but not in small towns, only in large metropolitan areas. ... Isn't that even harder to explain empirically? If you thought Walmart would have an effect, shouldn’t it lower wages in small towns rather than in large, I mean large towns have a lot more competition. And he looked at me and he dismissed me and said very confidently:

Well yeah, you have a neoclassical view of the world. In your view, of course that would be true, but ... [neoclassical meaning supply and demand, that in larger cities Walmart's share of demand would be relatively small, its share and affect on wages and employment would be relatively unimportant] ... well I just look at the data, I don’t need a theory. You have a neoclassical theory, that’s just one theory, could be wrong, this supply and demand thing ... but you know I’ve looked at the data, and the data show that Walmart lowers wages more in large cities than in small cities. I’m on better, firmer ground. 

My first reaction to that is: Are you kidding me? Do you really think you've been able to hold constant the other things you need to hold constant in evaluating Walmart's wages? It just seems like an absurd claim to me, but that’s not compelling as a counter-argument, right. It’s a difficult argument to sustain.

Roberts (and Hanson) on Truth and Economics (EconTalk) (timestamp 14:30)

Theory, well-constructed, encapsulates the entire life experience of the individual and/or profession. It is the summation of many hundreds of studies, experiments, lectures, readings, etc. It’s a very solid structure that has high explanatory (and predictive) value for many phenomena seen in the world. Individual studies confirm or deny the theory, but the vast predominance of evidence points toward its validity.

Beliefs are always harmonized. We have beliefs that come to us from the secular world, from our spiritual nature and from our social environment. We daily are bombarded by innumerable ‘signals’ that are filtered through our beliefs, and as we mature begin to suspect there is a consistency in message that spans all realms of belief. Changing one firmly-held belief entails revisiting the others. We are one person, with a many faceted view into the worlds around us.

We hear about revolutions in beliefs: shifts in paradigm. These are rightly viewed as revolutions given the deep roots many beliefs hold in our conscious and subconscious. The soil gets heavily disturbed when they are uprooted. But it should be that way. It’s our means for social organization. We build solid foundations to withstand the buffeting and turmoil of the charlatan and opportunists. Our belief structures must be enduring or we risk being torn down by that next so-called revolutionary (i.e., the demagogue+).

When we’re honest we are constantly challenging our beliefs. Why are they allowed such a gold pass to our assent? We know that beliefs are human constructs and as such are subject to constant revision, and even upset. So we’re always on the alert for any signals that indicate a change may be on the way. We welcome dissent. We welcome dis-confirming evidence+. We are always on our guard against our human weakness for retrospective distortion+: fitting past facts into our present understanding of a situation. Our beliefs are strong because we ourselves are constantly subjecting them to very severe tests, both in our thinking and in the individuals with whom we surround ourselves.

So along comes our interloper with his quantitative charts from some recent study. They point in exactly the opposite direction of our beliefs. The audience, the non-believers, have two choices. They can accept the results of the study, which are difficult, but with a little coaching are accessible to rational consideration. Or, they can accept the strength of your beliefs, as you profess them. The audience has experienced many false beliefs, and must have a reason to see yours as different from all the earlier false beliefs. The rational self sides with the study results. All you have to offset this ‘rationality’ is an appeal to beliefs on their side in the value of tradition and consistency: you must appeal to their conviction that extraordinary claims require extraordinary evidence, and that you stand on the side of the ordinary.

Who’s going to win? Often the interloper. Study results, even if they’re complete hooey, beat out theoretical arguments in public debates. The interloper points to numerical comparisons that shows product A is better than product B. The believer must appeal to esoteric arguments like Simpson’s Paradox+, or plead for the audience to remember that numerical associations do not mean causality. The interloper simply trots out disfigured children and societal wretches who would be cured or fixed by product A. The believer must argue these individuals are not representative of consumers of product A, that individuals switched from product B will be worse off, and that the interloper is merely playing the victim or pity card. It’s theory versus gut-wrenching theater. The believer hasn’t a chance.

It gets even worse in cases of latency. Our long term maintenance of a smooth and functioning marketplace are challenged by interlopers who claim that market fundamentals have changed and that protections are old and in-the-way of the ‘new economy’. The more the interlopers have their way, the riskier the situation grows. In the meantime the interlopers are making money hand-over-fist. The cautious, the believers in market retribution, are falling behind. They more and more look like the child crying wolf, and fewer and fewer in the audience will listen. But eventually the wolf does appear at the door: but the interlopers have already exited with their fortunes.

The interloper's favorite scenario is one where short term gains from discarding long-held beliefs are significant, and long term risks are difficult to measure. We play the game, reap our short term profits, and exit before the house of cards collapses. The theory was sound, in that the cards do eventually collapse, but theory was not deeply or broadly held by the paying public, and so was allowed to be submerged in the interest of short term individual gains.

We hold beliefs as to what is necessary and sufficient to achieve World Class R&D stature: the discontinuous nature of the pursuit of blockbusters, being viewed as a valuable partner to the rest of the corporation, etc. But we recognize that interlopers don’t care about long term success or R&D continuity: they care about immediate gains that translate into personal enrichment. So built into the tenets of World Class R&D are proactive protections against the interloper. We build institutional structures to make it tough for the interloper to play his or her game. We are constantly vigilant in our watch over non-believers, keeping our eye on them for their next clever attempt at personal enrichment at the expense of the long term vitality of the enterprise.

Nothing succeeds like success. Our best defense for R&D is outrageous progress toward blockbuster products. World Class R&D builds institutional structures designed to get the best out of our R&D leaders, and as a consequence out of those being led. We protect these structures against the outside interlopers in the inherent design and operation of the structures. Attempts to subvert the structure autonomously trigger overwhelming financial downsides for the interloper. R&D demonstrates great responsibility in its pursuit of blockbuster revenues for benefit of the corporation. R&D rightly expects great responsibility on the part of the corporation in its respect of the needs for continuity of R&D