Laboratory vs. Field Studies

Controlled variation is the foundation of empirical scientific knowledge Falk, A., & Heckman, J. J. (2009). Lab Experiments Are a Major Source of Knowledge in the Social Sciences

My first reaction to that is: Are you kidding me? Do you really think you've been able to hold constant the other things you need to hold constant in evaluating [this question]? It just seems like an absurd claim to me... Roberts, Russ (2009). Roberts (and Hanson) on Truth and Economics (EconTalk, timestamp 14:30)

We need to generalize our research to get blockbuster+ products. We build our products to attract the broadest possible consumer population. We test our generalizations in studies by holding constant many of the consumer variables, but are fully aware we often miss some pretty important ones (e.g., unk-unk). We run empirical (hands-on) studies to test the boundaries of our generalization, but we know the data coming out of the studies is tainted from both the design and the execution of the study. No study can be free from our beliefs and preconceptions (see Feyerabend), and yet we are obliged to move ahead.

Do we use laboratory or field studies? Much literature in the behavioral sciences tends to pooh-pooh the distinction, including the article from which the first citation above was extracted. These authors reach this conclusion because they leave vague the definition of a ‘field study’. For example, clinical trials in the pharmaceutical industry are in effect laboratory studies. It doesn’t matter whether or not you go to the patient or the patient comes to you. Post-marketing surveillance, watching the population of users after a drug is on the market, is an example of a field study. There is a qualitative difference in the data you get from each of these studies that no amount of hand waving will make go away.

McDonalds Corporation knows, nationally, how many more orders of french fries it will sell when it uses a 2-for-1 promotion on its hamburgers. They know this by region, by season, by the timing of the promotion with major sporting events and by many dozens of other important variables. It knows this because it conducts field studies. They don’t recruit volunteers and run them through controlled studies. The cash registers in each restaurant (and the advertising campaigns in each region ) are hooked up to teams of analysts with their quantitative models at national headquarters. They simply run the test: a field test.

 

Field studies trump laboratory studies, but are often quite difficult to run for innovative products. Laboratory studies trump theorizing. The human mind is too prone to false analogies without the tether of hands-on results. The theory-free empirical study is a myth: your theories (i.e., beliefs) are merely masked behind some façade of objectivity. You need all three: field, laboratory and theory. We stress-test our conclusions in that one or few of our study results or postulates are shown to be false and yet the overall body of evidence+ remains robust. There is rarely one overarching study, theory or result that clinches the case. It’s the overall impression from all the evidence, including the doubts from a few isolated results, that gives us the confidence to push forward.


Editor's Picks for October, 2010

Further Reading