Rigorously controlled scientific experiments often do not stand the test of time. Even interocular+ results are later found to be not replicable.

A recent article in the New Yorker magazine, The Truth Wears Off discusses how findings from rigorously controlled studies, repeated years later by the same researcher sometimes are not replicable. This happens in medical research, behavioral research, and ecological studies. Mostly in the life sciences.

Regular readers of World Class R&D articles will recognize the pattern. It has been discussed in many articles:

Many research examples cited in the New Yorker article were seemingly statistically solid — that is, they contained enough data so the effect due to ‘statistical fluctuations’ shouldn’t be as dramatic as those later found. But results were not replicable none-the-less.

The conclusion of the article is startling:

We like to pretend our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.

What this means for industrial R&D is that in order to commit to a line of inquiry we must first build a robust body of evidence+, one in which we can rip out (often compelling) results of any one or few studies and the conclusions still stand.1

Editor's Picks for September, 2011

  • 1. That is, we develop a robust body of evidence only after doing enough exploratory work to convince ourselves we are headed down the right path (our Skinny Path analogy). This article applies for our internal truth-seeking. For external consumption we rely on narratives and tiers of evidence+.
Further Reading
Reba Tull
Joined: 03/30/2011
It's just one study

This is one study (article) that comes to a very substantial conclusion (i.e., that scientific method doesn’t work). I’ll wait for a larger body of evidence+ better supporting this conclusion.