Quants Inadmissible

Our client lived in Ensenada. The firm where he worked was headquartered 1 hour to the north in Tijuana. My boss came to me with the following instructions:

Pull together an analysis that shows why this project should take place in Ensenada. Come up with ten or so criteria, so when we apply a weighted average to each criterion the summation will point to Ensenada. Try not to make it too obvious. Management Consulting Partner (~1991). Personal Communication

It was easy.

Another client needed a large project to be in charge of, in order to stand out from her peers in the race for a top position.

Pull together an industry survey that will show how our firm is at the bottom of the pack in terms of Customer Service. Management Consulting Partner (~1994). Personal Communication

Again it was quite easy, and the sole basis for a $50 million assignment that achieved the goals of both the client and the consultant.

Quantitative analyses are extremely compelling. Few stop to question the underlying basis for the analyses. Fewer still understand the how the numbers mask the underlying causality of the question at hand (i.e., through false generalization+). People mostly assume numbers don’t lie.

Quantitative analyses do perform a valuable social function in that they liberate and unburden the individual conscience and smooth our social interactions. They allow us to suppress awareness of 'the lie' that gnaws in the back of our heads. Our daily routines run with much less friction and energies. We don't need to stop and think about the how of our work -- it's handed down to us through ...

Our client lived in Ensenada. The firm where he worked was headquartered 1 hour to the north in Tijuana. My boss came to me with the following instructions:

Pull together an analysis that shows why this project should take place in Ensenada. Come up with ten or so criteria, so when we apply a weighted average to each criterion the summation will point to Ensenada. Try not to make it too obvious. Management Consulting Partner (~1991). Personal Communication

It was easy.

Another client needed a large project to be in charge of, in order to stand out from her peers in the race for a top position.

Pull together an industry survey that will show how our firm is at the bottom of the pack in terms of Customer Service. Management Consulting Partner (~1994). Personal Communication

Again it was quite easy, and the sole basis for a $50 million assignment that achieved the goals of both the client and the consultant.

Quantitative analyses are extremely compelling. Few stop to question the underlying basis for the analyses. Fewer still understand the how the numbers mask the underlying causality of the question at hand (i.e., through false generalization). People mostly assume numbers don’t lie.

Quantitative analyses do perform a valuable social function in that they liberate and unburden the individual conscience and smooth our social interactions. They allow us to suppress awareness of 'the lie' that gnaws in the back of our heads. Our daily routines run with much less friction and energies. We don't need to stop and think about the how of our work -- it's handed down to us through convention and tradition. But the essential person, the you, suffers existential damage. The lie is still present.

Mega-Phase III clinical trials in the pharmaceutical industry are huge number-crunching exercises. These trials are false – delusional to be precise. The delusion of these exercises has already been exposed:

We get 300% variability due to the initial conditions of the study, up to a 20% variability due to the placebo effect, and unknown variability due to the mystery of compliance. With Simpson's Paradox+ we can completely reverse any comparator study result with very little effort all the way up to a 40% difference. Narrow-minded individuals think they can fix these issues and salvage the exercise, but the issues are only manifestations of a fundamental flaw of using comparator studies as arbiters of truth in the life sciences. There is no reason to believe that humans in clinical trials should behave similarly to the roll of the dice (i.e., the foundation of statistics). We blaze a new path in World Class R&D.

The cynic could view this big government barrier-to-entry as a perk for big, well-financed pharmaceutical companies: it forces small companies to sell them their research as the only way to get products to market. Only the biggest firms can afford the expensive Phase III clinical trials required by the FDA and its international affiliates. Lawrence J. Lesko of the Office of Clinical Pharmacology, FDA in his 2006 DIA (pharmaceutical industry confab) presentation "Keeping Phased Development, but not the Lessons Learned, Is a Bad Idea", admitted he was unable to find where the whole concept of Clinical Phases originated. He opined it was an industry strategy that evolved from regulatory requirements over time. Doesn't the fact that almost all of these large trials are farmed out to the lowest-cost service provider speak volumes for the importance we should attach to them as a arbitrators of truth?

We’re dealing with drugs that work only in some of the people! We need Quants to tell us how many people we’re talking about. No! In World Class R&D we seek interocular+ differences. We don’t need comparator trials or large enrollments in our trials to know our drug is outstanding for many patients.

What about safety? That’s the value the Quants bring to the table. They’re the only ones able to tease out how many patients will suffer adverse effects once our drug is widely distributed after FDA approval. No! It should be quite clear that hiding behind a veil of FDA-approved numbers is a charade used by companies to forestall lawsuits. And this charade has cheapened the very profession we should be depending on to keep an eye on drug safety.

What about risk-benefit trade-offs? All drugs have risks. Even if your drug is deemed effective, I need assurance its additional benefits offsets any increased risks over existing therapies. Let’s take the case of Lipitor™. The ‘numbers’ from the clinical trials showed it could be sold at a lower dosage as a safer alternative to Merck’s Zocor™ or it could be sold at a higher dosage as a more effective treatment, but with the same level of risk (i.e., side effects). Never mind the inconvenient fact that lowering cholesterol below certain levels was of dubious clinical benefit, and indeed we're conflating HDL with LDL in this narrative.

It’s clear there was no calculation of the medical benefit of lowering cholesterol verses the higher risk of side effects. The calculation instead, was how much more could we sell of a more effective treatment versus a treatment labeled as having lower side effects (in a very small subset of the population, remember). We merely upped the dosage until enough patients complained of side effects – and in this case it was a dosage that dramatically lowered cholesterol (an interocular improvement). Never mind whether or not dramatically lowering cholesterol means anything for patient well-being. Risk-benefit trade-offs suffer from all the flaws described above for number-crunching exercises.

There is one legitimate argument for the use of Quants in the pharmaceutical (and medical) industry. Progress is often made in small increments. It's difficult, or impossible, to measure whether or not a new approach improves on the old. We only know over time after the new approach is tested in the market. So instead of trying to know ahead of time, we merely put in place very expensive hurdles to weed out the two-bit charlatans. We were successful in getting rid of patent medicines weren't we? Really? Quantitative Analysis is not so much an arbiter of truth as a protection against purveyors of lies.

The answer to the challenge of the Quants is twofold: we don’t fool ourselves; and, we get much better at fooling individuals who insist on these analyses. Internally, the Quants are barred from the decision making process: their evidence is declared inadmissible. Further, we put in place strict policing mechanisms to ensure they do not attempt end-runs around the restrictions.

Externally, we determine the best way to game a fraudulent system. We use the time-honored practices of patient or investigator enrichment. We use adaptive trials to tack and turn based on interim results. We subdivide adverse effects into finer and finer subsets of clinical diagnoses so there are no aggregate numbers that stand out. We make it clear to FDA investigators they might want us to keep out a lucrative welcome mat should their fortunes at the FDA wane. We staff the upper echelons of our statistics department with the best mathematicians from Reno and Las Vegas, back-filling lower-levels with bio-statisticians, for appearance sake.

Why is this important? Because this one small conclusion, that statistics has no place in decision making process for drug approval, will give an order of magnitude increase in R&D effectiveness (and patient safety) in the pharmaceutical industry. Instead we build a new tradition, one in which the conscience is liberated and unshackled in its disrespect for most quantitative analysis. Given the entrenched interests of the quants, that stretches from our industrial laboratories up to the American Medical Association and most large health-care payors, we use this new tradition not as an operating model, rather as freedom to disbelieve without moral hindrance. The priesthood will object, and we pay them the proper homage, but inside, where our creative spirit hides, we are now liberated from a large set of false beliefs in our search for the next blockbuster+ drug.