Statistics and A>B

If I run a clinical trial and A>B does that mean that if I run the same clinical trial again then I should have greater confidence that I’ll get A>B? You’ve read the story of the flour beetle.

If we’re talking about the flip of a coin the outcome of the first flip shouldn’t affect the chances of the 2nd flip – that’s because the coin is assumed to be balanced and each flip is independently 50-50. On the other hand, in clinical trials the results of the first trial may be a hint that the coin is not balanced – there’s a chance that fundamentally A>B. Since we don’t know if the coin is balanced or not and the first flip showed that A>B we are more justified in having a higher expectation of the second trial also giving A>B – and we are justified in assuming that after a few flips of the coin with most showing A>B that the coin is not balanced and that indeed fundamentally A>B. All the statistics does for you is it allows you to do the math to see how you compare to flipping a balanced coin. It does not prove that A>B since you could have randomly flipped a long series of Heads.

Well of course it’s pretty easy to generalize that all heads are heads, and all tails are tails. It’s not so easy with clinical trials. Not all the patients are the same and the disease itself can be in different states of progression. So when I run clinical trials lots of my coins end up standing on end.

Let’s take an example from the Life Sciences. Wheat yield has steadily increased year by year over the last few decades – the annual increase is small, based on statistically controlled variations, and yet over time we have doubled yield in the field. We don’t know all the causes of the variability in yield, but we don’t need to in order to gradually achieve dramatic results. It’s the same in drug development. We don’t know all the variables, but with statistics we can achieve steady incremental improvements and advance medicine for a particular disease over time.

Yes, but that’s just one dimension of the harvest, yield. Some would argue flavor was sacrificed. A drug is just a poison that is less effective. We can make the drug more effective, in effect grow the wheat taller, by increasing the drug’s potency. Just that we happen to be poisoning some of the patients also. That’s a key difference between humans and wheat. Wheat can be cloned. Also, we have very small numbers of wheat stalks in which the fertilizer now has serious adverse effects. You wouldn’t notice these stalks in a field of wheat, but the Trial Lawyers would notice it in a group of patients.

Further Reading