Quality and Effectiveness

How does quality assurance contribute to the effectiveness of R&D? At first blush, it would seem very little. Quality assurance is mostly about making sure the work we do is done well: a hallmark of efficiency. We want to avoid having to repeat shoddy work. It would seem that quality assurance has little to say about the direction of the work: are we working on the right things, aka effectiveness?

However, if we define effectiveness in terms of potential revenue for our product, as in the ability to achieve World Class R&D revenues, then quality can be viewed in a different light. We can view Quality Assurance in the same light that we view Product Safety. If the product is not perceived as safe, then this impacts sales. If the product is not perceived as being of high quality, then this can affect sales. So we’re not as much concerned with process quality, regulatory penalties and recalls, except to the extent these reflect on product quality, which shows up in product revenues. Note: you are not allowed to play the Time-to-Market+ trump card in this game. It’s been tagged as a Worst Practice.

We’ve also tagged Quality Hurdles+ as a Worst Practice in World Class R&D. It’s not that we’re unconcerned about the quality of our evidence; it’s rather that officious people think they can measure and improve the quality from the outside. We instead advocate that quality be internalized into each study and researcher. Individuals care about the quality of their evidence because they recognize that poor quality evidence is tantamount to counterfeiting, and that having the reputation of a counterfeiter will be very hard to overcome in future transactions. All our research transactions (i.e., evidence in-trade for decisions) will cost us more dearly from that point on.

In a competitive environment quality matters. Our evidence is essentially in competition with that of our neighbor team. If I practice poor quality, then I potentially (and will eventually) lose funding for my research. When I ask that my prototype+ be graduated into commercialization+ (i.e., Phase III in the pharmaceutical industry), the first thing that the prosecution will review, and will be sure to expose to the jury, is evidence that it is subpar in quality. In a competitive environment I care personally about the quality of my evidence: there is no need for an outside quality specialist or a quality hurdle.

So if we’re in Quality Assurance, that is, we practice quality for the sake of quality, then we most likely side with the prosecutor. On the side of the defense, we’re much more concerned about effectiveness in evidence gathering. Effectiveness means that we don’t care about quality per se. We mostly care about it to the extent it needs to be defended in a string of studies and experiments we will trade as part of our research transaction: that which we’ll trade for a favorable decision.

We’re looking for the skinny path, and if we skip a few ‘quality steps’ along the way, then it doesn’t matter. We’re not sure this particular study is even going to end up on the path for our final product. So instead we run our studies quick and dirty, first to see if we’re headed in the right direction. This is not carelessness, but it’s also not being tied to an artificial generalization+ of what constitutes quality. In the end we step back from a string of such studies and ask toward which direction they point. If one or more of these studies is viewed as crucial to our evidence trail, then we replicate them, probably using much more rigorous quality controls.

There is always the issue of having researchers that are ignorant as to the practice of quality. Quality is a discipline with its own special tools, techniques and perceptions. Similar to our concept of the compulsories+ for effectiveness in evidence gathering, we need assurances that individuals are knowingly stepping outside the conventions in their quality shortcuts. They have demonstrated a thorough understanding of common quality controls and have determined to skip them in the name of effectiveness. Show us you know the rules before we allow you to break them.

We need to be able to trust our results, especially for our internal decision-making, regardless whether or not this particular study makes it into the trail of evidence supporting our final prototype. Evidence is just as effective if it tells us where not to go as where to go. It’s effective if it reveals what we don’t know, or what we don’t need to know. It’s effective if it allows us to choose from amongst different avenues of research. In summary, we need to be able to trust what each and every piece of evidence is telling us, regardless of its final disposition. This happens only when we are assured our researchers are knowledgeable in the special tools and perceptions of Quality Assurance.

We eschew cheap heuristics+ in research. We no longer hide our ignorance of errors, biases, placebo effects and self-interest behind a veil of numbers. This means we put in place deliberate means to avoid these faults: independent replication of key studies, automation, video-taping, compliance containers and a host of other mechanical and procedural protections. We’ve seen the story of the clicker-counters gone bad: the prosecutor has access to the same list of potential study faults as our research teams, and will not hesitate to publicly disparage study results that do not incorporate deliberate protections.

As with much else that we propose in World Class R&D, there is no one-size-fits-all approach to quality. There is no master checklist; no quality system (except where mandated by law). We need to be reasonably assured we can trust interim results. We need to be very assured our final results are not open to an attack on their quality by a determined adversary. These ‘quality goals’ are very specific to each study. We intend to build measures of effectiveness into each and every study (see here). Our internal quality controls will cover these. These are of much less interest (or use) to an outside prosecutor.

To come full circle, in World Class R&D we practice quality control to the extent it contributes to effectiveness. Quality control is an additional step: additional work. By definition this means it is biased against effectiveness. But from the arguments above, there are many specific cases where this additional work can be reasonably viewed as contributing to effectiveness, and that’s the crucial connection.

Joined: 06/05/2010
Has nothing to do with human medicine

1. The FDA bases its reviews on risk management. Risk to patients is the bellwether for decision making by the agency. That which puts human patients at undue risk is not acceptable, hence the clinical quality requirements of the IND and the form FDA 1572.

2. Studies for which there is little utility outside of “nice to know” or “confirmation” or “proof of concept” are only acceptable where there can be statistical evidence generated.

3. The old models of giving amounts of drugs to few patients to see whether the concept works is no longer an acceptable one as it puts those patients at risk of (a) losing needed therapy and (b) experiencing serious adverse effects from questionable therapies.

4. Similarly, doing animal tests in a few animals is acceptable when you are in the clinical pharmacology phase. Safety trials need to be conducted to the same exacting standards as human studies do (21CFR58) so that the findings can be statistically analyzed.

5. Using the model as suggested in the quality article places humans in these small studies at the same level as laboratory animals, something which no IRB would accept, and which FDA would definitely have problems with in an IND. European countries which would allow such studies run the risk of FDA denying their studies of a larger nature being accepted by FDA in support of a larger submission.

6. This document makes questionable statements, e.g. “there is no quality system” – we have a number of sections of the code of federal regulations which defines quality systems, especially the 800 series for medical devices; there are the GMPs (21CFR200 series), the GCPs (Q6 of the ICH and 21CFR50, 56, 312.50-70), and so on. We cannot appear to put ourselves in that ethereal intellectual plane which gives a sense of “I am superior to all because of my great intellectual achievement.” While the role of R&D researcher is a good one, and the needs for great R&D should be run "out of the box" as it were, articles such as this should not give the impression that R&D even in an out of the box scenario puts the patient at risk. On the other hand, doing things out of the box for non-human endeavors should be aggressively pursued, so long as the resulting product is not later questioned by the CPSC or OSHA.