Measuring Interim Progress in R&D

Consider my conversation with the head of R&D at one major pharmaceutical company:

Here are my annual goals. This book lists all the projects we're working on in R&D. Each one has the name of the Vice President in charge. There are hundreds of projects. I have a personal goal to ensure each project moves from its opening position to the next milestone. I have three criteria: met goal, exceeded goal, or missed goal. If a project ends up two years in a row in the missed goal category I fire the Vice President in charge, but only after he or she has fired all the members of the team. Head of R&D for Major Pharmaceutical Company (2000). Personal Communication

When I first heard this (goal oriented evaluation) approach I was impressed with the clear line of responsibility that stretched from the Head of R&D down to individual team members. Upon further reflection, though, it was apparent this evaluation approach was not tenable over the long term. It was an exercise in deadline bias+ at its most extreme. It assumed a mechanical, linear, step-by-step approach to R&D and left little room for discovery as the project progressed. It assumed goals could be unambiguously defined at the beginning of the project. As a team member I’m highly motivated to find a legitimate scientific or clinical excuse to kill this project before that next milestone deadline.

R&D for innovative products should be an exercise in creativity, tacking & turning, exploration of many possible opportunities and being prepared for luck to happen unexpectedly. Teams should only be assessed, in the end, on their achievements in science and commercial pursuits, but these should be adjusted in the face of the opportunities and obstacles uncovered throughout the year. We do not want to crush creativity (or to push it underground into a skunkworks+) as a result of our team evaluations.

Teams should not have their options shut down too soon. If you have a notebook with all my milestones spelled out for the coming period, then the game is already lost. For a while I’ll play the game and mechanically move towards the next milestone. Eventually when I get the chance I’ll just leave.

Instead we should look for teams to open up new options. They should not settle on any particular option too soon, rather they should head into the market testing multiple alternatives. We eschew evaluations that focus on individual alternates, and instead assess teams on their progress and sense of commercial urgency across a broad set of activities. We applaud teams who practice evidentiary techniques designed to open up options, such as Alternate Competing Hypotheses+ or Domino Causality+. We encourage teams to identify, evaluate and to close-out particular alternatives, not as an activity in itself, but rather as a sign of good stewardship of our funds.

Measuring interim team progress in creative endeavors is not a matter of setting goals and then slavishly measuring the team’s progress towards those goals. Management does not have a clear picture of the goals at the start, and teams have an imperfect understanding of their ability to achieve any goals. R&D is many times all about designing experiments and studies just to be able to ask the right questions. If you set a goal on Jan. 1 and the team drives inexorably towards that goal then you’ve merely set yourself up for mechanical behaviors+: I’ll discount unexpected findings, and I’ll perfunctorily address unexpected obstacles. In the end I’ll achieve the artificial goal, because I was complicit in defining the goal in a manner that allows me to achieve it regardless of the intrinsic value of the research effort.

As a funder I need defined goals to be able to justify my up-front funding of the team, but truth-be-told in the end I’m more interested in results. Convince me you had my best interests in hand when you decided to deviate from our previously negotiated goals (or when you decided to stick with these goals despite evidence to the contrary), and I’ll give you the latitude you need to avoid mechanical behaviors.

Measuring interim team progress is definitely not a matter of setting targets for activities and then counting instances of those activities. The team will merely get good at doing the activities, for example, running clinical trials, or recruiting patients. They will find many creative ways of assuring you, the counter, that the activities meet your predefined standards for quality. In the end you’ll end up with a collection of activities divorced from the intrinsic value of the R&D effort.

Without firm fixed goals folks declare victory regardless of the outcome - essentially it becomes political-based evaluation. This leads to dissatisfaction on the part of the funder and to even firmer goals for the next round. Pharmaceutical Industry Colleague (2009). Personal Communication

Our approach for measuring interim team progress is a blend of negotiated goals and skill in adjusting those goals in the face of continual new information. We’ll negotiate the annual goals between the funder and the team on Jan. 1 fully recognizing the imperfect understanding of both parties to the goals. We know there will be mid-course corrections, and at times opportunities or obstacles will surface that require a complete change of direction. We allow teams to tack and turn throughout the year and we track those adjustments on-the-ground to make sure they are not merely the result of poor execution, or poor contingency planning.

We expect the team to take advantage of opportunities and to overcome obstacles uncovered as a result of the scientific and commercial pursuit. In the end, we assess the goals achieved in face of the tacking and turning. Did the team use the funds in a manner consistent with the expectations of the funding agent+? If the funding agent were under the hood or in the clinic, would he or she have reacted similarly? We’re interested in an assessment that takes the negotiated goals as the starting point, and adjusts those goals to accommodate legitimate obstacles and opportunities, all without falling into the trap of political-based evaluations.

Assent and consequences are key. Assessment of team progress is an exercise in which someone else decides my fate. You need my assent for your decision, even in cases where your decision is to shut down my team. My decisions, the thousands of tactical and operational decisions I made throughout the year, will eventually collapse into a single go, no-go decision. The consequences of all my past decisions are suddenly unambiguous. So a complete decision mechanism requires my assent to someone else’s evaluation of my progress, with clarity on how all my daily decisions will eventually collapse into that key moment. Our decision mechanism(s) for measuring team interim progress must incorporate assent and consequences as key design features.

I mentioned earlier that we must not assess interim team progress using artificial milestones. Extended, this means we will not allow teams to negotiate artificial milestones as their goals for the year. I am not interested in having a team promise to advance a product from Phase 1 to Phase 2. Instead, I’m interested in promises of scientific and commercial advances that take into account the inherent uncertainty, ambiguity and tenuousness of the R&D pursuit. I want a team to tell me how much they’ll advance the commercial equation+ for the portfolio of targets, compounds and indications within their scope. I really don’t care which products they use to advance the commercial equation. I want as much creativity in team goal setting as in the actual research activities that will be used to achieve those goals.