Independent Evaluation Defined

Independent Evaluation is an academic discipline with its own special tools, techniques and rigor, dedicated to evaluation of the progress of teams. Different evaluative techniques apply in different situations, and the trained independent evaluator+ understands these differences. Evaluation of team progress in creative endeavors, under great uncertainty, is a very special situation, and one in which Independent Evaluation has much to contribute. It is challenging or impossible to parachute in at the end of a long project and perform an assessment that captures all the nuances and challenges faced while the team worked on the original project goals. The independent evaluator being on-the-ground gives us these insights and allows for a more realistic assessment of interim progress.

Independent Evaluation finds its home at the American Evaluation Association (AEA) – an organization dedicated to the continuous improvement of this discipline. The beauty of this is that the academic heavy lifting has already been done: we merely need to adapt the findings from the AEA into the R&D setting. When you dive into the discipline of Independent Evaluation you find that this organization has made great progress in improving the quality and usefulness of evaluations – for example their insights on the limitations of goal-oriented evaluation and its relation to project execution. Much progress has been made on evaluative techniques that ensure final results are not due to compromises stemming from poor execution (i.e., settling for whatever results you can salvage from the train wreck) and that good processes aren’t leading to bad or no results. The mission of the AEA is to improve the evaluation process itself – to find the best formula for evaluations given the challenges the teams face. Given the success of the AEA over its 25 year history, our task is merely a matter of translating its findings into the needs for interim team assessments, which I start in this section.

When we talk about independent evaluation we’re talking about a role similar to a general contractor. This is the professional that stands between the owner-funder of the project and the tradesmen. The general contractor has a good relationship with the funder (i.e., the owner) and with the tradesmen. He or she is the conduit of trust between the two parties, smoothing out any minor differences that may escalate out of proportion in the mind of the funder, which then gets in the way of the passions and motivations of the tradesmen. The funder knows the contractor can faithfully represent his or her interests. The tradesmen know that the contractor understands the difficulties and the moments of creativity in the trade. The general contractor works with the passions of the tradesmen, allowing them to move beyond mere specifications, but always within the boundaries of the funder’s expectations. Similar to a construction contractor, it is the job of the independent evaluator to mediate between the two parties to avoid misunderstandings and to foster new approaches.

The beauty of the general contractor model is that it allows evaluation at a distance – arms-length+ evaluation: personalities get removed. It eliminates the need for the owner to be on site every day and yet gives the owner constant assurance that his or her wants are being cared for. The owner can step back from the detailed work and make objective decisions about the overall progress of the project. I have worked with this contractor before and I can rely on his judgment to evaluate the quality of the work of all the tradecrafts, and especially on how all the trade-crafts will come together as a finished product. I have much less a chance of being snowballed or being the victim of early goal-padding+. I reduce the chance of falling into the trap of the confirmation error+, where I only see evidence that confirms my preconceived notions about the competency of the team. The general contractor model reduces confusion from personal ties, animosities, and biases that can results in a direct relationship between the owner and the tradesmen. Finally, I am no longer forced to discount the implicit information that constitutes the bulk of the understanding of the teams. As the owner I can now judge the quality of work based on an objective, third-party assessment that is under-the-skin, so-to-speak, of the team being evaluated.

It’s the job of the general contractor to make sure the tradesmen are doing the best they can with the resources they have. We want to delight the funding agent+, to far exceed their expectations, but not at the risk of falling behind in our schedule or exceeding our budget. Or, if unexpected obstacles or opportunities present themselves, the general contractor provides a trusted conduit back to the owner to request more funds. The general contractor performs a daily subjective assessment of the work of the tradesmen, balancing their desire for creativity with the owner’s need for commercial urgency. World Class R&D is replete with uncertainty and unexpected findings. The general contractor steps into this uncertainty, and has the relationship with the funding agent to make sure that when the team deals with the uncertainty it is not perceived as mere sloppy execution

Why not just use the manager as the general contractor? After all isn’t that why we pay our managers? The clear advantage of Independent Evaluation is in its independence. As manager of this team my fate lies in the success of the team. As an independent evaluator my fate lies in the accurate assessment of the success or failure of the team. In purely mechanical activities, where there is less ambiguity on the assessment of progress, there is less need for an independent evaluator. But in creative environments this independent role is imperative. R&D managers are infinitely creative in performing their own self-assessments. The manager may go straight to the owner to plead his or her case. But in a well-designed decision mechanism, the independent evaluator stands in for the owner, and the manager will be politely rebuffed. The independent evaluator is responsible for judging the overall success of the manager and his team in meeting their goals in the face of uncertainty. The independent evaluator makes sure that when the team is off course, it is for legitimate scientific and/or commercial reasons, and not merely due to poor navigation skills.