Meta Analysis

Description Meta analysis is the practice of combining results from across several studies thereby deriving additional research insights that were not visible by looking at each study separately. For example, researchers and trial lawyers combine clinical trial results across many separate studies using drugs in the same class and draw conclusions about the safety or efficacy of the class of drugs.
Weaknesses

This practice leads to false generalization+ of the worst kind. A doctor from a prestigious cardiac center publishes a meta study showing that COX2 inhibitors (drugs that target a specific enzyme) cause heart attacks in 2% of patients. If you're taking a COX2 inhibitor and you see this study, you may naturally decide to stop taking the medication. After all, did the company that manufactures your COX2 inhibitor run a study (or can they even run a study) that shows their COX2 inhibitor does not cause heart attacks?

Trial lawyers, of course, look for anyone who has had a heart attack while taking the medication. The doctor from the prestigious cardiac center has a study filled with charts and numbers, and all you have is an academic dissertation on the flaws of meta studies. The trial lawyer will parade heart attack victims before the jury and all you can find are academics who discourse on the evils of meta analysis. Who do you think is going to win the trial?

A great danger of meta analysis is that they can be powerful tools in the hands of the demagogue+ or the self-interested. These individuals are quick to discard results that are unfavorable to their agenda, and are quick to ignore their own biases in the analysis of results that are favorable. The reports generated by these agenda-driven individuals have a patina of authenticity: after-all they are merely a reshuffling of results that have already gained acceptance. The opponent to these reports is placed in the disadvantageous position of having to argue about obscure fundamental flaws in the meta-analysis approach, while the demagogue argues from "the facts."

Appropriate Uses

Looking across studies to draw conclusions is fine. You voice your opinion on what this body of evidence+ tells you about the attractiveness of the research pursuit.

Using quantitative analysis across studies to reach conclusions should be banned - individuals presenting these false reports to decision makers should be ostracized. These individuals wreak immeasurable damage on the effectiveness of research as everyone scrambles to undo the misconceptions these reports place in the minds of decision makers.

Quantitative analysis leads individuals down the path of false generalizations. For internal research and decision making, we need to be ruthless in weeding out individuals who pretend to add knowledge through mere mathematical manipulation of pre-existing data. Mathematical manipulation is fine for uncovering new hypotheses, but not as a fount of new knowledge (nor even as a means to prioritize further research).

 

Further Reading