Methodological Challenges Associated with Meta-Analyses in Health Care and Behavioral Health Research
By Judith A. Shinogle, PhD, MSc
I. Executive Summary
Meta-analysis is used to inform a wide array of questions ranging from pharmaceutical safety to the relative effectiveness of different medical interventions. Meta-analysis also can be used to generate new hypotheses and reflect on the nature and possible causes of heterogeneity between studies. The wide range of applications has led to an increase in use of meta-analysis. When skillfully conducted, meta-analysis is one way researchers can combine and evaluate different bodies of research to determine the answers to research questions. However, as use of meta-analysis grows, it is imperative that the proper methods are used in order to draw meaningful conclusions from these analyses. This paper provides a framework for determining whether proper methods were used and thus for evaluating the quality of meta-analyses.
Meta-analysis refers to a joint analysis of at least two primary datasets after each has been initially analyzed and published. No new data are produced in a metaanalysis; it is a retrospective study of previous research. The growth of metaanalysis in recent years is due to many factors including an increasing number of systematic reviews to summarize evidence-based medicine, for use in comparative effectiveness studies and the desire to improve the power of clinical trials as well as the precision of studies.
Readers and researchers of meta-analysis should recognize the statistical sophistication needed to combine measures from various data and populations. The methods of meta-analysis should follow the same steps as any well thought out research project – development of research question; systematic review; and collection, evaluation and summarization of data. Each stage should be clearly thought out, and each section well documented. Standards for reporting metaanalysis exist and should be utilized by researchers and readers of studies to evaluate the quality of the study.
A well designed and implemented meta-analysis study can be a great benefit to society. When conducted appropriately, it can reveal the weaknesses and strengths of previous studies, present needed answers to difficult research questions and even provide suggestions for future research. A meta-analysis study can summarize from available studies the effects of interventions across many patients. It can hasten introduction of effective treatments into clinical practice by reducing 4 false negative results (Type II error). Further, it can determine if the effect of an intervention is sufficiently large in practical, as well as, statistical terms.
While meta-analysis has the potential to be a powerful tool in evaluating health care treatments and interventions, there are many potential pitfalls and problems that are yet to be resolved. A poorly performed meta-analysis can perpetuate biases from ill-conceived studies or lead to false conclusions. This, in turn, can cause consumers and caregivers, who frequently access results of meta-analysis through websites and popular press, to form incorrect conclusions and can result in inappropriate medical decisions. While the idea behind combining studies to improve precision and power is straightforward, the actual implementation of the process is difficult. Those who act or react based on meta-analysis should understand the various biases that could be incorporated into a review. Examples of some potential pitfalls in meta-analysis include publication bias, pipeline bias, and English language bias. In addition combining studies that are not similar in study design, population, methods of analysis or outcome definitions can lead to biases as well, which may result in spurious conclusions being drawn.
II. Introduction
A. Overview of paper
Meta-analysis is a commonly used methodology to combine findings from several studies in order to improve the power of these studies as well as the precision of the estimated effect. This method grew out of the psychology literature and is often used in conjunction with systematic reviews as a means to evaluate health care interventions. This paper will present an overview of meta-analysis technique, provide a description of a sound meta-analysis methodology and discuss challenges and issues involved with developing a meta-analysis study. The paper concludes with a discussion of the use and interpretation of the technique as well as questions that should be asked of a meta-analysis study. Throughout this paper several statistical terms will be used. A table from Koretz (2002) has been recreated in Appendix A and lists many of the key terms and their meanings.
B. Definition of Meta-Analysis
Meta comes from the Greek and means “after.” Meta-analysis refers to a joint analysis of at least two datasets, which maybe from published research or primary data, after each has been initially analyzed and published. The key issue is that meta-analysis is an analysis of analysis as opposed to a re-analysis of primary data. Meta-analysis is an arithmetic tool that is used in conjunction with a systematic review of the literature.
Download the full paper (PDF)
Please see http://www.umbc.edu/mipar/shinogle.php for information about the Judith A. Shinogle Memorial Fund, Baltimore, MD, and the AKC Canine Health Foundation (www.akcchf.org), Raleigh, NC.