November 24, 2014 •Natalie Patten
The beginning of the research and evaluation process can be daunting.
You know the topic or the question you are trying to answer, but maybe little else. You might begin by Googling your question. A Google search of “How effective are labor market policies?” produced about 5.4 million results. So, you might conduct a formal literature review and narrow your search to previous research studies.
Done correctly, a literature review provides context. It reveals what others have done to puzzle through similar questions or challenges. Done incorrectly, a literature review can confirm preconceived beliefs or give high and low quality studies equal weight.
When beginning the process, you may want to search for or conduct a meta-analysis: a special class of literature review that quantitatively synthesizes the results of many studies.
What is Meta-analysis?
A meta-analysis is considered the gold standard of literature reviews. It is a method to systematically review experimental and quasi-experimental studies with similar research topics. It can increase the validity of the literature review and can inform a study’s design and specification. David Card, Jochen Kluve, and Andrea Weber (2010) performed a meta-analysis on the effectiveness of labor market policies. Rather than sifting through 5.4 million Google hits, they developed a comprehensive sample of evaluations within a given time period (1995-2007) of all academic researchers affiliated with two leading research networks. Through a quantitative analysis of these studies, they were able to determine types of programs and policies that have short- and medium-term impacts on certain employment outcomes.
| In experimental and quasi-experimental studies, researchers assign participants to receive treatment or they otherwise structure conditions to determine whether a treatment has an effect on a specific outcome. Experimental study design requires random assignment of the treatment and control groups. |
Meta-analysis vs. the Single Study
A meta-analysis can be used to review the literature for a study or to answer a research question by itself. There are many benefits to conducting a meta-analysis as opposed to a single evaluation, including:
- A sample of many studies can:
- Balance out the measurement errors of individual studies, and
- Reduce type II errors (false negatives) by increasing the sample size of observations and improving the power of the study to detect smaller effects.
- It has external validity. The results of single evaluations on specific job training programs often cannot be generalized to other training programs. The meta-analysis can show the effective (and ineffective) components of job training programs as a whole.
- It can account for different study designs (randomize control trials (RCT) vs. quasi-experimental), different independent variables, interactions among independent variables, and even different outcomes. Card et al. found that program impacts were more likely to produce positive results for studies with registered unemployment as the outcome, rather than studies on employment and earnings.
- It can control for contaminating factors and other threats to internal validity. Studies with flawed specifications do not have to be excluded, and information can still be gleaned from them.
How Can We Use It?
Agencies and organizations wanting to answer questions about the effectiveness of policies and programs based on previous research can find value in conducting or sponsoring a meta-analysis on existing studies. We at Summit are adept at conducting this type of literature review to synthesize effects from multiple studies and identify factors that are consistently associated with outcomes.
This post was written with the help of Dr. Ed Dieterle.
