Summit's Dr. Shane Thompson writes about the importance of administrative data in impact evaluations in his three-part blog series. In his first blog, below, he writes about transforming summary statistics into actionable evidence. Those interested in reading more about program evaluation can download Summit's latest white paper here.

Today, Summit released a new white paper on estimation designs used in program evaluation.

On May 18, 2016, the White House and Department of Labor (DOL) released a final rule updating overtime pay regulations, raising the standard salary level for exempt workers from $455 weekly to $913. Nonexempt employees are entitled to overtime pay; exempt employees are not. Effective December 1, 2016, workers earning less than $913 per week will not be eligible for exempt status, extending overtime pay protection to 4.2 million currently exempt workers. (DOL Fact Sheet: https://www.dol.gov/whd/overtime/final2016/overtime-factsheet.htm.)

Program evaluation is important to Federal Departments and Agencies as discretionary budgets diminish and departments and agencies are required to justify their programs and budgets with evidence. As shown in the figure below, impact evaluations are distinct among other forms of program evaluation.

The Government Accountability Office defines impact evaluation as:

A form of outcome evaluation that assesses the net effect of a program by comparing program outcomes with an estimate of what would have happened in the absence of the program. This form of evaluation is employed when external factors are known to influence the program’s outcomes, in order to isolate the program’s contribution to achievement of its objectives. ^[1] 

Because impact evaluations can provide empirical evidence of the causal effects of programs or policies on important outcomes, Federal Agencies often prefer impact analyses to other forms of evaluation when evaluating mature programs. In the following video, Dr. George Cave, Summit's Senior Research Fellow, walks through a generic, but comprehensive, net impact analysis flow chart.

In an earlier post, Summit’s Program Evaluation Team argued why program evaluation is important to Federal Departments and Agencies. As discretionary budgets continue to tighten and departments and agencies are required to justify their programs and budgets, program evaluation can provide the evidence needed to make informed decisions about continuing, improving, or expanding programs.^[1]

As shown in Figure 1, impact evaluations are distinct among other forms of program evaluation. They can provide empirical evidence of the causal effects of programs or policies on important outcomes. For this reason, Federal Agencies often prefer impact analyses when evaluating mature programs. In this post, we describe four methodological techniques Summit uses to estimate program impacts from administrative data blended with observational and program data.

In our previous post, Summit’s Program Evaluation Team discussed the primary considerations in assessing what kinds of evaluations are suitable for a specific program or policy. Investigating data availability and comprehensiveness is a key component of the evaluation process. Data are the lifeblood of an evaluation study. Without relevant, high-quality data, the program evaluation may have little value, even with otherwise rigorous designs and large sample sizes. Evaluators can work with stakeholders to determine the quality of the data available for analysis and set expectations on what the evaluation can (and cannot) reveal.

In this post, we discuss the prospect of using a program's existing administrative data to conduct evaluations. We describe different types of administrative data as well as the advantages and challenges of using administrative data for program evaluation.

In the first post of our blog series on program evaluation, Summit’s Program Evaluation Team explored the importance of program evaluation in relation to Federal Departments and Federal Budgets. In our second post, we provided an overview of logic models, which depict how a program or policy solves an identified problem under specified conditions. In the current post, we discuss how to select the right type of evaluation given a program or policy’s maturity.

In the first post of our blog series from Summit’s Program Evaluation Team, we explored the importance of program evaluation in relation to Federal Departments and Federal Budgets. In this second post, we provide an overview of a fundamental activity associated with program evaluation: the construction of a logic model. A logic model depicts how a program or policy solves an identified problem under specified conditions.[1]

The process of developing a logic model helps evaluators and stakeholders (i.e. individuals or organizations that have a stake in a program or policy) describe a program or policy clearly, agree on key terms and assumptions, and identify outcomes and impacts.[2] The process involves a trained evaluator facilitating one or more guided conversations with stakeholders to help develop the logic model.

Figure 1 presents the generic logic model we use as part of our evaluation practice, describing each of its major components. Once developed, the model helps evaluators articulate evaluation questions and select appropriate methods to answer them. Additionally, it helps evaluators determine what to measure and how and when to measure it. We include the model Summit developed for study we conducted on  Veteran and Non-Veteran Job Seekers Analysis for the Depart of Labor’s Chief Evaluation Office. [3] The bottom of the diagram describes four different kinds of evaluations we will explore in future posts: needs assessment and process, outcome, and impact evaluation. [4] 

Over the past decade, the Federal Government has increasingly tied agency budgets to the impacts of their programs. The 2014 Economic Report of the President argued:

“Evaluating Federal programs and interventions to understand their impact, and developing the infrastructure within agencies to support a sustained level of high-quality evaluations, is an Administration priority.” 

By rigorously testing which programs and interventions are most effective at achieving important goals, the government can (a) improve its programs; (b) scale up the approaches that work best; and (c) discontinue those that are less effective.

Even so, Federal agencies' capacity for evaluating their programs–and effectively using the results of these evaluations–remains uneven. In a recent report, the Government Accountability Office showed that only 7 of 24 surveyed Federal agencies had central leaders responsible for evaluation. Perhaps more concerning is that 7 other agencies reported having had no recent evaluations of their performance goals.