Using Administrative Data for Program Evaluation

October 30, 2015 China Layne

In our previous post, Summit’s Program Evaluation Team discussed the primary considerations in assessing what kinds of evaluations are suitable for a specific program or policy. Investigating data availability and comprehensiveness is a key component of the evaluation process. Data are the lifeblood of an evaluation study. Without relevant, high-quality data, the program evaluation may have little value, even with otherwise rigorous designs and large sample sizes. Evaluators can work with stakeholders to determine the quality of the data available for analysis and set expectations on what the evaluation can (and cannot) reveal.

In this post, we discuss the prospect of using a program's existing administrative data to conduct evaluations. We describe different types of administrative data as well as the advantages and challenges of using administrative data for program evaluation.

Administrative data are increasingly useful for government agencies as the current administration continues to encourage data analytics and evidence-based program evaluations.[1]

Types of Administrative Data and its Benefits for Evaluation

Practically any data agencies or stakeholders collect about their program can be used for evaluation. These data come from a variety of sources:

  • Program start-up documents
  • Performance management documents
  • Case management data
  • Audit documents

Different sources of administrative data can inform different assessments. A needs assessment helps evaluators understand the target population and determine the gaps that the program will fill. Similarly, a process evaluation measures and monitors how the program is being implemented. For outcome and impact evaluations, the administrative data source must include information on the program's outcomes, participants, and a comparison group.

Advantages of Using Administrative Data for Evaluation

There are many advantages to using a program's administrative data for evaluation:

  1. Administrative data are readily available to the agency and immediately relevant to the program being evaluated.
  2. Administrative data are regularly collected and updated, providing the most recent information to use in evaluation.
  3. Administrative data often represent large samples of the underlying population, providing support for robust evaluation and analysis methods.

Challenges of Using Administrative Data for Evaluation

Despite the many advantages, using administrative data for evaluation also presents four specific challenges. Summit has developed a variety of methods for dealing with these challenges, as summarized in Figure 1.

Challenges of Using Administrative Data for Evaluation

Figure 1. Challenges of using administrative data for evaluation and potential solutions.

By understanding the specific advantages and challenges of using administrative data and developing appropriate methods for dealing with these challenges, trained evaluators working with agencies and stakeholders can fully leverage a program's own data for use in evaluation.

For example, the U.S. Department of Labor’s Occupational Safety and Health Administration (OSHA) contracted with Summit to learn more about the Federal Agency Targeting (FEDTARG) program. The FEDTARG program chooses Federal worksites for inspections based on the number of lost-time cases (LTCs) the worksite experienced in the previous fiscal year. Using administrative data, Summit conducted a descriptive study of the program. The results, available in the Federal Agency Targeting Inspection (FEDTARG) Program Study Final Report, provide details on the characteristics of the inspected sites, inspection information (including the violations cited during inspections), and the injury/illness counts before and after inspections.

Next week, we will discuss different types of program evaluation, with an emphasis on outcome and impact evaluations.

To learn more about program evaluation, check out other posts in this series: Is Program Evaluation Important to Federal Departments and Agencies?How to Use Logic Models in Program Evaluation, and Assessing Evaluability: Selecting the Right Type of Program Evaluation.

 

[1] U.S. Executive Office. “Fiscal Year 2016: Analytical Perspectives of the U.S. Government.” Washington, DC: U.S. Government Printing Office, 2015. http://www.whitehouse.gov/sites/default/files/omb/budget/fy2016/assets/spec.pdf

Share This: