The Alchemy of Program Evaluation, Part 1: Introduction

With contributions by Balint Peto

Welcome to our blog series on the alchemy of program evaluation. Program evaluation is the process of collecting and analyzing data to answer questions about how well and how effectively programs are working. Through mixed methods, we utilize both quantitative and qualitative data to answer such research questions as: How do you know if your program (or program modification) is achieving its mission? If program outcomes aren’t meeting expected goals, what isn’t working? What can be improved to increase program efficacy?

Evaluating the efficacy of programs is crucial in the federal sphere to ensure that federal dollars are spent wisely, and the Evidence-Based Policymaking Act aims to ensure programs achieve outcomes relevant to their missions. According to a 2020 Office of Management and Budget (OMB) memo, the Evidence Act “advances program evaluation as an essential component of Federal evidence building.”

There are a variety of different types of evaluations, including:

• Formative evaluations, which assess the feasibility of a program before implementation
• Process or implementation evaluations, which examine whether programs have been implemented as intended
• Outcome or effectiveness evaluations, which measure program effectiveness based on the target outcomes
• Impact evaluations, which evaluate the program’s overall effectiveness in achieving the program goals

Although Summit is well known for our work in data analytics and econometrics analyses, you might not know that we also have an experienced team of program evaluators who work with a wide range of methods. Summit’s program evaluators have conducted evaluations for federal and private clients on a wide variety of topics, such as succession planning, acquisition planning, and loan program uptake.

The program evaluation team came together to bring you this nine-part series on what we do. Throughout the series, you will hear from various members of our team, highlighting the program evaluation methods that we regularly use. Although there are many methods for evaluators to draw from, in this series we will look at seven key methods: literature reviews and environmental scans, focus groups, semi-structured interviews, surveys, logic models, administrative data analysis, and impact evaluations. Our final post will explore triangulation—how we bring together these mixed methods into a holistic program evaluation.