The Alchemy of Program Evaluation, Part 5: Surveys

Welcome back to our blog series on program evaluation. If you haven’t already, check out last week’s post on semi-structured interviews. In this installment, we will be looking at surveys.

Everyone has likely done a survey. Did you complete the U.S. Census last year? How about the customer satisfaction survey the cashier circles on your receipt from your favorite store? Or what about a poll for the last presidential election? A survey gathers information quickly from small samples (such as your soccer team’s favorite pizza toppings) or representative information on a large-scale population (such as the Census). Surveys are a common and very useful method of data collection in program evaluations.

You might decide to field a survey for a variety of reasons:

• To collect data on people’s knowledge, opinions, or perceptions
• To seek explanations for specific observations in a population
• When the current data isn’t enough to answer your research questions
• When the current data is not readily available or is out of date

For example, Summit has conducted surveys of Small Business Administration staff on how they think the acquisition process is going in their agency and their experiences with and opinions on leadership succession in the agency. In the case of the latter, there was administrative data available, but it was not sufficient to answer all of the research questions in the evaluation. Another survey example is one sent to state Supplemental Nutrition Assistance Program (SNAP) administrators on their perceptions of the development of SNAP mobile applications. Summit also conducted a survey designed as an experiment using Amazon’s Mechanical Turk platform as a respondent pool to see how they reacted to different sign-up flows (how you sign up for a service online). This experiment presented different ways to sign up for a service to different participants to see how the change in sign-up flow impacted how many individuals signed up.

Surveys are a powerful tool in the evaluator’s tool kit, and web-based survey platforms allow for quick and relatively inexpensive data collection for populations of any size. Surveys are most often used in conjunction with the other evaluation methods highlighted in this blog series, like administrative data and interviews.

Join us next week, when we will discuss logic and theory of change models.