The Alchemy of Program Evaluation, Part 3: Focus Groups

Posted by Teresa Kline on 4/19/21 10:00 AM

Find me on:

--------------------------------------------------------

12 crop

Welcome back to our blog series on program evaluation. If you haven’t already, check out last week’s post on literature reviews and environmental scans, the first step we take in conducting program evaluations. In this installment, we will be looking at focus groups.

Let’s say you’re conducting a program evaluation, and your research question is: What’s the efficacy of an antidrug program aimed at high school students? Or: How effective is a mammogram messaging campaign aimed at women over the age of 40? Focus groups are a great way to answer these types of questions, because they allow you to ask questions specific to your evaluation topic of the group most impacted by that topic. Focus groups are guided discussions with a homogeneous group of people (such as high school students or women over the age of 40) and are a great tool for collecting data on participants’ beliefs, opinions, and perceptions.

Mammogram messaging people

After focus groups have been conducted, we typically do a thematic analysis of findings, meaning our end goal is to highlight the major themes that were raised and determine if there are any patterns by subpopulation. The first step is to review a small percentage of the focus group transcripts to begin identifying themes. This step informs the creation of an a priori coding hierarchy—essentially, we are making educated guesses about what themes we think will be in the rest of the transcripts. But this is not a rigid process—as we conduct our analysis, we add new, emergent themes to this coding hierarchy.

After the creation of the hierarchy, our team begins coding the data. Coding is basically the process of “tagging” sections of text with the relevant thematic codes—we typically do this in a qualitative analysis software called NVivo. One coding is complete, we examine the major themes and look for patterns across groups—for example, do men and women have different opinions about a marketing campaign? Do older adults discuss different challenges in applying for personal loans than younger adults? These findings supplement the results from our other evaluation methods, contributing to our overall program evaluation. (For a more in-depth discussion of how qualitative analysis is conducted in NVivo, check out the second post from our series on the Federal Register.)

Special Considerations during COVID-19

Virtual Focus Groups

  • Smaller group size recommended (4–6 people)
  • Be prepared to troubleshoot video or audio issues
  • Be ready to deal with loss of nonverbal cues

While this post does not cover how to recruit for and conduct focus groups (that would take much more space than we have!), we do want to briefly discuss conducting focus groups during the COVID-19 era. Since we will be in this pandemic world for a little longer, this topic will have an ongoing impact on our work. Although virtual focus groups are not a new concept, they are more challenging to moderate than in-person groups. Program evaluators conducting virtual groups in lieu of in-person groups need to keep a few factors in mind:

  • Group size. A smaller than typical group is recommended: 4–6 participants rather than 8–12.
  • Technology. What service is the easiest for participants to use? Be ready to troubleshoot video or audio issues.
  • Loss of nonverbal cues. Be prepared to play “traffic cop” a little more frequently than usual, since participants and moderators will have a harder time picking up on nonverbal cues that someone has an opinion or is about to speak.

Virtual focus groups can work, and our team has successfully conducted them over the past year, but they do require some redesign.

Join us again next week, when we discuss another qualitative data collection methodology—semi-structured interviews.

Topics: program evaluation, Summit Blog

About the Summit Blog

Complexity simplified.

Summit is a specialized analytics advisory firm that guides clients as they decode their most complex analytical challenges. Our blog highlights the strategies and techniques we use, as well as relevant topics in current events.

Subscribe to Email Updates

Recent Posts

Posts by Topic

see all