April 15, 2021 •Kami Ehrich
Welcome to the second part of our blog series on program evaluation. If you have not already, check out last week’s post introducing the series and providing a description of what program evaluation is. In this installment, we will look at literature reviews and environmental scans.
Why do we care about literature reviews? Literature reviews are done so that you don’t reinvent the wheel during your analysis! They review relevant literature on similar topics and methods as your evaluation to provide a starting point. This is why they are often the very first step in conducting your evaluation. The review can also suggest important variables to consider or methods to employ. At Summit, we have completed literature reviews on:
- behavioral theories related to disaster relief that provided insights into how individuals make decisions in high-stress situations such as after a disaster;
- leadership and succession planning at government agencies that gave a reference point for developing interview and focus group topics to probe deeper into a single agency’s leadership and succession; and
- youth employment for youth with disabilities that provided knowledge on what peer-reviewed studies have to say on how to increase youth employment.
Literature reviews and environmental scans are slightly different:
- A literature review is a systematic search using online search engines and peer-reviewed publications of published or gray literature (materials and research produced by organizations outside of the traditional commercial or academic publishing and distribution channels, such as working papers, reports, white papers, and government documents) within a predetermined set of search parameters and inclusion/exclusion criteria, which leads to evidence-based conclusions.
- An environmental scan provides a current snapshot of the state of a particular topic from an examination of sources, which may include publicly available policy documents, published literature, gray literature, brochures, other public information, or primary and secondary data. An environmental scan usually has a recommendations section that includes any knowledge gaps based on current available sources that were identified.
However, in both an environmental scan and a literature review, there are a few similar steps and components:
- A review of all collected literature to ensure that it is relevant
- Summaries of all collected sources
- Organizing the summaries by source (literature reviews) or topic (environmental scans and literature reviews)
- A clear description of your methodology for searching and the inclusion and exclusion criteria
- A comprehensive bibliography, including summaries of all literature covered by the review
Literature reviews and environmental scans can help you understand what has been done on your topic and where your research can fit in the current landscape. They are a good starting point for any evaluation, but they should be accompanied by other research methods like focus groups, which are discussed in next week’s post.