November 8, 2024 •Hayley Carter
With contributions by Teresa Kline and Kyle Crane
In October, members from Summit’s Performance Improvement and Evaluation practice attended the American Evaluation Association’s Evaluation 2024: Amplifying and Empowering Voices in Evaluation conference in Portland, Oregon. The conference was a great opportunity to connect with and learn from researchers across academia; federal, state, and local government; private industry; and beyond.
Summit manager Kyle Crane and senior consultant Teresa Kline had the opportunity to deliver two Ignite-style presentations, sharing insights about underreporting of black lung in the Navajo Nation and opportunities for participatory data analysis to advance government evaluations.
While the conference sessions covered a wide variety of topics, a few key takeaways stood out to our team:
🤖 AI is here to stay—and must be used wisely. Unsurprisingly, the rise of generative AI was a big topic of conversation. Thursday’s plenary session, “Generative AI: Navigating the Ethical Frontier in Evaluation,” highlighted that AI is neither good nor bad; it’s a tool for evaluators with both potential and risks. AI can help us streamline evaluation tasks like coding to analyze survey data or categorizing hundreds of literature review sources. But we should remember that AI can’t do it alone. The human element is essential to using AI wisely. At the end of the day, we as evaluators should be the ones making judgments and drawing conclusions informed by our data. Furthermore, we have the responsibility to be mindful and intentional about what data we feed AI and the limitations. AI tools can be trained on large batches of data that may have built-in biases, and relying on AI tools blindly can amplify bias and discrimination—leading us down the wrong path. As evaluators, we should be doing our research to make sure we’re using AI carefully and ethically.
🔎 Evidence-based policy requires both quantitative and qualitative data. Some sessions underscored the increasing role of statistics in crafting robust evidence frameworks, showcasing approaches ranging from traditional statistical models to innovative data visualization tools. Others focused on the human aspect of evaluation—promoting equity across groups so that all voices are heard. However, an underlying theme throughout the conference was the need to use mixed-methods approaches, integrating quantitative rigor with qualitative insights for a more comprehensive and holistic understanding. Presenters shared examples of how quantitative methods are foundational to development and assessment of performance measurement but require qualitative insights to provide critical context and inform a robust set of findings and direction of policy decisions. This idea highlights both the opportunities and challenges evaluators face when designing and executing projects to address complex issues.
💻 Using data to inform decision-making. In addition to emphasizing the technical skills required for an evaluation, several sessions highlighted the importance of evidence- and data-informed decision-making. The first and most important step toward helping our clients integrate data into decision-making processes is to build trust in the data. Improving data literacy, maintaining transparency by involving clients throughout all stages of evaluations, and collaborating with clients through participatory data analysis sessions are key to building trust in the data itself and in the evaluation process. At the end of the day, we can’t decide how evaluation data will be used by our clients, but we can give them the resources they need to understand, trust, and advocate for evidence-informed policies and decision-making.
We look forward to next year’s conference, which will take place in Kansas City, Missouri!