February 23, 2018 •Laura Hoesly
In the fourth post of this series, we emphasize the importance of quality control. You can find the rest of the Strength In Numbers blog posts here. This blog was written by Summit Consultant Laura Hoesly and Summit Manager Kaye Burton.
Imagine you are a researcher tasked with understanding the effects of an oil spill in Alaska on the salmon population. Your teaming partner is responsible for collecting and combining the data and you are responsible for the analysis. When the data arrive, you read in the data, run your models, output your results, compile the results into a report, and send to your client. Your client calls and comments that the results are not making much sense.
You take a closer look at your data and realize you have fewer than expected pre-oil spill data files. Not only that, but there is an additional column showing the “scale” of the population and the scales are inconsistent across the data files.
There are many components to quality control that could have saved you from this situation. The following are a few key components:
- Know the risk points: Understand your data, process, and analysis to identify where potential pitfalls could come up. Create controls and responsibilities to mitigate against these risks.
- Communicate clearly: Establish clear communication lines with your team and partners. This will ensure everyone is working from the same, most updated data and analysis approaches.
- Conduct a sniff test: Compare your results to your expectations to catch mistakes. Think about your expected results and compare your results to this benchmark so you can easily flag questionable values.
- Update your process: Add items to your quality control procedures as you learn more and refine your approach. Document and explain procedures and decision points for future reference.
Understanding the potential risk points of a project may involve balancing the efficiency of automation with the value of human intervention. For example, in one of our cases, we were hired to perform rigorous, independent quality control of a process used to assess the damages associated with the 2010 BP Deepwater Horizon Oil Spill.
The initial damage estimation relied on survey results, photographs, and other data elements. For our team to assess the accuracy of the findings, we used a combination of manual and automated processes to import the information from the original files into electronic form, verify that the information was imported correctly, and then verify the calculations themselves. We planned and implemented the quality control process and made improvements along the way each time a new potential risk point was identified.
Open communication among your team and with your partners will help define responsibilities and roles so that everyone is on the same page. This is especially important when working in litigation matters involving multiple experts, as it is essential for all experts to have the same understanding about who owns each piece of analysis and how the analyses relate to one another.
Conducting a sniff test on your results and scrutinizing whether they are intuitive or plausible not only helps you detect coding or data errors, but it also will help you think of other avenues you may not have considered yet. Sniff tests could point you to exploring specific subsets of your data that may explain an overarching pattern in your results.
Finally, updating your process and documentation as you work through a project or case will centralize your understanding of the data and its potential issues. Not only will it safeguard you against making the same mistake twice but it will also help you keep all your accumulated knowledge in an easily accessible location.
Incorporating these practices into your analyses and deliverables, no matter the size, will decrease the likelihood of an error and increase your credibility. While it would be ideal to never have an error, they sometimes can happen. Facing these errors head-on, providing options to overcome the issue, and using what you have learned to improve your process will make the issues easier to overcome and less likely to happen again in the future.