Challenge: The U.S. Department of Energy (DOE) Office of Science funds and manages research programs in basic sciences through a grant program to approximately 250 universities. The Oak Ridge Institute for Science and Education (ORISE) was tasked with gathering information and performing a statistical analysis to track the output and performance of these grants in terms of human capital development. Results were included in the DOE’s annual budget report to Congress.
Solution: The Summit team worked with a group of subject matter experts and DOE staff to design a Web-based survey instrument that was sent to principal investigators of all DOE grants. Summit staff pretested the survey instrument in person with a group of principal investigators and then made alterations to the wording of questions and the design of the survey itself to ensure that the instrument was clear and easy to complete. Summit worked with an outside vendor to implement the survey through a Web-based platform. Summit hosted a help desk for survey respondents and provided follow up through additional modes (that is, e-mail, phone calls, and mailed letters) to ensure the required response rate. The final response rate to the survey was 84 percent. Summit designed a plan for testing and adjusting for non-response bias in the survey responses. This plan ensured that Summit used the most sophisticated techniques available to deal with any non-response bias and ensured a transparent and viable estimate of the supported work force that the DOE could confidently present to Congress. This data collection also had to gain Office of Management and Budget (OMB) approval under the Paperwork Reduction Act (PRA). The team authored both Part A and Part B of the OMB PRA request. The team also assisted the DOE in negotiations with OMB to make changes to the survey to deal with OMB privacy concerns, while ensuring that DOE could collect the data required for the study.