Case Study
Mitigation Loan Communication Strategies Evaluation for the U.S. Small Business Administration
Challenge: The U.S. Small Business Administration (SBA) wanted to conduct an experiment to determine whether new communication materials would increase uptake of the Disaster Assistance Loan mitigation increase option.
Historically, the percentage of eligible borrowers in the Disaster Assistance Loan program who utilize the additional mitigation increase option―which provides expanded financial support to purchase mitigation measures post-disaster―has been around 2%. One goal of the SBA’s Office of Disaster Assistance is to increase uptake of mitigation increase option, as stated in the SBA 2022–2023 fiscal year agency priority goals.
Based on findings from the Evaluation of Characteristics and Perceptions of Disaster Assistance Mitigation Loan Option Borrowers, which Summit completed in 2021, the SBA developed a new mitigation increase option communications plan, which includes an outreach email for eligible borrowers. In this follow-on evaluation, Summit used a quasi-experimental design to determine whether the new email communication increased the percentage of eligible borrowers choosing the mitigation increase option, whether people who opened the email or clicked the link for more information within the email chose the mitigation increase option at higher rates, and whether these outcomes varied across disaster types, states, borrower income, and other borrower characteristics.
Solution: Summit conducted an experiment to test the impact of newly developed communication materials.
Summit started this evaluation by testing several different email drafts. Based on cognitive testing results, the team selected the most effective email and made several updates to the language, layout, and format based on these findings.
The team then moved to the experiment phase, assigning borrowers to treatment and control groups using stratified random assignment. The treatment group received the updated email message in addition to existing communications, while the control group received only the existing SBA communications. Summit conducted quantitative analyses on the results of the experiment, including overall summary statistics and hypothesis testing using methods such as chi-squared and Fisher’s exact. The team then created two distinct logistic regression analyses models, as well as dot distribution and heat maps of borrower data using zip codes to visualize the concentration of borrowers who chose the mitigation increase option.
Summit found the email communication led to a statistically significant increase in the percentage of eligible borrowers choosing the mitigation increase option. Additionally, borrowers in the treatment group who opened the email or clicked the link chose the mitigation increase option at statistically significant higher rates.
Result: Summit empowered the SBA with evidence-based conclusions and recommendations to help improve uptake of the mitigation option.
The team accomplished this by (1) using additional email communications designed based on the cognitive testing findings; (2) increasing the frequency and type of communications that borrowers receive about the mitigation increase option; and (3) providing the SBA with a programmatic solution to continue tracking mitigation uptake over time. The team organized these recommendations based on feasibility and impact so the SBA could choose the easiest to implement and most impactful to the agency’s mission. A final report, titled SBA Mitigation Loan Communication Strategies Evaluation, can be viewed on the SBA’s website.