November 21, 2024 •Josh Goldberg
Regulators of financial institutions play a vital role in encouraging prudent risk management. Yet, designing an effective regulatory framework that balances innovation with accountability and establishes appropriate guardrails is no easy task. Best practices for engaging industry in shaping new regulations—like advance notices of proposed rulemakings and open comment periods—are standard. However, regulations can miss the mark and create layers of bureaucracy instead of fundamentally changing the way risk is managed.
Model risk management has emerged as a compelling example of effective, common-sense regulation. The best practices that evolved through bank regulation now apply broadly across industries. In this post, I’ll explore the 25‑year journey of model risk management’s development, which has closely paralleled my own career.
Early Foundations: The Introduction of Model Validation
In 2000, the Office of the Comptroller of the Currency (OCC) introduced model validation as a foundational practice for managing model risk. At that time, independent model validation was the only established method for model risk management. OCC Bulletin 2000-16 introduced the concept of effective challenge, which requires model validators to actively question a model’s explicit and implicit assumptions. Much like the audit principle of professional skepticism, the bulletin encouraged a deeper review process, systematically decomposing models for rigorous testing.
This model validation standard was in place when the Great Recession of 2008 struck. The crisis led many to question why models had not accurately predicted such significant losses. During this time, I was working in credit model risk management, gaining firsthand insight into potential shortcomings. Although models required more robust challenge, the deeper issue was a cultural one. Risk culture—especially the interaction between executive management and model results—often affected outcomes more profoundly than technical model shortcomings. Governance structures and cultural attitudes in the C-suite, along with pressure to present palatable results to investors, made it difficult to achieve genuine risk transparency.
Expanding the Scope: The Rise of Internal Audit
In 2009, Freddie Mac hired me to develop an approach that expanded beyond standard model validation. Their chief audit executive foresaw that regulators would no longer accept black-box approaches, which only tested inputs and outputs without probing model assumptions. Instead, the Internal Audit department was tasked with directly evaluating the model development and validation functions and assessing the reasonableness of model assumptions and results. While initially perceived as redundant, this additional level of independent review safeguarded against the biases that result from close relationships between model developers and validators.
Growth in Enterprise Risk Management
After the Great Recession of 2008, Enterprise Risk Management (ERM) departments expanded to cover a wider array of risks, from credit to market and liquidity. At organizations like Fannie Mae and Freddie Mac, this growth led to more oversight roles in the Legal, Internal Audit, and ERM departments, while trading floor activity waned. However, this expansion presented challenges, as ERM departments often became too broad and unfocused, resulting in redundancy and loss of agility.
A More Integrated Approach: The Three Lines of Defense
The release of the Supervisory Guidance on Model Risk Management, OCC 2011‑12 (also known as SR 11‑7), marked a significant turning point. This interagency guidance firmly established that model risk should be managed like other types of risk. Freddie Mac’s Internal Audit Division had already laid the groundwork, aligning its model risk management practices with what would later be recognized as industry best practice. The Federal Housing Finance Agency (FHFA), who regulates the government-sponsored enterprises, formalized their endorsement of SR 11-7 and included more prescriptions in some areas when they released Advisory Bulletin 2013‑07.
In 2014, the OCC’s Heightened Standards further formalized the Three Lines of Defense model, assigning clearer responsibilities to first-line risk management, or the business units closest to the processes. The goal was to shift more risk management responsibilities to the first line, ensuring that those closest to business operations were actively managing risk rather than relying on second-line oversight alone. This shift required an adjustment period as business lines that were accustomed to outsourcing risk management adapted to new expectations.
The Heightened Standards prompted a shift in headcount from the second line to the first line, as business lines took on more defined roles in risk management. Initially, some viewed this as a relief from oversight activities, but first-line risk management had to evolve and establish best practices for managing its expanded responsibilities.
The application of three lines of defense to model risk management is illustrated in Model Risk Management Framework graphic below. According to this framework, ERM operates as the second line of defense and owns the framework, which the Board of Directors approves. The first line of defense, or business line, consists of the revenue-generating divisions closest to the business processes. These units play an integral role in risk management by managing risks at the source and are critical participants across the model life cycle. The second line of defense, including ERM and model validation, coordinates with other risk functions to maintain oversight without directly overseeing every step. For instance, software development, implementation, operational control testing, and other similar functions fall outside the scope of model validation.
The Heightened Standards also introduced a subtle but significant change from effective challenge to credible challenge, recognizing that all challenges are not created equal. This change underscored the need for technically competent and practical oversight to ensure that risk challenges add real value. Critically, these standards addressed the need for a supportive risk culture, urging CEOs and front-line units to welcome credible challenges from risk management and internal audit in policy development, new products, strategy changes, and other areas.
Emerging Risks in a New Era of Model Risk
As the industry adopts modern modeling techniques, new risks have emerged, prompting updated regulatory guidance. In 2022, FHFA issued an Advisory Bulletin on Artificial Intelligence and Machine Learning Risk Management. Reflecting an ongoing partnership with industry stakeholders, this guidance anticipates the unique governance requirements that artificial intelligence (AI) and machine learning models bring, particularly around explainability and ethical use.
In 2023 and 2024, FHFA held a Model Risk Conference to address these emerging issues. At the 2024 conference (pictured here), I represented Summit Consulting, LLC on a panel discussing explainability, transition to new credit scores, and handling historical data from the pandemic. My focus was on the unique challenges explainability brings to model risk management, especially where regulatory requirements intersect with practical business needs. For certain applications, such as financial reporting or fair lending, explainability is non-negotiable. In other cases, like fraud detection, flexibility to adapt to new patterns may require less transparent methods. Managing explainability involves making complex methodologies more intuitive to users and stakeholders, balancing performance and interpretability as context demands.
Josh Goldberg of Summit Consulting (2nd from the left) and fellow panelists at the 2024 FHFA Model Risk Conference.
Dan Keating, Assistant Professor and Faculty Director of Academic Support at Simon Business School, presented on how Generative AI would transform the education of future business leaders. His insightful talk is part of a broader Generative AI initiative by Simon Dean Sevin Yeltekin. The breadth of the speakers—from technology providers (Blackrock, AWS, SAS) to first- and second-line risk management practitioners to educators—demonstrates the effect of AI on the practice of model risk management.
Looking Ahead
Over the past 25 years, model risk management regulations have evolved considerably, enhancing transparency and oversight. The timeline below depicts this evolution. As financial institutions adopt increasingly complex models, effective regulatory engagement with industry will be critical to addressing the challenges ahead. Bank regulators face the ongoing task of staying current with these rapid advancements in methodology, but they have demonstrated a strong track record thus far.
