Struggling with scenario definition

August 2024

Scenario analysis and stress testing have, over the last meanwhile nearly two decades, become something like tools of choice – due to global financial, economic, political, technological, societal, and last but not least climate change induced challenges. At their core stands, besides methodological choices and data issues, the design and specification of the stressed and unstressed scenarios which should be investigated. And this is a point where it usually gets difficult.

The setting

The scenarios to be defined are mostly multidimensional, interdependent across their dimensions, path-dependent, and can less and less draw on experiences we have already made at some point in the past (the more structural and fundamental the structural changes expected for the future compared to observed historical data will be, the more pronounced this problem is). Comparability of historical data would be necessary in order to apply our traditional risk management methodologies when trying to build the bridge from past developments to future developments. But now, under current circumstances, this is possible only to a limited degree. Rather, we are confronted with pending major developments, the consequences of which we do to a considerable degree not yet know and cannot sufficiently quantify. Their character will unveil only over time. An example for such a complex future development, already tangible by now, is climate change, including all its economic, financial and other consequences. Furthermore, we can and shall expect, as at any time, to be confronted with developments which are not yet on our radar at all and in respect of which we might not even have an idea how a quantification could look like for the time being.

But let us start from the beginning.

Scenario specification and historical statistical measures

Despite the background we have just sketched in the previous paragraph, stress scenarios for macro-economic variables can be and are obtained by looking on historical macro-economic data. For example, the US Federal Reserve System1 uses measures of economic activity and constructs on this basis stress scenarios by looking at realizations of these variables during recessions. By classifying the recession periods as mild, moderate, or severe, growth rates of GDP, unemployment, etc. the corresponding baseline and the severely adverse stress scenarios can be obtained from historical data.

For any stress-testing analysis, appropriate scenarios have to be defined. For example, a bank or a regulator might be interested in the effect of macro-economic shocks on the corresponding loan portfolio. Usually — so called — satellite models are used to link defaults or default intensities and macro-economic variables in order to obtain default forecasts. Furthermore, if the number of macro-economic variables considered is larger than the number of shocks (e.g., in multi-country models), then VAR or Global VAR models can be applied. They can be used to connect the shocks, or rather the shocked macro-economic variables, with the set of other macro-economic variables to estimate the effect of the shocks on them as well. In turn, the whole set of macro-economic variables, now all conveying the shock effects, can be used to forecast the, e.g., default probabilities.

Scenario derivation in the context of systematic stress testing

The following text on systematic stress testing is based on Breuer and Summer (2017).2

Current peculiarities: A problem related to stress testing is demonstrating the plausibility of stress scenarios. Furthermore, the number of scenarios is mostly restricted to a small, though carefully chosen, set. And finally, there is little indication concerning representativeness of the scenarios applied with regard to the investigated output size(s) like, for example, probability of default or sustained loss. It would be important to know, however, if one has chosen the scenario with the worst or the one with the weakest output effect of a given plausibility set.

Origin: These peculiarities of stress test scenario formulation can in principle be traced back to the original intention of stress tests, namely to be independent of modelling assumptions and underlying probability distributions – and as such to form a complement to other risk measurement approaches.

Given the increasing role stress tests have been playing over the last years and the growing importance of its results for diverse steering and supervision functions this particularity should be kept in mind. Plausibility quantification and identification of complete sets of stress scenarios will gain importance in order to avoid lack of informative value and lack of representativeness, stress testing experts expect.

Scenario derivation under systematic stress testing: What is required for a systematic scenario generation and evaluation? Basically three components: Plausibility quantification for each scenario, completeness of the applied scenario set, and quantification of scenario severity (in terms of consequences on output figures as, e.g., probability of default, sustained loss, capital, etc.).

• Plausibility: To be able to obtain a plausibility quantification of stress scenarios, one needs an enhanced information set compared to the current status. One needs, on top of the risk factors and the output function, also a probability distribution of the risk factors. With this step, distributional assumptions become relevant, and a conceptual issue also in stress tests. They seem, in some respect, to represent the price for the plausibility quantification of the scenarios. Methodology-wise, academic research proposes eligible statistical approaches to and measures for plausibility quantification.

This is contrary to non-systematic stress tests, as categorized by Breuer and Summer (2017). While the latter also take into consideration the plausibility of the scenarios and historical distributions of risk factors, though, this is done in a more informal way. In non-systematic stress tests the statistical distribution of the risk factors typically plays no formal role, plausibility is not rigorously quantified.3

• Completeness: To ensure that all important scenarios are considered in a stress test, the goal is to identify a complete set of sufficiently plausible scenarios and to apply them in the stress testing procedure. Again, academic research proposes methods to come up with estimations of such complete scenario sets which basically depend in their size on the chosen plausibility threshold and in their form on the risk factor distribution.

And for concluding the scenario selection process:

• Severity: The severity of the scenarios contained in the plausible scenario set has to be quantified. Severity means the effect of a scenario on the investigated output size. As already mentioned, examples for typical output categories are loss sustained, capital shortfall, probability of default, etc.

In sum: A systematic stress test should identify the worst-case scenario among the sufficiently plausible scenarios.

Scenario design in an environment of high uncertainty

And finally, we arrive again at the high complexity and uncertainty from where we started out at the beginning of this text. For example, climate change and all consequences associated with it, broadened and deepened use of AI and machine learning (including other technological disruptions like, e.g., quantum computing) or materially new climate-related technologies and fundamental industrial shifts are areas characterized by high uncertainty. In addition, far-reaching political shifts, and potential wars augment this list of fields with high uncertainty.

Furthermore, on top of the complexity of individual development strands (such as those just described), analyses of compound shocks are very important and have to be undertaken. For example, climate and war, or climate and pandemic, or political disruption and technological disruption, etc. Of course, such compound shocks are even harder to operationalize and implement than single-category shocks.

Scenario design and climate risks as a concrete high uncertainty example: Last year, the Financial Stability Institute published an executive summary document on scenario analysis for climate risks (FSI 2023). It emphasizes the importance of scenario analysis for evaluating the financial risks resulting from climate change – given the unprecedented nature of climate risks and the inherent uncertainty of future climate-related developments and occurrences. And of course, it also refers to the reference scenarios published by the Network for Greening the Financial System (NGFS), which are more and more used by public and private sector actors. These scenarios serve as a starting point for investigating economic and financial consequences of climate change. They are assigned to three categories (orderly, disorderly and hot-house world), and each category comprises two scenarios with different levels of physical and transition risks.

The FSI document delineates four steps to be carried out when undertaking a scenario analysis:4

1. Determine the objectives and the scope of the scenario analysis: Define which impacts to investigate (for example, financial firm´s risks, or system-wide risks, or other objectives), choose the most material risk drivers, select the time horizon, select the target audience.
2. Select and design the scenarios: Determine the type of climate risks covered (physical risks, transition risks, both); select the number of scenarios; select the level of granularity; choose the time intervals for assessments; define the level of severity of the scenarios; choose between static and dynamic balance sheet assumption; set the discount rates (especially for long time horizons and dynamic balance sheet important).
3. Assess the impact: Evaluate the economic effects on key macro variables, evaluate the financial effects (for example, loss of asset value), select exposure and potential loss metrics, choose between top-down-approach and bottom-up approach.
4. Use and communicate the results.

Some challenges with respect to scenarios and shocks in climate scenario analysis are:5

• Translating scenarios into shocks for a specific economy, a specific sector, a specific financial firm, a specific financial instrument, etc.
• Identifying plus figuring out the size of multiple transmission channels and of the ways companies react to climate shocks. In addition, detecting the degree of cost pass-through to clients.
• Detecting climate risk sensitive sectors and disaggregating the overall economic impact of a scenario and attributing it to specific sectors.

Comments from practical field experience: Comments from practical field experience in climate risk assessment and climate stress testing also allude to the effort it takes to translate the NGFS scenarios into scenarios used for specific individual application cases. The importance of transparency, which has to be established with respect to an individual object of investigation (i.e., clarification of taxonomy issues, detailed analysis of asset structure, etc.), in order to be able to come up with good, customized climate risk scenarios, is emphasized as well.

 

1 See https://www.federalregister.gov/documents/2017/12/15/2017-26858/policy-statement-on-the-scenario-design-framework-for-stress-testing and https://www.federalreserve.gov/publications/files/2024-stress-test-scenarios-20240215.pdf

2 Chapter 4.2 Second generation stress tests: plausibility and completeness.

3 See for this paragraph Breuer and Summer (2017), chapter 4.1, p.18.

4 See FSI (2023), Table: Scenario analysis process – main choices and assumptions, p.1.

5 For the following see FSI (2023), paragraph Common challenges, p.2.

 

References
BIS – FSI Connect (FSI 2023), Climate risks: scenario analysis – Executive Summary, FSI Executive Summaries, 31 May 2023.

T. Breuer and M. Summer (2017), Solvency stress testing of banks: Current practice and novel options, Report for the Sveriges Risksbank and Finansinspektionen, July 2017.

Federal Reserve System (2017), Policy Statement on the Scenario Design Framework for Stress Testing – A Proposed Rule by the Federal Reserve System on 12/15/2017 (12 CFR Part 252, Regulation YY; Docket No. OP-1588), see https://www.federalregister.gov/documents/2017/12/15/2017-26858/policy-statement-on-the-scenario-design-framework-for-stress-testing (accessed on August 28 2024)

Federal Reserve System (2024), Board of Governors of the Federal Reserve System – 2024 Stress Test Scenareos, see https://www.federalreserve.gov/publications/files/2024-stress-test-scenarios-20240215.pdf (accessed on August 28 2024)

 

Disclaimer
Ed.: LBMS Advisory Services GmbH, 1070 Vienna, Austria, FN 417881g, office@betainside.com, https://betainside.com.
The contents of this document are for general information and marketing purposes only and do not contain any recommendations whatsoever. In particular, the information does not constitute any offer or legal, tax, investment or financial advice and is not suitable as a substitute for individual business advice. The content has been compiled with the greatest possible care. However, we accept no liability whatsoever for the accuracy, completeness and up-to-dateness of the information and data contained herein, nor for any loss or damage of any kind (whether direct or indirect) arising from or in connection with this document. The information does not constitute a decision-making aid. The content created and published by the publisher is protected by copyright. All rights are reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Unauthorized use of such content is not permitted.