Book 1. Market Risk

FRM Part 2

MR 4. Backtesting VaR

Presented by: Sudhanshu

Module 1.  Backtesting VaR Models

Module 2.  Conditional Coverage And Basel Backtesting Rules

Module 1.  Backtesting VaR Models

Topic 1. Introduction to Backtesting VaR Models

Topic 2. Importance of Backtesting for Regulators and Risk Managers

Topic 3. Key Challenges in Backtesting VaR

Topic 4. Failure Rate – Measuring Accuracy of a VaR Model

Topic 5. z-Score Based Exception Testing

Topic 6. Understanding Type I and Type II Errors

Topic 7. Kupiec's Unconditional Coverage Test

Topic 1. Introduction to Backtesting VaR Models

• Definition: Backtesting is the process of comparing predicted losses (from a VaR model) with actual losses experienced.
• Purpose:
  • Evaluate model accuracy and adequacy.
  • Identify exceptions—instances where losses exceed VaR thresholds.
• Key Concept:
  • At a 95% confidence level, we expect exceptions in only 5% of observations.
• Regulatory Context:
  • Basel allows internal VaR models but mandates backtesting.
  • More than 4 exceptions in 250 observations → higher capital requirements.
• Example:
  • ​95% VaR model over 250 days ⇒ Expected exceptions = 0.05 × 250 = 12.5

Practice Questions: Q1

Q1. In backtesting a value at risk (VaR) model that was constructed using a 97.5% confidence level over
a 252-day period, how many exceptions are forecasted?
A. 2.5.
B. 3.7.
C. 6.3.
D. 12.6.

Practice Questions: Q1 Answer

Explanation:  C is correct.

(1 − 0.975) × 252 = 6.3

Topic 2.  Importance of Backtesting for Regulators and Risk Managers

• Why Backtest?
  • Validates model assumptions.
  • Helps detect underestimation or overestimation of risk.
• Actions Based on Results:
  • Too many exceptions → recalibrate model.
  • Too few exceptions → may be too conservative.

• Basel Committee Perspective:

  • Identifies risk underestimation.
  • Penalizes with capital multiplier increases.
• Penalizes with capital multiplier increases.
  • 4 exceptions in 250 observations triggers yellow/red zone penalties.

Topic 3. Key Challenges in Backtesting VaR

• Sample Representation Issues:Portfolio Dynamics:
  • VaR assumes static portfolios, but in reality, positions change daily.
• Noise in Actual Returns:
  • Includes fees, spreads, interest, commissions, not always modeled.
• Sample Representation Issues:
  • Limited sample → random variability in exceptions.
• Hypothetical Returns:
  • Should assume unchanged portfolio to isolate model accuracy.
  • Compare with cleaned returns (excluding non-market-related effects).

Practice Questions: Q2

Q2. Unconditional testing does not reflect the:
A. size of the portfolio.
B. number of exceptions.
C. confidence level chosen.
D. timing of the exceptions.

Practice Questions: Q2 Answer

Explanation:  D is correct.

Unconditional testing does not capture the timing of exceptions.

Topic 4. Failure Rate – Measuring Accuracy of a VaR Model

• Failure Rate
  • N = Number of exceptions
  • T = Total observations
• Expected Exceptions:
  • 99% CL → p = 0.01 → expect 2.5 exceptions in 250 days.
• Model Evaluation:
  • If failure rate is much higher or lower → model is inaccurate.
• Test Statistic:
  • Use z-score or log-likelihood ratio for formal test.

Practice Questions: Q3

Q3. Which of the following statements regarding verification of a VaR model by examining its failure
rates is false?
A. The frequency of exceptions can be determined with the confidence level used for the model.
B. According to Kupiec (1995), we should reject the hypothesis that the model is correct if the loglikelihood
ratio (LR) > 3.84.
C. Backtesting VaR models with a higher probability of exceptions is difficult because the number
of exceptions is not high enough to provide meaningful information.
D. The range for the number of exceptions must strike a balance between the chances of rejecting
an accurate model (a Type I error) and the chances of failing to reject an inaccurate model (a
Type II error).

Practice Questions: Q3 Answer

Explanation: C is correct.

Backtesting VaR models with a lower probability of exceptions is difficult because the number of exceptions is not high enough to provide meaningful information.

Topic 5. z-Score Based Exception Testing

• Formula:

• Decision Rule:
  • If |z| > 1.96 ⇒ Reject model at 95% confidence.
• Example:
  • x = 22 exceptions, T = 252, p = 0.05 → z = 2.72
  • Conclusion: Reject model.

Practice Questions: Q4

Q4. A risk manager is backtesting a sample at the 95% confidence level to see if a VaR model needs to
be recalibrated. He is using 252 daily returns for the sample and discovered 17 exceptions. What is
the z-score for this sample when conducting VaR model verification?
A. 0.62.
B. 1.27.
C. 1.64.
D. 2.86.

Practice Questions: Q4 Answer

Explanation: B is correct.

The z-score is calculated using x = 17, p = 0.05, c = 0.95, and N = 252, as follows:

Topic 6. Understanding Type I and Type II Errors

• Type I Error:
  • Incorrectly reject a valid model.
  • Probability increases as test becomes more sensitive.
• Type II Error:
  • Fail to reject an invalid model.
• Binomial Framework:
  • Used to calculate acceptable range of exceptions.
• Regulatory Focus: More weight is placed on minimizing Type II errors.

Topic 7. Kupiec's Unconditional Coverage Test

• Purpose:
  • Tests whether number of exceptions aligns with expected.
• Formula (LRuc):

• Test Statistic:
  • Reject model if LRuc > 3.84 (95% CL).
• Interpretation:
  • Doesn’t account for timing of exceptions—just count.

Module 2.  Conditional Coverage And Basel Backtesting Rules

Topic 1. Conditional Coverage and Exception Clustering

Topic 2. Basel Rules for Backtesting

Topic 3. Basel Penalty Zones and Capital Multiplier

Topic 4.  Practical Suggestions and Industry Insights

Topic 1. Conditional Coverage and Exception Clustering

• Unconditional Test Limitation:
  • Doesn’t consider when exceptions occur.
• Conditional Coverage:
  • Tests whether exceptions are independent over time.​
• Christoffersen’s Test:
  • Combines:
    • LRuc – coverage accuracy.
    • LRind – independence of exceptions.

• Rejection Rule:
  • Reject model if LRcc > 5.99
  • Reject independence if LRind > 3.84
• ​Use Case: Clustered exceptions = likely model misspecification.

Topic 2.  Basel Rules for Backtesting 

• Basel Requirement:
  • VaR at 99% CL.
  • Backtesting over past 250 trading days.
• Expected Exceptions: 2.5
• Penalty Mechanism:
  • 4 exceptions → capital multiplier k increases from 3 up to 4.
• Penalty Justification:
  • Type II error avoidance more important.

Topic 3.  Basel Penalty Zones and Capital Multiplier

• Causes of Exceptions:
  • Basic model integrity issues → Penalty applies.
  • Model inaccuracies → Penalty applies.
  • Intraday trading activity → Penalty considered.
  • Bad luck → Penalty may not apply.

Topic 4.  Practical Suggestions and Industry Insights

• Lower CL Proposal:
  • Use 95% CL with higher multiplier to get more exceptions and statistical reliability.
• Longer Backtest Periods:
  • Pros: Lower variance in exception rate.
  • Cons: May not reflect current portfolio dynamics.
• Tradeoff:
  • Balancing statistical significance vs regulatory simplicity.

Practice Questions: Q1

Q1. The Basel Committee has established four categories of causes for exceptions. Which of the
following does not apply to one of those categories?
A. The sample is small.
B. Intraday trading activity.
C. Model accuracy needs improvement.
D. The basic integrity of the model is lacking.

Practice Questions: Q1 Answer

Explanation: A is correct.

Causes include the following: bad luck, intraday trading activity, model accuracy needs improvement, and the basic integrity of the model is lacking.