risk.validation.binomialTest

Binomial test

Since R2025a

Syntax

hBinTest = risk.validation.binomialTest(Probability,NumEvents,NumTrials)

hBinTest = risk.validation.binomialTest(Probability,NumEvents,NumTrials,Name=Value)

[hBinTest,binTestOutput] = risk.validation.binomialTest(___)

Description

hBinTest = risk.validation.binomialTest(Probability,NumEvents,NumTrials) returns the binomial test result, hBinTest, for a given set of probabilities, events, and trials. The output is 1 if the test rejects the null hypothesis at the 95% confidence level, or 0 otherwise. Probability contains numeric values that represent quantities such as probability of default (PD) estimates.

hBinTest = risk.validation.binomialTest(Probability,NumEvents,NumTrials,Name=Value) specifies optional name-value arguments. For example, you can set a specific confidence level for the binomial test by using the ConfidenceLevel name-value argument.

[hBinTest,binTestOutput] = risk.validation.binomialTest(___) also returns a structure binTestOutput, that contains the method used in the binomial test and a table of summary metrics.

Examples

Apply Binomial Test on Probability of Default Data

Use the binomialTest function to determine whether the binomial test rejects the null hypothesis for a 0.95 confidence level in a probability of default (PD) data set. In this example, you use the credit validation data set, which includes a table, ScorecardValidationData, that contains PD values and their corresponding default status. Before you apply the test, it is common practice to:

• Group the probabilities by deciles.

• Compute the average probability of each group.

• Compute the total number of defaults and loans of each group.

After performing these steps, you then apply the test on each of the groups.

Load and display the credit validation data.

load CreditValidationData.mat
head(ScorecardValidationData)

    CreditScore      PD       Default
    ___________    _______    _______

      579.86       0.14182       0   
      563.65       0.17143       0   
      549.52       0.20106       0   
      546.25       0.20845       0   
      485.34       0.37991       0   
      482.07       0.39065       0   
      579.86       0.14182       1   
      451.73         0.494       0   

Group Probabilities by Deciles

Extract the variable, PD, from the table ScorecardValidationData and group the probabilities by using the risk.validation.groupNumberByQuantile function with the fully qualified namespace risk.validation. Specify "deciles" as the quantile type.

Probability = ScorecardValidationData.PD;
QuantileType = "deciles";
PDDecileNumber = risk.validation.groupNumberByQuantile(Probability,QuantileType);

Determine Average Probabilities by Group

Next, calculate the average probability for each group by using the groupsummary function.

PDAvgByDecile = groupsummary(Probability,PDDecileNumber,"mean");

Compute Total Defaults and Loans by Group

Extract the variable Default from the table ScorecardValidationData and use this variable as the default indicator. Then use the groupsummary function to compute the total number of defaults and loans for each group.

DefaultIndicator = ScorecardValidationData.Default;
[NumDefaultsByDecile,~,NumLoansByDecile] = groupsummary(DefaultIndicator,PDDecileNumber,"sum");

Apply Binomial Test

You can then apply the binomialTest function with the fully qualified namespace risk.validation for each group to see if the test rejects the null hypothesis. Use the average probabilities and the total number of defaults and loans as input arguments. Then, display a table with the results.

[hBinTest,binTestOutput] = risk.validation.binomialTest(PDAvgByDecile,NumDefaultsByDecile,NumLoansByDecile)

hBinTest = 10×1

     0
     0
     0
     0
     0
     0
     0
     0
     0
     0

binTestOutput = struct with fields:
     Method: "exact"
       Tail: "right"
    Results: [10×8 table]

disp(binTestOutput.Results)

    RejectBinTest     PValue     NumEvents    CriticalValue    ConfidenceLevel    NumTrials    Probability    ObservedProbability
    _____________    ________    _________    _____________    _______________    _________    ___________    ___________________

          0           0.21501        7              9               0.95             36          0.13683            0.19444      
          0          0.083864       11             12               0.95             36          0.19799            0.30556      
          0           0.51798        8             13               0.95             34          0.22656            0.23529      
          0           0.12711       10             12               0.95             27          0.25552            0.37037      
          0           0.12069       16             18               0.95             42           0.2868            0.38095      
          0           0.60973       13             19               0.95             41          0.32664            0.31707      
          0           0.92069       10             19               0.95             36          0.37627            0.27778      
          0           0.67016       11             17               0.95             29          0.40319            0.37931      
          0           0.58724       18             24               0.95             40          0.45525               0.45      
          0           0.55783       22             28               0.95             39          0.56233             0.5641      

Input Arguments

Probability — Probability values

numeric vector with values in the range (0,1)

Probability values, specified as a numeric vector with values in the range (0,1). Probability contains values that indicate quantities such as PD estimates.

NumEvents — Number of events observed

numeric vector of nonnegative integers

Number of events observed, specified as a numeric vector of nonnegative integers. For PD models, NumEvents contains the number of defaults observed.

NumTrials — Number of trials

numeric vector of positive integers

Number of trials, specified as a numeric vector of positive integers. Each element in NumTrials must be greater than or equal to the corresponding elements of NumEvents. For PD models, NumTrials contains the number of loans.

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: hBinTest = risk.validation.binomialTest(PDAvgByDecile,NumDefaultsByDecile,NumLoansByDecile,ConfidenceLevel=0.96) specifies a confidence level of 0.96.

ConfidenceLevel — Confidence level

0.95 (default) | numeric scalar in the range (0,1)

Confidence level of the hypothesis test, specified as a numeric scalar in the range (0,1).

Method — Hypothesis test method

"exact" (default) | "approximate"

Hypothesis test method, specified as either "approximate" or "exact".

• "exact" — binomialTest uses an exact probability based on a binomial distribution.

• "approximate" — binomialTest uses an approximate test based on a normal distribution.

Tail — Type of alternative hypothesis

"right" (default) | "left" | "both"

Since R2026a

Type of alternative hypothesis, specified as one of the following hypotheses.

• "right" — Test against the alternative hypothesis that the probability is greater than Probability.

• "left" — Test against the alternative hypothesis that the probability is smaller than Probability.

• "both" — Test against the alternative hypothesis that the probability is not Probability.

Data Types: string | char

Output Arguments

hBinTest — Hypothesis test result

numeric vector containing values of 1 or 0

Hypothesis test results, returned as a numeric vector.

• A value of 1 rejects the null hypothesis at the specified confidence level.

• A value of 0 fails to reject the null hypothesis at the specified confidence level.

binTestOutput — Output metrics

structure

Output metrics, returned as a structure with the following fields:

• Method — Method used for the hypothesis test.

• Results — Table with columns:

  • RejectBinTest — Numeric vector that indicates whether the null hypothesis was rejected. This column represents the same values as hBinTest.

  • NumEvents — Number of events observed. This column is the test statistic for the hypothesis test.

  • CriticalValue — Minimum number of events at which the test rejects the null hypothesis for the given probability and confidence level.

  • PValue — p-value for the hypothesis test returned as a scalar in the range [0,1]. A small value indicates that the null hypothesis might not be valid.

  • ConfidenceLevel — Confidence level for the hypothesis test.

  • NumTrials — Number of trials.

  • Probability — Probability values.

  • ObservedProbability — Ratio of NumEvents to NumTrials.

More About

Binomial Test

The binomial test [1] requires you to input the predicted probability P, the number of trials N, and the number of events observed, E. The null hypothesis is that the probability P is a correct estimate of the event frequency. The alternative hypothesis for the test depends on the value of the Tail name-value argument. The test statistic for the binomial test is the number of events E.

There are two versions of the test, the “exact” method, and the “approximate” method. In the “exact” method, the critical values for the right- and left-tailed tests are given by the following:

$$\begin{array}{l}{E}_{right}^{*}=\min \left\{k|{\displaystyle \sum _{i=k}^N\left(\begin{array}{c}N\\ i\end{array}\right)P^i{(1-P)}^{(N-i)}\le \alpha }\right\},\\ E_{left}^{*}=\max \left\{k|{\displaystyle \sum _{i=0}^k\left(\begin{array}{c}N\\ i\end{array}\right)P^i{(1-P)}^{(N-i)}\le \alpha }\right\},\end{array}$$

where α=1–ConfidenceLevel is the significance level for the test. The software rejects the null hypothesis for the right-tailed test when the test statistic is larger than or equal to E^*_right. The software rejects the null hypothesis for the left-tailed test when the test statistic is smaller than or equal to E^*_left. To calculate the exact critical values E^*_left and E^*_right for the two-tailed test, the software sorts the possible outcomes in ascending order of their probabilities. Then, it calculates the integer T such that

$${\displaystyle \sum _{j=1}^T{p}_j}\le \alpha \text{ and }{\displaystyle \sum _{j=1}^{T+1}{p}_j}>\alpha ,$$

where p_j is the jth smallest probability. The critical values are given by the equations

$$E_{left}^{*}=\underset{j\le T}{\max }\left\{E_j^{*}\right|E_j^{*}<M\}\text{ and }{E}_{right}^{*}=\underset{j\le T}{\max }\{E_j^{*}|E_j^{*}>M\},$$

where E_j is the outcome corresponding to p_j, and M is the median of the binomial distribution given by N and P. The software rejects the null hypothesis when the test statistic is outside of the interval [E^*_left+1,E^*_right-1].

In the “approximate” method, using the central limit theorem, the critical values for the left- and right-tailed tests are given by:

$$\begin{array}{l}{E}_{approx,right}^{{}^{*}}=NP+{\Phi }^{-1}(1-\alpha )\sqrt{NP(1-P)},\\ E_{approx,left}^{{}^{*}}=NP+{\Phi }^{-1}(\alpha )\sqrt{NP(1-P)},\end{array}$$

where Φ is the standard normal cumulative distribution function. The critical values for the two-tailed test are given by

$$E_{approx,left}^{{}^{*}}=NP+{\Phi }^{-1}\left(\frac{\alpha }{2}\right)\sqrt{NP(1-P)},E_{approx,right}^{{}^{*}}=NP+{\Phi }^{-1}\left(1-\frac{\alpha }{2}\right)\sqrt{NP(1-P)}$$

If the inputs to risk.validation.binomialTest are vectors of size G, then G statistical tests are run and the function returns the results for each of these tests.

For credit risk applications, P is typically a PD, N is the number of loans, and E the number of defaults. The G groups could correspond to different credit ratings or deciles based on the PD values of a continuous PD model. For market risk applications, P could represent the complement of the VaR level, N could represent the number of days in a test window such as 250 trading days, and E could represent the number of VaR exceptions.

Function

You can use the varbacktest object function bin to perform a binomial test on a timeseries of portfolio outcomes. bin allows you to perform binomial tests on multiple timeseries with different VaR confidence levels.

References

[1] Basel Committee on Banking Supervision, “Studies on the Validation of Internal Rating Systems”, Working Paper 14, May, 2005. https://www.bis.org/publ/bcbs_wp14.htm.

Version History

Introduced in R2025a

R2026a: Tail name-value argument to specify type of alternative hypothesis

Specify the type of alternative hypothesis using the Tail name-value argument.