VantageScore

February 2009

Overview

VantageScore Solutions LLC has conducted its annual revalidation of the credit risk score, VantageScore.

For the third consecutive year, the results of the revalidation show that VantageScore is highly predictive, consistent and has remained highly predictive despite the economic volatility seen in the U.S since 2007.

This paper provides an overview of credit risk score revalidation processes and the primary metrics for determining the quality of a credit risk score. VantageScore revalidation results for the 2006 to 2008 timeframe are also presented.

Background

Credit risk scores play an integral role in today’s bank lending processes. Scores are used to identify the likelihood that a consumer will repay a loan. Stated another way, a score with strong predictive power will effectively identify and separate good (likely to pay) consumers from those consumers that are unlikely to pay. This information contributes to the banks’ strategies for the type of loan to offer, loan pricing and terms, and the ongoing management of the consumers’ accounts.

A credit risk score synthesizes a consumers’ prior debt management behavior to estimate how they will manage the repayment of debts in the next two years. An underlying assumption in the design of all score algorithms is that the economic environment remains generally similar to the history on which the score was designed. Scores that have been architected in more recent timeframes like VantageScore will therefore reflect more relevant underlying drivers and behaviors of current economic conditions, and consequently maintain greater predictive power.

Validation routines are run at the time of score development to assess the score’s predictive power. Score revalidation analyses are run at subsequent intervals to insure the score retains its predictive power. If the predictive power falls substantially, observed by a significant reduction in statistical predictiveness or a failure to rank consumers according to increasing risk, then the score could expose the lending institution to increased and unnecessary risk.

Given the underlying design assumption presented above, recent economic turbulence and its impact on consumers may result in significant deterioration in credit risk score predictiveness. Score revalidation and monitoring processes are therefore essential to ensuring credit risk scores retain their predictiveness and aid risk mitigation within lending strategies. Lenders should also validate score predictiveness on their own portfolios to ensure the score addresses any nuances within their specific consumer base.

A key mission of VantageScore Solutions, LLC is to validate VantageScore on an annual basis and publish the results to the industry to facilitate understanding of VantageScore predictiveness as well as foster an awareness of credit risk score performance in general.

Revalidation Process

With the recent economic turbulence, an annual revalidation of a credit risk score is critical. In terms of the demographic configuration, size of the sample and the same seasonal timeframe, data selected for the revalidation should be reflective of the data used at time of development.

VantageScore was developed using a June 2003 – June 2005 timeframe on an anonymized sample of 7.5 million consumers, representing the entire U.S. demographic composition. For purposes of the annual revalidation, a similar number of consumers are randomly selected from the three national consumer reporting companies’ (CRC – Equifax, Experian and TransUnion) databases over the most recently available two-year window. For the 2008 revalidation of VantageScore, the sample timeframe for revalidation is June 2006 to June 2008.

Many measurements exist to evaluate the performance of a credit risk score on a variety of dimensions. Primary measurements used in the industry are:

• Statistical Validation – Kolmogorov-Smirnov (KS) test or GINI Index. The higher the value of either metric, the more effectively the score predicts consumer repayment performance. (The KS statistic is cited in this paper). Typical commercial credit risk scores have KS statistics in the range of 45 to 70.
• Rank Ordering – Consumers are ranked using the score such that increasing levels of default likelihood are observed in the higher deciles. An effective credit risk score ranks consumer risk such that the risk rate should monotonically increase for each increasing decile. Risk rates that do not monotonically increase are an indication that the score is failing to rank correctly, in other words, consumers with high risk profiles could be assigned to low risk lending strategies.
• Score Consistency across CRCs. Using a consistent data set and the same scoring algorithm a consumer should receive identical credit risk scores from multiple CRCs. VantageScore uses the same algorithm at each CRC, furthermore the characteristics used by the algorithm are leveled such that data submitted by lending institutions is interpreted in a highly consistent fashion. Differences in a consumer’s VantageScore are driven by variations in the consumer credit file that might exist at each CRC. Those variations are largely driven by two reasons: not all lenders submit customer payment information to all CRCs and payment information may be submitted at different timeframes.
• Score Reliability. Implementation of strategies requires significant resources so lenders require that credit risk scores maintain strong, stable performance over extended timeframes regardless of changes in economic conditions.

VantageScore Validation Results For The June 2006 To June 2008 Timeframe

Statistical Validation

The chart below demonstrates VantageScore is highly predictive. KS statistics are provided for VantageScore when validated at each CRC. Strong scores typically achieve a KS value in the range of 45 to 70.

Rank Ordering

VantageScore demonstrates strong rank ordering functionality, seen in the table below with interval risk rates that are monotonically increasing as the deciles increase. This ensures higher risk consumers are identified and assigned to higher deciles for more conservative lending practices.

Consumer Score Consistency

VantageScore continues to demonstrate very strong consumer consistency. Consumers were simultaneously scored at each of the three national CRCs and then analysis was conducted comparing two of the three results. Sixty-nine percent of these consumers received a VantageScore within 20 points of each other. In other words, the consumer’s risk threshold is the same from two independent sources. This result is especially important for the real estate industry, where lenders typically require a credit risk score from at least two CRCs. Large variations in the score can result in sub-optimal product and pricing offers for the consumer. The analysis was repeated using all pairwise combinations of data sourced from the three CRCs. Similar results were obtained for each combination.

Using the Score Consistency Index (see prior paper from VantageScore, Score Consistency Index, April, 2008), VantageScore is typically at least 30% more consistent than other comparable CRC proprietary generic credit risk scores, thereby enabling lenders to make more appropriate product and pricing offers to consumers.

Additionally, score consistency has remained stable over the last four annual validations.

Score Reliability

Four annual validations have been conducted on VantageScore since its development in 2005. The graph below shows the KS statistics for Existing and New account validations for each year. The predictive power of the score has remained extremely strong despite the economic volatility.

Conclusion

Robust revalidation processes are especially critical in periods of economic volatility. The transparency delivered to the market by publishing revalidation results provide the market with an effective tool for understanding risk model performance and stability. The analytics presented above clearly demonstrate VantageScore’s ability to deliver consistent performance despite changes in economic conditions.

All lenders who utilize consumer credit risk scores need to be assessing their models’ efficacy on an annual basis with measurements similar to the tests provided in this paper. Any significant shifts in score performance could require a corresponding shift in strategy.

VantageScore is a generic credit risk scoring model introduced to meet the market demands for a highly predictive consumer score. Developed as a joint venture among the three major credit reporting companies (CRCs) – Equifax, Experian and TransUnion, VantageScore offers more consistency across all three CRCs and has the ability to score a broad population.