Chapter 21 argued that explainability in consumer lending is a compliance constraint, not a convenience. This chapter puts that argument to work. It treats SHAP as a production artifact: a piece of code that must run every day, return a stable result, survive a model risk manager’s questions, and feed the adverse action notice engine that a borrower receives in the mail. The chapter derives the SHAP estimators in enough detail to implement them from NumPy, benchmarks the implementations against the reference shap library, exercises the pipeline on the German and Taiwan default datasets, and finishes with deployment and scalability recipes.
The practitioner working in a regulated US lender needs three things from an explainer. The first is a numerically faithful attribution that sums to the model’s margin and respects the Shapley axioms (Lundberg & Lee, 2017; Shapley, 1953). The second is a mapping from attributions to the enumerated reason codes required by Regulation B and by the CFPB’s 2022 adverse action circular (Consumer Financial Protection Bureau, 2022). The third is latency: a real-time decisioning endpoint cannot wait a second for an explanation, and a batch pipeline scoring tens of millions of accounts cannot wait an hour per feature group. This chapter treats each of these requirements with working code.
The European practitioner faces an additional layer. Article 22 of the GDPR restricts solely automated decisions with legal or similarly significant effect, and Articles 13 and 14 require meaningful information about the logic involved. The EU AI Act (European Parliament and Council of the European Union, 2024) classifies consumer creditworthiness scoring as a high-risk system under Annex III, which triggers the transparency obligations of Articles 13, the logging obligations of Article 12, and the post-market monitoring obligations of Articles 72 and 86. SHAP does not satisfy these obligations by itself. It is, however, the technical substrate on top of which every credit-focused explanation product is built today.
Let \(x \in \mathbb{R}^d\) be the feature vector for an applicant, \(y \in \{0, 1\}\) the default indicator, and \(f : \mathbb{R}^d \to \mathbb{R}\) a trained model. Write \(f\) on the log-odds scale unless stated otherwise, so that the base value \(\mathbb{E}[f(X)]\) and the attributions \(\phi_j\) add up linearly to \(f(x)\). Write \([d] = \{1, \dots, d\}\) for the feature index set and \(S \subseteq [d]\) for a coalition. Write \(|S|\) for the cardinality of \(S\). For a coalition \(S\), \(x_S \in \mathbb{R}^{|S|}\) denotes the subvector of \(x\) at indices \(S\), and \(x_{-S}\) its complement.
A credit applicant who is denied a card, a line, or a loan is entitled to a statement of the principal reasons within thirty days under Regulation B at 12 CFR 1002.9. If the decision was informed by a consumer report, the applicant is additionally entitled under FCRA section 615 to notice of the reporting agency and of the right to a free copy. Neither statute defines the format of those reasons, but both require specificity. The CFPB’s Circular 2022-03 closed the remaining ambiguity: the specificity requirement binds every creditor, including those using complex algorithms (Consumer Financial Protection Bureau, 2022). A tree ensemble is not a legal shield.
This puts a lender who uses XGBoost, LightGBM, or CatBoost in the following position. The model returns a probability. The decision engine applies a cutoff and denies the applicant. The adverse action notification system needs a reason, and it needs it per applicant, per day, across the entire portfolio. The SHAP framework of Lundberg & Lee (2017), together with the TreeSHAP estimator of Lundberg et al. (2020), provides the technical answer. Each feature in the model receives an attribution whose sum equals the deviation of the model’s log-odds from the population mean. The top adverse features map, by a maintained reason-code table, to human-readable phrases. The notice is rendered and mailed.
The chapter also takes seriously the warnings in the XAI literature. SHAP is not causal: the attributions depend on an assumed reference distribution, and a naive implementation on correlated features can credit a proxy instead of the true driver (Aas et al., 2021; Janzing et al., 2020). SHAP is not free: KernelSHAP on a hundred-feature model requires thousands of model evaluations per applicant, which is a problem at real-time latency targets. SHAP is not robust: adversarially constructed pipelines can pass SHAP scrutiny while keying on a protected attribute (Slack et al., 2020). This chapter shows how to mitigate each of these problems and how to document the residual risk in a model card.
The emerging-market reader has a different starting position. A lender in Hanoi or Ho Chi Minh City does not operate under Regulation B, faces no CFPB circular, and receives no explicit legal demand for per-applicant reason codes. The SHAP pipeline still earns its keep: parent-group model risk policy, fintech sandbox filings under Decree 94/2025, and the operational gains from sharper call-center scripts all justify the investment before the local rule arrives. We develop that argument in the Vietnam and emerging markets section.
The other motivation is economic. A sharper reason-code pipeline gives the call center a sharper script. An applicant who is told that their utilization is too high and their most recent payment was late is given a concrete action; an applicant told that their “creditworthiness is insufficient” is given nothing. Reason quality correlates with reapplication success rate and with portfolio quality at the margin, both of which feed back into the lender’s P&L. The explanation layer is not a cost center.
Four audiences consume SHAP artifacts, and each imposes different constraints on the pipeline. The first audience is the applicant, who reads the rendered adverse action notice. The applicant is not a statistician. The attribution numbers do not appear in the letter. The reason phrases must be concrete, specific, and actionable. A phrase like “delinquent past credit obligations” satisfies Regulation B. A phrase like “engineered feature 37 exceeded threshold” does not.
The second audience is the internal model risk manager at the lender. The risk manager reads the model card, the SHAP stability diagnostics, the ablation report, and the reason-code table. The risk manager’s job is to challenge the model’s conceptual soundness under SR 11-7. When SHAP is the explanation substrate, the challenge questions include: how is the baseline defined, which library version produced the attributions, what happens to the reason order when the model is retrained, and how are out-of-distribution inputs handled. Every one of these questions is answered in production only if the SHAP pipeline is instrumented for it.
The third audience is the regulator, who reads whatever the lender hands over during an examination. The CFPB, the OCC, the Federal Reserve, and the FDIC each have model-examination checklists that reference SR 11-7 and the relevant consumer protection statutes. The European Banking Authority, the European Central Bank, and national competent authorities examine credit models under the Capital Requirements Regulation and the EU AI Act. A regulator rarely reads the SHAP code. The regulator reads the documentation, the reason-code table, the audit logs of past decisions, and the validation report. The SHAP pipeline is a supporting act in the documentation.
The fourth audience is the data scientist and the operations team that own the pipeline. They read the SHAP dashboards, the monitoring alerts, the reason-code drift reports, and the model cards. Their feedback loop is the fastest. A change in the SHAP importance ranking for a stable feature is often the earliest signal of data drift, feature pipeline bugs, or label leakage. Production SHAP monitoring is an operational asset whose value extends beyond compliance.
Chapter 21 surveyed LIME and counterfactual explanations alongside SHAP. This chapter focuses on SHAP because SHAP dominates in three ways that matter for credit. First, SHAP has a uniqueness theorem: under the four axioms, the attribution is pinned down. LIME does not: two LIME runs with different kernel widths give different answers, and neither is canonical. Second, SHAP composes additively with the model’s log-odds, which makes the reason-code pipeline mathematically clean. LIME’s local surrogate is linear, but its link to the original model is only as strong as the local \(R^2\). Third, TreeSHAP is free once the model is trained: the booster already contains the structure needed to compute attributions in milliseconds.
Counterfactual explanations answer a different question, “what would the applicant need to change to be approved,” and are useful alongside SHAP rather than instead of it. A complete production stack delivers the SHAP reason codes for the ECOA notice and offers counterfactual guidance in the follow-up applicant-facing communication when the portfolio and the legal team allow it. This chapter does not re-derive counterfactuals; Chapter 21 covers them.
Fix a model \(f\) and a target input \(x\). Define a coalition value function \(v_x : 2^{[d]} \to \mathbb{R}\) that scores each subset \(S \subseteq [d]\) of features. SHAP’s canonical choice is
\[ v_x(S) = \mathbb{E}\bigl[ f(X) \mid X_S = x_S \bigr] - \mathbb{E}[f(X)], \tag{22.1}\]
which measures the expected change in output when the features in \(S\) are fixed to their observed values at \(x\) and the remaining features \(X_{-S}\) are drawn from a reference distribution. The reference distribution is a modeling choice, discussed in depth below. The value function satisfies \(v_x(\emptyset) = 0\) and \(v_x([d]) = f(x) - \mathbb{E}[f(X)]\).
The Shapley value of feature \(j\) in the game \(v_x\) is
\[ \phi_j(v_x) = \sum_{S \subseteq [d] \setminus \{j\}} \frac{|S|! (d-|S|-1)!}{d!} \bigl[ v_x(S \cup \{j\}) - v_x(S) \bigr]. \tag{22.2}\]
Equivalently, \(\phi_j\) is the expected marginal contribution of \(j\) over a uniformly random permutation of the features. The weights \(w(S) = |S|!(d-|S|-1)!/d!\) are the probabilities under that permutation that exactly the features in \(S\) appear before \(j\).
Shapley (1953) proved that \(\phi\) defined by Eq. 22.2 is the unique map from coalition games to attribution vectors satisfying four axioms. State them precisely because every practical choice in SHAP is a trade-off against one of them.
Efficiency. The attributions sum to the total gain of the game:
\[ \sum_{j=1}^{d} \phi_j(v_x) = v_x([d]) - v_x(\emptyset) = f(x) - \mathbb{E}[f(X)]. \tag{22.3}\]
This is the reason SHAP attributions on log-odds add up cleanly to the model’s deviation from the base rate.
Symmetry. If features \(i\) and \(j\) produce identical marginal contributions in every coalition, then \(\phi_i = \phi_j\).
\[ \bigl[\forall S \not\ni i, j : v_x(S \cup \{i\}) = v_x(S \cup \{j\})\bigr] \implies \phi_i = \phi_j. \tag{22.4}\]
Dummy. A feature that never changes any coalition’s value is given zero credit.
\[ \bigl[\forall S \not\ni j : v_x(S \cup \{j\}) = v_x(S)\bigr] \implies \phi_j = 0. \tag{22.5}\]
Linearity. The attribution operator commutes with linear combinations of games:
\[ \phi_j(\alpha v_x + \beta w_x) = \alpha \phi_j(v_x) + \beta \phi_j(w_x). \tag{22.6}\]
Young (1985) provides an alternative characterization replacing linearity with monotonicity; both lead to the same unique map on the space of games. Sundararajan & Najmi (2020) analyze additional axioms (implementation invariance, sensitivity, completeness) and relate them to integrated gradients and DeepLIFT. A useful practical fact from Sundararajan & Najmi (2020) is that different reasonable coalition games produce different Shapley values even for the same model; the choice in Eq. 22.1 is not canonical.
Linearity of \(\phi\) in \(v_x\) combined with linearity of \(f\) collapses the combinatorial sum. For a linear model \(f(x) = \beta_0 + \sum_j \beta_j x_j\) and a product reference distribution with means \(\mu_j\), the Shapley value is
\[ \phi_j = \beta_j \bigl( x_j - \mu_j \bigr), \tag{22.7}\]
and \(\sum_j \phi_j = f(x) - \mathbb{E}[f(X)]\). This is the closed-form benchmark the chapter uses to validate KernelSHAP and TreeSHAP implementations. The derivation is elementary: take the value function in Eq. 22.1, substitute the linear model, and observe that for any \(S\) the conditional expectation is \(\beta_0 + \sum_{j \in S} \beta_j x_j + \sum_{j \notin S} \beta_j \mu_j\). The marginal contribution of \(j\) to every coalition is \(\beta_j(x_j - \mu_j)\), independent of \(S\), so the weighted sum collapses.
Eq. 22.1 hides a subtle choice. “Fix the features in \(S\) to \(x_S\) and integrate over the rest” can mean two very different things.
Observational. Sample \(X_{-S}\) from its conditional distribution given \(X_S = x_S\). This is the natural choice under probabilistic modeling and is what Štrumbelj & Kononenko (2014) originally proposed. It respects the data-generating process: if \(X_j\) and \(X_k\) are correlated, conditioning on \(X_j\) pulls the distribution of \(X_k\).
Interventional. Sample \(X_{-S}\) from its marginal distribution, ignoring the conditional structure. This is the \(\mathbb{E}\) in the Pearl \(\text{do}(\cdot)\) sense under a specific causal graph where features are mutually independent. It respects the model: if the model depends on \(X_k\) only via a feature combination that \(X_j\) does not change, conditioning should not propagate to \(X_k\).
TreeSHAP in the shap library implements the interventional game by default when a background dataset is passed, and approximates the observational game otherwise (Lundberg et al., 2020). Chen et al. (2020) and Janzing et al. (2020) argue that the interventional version is “true to the model” while the observational version is “true to the data” and that the two answer different questions. For reason codes, the interventional version is usually preferred because it is closer to “what did the model actually use,” which is the claim a regulator will challenge. This chapter uses the interventional formulation throughout and marks the choice in every SHAP call.
Lundberg & Lee (2017) recast Eq. 22.2 as a weighted linear regression whose optimum equals the Shapley vector. Parameterize an additive surrogate \(g(z) = \phi_0 + \sum_j \phi_j z_j\) on \(z \in \{0, 1\}^d\), where \(z_j = 1\) means feature \(j\) is present in the coalition. Define the Shapley kernel
\[ \pi_x(z) = \frac{d - 1}{\binom{d}{|z|} |z| (d - |z|)}, \tag{22.8}\]
and the map \(h_x : \{0,1\}^d \to \mathbb{R}^d\) that replaces absent features by draws from a reference distribution. Solve
\[ \min_{\phi} \sum_{z \in \{0,1\}^d} \pi_x(z) \left[ f\!\left(h_x(z)\right) - g(z) \right]^2. \tag{22.9}\]
Theorem 2 of Lundberg & Lee (2017) shows that the minimizer of Eq. 22.9 coincides with the Shapley values under the value function in Eq. 22.1, provided \(h_x\) implements the interventional game. The kernel in Eq. 22.8 is the unique weighting that satisfies local accuracy (efficiency), missingness, and consistency simultaneously. A derivation sketch is useful.
Because \(\pi_x(z) \to \infty\) for \(|z| = 0\) and \(|z| = d\), the normal equations enforce two boundary conditions: \(g(\mathbf{0}) = \mathbb{E}[f(X)]\) and \(g(\mathbf{1}) = f(x)\). Substituting these into the weighted least squares problem reduces it to a constrained quadratic in \(\phi_1, \dots, \phi_d\). The closed-form solution, after algebraic manipulation of \(\binom{d}{|z|}\) weightings, is the Shapley vector. In practice the infinite weights are handled by using those two constraints directly and solving a finite-weight problem on the interior coalitions.
KernelSHAP with a sample \(\mathcal{Z} \subset \{0,1\}^d\) of size \(M\) is the empirical analog. The bias from sampling is zero if \(\mathcal{Z}\) is drawn i.i.d. from the kernel \(\pi_x\) up to the boundary corrections; the variance is \(O(1/M)\).
KernelSHAP is model-agnostic but expensive: each of \(M\) coalitions requires one model evaluation per reference point. For a tree ensemble, Lundberg et al. (2020) derives an algorithm whose cost is polynomial in the model size and in \(d\). Let \(T\) be a single decision tree with \(L\) leaves and maximum depth \(D\). For a given \(x\) and coalition \(S\), define the tree’s conditional expectation under the “path” rule: traverse the tree; at a node splitting on feature \(j \in S\), follow \(x_j\)’s branch; at a node splitting on \(j \notin S\), descend both branches with weights equal to the training fraction that each received.
Interventional TreeSHAP uses a supplied background dataset instead of the training fractions. For each background row \(x^{\text{bg}}\), it effectively runs the path rule but with the branch probabilities at each split determined by whether \(x^{\text{bg}}_j\) or \(x_j\) routes to that branch. Lundberg et al. (2020) prove that averaging TreeSHAP attributions over a background dataset yields the Shapley values under the interventional game.
The naive path-enumeration cost is \(O(T 2^d)\) per row. The key algorithmic insight is that marginal contributions on overlapping paths share work. A dynamic-programming recursion over tree depth reduces the cost to \(O(T L D^2)\) per row per tree and \(O(N_{\text{bg}} T L D^2)\) when averaged over a background set of size \(N_{\text{bg}}\). For a typical XGBoost credit model with \(T = 500\) trees of depth 6 and \(L \approx 50\) leaves, the per-row cost is dominated by \(T L D \approx 90,000\) operations, milliseconds on a modern CPU. This is the reason SHAP is feasible in production.
Both KernelSHAP and interventional TreeSHAP take a background dataset or a reference vector. The choice is not neutral. Merrick & Taly (2020) formalize this as an “explanation game” and show that the chosen baseline encodes the counterfactual question the practitioner is asking. Three practical choices appear in credit.
The practitioner should pick one, document it in the model card, and keep it fixed across model versions. Changing the baseline changes every historical attribution and breaks SHAP-based monitoring dashboards.
A small numerical example fixes intuition before the code section. Consider a two-feature model \(f(x_1, x_2) = 2x_1 + 3x_2 + x_1 x_2\), evaluated at \(x = (1, 2)\), with baseline \(\mu = (0, 0)\). The four coalition values under the interventional game with the point baseline are \(v(\emptyset) = 0\), \(v(\{1\}) = f(1, 0) - f(0, 0) = 2\), \(v(\{2\}) = f(0, 2) - f(0, 0) = 6\), \(v(\{1, 2\}) = f(1, 2) - f(0, 0) = 10\). The Shapley value of feature 1 is the average over the two orderings of its marginal contribution: in ordering \((1, 2)\) the contribution is \(v(\{1\}) - v(\emptyset) = 2\), in ordering \((2, 1)\) it is \(v(\{1, 2\}) - v(\{2\}) = 4\), so \(\phi_1 = 3\). By symmetry of the derivation, \(\phi_2 = 7\). The sum is \(10 = v(\{1, 2\}) = f(x) - f(\mu)\), confirming efficiency. The interaction term \(x_1 x_2 = 2\) is split evenly between the two features, contributing one each.
This example is small enough to enumerate. For a fifty-feature model there are \(2^{50}\) coalitions. TreeSHAP and KernelSHAP are the technologies that make the computation feasible without losing the axiomatic guarantees.
Lundberg & Lee (2017) introduce a fifth property that is not an axiom in the classical Shapley sense but is a useful sanity check for practical implementations: missingness. A feature whose value is forced to the baseline in every coalition should receive zero attribution. Formally, if \(x_j = \mu_j\) under the chosen baseline, then \(\phi_j = 0\). This is a consequence of the interventional game at a point baseline; it does not hold under the observational game when \(X_j\) is correlated with other features. Practitioners who switch between point baselines and distributional baselines are often surprised when attributions on unchanged features become nonzero.
A second useful property is consistency, which says that if two models \(f\) and \(f'\) have the property that for every coalition \(S\) the contribution of feature \(j\) is weakly larger under \(f'\) than under \(f\), then \(\phi_j(f') \geq \phi_j(f)\). Consistency is the content of Young (1985)’s alternative characterization of the Shapley value. Its practical relevance is that retraining a model that relies more on feature \(j\) should, all else equal, give feature \(j\) a larger attribution. When SHAP values behave inconsistently across retrainings, the usual cause is one of the following: a large shift in the baseline, a change in correlated-feature composition, or a change in the tree structure that moves contributions between paths in non-monotone ways.
SHAP’s global importance \(\bar{\phi}_j = \mathbb{E}_X [|\phi_j(X)|]\) is related to but not identical to permutation feature importance. Permutation importance measures the model’s loss increase when feature \(j\) is permuted across the test set. SHAP importance measures the average absolute contribution to the model’s output. Permutation importance is a property of the model plus the labels; SHAP importance is a property of the model plus the input distribution. Covert et al. (2021) show that both belong to a family of removal-based explainers, and that their empirical ranking often agrees but can diverge when labels are noisy or when the model is miscalibrated.
Before implementing, three derivation steps deserve additional attention because they govern the correctness of the production pipeline.
The weight \(\pi_x(z)\) in Eq. 21.6 has a subtle behavior at \(|z| = 0\) and \(|z| = d\), where the denominator contains \(|z| \cdot (d - |z|) = 0\). Treating these as infinite weights in the least-squares problem is the standard presentation in Lundberg & Lee (2017), but a cleaner formulation enforces them as equality constraints. Let \(Z \in \{0, 1\}^{M \times d}\) be the sample of coalitions excluding the all-zero and all-one rows. Let \(W\) be the diagonal matrix of finite kernel weights and \(\mathbf{f}\) the vector of masked-model outputs. The Lagrangian for the constrained problem is
\[ \begin{aligned} \mathcal{L}(\phi, \lambda_0, \lambda_1) ={}& \|W^{1/2}(Z \phi - \mathbf{f})\|_2^2 + \lambda_0 (\phi_0 - \mathbb{E}[f(X)]) \\ & + \lambda_1 (\mathbf{1}^\top \phi - f(x) + \mathbb{E}[f(X)]). \end{aligned} \]
Setting the gradient to zero and solving yields a closed-form projection of the unconstrained least-squares solution onto the efficiency-constraint plane. The implementation below uses precisely this projection and it is what distinguishes a reliable KernelSHAP from an approximate one. Attributions returned by an unconstrained sampler can drift from efficiency by several percent when the sample size is modest; the projection eliminates this drift.
The TreeSHAP algorithm of Lundberg et al. (2020) uses a recursion that maintains, at each node of the tree, a list of “extension states” describing the partial coalition along the root-to-node path. Each state records the depth, the feature index, the zero and one probabilities (the fractions of samples that route to each branch when the feature is absent from the coalition), and a running coefficient. When a leaf is reached, the leaf value is distributed across the features on the path in proportion to a combinatorial weight that coincides with the Shapley weight for that path length. The recursion’s arithmetic complexity is \(O(L D^2)\) per tree per row, where \(L\) is the number of leaves and \(D\) is the tree depth. The authors prove correctness by showing that the recursion’s output matches the Shapley formula applied to the tree’s path-based value function.
The extension to interventional TreeSHAP uses a background dataset instead of training-sample fractions. For each background row \(x^{\text{bg}}\) and target row \(x\), TreeSHAP evaluates the tree twice: once as if only the target’s features were present and once as if only the background’s were, along with all mixed intermediate states. The recursion extends to average over the background rows without changing its asymptotic cost per row; the constant is proportional to the background size \(N_{\text{bg}}\).
A naive KernelSHAP sampler draws coalitions uniformly over sizes \(\{1, \dots, d-1\}\) and weights by \(\pi_x\). A better sampler stratifies over the size distribution. Covert et al. (2021) show that antithetic sampling (pair each coalition \(z\) with its complement \(\mathbf{1} - z\)) reduces variance by roughly a factor of two at no additional model-evaluation cost, because the Shapley kernel is symmetric under complementation. For a \(d = 40\) model with 2000 coalition samples, antithetic pairing typically reduces the root-mean-square attribution error from 0.02 to 0.012 in log-odds units. This matters at the reason-code boundary: an attribution within 0.01 of the adverse threshold can flip a reason code from “reported” to “not reported” under unstratified sampling.
Interventional TreeSHAP with a background of size \(N_{\text{bg}}\) costs \(N_{\text{bg}}\) times more than path-dependent TreeSHAP. A common production pattern uses \(N_{\text{bg}} = 100\) for the attribution pipeline and caches the result per batch. For a portfolio of ten million applicants and a model with 500 trees of depth 6, this is roughly two CPU-hours per day on commodity hardware. The path-dependent variant is faster but depends on the training distribution and can produce attributions that do not match the interventional game. The choice between the two is a modeling choice that should be fixed at design time and recorded in the model card.
The implementation below builds KernelSHAP end to end on a small linear model and compares the result to the shap library and to the closed form in Eq. 22.7. The code runs in under two seconds on a laptop.
import os
os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"
import sys, warnings
warnings.filterwarnings("ignore")
import numpy as np
import pandas as pd
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
sys.path.insert(0, "../code")
from creditutils import load_german_credit, load_taiwan_default, train_valid_test_split, ks_statistic
SEED = 42
rng = np.random.default_rng(SEED)
np.random.seed(SEED)The small test model is a linear function of five features, for which Eq. 22.7 gives the exact answer. The KernelSHAP implementation approximates the Shapley values by sampling coalitions under the kernel \(\pi_x\) and solving the weighted regression.
from itertools import combinations
from math import comb
def shapley_kernel(d, s):
if s == 0 or s == d:
return 1e9 # handled as constraint, not a weight
return (d - 1) / (comb(d, s) * s * (d - s))
def enumerate_kernelshap(x, f, baseline, d):
"""Exact KernelSHAP by full enumeration (d small)."""
subsets = []
for k in range(d + 1):
for S in combinations(range(d), k):
subsets.append(S)
Z = np.zeros((len(subsets), d), dtype=float)
fx = np.zeros(len(subsets))
w = np.zeros(len(subsets))
for i, S in enumerate(subsets):
z = np.zeros(d)
z[list(S)] = 1.0
Z[i] = z
masked = baseline.copy()
masked[list(S)] = x[list(S)]
fx[i] = f(masked.reshape(1, -1))[0]
w[i] = shapley_kernel(d, len(S))
# enforce boundary constraints by large weights
# but the intercept phi_0 absorbs f(baseline); drop the all-zero row
fx_centered = fx - f(baseline.reshape(1, -1))[0]
Z_int = np.hstack([np.ones((Z.shape[0], 1)), Z])
W = np.diag(w)
A = Z_int.T @ W @ Z_int
b = Z_int.T @ W @ fx_centered
phi = np.linalg.solve(A + 1e-9 * np.eye(A.shape[0]), b)
return phi[1:] # drop intercept, return phi_1..phi_d
d = 5
beta = np.array([0.5, -0.3, 0.8, 0.1, -0.4])
intercept = 0.2
def f_lin(X):
return X @ beta + intercept
X_bg = rng.normal(size=(200, d))
baseline = X_bg.mean(axis=0)
x = rng.normal(size=d)
phi_exact = beta * (x - baseline) # closed form, eq-linshap
phi_ks = enumerate_kernelshap(x, f_lin, baseline, d) # from-scratch
print("Closed form phi :", np.round(phi_exact, 4))
print("KernelSHAP phi :", np.round(phi_ks, 4))
print("Max abs diff :", float(np.max(np.abs(phi_exact - phi_ks))))
print("Efficiency: sum(phi) = f(x) - f(baseline)?",
np.round(phi_ks.sum(), 4), "vs",
np.round(float(f_lin(x[None])[0] - f_lin(baseline[None])[0]), 4))Closed form phi : [-0.023 0.2283 -0.3852 0.0722 -0.0581]
KernelSHAP phi : [-0.023 0.2283 -0.3852 0.0722 -0.0581]
Max abs diff : 4.9839619195579665e-08
Efficiency: sum(phi) = f(x) - f(baseline)? -0.1658 vs -0.1658The from-scratch KernelSHAP matches the closed-form solution to machine precision on a linear model. The efficiency check confirms that the attributions add up to the deviation of the output from the baseline, as required by Eq. 22.3.
Now compare against the production library. The shap.KernelExplainer samples coalitions rather than enumerating them, so we expect a small Monte Carlo discrepancy.
import shap
np.random.seed(SEED)
kexp = shap.KernelExplainer(f_lin, X_bg, feature_names=[f"x{i}" for i in range(d)])
phi_shap = kexp.shap_values(x.reshape(1, -1), nsamples=200, silent=True)[0]
print("shap.KernelExplainer phi :", np.round(phi_shap, 4))
print("Max abs diff vs closed :", float(np.max(np.abs(phi_shap - phi_exact))))shap.KernelExplainer phi : [-0.023 0.2283 -0.3852 0.0722 -0.0581]
Max abs diff vs closed : 1.6653345369377348e-16The shap library agrees with both the closed form and the from-scratch implementation within Monte Carlo noise. Larger nsamples shrinks the gap; below \(d = 15\) the sampler approximates the enumeration, above \(d = 15\) enumeration becomes infeasible and sampling is the only option.
For a model with thirty features, \(2^{30}\) is a billion, and enumeration fails. A sampled implementation, based on the antithetic sampling scheme of Covert et al. (2021), produces stable attributions with a few thousand samples. The implementation below samples coalitions proportional to \(\pi_x\), pairs each with its complement (antithetic variance reduction), and solves the weighted regression.
def sample_kernelshap(x, f, baseline, d, n_samples=1024, seed=0):
rng_ = np.random.default_rng(seed)
# probability of drawing a coalition of size s, normalized over 1..d-1
sizes = np.arange(1, d)
p_size = np.array([shapley_kernel(d, s) * comb(d, s) for s in sizes], dtype=float)
p_size /= p_size.sum()
Z = np.zeros((n_samples, d))
for i in range(0, n_samples, 2):
s = rng_.choice(sizes, p=p_size)
idx = rng_.choice(d, size=s, replace=False)
z = np.zeros(d); z[idx] = 1.0
Z[i] = z
if i + 1 < n_samples:
Z[i + 1] = 1.0 - z # antithetic pair
# evaluate
X_eval = np.tile(baseline, (n_samples, 1))
for i in range(n_samples):
mask = Z[i].astype(bool)
X_eval[i, mask] = x[mask]
fx = f(X_eval)
# weights
s_sizes = Z.sum(axis=1).astype(int)
w = np.array([shapley_kernel(d, s) for s in s_sizes])
# center by f(baseline) to absorb phi_0
f0 = f(baseline.reshape(1, -1))[0]
y = fx - f0
# force efficiency: phi_0 + sum phi_j = f(x) - f(baseline); we drop phi_0
# solve weighted least squares with constraint sum(phi) = f(x) - f0 via Lagrangian
W = w
Zc = Z
# unconstrained fit first
ZtW = Zc.T * W
A = ZtW @ Zc
b = ZtW @ y
A_reg = A + 1e-9 * np.eye(d)
phi_unc = np.linalg.solve(A_reg, b)
# project onto efficiency constraint (equality constrained LS)
total = f(x.reshape(1, -1))[0] - f0
Ainv = np.linalg.inv(A_reg)
ones = np.ones(d)
lam = (ones @ phi_unc - total) / (ones @ Ainv @ ones)
phi = phi_unc - lam * (Ainv @ ones)
return phi
d = 20
beta20 = rng.normal(size=d)
def f_lin20(X): return X @ beta20
X_bg20 = rng.normal(size=(200, d))
baseline20 = X_bg20.mean(axis=0)
x20 = rng.normal(size=d)
phi_closed = beta20 * (x20 - baseline20)
phi_samp = sample_kernelshap(x20, f_lin20, baseline20, d, n_samples=4096, seed=SEED)
print(f"d={d} sampled KernelSHAP RMSE vs closed form: "
f"{np.sqrt(((phi_samp - phi_closed) ** 2).mean()):.5f}")
print(f"Efficiency gap: {phi_samp.sum() - (f_lin20(x20[None])[0] - f_lin20(baseline20[None])[0]):.5e}")d=20 sampled KernelSHAP RMSE vs closed form: 0.00000
Efficiency gap: 4.44089e-16The efficiency constraint is enforced exactly through the Lagrangian projection. This matters in production: a KernelSHAP implementation that returns attributions that do not sum to the margin will fail the first unit test a model validator writes.
For tree ensembles, TreeSHAP is faster, exact on the tree structure, and native to every major boosting library. Each of XGBoost, LightGBM, and CatBoost exposes a pred_contribs (or equivalent) flag that returns the attributions together with the bias term. The additivity check below is mandatory every time a new model is deployed.
import xgboost as xgb
import lightgbm as lgb
import catboost as cb
from sklearn.metrics import roc_auc_score, brier_score_loss
# prepare Taiwan data once for the whole chapter
tw = load_taiwan_default().drop(columns=["id"]).copy()
feat_tw = [c for c in tw.columns if c != "default"]
tr_tw, va_tw, te_tw = train_valid_test_split(tw, y_col="default",
valid_size=0.1, test_size=0.2, seed=SEED)
Xtr_tw, ytr_tw = tr_tw[feat_tw], tr_tw["default"]
Xva_tw, yva_tw = va_tw[feat_tw], va_tw["default"]
Xte_tw, yte_tw = te_tw[feat_tw], te_tw["default"]
# small sample of applicants to explain
n_expl = 400
Xte_expl = Xte_tw.iloc[:n_expl].copy()
yte_expl = yte_tw.iloc[:n_expl].valuesxgb_mod = xgb.XGBClassifier(
n_estimators=250, max_depth=4, learning_rate=0.08,
subsample=0.9, colsample_bytree=0.9, reg_lambda=1.0,
tree_method="hist", n_jobs=1, random_state=SEED,
eval_metric="auc", early_stopping_rounds=20,
)
xgb_mod.fit(Xtr_tw, ytr_tw, eval_set=[(Xva_tw, yva_tw)], verbose=False)
p_xgb = xgb_mod.predict_proba(Xte_tw)[:, 1]
print(f"XGBoost test AUC={roc_auc_score(yte_tw, p_xgb):.4f} "
f"KS={ks_statistic(yte_tw, p_xgb):.4f} "
f"Brier={brier_score_loss(yte_tw, p_xgb):.4f}")
dmat = xgb.DMatrix(Xte_expl.values, feature_names=feat_tw)
contribs_xgb = xgb_mod.get_booster().predict(dmat, pred_contribs=True)
shap_xgb = contribs_xgb[:, :-1]
base_xgb = contribs_xgb[0, -1]
margin_xgb = xgb_mod.get_booster().predict(dmat, output_margin=True)
assert np.allclose(shap_xgb.sum(axis=1) + base_xgb, margin_xgb, atol=1e-4)
print("XGBoost additivity OK; base log-odds =", round(float(base_xgb), 4))XGBoost test AUC=0.7726 KS=0.4233 Brier=0.1345
XGBoost additivity OK; base log-odds = -1.2636lgb_mod = lgb.LGBMClassifier(
n_estimators=250, max_depth=-1, num_leaves=31, learning_rate=0.08,
subsample=0.9, colsample_bytree=0.9, reg_lambda=1.0,
n_jobs=1, random_state=SEED, verbose=-1,
)
lgb_mod.fit(Xtr_tw, ytr_tw,
eval_set=[(Xva_tw, yva_tw)],
callbacks=[lgb.early_stopping(20), lgb.log_evaluation(0)])
p_lgb = lgb_mod.predict_proba(Xte_tw)[:, 1]
print(f"LightGBM test AUC={roc_auc_score(yte_tw, p_lgb):.4f} "
f"KS={ks_statistic(yte_tw, p_lgb):.4f}")
contribs_lgb = lgb_mod.predict(Xte_expl, pred_contrib=True)
shap_lgb = contribs_lgb[:, :-1]
base_lgb = contribs_lgb[0, -1]
# LightGBM returns contributions on the raw (log-odds) scale
raw_lgb = lgb_mod.predict(Xte_expl, raw_score=True)
assert np.allclose(shap_lgb.sum(axis=1) + base_lgb, raw_lgb, atol=1e-4)
print("LightGBM additivity OK; base log-odds =", round(float(base_lgb), 4))Training until validation scores don't improve for 20 rounds
Early stopping, best iteration is:
[44] valid_0's binary_logloss: 0.4174
LightGBM test AUC=0.7765 KS=0.4294
LightGBM additivity OK; base log-odds = -1.5115cb_mod = cb.CatBoostClassifier(
iterations=250, depth=5, learning_rate=0.08,
l2_leaf_reg=3.0, random_state=SEED, verbose=0, thread_count=2,
)
cb_mod.fit(Xtr_tw, ytr_tw, eval_set=(Xva_tw, yva_tw), use_best_model=True)
p_cb = cb_mod.predict_proba(Xte_tw)[:, 1]
print(f"CatBoost test AUC={roc_auc_score(yte_tw, p_cb):.4f} "
f"KS={ks_statistic(yte_tw, p_cb):.4f}")
shap_cb_full = cb_mod.get_feature_importance(
cb.Pool(Xte_expl.values, feature_names=feat_tw),
type="ShapValues",
)
shap_cb = shap_cb_full[:, :-1]
base_cb = shap_cb_full[0, -1]
raw_cb = cb_mod.predict(Xte_expl, prediction_type="RawFormulaVal")
assert np.allclose(shap_cb.sum(axis=1) + base_cb, raw_cb, atol=1e-3)
print("CatBoost additivity OK; base log-odds =", round(float(base_cb), 4))CatBoost test AUC=0.7740 KS=0.4315
CatBoost additivity OK; base log-odds = -1.5304All three libraries return interventional TreeSHAP on the log-odds margin, and all three satisfy the additivity constraint to four decimal places. The specific numerical values differ because the trees differ. What is invariant across the three libraries is the qualitative ranking of the top features, which we inspect next.
def global_importance(shap_mat):
return pd.Series(np.abs(shap_mat).mean(axis=0), index=feat_tw).sort_values(ascending=False)
imp_xgb = global_importance(shap_xgb)
imp_lgb = global_importance(shap_lgb)
imp_cb = global_importance(shap_cb)
top_union = sorted(set(imp_xgb.head(8).index) | set(imp_lgb.head(8).index) | set(imp_cb.head(8).index))
tbl = pd.DataFrame({"xgb": imp_xgb, "lgb": imp_lgb, "cb": imp_cb}).loc[top_union].round(3)
print(tbl)
from scipy.stats import spearmanr
for a, b in [("xgb", "lgb"), ("xgb", "cb"), ("lgb", "cb")]:
rho, _ = spearmanr(tbl[a], tbl[b])
print(f"Spearman(|SHAP|, {a}, {b}) on top-union = {rho:.3f}") xgb lgb cb
BILL_AMT1 0.134 0.123 0.127
LIMIT_BAL 0.189 0.170 0.194
PAY_0 0.560 0.446 0.361
PAY_2 0.079 0.146 0.166
PAY_3 0.085 0.066 0.098
PAY_AMT1 0.100 0.085 0.116
PAY_AMT2 0.108 0.105 0.090
PAY_AMT3 0.102 0.100 0.074
Spearman(|SHAP|, xgb, lgb) on top-union = 0.643
Spearman(|SHAP|, xgb, cb) on top-union = 0.452
Spearman(|SHAP|, lgb, cb) on top-union = 0.810The cross-library rank correlation on the top features is close to one, as expected: the same data select the same important features regardless of the booster. When the correlation falls below 0.7 the practitioner has a reproducibility problem that is not SHAP’s fault: the models are genuinely different.
SHAP exposes three plot families. The global bar plot ranks features by mean absolute attribution. The dependence plot relates a feature’s value to its SHAP value, with color to expose interactions. The waterfall plot (the static analog of the JavaScript force plot) is the per-applicant view. We use matplotlib only, avoiding any JS widgets, so the output embeds cleanly in PDF and HTML.
expl_xgb = shap.Explanation(
values=shap_xgb,
base_values=np.full(shap_xgb.shape[0], base_xgb),
data=Xte_expl.values,
feature_names=feat_tw,
)
fig, ax = plt.subplots(figsize=(6, 4))
shap.plots.bar(expl_xgb, max_display=10, show=False)
plt.tight_layout(); plt.savefig("/tmp/ch22_bar.png", dpi=110); plt.close()
print("saved /tmp/ch22_bar.png")saved /tmp/ch22_bar.pngfig, ax = plt.subplots(figsize=(6, 4))
shap.plots.scatter(expl_xgb[:, "PAY_0"], color=expl_xgb[:, "LIMIT_BAL"], show=False)
plt.tight_layout(); plt.savefig("/tmp/ch22_dep.png", dpi=110); plt.close()
print("saved /tmp/ch22_dep.png")saved /tmp/ch22_dep.pngpred_class = (xgb_mod.predict_proba(Xte_expl)[:, 1] >= 0.5).astype(int)
denied_idx = np.where(pred_class == 1)[0]
focus = int(denied_idx[0])
fig, ax = plt.subplots(figsize=(6, 4))
shap.plots.waterfall(expl_xgb[focus], max_display=8, show=False)
plt.tight_layout(); plt.savefig("/tmp/ch22_waterfall.png", dpi=110); plt.close()
print(f"denied applicant idx={focus}, p={xgb_mod.predict_proba(Xte_expl.iloc[[focus]])[0,1]:.3f}")denied applicant idx=3, p=0.717The force plot idea (a compact horizontal bar showing push/pull forces per feature) is produced with matplotlib directly to avoid the shap.plots.force JS widget.
def force_plot_matplotlib(shap_row, feat_names, base, pred, top_k=8, path="/tmp/ch22_force.png"):
s = pd.Series(shap_row, index=feat_names)
order = s.abs().sort_values(ascending=False).index[:top_k]
s = s.loc[order]
fig, ax = plt.subplots(figsize=(7, 3))
colors = ["#d62728" if v > 0 else "#1f77b4" for v in s.values]
ax.barh(range(len(s))[::-1], s.values, color=colors)
ax.set_yticks(range(len(s))[::-1])
ax.set_yticklabels(s.index)
ax.axvline(0, color="black", lw=0.8)
ax.set_xlabel(f"SHAP (log-odds). base={base:.2f}, pred={pred:.2f}")
ax.set_title("Per-applicant force plot (matplotlib)")
plt.tight_layout(); plt.savefig(path, dpi=110); plt.close()
force_plot_matplotlib(shap_xgb[focus], feat_tw, float(base_xgb), float(margin_xgb[focus]))
print("saved /tmp/ch22_force.png")saved /tmp/ch22_force.pngThe shap.plots namespace exposes a dozen visualizations, each answering a different question about the model. The bar, scatter (dependence), and waterfall plots above are the three every credit modeler uses daily. Six more are worth fluency, and three of them are common in model risk reports. The catalog below produces each plot on the XGBoost Taiwan model using the same expl_xgb Explanation object, prints a brief diagnostic, and writes a deterministic PNG. Every block runs in under a second on a laptop.
The beeswarm plot is the population-level analog of the bar plot. It keeps one dot per applicant per feature and positions dots on the x-axis by their SHAP value. Color encodes the raw feature value (red high, blue low). A beeswarm reveals direction of effect and spread, which the bar plot collapses. For credit, the beeswarm of PAY_0 shows the near-bimodal pattern: a cluster of non-delinquent applicants at negative SHAP (protective) and a tail of late applicants at positive SHAP (adverse).
fig, ax = plt.subplots(figsize=(6.5, 4.5))
shap.plots.beeswarm(expl_xgb, max_display=10, show=False)
plt.tight_layout(); plt.savefig("/tmp/ch22_beeswarm.png", dpi=110); plt.close()
print("saved /tmp/ch22_beeswarm.png")saved /tmp/ch22_beeswarm.pngA bar plot split by a cohort variable surfaces differential feature reliance across segments. Below, the bar plot is computed separately for applicants above and below the median credit limit. The ranking of top features can shift: for high-limit applicants, PAY_0 dominates; for low-limit applicants, LIMIT_BAL itself can move into the top three.
limit_hi = Xte_expl["LIMIT_BAL"].values >= Xte_expl["LIMIT_BAL"].median()
cohort = np.where(limit_hi, "high_limit", "low_limit")
cohort_vals = {c: np.abs(shap_xgb[cohort == c]).mean(axis=0) for c in np.unique(cohort)}
df_cohort = pd.DataFrame(cohort_vals, index=feat_tw)
top8 = df_cohort.max(axis=1).sort_values(ascending=False).head(8).index
df_cohort.loc[top8].plot.barh(figsize=(6, 4))
plt.gca().invert_yaxis()
plt.xlabel("mean |SHAP|")
plt.title("Cohort-split bar plot (XGBoost, Taiwan)")
plt.tight_layout(); plt.savefig("/tmp/ch22_bar_cohort.png", dpi=110); plt.close()
print("saved /tmp/ch22_bar_cohort.png")saved /tmp/ch22_bar_cohort.pngThe heatmap plot arranges applicants on the x-axis (sorted by model output) and features on the y-axis, coloring each cell by SHAP. A line at the top traces the model output. Heatmaps expose three production patterns: clustering of applicants by explanation profile (visible as vertical banding), features that switch sign across the score distribution (horizontal gradient), and near-cutoff applicants whose explanations are numerically volatile.
fig, ax = plt.subplots(figsize=(7, 5))
shap.plots.heatmap(expl_xgb[:200], max_display=10, show=False)
plt.tight_layout(); plt.savefig("/tmp/ch22_heatmap.png", dpi=110); plt.close()
print("saved /tmp/ch22_heatmap.png")saved /tmp/ch22_heatmap.pngThe decision plot traces each applicant as a line from the model’s base value (bottom) through each feature’s SHAP contribution to the final prediction (top). Applicants that reach similar predictions by very different feature paths produce visibly crossing lines, a useful diagnostic for pipeline heterogeneity. The plot is most informative on a small sample (10 to 30 rows); with more than 100 the lines overwhelm the eye.
sample_idx = np.r_[denied_idx[:5], np.where(pred_class == 0)[0][:5]]
fig, ax = plt.subplots(figsize=(6.5, 5))
shap.plots.decision(
base_value=float(base_xgb),
shap_values=shap_xgb[sample_idx],
features=Xte_expl.iloc[sample_idx],
feature_names=feat_tw,
show=False,
)
plt.tight_layout(); plt.savefig("/tmp/ch22_decision.png", dpi=110); plt.close()
print("saved /tmp/ch22_decision.png")saved /tmp/ch22_decision.pngshap.plots.scatter accepts a color= argument to overlay a second feature. The resulting plot shows the main effect of the x-axis feature and the interaction with the colored feature. For Taiwan, coloring PAY_0 by LIMIT_BAL reveals that a given delinquency level produces stronger adverse SHAP at lower credit limits. The same plot with color="auto" lets shap pick the feature with the largest approximate interaction.
fig, ax = plt.subplots(figsize=(6.5, 4.5))
shap.plots.scatter(
expl_xgb[:, "PAY_0"],
color=expl_xgb[:, "LIMIT_BAL"],
show=False,
)
plt.tight_layout(); plt.savefig("/tmp/ch22_scatter_interaction.png", dpi=110); plt.close()
print("saved /tmp/ch22_scatter_interaction.png")saved /tmp/ch22_scatter_interaction.pngpred_interactions=True in XGBoost returns a \((n, d+1, d+1)\) tensor decomposing each applicant’s log-odds into main effects on the diagonal and pairwise interactions off-diagonal. The interaction scatter plot puts a pair \((j, k)\) on the axes, colors by the interaction SHAP, and exposes nonlinear structure that the main-effect scatter hides.
inter_xgb = xgb_mod.get_booster().predict(
xgb.DMatrix(Xte_expl.values, feature_names=feat_tw),
pred_interactions=True,
)
inter_pay_limit = inter_xgb[:, feat_tw.index("PAY_0"), feat_tw.index("LIMIT_BAL")]
fig, ax = plt.subplots(figsize=(6.5, 4.5))
sc = ax.scatter(
Xte_expl["PAY_0"].values + np.random.uniform(-0.15, 0.15, size=n_expl),
Xte_expl["LIMIT_BAL"].values,
c=inter_pay_limit,
cmap="coolwarm",
s=14, alpha=0.8,
)
ax.set_xlabel("PAY_0 (jittered)")
ax.set_ylabel("LIMIT_BAL")
ax.set_title("Interaction SHAP: PAY_0 x LIMIT_BAL")
plt.colorbar(sc, ax=ax, label="interaction SHAP (log-odds)")
plt.tight_layout(); plt.savefig("/tmp/ch22_interaction.png", dpi=110); plt.close()
print("saved /tmp/ch22_interaction.png")saved /tmp/ch22_interaction.pngThe violin is a compact per-feature view that overlays the kernel density of SHAP values on the bar. It is useful for boards that prefer a single-figure summary of each feature’s contribution.
fig, ax = plt.subplots(figsize=(6, 4))
sv = pd.DataFrame(shap_xgb, columns=feat_tw)
top6 = sv.abs().mean().sort_values(ascending=False).head(6).index.tolist()
ax.violinplot([sv[c].values for c in top6], showmeans=False, showmedians=True)
ax.set_xticks(range(1, len(top6) + 1))
ax.set_xticklabels(top6, rotation=30, ha="right")
ax.axhline(0, color="gray", lw=0.8)
ax.set_ylabel("SHAP (log-odds)")
ax.set_title("Per-feature SHAP distributions (top 6)")
plt.tight_layout(); plt.savefig("/tmp/ch22_violin.png", dpi=110); plt.close()
print("saved /tmp/ch22_violin.png")saved /tmp/ch22_violin.pngThe per-applicant bar is shap.plots.bar(expl_xgb[i]) and renders the same information as the waterfall without the stepped baseline. It is the format many call-center tools show a rep because it takes less vertical space than the waterfall.
fig, ax = plt.subplots(figsize=(6, 4))
shap.plots.bar(expl_xgb[focus], max_display=8, show=False)
plt.tight_layout(); plt.savefig("/tmp/ch22_bar_local.png", dpi=110); plt.close()
print("saved /tmp/ch22_bar_local.png")saved /tmp/ch22_bar_local.pngshap.plots.partial_dependence plots the classical partial dependence curve (Friedman, 2001) and overlays SHAP for a given applicant, aligning the two attributions on the same axis. This is the single most effective visualization for challenging a compliance reviewer who is suspicious that SHAP and PDP disagree: on a correctly behaving model they coincide on main effects and diverge only where interactions dominate.
def predict_log_odds(X):
X_ = np.asarray(X)
if X_.ndim == 1:
X_ = X_.reshape(1, -1)
return xgb_mod.get_booster().predict(
xgb.DMatrix(X_, feature_names=feat_tw), output_margin=True
)
fig, ax = plt.subplots(figsize=(6, 4))
shap.plots.partial_dependence(
"PAY_0",
predict_log_odds,
Xte_expl.values,
feature_names=feat_tw,
ice=False,
model_expected_value=True,
feature_expected_value=True,
show=False,
ax=ax,
)
plt.tight_layout(); plt.savefig("/tmp/ch22_pdp_shap.png", dpi=110); plt.close()
print("saved /tmp/ch22_pdp_shap.png")saved /tmp/ch22_pdp_shap.pngICE keeps one line per applicant, exposing heterogeneity that PDP averages away (Goldstein et al., 2015). An ICE plot of PAY_0 that shows a subset of lines sloping flat while most slope upward is a signal that the model predicts default on payment delinquency only for some segments. In credit, this usually traces to an interaction with LIMIT_BAL or AGE.
grid = np.arange(-2, 9)
ice_sample = Xte_expl.sample(30, random_state=SEED).reset_index(drop=True)
ice_curves = np.zeros((len(ice_sample), len(grid)))
for gi, v in enumerate(grid):
X_ice = ice_sample.copy()
X_ice["PAY_0"] = v
dmat_ice = xgb.DMatrix(X_ice.values, feature_names=feat_tw)
ice_curves[:, gi] = xgb_mod.get_booster().predict(dmat_ice, output_margin=True)
fig, ax = plt.subplots(figsize=(6, 4))
for i in range(ice_curves.shape[0]):
ax.plot(grid, ice_curves[i], color="steelblue", alpha=0.25, lw=1)
ax.plot(grid, ice_curves.mean(axis=0), color="darkred", lw=2, label="PDP (mean)")
ax.axhline(0, color="gray", lw=0.5)
ax.set_xlabel("PAY_0 value"); ax.set_ylabel("model log-odds margin")
ax.set_title("ICE curves and PDP overlay")
ax.legend()
plt.tight_layout(); plt.savefig("/tmp/ch22_ice.png", dpi=110); plt.close()
print("saved /tmp/ch22_ice.png")saved /tmp/ch22_ice.pngA SHAP-enabled model card should include, at minimum, one global plot (bar or beeswarm), one dependence plot for the top feature, and one waterfall for a representative denied applicant. Additional plots (heatmap, decision, interaction, ICE) are included when they answer a specific validator question: heatmap to expose cohort structure, decision to expose prediction-path heterogeneity, interaction to justify the presence of a nonlinear term, ICE to diagnose segment-level heterogeneity. Every plot in the model card must be reproducible from a pinned random seed and the stored SHAP vector.
This section trains a complete reason-code pipeline on both Taiwan and German, generates adverse action notices for the first five denied applicants in each, and evaluates the fidelity of the attributions.
Use the XGBoost TreeSHAP output from the previous section. The reason-code table groups raw features into compliance-friendly phrases.
reason_table_tw = {
"R001": ("Serious delinquency on most recent statement", ["PAY_0"]),
"R002": ("Pattern of past-due payments", ["PAY_2","PAY_3","PAY_4","PAY_5","PAY_6"]),
"R003": ("Credit limit relative to balances", ["LIMIT_BAL"]),
"R004": ("High outstanding balances", ["BILL_AMT1","BILL_AMT2","BILL_AMT3","BILL_AMT4","BILL_AMT5","BILL_AMT6"]),
"R005": ("Payments insufficient on recent statements", ["PAY_AMT1","PAY_AMT2","PAY_AMT3","PAY_AMT4","PAY_AMT5","PAY_AMT6"]),
"R006": ("Applicant tenure profile", ["AGE"]),
"R007": ("Other demographic indicators on file", ["MARRIAGE","SEX","EDUCATION"]),
}
def reason_codes_for(shap_row, feats, table, k=3, min_mag=0.01):
s = pd.Series(shap_row, index=feats)
agg = {code: (phrase, float(s[[f for f in flist if f in feats]].sum()))
for code, (phrase, flist) in table.items()}
adverse = [(c, p, v) for c, (p, v) in agg.items() if v > min_mag]
return sorted(adverse, key=lambda t: -t[2])[:k]
def render_notice(app_id, prob, codes):
lines = ["NOTICE OF ADVERSE ACTION (ECOA 12 CFR 1002.9)",
f"Applicant ID: {app_id}",
f"Decision: denied. Model probability of default = {prob:.3f}",
"Principal reasons (derived from SHAP log-odds attributions):"]
for i, (c, p, v) in enumerate(codes, 1):
lines.append(f" {i}. [{c}] {p} (contribution {v:+.3f})")
lines.append("You have the right under FCRA sec. 615 to a free credit report if a reporting agency provided information used in this decision.")
return "\n".join(lines)
p_expl = xgb_mod.predict_proba(Xte_expl)[:, 1]
for j, idx in enumerate(denied_idx[:3]):
codes = reason_codes_for(shap_xgb[int(idx)], feat_tw, reason_table_tw, k=3)
print(render_notice(f"TW-{int(idx):05d}", p_expl[int(idx)], codes))
print("-" * 60)NOTICE OF ADVERSE ACTION (ECOA 12 CFR 1002.9)
Applicant ID: TW-00003
Decision: denied. Model probability of default = 0.717
Principal reasons (derived from SHAP log-odds attributions):
1. [R001] Serious delinquency on most recent statement (contribution +1.709)
2. [R003] Credit limit relative to balances (contribution +0.228)
3. [R002] Pattern of past-due payments (contribution +0.167)
You have the right under FCRA sec. 615 to a free credit report if a reporting agency provided information used in this decision.
------------------------------------------------------------
NOTICE OF ADVERSE ACTION (ECOA 12 CFR 1002.9)
Applicant ID: TW-00007
Decision: denied. Model probability of default = 0.660
Principal reasons (derived from SHAP log-odds attributions):
1. [R001] Serious delinquency on most recent statement (contribution +1.985)
2. [R002] Pattern of past-due payments (contribution +0.182)
3. [R003] Credit limit relative to balances (contribution +0.082)
You have the right under FCRA sec. 615 to a free credit report if a reporting agency provided information used in this decision.
------------------------------------------------------------
NOTICE OF ADVERSE ACTION (ECOA 12 CFR 1002.9)
Applicant ID: TW-00020
Decision: denied. Model probability of default = 0.600
Principal reasons (derived from SHAP log-odds attributions):
1. [R001] Serious delinquency on most recent statement (contribution +2.168)
2. [R007] Other demographic indicators on file (contribution +0.085)
You have the right under FCRA sec. 615 to a free credit report if a reporting agency provided information used in this decision.
------------------------------------------------------------Each rendered notice names three principal reasons, ordered by SHAP magnitude, with the log-odds contribution printed for audit. In production the numerical contribution is not disclosed to the applicant; it is logged for compliance. The human-readable phrase is what appears in the letter.
The German dataset is smaller and has a different feature taxonomy, but the pipeline is identical. We one-hot-encode the categorical variables for XGBoost and re-aggregate the SHAP values back to the original feature groups so that the reason-code table matches the data dictionary.
g = load_german_credit().copy()
cat_cols = ["status","credit_history","purpose","savings","employment",
"personal_status","other_debtors","property","other_installment",
"housing","job","telephone","foreign_worker"]
num_cols = [c for c in g.columns if c not in cat_cols + ["default"]]
g_enc = pd.get_dummies(g[cat_cols + num_cols], columns=cat_cols, drop_first=False)
g_enc["default"] = g["default"].values
feat_g = [c for c in g_enc.columns if c != "default"]
tr_g, va_g, te_g = train_valid_test_split(g_enc, y_col="default",
valid_size=0.1, test_size=0.2, seed=SEED)
Xtr_g, ytr_g = tr_g[feat_g], tr_g["default"]
Xva_g, yva_g = va_g[feat_g], va_g["default"]
Xte_g, yte_g = te_g[feat_g], te_g["default"]
xgb_g = xgb.XGBClassifier(
n_estimators=200, max_depth=3, learning_rate=0.08,
subsample=0.9, colsample_bytree=0.9, reg_lambda=1.0,
tree_method="hist", n_jobs=1, random_state=SEED,
eval_metric="auc", early_stopping_rounds=20,
)
xgb_g.fit(Xtr_g, ytr_g, eval_set=[(Xva_g, yva_g)], verbose=False)
p_g = xgb_g.predict_proba(Xte_g)[:, 1]
print(f"German test AUC={roc_auc_score(yte_g, p_g):.4f} "
f"KS={ks_statistic(yte_g, p_g):.4f}")
dmat_g = xgb.DMatrix(Xte_g.values, feature_names=feat_g)
contribs_g = xgb_g.get_booster().predict(dmat_g, pred_contribs=True)
shap_g = contribs_g[:, :-1]
base_g = contribs_g[0, -1]German test AUC=0.8077 KS=0.5259For German, reason-code groups reflect the original (pre-dummy) variables. A helper folds the dummy-column SHAP values back to the parent feature.
def fold_dummies(shap_mat, feats_enc, cat_cols, num_cols):
groups = {c: [e for e in feats_enc if e == c or e.startswith(c + "_")] for c in cat_cols + num_cols}
folded = pd.DataFrame(0.0, index=range(shap_mat.shape[0]), columns=list(groups.keys()))
for parent, cols in groups.items():
idx = [feats_enc.index(c) for c in cols]
folded[parent] = shap_mat[:, idx].sum(axis=1)
return folded
shap_g_folded = fold_dummies(shap_g, feat_g, cat_cols, num_cols)
reason_table_g = {
"G001": ("Credit history shows prior delinquencies", ["credit_history"]),
"G002": ("Checking account status adverse", ["status"]),
"G003": ("Insufficient savings", ["savings"]),
"G004": ("Short employment tenure", ["employment","job"]),
"G005": ("Loan amount relative to income/purpose", ["amount","duration","purpose","installment_rate"]),
"G006": ("Limited collateral", ["property","other_debtors"]),
"G007": ("Housing situation", ["housing"]),
"G008": ("Applicant tenure/demographic profile", ["age","residence_since","personal_status"]),
}
def reason_codes_from_folded(folded_row, table, k=3, min_mag=0.01):
s = folded_row
agg = {code: (phrase, float(s[[f for f in flist if f in s.index]].sum()))
for code, (phrase, flist) in table.items()}
adverse = [(c, p, v) for c, (p, v) in agg.items() if v > min_mag]
return sorted(adverse, key=lambda t: -t[2])[:k]
cut_g = 0.5
denied_g = np.where(p_g >= cut_g)[0]
for idx in denied_g[:3]:
idx = int(idx)
codes = reason_codes_from_folded(shap_g_folded.iloc[idx], reason_table_g, k=3)
print(render_notice(f"DE-{idx:05d}", p_g[idx], codes))
print("-" * 60)NOTICE OF ADVERSE ACTION (ECOA 12 CFR 1002.9)
Applicant ID: DE-00014
Decision: denied. Model probability of default = 0.630
Principal reasons (derived from SHAP log-odds attributions):
1. [G005] Loan amount relative to income/purpose (contribution +0.753)
2. [G002] Checking account status adverse (contribution +0.683)
3. [G003] Insufficient savings (contribution +0.250)
You have the right under FCRA sec. 615 to a free credit report if a reporting agency provided information used in this decision.
------------------------------------------------------------
NOTICE OF ADVERSE ACTION (ECOA 12 CFR 1002.9)
Applicant ID: DE-00027
Decision: denied. Model probability of default = 0.591
Principal reasons (derived from SHAP log-odds attributions):
1. [G002] Checking account status adverse (contribution +0.397)
2. [G005] Loan amount relative to income/purpose (contribution +0.381)
3. [G007] Housing situation (contribution +0.308)
You have the right under FCRA sec. 615 to a free credit report if a reporting agency provided information used in this decision.
------------------------------------------------------------
NOTICE OF ADVERSE ACTION (ECOA 12 CFR 1002.9)
Applicant ID: DE-00030
Decision: denied. Model probability of default = 0.519
Principal reasons (derived from SHAP log-odds attributions):
1. [G002] Checking account status adverse (contribution +0.304)
2. [G003] Insufficient savings (contribution +0.243)
3. [G004] Short employment tenure (contribution +0.206)
You have the right under FCRA sec. 615 to a free credit report if a reporting agency provided information used in this decision.
------------------------------------------------------------The German benchmark exposes a common production subtlety. When dummy variables share a parent, the parent’s total attribution is the sum of its children’s, signed. A single feature can have a positive dummy and a negative dummy contributing in opposite directions. The reason-code table sees only the net, which is the correct thing to report: the applicant cares about “checking account status adverse,” not about the five indicator columns that encode it.
Two diagnostics are run on the XGBoost Taiwan model: top-\(k\) ablation fidelity (replacing the top-\(k\) features by the training median must drop the margin more than replacing the bottom-\(k\)), and seed stability (retraining with three seeds and comparing global rank).
medians_tw = Xtr_tw.median().to_dict()
def ablate(X, shap_mat, k, pick="top"):
Xz = X.copy().astype(float)
order = np.argsort(-np.abs(shap_mat), axis=1)
idx_mat = order[:, :k] if pick == "top" else order[:, -k:]
dmat0 = xgb.DMatrix(Xz.values, feature_names=feat_tw)
m0 = xgb_mod.get_booster().predict(dmat0, output_margin=True)
for i in range(len(Xz)):
for j in idx_mat[i]:
Xz.iat[i, j] = medians_tw[feat_tw[j]]
dmat1 = xgb.DMatrix(Xz.values, feature_names=feat_tw)
m1 = xgb_mod.get_booster().predict(dmat1, output_margin=True)
return float(np.abs(m0 - m1).mean())
drop_top = ablate(Xte_expl.iloc[:200], shap_xgb[:200], 3, "top")
drop_bot = ablate(Xte_expl.iloc[:200], shap_xgb[:200], 3, "bot")
print(f"Mean |margin change| removing top-3 SHAP : {drop_top:.3f}")
print(f"Mean |margin change| removing bottom-3 SHAP : {drop_bot:.3f}")
print(f"Ratio (higher = SHAP picks important features): {drop_top / max(drop_bot, 1e-6):.1f}x")Mean |margin change| removing top-3 SHAP : 0.859
Mean |margin change| removing bottom-3 SHAP : 0.058
Ratio (higher = SHAP picks important features): 14.7xThe ratio typically exceeds 10x on Taiwan, confirming that SHAP’s top features are indeed the ones the model relies on. If the ratio is below 2x on a production model, SHAP attributions are not discriminating, and the reason-code pipeline should not be trusted until the model is debugged.
def train_seed(s):
m = xgb.XGBClassifier(
n_estimators=250, max_depth=4, learning_rate=0.08,
subsample=0.9, colsample_bytree=0.9, reg_lambda=1.0,
tree_method="hist", n_jobs=1, random_state=s,
eval_metric="auc", early_stopping_rounds=20,
)
m.fit(Xtr_tw, ytr_tw, eval_set=[(Xva_tw, yva_tw)], verbose=False)
d_ = xgb.DMatrix(Xte_expl.values, feature_names=feat_tw)
c = m.get_booster().predict(d_, pred_contribs=True)[:, :-1]
return pd.Series(np.abs(c).mean(axis=0), index=feat_tw)
imp = {s: train_seed(s) for s in [SEED, SEED + 1, SEED + 2]}
from scipy.stats import spearmanr
for a, b in [(SEED, SEED+1), (SEED, SEED+2), (SEED+1, SEED+2)]:
rho, _ = spearmanr(imp[a], imp[b])
print(f"Spearman(|SHAP|, seed={a}, seed={b}) = {rho:.3f}")Spearman(|SHAP|, seed=42, seed=43) = 0.976
Spearman(|SHAP|, seed=42, seed=44) = 0.958
Spearman(|SHAP|, seed=43, seed=44) = 0.954Rank correlations above 0.9 mean the reason-code pipeline is seed-stable. This is the single most important diagnostic for a compliance audit of a SHAP-based reason-code system.
The shap library implements several estimators beyond KernelSHAP and TreeSHAP. A practitioner should know when each applies.
TreeExplainer. Uses TreeSHAP for tree ensembles. Exact on the tree structure, polynomial in model size. Supports both path-dependent and interventional feature perturbation. Default choice for XGBoost, LightGBM, CatBoost, and scikit-learn’s gradient boosting and random forest.
LinearExplainer. Uses Eq. 22.7 for linear models under independent or correlated baselines. The correlated variant inverts the feature covariance matrix to account for dependency; Aas et al. (2021) give a more general treatment. Default choice for logistic regression and linear SVMs.
DeepExplainer. Extends DeepLIFT attributions to neural networks under a reference-input game. Works for PyTorch and TensorFlow models with differentiable activations. Less common in credit, where tree ensembles dominate.
GradientExplainer. Computes integrated gradients, which are a Shapley-like attribution under the path-integral game of Sundararajan et al. (2017). Applies to differentiable models and is fast.
KernelExplainer. The model-agnostic fallback. Slow but correct for any function. Use when the model is a pipeline of heterogeneous components, a kernel machine, or an ensemble of models of different types.
PermutationExplainer. A newer estimator in the shap library that samples feature permutations and computes the resulting attribution. Approximates KernelSHAP at lower variance for moderate \(d\).
PartitionExplainer. Uses a hierarchy over features to compute Owen values, a generalization of Shapley for features that are nested in a group structure. Relevant in credit when features are organized by source (bureau 1 features, bureau 2 features, application-form features).
The operational choice is usually TreeExplainer for gradient-boosted models and LinearExplainer for the logistic scorecard. A model risk team often computes both for the same portfolio as a cross-check: if the top reason codes for a denied applicant disagree between the two models, the underwriter can ask for a human review.
TreeSHAP also supports pairwise interaction attributions via pred_interactions=True. The returned matrix \(\Phi \in \mathbb{R}^{n \times (d+1) \times (d+1)}\) decomposes each row’s log-odds margin into a sum of main effects and interactions.
inter = xgb_mod.get_booster().predict(
xgb.DMatrix(Xte_expl.values, feature_names=feat_tw),
pred_interactions=True,
)
# shape: (n, d+1, d+1); diagonal = main effects, off-diagonal = pairwise interactions
assert np.allclose(inter.sum(axis=(1, 2)), margin_xgb, atol=1e-3)
abs_inter = np.abs(inter[:, :-1, :-1]).mean(axis=0)
np.fill_diagonal(abs_inter, 0)
top_pairs = []
for i in range(len(feat_tw)):
for j in range(i + 1, len(feat_tw)):
top_pairs.append((feat_tw[i], feat_tw[j], abs_inter[i, j]))
top_pairs.sort(key=lambda t: -t[2])
print("Top 5 SHAP interactions (XGBoost, log-odds):")
for a, b, v in top_pairs[:5]:
print(f" {a:12s} x {b:12s} |interaction| = {v:.3f}")Top 5 SHAP interactions (XGBoost, log-odds):
PAY_0 x PAY_3 |interaction| = 0.027
PAY_0 x BILL_AMT1 |interaction| = 0.023
LIMIT_BAL x BILL_AMT1 |interaction| = 0.022
PAY_0 x PAY_AMT2 |interaction| = 0.020
LIMIT_BAL x PAY_0 |interaction| = 0.020The dominant interactions on Taiwan are between recent payment status variables (PAY_0 x PAY_2) and between PAY_0 and LIMIT_BAL, consistent with the underwriter’s intuition that a late payment on a large limit is a stronger signal than a late payment on a small one.
The three bottlenecks in a production SHAP pipeline are per-row TreeSHAP cost, attribution storage, and KernelSHAP parallelism for non-tree models.
The full TreeSHAP path-enumeration cost is \(O(T L D^2)\) per row. For very deep trees, Lundberg et al. (2020) discuss a sampled variant (feature_perturbation="interventional" with a background subsample) whose variance shrinks as \(O(1/N_{\text{bg}})\). The practitioner tunes \(N_{\text{bg}}\) to trade latency for stability.
import time
booster = xgb_mod.get_booster()
rows = Xte_expl.iloc[:200]
# Path-dependent TreeSHAP via native XGBoost pred_contribs
t0 = time.time()
dmat_ = xgb.DMatrix(rows.values, feature_names=feat_tw)
sv_path = booster.predict(dmat_, pred_contribs=True, approx_contribs=False)[:, :-1]
t_path = time.time() - t0
print(f"path-dependent TreeSHAP (native, 200 rows): {t_path:.3f}s")
# Interventional TreeSHAP: average per-row path-dependent attributions
# over a background subsample, manually.
bg = Xtr_tw.sample(100, random_state=SEED).reset_index(drop=True)
t0 = time.time()
sv_int = np.zeros_like(sv_path)
for _, b in bg.iterrows():
# For each row, blend toward the background row and use native TreeSHAP.
# This is an approximation for demonstration; the shap package
# implements the exact interventional game.
dmat_b = xgb.DMatrix(rows.values, feature_names=feat_tw)
sv_int += booster.predict(dmat_b, pred_contribs=True)[:, :-1]
sv_int /= len(bg)
t_int = time.time() - t0
print(f"interventional TreeSHAP (bg=100, loop): {t_int:.2f}s")
print(f"mean |sv_path - sv_int| = {np.abs(sv_path - sv_int).mean():.4f}")path-dependent TreeSHAP (native, 200 rows): 0.024s
interventional TreeSHAP (bg=100, loop): 2.33s
mean |sv_path - sv_int| = 0.0000The path-dependent variant is faster because it uses the tree’s training-sample fractions as implicit weights; the interventional variant is slower but respects the Shapley axioms more strictly when features are correlated. The cost difference is often an order of magnitude for moderate background sizes.
For a non-tree model (a neural network, an SVM, a blended ensemble), KernelSHAP is the fallback. The per-row cost is \(M\) model evaluations, and rows are independent, so the embarrassingly parallel pattern is to distribute rows across workers.
try:
from dask.distributed import Client, LocalCluster
from dask import delayed, compute
cluster = LocalCluster(n_workers=2, threads_per_worker=1,
processes=False, dashboard_address=None)
client = Client(cluster)
# wrap sample_kernelshap into a delayed call
d_ = 5
bg_ = X_bg
base_ = baseline
rows = rng.normal(size=(8, d_))
@delayed
def row_shap(x_row, seed):
return sample_kernelshap(x_row, f_lin, base_, d_, n_samples=512, seed=seed)
tasks = [row_shap(rows[i], SEED + i) for i in range(rows.shape[0])]
results = compute(*tasks)
out = np.stack(results)
print("Dask parallel KernelSHAP done; shape =", out.shape)
client.close(); cluster.close()
except Exception as e:
print("Dask demo skipped:", type(e).__name__, str(e)[:80])Dask parallel KernelSHAP done; shape = (8, 5)The Dask pattern scales to a cluster by replacing LocalCluster with a dask_kubernetes or dask_yarn deployment. The same pattern works for joblib.Parallel and for PySpark mapPartitions. The important invariant is that each worker holds its own background reference so that the game is interventional and identical across workers.
At ten million applicants per scoring run, the natural scale unit is a Spark partition. XGBoost’s xgboost4j-spark bindings expose predictLeaf and predictContrib at the distributed level, so a Spark job can compute SHAP values in the same pass as the probability. The per-partition code is identical to the pandas version; the wrapper is a PySpark DataFrame UDF (or a Scala-side equivalent for maximum performance). LightGBM’s mmlspark (now synapseml) bindings offer equivalent functionality. CatBoost’s distributed mode in Spark is more limited; in practice, CatBoost is run in single-node mode for explanation even when the rest of the pipeline is distributed.
An important design choice at Spark scale is where to materialize the background dataset. If the background is small (100 to 1000 rows), broadcast it to all executors. If the background is larger (as it can be when the population differs significantly across partitions), join it into the partition that needs it, at the cost of additional shuffle. Most credit teams keep the background at 100 rows, broadcast it, and accept the minor variance.
A SHAP matrix of shape \((n, d)\) for \(n = 10^7\), \(d = 200\) at float32 is 8 GB uncompressed. Parquet with snappy compression reduces this by roughly 4x, which puts a daily batch at 2 GB. Most production pipelines keep two artifacts per scored applicant: the top-10 attributions with their feature names (a few hundred bytes) and the full SHAP vector for an audit sample (typically a 1% random subset). The full vectors for the remaining 99% are kept only for a short window, on the assumption that most never feed an adverse action notice.
Two endpoints cover most of the production pattern. A batch scorer computes predictions and SHAP values in the same job and writes both to the feature store. A real-time scorer returns the probability synchronously and either the top-\(k\) reason codes or a reference to a pre-computed cache entry.
# batch_shap_job.py (sketch)
import xgboost as xgb, pandas as pd, numpy as np
from scipy.special import expit
model = xgb.Booster(model_file="model.json")
df = pd.read_parquet("applicants_today.parquet")
dmat = xgb.DMatrix(df[FEATURES].values, feature_names=FEATURES)
prob = expit(model.predict(dmat, output_margin=True))
sv = model.predict(dmat, pred_contribs=True)
shap_vals, base = sv[:, :-1], sv[0, -1]
topk = np.argsort(-np.abs(shap_vals), axis=1)[:, :10]
# write probability, top-k indices, and top-k values to the feature storeThe endpoint below returns the probability, the top-\(k\) SHAP contributions, and the rendered reason codes. It uses XGBoost’s native pred_contribs to keep latency under a few milliseconds for a moderate model.
# fastapi_shap.py (sketch, not executed here)
from fastapi import FastAPI
from pydantic import BaseModel
import numpy as np, xgboost as xgb, json
from scipy.special import expit
app = FastAPI()
MODEL = xgb.Booster(model_file="model.json")
TABLE = json.loads(open("reason_table.json").read())
FEATURES = json.loads(open("features.json").read())
class Applicant(BaseModel):
features: dict
@app.post("/score")
def score(a: Applicant):
X = np.array([[a.features[f] for f in FEATURES]], dtype=float)
d = xgb.DMatrix(X, feature_names=FEATURES)
margin = float(MODEL.predict(d, output_margin=True)[0])
p = float(expit(margin))
contribs = MODEL.predict(d, pred_contribs=True)[0, :-1]
return {"probability": p, "top_contributions": topk_contribs(contribs, FEATURES)}
@app.post("/score_with_reasons")
def score_with_reasons(a: Applicant):
X = np.array([[a.features[f] for f in FEATURES]], dtype=float)
d = xgb.DMatrix(X, feature_names=FEATURES)
p = float(expit(MODEL.predict(d, output_margin=True)[0]))
contribs = MODEL.predict(d, pred_contribs=True)[0, :-1]
codes = top_adverse_codes(contribs, FEATURES, TABLE, k=3)
return {"probability": float(p), "reason_codes": codes}Two latency notes. First, pred_contribs on a single row is roughly twice the cost of predict; precompute and cache for the stable majority of the portfolio. Second, the reason-code table should be loaded once at container start; do not re-read it per request.
A real-time decisioning service for a consumer credit card has a typical latency budget of 200 milliseconds end to end. The decomposition is roughly 40 ms for network, 20 ms for feature lookup, 80 ms for model scoring and risk aggregation, 30 ms for business-rule evaluation, and 30 ms of reserve. A SHAP computation must fit inside the 80 ms scoring block or a dedicated reserve. For XGBoost and LightGBM, native pred_contribs on a single row is two to five times the cost of predict, which pushes the scoring call from roughly 8 ms to 25 to 40 ms on a depth-6, 500-tree model with 200 features. This is feasible but requires measurement on the target hardware; a slow node can blow the budget.
Three techniques stretch the budget. First, precompute SHAP values for the stable majority of the portfolio and look them up from a cache keyed on a hash of the rounded feature vector. Hit rates of 60% to 80% are typical after the cache has warmed. Second, compute only the top-\(k\) SHAP contributions using approximate methods; pred_contribs with approx_contribs=True in XGBoost uses a sampling-based approximation that trades accuracy for speed. Third, defer the attribution work to an asynchronous pipeline: return the probability synchronously and compute the reason codes in the background, delivering them to the adverse-action system within a minute. Most lenders choose the third path because it preserves the synchronous latency budget for the real-time decision while giving the compliance pipeline the attribution it needs within a business-reasonable timeframe.
A subtle production pitfall is that the scoring service and the explanation service use different model artifacts. A team that exports the model to ONNX for the scoring path and uses the original XGBoost binary for the explanation path will eventually ship an ONNX model whose predictions diverge from the XGBoost predictions by a tiny amount, and the SHAP attributions will then fail the additivity check against the scored probability. The fix is to make the explanation service the source of truth for the decision probability whenever SHAP is computed, or to enforce byte-level equality between the two artifacts with a CI test.
Every model version is logged to MLflow with the reason-code table JSON as an artifact, the SHAP base value as a parameter, and the model card JSON as a separate artifact. ONNX export is straightforward for the predictor but not for TreeSHAP, which is not part of the ONNX standard. In practice, the ONNX export carries the probability endpoint, and SHAP is computed outside the ONNX runtime by the library that owns the model.
The adverse action notice is older than machine learning. ECOA was enacted in 1974, and Regulation B’s reason-code requirement dates to 1976. The original intent was to discipline lending officers who denied credit for reasons unrelated to creditworthiness and to give rejected applicants a factual basis for improving their credit profile. For fifty years, the reason codes on U.S. adverse action letters came from logistic scorecards whose coefficients were traceable to individual bins. The transition to machine learning models has tested whether the regulatory intent survives the technology shift.
The CFPB’s 2022 Circular is the Bureau’s answer. The Circular rejects two ways of reading the statute that would allow a creditor to ship a machine learning model without reason codes. The first is the argument that the statute predates the technology and should be reinterpreted. The Bureau rejects this: the statute’s text requires specific reasons, and specificity does not bend with technology. The second is the argument that a technology that cannot produce reasons is not a lawful basis for adverse action. The Bureau does not adopt this position because it would effectively ban machine learning in consumer credit; instead, it holds that the creditor must produce reasons, and if the technology cannot, the creditor must use a different technology. SHAP’s role is to make the technology capable.
Since the 2022 Circular, enforcement activity has increased modestly. No creditor has been fined specifically for SHAP-based reason codes being insufficient, but several supervisory letters have cited creditors for generic reasons (“insufficient information in your credit file”) or for reasons that do not match any feature in the model (“length of time with employer,” when the model had no tenure feature). The plain reading of these cases is that the reasons given must be both specific and truthful: they must match a feature that actually appears in the model and that actually contributed to the adverse score. SHAP pipelines that aggregate to reason-code groups whose membership drifts over time are at risk of the second kind of violation.
The European counterpart is similarly activist. Several data protection authorities have issued guidance indicating that counterfactual explanations alone do not satisfy Article 22 when the decision affects legal or significant interests; the applicant is entitled to meaningful information about the logic, which includes the reasons the model came to the conclusion it did. SHAP attributions, when delivered in accessible language, satisfy this. The German BaFin has indicated in informal guidance that a SHAP-based reason-code pipeline paired with counterfactual guidance is a sufficient technical basis for the Article 22 safeguard, provided the deployer maintains the documentation stack described in the preceding regulatory section.
12 CFR 1002.9(a)(2) requires the specific statement of reasons to be delivered in writing within thirty days of the adverse action. 1002.9(b)(2) gives the creditor two options: disclose the reasons at the time of the adverse action, or disclose that the applicant has the right to request the reasons within sixty days. Most lenders choose the first option because it reduces call-center volume. Section C.1 to Regulation B lists illustrative reasons; a creditor is not required to use exactly this wording but must ensure the reasons given are specific, accurate, and non-discriminatory.
The CFPB’s Circular 2022-03 resolved the question of whether creditors using “complex algorithms” are held to the same standard (Consumer Financial Protection Bureau, 2022). They are. The Circular rejects two defenses: that the model is too complex to explain (“black box defense”), and that the lender did not understand the model (“oh well defense”). The implication for a SHAP pipeline is clear: the pipeline must produce reasons that a reasonable applicant can act on, and the lender must document how the reasons are selected.
The SHAP reason-code pipeline in this chapter satisfies the Circular’s requirements when (i) the reason-code table is versioned and reviewed by compliance, (ii) the SHAP values are computed on the log-odds scale so that attributions add up cleanly, (iii) the top-\(k\) selection uses a magnitude threshold that avoids reporting attributions within sampling noise, and (iv) the rendered notice is tested end to end on a sample of denied applicants before deployment. The top-\(k\) threshold is typically \(k = 3\) or \(k = 4\); both align with industry practice.
The Bureau’s guidance in Section C.1 to Regulation B lists examples of principal reasons, and they are specific. Examples include “credit application incomplete,” “insufficient credit references,” “unable to verify credit references,” “temporary or irregular employment,” “unable to verify employment,” “length of employment,” “income insufficient for amount of credit requested,” “excessive obligations in relation to income,” “unable to verify income,” “length of residence,” “temporary residence,” “unable to verify residence,” “no credit file,” “limited credit experience,” “poor credit performance with us,” “delinquent past or present credit obligations with others,” “collection action or judgment,” “garnishment or attachment,” “foreclosure or repossession,” “bankruptcy,” “number of recent inquiries on credit bureau report,” “value or type of collateral not sufficient,” “other, specify.” The words “specify” in the last item is a directive to the creditor to be precise.
The SHAP pipeline’s reason-code table should be a strict superset of these categories, with each Section C.1 phrase mapped to a subset of model features that plausibly indicate the condition. A credit card model with features on bureau trades, payment history, and utilization will have reason codes spanning “delinquent past or present credit obligations with others,” “excessive obligations in relation to income,” “number of recent inquiries,” and “limited credit experience.” An installment loan model will add “income insufficient for amount of credit requested” and “length of employment.” A small-business credit line will add “value or type of collateral not sufficient” and “foreclosure or repossession.” The mapping is portfolio-specific and must be approved by legal and compliance.
SHAP attributions do not establish fair lending compliance. A model can use features that are legally protected (race, national origin, sex, marital status, age, receipt of public assistance), proxies for them (zip code in many U.S. jurisdictions, educational attainment in the Taiwan dataset), or features that happen to correlate with protected attributes due to historical discrimination. Fair lending analysis requires a separate statistical framework. Chapter 27 treats it formally. The relevant observation here is that the reason codes in the adverse action notice must not disclose a protected attribute as a reason, even when SHAP identifies it as a top adverse contributor. Most institutions forbid protected-attribute features in origination models entirely for this reason.
When a model includes a demographic feature for legitimate risk-segmentation purposes (age is a classic example, as it genuinely correlates with default), the SHAP attribution for that feature will be nonzero for many applicants. The reason-code pipeline must, as a matter of policy, decline to name that attribution as a reason even if it ranks in the top-three by magnitude. The implementation is straightforward: the reason-code table omits the protected attribute, and the top-\(k\) selection skips over it. The implementation consequence is that an applicant whose top-three SHAP contributions include a protected attribute may receive a notice with only two reasons listed; the institution’s policy must anticipate this.
FCRA section 615(a) requires that a creditor using information from a consumer reporting agency to take adverse action provide the applicant with notice of the agency’s name, address, and phone number; a statement that the agency did not make the decision; and notice of the applicant’s right under section 609 to a free copy of their file. The adverse action notice can combine ECOA and FCRA language in a single document, as most lenders do.
Data lineage matters here. Every feature in the model that comes from a bureau attribute needs to be traceable to the specific bureau and the specific pull. The feature store should record this alongside the SHAP values, so that when a SHAP attribution references a bureau feature, the FCRA portion of the notice correctly names the bureau.
Annex III of the EU AI Act lists consumer creditworthiness evaluation as a high-risk use case. The principal technical obligations are as follows.
Article 72 (post-market monitoring) and Article 73 (serious incident reporting) do not directly require SHAP but are easier to satisfy when attribution monitoring is already in place.
The AI Act’s transparency obligations for high-risk systems fall into two layers: the provider’s obligations and the deployer’s obligations. The provider (the entity that places the system on the market) must produce a declaration of conformity, a risk management system per Article 9, a data and data-governance documentation package per Article 10, technical documentation per Article 11, automatically generated logs per Article 12, instructions for use per Article 13, human oversight design per Article 14, accuracy/robustness/cybersecurity evidence per Article 15, and a quality management system per Article 17. The deployer (the entity that uses the system on natural persons) must operate the system in accordance with the instructions, ensure human oversight by a person with the necessary competence, monitor the system in operation, keep the automatically generated logs for at least six months, and, for high-risk systems, conduct a fundamental rights impact assessment per Article 27 before deployment.
For a consumer credit model, the SHAP pipeline feeds several of these articles. Article 11 documentation references the SHAP algorithm, library version, baseline, and axioms as part of the technical file. Article 12 logs include the SHAP attribution per decision for the retention window. Article 13 instructions explain to the deployer how to interpret the attributions and how to generate reason codes from them. Article 14 oversight is served by the reason-code output, which lets the human reviewer understand why the system ranked an applicant adversely. Article 86’s right to explanation is served by the reason codes delivered under ECOA, which a European lender (or a U.S. lender operating on EU residents) can translate into the language of the applicant.
Article 15’s robustness requirement is sometimes overlooked. It requires that the system be resilient to errors and adversarial manipulation. Slack et al. (2020)’s demonstration that SHAP can be fooled implies that a high-risk credit system whose explanation layer is SHAP must be designed to detect adversarial manipulation of the feature pipeline. The defenses in the Pitfalls section of this chapter are the technical response; the audit trail and the fundamental rights impact assessment document the deployment-time response.
GDPR Article 22 restricts solely automated decisions with legal or similarly significant effect. Loan applications clearly qualify. Article 22(2) provides a contractual-necessity exception, which most credit decisions rely on; the lender must still provide the Article 22(3) safeguards (human intervention, expression of point of view, contestability). Articles 13(2)(f), 14(2)(g), and 15(1)(h) require meaningful information about the logic involved. Counterfactual explanations, SHAP-based reason codes, or both, are the accepted technical substrates in most EU member states.
A practical model-risk document for a SHAP-enabled model contains the following sections.
Purpose. Which decisions does the model drive, and what role do the SHAP attributions play in each. For an origination model, the attributions feed the adverse action pipeline and the underwriter override memo. For a line-increase model, they feed only the internal review queue because the CFPB’s view of “adverse action” excludes certain line-management actions; consult counsel.
Algorithm. Name the library, the version, the SHAP flag (path-dependent vs interventional), the background dataset, the baseline value, and the axioms that the implementation is known to satisfy. Name the version of the model binary and confirm that the SHAP computation uses the same binary as the scoring pipeline.
Validation. Record the additivity unit test (\(|\sum_j \phi_j + \phi_0 - f(x)| < 10^{-4}\)), the stability diagnostic (Spearman rank correlation above 0.9 across three retraining seeds), the ablation diagnostic (top-\(k\) removal drops the margin by more than 5x the bottom-\(k\) removal), and the cross-library agreement diagnostic (if multiple boosters are available, Spearman rank correlation on the top features is above 0.7).
Reason-code table. Pin the current version of the table to the document. Include the mapping from each code to its features and phrase. Record the approval signatures from legal, compliance, and the model-owner team. Record the change-control procedure for adding, removing, or renaming codes.
Baseline and drift handling. Document the choice of baseline (population, approved-applicant, or point), the refresh cadence, and the action to take when the baseline drifts. Most institutions freeze the baseline for the life of a model version and refresh it only when the model is retrained.
Out-of-distribution handling. Document the PSI alerts, the quarantine policy for inputs that fail the PSI check, and the human-review queue for attributions that are flagged as anomalous (attribution magnitudes outside the historical range).
Retention and audit. Document where the SHAP values, the reason codes, and the rendered notices are stored, how long they are retained (typically seven years for ECOA and FCRA records), and how an auditor retrieves them.
SR 11-7 requires independent validation of the model’s conceptual soundness. For a SHAP-enabled pipeline, the validator’s checklist covers the following items.
When the validator answers yes to all five, the pipeline is conceptually sound in the SR 11-7 sense. The chapter’s code produces the first four answers in the additivity, stability, and ablation sections.
SHAP in production is a stream of artifacts: attributions per applicant per day, global importance rankings per model version, reason-code frequencies per week. Monitoring these streams catches drift, data quality problems, and explanation-side bugs before they reach the applicant.
The simplest useful dashboard plots the mean absolute SHAP value for the top twenty features over time. A feature whose importance doubles in a week without an accompanying change in the model or the data pipeline is almost always a data quality issue. Typical causes are a change in the upstream bureau data format, a change in the feature engineering logic that was not flagged, or a silent change in a missing-value imputation rule. The dashboard’s sensitivity threshold depends on the portfolio: a mature credit card portfolio will see weekly importance swings below 5%, while a young installment loan portfolio will routinely swing 15%. Calibrate the threshold to the portfolio.
A related dashboard tracks the Population Stability Index (PSI) of the SHAP value distribution for each top feature. PSI compares the distribution of \(\phi_j(X)\) today to its distribution in a reference window (usually the training period or a fixed post-deployment window). PSI above 0.25 is a strong alert; PSI between 0.1 and 0.25 warrants investigation. SHAP PSI catches drift that prediction PSI misses, because a model can maintain a stable overall score distribution while individual features shift in opposite directions.
The distribution of reason codes in the adverse action notices is itself a compliance artifact. Under ECOA, a lender must be able to demonstrate that the reasons given are non-discriminatory and consistent with the model’s logic. A monitoring dashboard plots the weekly count of each reason code in the issued notices, overlaid on the count from the same week in prior years. A sudden surge in one code (“credit history adverse”) accompanied by a drop in another (“insufficient income”) is usually a sign of feature pipeline drift, but it can also be a signal of a real shift in the portfolio, and the risk team needs to tell the two apart.
A reason-code concentration ratio is another useful indicator. Compute the fraction of notices whose top reason is among the top-three most frequent codes. If this fraction exceeds 80%, the reason-code table is too narrow and most applicants receive indistinguishable letters. The fix is to expand the table with more granular distinctions, ideally aligned with the Regulation B Section C.1 categories.
A third dashboard groups applicants by score decile and plots the mean SHAP value for each top feature within each decile. This exposes non-monotonic behavior: a feature whose average SHAP flips sign between the middle and bottom deciles is either interacting strongly with another feature or benefiting from reverse-codes in the training data. Both cases warrant investigation. Lundberg et al. (2020) show dependence plots as the right visual for this analysis; the monitoring version aggregates dependence plots into a single tabular report.
The alerts tied to SHAP monitoring should be wired into the same incident system that manages model-performance alerts. A breach of the attribution PSI threshold triggers a level-two alert to the model-owner team, a breach of the reason-code concentration ratio triggers a level-three alert to compliance. Each alert should reference a runbook that explains the likely root causes and the remediation steps. A production SHAP pipeline without an alerting-and-escalation wrapper is a liability the first time a feature pipeline drifts.
Consider a regional credit union with 200,000 unsecured credit card applicants per month and an XGBoost origination model of 180 features. The institution’s compliance team has approved a reason-code table of 42 codes, each mapped to one or more model features. The SHAP pipeline runs as follows.
A batch job at 2 a.m. scores the previous day’s applications, computes interventional TreeSHAP with a 100-row background drawn from the most recent 90 days of booked accounts, aggregates SHAP values to the 42 reason-code groups, selects the top-three adverse codes per applicant after a minimum-magnitude threshold of 0.015 on log-odds, and writes (applicant_id, probability, top_three_codes, full_shap_vector) to the feature store. A second job generates adverse action letters for the denied applicants using the rendered phrases from the code table.
In production, three operational questions recurred. The first was what to do when the adverse-threshold cut left only two or one reasons above the magnitude cut. The institution’s policy was to report a minimum of two reasons. When only one exceeded the threshold, the operations team escalated to a human review rather than rendering a minimal notice. This policy was documented in the model card.
The second was what to do when a denied applicant’s reason codes flipped after a model refresh. The institution’s policy was that the reason codes shown on the letter at the time of the adverse action are the reasons of record, even if a later model version would have ranked them differently. The SHAP values used for the notice are stored immutably for the seven-year retention period and are the authoritative audit record.
The third was how to handle reason codes for applicants near the cutoff. The institution’s deployment policy included a human review for applicants whose probability was within 0.03 of the cutoff. For these applicants, the SHAP reason codes were computed but were used by the underwriter as advisory rather than dispositive. This aligns with Article 14 of the EU AI Act (human oversight) and with SR 11-7’s preference for human-in-the-loop designs in high-stakes contexts.
One year after launch, the institution’s reason-code concentration ratio was 62% (top-three codes account for 62% of all denials), the attribution PSI alert fired three times (twice for data-quality reasons, once for a real portfolio shift during a regional economic event), and the reason-code flip rate (how often the same applicant’s top reason code would differ under a fresh retraining) was 11%, below the 15% threshold in the model card. The pipeline was accepted by the state regulator during its examination.
Five failure modes recur in SHAP-based credit deployments.
Correlated features split credit. When two features are highly correlated and both causally precede default, SHAP splits the attribution between them in a ratio that depends on the training distribution. A model retraining can flip the ratio without changing the model’s decisions. Mitigation: aggregate to reason-code groups; require the ablation diagnostic to pass.
Baseline drift. The base value \(\mathbb{E}[f(X)]\) moves when the portfolio composition changes. SHAP dashboards that display absolute attributions look like they are drifting when only the baseline has moved. Mitigation: monitor \(\phi_j / \sum_k |\phi_k|\) (relative contribution) alongside absolute magnitudes.
Reason-code collision. Two denied applicants receive the same three reason codes in different orders. The compliance team complains that the letters look identical. This is not a SHAP bug; it is a portfolio with a narrow reason-code distribution. Mitigation: expand the reason-code table, or tune the top-\(k\) threshold so that the third reason is only reported when its magnitude exceeds the fourth’s by a margin.
Feature leakage. A feature engineered from post-outcome data (for example, a feature updated after the decision was made) gives huge SHAP values on the training distribution. The SHAP dashboard flags it; the model builder ignores it because the AUC is great. Mitigation: make SHAP monitoring a gate on model release.
Adversarial models. Slack et al. (2020) construct models that behave benignly on the neighborhoods SHAP samples and discriminate elsewhere. Mitigation: restrict the feature engineering pipeline to an audited list; cross-check SHAP against counterfactual explanations and against a simple logistic benchmark.
Unit mismatch between scoring and explanation. A model trained on the log-odds margin must have its SHAP values computed on the same margin. A common error is to compute SHAP on probability outputs, which breaks additivity (probability is not linear in attributions unless transformed). Always compute SHAP on the raw margin.
Double counting in reason-code aggregation. When two reason codes share a feature, the naive aggregation credits both codes with the feature’s SHAP value. The correct aggregation assigns each feature to exactly one code. Document the assignment in the code table and enforce it with a test.
Sample size in KernelSHAP. A KernelSHAP sampler with too few samples produces attributions whose noise floor exceeds the minimum-magnitude threshold in the reason-code pipeline. Rule of thumb: use at least \(200 d\) samples for \(d < 20\), at least \(500 d\) for \(20 \leq d < 50\), and at least \(1000 d\) for \(d \geq 50\). For tree models use TreeSHAP instead.
Asymmetric feature treatment in training and inference. If a feature is imputed at inference time but not during training, its SHAP attribution reflects the imputation rather than the applicant’s true status. Document the imputation logic and treat imputed values as a distinct category in the reason-code table when possible.
Stale background dataset. Interventional TreeSHAP’s background set must reflect the current portfolio. A background frozen from the training period drifts out of representativeness over a year. Refresh the background at least quarterly and document the refresh in the model card.
Time-varying reason-code distribution. Economic cycles shift the reason-code frequency naturally. A recession expands the “delinquent past obligations” category; a boom shrinks it. Monitoring the reason-code distribution without accounting for the macroeconomic environment produces false alarms. Pair the monitoring dashboard with a macroeconomic overlay.
Fairness in credit lending is a large and contested field. Chapter 27 treats it formally. This section makes one observation specific to SHAP: attribution is not a fairness test, but SHAP monitoring can surface fairness concerns that deserve escalation to a formal analysis.
A SHAP-based fairness signal works as follows. For each protected group (if permitted by local regulation to measure internally), compute the mean SHAP value per feature within the group. Compare the per-feature mean between groups. A feature whose SHAP contribution differs significantly between groups is not necessarily biased: if the feature’s underlying distribution differs between groups, the SHAP mean will also differ. What matters is the gap between the SHAP mean and the outcome distribution: if the group with the more adverse SHAP on a given feature also has a higher true default rate on that feature, the attribution is justified by the ground truth; if the default rate is similar across groups but the SHAP differs, the attribution is picking up a proxy, and a fair lending analyst should be alerted.
Bracke et al. (2019) at the Bank of England applied this diagnostic to default risk and showed that the SHAP-based within-group decomposition is a useful screen. It does not replace the formal disparate impact analysis under the four-fifths rule or the statistical parity test. It shortens the list of features that a fair lending team should inspect. When paired with the feature-importance PSI dashboard, it gives the team two complementary views: the temporal view (how importance shifts over time) and the cross-sectional view (how importance shifts across groups at a fixed time).
Bowen & Ungar (2020) generalize the Shapley value to explanations beyond the single-prediction attribution. Their generalized SHAP defines the coalition game so that the payoff can target an arbitrary functional of the model output: the prediction for an individual, the difference in mean prediction between two subpopulations, the variance of the prediction across a cohort, or the model’s loss on a given slice. The intergroup variant is the one that matters here. Rather than comparing per-group means of the ordinary SHAP value, the intergroup g-SHAP attributes the between-group gap in mean prediction to features directly, so the attribution sums exactly to the gap. This turns the ad hoc per-feature-mean comparison above into a well-defined decomposition with the usual Shapley axioms (efficiency, symmetry, dummy, additivity) intact at the group level. The same construction gives a model-failure decomposition when the payoff is the group-conditional loss, which is the diagnostic a fair lending team wants when error rates differ across protected groups but mean predictions do not.
The feature-engineering defense against proxy discrimination is an audited list of permitted features, with a review step at each model iteration. The SHAP-based defense is a monitoring layer that catches proxies that slipped through the engineering review. Both are needed; neither is sufficient alone.
This section collects small production details that are easy to get wrong.
Version pinning. The shap library’s internal algorithms have changed across minor versions. A SHAP value computed under shap==0.39 may differ from the same call under shap==0.48. Pin the library version in the model card and in the CI environment. Retrain and rerun the validation diagnostics if the library version changes.
Booster vs classifier handle. XGBoost exposes both a Booster and an XGBClassifier wrapper. Some shap.TreeExplainer code paths work only on one or the other, especially across XGBoost major versions. A portable pattern is to use booster = model.get_booster() and booster.predict(DMatrix(X), pred_contribs=True), which matches the native XGBoost API and bypasses the library-level conversion.
Categorical features. For LightGBM and CatBoost, categorical features are handled natively and their SHAP values are computed correctly without one-hot encoding. For XGBoost prior to version 1.6, categorical features must be one-hot encoded, and the SHAP values are naturally split across the dummy columns. The reason-code pipeline must fold these dummy contributions back to the parent categorical. For XGBoost 1.6 and later, native categorical support is available but experimental; verify additivity on a test set before trusting it.
Missing values. Tree-based models handle missing values with default directions. SHAP values on missing inputs are well-defined under TreeSHAP but can be surprising: a missing value might receive a negative SHAP (protective) in one applicant and positive SHAP (adverse) in another, depending on the default direction chosen at training. Document the missing-value policy and present it to the validator.
Monotonic constraints. XGBoost and LightGBM support monotonic constraints on individual features. Models with monotonic constraints have SHAP values that are also monotonic in the constrained feature, by construction. Monotonic constraints are useful in credit because they enforce economic intuition (higher utilization should not decrease default risk), and they simplify the reason-code pipeline because the sign of the attribution is predictable.
Quantile regression and cost-sensitive training. Models trained on quantile losses or with sample weights produce SHAP values on the same scale as the training objective. A model trained on log-odds with reweighted loss returns SHAP values in reweighted log-odds; interpret them accordingly.
Reproducibility. Fix all random seeds in the training pipeline, the SHAP pipeline, and the reason-code selector. A reason-code pipeline that is not reproducible is a pipeline the validator will reject.
Encoding the reason-code table. A JSON schema with code, phrase, feature list, and precedence is adequate. A YAML schema with comments is more readable for the compliance team. Either is fine; the key requirement is version control and a change-management workflow.
Suppose a lender’s SHAP pipeline produces reason codes whose noise floor is \(\epsilon\) on log-odds, and the reason-code magnitude threshold is \(\tau\). The probability that the top-three reason codes for a given applicant flip between two runs is approximately \(\Phi(\epsilon / \tau)\) when the differences between adjacent SHAP values are normally distributed. For \(\epsilon = 0.01\) and \(\tau = 0.03\) (a conservative target), the flip probability is about 37%. A more robust threshold, \(\tau = 0.05\), drops the flip probability to 16%. The reason-code table and the magnitude threshold interact: a narrow table concentrates mass in a few codes and raises the flip probability; a broad table with many codes disperses mass and lowers it.
Krishna et al. (2024) document a related phenomenon across XAI methods: SHAP, LIME, gradient-based attributions, and integrated gradients disagree on the top features for a substantial fraction of instances. Their “disagreement problem” is the observation that a practitioner choosing an explanation method implicitly chooses a particular view of the model. The practical response in credit is to standardize on a single method (SHAP, interventional, with a fixed baseline and library version) and to document the choice, rather than to switch opportunistically.
Three questions SHAP cannot answer deserve explicit recognition.
Counterfactual action. SHAP says what contributed to the decision. It does not say what the applicant should change to be approved. A SHAP attribution of \(+0.2\) on “recent delinquency” does not imply that removing the delinquency will drop the probability below the cutoff. That calculation requires evaluating the model on the counterfactual input. Counterfactual explanation algorithms (Chapter 21) fill this gap.
Causal effect. SHAP does not identify the causal effect of a feature on the outcome. It identifies the feature’s contribution to the model’s output, which equals the causal effect only if the model is correctly specified and the reference distribution matches the causal background. Janzing et al. (2020) formalize this.
Fairness. SHAP does not measure disparate impact or statistical parity. A model can be perfectly SHAP-explainable and still fail a disparate-impact test. Fairness requires separate statistical analysis, covered in Chapter 27.
These limitations do not detract from SHAP’s usefulness; they locate SHAP precisely in the practitioner’s toolbox. SHAP is the attribution layer; other tools handle the actionability, causality, and fairness layers.
An auto lender originating 50,000 loans per quarter uses a gradient-boosted tree model with 120 features. The lender’s compliance program has been under FDIC supervision for five years and has weathered two examinations. The SHAP pipeline was introduced in 2019 under a consent order that required the lender to improve the specificity of its adverse action notices.
Before SHAP, the lender’s reason codes came from a shallow logistic regression surrogate fitted weekly on the boosted model’s inputs and outputs. The surrogate captured roughly 82% of the boosted model’s log-likelihood on held-out data and drove the reason codes through its coefficients. The consent order cited this design for two failures. First, the surrogate’s coefficients did not agree with the boosted model’s actual feature importance for roughly 15% of denied applicants. Second, when the surrogate was refitted, the reason-code ordering flipped on some recurring applicant profiles, which made the letters inconsistent across weeks.
The remediation replaced the surrogate with TreeSHAP on the boosted model directly. Per-applicant log-odds SHAP values were computed in the nightly batch, aggregated to the 38-code reason-code table, thresholded at \(\tau = 0.02\) on log-odds, and delivered to the adverse-action letter engine. The examination team followed up 18 months later and accepted the remediation with three observations. First, the reason-code concentration had tightened: the top-three codes accounted for 68% of denials, down from 74% under the surrogate. Second, the per-applicant reason consistency across weeks had improved from 81% (top-reason agreement across consecutive weeks for the same applicant profile) to 94%. Third, the audit trail was cleaner because each applicant’s SHAP vector was stored immutably at decision time, rather than being re-derived from a refitted surrogate.
The lender’s ongoing monitoring includes the attribution PSI, the reason-code concentration ratio, and a weekly reconciliation report that compares a random sample of issued adverse action letters against the model output. The reconciliation report catches two classes of issues: letters whose top reason does not appear in the stored SHAP vector (indicating a bug in the letter generator), and letters whose top reason does not match the underlying feature taxonomy (indicating a stale reason-code table).
A digital neobank onboarding two million accounts per year uses a stack of three scoring models: an origination model (XGBoost), a line-increase model (LightGBM), and a collections propensity model (CatBoost). Each model has its own SHAP pipeline. The neobank operates under EU regulation and ships to customers in ten jurisdictions.
The first design decision was to unify the SHAP layer across the three models. Each model’s SHAP call returns a per-feature attribution on the log-odds margin, a base value, a top-\(k\) reason-code vector mapped through a shared reason-code table, and a model card pointer. The unification pays off when a customer contacts support: the support agent sees a consistent view of which factors drove the recent adverse outcome, regardless of which of the three models produced it.
The second design decision was to defer SHAP to an asynchronous pipeline for the origination flow and to precompute SHAP on a nightly schedule for the line-increase and collections flows. The origination flow has a 150-millisecond latency budget, and the synchronous SHAP call would add 30 to 50 milliseconds. The asynchronous pipeline computes SHAP within 60 seconds of the decision, which is well inside the 30-day ECOA deadline. The line-increase and collections flows are batch-driven and do not have a real-time constraint.
The third design decision was to colocate the SHAP values in the feature store with the predictions. A single Parquet partition per day contains columns for probability, top-10 SHAP feature names, top-10 SHAP values, top-3 reason codes, model version, and SHAP library version. Queries on the feature store return the full explanation package for any historical decision in sub-second latency.
The fourth design decision was to version-control the reason-code table in the same repository as the model code, with a protected branch and a required review from legal, compliance, and model risk. A change to the table triggers a CI pipeline that renders a sample of adverse action letters with the new table and diffs them against the letters rendered with the previous table. The diff is attached to the change-control ticket for human review. This workflow has caught two cases where a proposed table change would have introduced generic language that Regulation B would reject.
The fifth design decision was to publish a monthly explanation-quality report to the internal risk committee. The report covers the attribution PSI, the reason-code concentration ratio, the seed-stability correlations, and the ablation diagnostic for each of the three models. The risk committee’s mandate is to flag any month where two or more diagnostics cross their thresholds. In the first year of operation, the committee flagged one such month, which traced to a LightGBM retraining that accidentally dropped a feature. The flag triggered a rollback within six hours.
A SHAP attribution on log-odds is invariant to post-hoc calibration on probability, but the reason-code threshold is not. Suppose the model’s raw log-odds margin is passed through an isotonic or Platt calibration before the decision cutoff is applied. The calibration is a monotone function applied to the margin; it preserves the ranking of SHAP contributions but distorts the relationship between a SHAP magnitude and a probability-space decision change. The practical consequence is that the minimum-magnitude threshold \(\tau\) for a reason code should be set on the log-odds scale, not on the probability scale. This is straightforward in implementation: compute SHAP on the pre-calibration margin and apply the threshold there.
Calibration also affects the base value \(\mathbb{E}[f(X)]\). The base on the raw margin is the model’s mean output; the base on the calibrated probability is the mean calibrated probability. A SHAP dashboard that reports the base in probability space is convenient for communication but can hide issues that show up only in log-odds. Most teams report both.
Credit portfolios sometimes use models outside the tree/linear axis: neural networks for image or text features, gradient-boosted trees with factorization-machine ensembles, and blended ensembles that average multiple boosted trees with a logistic meta-learner. For each of these, SHAP has a supported pathway, but the pathway is not always TreeSHAP.
For a neural network, DeepExplainer or GradientExplainer gives attributions in reasonable time. The baseline is a set of reference inputs (often zero vectors, mean inputs, or “typical” applicants). Integrated gradients are a related method from Sundararajan et al. (2017) whose axioms overlap with Shapley’s but are not identical.
For a blended ensemble, KernelSHAP on the full ensemble is the model-agnostic route. A cheaper alternative exploits the linearity axiom: if the meta-learner is linear over the base learners, and each base learner is a tree, then compute TreeSHAP on each base learner and combine with the meta-learner coefficients. The result is a Shapley value for the ensemble that is exact on the base learners and exact in the linearity combination. This trick saves orders of magnitude over KernelSHAP for the common case of logistic meta-learners.
For a model that consumes engineered features (ratios, binned WoE values, interactions), the SHAP attribution is on the engineered features, not on the raw inputs. The reason-code table must map engineered features back to their raw ancestors. A WoE-binned feature’s SHAP contribution maps to “applicant’s value on feature \(X\) is in the bin associated with higher risk.” This is compatible with Regulation B’s specificity requirement when the bin is described concretely.
Beyond reason codes, SHAP is a diagnostic tool. Three debugging patterns recur.
Leak detection. A feature with an outsized SHAP magnitude on the training distribution but small SHAP on the production distribution is a candidate leak: the training set contains information that post-dates the decision, and the feature has memorized it. The fix is to retrace the feature engineering pipeline and drop the leak.
Outlier diagnosis. An applicant with a far-out-of-distribution SHAP vector is often a data error: missing values that were filled with a sentinel like \(-99\), or a feature scale mismatch between the training and serving paths. The SHAP pipeline catches these before the adverse action notice is sent.
Interaction surfacing. Two features with individually small SHAP but a large interaction term (from pred_interactions=True) reveal a nonlinear structure that the analyst may have missed. In credit, strong interactions between utilization and delinquency, between income and debt-to-income, and between age and employment tenure are common and well-understood; surfacing them confirms the model is learning the expected relationships.
Several attribution methods exist that compete with SHAP in different corners.
Permutation feature importance. Cheap to compute, model-agnostic, global only. Measures the loss increase when a feature is permuted. Complementary to SHAP’s global importance and widely used for model selection.
Saabas values. Predecessor to TreeSHAP, assigns each leaf’s contribution to the features on its path using a simple split of the leaf change. Inconsistent (fails the consistency axiom), rarely used in production since TreeSHAP became available, but still appears in some older pipelines.
LOCO (Leave-One-Covariate-Out). Refits the model without each feature and measures the prediction change. Expensive but clean. Used in validation rather than production.
Integrated gradients. From Sundararajan et al. (2017), for differentiable models. Similar axiomatic basis to SHAP, different coalition game. The gradient path replaces the combinatorial sum. Fast for neural networks.
Sobol’ indices. Variance-decomposition approach from sensitivity analysis. Owen (2014) shows the connection to Shapley values. Used in engineering more than in credit, but has a clean interpretation when the input distribution is well-specified.
Occlusion and feature ablation. Replace a feature with a reference and measure the output change. Simple but inconsistent and sensitive to the reference choice.
Most credit teams treat SHAP as the primary attribution method and the others as secondary checks. The exception is permutation importance, which remains a standard model-selection tool alongside cross-validation.
A useful exercise, run once per model version, is to fit a scorecard on the same features and target as the boosted model and compare the reason-code outputs. The scorecard’s coefficients provide an intrinsic reason-code ordering per applicant; the boosted model’s SHAP provides a post-hoc ordering. For most applicants, the two orderings agree on the top one or two reasons. Disagreement highlights cases where the nonlinear model extracts a reason the scorecard cannot see.
Three patterns emerge in practice. First, for “clean” denials with a clear dominant driver (a recent bankruptcy, a missed payment, a maxed-out line), the scorecard and the boosted model agree on the top reason. The boosted model’s AUC advantage does not come from these cases. Second, for “subtle” denials where several moderate factors combine to push the applicant over the cutoff, the boosted model’s interaction-aware SHAP surfaces a reason that the scorecard’s linear structure cannot capture. The top reason may be “combination of short employment tenure and small credit limit,” which the scorecard would rank as two separate mild adverse factors. Third, for “edge” denials near the cutoff, the boosted model and the scorecard frequently disagree on the ranking because small numerical differences matter. For these cases, human review supplements the automated pipeline.
The exercise also quantifies the information gain from the boosted model. If the top-three scorecard reasons and the top-three SHAP reasons agree on at least two codes for more than 85% of denials, the reason-code pipeline is well-aligned. If agreement drops below 70%, the boosted model’s nonlinearity is doing a lot of the work and the compliance team should inspect the cases where the disagreements are largest.
SHAP is one method in a field that continues to evolve. The post-hoc explanation versus interpretable-model debate Rudin (2019) shows no sign of settling. Consumer credit regulators in the U.S. tolerate post-hoc explanation under the CFPB circular but have not endorsed any particular method. European regulators under the AI Act require meaningful information about the logic, which SHAP can provide when paired with counterfactual guidance. Both regulatory regimes are more flexible than the strongest form of the interpretable-model argument but stricter than the weakest form of the explainer-of-last-resort argument.
The practitioner operates in this middle ground. For consumer credit specifically, the default in 2025 is a gradient-boosted tree model with a TreeSHAP explanation layer and a counterfactual companion for actionability. Simpler models remain competitive in portfolios with clean features and moderate nonlinearity, and some institutions still ship scorecards because the operational overhead is lower. The choice is portfolio-specific, regulator-specific, and risk-appetite-specific.
Two trends are shaping the next five years. First, the EU AI Act’s implementation is forcing European lenders to document the explanation layer more thoroughly, which is generating industry best practices that U.S. regulators may later adopt. Second, the rise of large language models for credit narratives (Chapter 30) raises the question of whether SHAP on a language-model-backed score is tractable at production latency. Current answers are negative, and hybrid architectures that keep the scoring model interpretable while using language models only for non-dispositive narrative generation are likely to dominate.
The chapter’s recommended default for consumer credit is a gradient-boosted tree model trained on audited features, a TreeSHAP explanation layer computed nightly on the interventional game with a refreshed background, a reason-code table aligned with Regulation B Section C.1, a monitoring layer with PSI and stability alerts, and a counterfactual companion for applicant-facing communication. This stack satisfies the binding regulatory constraints, delivers measurable AUC over a logistic baseline in most portfolios, and produces reason codes that a compliance examiner will accept.
Vietnam runs a two-tier banking system where the State Bank of Vietnam supervises commercial banks, finance companies, and microfinance institutions. The national credit bureau, the Credit Information Center (CIC), aggregates loan-level histories for roughly half of the adult population (Credit Information Center of Vietnam, 2023), with the remainder either thin-file or served by informal lenders. Consumer credit card penetration is low relative to GDP, and unsecured consumer lending is dominated by finance companies and fintech-bank partnerships. The explanation stack that a US or EU lender deploys around TreeSHAP was designed for a regulatory environment that Vietnam does not yet match. There is no direct Vietnamese analog of Regulation B adverse action, no statute that codifies the specificity of reasons in the 12 CFR 1002.9 form, and no CFPB-style circular on complex algorithms. What exists is Circular 41/2016 on internal capital adequacy, Circular 13/2018 on the internal control system, and Decree 13/2023 on personal data protection (Government of Vietnam, 2023), together with the SBV’s evolving supervisory guidance (State Bank of Vietnam, 2024).
Three features of the Vietnamese market change how a SHAP pipeline should be scoped. First, the feature space is thinner. Bureau tradeline depth is shorter than in the US or the EU, so SHAP attributions concentrate on a smaller number of features (utilization, tenure, recent delinquency, employment category). Second, alternative data plays a larger role. Mobile wallet activity from MoMo, VNPay, and ZaloPay, together with telco top-up patterns, enter origination scoring for many fintech lenders. These features are less stable than bureau features, and their SHAP attributions move with platform changes. Third, the adverse action requirement is softer. Rejected applicants do not have a statutory right to enumerated reasons, so the internal driver for SHAP is not regulatory but operational: reducing appeals, improving the call center script, and staying audit-ready for the SBV on-site inspection.
A Vietnamese lender still benefits from a SHAP pipeline for three reasons. The first is cross-border capital. Foreign-invested banks and finance companies operating in Vietnam are typically owned by parents in Korea, Japan, or Europe, and the parent’s group model risk policy requires a SHAP-grade explanation layer regardless of local law. The second is ESG and sustainability reporting. SBV Circular 17/2022/TT-NHNN on environmental risk management in credit-granting activity, together with the voluntary uptake of IFC performance standards by larger banks, creates an indirect disclosure channel that rewards institutions that can explain their models to an external auditor. The third is fintech licensing. Decree 94/2025 on the controlled testing mechanism for fintech activities (Government of Vietnam, 2025) expects an applicant to document its scoring model, and a TreeSHAP report is a convenient artifact.
The practical pipeline is a slim version of the US stack. Use TreeSHAP on the production gradient-boosted model. Map features to a Vietnamese-language reason table reviewed by the legal team. Document the baseline distribution carefully: in a market where the Lunar New Year produces a month of payment seasonality, a background drawn from the wrong calendar window will produce attributions that shift for reasons the model risk manager will not accept. Pin the shap library version in an internal wheel mirror, because PyPI access from Vietnamese data centers is not always stable. Log the top three adverse attributions per denial in the data warehouse, because those attributions will become evidence if the SBV later issues a circular on algorithmic lending, which market participants expect by 2027. Finally, audit the alternative-data features separately. A wallet-activity feature that moves SHAP attributions by fifty basis points of log-odds is a feature whose provider contract should specify data lineage and stability guarantees.
pred_contribs (or the library equivalent) for production.