62  Data Validation and Testing

Machine learning systems inherit the quality of the data that flows through them. A model trained on corrupted features, a feature store populated with stale records, or a serving pipeline fed malformed input will produce confident and wrong predictions. Unlike a software bug that throws an exception, a data quality defect often passes silently through every layer of a system until it surfaces as degraded business metrics weeks later. Data validation and testing exist to make these defects loud, early, and cheap to fix.

This chapter treats data as a first class artifact that deserves the same engineering discipline applied to source code. We examine the spectrum of validation techniques, from rigid schema enforcement to probabilistic statistical checks, run five open source validation libraries (Pandera, Patito, Pointblank, Validoopsie, and Dataframely) against one shared defective batch, and discuss how validation behaves differently in batch and streaming production pipelines. Where the topic warrants precision, we state the underlying mathematics: the drift statistics in particular have exact definitions and properties that determine when each is appropriate.

What this chapter covers

A layered model of data testing. Schema validation pins down structure, constraint checks encode domain logic, statistical validation guards distributions, and data unit tests verify the transformation code itself. A hands on tooling section then expresses one contract in five libraries, executes each against the same batch of 1,000 records with seven planted defects, and compares what each one catches, isolates, and reports. We then place these checks in a production pipeline, formalize the drift metrics that statistical validation relies on, and close with a pragmatic strategy for growing a validation suite without drowning in maintenance.

62.1 Why Data Validation Matters

62.1.1 The Silent Failure Mode

Traditional software fails loudly. A null pointer dereference crashes, a type mismatch refuses to compile, and a failed assertion halts execution. Data pipelines fail quietly. A column that should range from zero to one hundred suddenly contains values in the millions because an upstream service changed its units from dollars to cents. The pipeline does not crash. It computes averages, trains models, and serves predictions, all built on a silent defect.

The cost of a data defect grows with the distance it travels before detection. Catching a malformed record at ingestion costs a retry. Catching it after it has trained a production model costs a retraining cycle, a redeployment, and possibly an incident review. Validation pushes detection as close to the source as possible.

62.1.2 Data Quality Dimensions

Practitioners commonly decompose data quality into several dimensions. Completeness asks whether required values are present. Validity asks whether values conform to their expected type and format. Consistency asks whether related values agree with each other across tables or time. Accuracy asks whether values reflect reality, which is the hardest dimension to test automatically. Timeliness asks whether data arrived within its expected window. Uniqueness asks whether records that should be distinct are in fact distinct.

A mature validation strategy maps each dimension to concrete, automatable checks. Completeness becomes a null count threshold. Validity becomes a type and regex check. Uniqueness becomes a primary key duplication check. Accuracy usually requires a trusted reference source or human review, so teams often approximate it with plausibility bounds.

It helps to see these dimensions as predicates over a dataset. Let \(D\) be a dataset, viewed as a multiset of records drawn from a record space \(\mathcal{R}\). A check is a function \(c : \mathcal{R}^{*} \to \{0, 1\}\) that returns \(1\) when the dataset satisfies the property and \(0\) otherwise. Schema and constraint checks are deterministic predicates: they evaluate the same way every time on the same input. Statistical checks are distributional predicates: they compare a summary of \(D\) against a reference and pass only when the discrepancy stays within a tolerance. This distinction, deterministic versus distributional, is the organizing axis of the chapter, because it dictates how a check is written, where it can run, and how its failures should be interpreted.

62.2 Schema Validation

62.2.1 What a Schema Captures

A schema is a contract that describes the structure of a dataset: the set of columns, their names, their data types, their nullability, and sometimes their permissible value ranges. Schema validation verifies that an incoming dataset honors this contract before any downstream code touches it.

Schema validation is the cheapest and highest leverage form of data testing because it catches an entire class of structural defects with a single declarative specification. When an upstream team renames a column, drops a field, or changes an integer to a string, a schema check fails immediately rather than letting the change propagate.

62.2.2 Explicit Schemas with Pandera

Pandera lets you express a schema as code over pandas, Polars, PySpark, and other dataframe engines, with one import path per engine (pandera.pandas, pandera.polars, and so on). A DataFrameSchema declares columns with types and constraints, and validation either returns the validated frame or raises a SchemaError describing exactly which rows and checks failed. Section 6 executes this style of schema, along with four alternative libraries, on a live batch.

import pandera.pandas as pa
from pandera.pandas import Column, Check

schema = pa.DataFrameSchema({
    "user_id": Column(int, Check.gt(0), unique=True),
    "age": Column(int, Check.in_range(0, 120), nullable=False),
    "email": Column(str, Check.str_matches(r".+@.+\..+")),
    "signup_country": Column(str, Check.isin(["US", "CA", "GB", "DE"])),
})

validated = schema.validate(raw_df, lazy=True)

The lazy=True flag is important in practice. Without it, validation stops at the first failure. With it, Pandera collects every failing check across the frame and reports them together, which dramatically shortens the debugging loop when an unfamiliar dataset arrives.

62.2.3 Class Based Schemas and Type Integration

Pandera also offers a class based API using DataFrameModel, which reads like a dataclass and integrates with static type checkers. This style keeps the schema definition close to the rest of your typed code and supports inheritance for shared field definitions.

import pandera.pandas as pa
from pandera.pandas import DataFrameModel, Field

class Transaction(DataFrameModel):
    amount: float = Field(ge=0, le=1_000_000)
    currency: str = Field(isin=["USD", "EUR", "GBP"])
    timestamp: pa.DateTime = Field(nullable=False)

    class Config:
        strict = True  # reject unexpected columns

The strict = True configuration is a deliberate choice. It rejects any column not declared in the schema, which protects against silent additions of unexpected fields. The opposite default, permissive validation, is more forgiving but lets schema drift accumulate unnoticed.

62.2.4 Serialized Schemas for Cross Language Pipelines

When a pipeline spans multiple languages or services, an in code Python schema does not travel well. Serialized schema formats such as JSON Schema, Avro, and Protobuf describe structure in a language neutral way. A producer written in Java and a consumer written in Python can share one Avro schema and enforce it at the boundary. Schema registries, common in Kafka deployments, store these definitions centrally and enforce compatibility rules so that a producer cannot publish a breaking change without explicit approval.

62.3 Constraint Checks

62.3.1 Beyond Structure

Schema validation confirms that a column is an integer. Constraint checks confirm that the integer makes sense. A column may be a perfectly valid float and still be wrong if it represents an age of negative five or a probability of one hundred. Constraint checks encode domain knowledge that no type system can express.

Common constraint categories include range checks (a value falls within known bounds), set membership checks (a category belongs to an allowed list), uniqueness checks (a key does not repeat), referential checks (a foreign key exists in a parent table), and cross column checks (a discount price does not exceed a list price).

62.3.2 Cross Field and Cross Row Constraints

The most valuable constraints often span multiple fields. A record where end_date precedes start_date is structurally valid but logically impossible. A row where total does not equal the sum of its line items signals a computation bug upstream. These relationships are invisible to per column validation and require checks that reason about the whole record.

schema = pa.DataFrameSchema(
    columns={
        "start_date": Column(pa.DateTime),
        "end_date": Column(pa.DateTime),
    },
    checks=Check(
        lambda df: df["end_date"] >= df["start_date"],
        error="end_date must not precede start_date",
    ),
)

62.3.3 Choosing Hard versus Soft Constraints

Not every constraint violation should halt a pipeline. A hard constraint, such as a null primary key, justifies rejecting the batch outright because downstream joins will break. A soft constraint, such as a slightly elevated null rate in an optional field, may warrant a warning and a logged metric rather than a full stop. Encoding this distinction explicitly prevents two failure modes: a brittle pipeline that halts on trivial issues, and a permissive pipeline that ignores real corruption. Most validation tools support both raising errors and emitting warnings, and the choice should reflect the genuine downstream impact of each rule.

62.4 Statistical Validation

62.4.1 From Deterministic Rules to Distributions

Schema and constraint checks are deterministic: a value either satisfies the rule or it does not. Many real defects are statistical. The data is individually valid but collectively wrong. A feature whose mean shifts by three standard deviations, a categorical column whose distribution suddenly skews toward one value, or a join that quietly drops half its rows all produce data that passes every per row check while being deeply broken.

Statistical validation compares the shape of incoming data against an expected profile, usually derived from historical reference data. Rather than asking “is this value valid,” it asks “does this dataset look like the data we expect.”

62.4.2 Distributional Checks and Drift

The core technique is to summarize a column with statistics, the mean, standard deviation, quantiles, null fraction, and cardinality, then compare those summaries against a baseline. When the deviation exceeds a threshold, the check fails or raises an alert. Drift is the phenomenon being detected: the distribution of a column in the current window has moved away from the distribution it had in a trusted reference window. Several standard metrics quantify how far it has moved, and they differ in what they assume about the data and in how they weight discrepancies.

Kolmogorov Smirnov statistic. For a continuous column, the two sample Kolmogorov Smirnov (KS) statistic measures the largest gap between two empirical cumulative distribution functions. If \(F_r\) is the empirical CDF of the reference sample and \(F_c\) that of the current sample, then

\[ D_{\mathrm{KS}} = \sup_{x} \, \bigl| F_r(x) - F_c(x) \bigr|. \]

The statistic lies in \([0, 1]\), is zero only when the two empirical CDFs coincide, and is invariant to any monotone reparameterization of \(x\) because it depends only on ranks. Under the null hypothesis that both samples come from the same continuous distribution, the rescaled statistic has a known limiting distribution, which yields a \(p\) value. The KS test is sensitive to shifts in location and scale but comparatively insensitive to differences in the tails, since the supremum of the CDF gap is usually attained near the center of the distribution.

Chi squared test. For a categorical column with \(k\) categories, partition both samples into category counts. Let \(O_i\) be the observed count of category \(i\) in the current sample and \(E_i\) the count expected if the current sample followed the reference proportions. The statistic

\[ \chi^2 = \sum_{i=1}^{k} \frac{(O_i - E_i)^2}{E_i} \]

follows, under the null hypothesis of no change, an approximate chi squared distribution with \(k - 1\) degrees of freedom. The approximation degrades when expected counts are small, a common rule of thumb requires \(E_i \geq 5\) for every category, so rare categories should be merged before applying the test.

Population stability index. The population stability index (PSI) is the workhorse of production drift monitoring because it produces a single, threshold friendly number. Bin the column into \(B\) buckets (for a continuous column, fixed quantile edges from the reference; for a categorical column, one bucket per category). Let \(p_i\) be the fraction of reference records in bucket \(i\) and \(q_i\) the fraction of current records in the same bucket. Then

\[ \mathrm{PSI} = \sum_{i=1}^{B} (q_i - p_i) \, \ln\!\frac{q_i}{p_i}. \]

PSI is symmetric in its two arguments and non negative, reaching zero only when \(q_i = p_i\) for every bucket. It is in fact the symmetrized relative entropy between the two binned distributions, sometimes written \(\mathrm{PSI} = D_{\mathrm{KL}}(q \,\|\, p) + D_{\mathrm{KL}}(p \,\|\, q)\), where \(D_{\mathrm{KL}}\) is the Kullback Leibler divergence. A widely used convention treats \(\mathrm{PSI} < 0.1\) as no material shift, \(0.1 \leq \mathrm{PSI} < 0.25\) as a moderate shift worth investigating, and \(\mathrm{PSI} \geq 0.25\) as a significant shift. Empty buckets force a division by zero, so implementations add a small floor \(\epsilon\) to every \(p_i\) and \(q_i\), which is why bucket counts and the choice of \(\epsilon\) must be fixed and recorded for the metric to be reproducible.

Jensen Shannon divergence. When you want a bounded, smooth distance between two probability distributions \(P\) and \(Q\), the Jensen Shannon divergence is a good choice. With \(M = \tfrac{1}{2}(P + Q)\) the average distribution,

\[ \mathrm{JSD}(P \,\|\, Q) = \tfrac{1}{2} D_{\mathrm{KL}}(P \,\|\, M) + \tfrac{1}{2} D_{\mathrm{KL}}(Q \,\|\, M). \]

Unlike KL divergence, JSD is symmetric, is always finite (the averaged \(M\) never assigns zero probability where \(P\) or \(Q\) is positive), and lies in \([0, \ln 2]\) when using natural logarithms or \([0, 1]\) when using base two logarithms. Its square root is a true metric, the Jensen Shannon distance, which makes it convenient when drift scores are compared or thresholded across many columns.

The practical takeaway is that these metrics answer slightly different questions. KS and chi squared come with hypothesis tests and \(p\) values, which tempt teams to treat drift as a binary significance decision; but with large samples even a trivial, harmless shift becomes statistically significant, so a raw \(p\) value is a poor production gate. PSI and JSD instead report an effect size on a fixed scale, which is what a monitoring dashboard actually needs.

from scipy.stats import ks_2samp

stat, p_value = ks_2samp(reference["feature"], current["feature"])
if p_value < 0.01:
    raise DataDriftError(f"feature distribution shifted, KS p={p_value:.4f}")

62.4.3 A Worked PSI Calculation

Suppose a feature is bucketed into four bins. The reference window and the current window distribute their mass across those bins as follows.

Bucket	Reference \(p_i\)	Current \(q_i\)	\(q_i - p_i\)	\(\ln(q_i / p_i)\)	Contribution
1	0.40	0.25	-0.15	-0.470	0.0705
2	0.30	0.30	0.00	0.000	0.0000
3	0.20	0.25	0.05	0.223	0.0112
4	0.10	0.20	0.10	0.693	0.0693

Summing the contributions gives \(\mathrm{PSI} \approx 0.151\). By the conventional bands this falls in the moderate range, \(0.1 \leq \mathrm{PSI} < 0.25\), so the column warrants investigation but does not on its own justify halting the pipeline. Notice that bucket 2 contributes exactly zero because its mass is unchanged, while bucket 4 contributes heavily despite a modest absolute change of \(0.10\), because the log ratio amplifies a doubling of probability. This asymmetric weighting, large relative changes in small buckets dominate the score, is the defining behavior of PSI and the reason rare buckets must be sized carefully.

62.4.4 The Threshold Problem

Statistical validation introduces a tuning challenge absent from deterministic checks. Set thresholds too tight and the pipeline alerts on normal seasonal variation, training the team to ignore alarms. Set them too loose and genuine drift slips through. There is no universally correct threshold. Effective teams derive thresholds empirically by replaying historical data and measuring how often each candidate threshold would have fired, then tuning so that alerts correlate with real incidents rather than noise. Statistical checks should be treated as monitoring signals that often warrant investigation rather than hard gates that block deployment.

62.4.5 Validating Against Reference Data

A powerful pattern compares a derived dataset against an independent source of truth. If a feature pipeline computes daily active users, an independent analytics warehouse should report a closely matching count. Reconciliation checks of this kind catch logic bugs that no internal consistency check could find, because they compare two independent computations of the same quantity.

62.5 Data Unit Tests

62.5.1 Treating Data Logic Like Code

The transformations that produce data are code, and code deserves unit tests. A data unit test runs a transformation against a small, fixed input and asserts an exact expected output. Unlike validation, which runs against live production data, a data unit test runs in continuous integration against synthetic fixtures and verifies that the transformation logic itself is correct.

def test_revenue_excludes_refunds():
    input_df = pd.DataFrame({
        "order_id": [1, 2, 3],
        "amount": [100, 50, 30],
        "status": ["paid", "refunded", "paid"],
    })
    result = compute_revenue(input_df)
    assert result == 130  # refunded order excluded

This test pins the business rule that refunds do not count toward revenue. If a future change accidentally includes refunds, the test fails in continuous integration before the bad logic ever reaches production data.

62.5.2 Fixtures, Edge Cases, and Property Based Testing

Good data unit tests deliberately include edge cases: empty inputs, single row inputs, nulls in every nullable field, duplicate keys, and boundary values. Property based testing libraries such as Hypothesis generate many randomized inputs and check that invariants hold across all of them, which surfaces edge cases a human would not think to enumerate.

A useful property to assert is that a transformation never introduces nulls into a non nullable output column, or that an aggregation never produces a count larger than its input row count. These structural invariants hold regardless of the specific input values. Several validation libraries can generate such inputs directly from the schema, so fixtures stay in sync with the contract: Patito’s Model.examples fills in valid defaults around the columns a test specifies, and Dataframely’s Schema.sample draws extreme but valid frames that respect every rule, both demonstrated in Section 6.

62.5.3 The dbt Testing Model

In analytics engineering, dbt has popularized declarative data tests defined in YAML alongside model definitions. Built in generic tests cover the most common assertions, and custom tests are expressed as SQL queries that should return zero rows, where any returned row represents a failure.

models:
  - name: orders
    columns:
      - name: order_id
        tests:
          - unique
          - not_null
      - name: status
        tests:
          - accepted_values:
              values: ["placed", "shipped", "delivered", "returned"]

This model blurs the line between unit testing and production validation. The same assertions run during development against sample data and in production against the full warehouse, which is one of dbt’s most practical strengths.

62.6 Tooling: Five Open Source Validators, Side by Side

The ideas of Sections 2 through 5 pay off only once they are wired into code that runs on every batch. The Python ecosystem now offers several mature, MIT licensed libraries for this, and they make genuinely different design choices about how a contract is written, what happens when it is broken, and what evidence is left behind. This section runs five of them, Pandera, Patito, Pointblank, Validoopsie, and Dataframely, against one contract and one defective batch, so that their differences show up as differences in output rather than in feature lists. All five accept Polars frames. Patito and Dataframely are built directly on Polars; Pandera has a dedicated Polars backend alongside pandas, PySpark, and Ibis; Pointblank and Validoopsie reach Polars, pandas, and other engines through the Narwhals dataframe compatibility layer.

62.6.1 A Shared Contract and a Defective Batch

Section 1.2 described a check as a predicate over a dataset. To compare libraries we need one more piece of notation. Let the batch be \(D = (r_1, \dots, r_n)\) and the contract a set of \(K\) checks, and write \(F_k \subseteq \{1, \dots, n\}\) for the set of rows that violate check \(k\). Most checks are row local: whether row \(j\) violates “age lies in \([0, 120]\)” depends on \(r_j\) alone. Uniqueness is not. Row \(j\) violates “user_id is unique” exactly when some other row \(i \neq j\) carries the same key, so \(F_{\text{uniq}}\) is either empty or has at least two elements, and it can only be computed by looking at the whole column. This distinction explains several of the differences between libraries below.

A quarantine policy (Section 7.2) diverts the union of the failure sets and promotes the rest,

\[ Q = \bigcup_{k=1}^{K} F_k, \qquad |Q| \le \sum_{k=1}^{K} |F_k|, \]

with equality exactly when no row breaks two rules at once. A soft check with tolerance \(\tau_k\) passes when its failure fraction stays at or below that tolerance, \(|F_k| / n \le \tau_k\). This is how Pointblank and Validoopsie encode the hard versus soft distinction of Section 3.3: a hard check is simply one with \(\tau_k = 0\).

The batch is 1,000 synthetic user records. After generating a clean frame, the cell injects seven defects at known rows, one per rule: a duplicated key, an impossible age, a missing age, a malformed email, a country outside the allowed set, an activity date that precedes signup, and a negative spend.

from datetime import date, timedelta

import numpy as np
import polars as pl

rng = np.random.default_rng(55)
n = 1_000
EMAIL_RE = r"^[\w.+-]+@[\w-]+\.[\w.]+$"
ALLOWED = ["US", "CA", "GB", "DE"]

signup = [date(2024, 1, 1) + timedelta(days=int(d)) for d in rng.integers(0, 365, n)]
clean = pl.DataFrame({
    "user_id": np.arange(1, n + 1, dtype=np.int64),
    "age": rng.integers(18, 90, n, dtype=np.int64),
    "email": [f"user{i}@example.com" for i in range(1, n + 1)],
    "country": rng.choice(ALLOWED, n).tolist(),
    "signup_date": signup,
    "last_active": [s + timedelta(days=int(d)) for s, d in zip(signup, rng.integers(0, 400, n))],
    "spend": np.round(rng.gamma(shape=2.0, scale=40.0, size=n), 2),
})

# Inject one defect per rule at known row positions.
row = pl.int_range(pl.len())
batch = clean.with_columns(
    user_id=pl.when(row == 10).then(pl.lit(10, pl.Int64)).otherwise("user_id"),
    age=pl.when(row == 100).then(250).when(row == 200).then(None).otherwise("age"),
    email=pl.when(row == 300).then(pl.lit("bob_at_example.com")).otherwise("email"),
    country=pl.when(row == 400).then(pl.lit("FR")).otherwise("country"),
    last_active=pl.when(row == 500)
    .then(pl.col("signup_date") - pl.duration(days=30))
    .otherwise("last_active"),
    spend=pl.when(row == 600).then(-12.5).otherwise("spend"),
)
batch.slice(8, 4)

shape: (4, 7)

user_id	age	email	country	signup_date	last_active	spend
i64	i64	str	str	date	date	f64
9	56	"user9@example.com"	"GB"	2024-08-02	2024-08-19	26.05
10	44	"user10@example.com"	"DE"	2024-06-28	2024-08-27	134.22
10	57	"user11@example.com"	"DE"	2024-05-03	2025-05-13	177.05
12	70	"user12@example.com"	"CA"	2024-10-01	2025-05-30	80.77

Rows 9 and 10 now share user_id 10, and the other six defects sit at rows 100 through 600. Before reaching for any library, it is worth writing the contract in the plainest form available: one Polars expression per rule that evaluates to true on a violation. This reference implementation is the ground truth against which the five libraries are checked, and because it uses nothing but expressions it ports directly to Julia and Rust.

Python

# Library free reference: each rule is an expression that is True on a violation.
RULES = {
    "user_id unique": pl.col("user_id").is_duplicated(),
    "age not null": pl.col("age").is_null(),
    "age in [0, 120]": ~pl.col("age").is_between(0, 120),
    "email format": ~pl.col("email").str.contains(EMAIL_RE),
    "country allowed": ~pl.col("country").is_in(ALLOWED),
    "active after signup": pl.col("last_active") < pl.col("signup_date"),
    "spend >= 0": pl.col("spend") < 0,
}
HARD = {"user_id unique", "age not null", "active after signup"}

flags = {name: expr.fill_null(False) for name, expr in RULES.items()}
quarantine = batch.with_row_index("row").filter(pl.any_horizontal(list(flags.values())))
truth = (
    batch.select(**flags)
    .sum()
    .transpose(include_header=True, header_name="rule", column_names=["rows_failing"])
    .with_columns(severity=pl.when(pl.col("rule").is_in(list(HARD))).then(pl.lit("hard")).otherwise(pl.lit("soft")))
)
print(f"{truth.height} rules violated; quarantine rows: {quarantine['row'].to_list()}")
truth

7 rules violated; quarantine rows: [9, 10, 100, 200, 300, 400, 500, 600]

shape: (7, 3)

rule	rows_failing	severity
str	u32	str
"user_id unique"	2	"hard"
"age not null"	1	"hard"
"age in [0, 120]"	1	"soft"
"email format"	1	"soft"
"country allowed"	1	"soft"
"active after signup"	1	"hard"
"spend >= 0"	1	"soft"

Julia

# Illustrative: the same seven rules over DataFrames.jl.
# `batch` exported from Python with batch.write_parquet("users_batch.parquet").
using DataFrames, Dates, Parquet2

const EMAIL_RE = r"^[\w.+-]+@[\w-]+\.[\w.]+$"
const ALLOWED = Set(["US", "CA", "GB", "DE"])

batch = DataFrame(Parquet2.Dataset("users_batch.parquet"); copycols = false)

function violations(df::AbstractDataFrame)
    DataFrame(
        "user_id unique"      => nonunique(df, :user_id; keep = :noduplicates),
        "age not null"        => ismissing.(df.age),
        "age in [0, 120]"     => coalesce.((df.age .< 0) .| (df.age .> 120), false),
        "email format"        => .!occursin.(Ref(EMAIL_RE), df.email),
        "country allowed"     => .!in.(df.country, Ref(ALLOWED)),
        "active after signup" => df.last_active .< df.signup_date,
        "spend >= 0"          => df.spend .< 0,
    )
end

v = violations(batch)
truth = [String(name) => sum(col) for (name, col) in pairs(eachcol(v))]
quarantine = findall(any.(eachrow(v))) .- 1   # zero based, to match Polars
println(truth)
println("quarantine rows: ", quarantine)

Rust

// Illustrative: the same seven rules as Polars expressions in Rust.
// Cargo.toml: polars = { version = "0.51", features = ["lazy", "parquet", "regex",
//                        "strings", "is_unique", "dtype-date"] }
use polars::prelude::*;

const EMAIL_RE: &str = r"^[\w.+-]+@[\w-]+\.[\w.]+$";
const ALLOWED: [&str; 4] = ["US", "CA", "GB", "DE"];

fn rules() -> Vec<Expr> {
    let allowed = ALLOWED
        .iter()
        .fold(lit(false), |acc, c| acc.or(col("country").eq(lit(*c))));
    vec![
        col("user_id").is_duplicated().alias("user_id unique"),
        col("age").is_null().alias("age not null"),
        col("age").lt(lit(0)).or(col("age").gt(lit(120))).alias("age in [0, 120]"),
        col("email").str().contains(lit(EMAIL_RE), true).not().alias("email format"),
        allowed.not().alias("country allowed"),
        col("last_active").lt(col("signup_date")).alias("active after signup"),
        col("spend").lt(lit(0.0)).alias("spend >= 0"),
    ]
}

fn main() -> PolarsResult<()> {
    let batch = LazyFrame::scan_parquet("users_batch.parquet", ScanArgsParquet::default())?;
    let flags: Vec<Expr> = rules().into_iter().map(|e| e.fill_null(lit(false))).collect();

    let truth = batch
        .clone()
        .select(flags.iter().cloned().map(|e| e.sum()).collect::<Vec<_>>())
        .collect()?;
    println!("{truth}");

    let quarantine = batch
        .with_row_index("row", None)
        .filter(any_horizontal(&flags)?)
        .select([col("row")])
        .collect()?;
    println!("quarantine rows: {quarantine}");
    Ok(())
}

Seven rules fail. The duplicated key contributes two rows to \(F_{\text{uniq}}\) and every other rule contributes one, and since no row breaks two rules the quarantine set has \(|Q| = \sum_k |F_k| = 8\) rows. A correct validator must find all seven violations; a validator that quarantines should isolate these eight rows. Following Section 3.3, the three rules marked hard (a duplicated key breaks joins, a missing age and a time reversal signal a corrupt record) should stop the pipeline, while the four soft rules should be recorded without necessarily blocking it.

62.6.2 Pandera: Schemas as Typed Classes

Pandera, introduced in Section 2 on pandas frames, provides a Polars backend under pandera.polars (Bantilan 2020). A DataFrameModel declares one typed attribute per column, Field arguments attach built in checks, and a method decorated with @pa.dataframe_check expresses the cross column rule. Setting strict = True makes an undeclared column a failure in its own right.

import pandera.polars as pa
from pandera.errors import SchemaErrors


class UserSchema(pa.DataFrameModel):
    user_id: pl.Int64 = pa.Field(gt=0, unique=True)
    age: pl.Int64 = pa.Field(in_range={"min_value": 0, "max_value": 120}, nullable=False)
    email: pl.String = pa.Field(str_matches=EMAIL_RE)
    country: pl.String = pa.Field(isin=ALLOWED)
    signup_date: pl.Date
    last_active: pl.Date
    spend: pl.Float64 = pa.Field(ge=0)

    @pa.dataframe_check
    def active_after_signup(cls, data: pa.PolarsData) -> pl.LazyFrame:
        return data.lazyframe.select(pl.col("last_active") >= pl.col("signup_date"))

    class Config:
        strict = True  # an unexpected column is itself a failure


UserSchema.validate(clean)  # the clean frame passes silently

try:
    UserSchema.validate(batch, lazy=True)
except SchemaErrors as exc:
    pandera_failures = exc.failure_cases

pandera_failures.select("column", "check", "index", "failure_case")

shape: (8, 4)

column	check	index	failure_case
str	str	i32	str
"user_id"	"field_uniqueness"	9	"10"
"user_id"	"field_uniqueness"	10	"10"
"age"	"not_nullable"	200	null
"age"	"in_range(0, 120)"	100	"250"
"email"	"str_matches('^[\w.+-]+@[\w-]+\…	300	"bob_at_example.com"
"country"	"isin(['US', 'CA', 'GB', 'DE'])"	400	"FR"
"spend"	"greater_than_or_equal_to(0)"	600	"-12.5"
"UserSchema"	"active_after_signup"	500	"{"user_id":501,"age":33,"email…

With lazy=True, Pandera collects every failure into a single SchemaErrors exception whose failure_cases attribute is itself a Polars frame: one row per failing value, with the check that failed and the row index where it failed. Both members of the duplicated key appear, rows 9 and 10, because uniqueness is a property of the whole column, and the cross column rule reports row 500 with the offending record serialized as its failure case. The eight distinct indices are exactly the reference quarantine set.

The schema also doubles as a type annotation. Decorating a function with @pa.check_types validates its annotated arguments at call time, so a transformation cannot begin on a batch that breaks its input contract. This is the inline, fail fast placement that Section 7.1 recommends after each transformation.

from pandera.typing.polars import DataFrame


@pa.check_types(lazy=True)
def spend_by_country(users: DataFrame[UserSchema]) -> pl.DataFrame:
    return users.group_by("country").agg(pl.col("spend").sum().round(2)).sort("country")


print(spend_by_country(clean))
try:
    spend_by_country(batch)
except SchemaErrors as exc:
    print(f"rejected before aggregating: {exc.failure_cases.height} failure cases")

shape: (4, 2)
┌─────────┬──────────┐
│ country ┆ spend    │
│ ---     ┆ ---      │
│ str     ┆ f64      │
╞═════════╪══════════╡
│ CA      ┆ 19816.72 │
│ DE      ┆ 20285.11 │
│ GB      ┆ 18467.98 │
│ US      ┆ 19434.31 │
└─────────┴──────────┘
rejected before aggregating: 8 failure cases

62.6.3 Patito: Pydantic Models for DataFrames

Patito reaches the same contract from the other direction: it is written as a Pydantic model, so the class that validates a whole frame is also an ordinary row model that can validate an API payload or a configuration record. Nullability follows Python typing (an int field rejects nulls, an int | None field accepts them), a Literal type replaces an allowed set, and pt.field stands for the column being defined when a constraint needs an arbitrary Polars expression.

from typing import Literal

import patito as pt
from patito.exceptions import DataFrameValidationError


class User(pt.Model):
    user_id: int = pt.Field(gt=0, unique=True)
    age: int = pt.Field(ge=0, le=120)  # not Optional, so a null is an error
    email: str = pt.Field(pattern=EMAIL_RE)
    country: Literal["US", "CA", "GB", "DE"]
    signup_date: date
    last_active: date = pt.Field(constraints=pt.field >= pl.col("signup_date"))
    spend: float = pt.Field(ge=0)


User.validate(clean)

try:
    User.validate(batch)
except DataFrameValidationError as exc:
    patito_errors = exc.errors()
    print(exc)

7 validation errors for User
age
  1 missing value (type=value_error.missingvalues)
user_id
  2 rows with duplicated values. (type=value_error.rowvalue)
age
  1 row with out of bound values. (type=value_error.rowvalue)
email
  1 row with out of bound values. (type=value_error.rowvalue)
country
  Rows with invalid values: {'FR'}. (type=value_error.rowvalue)
last_active
  1 row does not match custom constraints. (type=value_error.rowvalue)
spend
  1 row with out of bound values. (type=value_error.rowvalue)

Patito reports all seven violations in the familiar Pydantic error format, grouped by column. The messages carry counts (“2 rows with duplicated values”) and, for categorical violations, the offending values, but not row positions: Patito answers whether a frame conforms and leaves locating bad rows to the caller.

Its distinctive extra is fixture generation for the data unit tests of Section 5. Model.examples builds a valid frame from only the columns a test cares about and fills every other column with a value that satisfies its type and constraints. It cannot invent strings that match a regular expression, so a pattern constrained column such as email must be supplied explicitly.

fixture = User.examples({
    "user_id": [1, 2, 3],
    "email": ["a@x.io", "b@y.io", "c@z.io"],
    "spend": [0.0, 19.99, 1_000_000.0],  # boundary values the test cares about
})
User.validate(fixture)
fixture.select(User.columns)

shape: (3, 7)

user_id	age	email	country	signup_date	last_active	spend
i64	i64	str	enum	date	date	f64
1	60	"a@x.io"	"US"	1970-01-01	1970-01-01	0.0
2	60	"b@y.io"	"US"	1970-01-01	1970-01-01	19.99
3	60	"c@z.io"	"US"	1970-01-01	1970-01-01	1e6

62.6.4 Pointblank: Validation Plans with Severity Thresholds

Pointblank, from Posit, treats validation as a plan: a chain of named steps, each a check over one or more columns, interrogated together and summarized in a report. Its central idea is graded severity. Every step carries Thresholds for warning, error, and critical states, given either as absolute counts of failing rows (integers) or as failure fractions (floats), which is precisely the tolerance \(\tau_k\) of Section 6.1. Below, the three hard rules get critical=1, so a single failing row is critical, while soft rules escalate from warning at one failing row to error at 1 percent and critical at 5 percent. The plan opens with a schema step whose expected column names and types are read from the clean reference frame.

import pointblank as pb

HARD_T = pb.Thresholds(critical=1)  # one failing row is fatal
SOFT_T = pb.Thresholds(warning=1, error=0.01, critical=0.05)  # escalate with the rate

validation = (
    pb.Validate(data=batch, tbl_name="users", label="Nightly signup batch", thresholds=SOFT_T)
    .col_schema_match(schema=pb.Schema(tbl=clean), brief="schema")
    .rows_distinct(columns_subset=["user_id"], thresholds=HARD_T, brief="user_id unique")
    .col_vals_not_null(columns="age", thresholds=HARD_T, brief="age not null")
    .col_vals_between(columns="age", left=0, right=120, na_pass=True, brief="age in [0, 120]")
    .col_vals_regex(columns="email", pattern=EMAIL_RE, brief="email format")
    .col_vals_in_set(columns="country", set=ALLOWED, brief="country allowed")
    .col_vals_ge(
        columns="last_active", value=pb.col("signup_date"),
        thresholds=HARD_T, brief="active after signup",
    )
    .col_vals_ge(columns="spend", value=0, brief="spend >= 0")
    .interrogate()
)
validation.get_tabular_report()

Pointblank Validation
Nightly signup batch PolarsusersWARNING1ERROR0.01CRITICAL0.05
		STEP	COLUMNS	VALUES	TBL	EVAL	UNITS	PASS	FAIL	W	E	C	EXT
#4CA64C	1	col_schema_match() schema	—	SCHEMA		✓	1	1 1.00	0 0.00	○	○	○	—
#FF3300	2	rows_distinct() user_id unique	user_id	—		✓	1000	998 1.00	2 0.00	—	—	●
#FF3300	3	col_vals_not_null() age not null	age	—		✓	1000	999 1.00	1 0.00	—	—	●
#AAAAAA	4	col_vals_between() age in [0, 120]	age	[0, 120]		✓	1000	999 1.00	1 0.00	●	○	○
#AAAAAA	5	col_vals_regex() email format	email	^[\w.+-]+@[\w-]+\.[\w.]+$		✓	1000	999 1.00	1 0.00	●	○	○
#AAAAAA	6	col_vals_in_set() country allowed	country	US, CA, GB, DE		✓	1000	999 1.00	1 0.00	●	○	○
#FF3300	7	col_vals_ge() active after signup	last_active	signup_date		✓	1000	999 1.00	1 0.00	—	—	●
#AAAAAA	8	col_vals_ge() spend >= 0	spend	0		✓	1000	999 1.00	1 0.00	●	○	○
2026-10-17 23:20:26 UTC< 1 s2026-10-17 23:20:26 UTC
Notes Step 1 (schema_check) ✓ Schema validation passed. Schema Comparison TARGET EXPECTED COLUMN DATA TYPE COLUMN DATA TYPE 1 user_id Int64 1 user_id ✓ Int64 ✓ 2 age Int64 2 age ✓ Int64 ✓ 3 email String 3 email ✓ String ✓ 4 country String 4 country ✓ String ✓ 5 signup_date Date 5 signup_date ✓ Date ✓ 6 last_active Date 6 last_active ✓ Date ✓ 7 spend Float64 7 spend ✓ Float64 ✓ Supplied Column Schema: [('user_id', 'Int64'), ('age', 'Int64'), ('email', 'String'), ('country', 'String'), ('signup_date', 'Date'), ('last_active', 'Date'), ('spend', 'Float64')] Schema Match Settings COMPLETE IN ORDER COLUMN ≠ column DTYPE ≠ dtype float ≠ float64 Step 2 (local_thresholds) Step-specific thresholds set with C:1. Step 3 (local_thresholds) Step-specific thresholds set with C:1. Step 7 (local_thresholds) Step-specific thresholds set with C:1.

The report is a self contained HTML table, built with the Great Tables library, listing every step with its pass and fail counts, failure fractions, and threshold states. It is an artifact a data steward or analyst can read without opening Python, the communication role that Data Docs play in Great Expectations (Section 6.9).

Two calls turn the plan into pipeline behavior. assert_below_threshold is the gate: asked about the critical level it fails on exactly the three hard rules, while the four soft violations, each one row in a thousand, sit at warning and do not block. get_sundered_data is the quarantine, splitting the batch into passing and failing partitions.

try:
    validation.assert_below_threshold(level="critical")
except AssertionError as exc:
    print(exc)

passed = validation.get_sundered_data(type="pass")
failed = validation.get_sundered_data(type="fail")
print(f"pass partition: {passed.height} rows, fail partition: {failed.height} rows")
print(f"duplicated user_id rows left in the pass partition: {passed['user_id'].is_duplicated().sum()}")

The following steps exceeded the critical threshold level:
Step 2: Expect entirely distinct rows across `user_id`.
Step 3: Expect that all values in `age` should not be Null.
Step 7: Expect that values in `last_active` should be >= `signup_date`.
pass partition: 994 rows, fail partition: 6 rows
duplicated user_id rows left in the pass partition: 2

Note what the split does with the duplicated key. rows_distinct is a judgment about the table rather than about any single row (the \(F_{\text{uniq}}\) case of Section 6.1), so it does not take part in the row level split and both duplicate rows remain in the pass partition. When keys must be enforced at quarantine time, pair Pointblank’s report with an explicit deduplication step, or with a library that treats key violations as row failures, such as Pandera or Dataframely.

62.6.5 Validoopsie: Impact Levels and Tolerances on Any Backend

Validoopsie is the lightest of the five. It is written against Narwhals, so one validation runs unchanged on pandas, Polars, PyArrow, and the other engines Narwhals supports, and its catalog is organized into themed groups (NullValidation, ValuesValidation, StringValidation, and so on) chained from a single Validate object. Each check takes an impact level and a fractional threshold. Impact sets the consequence when validate() runs: a failed high impact check raises, a medium one is logged as an error, and a low one as a warning. The threshold is the tolerance \(\tau_k\), and a check whose failure fraction stays within it reports success.

For the low impact rules the tolerance should come from the reasoning of Section 4.4 rather than a guess. If a healthy batch fails a soft rule at a benign rate \(p_0\), the failure count is binomial with mean \(n p_0\) and variance \(n p_0 (1 - p_0)\), so a tolerance three standard errors above the benign rate is

\[ \tau = p_0 + 3 \sqrt{\frac{p_0 (1 - p_0)}{n}}. \]

With \(p_0 = 0.002\) and \(n = 1000\) this gives \(\tau \approx 0.0062\), about six failing rows per thousand before the check fails. Validoopsie has no built in check that orders two columns, so the date rule is written as a custom check by subclassing BaseValidation. Its __call__ receives a Narwhals frame and returns the failing values grouped with their counts, which keeps the custom rule backend agnostic as well.

import math

import narwhals as nw
from validoopsie import Validate
from validoopsie.base import BaseValidation


class ColumnNotBefore(BaseValidation):
    """Fail rows where `column` precedes `other`: a cross column ordering rule."""

    def __init__(self, column, other, impact="low", threshold=0.0, **kwargs):
        super().__init__(column, impact, threshold, **kwargs)
        self.other = other

    @property
    def fail_message(self) -> str:
        return f"'{self.column}' precedes '{self.other}' in some rows."

    def __call__(self, frame):
        return (
            frame.filter(nw.col(self.column) < nw.col(self.other))
            .group_by(self.column)
            .agg(nw.col(self.column).count().alias(f"{self.column}-count"))
        )


p0 = 0.002
tau = p0 + 3 * math.sqrt(p0 * (1 - p0) / batch.height)

vd = (
    Validate(batch)
    .UniqueValidation.ColumnUniquePair(column_list=["user_id"], impact="high")
    .NullValidation.ColumnNotBeNull(column="age", impact="high")
    .add_validation(ColumnNotBefore(column="last_active", other="signup_date", impact="high"))
    .ValuesValidation.ColumnValuesToBeBetween(column="age", min_value=0, max_value=120, impact="medium")
    .StringValidation.PatternMatch(column="email", pattern=EMAIL_RE, impact="medium")
    .UniqueValidation.ColumnUniqueValuesToBeInList(
        column="country", values=ALLOWED, impact="low", threshold=tau
    )
    .ValuesValidation.ColumnValuesToBeBetween(column="spend", min_value=0, impact="low", threshold=tau)
)

print(f"tolerance for low impact rules: tau = {tau:.4f}")
vd_summary = pl.DataFrame([
    {
        "check": key,
        "impact": res["impact"],
        "status": res["result"]["status"],
        "failed": res["result"].get("failed_number", 0),
        "fraction": res["result"].get("failed_percentage", 0.0),
    }
    for key, res in vd.results.items()
    if key != "Summary"
])
vd_summary

tolerance for low impact rules: tau = 0.0062

shape: (7, 5)

check	impact	status	failed	fraction
str	str	str	i64	f64
"ColumnUniquePair_user_id"	"high"	"Fail"	2	0.002
"ColumnNotBeNull_age"	"high"	"Fail"	1	0.001
"ColumnNotBefore_last_active"	"high"	"Fail"	1	0.001
"ColumnValuesToBeBetween_age"	"medium"	"Fail"	1	0.001
"PatternMatch_email"	"medium"	"Fail"	1	0.001
"ColumnUniqueValuesToBeInList_c…	"low"	"Success"	1	0.001
"ColumnValuesToBeBetween_spend"	"low"	"Success"	1	0.001

The summary shows all three tiers at work. The two low impact checks each see one failing row, a fraction of 0.001 that falls inside \(\tau\), so they report success while still recording the count for monitoring. The two medium checks fail with zero tolerance but are only logged. Validoopsie logs through Loguru, so routing that log to a compact sink at warning level and above makes the tiers visible: three critical lines for the high impact failures, two error lines for the medium ones, and nothing for the low impact checks that stayed within tolerance. Only the high impact failures stop the pipeline:

import sys

from loguru import logger

logger.remove()
logger.add(sys.stdout, level="WARNING", format="{level: <8} {message}", colorize=False)

try:
    vd.validate()
except ValueError as exc:
    print(f"\nraised: {exc}")

CRITICAL Failed validation: ColumnUniquePair_user_id - Duplicate entries found: The combination of columns [user_id] contains non-unique values.
CRITICAL Failed validation: ColumnNotBeNull_age - The column 'age' has values that are null.
CRITICAL Failed validation: ColumnNotBefore_last_active - 'last_active' precedes 'signup_date' in some rows.
ERROR    Failed validation: ColumnValuesToBeBetween_age - The column 'age' has values that are not between 0 and 120.
ERROR    Failed validation: PatternMatch_email - The column 'email' has entries that do not match the pattern '^[\w.+-]+@[\w-]+\.[\w.]+$'.

raised: Failed Validation(s): ['ColumnUniquePair_user_id', 'ColumnNotBeNull_age', 'ColumnNotBefore_last_active']

Because every check is expressed through Narwhals, the same validation runs on a pandas copy of the batch without modification:

on_pandas = Validate(batch.to_pandas()).ValuesValidation.ColumnValuesToBeBetween(
    column="spend", min_value=0, impact="high"
)
on_pandas.results["ColumnValuesToBeBetween_spend"]["result"]

{'status': 'Fail',
 'threshold_pass': False,
 'message': "The column 'spend' has values that are not between 0 and None.",
 'failing_items': [-12.5],
 'failed_number': 1,
 'frame_row_number': 1000,
 'threshold': 0.0,
 'failed_percentage': 0.001}

62.6.6 Dataframely: Polars Native Schemas, Quarantine, and Sampling

Dataframely, from QuantCo, is Polars native and the most opinionated about types. Columns are non nullable unless declared otherwise, a primary_key flag replaces a separate uniqueness check, and cross column logic is a method decorated with @dy.rule() that returns a Polars expression evaluated per row. Its central operation is filter, which never raises: it returns the rows that satisfy every rule together with a FailureInfo object describing the rest. The quarantine pattern of Section 7.2 is built into the API.

import dataframely as dy


class UserTable(dy.Schema):
    user_id = dy.Int64(primary_key=True, min=1)
    age = dy.Int64(min=0, max=120)  # non nullable by default
    email = dy.String(regex=EMAIL_RE)
    country = dy.String(check=lambda c: c.is_in(ALLOWED))
    signup_date = dy.Date()
    last_active = dy.Date()
    spend = dy.Float64(min=0)

    @dy.rule()
    def active_after_signup(cls) -> pl.Expr:
        return pl.col("last_active") >= pl.col("signup_date")


valid, failure = UserTable.filter(batch)
print(f"{valid.height} rows promoted, {failure.invalid().height} rows quarantined")
failure.counts()

992 rows promoted, 8 rows quarantined

{'active_after_signup': 1,
 'primary_key': 2,
 'age|nullability': 1,
 'age|max': 1,
 'email|regex': 1,
 'country|check': 1,
 'spend|min': 1}

The counts match the reference implementation rule for rule, including two rows for the primary key, and the eight quarantined rows are exactly the reference quarantine set. FailureInfo can also report which rules fail together (cooccurrence_counts) and be written to Parquet for later inspection. When fail fast behavior is wanted instead, validate raises with a summary of the same counts:

try:
    UserTable.validate(batch)
except dy.exc.ValidationError as exc:
    print(exc)

7 rules failed validation:
 - 'active_after_signup' failed for 1 rows
 - 'primary_key' failed for 2 rows
 * Column 'age' failed validation for 2 rules:
   - 'nullability' failed for 1 rows
   - 'max' failed for 1 rows
 * Column 'email' failed validation for 1 rules:
   - 'regex' failed for 1 rows
 * Column 'country' failed validation for 1 rules:
   - 'check' failed for 1 rows
 * Column 'spend' failed validation for 1 rules:
   - 'min' failed for 1 rows

Two further features set Dataframely apart. A validated frame is typed as dy.DataFrame[UserTable], so a function signature can demand a frame that has already passed the schema and a static type checker flags code that passes an unvalidated one. And sample generates synthetic frames that satisfy the entire schema, including the cross column rule and the primary key, which turns the schema into a source of property based test fixtures (Section 5.2). The generator deliberately explores extreme but valid values, very large keys and amounts and dates centuries apart, because those are the inputs that break transformation code. Columns with a custom check cannot be sampled automatically, so country receives explicit values.

sample = UserTable.sample(
    num_rows=5,
    overrides={"country": ["US", "CA", "GB", "DE", "US"]},
    generator=dy.random.Generator(seed=55),
)
print(f"sample satisfies the schema: {UserTable.is_valid(sample)}")
sample.select("user_id", "age", "country", "signup_date", "last_active", "spend")

sample satisfies the schema: True

shape: (5, 6)

user_id	age	country	signup_date	last_active	spend
i64	i64	str	date	date	f64
7748439085172366648	29	"US"	0620-10-04	8594-12-30	8.8547e307
8030288389114823676	91	"CA"	0678-10-14	3988-07-22	1.4687e307
6860717400408217482	20	"GB"	2725-12-06	8229-03-05	9.5329e307
5482772210614667371	81	"DE"	7333-11-05	7404-05-05	1.5018e308
4467195370108145635	110	"US"	5588-06-02	9966-03-04	9.3465e307

Beyond single tables, a dy.Collection groups several schemas and adds rules across them, such as requiring every order to reference an existing user, which brings the referential checks of Section 3.1 into the same declarative framework.

62.6.7 Scorecard: One Contract, Five Verdicts

The cell below gathers what each library reported on the defective batch.

vo_results = [res for key, res in vd.results.items() if key != "Summary"]
scorecard = pl.DataFrame({
    "library": ["reference", "pandera", "patito", "pointblank", "validoopsie", "dataframely"],
    "rules_flagged": [
        truth.height,
        pandera_failures.select("column", "check").n_unique(),
        len(patito_errors),
        sum(count > 0 for count in validation.n_failed().values()),
        sum(res["result"].get("failed_number", 0) > 0 for res in vo_results),
        len(failure.counts()),
    ],
    "rows_isolated": [
        quarantine.height,
        pandera_failures["index"].n_unique(),
        None,
        validation.get_sundered_data(type="fail").height,
        None,
        failure.invalid().height,
    ],
    "default_on_failure": [
        "none (plain expressions)",
        "raise (all failures with lazy=True)",
        "raise",
        "report; assert by severity level",
        "raise on high impact only",
        "filter splits; validate raises",
    ],
})
scorecard

shape: (6, 4)

library	rules_flagged	rows_isolated	default_on_failure
str	i64	i64	str
"reference"	7	8	"none (plain expressions)"
"pandera"	7	8	"raise (all failures with lazy=…
"patito"	7	null	"raise"
"pointblank"	7	6	"report; assert by severity lev…
"validoopsie"	7	null	"raise on high impact only"
"dataframely"	7	8	"filter splits; validate raises"

All five libraries flag the same seven rules as the reference. They differ in what they hand back. Pandera and Dataframely isolate exactly the eight rows of the reference quarantine set. Pointblank’s row level split isolates six, because its uniqueness step judges the table rather than individual rows. Patito and Validoopsie report rule level verdicts and counts rather than row positions, which is enough for a gate and for monitoring but not, on its own, for quarantine. The broader differences are summarized below.

	Pandera	Patito	Pointblank	Validoopsie	Dataframely
Contract style	Class or object schema	Pydantic model	Chained validation plan	Chained validation plan	Class schema with rules
Engines	pandas, Polars, PySpark, Ibis, and others	Polars	Polars, pandas, SQL databases via Ibis	Any Narwhals backend	Polars
On failure	Raises; collects all with lazy=True	Raises with all errors	Reports; asserts by severity on request	Raises for high impact only	filter splits; validate raises
Severity model	Error, or warning per check	Error only	Warning, error, critical thresholds	Impact level plus tolerance	Error only
Row level output	Failure cases with row index	Counts and values	Pass and fail partitions	Failing values and counts	Valid frame plus FailureInfo
Human facing report	Failure case frame	Pydantic style summary	HTML report table	Text summary table	Rule count summary
Test data generation	Hypothesis strategies (pandas)	Model.examples	generate_dataset	None built in	Schema.sample
Distinctive extra	check_types decorator	One model for rows and frames	YAML plans, step reports	Small custom check API	Multi table Collection

62.6.8 Choosing Among Them

The choice follows the shape of the codebase and the audience for the results. Reach for Pandera when one validation vocabulary must span several engines (pandas in notebooks, Polars in pipelines, PySpark at scale) or when typed function boundaries matter, since check_types validates at every call. Reach for Patito when Pydantic models are already the lingua franca, for example when the same record shape arrives through an API and leaves through a dataframe, and one class should govern both. Reach for Pointblank when the people who must act on validation results are analysts, stewards, or managers, when graded severity matters more than pass or fail, or when the data lives in a SQL database. Reach for Validoopsie when the goal is a small dependency footprint, backend portability, and tiered severity in a few lines. Reach for Dataframely in Polars native pipelines that need quarantine, typed frames, multi table rules, or schema driven test data.

The libraries also compose. A realistic arrangement for the zones of Section 7.1 places Dataframely or Pandera at the schema and constraint gates inside transformation code, where typed frames and row level quarantine matter, and Pointblank at the statistical and reporting gate, where results must be read and acted on by people. The cost of combining tools is that the contract is written more than once. Keep one authoritative definition, such as the reference expressions of Section 6.1, and test in continuous integration that every library’s translation agrees with it on a fixture batch, exactly as the scorecard does.

62.6.9 Adjacent Tooling

Great Expectations remains the most widely deployed framework in this space. It is built around Expectations, declarative assertions such as expect_column_values_to_not_be_null, grouped into suites, run through checkpoints, and rendered as Data Docs, readable HTML reports that serve as both documentation and test results. It suits organizations where many teams share one catalog of expectations, at the cost of more configuration and a heavier conceptual footprint than the five libraries above. Soda offers a declarative checks language aimed at data reliability monitoring. Deequ, built on Spark, brings constraint verification and data profiling to very large datasets and can suggest constraints automatically (Schelter et al. 2018). Evidently focuses on drift and model monitoring with visual reports, the natural companion to the statistical validation of Section 4. The right combination depends on data scale, the prevailing language stack, and whether the emphasis is on batch validation or continuous monitoring.

62.7 Validating Data in Production Pipelines

62.7.1 Where to Place Validation Gates

Validation in production is about placement as much as technique. The standard pattern places gates at every boundary where data changes ownership: at ingestion, where external data enters the system; after each major transformation, where bugs are introduced; and before serving, where bad data would directly harm users or models. Validating only at the end leaves you knowing that something broke without knowing where. Validating at each boundary localizes failures to a specific stage.

A widely used architecture routes data through quality zones, often named for metals. Raw data lands in a bronze zone with minimal validation. Cleaned and conformed data moves to a silver zone after passing schema and constraint checks. Business level aggregates reach a gold zone only after statistical validation confirms they look reasonable. Each promotion between zones is a validation gate, and a record that fails a gate is diverted rather than silently promoted.

flowchart LR
    SRC["External source"] --> G1{"Schema gate"}
    G1 -->|"pass"| BRONZE["Bronze zone raw"]
    G1 -->|"fail"| Q1["Quarantine"]
    BRONZE --> G2{"Constraint gate"}
    G2 -->|"pass"| SILVER["Silver zone conformed"]
    G2 -->|"fail"| Q2["Quarantine"]
    SILVER --> G3{"Statistical gate"}
    G3 -->|"pass"| GOLD["Gold zone aggregates"]
    G3 -->|"alert"| MON["Monitoring and review"]
    GOLD --> SERVE["Serving and models"]

The gates are ordered from cheapest and strictest to most expensive and most forgiving. The schema gate is a hard structural check at the boundary. The constraint gate adds domain logic. The statistical gate, placed last, is usually a monitoring signal rather than a hard stop, which is why its failure edge routes to review rather than to quarantine.

62.7.2 Fail Fast versus Quarantine

When validation fails in production, the pipeline must decide what to do with the offending data. Two strategies dominate. The fail fast strategy halts the pipeline, alerts an operator, and prevents bad data from propagating, which is appropriate when downstream consumers cannot tolerate any corruption. The quarantine strategy diverts failing records to a separate location for later inspection while allowing valid records to proceed, which keeps the pipeline flowing when partial data is better than none.

The right choice depends on the use case. A financial reconciliation pipeline should fail fast because a wrong number is worse than a late number. A recommendation feature pipeline might quarantine a small fraction of malformed events and continue, because serving slightly incomplete recommendations beats serving none. Encoding this policy explicitly, rather than letting it emerge from where exceptions happen to propagate, is a mark of a mature pipeline. The libraries of Section 6 make the choice an API decision: Pandera’s validate, Patito’s validate, and Dataframely’s validate fail fast, while Dataframely’s filter and Pointblank’s get_sundered_data quarantine, returning the passing rows and the failing rows separately.

62.7.3 Batch versus Streaming Validation

Batch pipelines validate a whole dataset at once, which makes statistical checks straightforward because the full distribution is available. Streaming pipelines validate one record or a small window at a time, which constrains the kinds of checks that are feasible. A per record schema check is trivial in a stream. A distributional drift check is not, because no single record reveals a distribution.

Streaming systems address this by maintaining rolling windows and incremental statistics, computing approximate distributions over a recent window and comparing them against a baseline. Per record validation handles structural and constraint checks at the edge, while windowed aggregation handles statistical checks slightly behind the live edge. Records that fail structural checks are typically routed to a dead letter queue, the streaming analog of quarantine.

62.7.4 Validation as Monitoring

In production, validation results are themselves a stream of metrics. The null rate of a column, the number of quarantined records, the drift score against baseline, and the pass rate of each expectation suite should all be emitted to a monitoring system and tracked over time. This transforms validation from a binary gate into an observable signal. A slow upward creep in a null rate, invisible to any single threshold check, becomes obvious on a time series chart and lets teams intervene before the rate crosses a failure boundary.

Effective teams alert on validation metrics with the same rigor they apply to service latency and error rates. A failing expectation suite should page someone, a rising quarantine rate should open a ticket, and a sustained drift signal should trigger an investigation into whether a model needs retraining.

62.7.5 Versioning Expectations with Data

Validation logic evolves as understanding of the data deepens. New constraints get added, thresholds get tuned, and obsolete checks get removed. Because validation suites are code, they belong in version control alongside the pipeline they protect, reviewed through the same pull request process. When a validation rule changes, the change history explains why, which is invaluable during an incident review months later. Treating expectations as versioned artifacts also enables a disciplined workflow: propose a new constraint, test it against historical data to estimate its failure rate, and only then promote it to a blocking gate.

62.8 Building a Validation Strategy

62.8.1 Start at the Boundaries

A team cannot validate everything at once, and attempting to do so produces a brittle thicket of checks that nobody maintains. The pragmatic starting point is to enforce schemas at the boundaries where external data enters the system, because that is where the most defects originate and where a single check protects the most downstream code. Schema enforcement at ingestion delivers the highest return for the least effort.

62.8.2 Layer in Depth Over Time

Once boundary schemas are in place, layer additional validation where incidents reveal it is needed. Each production data incident is a lesson that should become a permanent check, so that the same defect can never recur silently. This incident driven growth keeps the validation suite focused on real risks rather than hypothetical ones, and it ensures that every check earns its maintenance cost by guarding against a failure that actually happened.

62.8.3 Balancing Coverage and Maintenance

Every validation rule is code that must be maintained, and an overzealous suite that fires constantly on benign variation trains a team to ignore it, which is worse than having no validation at all. The goal is not maximal coverage but calibrated coverage, where each check that fires reliably signals a real problem worth a human’s attention. Periodically auditing which checks have fired, which have never fired, and which fire so often they are ignored keeps the suite healthy and trusted.

62.8.4 When to Use Each Layer, and the Pitfalls of Each

The four validation layers are not interchangeable, and a healthy strategy reaches for each where it is strongest.

Reach for schema validation at every boundary where data enters or changes ownership. It is cheap, declarative, and catches the structural defects that cause the loudest downstream failures. Its pitfall is over strictness: a schema that rejects any unexpected column will halt a pipeline when an upstream team adds a harmless field, so reserve strict mode for boundaries where unexpected columns genuinely signal a problem.

Reach for constraint checks when domain logic constrains values beyond their type, especially for cross field invariants that no schema can express. Their pitfall is brittleness, a constraint encoding an assumption that was only ever approximately true (every order has a positive amount, every user has exactly one country) will fire on legitimate edge cases and erode trust.

Reach for statistical validation to catch the collectively wrong data that passes every per record check. Its pitfalls are the threshold problem and the significance trap. Tune thresholds against replayed history rather than guessing, prefer effect size metrics such as PSI and JSD over raw \(p\) values for production gates, and treat most statistical checks as monitoring signals rather than hard blocks.

Reach for data unit tests to verify the transformation code itself, in continuous integration, against fixed fixtures. Their pitfall is fixture rot, a fixture that no longer resembles production data will pass while the real pipeline breaks, so refresh fixtures from sampled production records and include the edge cases that past incidents revealed.

The deepest pitfall spans all four layers: a validation suite that fires constantly on benign variation trains a team to ignore it, which is strictly worse than no validation at all. Calibrated coverage, where every firing check reliably signals a real problem, is the goal, not maximal coverage.

62.9 References

1. Bantilan, N. “pandera: Statistical Data Validation of Pandas Dataframes.” Proceedings of the 19th Python in Science Conference (SciPy 2020), 116-124. https://doi.org/10.25080/Majora-342d178e-010
2. Pandera Documentation. https://pandera.readthedocs.io/
3. Martinussen, J. G., and Aarholt, T. Patito: A Dataframe Modelling Library Built on Polars and Pydantic. https://github.com/JakobGM/patito
4. Iannone, R. Pointblank for Python: Data Validation Made Beautiful and Powerful. Posit Software. https://github.com/posit-dev/pointblank
5. Soliev, A. Validoopsie: A Simple and Light Data Validation Library. https://github.com/akmalsoliev/Validoopsie
6. Albert, A., Elsner, D., and Borchert, O. Dataframely: A Declarative, Polars Native Data Frame Validation Library. QuantCo. https://github.com/quantco/dataframely
7. Vink, R., et al. Polars. Zenodo. https://doi.org/10.5281/zenodo.7697217
8. Gorelli, M., et al. Narwhals: A Lightweight Compatibility Layer Between Dataframe Libraries. https://github.com/narwhals-dev/narwhals
9. Great Expectations Documentation. https://docs.greatexpectations.io/
10. Great Expectations, “Data Docs.” https://docs.greatexpectations.io/docs/reference/learn/terms/data_docs/
11. dbt Tests Documentation. https://docs.getdbt.com/docs/build/data-tests
12. Soda Core Documentation. https://docs.soda.io/
13. Amazon Deequ: Unit Tests for Data. https://github.com/awslabs/deequ
14. Evidently AI Documentation. https://docs.evidentlyai.com/
15. MacIver, D. R., Hatfield-Dodds, Z., et al. “Hypothesis: A New Approach to Property-Based Testing.” Journal of Open Source Software 4(43), 1891, 2019. https://doi.org/10.21105/joss.01891
16. Schelter, S., Lange, D., Schmidt, P., Celikel, M., Biessmann, F., and Grafberger, A. “Automating Large-Scale Data Quality Verification.” Proceedings of the VLDB Endowment 11(12), 1781-1794, 2018. https://doi.org/10.14778/3229863.3229867
17. Breck, E., Polyzotis, N., Roy, S., Whang, S. E., and Zinkevich, M. “Data Validation for Machine Learning.” Proceedings of Machine Learning and Systems (MLSys), 2019. https://mlsys.org/Conferences/2019/doc/2019/167.pdf
18. Polyzotis, N., Roy, S., Whang, S. E., and Zinkevich, M. “Data Lifecycle Challenges in Production Machine Learning: A Survey.” ACM SIGMOD Record 47(2), 17-28, 2018. https://doi.org/10.1145/3299887.3299891
19. Apache Avro Schema Specification. https://avro.apache.org/docs/current/specification/
20. Confluent Schema Registry Documentation. https://docs.confluent.io/platform/current/schema-registry/index.html
21. Massey, F. J. “The Kolmogorov-Smirnov Test for Goodness of Fit.” Journal of the American Statistical Association, 1951. https://doi.org/10.1080/01621459.1951.10500769
22. Lin, J. “Divergence Measures Based on the Shannon Entropy.” IEEE Transactions on Information Theory, 1991. https://doi.org/10.1109/18.61115
23. Gama et al., “A Survey on Concept Drift Adaptation.” ACM Computing Surveys, 2014. https://doi.org/10.1145/2523813