featureguard

Catch schema breaks, numerical faults and distribution drift before they reach a model.

Three ways a feature pipeline breaks without raising anything.

Columns get reordered. Nothing crashes. A linear layer has no idea that column 3 used to be flow_iat_mean and is now bwd_packet_len_std, so it keeps producing confident, meaningless predictions. Shape, dtype and value ranges all still look correct, which is exactly why this one survives code review.

A NaN appears. It propagates through a dense layer to every output of that layer, so one bad cell poisons the whole prediction. NaNs arrive from a division by a zero-duration flow, from a join that missed, from a sensor dropping out.

The distribution moves. The model starts extrapolating and nothing tells you.

featureguard checks for all three on the way in.

from featureguard import FeatureGuard, Schema

guard = FeatureGuard(Schema.flowlens()).fit(training_features)

report = guard.check(batch)
if not report.ok:
    print(report)

Cost

Measured on 56-column batches, median over 200 runs, against a 50,000-row reference.

Batch rows	Schema	+ sanity	+ drift	p99
1	2.2 us	9.5 us	442 us	521 us
32	2.2 us	21 us	1,078 us	1,271 us
256	2.2 us	66 us	1,190 us	1,395 us
4,096	2.2 us	939 us	3,339 us	3,642 us

The schema column is the interesting one: 2.2 us whether the batch is one row or four thousand. It never touches the data. It reads dtype, shape and the column index, which is what "zero-copy validator" actually means. Copying a 4,096 by 56 matrix to check its column order would cost about 1.8 MB of churn per request.

Sanity scales with rows, as it must, since it inspects every cell.

Drift is expensive, roughly a millisecond even on small batches, because binning is a pass over the data with per-column work. If your latency budget cannot absorb that, pass check_drift=False and run drift on a sampled side-channel instead. The library does not pretend this is free.

Bounded memory

Keeping the last N batches of raw features to compare against a reference costs memory proportional to throughput. featureguard keeps histograms.

After 128,000 rows through a 200-batch window:

Window memory	879 KiB
Equivalent raw history	57 MB
Ratio	64x

And it is flat. Bin edges are fitted once, each batch is reduced to counts, and a ring buffer holds per-batch counts with a running total updated incrementally, so eviction is a subtraction rather than a recomputation. Memory is window_batches x n_features x n_bins x 8 bytes whatever the traffic.

The cost is resolution. Everything is measured on the reference's bin grid, so drift happening entirely inside one bin is invisible.

The three modules

Schema lock

schema = Schema.flowlens()          # the 56 columns flowlens emits
schema.save("model/schema.json")    # ship it next to the model

schema.validate(batch)                       # array, tensor or DataFrame
schema.validate(batch, names=column_names)   # when the array cannot carry names

Order is part of the contract, so comparison is order-sensitive, and a reordering gets its own message rather than a generic failure:

columns are reordered: 12 of 56 are in the wrong position, e.g. 'flow_iat_mean'
expected at index 15 but found at 22. Values and dtype look correct, which is
exactly why this would otherwise pass unnoticed.

schema.reorder(frame) and schema.align(array, names) fix it once you have decided a reordering is acceptable rather than a bug.

Sanity

from featureguard import SanityChecker, SanityConfig, Action

checker = SanityChecker(names, SanityConfig(
    on_non_finite=Action.RAISE,      # NaN and inf
    on_out_of_bounds=Action.WARN,    # outside the fitted range
    on_dead=Action.WARN,             # constant across the batch
)).fit(reference)

Bounds are fitted from reference quantiles rather than hand-written, because nobody maintains hand-written bounds for 56 columns. The 0.1st and 99.9th percentiles are used rather than min and max, so a single outlier in the reference cannot set the range, plus a 10% margin because reference data never covers everything legitimate.

Action.REPAIR clips out-of-range values and imputes non-finite ones with the midpoint of the fitted bounds, deliberately not zero. Zero is a real and meaningful value for most of these features, so imputing it invents a plausible-looking observation. A repaired batch still reports what was repaired.

Action.IGNORE records the issue without failing the batch, so a check you have turned down to reporting-only is still visible to a metrics endpoint.

Drift

from featureguard import DriftDetector, DriftConfig

detector = DriftDetector(names, DriftConfig(n_bins=10, window_batches=200)).fit(reference)
for batch in stream:
    detector.update(batch)
report = detector.report()

Two statistics, and reporting both is deliberate.

PSI is sum((a - e) * ln(a / e)) over bins. It reacts sharply when mass appears in a previously empty bin. Its conventional thresholds (0.1 warn, 0.25 alarm) are folklore rather than theory, and they are defaults here, not truths.

Wasserstein distance is the earth-mover's distance from the CDFs, sum(|CDF_a - CDF_e| * width). It is in the units of the feature, so it is interpretable, and it reacts to mass moving a long way rather than to mass appearing somewhere new.

A shift can trip one and not the other, and which one fired tells you what kind of shift it was.

PSI has a noise floor, and it bites

This surfaced from a failing test rather than from theory. Under the null hypothesis, where the window genuinely is drawn from the reference distribution, the expected PSI is roughly

E[PSI] ~ (n_bins - 1) / n_rows

because the multinomial counts fluctuate around the reference proportions. With 10 bins and a 200-row window that is about 0.045, close enough to the 0.1 warn threshold to fire on data that never moved.

So min_rows defaults to 10 * (n_bins - 1) / psi_warn, which is 900 rows for the defaults, holding the floor an order of magnitude below the threshold. Below that the detector returns an explicitly unscored report rather than a number. DriftConfig exposes effective_min_rows and noise_floor so you can check the arithmetic for your own settings.

A drift detector that alarms on stationary data is worse than none, because people stop reading it.

Wiring it in

from featureguard import FeatureGuard, Schema, guard_input

guard = FeatureGuard(Schema.flowlens()).fit(training_features)

# as a decorator, on a serving handler or a forward pass
@guard_input(guard)
def forward(self, features):
    return self.net(features)

# or explicitly
report = guard.check(batch)
guard.stats()   # counters for a metrics endpoint

The decorator handles bound methods: args[0] is self on a forward, so it looks one along rather than making you say so.

Checks run schema, then sanity, then drift, and the order is not arbitrary. A schema failure stops the rest, because if the columns are wrong then every downstream number describes the wrong feature and the sanity report would be actively misleading.

With flowlens

import flowlens
from featureguard import FeatureGuard, Schema

reference = flowlens.extract("reference.pcap")
guard = FeatureGuard(Schema.flowlens()).fit(reference.features)

live = flowlens.extract("live.pcap")
report = guard.check(live.features)

Schema.flowlens() matches flowlens's 56-column output exactly. The column list is duplicated here rather than imported, so featureguard has no hard dependency on flowlens being installed.

Install

pip install -e ".[dev]"

Python 3.10+. Runtime dependencies are numpy and scipy. pandas and torch are optional and only needed for the paths that use them.

Limitations

• Drift is not cheap. About a millisecond per batch. Turn it off or sample it if that does not fit.
• Binned drift misses within-bin movement. Everything is measured on the reference's bin grid.
• PSI thresholds are convention. 0.1 and 0.25 come from credit-scoring practice, not from a derivation. Calibrate against your own data.
• Univariate drift only. Each feature is scored independently, so a change in the correlation structure with every marginal unchanged goes undetected.
• Bounds are quantile-based, so they assume the reference is clean. A reference containing the incident you want to catch will not catch it.
• No categorical support. Everything is treated as continuous, which suits flow features but not one-hot or high-cardinality columns.

Development

pytest              # 126 tests, 96% coverage
ruff check .
mypy                # strict

Twelve of those are property-based, in tests/test_properties.py. The failures this library exists for are the ones nobody wrote down — the reordering that took a model from 99.87% to 26.01% was not on anyone's list of cases to try — so those tests state the property and let Hypothesis hunt for a counterexample, shrinking it to a minimal frame when it finds one:

• a matching batch is never rejected, at any width or row count
• every column permutation is caught, and align inverts each one exactly
• PSI never alarms on stationary data at the detector's own min_rows, so the noise-floor derivation is tested as a property rather than at one configuration
• a large shift is always caught, so the quiet half does not come for free
• memory stays under the window's ceiling however long the stream runs

_{If pytest fails to start with a PluginValidationError mentioning nengo, an unrelated package in your environment ships an incompatible pytest plugin. Run pytest -p no:nengo, or use a clean virtual environment.}

License

MIT. See LICENSE.