Repository navigation
A/B experiments: a flag measured against a metric - #10
Merged
Merged
Conversation
An experiment binds a flag to a conversion metric; assignment reuses the engine's deterministic bucketing, so the new module is pure data and pure math. Results carry a 95% Wilson interval per variant — chosen over the normal approximation because early experiments live exactly where that one misbehaves — and a pooled two-proportion z-test against the control, with the normal CDF via Abramowitz & Stegun 7.1.26 rather than a statistics dependency. Snapshots gain a default-empty `experiments` list holding the running ones, their versions folded into the snapshot version so starting or stopping an experiment is a visible change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two tables. `experiments` holds the definition — flag, metric, control variant — with a one-way draft/running/stopped lifecycle: a stopped experiment cannot restart, because reopening a measurement window would average two populations into an answer about neither. `experiment_counters` holds hourly (variant, kind) tallies incremented in place, so the table is bounded by hours times variants rather than by traffic, and carries no notify trigger — a cache invalidation per ingest would be a storm. Ingest collapses a batch into cells first (Postgres refuses ON CONFLICT touching a row twice in one statement) and joins on running experiments, so deltas straddling a stop are dropped as routine traffic rather than punished as errors. Deleting a flag is refused while experiments reference it, same contract as segments, with a RESTRICT foreign key as the backstop. Running experiments now travel in the snapshot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Management routes mirror segments — environment-scoped, audited, with the storage lifecycle surfacing as 409s that say what to do rather than what went wrong. The measurement fields are fixed at creation: changing what a running experiment measures would splice two questions into one answer, so PATCH takes presentation fields only. POST /api/v1/events is the SDK half: batches of counter increments, authenticated by SDK key, aggregated into hourly cells in one UNNEST round trip. Deltas naming an unknown or stopped experiment are dropped and reported, not errored — a batch straddling a stop is routine. GET .../results runs the core statistics over the counters, zero-filled in flag-variant order so every arm renders before its first exposure. The snapshot response now carries running experiments, and the seed plants one per environment on checkout.v2: running with a week of deterministic counters in production, a draft in staging. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The client never sends individual events. Exposures land in an in-process counter map as flags under experiment are evaluated; track(metric, ctx) attributes conversions by re-running the same deterministic evaluation the exposure went through, so both sides of a rate land on the same arm with no per-user state. A background task ships the counts every ten seconds, chunked to the server's batch cap; a failed flush restores its counts and retries, so an outage delays counters rather than losing them. The cap on distinct cells refuses new cells but never existing ones — a runaway key generator is contained while healthy experiments keep counting. The agreement suite gains the property the design rests on: the counters the server ends up holding must equal a tally the test keeps by hand from local assignment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Organised like the segments page — pick an environment, work inside it — with lifecycle actions that mirror the API's state machine. The results view puts one meter per arm on a shared scale snapped to the next 5%, the 95% interval as end ticks over the fill, and the exact numbers printed beside every bar: the meter is the shape, never the source. Fill colours reuse the distribution bar's position-keyed tokens, so an arm keeps its colour between the two views, and the z-test verdict is words with a badge, never colour alone. The create form defaults the control to the flag's off variant — the behaviour traffic outside the experiment would have seen. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A feature bullet, a dashboard section ending in a verdict, a design note on why the counters are bounded by time rather than traffic, the SDK example gaining track(), and the test count moving to 244 with the new layers accounted for. All eight screenshots come from one scripted pass over a fresh seed — the project header gained an Experiments button, which dates every shot that shows it, and a gallery mixing UI states looks worse than a dated one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ssebv
force-pushed
the
feat/experiments
branch
from
August 11, 2026 03:21
2d0c674 to
3e13e88
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An experiment binds a flag to a conversion metric; the flag's variants become the arms and the traffic already being served becomes the sample. Six commits, each buildable on its own, bottom of the stack upward:
ExperimentSpecplus pure statistics: a 95% Wilson interval per arm (chosen over the normal approximation because young experiments live exactly where that one misbehaves) and a pooled two-proportion z-test against the control, with the normal CDF via Abramowitz & Stegun rather than a statistics crate. Snapshots carry the running experiments, versions folded in so starting or stopping one is a visible change.experimentswith a one-way draft → running → stopped lifecycle (a reopened measurement window would average two populations into an answer about neither), andexperiment_counters: hourly cells incremented in place, so the table is bounded by hours × variants rather than by traffic. Deleting a flag is refused while experiments reference it, same contract as segments, with a RESTRICT foreign key as backstop.POST /api/v1/eventsfor SDK counter batches (one UNNEST round trip; deltas straddling a stop are dropped and reported, not errored), judged results zero-filled in flag-variant order, and a seeded experiment with a week of deterministic counters so the first screen anyone opens has a verdict on it.track(metric, ctx)attributes conversions by re-running the same deterministic evaluation, so both sides of the rate land on the same arm with no per-user state. A background flusher ships counts every ten seconds; failed flushes restore their counts and retry.track()in the SDK example, test count to 244, and all eight screenshots regenerated in one pass.The agreement suite gains the property the design rests on: the counters the server ends up holding must equal a tally the test keeps by hand from local assignment.
🤖 Generated with Claude Code