Repository navigation
Build a baseline pilot clustering model #216
Description
Is your feature request related to a problem? Please describe.
The client wants 10 to 20 pilots grouped by performance across heading, vertical speed, altitude and airspeed combined, with pilot rating and flight hours factored in. There is no clustering code in the repo and no unsupervised model of any kind.
Describe the solution you'd like
A baseline that forms performance groups (see GLOSSARY.md) from the per-pilot deviation features of #215. It runs on synthetic pilots now so it is ready when the client's telemetry arrives.
Pipeline: generate_synthetic_pilots.py → compute_features.py → cluster_pilots.py.
- Inputs.
stdand|bias|for each flight dynamics factor, 8 features in total, read frompilot_features.csv. Each feature is clustered as log(x + floor) (floors 1 ft, 0.1°, 2 ft/min, 0.1 kt) and then standardized over the cohort, because linear features recovered the synthetic tiers at only about 0.40 agreement. See ADR 0002. Notmad, signedbias,rows_usedorflights. Seedocs/adr/0002-cluster-on-std-and-absolute-bias.md. - Pilot metadata is not an input. It is compared against the groups afterwards, once Collect pilot metadata and join it to telemetry for clustering #214 lands.
pilot_idandflight_idare identifiers only. Seedocs/adr/0001-metadata-checks-clusters-not-shapes-them.md. - Scaling. One scaler fit on the whole cohort, never per pilot.
- Missing values. If any pilot has a NaN feature, the script stops with an error that names the pilots and metrics. Nothing is dropped or imputed.
- Choosing k. KMeans, never picking k automatically:
- With no
--k: print a sweep over k = 2..min(6, n // 3) with silhouette and subsample stability. Subsample stability is the mean adjusted Rand index over about 50 refits on about 80% of pilots, at a fixed seed. Nothing is written. - With
--k N: fit once and writepilot_clusters.csv(pilot_id,group) plus each group's centre in raw units (ft, deg, ft/min, kts).
- With no
- Synthetic pilots. A new
generate_synthetic_pilots.py.generate_balanced_data.pystays untouched because the labeler and the Random Forest depend on it.- 15 pilots in 3 hidden skill tiers (5 per tier), each with 2 to 4 flights of 2 minutes at 20 Hz, and a different constant target per flight.
- Deviation per metric is a steady bias with a random sign per flight, plus wobble that is correlated over time (AR(1)). The tier sets the size of both, with variation per metric (for example good altitude, poor heading (not yet; tracked in the mixed-skill follow-up)").
- Values use post-Plugin applies metric conversion factors to pilot altitude and airspeed that are already in feet and knots #196 units and a fixed seed.
- The true tier is written to a separate file (
pilot_id,tier) and never appears in the telemetry.
- Layout. A pure
clustering.py(DataFrame in, results out), a thin CLIcluster_pilots.py, andtests/test_clustering.py, followingfeatures.py/compute_features.py.
Acceptance criteria
-
generate_synthetic_pilots.pywrites telemetry thatcompute_features.pyaccepts, plus the separate tier file -
cluster_pilots.pywith no--kprints silhouette and subsample stability per k and writes nothing -
cluster_pilots.py --k Nwrites assignments and centres in raw units - Given the tier file, the script reports the adjusted Rand index against the true tier
- A test checks that the 3 tiers are recovered (mean adjusted Rand index ≥ 0.8 at k = 3 over seeds 0–5; a single 15-pilot cohort ranges from 0.64 to 1.0)
- A test checks that NaN features stop the script with an error naming the pilots
Describe alternatives you've considered
- Using metadata as clustering features, and clustering on all 12 deviation features. Both were rejected; see the two ADRs.
- Measuring stability across seeds only. Rejected because KMeans with
n_init=10gives nearly the same partition for any seed, so the measure would always look stable. - Saving the fitted scaler and centres. Deferred to the live-clustering issue, which is the first thing that needs them.
- Clustering on linear features. Rejected after testing; see ADR 0002.
Additional context
With 10 to 20 pilots the sample is far below the roughly 70 × features rule of thumb for segmentation (Dolnicar et al., 2014), so subsample stability is the main signal of whether the groups are real. If they are unstable, drop to 4 features. Live, in-flight assignment is tracked in #223 .
Activity
jmacd867 commentedon Oct 3, 2026
generate_synthetic_pilots.py. Leavegenerate_balanced_data.pyuntouched, because the labeler and Random Forest training depend on it.jmacd867 commentedon Oct 9, 2026
Superseded: the body above now defines stability as subsample stability (mean adjusted Rand index over about 50 refits on 80% of pilots). Seed-only stability was rejected because KMeans with
n_init=10returns nearly the same partition for any seed.jmacd867 commentedon Oct 12, 2026
Figures in the acceptance criteria changed in #225:
See the Evidence section of
docs/adr/0002-cluster-on-std-and-absolute-bias.md.jmacd867 commentedon Oct 15, 2026
How the client's data lines up with this design:
candidate_ksraises below 6 pilots. Flights per pilot are uneven: 6, 6, 1, 4 and 2.