[jmacd867]

Fixes #216

---

What was changed?

• Added inference/generate_synthetic_pilots.py: Synthetic pilots in 3 hidden skill tiers, with targets, in post-Plugin applies metric conversion factors to pilot altitude and airspeed that are already in feet and knots #196 units.
• Added inference/clustering.py and inference/cluster_pilots.py: forms Performance groups from per-pilot Deviation features.
• Added tests for all three: inference/tests/test_clustering.py, test_cluster_pilots.py, test_generate_synthetic_pilots.py.
• Added docs/adr/0001 (pilot metadata checks groups, it does not form them) and docs/adr/0002 (which features, and why a log scale).
• Updated GLOSSARY.md (Performance group, Synthetic pilot) and inference/README.md (steps 7a and 8).
• No existing code changed. features.py, compute_features.py and generate_balanced_data.py are untouched.

---

Why was it changed?

The client wants 10 to 20 pilots grouped by performance across heading, vertical speed, altitude and airspeed. The repo had no clustering and no way to test it, since real telemetry with targets doesn't exist yet (#213, #211). This adds a baseline that runs on synthetic pilots now and on real flights later.

Decisions that shape the result:

• Features are std and |bias| per metric (8 total), not mad or signed bias. Signed bias would split a pilot 50 ft high from one 50 ft low, and mad mostly repeats the other two. See ADR 0002.
• Features are clustered on log(x + floor). Linear features recovered the synthetic tiers at about 0.40 agreement (adjusted Rand index), the log scale at 0.87.
• Pilot metadata is not a feature. It is compared against the groups afterwards, once Collect pilot metadata and join it to telemetry for clustering #214 lands. See ADR 0001.
• The script never picks k. A person chooses it from the sweep.
• Stability is measured on subsamples, not seeds. KMeans with n_init=10 returns nearly the same partition for any seed, so a seed-based score would always look stable.

---

How was it changed?

generate_synthetic_pilots.py
  -> telemetry (pilot_id, flight_id, actuals, target_*)   and   tiers (pilot_id, tier)
compute_features.py                      # existing, unchanged
  -> pilot_features.csv
cluster_pilots.py
  no --k : sweep(k = 2..min(6, n // 3))  -> prints silhouette + subsample stability, writes nothing
  --k N  : fit(k)                        -> pilot_clusters.csv, group_centres.csv (raw units)
  --truth: also prints agreement with the true tiers

inference/
├── clustering.py                 # pure: clustering_features, sweep, fit, truth_agreement
├── cluster_pilots.py             # thin CLI
├── generate_synthetic_pilots.py  # tiers, per-flight bias + AR(1) wobble
└── tests/                        # three new test files

• NaN features stop the run with an error naming each pilot and metric. Nothing is dropped or imputed.
• Group centres are raw-unit means of each group's members, since KMeans centres live in log-scaled space.
• Synthetic deviation per flight is a steady bias with a random sign plus AR(1) wobble. The tier sets the size of both.

---

Screenshots (if applicable):

Before: no clustering code; uv run pytest ran 40 tests. After: 59 pass.

Tier recovery, the test that gates the PR:

for seed in 0..5:
    telemetry, truth = generate_pilots(seed)
    groups = fit(compute_features(telemetry), k=3)
    scores.append(truth_agreement(groups, truth))
assert mean(scores) >= 0.8        # 0.906 (single cohorts range 0.64 to 1.0)

End-to-end run on 15 synthetic pilots (seed 0):

$ python inference/cluster_pilots.py
 k  silhouette  stability
 2       0.429      0.882
 3       0.382      1.000
 4       0.347      0.951
 5       0.308      0.737

$ python inference/cluster_pilots.py --k 3 --truth inference/Data/synthetic_pilot_tiers.csv
 group  pilots  altitude_std  altitude_abs_bias  heading_std  ...
     0       5         82.46              67.96         5.60  ...
     1       5         15.99               8.84         0.78  ...
     2       5         41.22              32.23         2.80  ...
Agreement with true tiers (adjusted Rand index): 1.000

Silhouette favours k=2, stability peaks at k=3. That is the case for leaving the choice of k to a person.

---

Merge Danger

Door: two-way. All new files plus doc edits, so a revert is clean.

Blast Radius: inference only.

• Nothing in the plugin, relay or existing inference/ scripts changed.
• Running the scripts with defaults writes CSVs into inference/Data/ (about 100k rows for the synthetic telemetry). They are not gitignored, so don't commit them.
• Real-data runs are blocked on Record pilot_id and flight_id in the logged CSV #211 and Capture designated (target) values for altitude, heading, vertical speed and airspeed #213. Only synthetic pilots exercise this today. Hardware-bound parts were not touched, so nothing here needs X-Plane or iMotions.
• Not covered here: synthetic pilots who are strong on one metric and weak on another (follow-up issue), and live in-flight assignment (Assign a pilot to a performance group live, during a flight #223).

[Sanjana-Siddani]

Nice work! All tests pass and everything runs fine for me.

One fix before merging: abs_bias averages the bias across flights first and then takes the absolute value. So if a pilot is 50 ft too high on one flight and 50 ft too low on another, they cancel out and the pilot looks perfect. Taking the absolute value for each flight first and then averaging fixes this. When I tried it, the groups matched the real tiers on all 12 test seeds instead of only some.

Also, maybe add the generated CSVs to .gitignore. Will approve after the fix

[Phlabry]

ADR 0002's linear-versus-log numbers don't hold at this head. 0.408 and 0.873 reproduce only
against the old abs(mean bias). With the per-flight abs_bias, log and linear both score 1.000,
and so do mad only and mad plus std, the options the ADR rejects.

Cause is the generator: TIER_SCALE 1.0 / 2.5 / 6.0 with METRIC_VARIATION 0.7 to 1.4 gives tier
bands that never overlap, so once the fix removed the noise the tiers are trivially separable. That
also makes test_synthetic_pilots_are_grouped_by_their_hidden_tier's >= 0.8 pass for every option
here. Overlap the tiers, then re-measure the ADR.

Two smaller ones:

• --k isn't validated. --k 1 gives one group for everyone, --k 15 on 15 pilots gives one pilot
  each, while the sweep path enforces 3 per group. Check k against candidate_ks.
• truth_agreement merges how="left", so a duplicate pilot_id in --truth turns 15 pilots into
  16 rows and shifts the score. truth["pilot_id"].is_unique would catch it.

Otherwise it runs clean: 61 tests pass, the README commands work from a fresh checkout, and nothing
generated lands in git status.

[jmacd867]

Follow-ups that came out of this review. They are tracked separately and none is needed to merge this PR:

• Synthetic pilots whose bias and wobble vary independently #230 Synthetic pilots whose bias and wobble vary independently
• Make the comparison behind ADR 0002's Evidence table reproducible #231 Make the comparison behind ADR 0002's Evidence table reproducible
• cluster_pilots.py shows a traceback for a bad --k and leaves outputs when --truth is invalid #232 cluster_pilots.py shows a traceback for a bad --k and leaves outputs when --truth is invalid
• Re-check ADR 0002's feature choices on real pilot data #233 Re-check ADR 0002's feature choices on real pilot data

The figures quoted in #216 and #224 changed with this PR, so I commented on both.

[jmacd867]

Summary

Addresses the three review points: overlap the tiers and re-measure ADR 0002, validate --k, and reject a repeated pilot_id in --truth.

 inference/
 ├── generate_synthetic_pilots.py
-│     TIER_SCALE 1 / 2.5 / 6, METRIC_VARIATION 0.7–1.4   # tier bands never overlap
+│     TIER_SCALE 1 / 2 / 4,   METRIC_VARIATION 0.5–2.0   # adjacent tiers overlap
 ├── clustering.py
 │     fit(pilots, k)
+│       if k not in candidate_ks(n): raise ValueError     # the range the sweep offers
 │     truth_agreement(assignments, truth)
+│       if a pilot_id repeats in truth: raise ValueError
 └── tests/
docs/adr/0002-cluster-on-std-and-absolute-bias.md   Evidence table re-measured
GLOSSARY.md                                         + Hidden tier

Evidence

• Before: with the per-flight abs_bias, every option except the per-pilot variant scored 1.000, so ADR 0002's 0.41 / 0.87 described the old definition.
  After: the tiers overlap and the Evidence table is re-measured (mean adjusted Rand index, k = 3, seeds 0–11, log / linear):

  Features	Before	After
  std + per-flight abs_bias (chosen)	1.00 / 1.00	0.78 / 0.51
  mad only	1.00 / 1.00	0.76 / 0.50
  mad + std	1.00 / 1.00	0.74 / 0.48
  std + abs of pilot's mean signed bias	0.87 / 0.41	0.55 / 0.36

  The recovery test's bar moves from 0.8 to 0.7, which now rejects linear and the per-pilot variant. The runs cannot separate the chosen features from the mad options, and the ADR now says so.

• Before: fit(15 pilots, k=1) returned one group for everyone, and a repeated pilot in --truth merged into an extra row.
  After:

  fit(15 pilots, k=1)   ValueError: k must be 2 to 5 for 15 pilots, got 1
  truth with P01 twice  ValueError: Pilot(s) repeated in truth: P01
  

• Tests: test_adjacent_tiers_overlap_on_some_std_feature fails on the old constants ("seed 0: no adjacent tiers overlap on any std feature") and passes now. The full inference suite is 66 passed, up from 61.

Merge Danger

Door: two-way

Docs, a synthetic-data generator and two input checks. No data migration and nothing destructive.

Blast Radius: Narrow

• fit now raises for k outside 2 to min(6, n // 3), and for fewer than 6 pilots. The only caller in the repo is cluster_pilots.py.
• The same generator seed now produces different synthetic pilots, so any synthetic CSVs generated earlier under inference/Data/ are stale (they are gitignored).
• Not needed to merge, tracked separately: Synthetic pilots whose bias and wobble vary independently #230 (the generator can't separate mad), Make the comparison behind ADR 0002's Evidence table reproducible #231 (reproduce the table), cluster_pilots.py shows a traceback for a bad --k and leaves outputs when --truth is invalid #232 (CLI error output), Re-check ADR 0002's feature choices on real pilot data #233 (check on real pilots).

[jmacd867]

New commit adds inference/data/client to .gitignore fixing #227

[Sanjana-Siddani]

lgtm!