You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Synthetic pilots strong on one metric and weak on another #224
Is your feature request related to a problem? Please describe. #216 asked for synthetic pilots with "variation per metric (for example good altitude, poor heading)". generate_synthetic_pilots.py scales each pilot's deviation per metric by 0.7 to 1.4 around its tier, and the tier scales are 1, 2.5 and 6. The per-metric ranges (0.7–1.4, 1.75–3.5, 4.2–8.4) never overlap, so every metric of a pilot stays inside its tier. No synthetic pilot is tier 1 on altitude and tier 3 on heading, so performance groups are never tested on the mixed pilots real cohorts will contain.
Describe the solution you'd like
Give some synthetic pilots one or two flight dynamics factors from a different tier, for example a --mixed-share fraction of pilots with one metric drawn from another tier's scale. Then report what that does to tier recovery.
Keep the hidden tier as the pilot's main tier, so --truth still works.
Measure the mean adjusted Rand index over seeds 0–5 with mixed pilots, against the current 0.906.
If recovery drops, decide whether the k = 3 test should keep 0.8 on the current cohort and add a separate, looser test for mixed cohorts.
Describe alternatives you've considered
Widening METRIC_VARIATION until the tiers overlap. Simpler, but it blurs every pilot rather than producing pilots with a clearly weak metric.
Leaving the generator as is. The tests then only cover pilots who are uniformly good or uniformly poor.
Additional context
Follow-up to #216. The clustering features and their log scale are in docs/adr/0002-cluster-on-std-and-absolute-bias.md; this issue only changes the synthetic data, not clustering.
#225 widened METRIC_VARIATION to 0.5–2.0 and set TIER_SCALE to 1 / 2 / 4, so the per-metric ranges now overlap (0.5–2.0, 1.0–4.0, 2.0–8.0). The baseline is 0.78 over seeds 0–11, not 0.906, and the recovery test's bar is 0.7.
Mixed pilots are still worth adding: widening blurs every pilot, while a mixed pilot has one clearly weak metric.
Activity
jmacd867 commentedon Oct 12, 2026
#225 widened
METRIC_VARIATIONto 0.5–2.0 and setTIER_SCALEto 1 / 2 / 4, so the per-metric ranges now overlap (0.5–2.0, 1.0–4.0, 2.0–8.0). The baseline is 0.78 over seeds 0–11, not 0.906, and the recovery test's bar is 0.7.Mixed pilots are still worth adding: widening blurs every pilot, while a mixed pilot has one clearly weak metric.