Skip to content

Synthetic pilots strong on one metric and weak on another #224

Open

Description

@jmacd867

Is your feature request related to a problem? Please describe.
#216 asked for synthetic pilots with "variation per metric (for example good altitude, poor heading)". generate_synthetic_pilots.py scales each pilot's deviation per metric by 0.7 to 1.4 around its tier, and the tier scales are 1, 2.5 and 6. The per-metric ranges (0.7–1.4, 1.75–3.5, 4.2–8.4) never overlap, so every metric of a pilot stays inside its tier. No synthetic pilot is tier 1 on altitude and tier 3 on heading, so performance groups are never tested on the mixed pilots real cohorts will contain.

Describe the solution you'd like
Give some synthetic pilots one or two flight dynamics factors from a different tier, for example a --mixed-share fraction of pilots with one metric drawn from another tier's scale. Then report what that does to tier recovery.

  • Keep the hidden tier as the pilot's main tier, so --truth still works.
  • Measure the mean adjusted Rand index over seeds 0–5 with mixed pilots, against the current 0.906.
  • If recovery drops, decide whether the k = 3 test should keep 0.8 on the current cohort and add a separate, looser test for mixed cohorts.

Describe alternatives you've considered

  • Widening METRIC_VARIATION until the tiers overlap. Simpler, but it blurs every pilot rather than producing pilots with a clearly weak metric.
  • Leaving the generator as is. The tests then only cover pilots who are uniformly good or uniformly poor.

Additional context
Follow-up to #216. The clustering features and their log scale are in docs/adr/0002-cluster-on-std-and-absolute-bias.md; this issue only changes the synthetic data, not clustering.

Activity

  1. jmacd867 commented on Oct 12, 2026

    @jmacd867
    CollaboratorAuthor

    #225 widened METRIC_VARIATION to 0.5–2.0 and set TIER_SCALE to 1 / 2 / 4, so the per-metric ranges now overlap (0.5–2.0, 1.0–4.0, 2.0–8.0). The baseline is 0.78 over seeds 0–11, not 0.906, and the recovery test's bar is 0.7.

    Mixed pilots are still worth adding: widening blurs every pilot, while a mixed pilot has one clearly weak metric.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions