Skip to content

Build a baseline pilot clustering model #216

Closed

Description

@Phlabry

Is your feature request related to a problem? Please describe.
The client wants 10 to 20 pilots grouped by performance across heading, vertical speed, altitude and airspeed combined, with pilot rating and flight hours factored in. There is no clustering code in the repo and no unsupervised model of any kind.

Describe the solution you'd like
A baseline that forms performance groups (see GLOSSARY.md) from the per-pilot deviation features of #215. It runs on synthetic pilots now so it is ready when the client's telemetry arrives.

Pipeline: generate_synthetic_pilots.py → compute_features.py → cluster_pilots.py.

  • Inputs. std and |bias| for each flight dynamics factor, 8 features in total, read from pilot_features.csv. Each feature is clustered as log(x + floor) (floors 1 ft, 0.1°, 2 ft/min, 0.1 kt) and then standardized over the cohort, because linear features recovered the synthetic tiers at only about 0.40 agreement. See ADR 0002. Not mad, signed bias, rows_used or flights. See docs/adr/0002-cluster-on-std-and-absolute-bias.md.
  • Pilot metadata is not an input. It is compared against the groups afterwards, once Collect pilot metadata and join it to telemetry for clustering #214 lands. pilot_id and flight_id are identifiers only. See docs/adr/0001-metadata-checks-clusters-not-shapes-them.md.
  • Scaling. One scaler fit on the whole cohort, never per pilot.
  • Missing values. If any pilot has a NaN feature, the script stops with an error that names the pilots and metrics. Nothing is dropped or imputed.
  • Choosing k. KMeans, never picking k automatically:
    • With no --k: print a sweep over k = 2..min(6, n // 3) with silhouette and subsample stability. Subsample stability is the mean adjusted Rand index over about 50 refits on about 80% of pilots, at a fixed seed. Nothing is written.
    • With --k N: fit once and write pilot_clusters.csv (pilot_id, group) plus each group's centre in raw units (ft, deg, ft/min, kts).
  • Synthetic pilots. A new generate_synthetic_pilots.py. generate_balanced_data.py stays untouched because the labeler and the Random Forest depend on it.
    • 15 pilots in 3 hidden skill tiers (5 per tier), each with 2 to 4 flights of 2 minutes at 20 Hz, and a different constant target per flight.
    • Deviation per metric is a steady bias with a random sign per flight, plus wobble that is correlated over time (AR(1)). The tier sets the size of both, with variation per metric (for example good altitude, poor heading (not yet; tracked in the mixed-skill follow-up)").
    • Values use post-Plugin applies metric conversion factors to pilot altitude and airspeed that are already in feet and knots #196 units and a fixed seed.
    • The true tier is written to a separate file (pilot_id, tier) and never appears in the telemetry.
  • Layout. A pure clustering.py (DataFrame in, results out), a thin CLI cluster_pilots.py, and tests/test_clustering.py, following features.py / compute_features.py.

Acceptance criteria

  • generate_synthetic_pilots.py writes telemetry that compute_features.py accepts, plus the separate tier file
  • cluster_pilots.py with no --k prints silhouette and subsample stability per k and writes nothing
  • cluster_pilots.py --k N writes assignments and centres in raw units
  • Given the tier file, the script reports the adjusted Rand index against the true tier
  • A test checks that the 3 tiers are recovered (mean adjusted Rand index ≥ 0.8 at k = 3 over seeds 0–5; a single 15-pilot cohort ranges from 0.64 to 1.0)
  • A test checks that NaN features stop the script with an error naming the pilots

Describe alternatives you've considered

  • Using metadata as clustering features, and clustering on all 12 deviation features. Both were rejected; see the two ADRs.
  • Measuring stability across seeds only. Rejected because KMeans with n_init=10 gives nearly the same partition for any seed, so the measure would always look stable.
  • Saving the fitted scaler and centres. Deferred to the live-clustering issue, which is the first thing that needs them.
  • Clustering on linear features. Rejected after testing; see ADR 0002.

Additional context
With 10 to 20 pilots the sample is far below the roughly 70 × features rule of thumb for segmentation (Dolnicar et al., 2014), so subsample stability is the main signal of whether the groups are real. If they are unstable, drop to 4 features. Live, in-flight assignment is tracked in #223 .

Activity

  1. jmacd867 commented on Oct 3, 2026

    @jmacd867
    Collaborator
    • Add a new generate_synthetic_pilots.py. Leave generate_balanced_data.py untouched, because the labeler and Random Forest training depend on it.
    • Synthetic pilots have 3 latent skill tiers. The tier sets per-metric noise around a constant per-flight target, with some metric-specific variation (e.g. good altitude, poor heading). Each pilot has 2-4 flights with different targets.
    • Scale the per-pilot features here (the scaler is fit on the whole cohort).
    • Sweep k from 2 to min(6, n // 3). Report silhouette per k and stability across several seeds, with a fixed seed. Do not auto-select k. A human chooses it.
  2. jmacd867 commented on Oct 9, 2026

    @jmacd867
    Collaborator
    • Add a new generate_synthetic_pilots.py. Leave generate_balanced_data.py untouched, because the labeler and Random Forest training depend on it.
    • Synthetic pilots have 3 latent skill tiers. The tier sets per-metric noise around a constant per-flight target, with some metric-specific variation (e.g. good altitude, poor heading). Each pilot has 2-4 flights with different targets.
    • Scale the per-pilot features here (the scaler is fit on the whole cohort).
    • Sweep k from 2 to min(6, n // 3). Report silhouette per k and stability across several seeds, with a fixed seed. Do not auto-select k. A human chooses it.

    Superseded: the body above now defines stability as subsample stability (mean adjusted Rand index over about 50 refits on 80% of pilots). Seed-only stability was rejected because KMeans with n_init=10 returns nearly the same partition for any seed.

  3. jmacd867 commented on Oct 12, 2026

    @jmacd867
    Collaborator

    Figures in the acceptance criteria changed in #225:

    • Linear scale is now 0.51 on the current generator (was "about 0.40").
    • The recovery test is mean adjusted Rand index ≥ 0.7 over seeds 0–11 (was ≥ 0.8 over seeds 0–5).
    • A single 15-pilot cohort ranges from 0.37 to 1.00 (was 0.64 to 1.0).

    See the Evidence section of docs/adr/0002-cluster-on-std-and-absolute-bias.md.

  4. jmacd867 commented on Oct 15, 2026

    @jmacd867
    Collaborator

    How the client's data lines up with this design:

    • 5 pilots, so the current code refuses to run. candidate_ks raises below 6 pilots. Flights per pilot are uneven: 6, 6, 1, 4 and 2.
    • Five groups would need about 15 pilots. The client may expect five performance groups (to be confirmed at the meeting), and the k sweep only goes up to n // 3.
    • Pilot facts in the grouping. In the first meeting the client asked for pilot rating and flight hours to factor into the clustering. ADR 0001 decided the opposite. This is being put to her directly at the upcoming meeting; if she insists, the ADR needs revisiting.
    • Two simulators. The data is from Prepar3D and the client plans to move to X-Plane later. Pilots flown in different simulators (different flight models, about 1 Hz vs 20 Hz) shouldn't be clustered together until we've checked that the same flying gives the same features in both. Whether the 10–20 pilots will be split across simulators is being asked at the meeting.
  5. added this to the v1.0 milestone on Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

High PriorityNeeds to be fixed as soon as possiblearea:mlML / inference workenhancementNew feature or requestpython

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions