MLmodel / clustering

.gitignore

@@ -45,9 +45,20 @@ inference/evaluation_metrics.json
 inference/Models/*.pkl
 inference/dataset/test.csv
 
+# Generated clustering inputs and outputs (regenerated by the scripts under inference/)
+inference/Data/synthetic_pilot_telemetry.csv
+inference/Data/synthetic_pilot_tiers.csv
+inference/Data/flight_features.csv
+inference/Data/pilot_features.csv
+inference/Data/pilot_clusters.csv
+inference/Data/group_centres.csv
+inference/Data/client/*
 # Collected telemetry (one file per collection session, written by data_logger.py)
 inference/Data/session_flight_data.csv
 
 # Local build archives
 /xplane_plugin.zip
 /src.zip
+
+## Client Data
+inference/data/Client/*

GLOSSARY.md

@@ -56,6 +56,10 @@ _Avoid_: Performance score, error metrics
 The four metrics the client cares about: heading, vertical speed, altitude and airspeed.
 _Avoid_: Core metrics, KPIs
 
+**Performance group**:
+A set of pilots whose **Deviation features** resemble each other. Groups are not ranked.
+_Avoid_: Tier (a **Synthetic pilot**'s known level is its **Hidden tier**, a different thing), skill level, class
+
 ### Pilots and flights
 
 **Pilot**:
@@ -84,13 +88,23 @@ _Avoid_: Stage, segment
 Generated telemetry, balanced across flight event labels, used when real flights are not available.
 _Avoid_: Fake data, dummy data, mock data
 
+**Synthetic pilot**:
+A generated pilot with generated flights, built to deviate by a known amount, used to check that **Performance groups** find real differences before real pilots are available.
+_Avoid_: Test pilot, fake pilot, mock pilot
+
+**Hidden tier**:
+The known deviation level a **Synthetic pilot** is generated at, kept apart from its telemetry so **Performance groups** can be compared against it. Real pilots have none.
+_Avoid_: True group, skill level
+
 ## Relationships
 
 - A **Pilot** flies many **Flights**; a **Flight** produces many **Samples**.
 - A **Sample** has one value per telemetry field, and one **Deviation** per **Flight dynamics factor** once a **Target** exists.
 - A **Sample** receives exactly one **Flight event label**; a **Flight phase** is a kind of flight event label.
+- A **Pilot** falls into exactly one **Performance group**, decided by their **Deviation features** alone.
+- **Pilot metadata** is compared against **Performance groups** after they are formed; it does not form them.
 
 ## Flagged ambiguities
 
-- **Pilot rating** is unresolved: the client may mean a licensing rating (instrument, multi-engine) or a performance score. Do not use it as a precise term until clarified.
-- **Target** source is undecided: instructor-assigned, or taken from the autopilot. No targets are captured yet.
+- **Pilot rating** is the type of certification, not a performance score (client, see `docs/client_meeting_notes.md`). Whether it means the same as certification level, or an extra rating such as instrument, is still unclear.
+- **Target** source is most likely instructor-assigned: the client gives standardized, timestamped instructions during the flight. No targets are captured yet, and the heading reference (true vs. magnetic) is unresolved.

docs/adr/0001-metadata-checks-clusters-not-shapes-them.md

@@ -0,0 +1,12 @@
+# Pilot metadata checks performance groups; it does not form them
+
+The client asked for pilot rating and flight hours to be "factored in" to pilot clustering (#216). We form **Performance groups** from **Deviation features** only. **Pilot metadata** (certification level, flight hours, pilot rating) is compared against the groups afterwards, for example by checking whether high-hours pilots land in the low-deviation group. Identifiers (`pilot_id`, `flight_id`) are never features either way.
+
+## Considered Options
+
+- **Metadata as clustering features.** Rejected. Groups would mix how a pilot flies with who the pilot is: two pilots who fly identically but have 50 and 5,000 hours could be split apart. It would also make clustering wait on #214, where the type of pilot rating is still unresolved.
+- **Metadata afterwards (chosen).** Groups describe flying behaviour alone. Comparing them with metadata is an independent check that they mean something, and it still answers the client's question of how rating and hours relate to performance.
+
+## Consequences
+
+When presenting results, explain to the client that metadata informs the interpretation of performance groups rather than their formation. If the client insists on metadata as an input, revisit this ADR, since it changes what every group means.

docs/adr/0002-cluster-on-std-and-absolute-bias.md

@@ -0,0 +1,33 @@
+# Cluster on the log of std and |bias|, not mad or signed bias
+
+`inference/features.py` emits `mad`, `std`, signed `bias` and `abs_bias` per flight dynamics factor. Pilot clustering (#216) uses only `std` (how much the pilot wobbles) and `|bias|` (how far the pilot sits off target), 8 features in total. `|bias|` is the mean of each flight's own `|bias|`, not the absolute value of the mean signed bias, which would cancel a pilot's flights on opposite sides of the target. These are the two independent parts of deviation: mean squared deviation is exactly `std² + bias²`. Each feature is clustered on a log scale, `log(x + floor)`, and then standardized over the cohort. The floors are 1 ft, 0.1°, 2 ft/min and 0.1 kt.
+
+## Considered Options
+
+- **All 12 (`mad`, `std`, signed `bias`).** Rejected. Signed bias puts a pilot 50 ft high and one 50 ft low at opposite ends of an axis, although they performed equally well. `mad` mostly restates `std` and `|bias|`, so with KMeans's Euclidean distance it counts each metric twice.
+- **`mad` + `std`.** Rejected because of that overlap. The synthetic pilots cannot confirm it (see Evidence).
+- **`mad` only (4).** The fewest dimensions, but it cannot separate a steady pilot who sits off target from an on-target pilot who oscillates. This is a structural argument, not a measured one: synthetic pilots draw bias and wobble from one scale, so they move together and never present that case (see Evidence).
+- **Absolute value of each pilot's mean signed bias.** Rejected. A pilot 60 ft high on one flight and 60 ft low on the next averages to zero bias and looks on target. Tested on synthetic pilots (see Evidence).
+- **Linear scale.** Rejected after testing on synthetic pilots (see Evidence). Deviation sizes are positive and pilots differ by multiples, so on a linear scale the worst pilots stretch every axis and the rest bunch together.
+- **Log with no floor.** Rejected. It would make 0.01 ft vs 0.1 ft of bias as far apart as 10 ft vs 100 ft.
+
+## Evidence
+
+Mean adjusted Rand index against the hidden tier, k = 3, synthetic seeds 0 to 11, on synthetic pilots whose adjacent tiers overlap. To reproduce, run `generate_pilots` for each seed, `compute_features`, then the clustering steps in `clustering.py` with each feature set.
+
+| Features | Log | Linear |
+| --- | --- | --- |
+| `std` + per-flight `abs_bias` (chosen) | 0.78 | 0.51 |
+| `mad` only | 0.76 | 0.50 |
+| `mad` + `std` | 0.74 | 0.48 |
+| `std` + abs of each pilot's mean signed bias | 0.55 | 0.36 |
+
+Per-seed scores for the chosen option run from 0.37 to 1.00, so differences of a few hundredths are noise. The runs support two choices: log over linear, and per-flight `abs_bias` over the per-pilot variant. Both orderings also held at each of two heavier overlap settings tried, though the gap between log and linear narrowed to 0.03 at the first. The runs do not separate the chosen features from the `mad` options, so those rejections rest on the structural arguments above.
+
+Two limits. The generator scales deviations multiplicatively, which is the assumption the log scale encodes, so this shows that log handles that structure and not that real pilots have it. And the overlap was set after trying a few settings, not derived from real flights.
+
+An earlier generator kept the tiers' scale bands apart. On it the chosen features and both `mad` options scored 1.00 on either scale, so it could not compare them, while the per-pilot variant scored 0.87 on log and 0.41 on linear. Figures from before the tiers overlapped do not apply here.
+
+## Consequences
+
+With 10 to 20 pilots, even 8 features is far below the roughly 70 × features sample size recommended for segmentation (Dolnicar et al., 2014). The k sweep's subsample stability is how we find out whether the performance groups hold up. If they don't, drop to 4 features before adding any. `rows_used` and `flights` measure how much data a pilot has, not performance, and are excluded. Group centres are reported as raw-unit means of their members, since KMeans centres live in log-scaled space.

docs/client_meeting_notes.md

@@ -0,0 +1,75 @@
+# Client meeting notes: Q&A and simulator visit
+
+Source: transcript of a meeting with the client (Dr. Tamilselvan) in the simulator lab, with a walkthrough of the Prepar3D machine. The meeting date is not recorded in the transcript. The transcript is auto-generated and partly garbled, so anything unclear is marked **unconfirmed**. The earlier kickoff notes are referenced from [telemetry_schema.md](telemetry_schema.md).
+
+Vocabulary follows [GLOSSARY.md](../GLOSSARY.md).
+
+## Where the data comes from today
+
+- The client's existing data was collected in **Prepar3D**, not X-Plane. Prepar3D is closed software, so it cannot be reconfigured; the client's own plugin extracts data from it.
+- The Prepar3D export is TSV, and the export rate is one sample per second. A sample rate of roughly 90 to 93 Hz was also mentioned for the simulator. **Unconfirmed**, the two statements conflict.
+- The client has 44 pilots. We received a de-identified sample.
+- Two pilots in the full set have missing data, and the client will find out why. **Which two is unconfirmed**: the transcript says "six and seven" and "04 and 07", but P004 and P007 are in the sample we received (see #228), so those numbers are probably misheard.
+- Every pilot flew one scenario, between one and six trials each. More scenarios and routes are planned.
+- The client already segmented each flight into 7 phases (takeoff through landing) and will share the segmentation.
+- X-Plane is not installed on the lab machine yet. The client has no date and will give an update the following week. Development proceeds on Prepar3D data in the meantime.
+- The simulator machine runs Windows only.
+
+## Scenario and what counts as on target
+
+- The scenario's headings: climbs on heading **190 and 40**, to **3100 ft**; approach legs on heading **220 and 0**.
+- Judge every leg on all four **Flight dynamics factors**: altitude, airspeed, heading and vertical speed. Approaches are judged on all four too, not on glide path or touchdown point alone.
+- The client describes **Performance** as the gap between asked-for and achieved values, for example a pilot who climbs to 3000 ft when asked for 3100 ft.
+- Route headings are **true headings**. The plugin currently sends **magnetic** heading (`heading_AHARS_deg_mag_pilot`). The mismatch matters once **Targets** exist, since the two differ by the magnetic variation. **Needs resolving before computing heading Deviation.**
+- Units: feet, knots and feet per minute.
+
+## Flights
+
+- Time on the ground does not matter. Measure each **Flight** from takeoff to the final landing and leave out the hour or more spent on the ground.
+- Stopping and restarting a recording partway through a flight is still **one Flight**. Join the parts by timestamp.
+- A full scenario run is about 40 minutes.
+
+## Targets
+
+- Pilots receive **standardized instructions** during the flight (for example "change heading to 20"). The instructions are timestamped in iMotions, and the eye tracker records audio and video.
+- This partly answers the **Target** source question: targets would be instructor-assigned, not taken from the autopilot. We have none of the timestamps yet, and the client may not be able to share iMotions data at first.
+- Without instruction timestamps the pipeline cannot tell a late turn from an early one. A later goal is a model that detects the instruction and segments automatically. Until then, use the client's manual segmentation.
+
+## Pilot metadata
+
+- **Pilot rating** means the type of certification, not a score. The client will send certification level, total flight hours and rating for the pilots in the sample.
+- For performance groups, the client prioritizes **experience (total flight hours) over certification level**.
+- The client agreed to metadata being used **at the end**, to interpret groups and not to form them. This confirms [ADR 0001](adr/0001-metadata-checks-clusters-not-shapes-them.md).
+- The client did clustering by hand last year and wants it automated.
+
+## Clustering and where it runs
+
+- The goal is real-time clustering. It is not expected to be computationally heavy.
+- It should run on the **iMotions computer**, which already shows the pilots' views, and not on the simulator machine.
+- Unresolved: where the result is shown (we assume not to the pilot, since it could add stress), and how often to cluster.
+- The sensors run at different rates: simulator about 90 Hz, EEG 256 Hz, EDA 128 Hz and eye tracking 51 Hz (as transcribed, **unconfirmed**). The client is not sure there is a common rate to cluster at. iMotions synchronizes the timestamps.
+- Later work combines flight data with the three physiological streams (eye tracking, EDA, EEG). These are kept in separate files, separate from demographics. Flight data alone is the starting point.
+
+## Data handling
+
+- The data is covered by an IRB protocol. Only people named on the protocol may access the full data. Copies may not be made or published.
+- The sample the client shared is de-identified, and use in this class project is fine. Publishing results in a journal or conference is allowed.
+- The repository is public, so **never commit raw client data**.
+
+## Network port
+
+- The X-Plane plugin sent to a placeholder port in the code. A new port was set, but it is **still unverified** that it matches the port iMotions listens on. The client, who handles the iMotions machine, will check. See [README.md](../README.md) for the open question.
+- Whether developers get administrator rights on the simulator machine was not settled.
+
+## Open items
+
+| Item | Owner |
+| --- | --- |
+| Confirm which port iMotions listens on | Client |
+| Share the 7-phase segmentation and the scenario briefing | Client |
+| Send certification level, hours and rating per pilot | Client |
+| Find out which pilots have missing data, and why | Client |
+| X-Plane installation date | Client |
+| Decide magnetic vs. true heading for heading **Targets** | Team |
+| Decide where clustering results are shown, and how often to cluster | Team and client |
+| Whether the team continues next semester | Unknown |

docs/telemetry_schema.md

@@ -132,11 +132,17 @@ alongside the telemetry data) and joined in at feature-engineering time.
 | `pilot_id` | string | — | join key to telemetry table |
 | `certification_level` | string (categorical) | — | e.g. Student, Private, Commercial, ATP — exact category set TBD with client |
 | `flight_hours` | float | hours | total logged flight hours |
-| `pilot_rating` | string or numeric | — | client mentioned "pilot rating" as distinct from certification; needs clarification on whether this is a licensing rating (e.g. instrument rating) or a performance score |
+| `pilot_rating` | string or numeric | — | client mentioned "pilot rating" as distinct from certification; the client says it is the type of certification, not a performance score (see below) |
 
-**Open question for the client:** clarify whether `pilot_rating` is a
-licensing qualification (instrument, multi-engine, etc.) or an evaluative
-score, since that changes its type and how it factors into clustering.
+**Answered by the client:** `pilot_rating` is the type of certification, not an
+evaluative score (see [client_meeting_notes.md](client_meeting_notes.md)). It
+is still unclear whether it differs from `certification_level`. The client
+prioritizes `flight_hours` over certification, and agreed it is used after
+clustering, not as a feature.
+
+**Heading reference:** the client says the scenario's route headings are
+**true** headings, but `heading` above is magnetic. Resolve this before
+targets are populated.
 
 ## 5. Deviation formulas
 
@@ -175,9 +181,12 @@ raw telemetry column keeps its name, `velocity`.
 
 `inference/features.py` summarises deviation per flight and per pilot in raw
 units: mean absolute deviation (`_mad`), standard deviation of the signed
-deviation (`_std`) and mean signed deviation (`_bias`) for each metric, plus
-the number of samples used (`_rows_used`, per flight). A pilot's features are
-the mean of their flights' features, with a `flights` count. Units differ in
+deviation (`_std`), mean signed deviation (`_bias`) and the absolute value of
+each flight's `_bias` (`_abs_bias`) for each metric, plus the number of samples
+used (`_rows_used`, per flight). A pilot's features are the mean of their
+flights' features, with a `flights` count. `_abs_bias` is taken per flight
+before the average, so a pilot 50 ft high on one flight and 50 ft low on
+another has `_bias` 0 but `_abs_bias` 50. Units differ in
 scale (ft vs. deg vs. kts), so scaling across pilots is done by the clustering
 step, not here; do not z-score per pilot, which would erase the differences
 between pilots.