From 2d313a6bf9e8d6702e0afe415972c6f87673b4de Mon Sep 17 00:00:00 2001 From: 1bananas1 Date: Fri, 9 Oct 2026 10:18:58 -0500 Subject: [PATCH 1/4] MLmodel / clustering --- GLOSSARY.md | 10 ++ ...etadata-checks-clusters-not-shapes-them.md | 12 ++ .../0002-cluster-on-std-and-absolute-bias.md | 15 ++ inference/README.md | 44 ++++- inference/cluster_pilots.py | 74 ++++++++ inference/clustering.py | 148 ++++++++++++++++ inference/generate_synthetic_pilots.py | 125 ++++++++++++++ inference/tests/test_cluster_pilots.py | 55 ++++++ inference/tests/test_clustering.py | 160 ++++++++++++++++++ .../tests/test_generate_synthetic_pilots.py | 38 +++++ 10 files changed, 677 insertions(+), 4 deletions(-) create mode 100644 docs/adr/0001-metadata-checks-clusters-not-shapes-them.md create mode 100644 docs/adr/0002-cluster-on-std-and-absolute-bias.md create mode 100644 inference/cluster_pilots.py create mode 100644 inference/clustering.py create mode 100644 inference/generate_synthetic_pilots.py create mode 100644 inference/tests/test_cluster_pilots.py create mode 100644 inference/tests/test_clustering.py create mode 100644 inference/tests/test_generate_synthetic_pilots.py diff --git a/GLOSSARY.md b/GLOSSARY.md index cebfbe9..182e259 100644 --- a/GLOSSARY.md +++ b/GLOSSARY.md @@ -56,6 +56,10 @@ _Avoid_: Performance score, error metrics The four metrics the client cares about: heading, vertical speed, altitude and airspeed. _Avoid_: Core metrics, KPIs +**Performance group**: +A set of pilots whose **Deviation features** resemble each other. Groups are not ranked. +_Avoid_: Tier, skill level, class + ### Pilots and flights **Pilot**: @@ -84,11 +88,17 @@ _Avoid_: Stage, segment Generated telemetry, balanced across flight event labels, used when real flights are not available. _Avoid_: Fake data, dummy data, mock data +**Synthetic pilot**: +A generated pilot with generated flights, built to deviate by a known amount, used to check that **Performance groups** find real differences before real pilots are available. +_Avoid_: Test pilot, fake pilot, mock pilot + ## Relationships - A **Pilot** flies many **Flights**; a **Flight** produces many **Samples**. - A **Sample** has one value per telemetry field, and one **Deviation** per **Flight dynamics factor** once a **Target** exists. - A **Sample** receives exactly one **Flight event label**; a **Flight phase** is a kind of flight event label. +- A **Pilot** falls into exactly one **Performance group**, decided by their **Deviation features** alone. +- **Pilot metadata** is compared against **Performance groups** after they are formed; it does not form them. ## Flagged ambiguities diff --git a/docs/adr/0001-metadata-checks-clusters-not-shapes-them.md b/docs/adr/0001-metadata-checks-clusters-not-shapes-them.md new file mode 100644 index 0000000..e524749 --- /dev/null +++ b/docs/adr/0001-metadata-checks-clusters-not-shapes-them.md @@ -0,0 +1,12 @@ +# Pilot metadata checks performance groups; it does not form them + +The client asked for pilot rating and flight hours to be "factored in" to pilot clustering (#216). We form **Performance groups** from **Deviation features** only. **Pilot metadata** (certification level, flight hours, pilot rating) is compared against the groups afterwards, for example by checking whether high-hours pilots land in the low-deviation group. Identifiers (`pilot_id`, `flight_id`) are never features either way. + +## Considered Options + +- **Metadata as clustering features.** Rejected. Groups would mix how a pilot flies with who the pilot is: two pilots who fly identically but have 50 and 5,000 hours could be split apart. It would also make clustering wait on #214, where the type of pilot rating is still unresolved. +- **Metadata afterwards (chosen).** Groups describe flying behaviour alone. Comparing them with metadata is an independent check that they mean something, and it still answers the client's question of how rating and hours relate to performance. + +## Consequences + +When presenting results, explain to the client that metadata informs the interpretation of performance groups rather than their formation. If the client insists on metadata as an input, revisit this ADR, since it changes what every group means. diff --git a/docs/adr/0002-cluster-on-std-and-absolute-bias.md b/docs/adr/0002-cluster-on-std-and-absolute-bias.md new file mode 100644 index 0000000..96f39d3 --- /dev/null +++ b/docs/adr/0002-cluster-on-std-and-absolute-bias.md @@ -0,0 +1,15 @@ +# Cluster on the log of std and |bias|, not mad or signed bias + +`inference/features.py` emits `mad`, `std` and signed `bias` per flight dynamics factor. Pilot clustering (#216) uses only `std` (how much the pilot wobbles) and `|bias|` (how far the pilot sits off target), 8 features in total. These are the two independent parts of deviation: mean squared deviation is exactly `std² + bias²`. Each feature is clustered on a log scale, `log(x + floor)`, and then standardized over the cohort. The floors are 1 ft, 0.1°, 2 ft/min and 0.1 kt. + +## Considered Options + +- **All 12 (`mad`, `std`, signed `bias`).** Rejected. Signed bias puts a pilot 50 ft high and one 50 ft low at opposite ends of an axis, although they performed equally well. `mad` mostly restates `std` and `|bias|`, so with KMeans's Euclidean distance it counts each metric twice. +- **`mad` + `std`.** Rejected because of that overlap. +- **`mad` only (4).** The fewest dimensions, but it cannot separate a steady pilot who sits off target from an on-target pilot who oscillates. +- **Linear scale.** Rejected after testing on synthetic pilots. Deviation sizes are positive and pilots differ by multiples, so on a linear scale the worst pilots stretch every axis and the rest bunch together. KMeans then splits the worst tier and merges the other two (mean adjusted Rand index against the true tier about 0.40, against 0.87 on the log scale, over synthetic seeds 0 to 11). +- **Log with no floor.** Rejected. It would make 0.01 ft vs 0.1 ft of bias as far apart as 10 ft vs 100 ft. + +## Consequences + +With 10 to 20 pilots, even 8 features is far below the roughly 70 × features sample size recommended for segmentation (Dolnicar et al., 2014). The k sweep's subsample stability is how we find out whether the performance groups hold up. If they don't, drop to 4 features before adding any. `rows_used` and `flights` measure how much data a pilot has, not performance, and are excluded. Group centres are reported as raw-unit means of their members, since KMeans centres live in log-scaled space. diff --git a/inference/README.md b/inference/README.md index 94dacb2..ac3935f 100644 --- a/inference/README.md +++ b/inference/README.md @@ -57,7 +57,9 @@ the project root** (`PilotDataSynchronization/`), with the virtual environment a | 4 | `inference/prepare_data.py` | (Optional) Cleans, median-fills, standardizes, and splits the labeled data into train/test CSVs — independent of the training step below | `inference/Data/labeled_flight_data.csv` | `inference/dataset/*.csv`, `label_mapping.json`, `scaler_params.json` | | 5 | `inference/train_model.py` | Trains a Random Forest classifier (80/20 stratified split) and reports accuracy/precision/recall | `inference/Data/labeled_flight_data.csv` | `inference/Models/bestModel.pkl`, `inference/Models/finalModel.pkl`, `inference/dataset/test.csv` | | 6 | `inference/test_model.py` | Loads a trained model, runs inference, and (if ground-truth labels are present) evaluates it | Trained model + held-out test data (see below) | `inference/predictions_output.csv`, `inference/evaluation_metrics.json` | -| 7 | `inference/compute_features.py` | (Optional, clustering) Computes deviation features per flight and per pilot from telemetry that carries targets | Telemetry CSV with `pilot_id`, `flight_id` and `target_*` columns (none exist yet)| `inference/Data/flight_features.csv`, `inference/Data/pilot_features.csv` | +| 7a | `inference/generate_synthetic_pilots.py` | (Optional, clustering, no hardware needed) Generates synthetic pilots in 3 hidden skill tiers, with targets | none | `inference/Data/synthetic_pilot_telemetry.csv`, `inference/Data/synthetic_pilot_tiers.csv` | +| 7 | `inference/compute_features.py` | (Optional, clustering) Computes deviation features per flight and per pilot from telemetry that carries targets | Telemetry CSV with `pilot_id`, `flight_id` and `target_*` columns (only step 7a produces these today)| `inference/Data/flight_features.csv`, `inference/Data/pilot_features.csv` | +| 8 | `inference/cluster_pilots.py` | (Optional, clustering) Sweeps k, or with `--k` forms performance groups | `inference/Data/pilot_features.csv` | Sweep: console only. `--k`: `inference/Data/pilot_clusters.csv`, `inference/Data/group_centres.csv` | ### Step 4 and step 5/6 are independent @@ -76,8 +78,7 @@ which `test_model.py` reads first (see "Model details" below) — this is a diff flight and per pilot (see `docs/telemetry_schema.md` section 5). The input needs `pilot_id`, `flight_id`, the four actual values and `target_altitude`, `target_heading`, `target_vertical_speed` and `target_airspeed`, in post-#196 units. The committed CSVs lack these -columns, and no script in this repo generates them yet, so this step has only been run on -hand-built test data. +columns; until real flights carry them, use `generate_synthetic_pilots.py` (step 7a). ``` python inference/compute_features.py --input @@ -93,6 +94,30 @@ A flight with no valid target for a metric gets NaN for it and a warning, and it metrics are kept. A pilot's feature averages over the flights that have it; a pilot with none gets NaN and a warning. Rows with a null `pilot_id` or `flight_id` raise an error. +### Performance groups (clustering) + +`inference/cluster_pilots.py` groups pilots with KMeans on `std` and `|bias|` for each metric, +log-scaled and then standardized over the cohort (`docs/adr/0002`). Pilot metadata is not an +input; compare it against the groups afterwards (`docs/adr/0001`). Any pilot with a NaN feature +stops the run with an error naming the pilot and metric. + +The script never picks k. Run it once without `--k` to print, for k = 2 to min(6, pilots // 3): + +- `silhouette`: how cohesive and separated the groups are, from -1 to 1 +- `stability`: mean adjusted Rand index between the full fit and refits on random 80% subsamples + of pilots; 1 means the same groups every time + +Then rerun with the chosen `--k`. It writes each pilot's group and each group's centre, as the +mean of its members in raw units (ft, deg, ft/min, kts). For synthetic pilots, `--truth` also +prints the adjusted Rand index against the hidden tiers. + +``` +python inference/generate_synthetic_pilots.py +python inference/compute_features.py --input inference/Data/synthetic_pilot_telemetry.csv +python inference/cluster_pilots.py +python inference/cluster_pilots.py --k 3 --truth inference/Data/synthetic_pilot_tiers.csv +``` + ### Tests ``` @@ -100,7 +125,8 @@ uv sync --project inference # installs pytest from the dev group; the plain-v uv run --project inference pytest inference/tests ``` -Covers the angle helper, the deviation features and the labeler's heading change. +Covers the angle helper, the deviation features, the labeler's heading change, the synthetic +pilot generator and performance groups (including recovering the synthetic tiers end to end). ## Commands (run from the project root) @@ -128,8 +154,15 @@ python inference/train_model.py # 6. Test the model and generate predictions python inference/test_model.py +# 7a. (Optional) Generate synthetic pilots with targets, no hardware needed +python inference/generate_synthetic_pilots.py + # 7. (Optional) Compute deviation features from telemetry that carries targets python inference/compute_features.py --input + +# 8. (Optional) Sweep k, then form performance groups with the chosen k +python inference/cluster_pilots.py +python inference/cluster_pilots.py --k ``` Each script can also be run from inside `inference/` (e.g. `cd inference && python @@ -229,6 +262,9 @@ held-out split exists and the first candidate is used. - `prepare_data.py` — Cleans, standardizes, and splits labeled data into train/test CSVs (optional, not used by `train_model.py`/`test_model.py`) - `train_model.py` — Trains and saves a Random Forest classifier - `test_model.py` — Loads a trained model, runs inference, evaluates, and saves predictions +- `generate_synthetic_pilots.py` — Generates synthetic pilots with targets and hidden skill tiers, for clustering without hardware +- `features.py` / `compute_features.py` — Deviation features per flight and per pilot (module / CLI) +- `clustering.py` / `cluster_pilots.py` — Performance groups from per-pilot deviation features (module / CLI) - `requirements.txt` / `pyproject.toml` / `uv.lock` — Python dependencies - `Data/raw_flight_data.csv` — Raw telemetry input (not tracked for new data; a sample file is present in this repo); written by step 1 or renamed from step 1b's output - `Data/synthetic_flight_data.csv` — Output of `generate_balanced_data.py` (step 1b); rename to `raw_flight_data.csv` to feed it into step 2 diff --git a/inference/cluster_pilots.py b/inference/cluster_pilots.py new file mode 100644 index 0000000..af18e7f --- /dev/null +++ b/inference/cluster_pilots.py @@ -0,0 +1,74 @@ +""" +Form performance groups from per-pilot deviation features. + +Without --k, prints silhouette and subsample stability for each candidate k +and writes nothing; a person picks k from that report. With --k, fits once and +writes each pilot's group and each group's centre in raw units. + +Usage (from the project root): + python inference/cluster_pilots.py + python inference/cluster_pilots.py --k 3 --truth inference/Data/synthetic_pilot_tiers.csv +""" + +import argparse +from pathlib import Path + +import pandas as pd + +from clustering import fit, sweep, truth_agreement + +DATA_DIR = Path(__file__).parent / "Data" + + +def main(argv=None) -> None: + parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + parser.add_argument( + "--input", + type=Path, + default=DATA_DIR / "pilot_features.csv", + help="Per-pilot features CSV from compute_features.py", + ) + parser.add_argument("--k", type=int, help="Number of groups; omit to print the sweep") + parser.add_argument("--seed", type=int, default=0, help="Random seed") + parser.add_argument( + "--subsamples", type=int, default=50, help="Refits per k for the stability score" + ) + parser.add_argument( + "--truth", type=Path, help="True tier per pilot, for synthetic pilots only" + ) + parser.add_argument( + "--assignments-output", + type=Path, + default=DATA_DIR / "pilot_clusters.csv", + help="Group per pilot CSV", + ) + parser.add_argument( + "--centres-output", + type=Path, + default=DATA_DIR / "group_centres.csv", + help="Group centres CSV, in raw units", + ) + args = parser.parse_args(argv) + if args.truth is not None and args.k is None: + parser.error("--truth needs --k: agreement is reported for one fitted k") + + pilots = pd.read_csv(args.input) + if args.k is None: + report = sweep(pilots, seed=args.seed, n_subsamples=args.subsamples) + print(report.to_string(index=False, float_format="%.3f")) + print("\nPick k from this report and rerun with --k.") + return + + assignments, centres = fit(pilots, args.k, seed=args.seed) + assignments.to_csv(args.assignments_output, index=False) + centres.to_csv(args.centres_output, index=False) + print(f"Wrote {len(assignments)} pilots in {args.k} groups to {args.assignments_output}") + print(f"Wrote group centres to {args.centres_output}") + print(centres.to_string(index=False, float_format="%.2f")) + if args.truth is not None: + score = truth_agreement(assignments, pd.read_csv(args.truth)) + print(f"Agreement with true tiers (adjusted Rand index): {score:.3f}") + + +if __name__ == "__main__": + main() diff --git a/inference/clustering.py b/inference/clustering.py new file mode 100644 index 0000000..c4a70a4 --- /dev/null +++ b/inference/clustering.py @@ -0,0 +1,148 @@ +""" +Performance groups from per-pilot deviation features. + +Clusters pilots on the log of std and |bias| for each flight dynamics factor +(docs/adr/0002). Pilot metadata is not an input (docs/adr/0001). Pure +DataFrame-in functions; cluster_pilots.py is the CLI wrapper. +""" + +import numpy as np +import pandas as pd +from sklearn.cluster import KMeans +from sklearn.metrics import adjusted_rand_score, silhouette_score +from sklearn.preprocessing import StandardScaler + +from features import METRICS + +FEATURE_PARTS = ("std", "abs_bias") +FEATURE_COLUMNS = [f"{metric}_{part}" for metric in METRICS for part in FEATURE_PARTS] + +# Added before taking the log, in raw units, so near-zero differences (0.01 vs +# 0.1 ft of bias) don't count as much as real ones (10 vs 100 ft). +LOG_FLOOR = {"altitude": 1.0, "heading": 0.1, "vertical_speed": 2.0, "airspeed": 0.1} + +# Share of pilots each stability refit keeps. +SUBSAMPLE_FRACTION = 0.8 + + +def clustering_features(pilots: pd.DataFrame) -> pd.DataFrame: + """ + The clustering inputs, one row per pilot indexed by pilot_id, in raw units. + + Raises if any pilot has a NaN feature, naming each pilot and metric. + Dropping or imputing would quietly reshape a cohort of 10 to 20 pilots. + """ + features = pd.DataFrame(index=pilots["pilot_id"]) + for metric in METRICS: + features[f"{metric}_std"] = pilots[f"{metric}_std"].to_numpy() + features[f"{metric}_abs_bias"] = pilots[f"{metric}_bias"].abs().to_numpy() + + gaps = [] + for pilot_id, row in features.iterrows(): + metrics = [m for m in METRICS if row[[f"{m}_std", f"{m}_abs_bias"]].isna().any()] + if metrics: + gaps.append(f"{pilot_id} ({', '.join(metrics)})") + if gaps: + raise ValueError(f"NaN deviation features for pilot(s): {'; '.join(gaps)}") + return features[FEATURE_COLUMNS] + + +def _scaled(features: pd.DataFrame) -> np.ndarray: + """ + Log, then standardize over the cohort being clustered (never per pilot). + + Deviation sizes are positive and pilots differ by multiples, so on a + linear scale the worst pilots stretch the axis and the rest bunch up. + """ + floors = pd.Series( + {f"{metric}_{part}": LOG_FLOOR[metric] for metric in METRICS for part in FEATURE_PARTS} + ) + return StandardScaler().fit_transform(np.log(features + floors[features.columns])) + + +def _kmeans_labels(scaled: np.ndarray, k: int, seed: int) -> np.ndarray: + return KMeans(n_clusters=k, n_init=10, random_state=seed).fit_predict(scaled) + + +def candidate_ks(n_pilots: int) -> range: + """k from 2 to min(6, n // 3), so every group can average 3 or more pilots.""" + if n_pilots < 6: + raise ValueError(f"Need at least 6 pilots to sweep k, got {n_pilots}") + return range(2, min(6, n_pilots // 3) + 1) + + +def sweep( + pilots: pd.DataFrame, + seed: int = 0, + n_subsamples: int = 50, +) -> pd.DataFrame: + """ + Report how well each candidate k fits, without choosing one. + + One row per k with: + silhouette cohesion vs separation of the full-cohort fit, in [-1, 1] + stability mean adjusted Rand index between the full-cohort fit and + refits on random subsamples (SUBSAMPLE_FRACTION of the pilots), + compared on the pilots each subsample kept; 1 is identical + + Stability is measured on subsamples rather than KMeans seeds, because with + n_init=10 a small cohort gets nearly the same partition from any seed. + """ + features = clustering_features(pilots) + scaled = _scaled(features) + n = len(features) + size = round(SUBSAMPLE_FRACTION * n) + rng = np.random.default_rng(seed) + rows = [] + for k in candidate_ks(n): + labels = _kmeans_labels(scaled, k, seed) + scores = [] + for _ in range(n_subsamples): + kept = np.sort(rng.choice(n, size=size, replace=False)) + # Rescale on the subsample, as a fresh run on those pilots would. + refit = _kmeans_labels(_scaled(features.iloc[kept]), k, seed) + scores.append(adjusted_rand_score(labels[kept], refit)) + rows.append( + { + "k": k, + "silhouette": silhouette_score(scaled, labels), + "stability": float(np.mean(scores)), + } + ) + return pd.DataFrame(rows) + + +def fit(pilots: pd.DataFrame, k: int, seed: int = 0): + """ + Cluster pilots into k performance groups. + + Returns (assignments, centres): assignments has pilot_id and group; + centres has one row per group with its pilot count and the mean of each + feature in raw units (ft, deg, ft/min, kts), so a group can be read + without undoing the scaling. + """ + features = clustering_features(pilots) + labels = _kmeans_labels(_scaled(features), k, seed) + assignments = pd.DataFrame({"pilot_id": features.index, "group": labels}) + + # The raw-unit mean of each group's members, not the KMeans centre, which + # lives in log-scaled space. + grouped = features.groupby(labels) + centres = grouped.mean() + centres.insert(0, "pilots", grouped.size()) + centres = centres.rename_axis("group").reset_index() + return assignments, centres + + +def truth_agreement(assignments: pd.DataFrame, truth: pd.DataFrame) -> float: + """ + Adjusted Rand index between performance groups and known tiers. + + For synthetic pilots only: truth has pilot_id and tier. 1 means the groups + match the tiers exactly, however they are numbered; about 0 is chance. + """ + merged = assignments.merge(truth, on="pilot_id", how="left") + missing = merged.loc[merged["tier"].isna(), "pilot_id"].tolist() + if missing: + raise ValueError(f"Pilot(s) missing from truth: {', '.join(map(str, missing))}") + return float(adjusted_rand_score(merged["tier"], merged["group"])) diff --git a/inference/generate_synthetic_pilots.py b/inference/generate_synthetic_pilots.py new file mode 100644 index 0000000..15c5501 --- /dev/null +++ b/inference/generate_synthetic_pilots.py @@ -0,0 +1,125 @@ +""" +Generate synthetic pilots for testing performance groups (#216). + +Each synthetic pilot belongs to one of 3 hidden skill tiers (1 holds targets +best, 3 worst) and flies 2 to 4 flights at 20 Hz, each with its own constant +targets. Per metric, a flight's deviation is a steady bias with a random sign +plus time-correlated AR(1) wobble. The tier sets the size of both, scaled per +pilot and metric so a pilot can be better at altitude than at heading. + +Writes telemetry in post-#196 units (ft, kts) that compute_features.py +accepts, and the true tiers to a separate file that clustering may compare +against. generate_balanced_data.py is separate and serves the labeler. + +Usage (from the project root): + python inference/generate_synthetic_pilots.py +""" + +import argparse +from pathlib import Path + +import numpy as np +import pandas as pd + +from features import ACTUAL_COLUMN, METRICS + +DATA_DIR = Path(__file__).parent / "Data" +HZ = 20 +TIERS = (1, 2, 3) +TIER_SCALE = {1: 1.0, 2: 2.5, 3: 6.0} +# Per-pilot, per-metric spread around the tier scale. +METRIC_VARIATION = (0.7, 1.4) +# Per-flight spread of the bias size. +BIAS_VARIATION = (0.7, 1.3) +# AR(1) coefficient per sample; at 20 Hz a wobble lasts a few seconds. +WOBBLE_PERSISTENCE = 0.98 + +# Tier 1 sizes in raw units: altitude ft, heading deg, vertical speed ft/min, airspeed kts. +BASE_BIAS = {"altitude": 20.0, "heading": 1.5, "vertical_speed": 40.0, "airspeed": 2.0} +BASE_WOBBLE = {"altitude": 15.0, "heading": 1.0, "vertical_speed": 60.0, "airspeed": 1.5} + + +def _targets(rng) -> dict: + return { + "altitude": float(rng.integers(6, 19) * 500), + "heading": float(rng.integers(0, 360)), + "vertical_speed": float(rng.choice([-500, 0, 500])), + "airspeed": float(rng.integers(18, 29) * 5), + } + + +def _ar1(rng, n: int, sd: float) -> np.ndarray: + """Stationary AR(1) series with standard deviation sd.""" + innovations = rng.normal(0.0, sd * np.sqrt(1 - WOBBLE_PERSISTENCE**2), n) + series = np.empty(n) + series[0] = rng.normal(0.0, sd) + for t in range(1, n): + series[t] = WOBBLE_PERSISTENCE * series[t - 1] + innovations[t] + return series + + +def _flight(rng, pilot_id: str, flight_id: str, deviation_scale: dict, n: int) -> pd.DataFrame: + targets = _targets(rng) + columns = {"pilot_id": pilot_id, "flight_id": flight_id} + for metric in METRICS: + bias = BASE_BIAS[metric] * deviation_scale[metric] * rng.uniform(*BIAS_VARIATION) + bias *= rng.choice([-1.0, 1.0]) + deviation = bias + _ar1(rng, n, BASE_WOBBLE[metric] * deviation_scale[metric]) + actual = targets[metric] + deviation + if metric == "heading": + actual = np.mod(actual, 360.0) + columns[ACTUAL_COLUMN[metric]] = actual + columns[f"target_{metric}"] = targets[metric] + return pd.DataFrame(columns) + + +def generate_pilots(n_pilots: int = 15, seed: int = 0, flight_seconds: int = 120): + """ + Returns (telemetry, truth). + + telemetry has pilot_id, flight_id, the four actual values and their + target_* columns. truth has pilot_id and tier, with tiers split as evenly + as n_pilots allows. + """ + rng = np.random.default_rng(seed) + n_samples = flight_seconds * HZ + flights, truth = [], [] + for i in range(n_pilots): + pilot_id = f"P{i + 1:02d}" + tier = TIERS[i * len(TIERS) // n_pilots] + truth.append({"pilot_id": pilot_id, "tier": tier}) + # Multiplies deviation size, so larger is worse. + deviation_scale = {m: TIER_SCALE[tier] * rng.uniform(*METRIC_VARIATION) for m in METRICS} + for f in range(rng.integers(2, 5)): + flights.append(_flight(rng, pilot_id, f"{pilot_id}-F{f + 1}", deviation_scale, n_samples)) + return pd.concat(flights, ignore_index=True), pd.DataFrame(truth) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + parser.add_argument("--pilots", type=int, default=15, help="Number of pilots") + parser.add_argument("--seed", type=int, default=0, help="Random seed") + parser.add_argument("--flight-seconds", type=int, default=120, help="Length of each flight") + parser.add_argument( + "--output", + type=Path, + default=DATA_DIR / "synthetic_pilot_telemetry.csv", + help="Telemetry CSV", + ) + parser.add_argument( + "--truth-output", + type=Path, + default=DATA_DIR / "synthetic_pilot_tiers.csv", + help="True tier per pilot", + ) + args = parser.parse_args() + + telemetry, truth = generate_pilots(args.pilots, args.seed, args.flight_seconds) + telemetry.to_csv(args.output, index=False) + truth.to_csv(args.truth_output, index=False) + print(f"Wrote {len(telemetry)} samples from {len(truth)} pilots to {args.output}") + print(f"Wrote true tiers to {args.truth_output}") + + +if __name__ == "__main__": + main() diff --git a/inference/tests/test_cluster_pilots.py b/inference/tests/test_cluster_pilots.py new file mode 100644 index 0000000..6f60a5f --- /dev/null +++ b/inference/tests/test_cluster_pilots.py @@ -0,0 +1,55 @@ +import pandas as pd +import pytest + +from cluster_pilots import main +from features import compute_features +from generate_synthetic_pilots import generate_pilots + + +@pytest.fixture +def cohort(tmp_path): + """A synthetic pilot_features.csv and its true tiers, in an otherwise empty directory.""" + telemetry, truth = generate_pilots(n_pilots=9, seed=0, flight_seconds=20) + _, pilots = compute_features(telemetry) + pilots.to_csv(tmp_path / "pilot_features.csv", index=False) + truth.to_csv(tmp_path / "tiers.csv", index=False) + return tmp_path + + +def outputs(directory): + return {"assignments": directory / "groups.csv", "centres": directory / "centres.csv"} + + +def run(directory, *extra): + paths = outputs(directory) + main( + [ + "--input", str(directory / "pilot_features.csv"), + "--assignments-output", str(paths["assignments"]), + "--centres-output", str(paths["centres"]), + "--subsamples", "5", + *extra, + ] + ) + return paths + + +def test_without_k_prints_the_sweep_and_writes_nothing(cohort, capsys): + paths = run(cohort) + + report = capsys.readouterr().out + assert "silhouette" in report and "stability" in report + assert not paths["assignments"].exists() + assert not paths["centres"].exists() + + +def test_with_k_writes_groups_and_centres_and_reports_truth(cohort, capsys): + paths = run(cohort, "--k", "3", "--truth", str(cohort / "tiers.csv")) + + groups = pd.read_csv(paths["assignments"]) + assert list(groups.columns) == ["pilot_id", "group"] + assert len(groups) == 9 + centres = pd.read_csv(paths["centres"]) + assert len(centres) == 3 + assert centres["pilots"].sum() == 9 + assert "Agreement with true tiers" in capsys.readouterr().out diff --git a/inference/tests/test_clustering.py b/inference/tests/test_clustering.py new file mode 100644 index 0000000..fdadd02 --- /dev/null +++ b/inference/tests/test_clustering.py @@ -0,0 +1,160 @@ +import numpy as np +import pandas as pd +import pytest + +from clustering import fit, sweep, truth_agreement +from features import METRICS, STATISTICS, compute_features +from generate_synthetic_pilots import generate_pilots + + +def make_pilots(rows): + """ + A per-pilot feature table shaped like features.pilot_features output. + + Each row is (pilot_id, {column: value}); unspecified features are 1.0 for + std and 0.0 for mad and bias. + """ + records = [] + for pilot_id, overrides in rows: + record = {"pilot_id": pilot_id, "flights": 2} + for metric in METRICS: + for stat in STATISTICS: + record[f"{metric}_{stat}"] = 1.0 if stat == "std" else 0.0 + record.update(overrides) + records.append(record) + return pd.DataFrame(records) + + +def group_of(assignments, pilot_id): + return assignments.set_index("pilot_id").loc[pilot_id, "group"] + + +def steady_and_wobbly_pilots(): + """Four pilots holding steady 60 ft off target (two high, two low), four wobbling around it.""" + return make_pilots( + [ + ("high1", {"altitude_bias": 60.0, "altitude_std": 5.0}), + ("low1", {"altitude_bias": -60.0, "altitude_std": 5.0}), + ("high2", {"altitude_bias": 60.0, "altitude_std": 5.0}), + ("low2", {"altitude_bias": -60.0, "altitude_std": 5.0}), + ("wobbly1", {"altitude_std": 60.0}), + ("wobbly2", {"altitude_std": 60.0}), + ("wobbly3", {"altitude_std": 60.0}), + ("wobbly4", {"altitude_std": 60.0}), + ] + ) + + +def test_pilots_off_target_in_opposite_directions_share_a_group(): + assignments, _ = fit(steady_and_wobbly_pilots(), k=2) + + steady = {group_of(assignments, p) for p in ("high1", "low1", "high2", "low2")} + wobbly = {group_of(assignments, p) for p in ("wobbly1", "wobbly2", "wobbly3", "wobbly4")} + assert len(steady) == 1 + assert len(wobbly) == 1 + assert steady != wobbly + + +def test_sub_foot_bias_differences_do_not_decide_groups(): + # Tight and loose pilots differ by 4 ft of wobble, and within each, by a + # meaningless 0.09 ft of bias. Pilots 50 ft off target give the bias axis + # a real spread; without a floor on the log, 0.01 vs 0.1 ft would + # look as far apart as 5 vs 50 ft and split the groups the wrong way. + pilots = make_pilots( + [ + (f"{size}{i}", {"altitude_std": std, "altitude_bias": bias}) + for size, std in (("tight", 8.0), ("loose", 12.0)) + for i, bias in enumerate((0.01, 0.1, 0.01, 0.1)) + ] + + [ + (f"off{i}", {"altitude_std": 60.0, "altitude_bias": bias}) + for i, bias in enumerate((50.0, -50.0, 50.0, -50.0)) + ] + ) + assignments, _ = fit(pilots, k=3) + + tight = {group_of(assignments, f"tight{i}") for i in range(4)} + loose = {group_of(assignments, f"loose{i}") for i in range(4)} + assert len(tight) == 1 + assert len(loose) == 1 + assert tight != loose + + +def test_centres_are_in_raw_units(): + assignments, centres = fit(steady_and_wobbly_pilots(), k=2) + centres = centres.set_index("group") + + steady = centres.loc[group_of(assignments, "high1")] + assert steady["altitude_abs_bias"] == pytest.approx(60.0) + assert steady["altitude_std"] == pytest.approx(5.0) + assert steady["pilots"] == 4 + + wobbly = centres.loc[group_of(assignments, "wobbly1")] + assert wobbly["altitude_abs_bias"] == pytest.approx(0.0) + assert wobbly["altitude_std"] == pytest.approx(60.0) + assert wobbly["pilots"] == 4 + + +@pytest.mark.parametrize( + "n_pilots, expected_ks", + [(20, [2, 3, 4, 5, 6]), (15, [2, 3, 4, 5]), (9, [2, 3]), (6, [2])], +) +def test_sweep_covers_k_from_2_to_min_of_6_and_a_third_of_pilots(n_pilots, expected_ks): + rng = np.random.default_rng(0) + pilots = make_pilots( + [(f"p{i}", {"altitude_std": rng.uniform(1, 100)}) for i in range(n_pilots)] + ) + assert sweep(pilots, n_subsamples=5)["k"].tolist() == expected_ks + + +def test_sweep_needs_at_least_6_pilots(): + pilots = make_pilots([(f"p{i}", {"altitude_std": float(i)}) for i in range(5)]) + with pytest.raises(ValueError, match="6"): + sweep(pilots) + + +def test_well_separated_groups_score_as_stable_and_cohesive(): + report = sweep(steady_and_wobbly_pilots()).set_index("k") + assert report.loc[2, "stability"] == pytest.approx(1.0) + assert report.loc[2, "silhouette"] > 0.9 + + +def test_truth_agreement_ignores_how_groups_are_numbered(): + assignments = pd.DataFrame({"pilot_id": ["a", "b", "c", "d"], "group": [0, 0, 1, 1]}) + truth = pd.DataFrame({"pilot_id": ["d", "c", "b", "a"], "tier": [1, 1, 3, 3]}) + assert truth_agreement(assignments, truth) == pytest.approx(1.0) + + +def test_truth_agreement_fails_naming_pilots_missing_from_truth(): + assignments = pd.DataFrame({"pilot_id": ["a", "b", "c"], "group": [0, 1, 1]}) + truth = pd.DataFrame({"pilot_id": ["a", "b"], "tier": [1, 2]}) + with pytest.raises(ValueError, match="c"): + truth_agreement(assignments, truth) + + +def test_synthetic_pilots_are_grouped_by_their_hidden_tier(): + # Averaged over cohorts, since 15 pilots make any single cohort noisy. + scores = [] + for seed in range(6): + telemetry, truth = generate_pilots(seed=seed) + _, pilots = compute_features(telemetry) + assignments, _ = fit(pilots, k=3) + scores.append(truth_agreement(assignments, truth)) + assert np.mean(scores) >= 0.8 + + +def test_nan_feature_fails_naming_pilot_and_metric(): + pilots = make_pilots( + [ + ("p1", {}), + ("p2", {"heading_std": float("nan"), "heading_bias": float("nan")}), + ("p3", {}), + ("p4", {"airspeed_bias": float("nan")}), + ] + ) + with pytest.raises(ValueError) as error: + fit(pilots, k=2) + message = str(error.value) + assert "p2" in message and "heading" in message + assert "p4" in message and "airspeed" in message + assert "p1" not in message and "p3" not in message diff --git a/inference/tests/test_generate_synthetic_pilots.py b/inference/tests/test_generate_synthetic_pilots.py new file mode 100644 index 0000000..30ebcff --- /dev/null +++ b/inference/tests/test_generate_synthetic_pilots.py @@ -0,0 +1,38 @@ +import pandas as pd + +from features import compute_features +from generate_synthetic_pilots import generate_pilots + + +def test_telemetry_feeds_compute_features(): + telemetry, _ = generate_pilots(n_pilots=6, seed=1, flight_seconds=5) + flights, pilots = compute_features(telemetry) + + assert len(pilots) == 6 + assert flights.groupby("pilot_id").size().between(2, 4).all() + # 5 seconds at 20 Hz. + assert (flights["altitude_rows_used"] == 100).all() + + +def test_values_are_in_feet_and_knots(): + telemetry, _ = generate_pilots(n_pilots=6, seed=1, flight_seconds=5) + # Pre-#196 data was in metres and m/s; feet and knots put a light aircraft here. + assert telemetry["target_altitude"].between(1000, 15000).all() + assert telemetry["target_airspeed"].between(60, 200).all() + + +def test_truth_is_separate_with_three_equal_tiers(): + telemetry, truth = generate_pilots(n_pilots=15, seed=1, flight_seconds=5) + + assert "tier" not in telemetry.columns + assert sorted(truth.columns) == ["pilot_id", "tier"] + assert truth["pilot_id"].is_unique + assert set(truth["pilot_id"]) == set(telemetry["pilot_id"]) + assert truth["tier"].value_counts().to_dict() == {1: 5, 2: 5, 3: 5} + + +def test_same_seed_gives_same_pilots(): + first = generate_pilots(n_pilots=6, seed=3, flight_seconds=5) + second = generate_pilots(n_pilots=6, seed=3, flight_seconds=5) + pd.testing.assert_frame_equal(first[0], second[0]) + pd.testing.assert_frame_equal(first[1], second[1]) From 5bd37d070a262df8e6aaeef0a8614e7fa763bb82 Mon Sep 17 00:00:00 2001 From: 1bananas1 Date: Sun, 11 Oct 2026 20:09:39 -0500 Subject: [PATCH 2/4] Requested changes --- .gitignore | 8 +++ .../0002-cluster-on-std-and-absolute-bias.md | 2 +- docs/telemetry_schema.md | 9 ++- inference/README.md | 8 ++- inference/clustering.py | 15 +++-- inference/features.py | 8 ++- inference/tests/test_clustering.py | 62 ++++++++++++++++--- inference/tests/test_features.py | 19 ++++++ 8 files changed, 107 insertions(+), 24 deletions(-) diff --git a/.gitignore b/.gitignore index e50966b..2c5bfbc 100644 --- a/.gitignore +++ b/.gitignore @@ -45,6 +45,14 @@ inference/evaluation_metrics.json inference/Models/*.pkl inference/dataset/test.csv +# Generated clustering inputs and outputs (regenerated by the scripts under inference/) +inference/Data/synthetic_pilot_telemetry.csv +inference/Data/synthetic_pilot_tiers.csv +inference/Data/flight_features.csv +inference/Data/pilot_features.csv +inference/Data/pilot_clusters.csv +inference/Data/group_centres.csv + # Local build archives /xplane_plugin.zip /src.zip diff --git a/docs/adr/0002-cluster-on-std-and-absolute-bias.md b/docs/adr/0002-cluster-on-std-and-absolute-bias.md index 96f39d3..a29c57f 100644 --- a/docs/adr/0002-cluster-on-std-and-absolute-bias.md +++ b/docs/adr/0002-cluster-on-std-and-absolute-bias.md @@ -1,6 +1,6 @@ # Cluster on the log of std and |bias|, not mad or signed bias -`inference/features.py` emits `mad`, `std` and signed `bias` per flight dynamics factor. Pilot clustering (#216) uses only `std` (how much the pilot wobbles) and `|bias|` (how far the pilot sits off target), 8 features in total. These are the two independent parts of deviation: mean squared deviation is exactly `std² + bias²`. Each feature is clustered on a log scale, `log(x + floor)`, and then standardized over the cohort. The floors are 1 ft, 0.1°, 2 ft/min and 0.1 kt. +`inference/features.py` emits `mad`, `std`, signed `bias` and `abs_bias` per flight dynamics factor. Pilot clustering (#216) uses only `std` (how much the pilot wobbles) and `|bias|` (how far the pilot sits off target), 8 features in total. `|bias|` is the mean of each flight's own `|bias|`, not the absolute value of the mean signed bias, which would cancel a pilot's flights on opposite sides of the target. These are the two independent parts of deviation: mean squared deviation is exactly `std² + bias²`. Each feature is clustered on a log scale, `log(x + floor)`, and then standardized over the cohort. The floors are 1 ft, 0.1°, 2 ft/min and 0.1 kt. ## Considered Options diff --git a/docs/telemetry_schema.md b/docs/telemetry_schema.md index 91e7fd3..2da39c9 100644 --- a/docs/telemetry_schema.md +++ b/docs/telemetry_schema.md @@ -170,9 +170,12 @@ raw telemetry column keeps its name, `velocity`. `inference/features.py` summarises deviation per flight and per pilot in raw units: mean absolute deviation (`_mad`), standard deviation of the signed -deviation (`_std`) and mean signed deviation (`_bias`) for each metric, plus -the number of samples used (`_rows_used`, per flight). A pilot's features are -the mean of their flights' features, with a `flights` count. Units differ in +deviation (`_std`), mean signed deviation (`_bias`) and the absolute value of +each flight's `_bias` (`_abs_bias`) for each metric, plus the number of samples +used (`_rows_used`, per flight). A pilot's features are the mean of their +flights' features, with a `flights` count. `_abs_bias` is taken per flight +before the average, so a pilot 50 ft high on one flight and 50 ft low on +another has `_bias` 0 but `_abs_bias` 50. Units differ in scale (ft vs. deg vs. kts), so scaling across pilots is done by the clustering step, not here; do not z-score per pilot, which would erase the differences between pilots. diff --git a/inference/README.md b/inference/README.md index ac3935f..40dea07 100644 --- a/inference/README.md +++ b/inference/README.md @@ -86,9 +86,11 @@ python inference/compute_features.py --input Outputs `inference/Data/flight_features.csv` and `inference/Data/pilot_features.csv`, in raw units. Per metric (altitude, heading, vertical speed, airspeed) it gives `_mad` (mean absolute -deviation), `_std` (standard deviation of the signed deviation) and `_bias` (mean signed -deviation, not meant for clustering by default). The flight file also has `_rows_used`, the -samples that had a valid deviation; the pilot file has a `flights` count instead. +deviation), `_std` (standard deviation of the signed deviation), `_bias` (mean signed +deviation) and `_abs_bias` (the absolute value of each flight's `_bias`, averaged per pilot, +so flights on opposite sides of the target do not cancel). Clustering uses `_std` and +`_abs_bias`. The flight file also has `_rows_used`, the samples that had a valid deviation; +the pilot file has a `flights` count instead. A flight with no valid target for a metric gets NaN for it and a warning, and its other metrics are kept. A pilot's feature averages over the flights that have it; a pilot with none diff --git a/inference/clustering.py b/inference/clustering.py index c4a70a4..f5241ce 100644 --- a/inference/clustering.py +++ b/inference/clustering.py @@ -1,8 +1,8 @@ """ Performance groups from per-pilot deviation features. -Clusters pilots on the log of std and |bias| for each flight dynamics factor -(docs/adr/0002). Pilot metadata is not an input (docs/adr/0001). Pure +Clusters pilots on the log of std and |bias| for each flight dynamics factor, +where |bias| is the mean of each flight's own |bias| (docs/adr/0002). Pilot metadata is not an input (docs/adr/0001). Pure DataFrame-in functions; cluster_pilots.py is the CLI wrapper. """ @@ -32,10 +32,15 @@ def clustering_features(pilots: pd.DataFrame) -> pd.DataFrame: Raises if any pilot has a NaN feature, naming each pilot and metric. Dropping or imputing would quietly reshape a cohort of 10 to 20 pilots. """ + missing = [c for c in FEATURE_COLUMNS if c not in pilots.columns] + if missing: + raise ValueError( + f"Missing column(s): {', '.join(missing)}. " + "Regenerate the per-pilot features with compute_features.py." + ) features = pd.DataFrame(index=pilots["pilot_id"]) - for metric in METRICS: - features[f"{metric}_std"] = pilots[f"{metric}_std"].to_numpy() - features[f"{metric}_abs_bias"] = pilots[f"{metric}_bias"].abs().to_numpy() + for column in FEATURE_COLUMNS: + features[column] = pilots[column].to_numpy() gaps = [] for pilot_id, row in features.iterrows(): diff --git a/inference/features.py b/inference/features.py index 2a9e29b..60b72b5 100644 --- a/inference/features.py +++ b/inference/features.py @@ -30,7 +30,7 @@ } TARGET_COLUMN = {metric: f"target_{metric}" for metric in METRICS} ID_COLUMNS = ["pilot_id", "flight_id"] -STATISTICS = ("mad", "std", "bias") +STATISTICS = ("mad", "std", "bias", "abs_bias") REQUIRED_COLUMNS = ID_COLUMNS + list(ACTUAL_COLUMN.values()) + list(TARGET_COLUMN.values()) @@ -78,7 +78,10 @@ def flight_features(df: pd.DataFrame) -> pd.DataFrame: _mad mean absolute deviation (size of the deviation) _std standard deviation of signed deviation, ddof=0 (steadiness) - _bias mean signed deviation (not meant for clustering by default) + _bias mean signed deviation + _abs_bias absolute value of this flight's mean signed deviation, so + a pilot's flights on opposite sides of the target do not + cancel when they are averaged per pilot _rows_used samples with a valid deviation NaN deviations are skipped. A flight with no valid sample for a metric gets @@ -108,6 +111,7 @@ def flight_features(df: pd.DataFrame) -> pd.DataFrame: row[f"{metric}_mad"] = deviations.abs().mean() row[f"{metric}_std"] = deviations.std(ddof=0) row[f"{metric}_bias"] = deviations.mean() + row[f"{metric}_abs_bias"] = abs(row[f"{metric}_bias"]) row[f"{metric}_rows_used"] = len(deviations) rows.append(row) return pd.DataFrame(rows) diff --git a/inference/tests/test_clustering.py b/inference/tests/test_clustering.py index fdadd02..f7054fe 100644 --- a/inference/tests/test_clustering.py +++ b/inference/tests/test_clustering.py @@ -33,10 +33,10 @@ def steady_and_wobbly_pilots(): """Four pilots holding steady 60 ft off target (two high, two low), four wobbling around it.""" return make_pilots( [ - ("high1", {"altitude_bias": 60.0, "altitude_std": 5.0}), - ("low1", {"altitude_bias": -60.0, "altitude_std": 5.0}), - ("high2", {"altitude_bias": 60.0, "altitude_std": 5.0}), - ("low2", {"altitude_bias": -60.0, "altitude_std": 5.0}), + ("high1", {"altitude_abs_bias": 60.0, "altitude_std": 5.0}), + ("low1", {"altitude_abs_bias": 60.0, "altitude_std": 5.0}), + ("high2", {"altitude_abs_bias": 60.0, "altitude_std": 5.0}), + ("low2", {"altitude_abs_bias": 60.0, "altitude_std": 5.0}), ("wobbly1", {"altitude_std": 60.0}), ("wobbly2", {"altitude_std": 60.0}), ("wobbly3", {"altitude_std": 60.0}), @@ -45,7 +45,7 @@ def steady_and_wobbly_pilots(): ) -def test_pilots_off_target_in_opposite_directions_share_a_group(): +def test_pilots_equally_far_off_target_share_a_group(): assignments, _ = fit(steady_and_wobbly_pilots(), k=2) steady = {group_of(assignments, p) for p in ("high1", "low1", "high2", "low2")} @@ -62,13 +62,13 @@ def test_sub_foot_bias_differences_do_not_decide_groups(): # look as far apart as 5 vs 50 ft and split the groups the wrong way. pilots = make_pilots( [ - (f"{size}{i}", {"altitude_std": std, "altitude_bias": bias}) + (f"{size}{i}", {"altitude_std": std, "altitude_abs_bias": bias}) for size, std in (("tight", 8.0), ("loose", 12.0)) for i, bias in enumerate((0.01, 0.1, 0.01, 0.1)) ] + [ - (f"off{i}", {"altitude_std": 60.0, "altitude_bias": bias}) - for i, bias in enumerate((50.0, -50.0, 50.0, -50.0)) + (f"off{i}", {"altitude_std": 60.0, "altitude_abs_bias": bias}) + for i, bias in enumerate((50.0, 50.0, 50.0, 50.0)) ] ) assignments, _ = fit(pilots, k=3) @@ -147,9 +147,9 @@ def test_nan_feature_fails_naming_pilot_and_metric(): pilots = make_pilots( [ ("p1", {}), - ("p2", {"heading_std": float("nan"), "heading_bias": float("nan")}), + ("p2", {"heading_std": float("nan"), "heading_abs_bias": float("nan")}), ("p3", {}), - ("p4", {"airspeed_bias": float("nan")}), + ("p4", {"airspeed_abs_bias": float("nan")}), ] ) with pytest.raises(ValueError) as error: @@ -158,3 +158,45 @@ def test_nan_feature_fails_naming_pilot_and_metric(): assert "p2" in message and "heading" in message assert "p4" in message and "airspeed" in message assert "p1" not in message and "p3" not in message + + +def _telemetry_with_altitude_offsets(offsets_by_pilot, seed=0): + """Two-flight telemetry; each pilot's altitude sits at target + that flight's offset.""" + rng = np.random.default_rng(seed) + frames = [] + for pilot_id, offsets in offsets_by_pilot.items(): + for flight, offset in enumerate(offsets): + n = 60 + frames.append( + pd.DataFrame( + { + "pilot_id": pilot_id, + "flight_id": f"{pilot_id}-{flight}", + "altitude": 5000.0 + offset + rng.normal(0, 5.0, n), + "heading": 90.0, + "vertical_speed": 0.0, + "velocity": 120.0, + "target_altitude": 5000.0, + "target_heading": 90.0, + "target_vertical_speed": 0.0, + "target_airspeed": 120.0, + } + ) + ) + return pd.concat(frames, ignore_index=True) + + +def test_pilot_off_target_in_opposite_directions_on_different_flights_is_not_on_target(): + # Averaging signed bias across flights first would make the alternating + # pilots look exactly like the on-target ones. + offsets = {f"alt{i}": (60.0, -60.0) for i in range(4)} + offsets.update({f"ontarget{i}": (0.0, 0.0) for i in range(4)}) + _, pilots = compute_features(_telemetry_with_altitude_offsets(offsets)) + + assignments, _ = fit(pilots, k=2) + + alternating = {group_of(assignments, f"alt{i}") for i in range(4)} + on_target = {group_of(assignments, f"ontarget{i}") for i in range(4)} + assert len(alternating) == 1 + assert len(on_target) == 1 + assert alternating != on_target diff --git a/inference/tests/test_features.py b/inference/tests/test_features.py index 8a786bb..16d6d2e 100644 --- a/inference/tests/test_features.py +++ b/inference/tests/test_features.py @@ -212,3 +212,22 @@ def test_one_row_flight_has_zero_std(): assert row["altitude_mad"] == pytest.approx(100.0) assert row["altitude_std"] == 0.0 assert row["altitude_rows_used"] == 1 + + +def test_abs_bias_is_per_flight_so_opposite_flights_do_not_cancel(): + # 50 ft high on one flight, 50 ft low on the next: signed bias cancels, + # absolute bias must not. + df = pd.concat( + [ + make_flight(flight_id="high", altitude=5050.0), + make_flight(flight_id="low", altitude=4950.0), + ], + ignore_index=True, + ) + flights, pilots = compute_features(df) + flights = flights.set_index("flight_id") + assert flights.loc["high", "altitude_abs_bias"] == pytest.approx(50.0) + assert flights.loc["low", "altitude_abs_bias"] == pytest.approx(50.0) + pilot = pilots.set_index("pilot_id").loc["p1"] + assert pilot["altitude_bias"] == pytest.approx(0.0) + assert pilot["altitude_abs_bias"] == pytest.approx(50.0) From f214b124a1c9e0d42f66f1e0983d377e504cf37f Mon Sep 17 00:00:00 2001 From: 1bananas1 Date: Mon, 12 Oct 2026 09:48:07 -0500 Subject: [PATCH 3/4] Changes requested, see comment for summary --- GLOSSARY.md | 6 +++- .../0002-cluster-on-std-and-absolute-bias.md | 24 +++++++++++-- inference/README.md | 7 ++-- inference/clustering.py | 15 +++++++- inference/generate_synthetic_pilots.py | 8 +++-- inference/tests/test_clustering.py | 36 +++++++++++++++++-- .../tests/test_generate_synthetic_pilots.py | 22 +++++++++++- 7 files changed, 104 insertions(+), 14 deletions(-) diff --git a/GLOSSARY.md b/GLOSSARY.md index 182e259..eaeb0d2 100644 --- a/GLOSSARY.md +++ b/GLOSSARY.md @@ -58,7 +58,7 @@ _Avoid_: Core metrics, KPIs **Performance group**: A set of pilots whose **Deviation features** resemble each other. Groups are not ranked. -_Avoid_: Tier, skill level, class +_Avoid_: Tier (a **Synthetic pilot**'s known level is its **Hidden tier**, a different thing), skill level, class ### Pilots and flights @@ -92,6 +92,10 @@ _Avoid_: Fake data, dummy data, mock data A generated pilot with generated flights, built to deviate by a known amount, used to check that **Performance groups** find real differences before real pilots are available. _Avoid_: Test pilot, fake pilot, mock pilot +**Hidden tier**: +The known deviation level a **Synthetic pilot** is generated at, kept apart from its telemetry so **Performance groups** can be compared against it. Real pilots have none. +_Avoid_: True group, skill level + ## Relationships - A **Pilot** flies many **Flights**; a **Flight** produces many **Samples**. diff --git a/docs/adr/0002-cluster-on-std-and-absolute-bias.md b/docs/adr/0002-cluster-on-std-and-absolute-bias.md index a29c57f..9701201 100644 --- a/docs/adr/0002-cluster-on-std-and-absolute-bias.md +++ b/docs/adr/0002-cluster-on-std-and-absolute-bias.md @@ -5,11 +5,29 @@ ## Considered Options - **All 12 (`mad`, `std`, signed `bias`).** Rejected. Signed bias puts a pilot 50 ft high and one 50 ft low at opposite ends of an axis, although they performed equally well. `mad` mostly restates `std` and `|bias|`, so with KMeans's Euclidean distance it counts each metric twice. -- **`mad` + `std`.** Rejected because of that overlap. -- **`mad` only (4).** The fewest dimensions, but it cannot separate a steady pilot who sits off target from an on-target pilot who oscillates. -- **Linear scale.** Rejected after testing on synthetic pilots. Deviation sizes are positive and pilots differ by multiples, so on a linear scale the worst pilots stretch every axis and the rest bunch together. KMeans then splits the worst tier and merges the other two (mean adjusted Rand index against the true tier about 0.40, against 0.87 on the log scale, over synthetic seeds 0 to 11). +- **`mad` + `std`.** Rejected because of that overlap. The synthetic pilots cannot confirm it (see Evidence). +- **`mad` only (4).** The fewest dimensions, but it cannot separate a steady pilot who sits off target from an on-target pilot who oscillates. This is a structural argument, not a measured one: synthetic pilots draw bias and wobble from one scale, so they move together and never present that case (see Evidence). +- **Absolute value of each pilot's mean signed bias.** Rejected. A pilot 60 ft high on one flight and 60 ft low on the next averages to zero bias and looks on target. Tested on synthetic pilots (see Evidence). +- **Linear scale.** Rejected after testing on synthetic pilots (see Evidence). Deviation sizes are positive and pilots differ by multiples, so on a linear scale the worst pilots stretch every axis and the rest bunch together. - **Log with no floor.** Rejected. It would make 0.01 ft vs 0.1 ft of bias as far apart as 10 ft vs 100 ft. +## Evidence + +Mean adjusted Rand index against the hidden tier, k = 3, synthetic seeds 0 to 11, on synthetic pilots whose adjacent tiers overlap. To reproduce, run `generate_pilots` for each seed, `compute_features`, then the clustering steps in `clustering.py` with each feature set. + +| Features | Log | Linear | +| --- | --- | --- | +| `std` + per-flight `abs_bias` (chosen) | 0.78 | 0.51 | +| `mad` only | 0.76 | 0.50 | +| `mad` + `std` | 0.74 | 0.48 | +| `std` + abs of each pilot's mean signed bias | 0.55 | 0.36 | + +Per-seed scores for the chosen option run from 0.37 to 1.00, so differences of a few hundredths are noise. The runs support two choices: log over linear, and per-flight `abs_bias` over the per-pilot variant. Both orderings also held at each of two heavier overlap settings tried, though the gap between log and linear narrowed to 0.03 at the first. The runs do not separate the chosen features from the `mad` options, so those rejections rest on the structural arguments above. + +Two limits. The generator scales deviations multiplicatively, which is the assumption the log scale encodes, so this shows that log handles that structure and not that real pilots have it. And the overlap was set after trying a few settings, not derived from real flights. + +An earlier generator kept the tiers' scale bands apart. On it the chosen features and both `mad` options scored 1.00 on either scale, so it could not compare them, while the per-pilot variant scored 0.87 on log and 0.41 on linear. Figures from before the tiers overlapped do not apply here. + ## Consequences With 10 to 20 pilots, even 8 features is far below the roughly 70 × features sample size recommended for segmentation (Dolnicar et al., 2014). The k sweep's subsample stability is how we find out whether the performance groups hold up. If they don't, drop to 4 features before adding any. `rows_used` and `flights` measure how much data a pilot has, not performance, and are excluded. Group centres are reported as raw-unit means of their members, since KMeans centres live in log-scaled space. diff --git a/inference/README.md b/inference/README.md index 40dea07..21c2618 100644 --- a/inference/README.md +++ b/inference/README.md @@ -109,9 +109,12 @@ The script never picks k. Run it once without `--k` to print, for k = 2 to min(6 - `stability`: mean adjusted Rand index between the full fit and refits on random 80% subsamples of pilots; 1 means the same groups every time -Then rerun with the chosen `--k`. It writes each pilot's group and each group's centre, as the +Then rerun with the chosen `--k`, which must be one of the swept values (so at least 6 pilots, and +3 or more per group on average). It writes each pilot's group and each group's centre, as the mean of its members in raw units (ft, deg, ft/min, kts). For synthetic pilots, `--truth` also -prints the adjusted Rand index against the hidden tiers. +prints the adjusted Rand index against the hidden tiers; the file must list each pilot once. +The synthetic tiers overlap, so this is below 1: the default seed gives 0.79, and generator seeds 0 +to 11 range from 0.37 to 1.00. ``` python inference/generate_synthetic_pilots.py diff --git a/inference/clustering.py b/inference/clustering.py index f5241ce..19f7f5d 100644 --- a/inference/clustering.py +++ b/inference/clustering.py @@ -72,7 +72,7 @@ def _kmeans_labels(scaled: np.ndarray, k: int, seed: int) -> np.ndarray: def candidate_ks(n_pilots: int) -> range: """k from 2 to min(6, n // 3), so every group can average 3 or more pilots.""" if n_pilots < 6: - raise ValueError(f"Need at least 6 pilots to sweep k, got {n_pilots}") + raise ValueError(f"Need at least 6 pilots to form groups, got {n_pilots}") return range(2, min(6, n_pilots // 3) + 1) @@ -125,8 +125,16 @@ def fit(pilots: pd.DataFrame, k: int, seed: int = 0): centres has one row per group with its pilot count and the mean of each feature in raw units (ft, deg, ft/min, kts), so a group can be read without undoing the scaling. + + Raises if k is outside candidate_ks, the range the sweep offers. That + needs at least 6 pilots. """ features = clustering_features(pilots) + allowed = candidate_ks(len(features)) + if k not in allowed: + raise ValueError( + f"k must be {allowed.start} to {allowed.stop - 1} for {len(features)} pilots, got {k}" + ) labels = _kmeans_labels(_scaled(features), k, seed) assignments = pd.DataFrame({"pilot_id": features.index, "group": labels}) @@ -145,7 +153,12 @@ def truth_agreement(assignments: pd.DataFrame, truth: pd.DataFrame) -> float: For synthetic pilots only: truth has pilot_id and tier. 1 means the groups match the tiers exactly, however they are numbered; about 0 is chance. + + Raises if a pilot is missing from truth or appears in it more than once. """ + repeated = truth.loc[truth["pilot_id"].duplicated(), "pilot_id"].unique().tolist() + if repeated: + raise ValueError(f"Pilot(s) repeated in truth: {', '.join(map(str, repeated))}") merged = assignments.merge(truth, on="pilot_id", how="left") missing = merged.loc[merged["tier"].isna(), "pilot_id"].tolist() if missing: diff --git a/inference/generate_synthetic_pilots.py b/inference/generate_synthetic_pilots.py index 15c5501..8c1f39d 100644 --- a/inference/generate_synthetic_pilots.py +++ b/inference/generate_synthetic_pilots.py @@ -26,9 +26,11 @@ DATA_DIR = Path(__file__).parent / "Data" HZ = 20 TIERS = (1, 2, 3) -TIER_SCALE = {1: 1.0, 2: 2.5, 3: 6.0} -# Per-pilot, per-metric spread around the tier scale. -METRIC_VARIATION = (0.7, 1.4) +TIER_SCALE = {1: 1.0, 2: 2.0, 3: 4.0} +# Per-pilot, per-metric spread around the tier scale. Wide enough that adjacent +# tiers overlap: with separate bands most clustering options score perfectly +# and cannot be compared (docs/adr/0002). +METRIC_VARIATION = (0.5, 2.0) # Per-flight spread of the bias size. BIAS_VARIATION = (0.7, 1.3) # AR(1) coefficient per sample; at 20 Hz a wobble lasts a few seconds. diff --git a/inference/tests/test_clustering.py b/inference/tests/test_clustering.py index f7054fe..6f92a13 100644 --- a/inference/tests/test_clustering.py +++ b/inference/tests/test_clustering.py @@ -107,6 +107,24 @@ def test_sweep_covers_k_from_2_to_min_of_6_and_a_third_of_pilots(n_pilots, expec assert sweep(pilots, n_subsamples=5)["k"].tolist() == expected_ks +@pytest.mark.parametrize( + "k", + [ + pytest.param(1, id="one-group-for-everyone"), + pytest.param(6, id="just-over-a-third-of-the-pilots"), + pytest.param(15, id="one-pilot-per-group"), + ], +) +def test_fit_rejects_k_the_sweep_would_not_offer(k): + # 15 pilots allow k of 2 to 5, so that groups average 3 or more pilots. + rng = np.random.default_rng(0) + pilots = make_pilots( + [(f"p{i}", {"altitude_std": rng.uniform(1, 100)}) for i in range(15)] + ) + with pytest.raises(ValueError, match=r"2 to 5"): + fit(pilots, k=k) + + def test_sweep_needs_at_least_6_pilots(): pilots = make_pilots([(f"p{i}", {"altitude_std": float(i)}) for i in range(5)]) with pytest.raises(ValueError, match="6"): @@ -132,15 +150,27 @@ def test_truth_agreement_fails_naming_pilots_missing_from_truth(): truth_agreement(assignments, truth) +def test_truth_agreement_fails_naming_pilots_repeated_in_truth(): + # A repeat would turn 3 pilots into 4 rows and quietly shift the score. + assignments = pd.DataFrame({"pilot_id": ["a", "b", "c"], "group": [0, 1, 1]}) + truth = pd.DataFrame({"pilot_id": ["a", "b", "c", "b"], "tier": [1, 2, 2, 3]}) + with pytest.raises(ValueError, match="repeated in truth: b"): + truth_agreement(assignments, truth) + + def test_synthetic_pilots_are_grouped_by_their_hidden_tier(): - # Averaged over cohorts, since 15 pilots make any single cohort noisy. + # Averaged over the 12 cohorts of docs/adr/0002, since 15 pilots make any + # single cohort noisy. The tiers overlap, so this is well short of 1. The + # bar sits above a linear scale and the per-pilot |mean bias| (both about + # 0.5 here) and below the chosen features; it does not separate the mad + # options, which score about the same. scores = [] - for seed in range(6): + for seed in range(12): telemetry, truth = generate_pilots(seed=seed) _, pilots = compute_features(telemetry) assignments, _ = fit(pilots, k=3) scores.append(truth_agreement(assignments, truth)) - assert np.mean(scores) >= 0.8 + assert np.mean(scores) >= 0.7 def test_nan_feature_fails_naming_pilot_and_metric(): diff --git a/inference/tests/test_generate_synthetic_pilots.py b/inference/tests/test_generate_synthetic_pilots.py index 30ebcff..b6bf8bb 100644 --- a/inference/tests/test_generate_synthetic_pilots.py +++ b/inference/tests/test_generate_synthetic_pilots.py @@ -1,7 +1,7 @@ import pandas as pd from features import compute_features -from generate_synthetic_pilots import generate_pilots +from generate_synthetic_pilots import TIERS, generate_pilots def test_telemetry_feeds_compute_features(): @@ -36,3 +36,23 @@ def test_same_seed_gives_same_pilots(): second = generate_pilots(n_pilots=6, seed=3, flight_seconds=5) pd.testing.assert_frame_equal(first[0], second[0]) pd.testing.assert_frame_equal(first[1], second[1]) + + +def test_adjacent_tiers_overlap_on_some_std_feature(): + # With every tier cleanly apart on every feature, most clustering options + # score perfectly and cannot be compared (docs/adr/0002). This only rules + # that out; how hard the overlap makes grouping is measured in + # test_clustering.py. + for seed in range(6): + telemetry, truth = generate_pilots(n_pilots=15, seed=seed) + _, pilots = compute_features(telemetry) + std_columns = [c for c in pilots.columns if c.endswith("_std")] + spread = pilots.merge(truth, on="pilot_id").groupby("tier")[std_columns].agg(["min", "max"]) + + overlapping = [ + column + for column in std_columns + for better, worse in zip(TIERS, TIERS[1:]) + if spread.loc[better, (column, "max")] > spread.loc[worse, (column, "min")] + ] + assert overlapping, f"seed {seed}: no adjacent tiers overlap on any std feature" From c36b06dcf699817af3af954ee0111ff61bd1f44c Mon Sep 17 00:00:00 2001 From: 1bananas1 Date: Fri, 16 Oct 2026 09:20:59 -0500 Subject: [PATCH 4/4] Meeting Notes Added --- .gitignore | 3 ++ GLOSSARY.md | 4 +- docs/client_meeting_notes.md | 75 ++++++++++++++++++++++++++++++++++++ docs/telemetry_schema.md | 14 +++++-- 4 files changed, 90 insertions(+), 6 deletions(-) create mode 100644 docs/client_meeting_notes.md diff --git a/.gitignore b/.gitignore index 5e7b35e..f6812ae 100644 --- a/.gitignore +++ b/.gitignore @@ -59,3 +59,6 @@ inference/Data/session_flight_data.csv # Local build archives /xplane_plugin.zip /src.zip + +## Client Data +inference/data/Client/* \ No newline at end of file diff --git a/GLOSSARY.md b/GLOSSARY.md index eaeb0d2..2526802 100644 --- a/GLOSSARY.md +++ b/GLOSSARY.md @@ -106,5 +106,5 @@ _Avoid_: True group, skill level ## Flagged ambiguities -- **Pilot rating** is unresolved: the client may mean a licensing rating (instrument, multi-engine) or a performance score. Do not use it as a precise term until clarified. -- **Target** source is undecided: instructor-assigned, or taken from the autopilot. No targets are captured yet. +- **Pilot rating** is the type of certification, not a performance score (client, see `docs/client_meeting_notes.md`). Whether it means the same as certification level, or an extra rating such as instrument, is still unclear. +- **Target** source is most likely instructor-assigned: the client gives standardized, timestamped instructions during the flight. No targets are captured yet, and the heading reference (true vs. magnetic) is unresolved. diff --git a/docs/client_meeting_notes.md b/docs/client_meeting_notes.md new file mode 100644 index 0000000..7656ab5 --- /dev/null +++ b/docs/client_meeting_notes.md @@ -0,0 +1,75 @@ +# Client meeting notes: Q&A and simulator visit + +Source: transcript of a meeting with the client (Dr. Tamilselvan) in the simulator lab, with a walkthrough of the Prepar3D machine. The meeting date is not recorded in the transcript. The transcript is auto-generated and partly garbled, so anything unclear is marked **unconfirmed**. The earlier kickoff notes are referenced from [telemetry_schema.md](telemetry_schema.md). + +Vocabulary follows [GLOSSARY.md](../GLOSSARY.md). + +## Where the data comes from today + +- The client's existing data was collected in **Prepar3D**, not X-Plane. Prepar3D is closed software, so it cannot be reconfigured; the client's own plugin extracts data from it. +- The Prepar3D export is TSV, and the export rate is one sample per second. A sample rate of roughly 90 to 93 Hz was also mentioned for the simulator. **Unconfirmed**, the two statements conflict. +- The client has 44 pilots. We received a de-identified sample. +- Two pilots in the full set have missing data, and the client will find out why. **Which two is unconfirmed**: the transcript says "six and seven" and "04 and 07", but P004 and P007 are in the sample we received (see #228), so those numbers are probably misheard. +- Every pilot flew one scenario, between one and six trials each. More scenarios and routes are planned. +- The client already segmented each flight into 7 phases (takeoff through landing) and will share the segmentation. +- X-Plane is not installed on the lab machine yet. The client has no date and will give an update the following week. Development proceeds on Prepar3D data in the meantime. +- The simulator machine runs Windows only. + +## Scenario and what counts as on target + +- The scenario's headings: climbs on heading **190 and 40**, to **3100 ft**; approach legs on heading **220 and 0**. +- Judge every leg on all four **Flight dynamics factors**: altitude, airspeed, heading and vertical speed. Approaches are judged on all four too, not on glide path or touchdown point alone. +- The client describes **Performance** as the gap between asked-for and achieved values, for example a pilot who climbs to 3000 ft when asked for 3100 ft. +- Route headings are **true headings**. The plugin currently sends **magnetic** heading (`heading_AHARS_deg_mag_pilot`). The mismatch matters once **Targets** exist, since the two differ by the magnetic variation. **Needs resolving before computing heading Deviation.** +- Units: feet, knots and feet per minute. + +## Flights + +- Time on the ground does not matter. Measure each **Flight** from takeoff to the final landing and leave out the hour or more spent on the ground. +- Stopping and restarting a recording partway through a flight is still **one Flight**. Join the parts by timestamp. +- A full scenario run is about 40 minutes. + +## Targets + +- Pilots receive **standardized instructions** during the flight (for example "change heading to 20"). The instructions are timestamped in iMotions, and the eye tracker records audio and video. +- This partly answers the **Target** source question: targets would be instructor-assigned, not taken from the autopilot. We have none of the timestamps yet, and the client may not be able to share iMotions data at first. +- Without instruction timestamps the pipeline cannot tell a late turn from an early one. A later goal is a model that detects the instruction and segments automatically. Until then, use the client's manual segmentation. + +## Pilot metadata + +- **Pilot rating** means the type of certification, not a score. The client will send certification level, total flight hours and rating for the pilots in the sample. +- For performance groups, the client prioritizes **experience (total flight hours) over certification level**. +- The client agreed to metadata being used **at the end**, to interpret groups and not to form them. This confirms [ADR 0001](adr/0001-metadata-checks-clusters-not-shapes-them.md). +- The client did clustering by hand last year and wants it automated. + +## Clustering and where it runs + +- The goal is real-time clustering. It is not expected to be computationally heavy. +- It should run on the **iMotions computer**, which already shows the pilots' views, and not on the simulator machine. +- Unresolved: where the result is shown (we assume not to the pilot, since it could add stress), and how often to cluster. +- The sensors run at different rates: simulator about 90 Hz, EEG 256 Hz, EDA 128 Hz and eye tracking 51 Hz (as transcribed, **unconfirmed**). The client is not sure there is a common rate to cluster at. iMotions synchronizes the timestamps. +- Later work combines flight data with the three physiological streams (eye tracking, EDA, EEG). These are kept in separate files, separate from demographics. Flight data alone is the starting point. + +## Data handling + +- The data is covered by an IRB protocol. Only people named on the protocol may access the full data. Copies may not be made or published. +- The sample the client shared is de-identified, and use in this class project is fine. Publishing results in a journal or conference is allowed. +- The repository is public, so **never commit raw client data**. + +## Network port + +- The X-Plane plugin sent to a placeholder port in the code. A new port was set, but it is **still unverified** that it matches the port iMotions listens on. The client, who handles the iMotions machine, will check. See [README.md](../README.md) for the open question. +- Whether developers get administrator rights on the simulator machine was not settled. + +## Open items + +| Item | Owner | +| --- | --- | +| Confirm which port iMotions listens on | Client | +| Share the 7-phase segmentation and the scenario briefing | Client | +| Send certification level, hours and rating per pilot | Client | +| Find out which pilots have missing data, and why | Client | +| X-Plane installation date | Client | +| Decide magnetic vs. true heading for heading **Targets** | Team | +| Decide where clustering results are shown, and how often to cluster | Team and client | +| Whether the team continues next semester | Unknown | diff --git a/docs/telemetry_schema.md b/docs/telemetry_schema.md index e0a26c9..b2419f7 100644 --- a/docs/telemetry_schema.md +++ b/docs/telemetry_schema.md @@ -132,11 +132,17 @@ alongside the telemetry data) and joined in at feature-engineering time. | `pilot_id` | string | — | join key to telemetry table | | `certification_level` | string (categorical) | — | e.g. Student, Private, Commercial, ATP — exact category set TBD with client | | `flight_hours` | float | hours | total logged flight hours | -| `pilot_rating` | string or numeric | — | client mentioned "pilot rating" as distinct from certification; needs clarification on whether this is a licensing rating (e.g. instrument rating) or a performance score | +| `pilot_rating` | string or numeric | — | client mentioned "pilot rating" as distinct from certification; the client says it is the type of certification, not a performance score (see below) | -**Open question for the client:** clarify whether `pilot_rating` is a -licensing qualification (instrument, multi-engine, etc.) or an evaluative -score, since that changes its type and how it factors into clustering. +**Answered by the client:** `pilot_rating` is the type of certification, not an +evaluative score (see [client_meeting_notes.md](client_meeting_notes.md)). It +is still unclear whether it differs from `certification_level`. The client +prioritizes `flight_hours` over certification, and agreed it is used after +clustering, not as a feature. + +**Heading reference:** the client says the scenario's route headings are +**true** headings, but `heading` above is magnetic. Resolve this before +targets are populated. ## 5. Deviation formulas