Skip to content

Synthetic pilots whose bias and wobble vary independently #230

Open

Description

@jmacd867

Is your feature request related to a problem? Please describe.
generate_synthetic_pilots.py draws each pilot's steady bias and its wobble from the same deviation scale, so the two always rise and fall together. Pilots who sit steadily off target, or who are on target but oscillate, are never generated. ADR 0002 rejects mad only because it cannot tell those two apart, but with this generator that argument cannot be tested: mad only scores 0.76 against 0.78 for the chosen features, within noise.

Describe the solution you'd like
Draw bias scale and wobble scale independently for some synthetic pilots so both kinds exist, keeping the hidden tier as the pilot's main tier so --truth still works. Measure std + abs_bias against mad only on that cohort and update the ADR's Evidence section with the result.

Describe alternatives you've considered

Additional context
Different from #224, which varies the tier per metric; this varies bias against wobble within a metric. Follow-up to #216 and #225.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions