Skip to content

[Seatbelt] Add learning mode (captureDenials) - #1668

Open
Elliot (theelliotm) wants to merge 10 commits into
mainfrom
user/emichlin/macos-learning-mode
Open

Elliot (theelliotm) wants to merge 10 commits into
mainfrom
user/emichlin/macos-learning-mode

Conversation

@theelliotm

@theelliotm Elliot (theelliotm) commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

📖 Description

Adds learning mode to the macOS Seatbelt backend, following the approved spec (#1609). MXC records what a workload tried to do that the policy blocked, and writes a report in the same format as Windows processContainer.captureDenials. Replaces #1580.

  • Config: seatbelt.captureDenials { mode, outputPath, retainEtl }, in the 1.1.0-alpha contract only. 1.0.0 rejects it. Schema and Node wire types regenerated.
  • Modes: block keeps blocked accesses blocked and records them. allow permits default-denied accesses and records them; explicit denies (deniedPaths, ui) still apply.
  • How it works: MXC tags its profile's deny rules with a per-run ID and reads matching entries from log stream. A start marker confirms log stream is receiving before the workload runs; an end marker confirms late events arrived. A small watchdog stops log stream if MXC is killed.
  • Repeats: denials are analyzed as they arrive and repeats are counted, so a noisy workload can't push other denials out of the report.
  • retainEtl: keeps the raw log stream output as an owner-only .seatbelt.log, reported in etlPath. Capped at 1 GiB with a warning.
  • Warnings: incomplete captures and trace truncation surface as process warnings, including through the SDK streaming path.
  • Discovery: Seatbelt advertises captureDenials when /usr/bin/log is available.
  • mxc-exec-mac --audit: shortcut for allow-mode capture to a temp file.
  • Rejected: captureDenials with profileOverride.
  • Docs: Seatbelt guide learning-mode section, schema.md. logging-access-denied.md renamed to learning-mode.md. A new "Required V2 changes" section in versioning.md.

Limitations (documented):

  • macOS hides remote addresses in network reports (port only).
  • Reads of missing paths aren't reported.
  • The typed .NET/Node/Rust SDK fields wait until contract 1.1.0 is published (spec step 5); until then, use the V1.Dev JSON entry points.

🔗 References

Resolves #1591. After this PR, open a follow up for spec steps 4/5.

🔍 Validation

Run on macOS (Apple Silicon):

  • cargo fmt --all -- --check: passed
  • cargo clippy -p mxc-sdk -p mxc_darwin --all-targets -- -D warnings: passed
  • cargo clippy -p mxc-sdk --lib --tests --target x86_64-unknown-linux-gnu -- -D warnings: passed
  • cargo clippy -p mxc-sdk --lib --tests --target x86_64-pc-windows-msvc -- -D warnings: passed
  • cargo test -p mxc-sdk --lib: passed (1913)
  • cargo test -p mxc-sdk --test mxc_contract_v1_1_0_alpha: passed (380)
  • cargo test -p mxc-sdk --test mxc_contract_v1_0_0: passed (281)
  • cargo test -p mxc-sdk --test streaming: passed (23). seatbelt_pty_preserves_explicit_timeout_kill fails about 1 run in 6 here and on main; not caused by this PR.
  • node scripts/versioning/check-contract-codegen.js: passed
  • node scripts/versioning/validate-configs.js: passed
  • tests/scripts/run_seatbelt_capture_denials_test.sh: passed (99 assertions). New functional suite, wired into the nightly macOS Seatbelt validation through run_seatbelt_all_tests.sh.
  • tests/scripts/run_seatbelt_all_tests.sh: passed (16/16)
  • Heavy workload (200,000 denials over 5,000 files): about 7.5 s, ~14 MB peak memory, report not truncated.

✅ Checklist

  • Signed the Contributor License Agreement
  • Linked to an issue
  • For features or changes requiring a spec: linked the issue with maintainer-applied feature-accepted
  • For implementation: linked the required approved and merged design/API spec, or explained why no spec is required
  • Feature issue remains open unless this PR completes all agreed implementation, validation, and documentation
  • Updated documentation (if applicable)
  • Updated Copilot instructions (if build, architecture, or conventions changed)
  • If this PR changes Cargo.lock, the dependency-feed-check check passes (see pull request builds)

📋 Issue Type

  • Bug fix
  • Feature
  • Task

Copilot AI balanced review requested due to automatic review settings November 4, 2026 00:36
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

An orphaned log collector can block SDK wait() or Drop indefinitely, while a startup race in the overlap test can cause spurious nightly failures.

2 open issues
What changed in this PR

Adds learning mode to MXC’s macOS Seatbelt backend, using the shared Windows denial-report format. Configuration is limited to 1.1.0-alpha; typed SDK support remains deferred until contract publication.

Changes:

  • Adds block/allow capture, retained traces, capability discovery, and the --audit shortcut.
  • Shares report-writing helpers and propagates capture warnings and teardown errors.
  • Adds contract and functional coverage, nightly-suite integration, and usage documentation.
File Description
tests/​scripts/​run_seatbelt_capture_denials_test.sh Exercises capture behavior, reports, isolation, and cleanup.
tests/​scripts/​run_seatbelt_all_tests.sh Includes capture tests in the Seatbelt suite.
tests/​configs/​seatbelt_capture_denials_v1_0.json Tests rejection by contract 1.0.0.
tests/​configs/​seatbelt_capture_denials_timeout.json Exercises timeout with report-write failure.
tests/​configs/​seatbelt_capture_denials_sentinel.json Supports overlap and collector-lifecycle tests.
tests/​configs/​seatbelt_capture_denials_repeats.json Generates repeated and late denials.
tests/​configs/​seatbelt_capture_denials_profile_override.json Tests incompatible profile override.
tests/​configs/​seatbelt_capture_denials_network.json Exercises loopback and proxy capture.
tests/​configs/​seatbelt_capture_denials_gui_clipboard_none.json Exercises explicit clipboard denial.
tests/​configs/​seatbelt_capture_denials_gui_app.json Exercises GUI grants during capture.
tests/​configs/​seatbelt_capture_denials_gui_app_no_gui_access.json Captures GUI dependencies without broad grants.
tests/​configs/​seatbelt_capture_denials_block.json Exercises block mode and unterminated stderr.
tests/​configs/​seatbelt_capture_denials_audit.json Provides an audit-shortcut workload.
tests/​configs/​seatbelt_capture_denials_allow.json Checks permissive capture preserves explicit denies.
src/​tools/​mxc_darwin/​src/​main.rs Adds --audit and report-pointer output.
src/​tools/​mxc_darwin/​Cargo.toml Adds JSON serialization dependency.
src/​mxc-sdk/​tests/​streaming.rs Checks capture-failure warnings after timeout.
src/​mxc-sdk/​tests/​mxc_contract/​v1_1_0_alpha/​seatbelt.rs Tests development capture settings.
src/​mxc-sdk/​tests/​mxc_contract/​v1_1_0_alpha/​fixtures/​one_shot/​valid/​seatbelt_capture_denials.json Adds a valid capture request.
src/​mxc-sdk/​tests/​mxc_contract/​v1_1_0_alpha/​fixtures/​one_shot/​valid/​seatbelt_capture_denials_retain_etl.json Adds a valid trace-retention request.
src/​mxc-sdk/​tests/​mxc_contract/​v1_0_0/​seatbelt.rs Verifies published-contract rejection.
src/​mxc-sdk/​src/​policy.rs Updates a documentation link in a comment.
src/​mxc-sdk/​src/​core/​mxc_engine/​verbose_telemetry.rs Recognizes the Seatbelt report provider.
src/​mxc-sdk/​src/​core/​mxc_engine/​probe.rs Advertises Seatbelt capture capability.
src/​mxc-sdk/​src/​core/​mxc_contract/​dev/​stable.rs Defines development capture fields.
src/​mxc-sdk/​src/​core/​mxc_contract/​dev/​mod.rs Exports the development capture type.
src/​mxc-sdk/​src/​core/​mxc_common/​wire.rs Represents Seatbelt capture settings internally.
src/​mxc-sdk/​src/​core/​mxc_common/​sandbox_process.rs Preserves wait diagnostics and process warnings.
src/​mxc-sdk/​src/​core/​mxc_common/​policy_identity.rs Excludes report destination from policy hashing.
src/​mxc-sdk/​src/​core/​mxc_common/​models.rs Adds capture configuration and output conversion.
src/​mxc-sdk/​src/​core/​mxc_common/​config_parser.rs Maps capture settings and shares path validation.
src/​mxc-sdk/​src/​core/​mxc_common/​config_contract_adapters/​v1_0/​one_shot.rs Keeps capture absent for contract 1.0.0.
src/​mxc-sdk/​src/​core/​mxc_common/​config_contract_adapters/​v0_9/​one_shot.rs Keeps capture absent for contract 0.9.
src/​mxc-sdk/​src/​core/​mxc_common/​config_contract_adapters/​dev/​one_shot.rs Adapts development capture settings.
src/​mxc-sdk/​src/​core/​learning_mode_windows/​etl_decode.rs Reuses shared saturation handling.
src/​mxc-sdk/​src/​core/​learning_mode_core/​verbose_logging.rs Generalizes sibling artifact naming.
src/​mxc-sdk/​src/​core/​learning_mode_core/​mod.rs Exposes shared capture helpers.
src/​mxc-sdk/​src/​core/​learning_mode_core/​capture_output.rs Centralizes report writing, paths, and error composition.
src/​mxc-sdk/​src/​core/​learning_mode_core/​capture_diagnostics.rs Adds version 6 and Seatbelt signatures.
src/​mxc-sdk/​src/​backends/​seatbelt/​common/​seatbelt_runner.rs Integrates capture startup, reporting, and teardown.
src/​mxc-sdk/​src/​backends/​seatbelt/​common/​seatbelt_policy.rs Rejects capture with profile overrides.
src/​mxc-sdk/​src/​backends/​seatbelt/​common/​profile_preparation.rs Prepares and validates report paths.
src/​mxc-sdk/​src/​backends/​seatbelt/​common/​profile_builder.rs Tags rules and implements capture modes.
src/​mxc-sdk/​src/​backends/​seatbelt/​common/​mod.rs Registers capture modules with platform gates.
src/​mxc-sdk/​src/​backends/​seatbelt/​common/​denial_capture.rs Collects tagged logs and manages readiness and draining.
src/​mxc-sdk/​src/​backends/​seatbelt/​common/​capture_analysis.rs Classifies and aggregates captured events.
src/​mxc-sdk/​src/​backends/​process_container/​common/​capture_output.rs Delegates report writing to shared helpers.
src/​mxc-sdk/​src/​backends/​process_container/​common/​base_container_runner.rs Uses relocated capture helpers.
src/​Cargo.lock Records the CLI dependency change.
sdk/​node/​tests/​integration/​macos-seatbelt.test.ts Checks capture capability discovery.
sdk/​node/​src/​generated/​v1_1_0_alpha/​wire.ts Adds generated development capture types.
scripts/​versioning/​config-validation-exemptions.json Exempts the intentional invalid-contract fixture.
schemas/​dev/​mxc-config.schema.1.1.0-alpha.json Adds generated capture schema definitions.
README.md Updates learning-mode documentation links.
docs/​schema.md Documents Seatbelt capture configuration.
docs/​learning-mode.md Adds macOS guidance and report-version updates.
docs/​development/​guides/​diagnostics.md Updates the learning-mode link.
docs/​development/​architecture/​versioning.md Records development availability and deferred V2 changes.
docs/​development/​architecture/​telemetry.md Documents version 6 report handling.
docs/​development/​architecture/​designs/​1591-seatbelt-learning-mode/​README.md Updates report-format links.
docs/​containment-policy/​dev/​policy.md Lists capture fields and incompatibility.
docs/​backends/​seatbelt/​seatbelt-backend.md Documents capture usage, outputs, and limitations.

Review effort: Balanced

self.recorder.summary.actionable_limit_reached = true;
return;
}
self.seen.insert((resource.clone(), access_type));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider enforcing the approved 16 MiB budget for retained denial data, not only verbose signatures.

@theelliotm Elliot (theelliotm) Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather not add it. I optimized how MXC stores repeated denials in memory, so memory grows only with unique denials and verbose groups. Denials are capped at 10,000 entries, like Windows. Each holds one path, and macOS only reports paths up to about 1 KB. A test with a 3 KB path reported at most 824 bytes. That puts the worst case around 20 MiB, so a 16 MiB budget would rarely trigger.

Copilot AI balanced review requested due to automatic review settings November 4, 2026 22:51
@theelliotm
Elliot (theelliotm) marked this pull request as ready for review November 4, 2026 22:57
@theelliotm
Elliot (theelliotm) requested a review from a team November 4, 2026 22:57
@theelliotm
Elliot (theelliotm) requested a review from a team as a code owner November 4, 2026 22:57
@theelliotm Elliot (theelliotm) self-assigned this Nov 4, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Capture tagging can mistake (deny inside a newline-containing path for a rule, corrupting the generated Seatbelt profile and preventing the workload from starting.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Avoid matching deny rules inside quoted path literals

src/​mxc-sdk/​src/​backends/​seatbelt/​common/​profile_builder.rs:208

[P2] This line-based pass also matches (deny inside quoted path literals: a path containing secret\n(deny data) keeps its newline through quote_scheme, so the inserted modifier adds unescaped quotes inside the subpath string. Enabling capture then corrupts the generated profile and prevents the workload from starting. Add tags while emitting deny rules, or make this pass respect Scheme string boundaries, and add a regression scenario with a newline-containing path.

2 resolved since last review

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Seatbelt] Extend learning mode and denial capture to macOS

3 participants