Skip to content

docs: accept SEP 47, backend-owned execution sessions - #589

Merged
jtravs merged 10 commits into
mainfrom
docs/sep-0047-run-ownership
Nov 9, 2026
Merged

jtravs merged 10 commits into
mainfrom
docs/sep-0047-run-ownership

Conversation

@jtravs

@jtravs jtravs commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

SEP 47 makes each backend the only owner of a propagation's mutable numerical state and schedule. Python keeps the scientific construction and external services; native exposes its complete session, and the independent NumPy oracle uses a host session in both precisions. MLX is retired early. #346 and #361 are included workstreams.

Status: Accepted by John Travers on 2026-11-09, after two AI reviews (Opus 5.5) and the drafting agent's responses. Before implementation, Opus 5.5 checked Part I against main at 0901bf6f. The maintainer then decided the questions that check raised (Part III, "Maintainer decisions at acceptance"):

Delivery uses the recorded SEP 0 exception: a few coherent phase PRs, feature/sep-0047-<milestone>.

Tracks #588; addresses #110's design questions. Accepting the proposal closes no implementation issue.

Validation:

  • git diff --check and codespell on the changed files passed.
  • pytest tests/test_seps.py: 342 passed, 3 expected process-SEP skips.
  • poe docs built cleanly with warnings as errors.

Documentation only. No native build was changed, and no benchmark or full gate was run.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XGENPwVEY6NQKGaDDptaj9

Copilot AI balanced review requested due to automatic review settings November 9, 2026 00:01

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-09T00:02:42.485940Z 2853d31 PR opened
🔒 Security Review ✅ Completed 2026-11-09T00:03:56.220178Z 2853d31 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@jtravs jtravs added area: native The native backend, threading and FFT planning documentation Improvements or additions to documentation labels Nov 9, 2026 — with ChatGPT Codex Connector
@jtravs jtravs mentioned this pull request Nov 9, 2026
4 of 7 tasks
jtravs and others added 3 commits November 9, 2026 00:32
Record the review in place under Part III as points R1-R17, with the
questions for the maintainer. Part I is unchanged. Review status moves to
In review with Opus 5.5 (AI) as reviewer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jtravs commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the Opus 5.5 review in a6a2130, after updating the branch to main at eee57c6 (SEP 46 / #582 is now Final).

The response to all 17 findings is in Part III. The original review text is preserved; separately labelled maintainer decisions sit beside R4, R5, R6, R9 and R12. Part I and the roadmap were revised accordingly.

John Travers resolved the arbitrations:

  • R4: allow a small number of coherent phase PRs as an exception for SEP 47, each leaving main usable; remove temporary internal adapters by Final.
  • R5/R6: native exposes only its complete session. Keep custom Python controllers and benchmark comparison methods reference-only; port built-in production capabilities, including coupled flows and carrier tables.
  • R9: record restart retention/restoration/reset choices and reasons, warning once on resume for unexpectedly omitted Adams history. Full RK continuation is deferred to Qualify full Runge–Kutta checkpoint continuation #596.
  • R12: use an atomic native stop token checked between attempts, finishing the current attempt safely. Keep bounded returns for Python signals. Snapshot requests become independent of step_planes=1, with valid checkpoint/history persistence before copying.
Finding Disposition in the revised proposal
R1 Concrete remaining crossings, historical cost ceilings and event-rich measurement cases added. Where no isolated timing exists, phase 1 must measure it; no speculative ceiling is supplied.
R2 Exact identical-input control traces and strict-arithmetic witnesses required. Clarified that the existing RK bitwise diagnostic shares native norms/kernels; independent NumPy field arithmetic still follows SEP 25.
R3 Monitor state is controller input; cap order and restart state are explicit. Independent exact watch tests replace production shadow checking.
R4 Phased-merge exception and concurrent-feature policy recorded.
R5 Every fallback has a port/refuse/retire disposition and qualifying witness. Built-in coupled capabilities are retained without requiring a pre-existing performance case.
R6 Removed the host-over-native executor; corrected the custom-controller and MLX premises; specified retirement of the StepDriver hierarchy.
R7 Yields, acknowledgments and observational reporting are exact lifecycle no-ops, with deterministic CPU and scripted-input witnesses.
R8 Runtime batch-axis/member contracts and checkpoint serialization specified. Preserved existing per-member boundary versus group-fatal numerical failure semantics, including partially advanced groups.
R9 Corrected the current restart claims; added explicit records/warnings and separately tracked full RK continuation. Controller scalars alone do not establish exact continuation.
R10 Defined extension procedures for routes, responses, statistics and auxiliary state. Structural manifests replace old seam checks; the already-removed schema version is not restored.
R11 Added precision scaling, spatial-route rules and one output-owned conversion recipe. Identical exports are shared; raw restart state and a converted/dense saved field may legitimately need different payloads.
R12 Specified safe-boundary token latency and independent snapshots. Declined in-place mid-stage interruption; Python signal handlers still require bounded returns to Python.
R13 Added explicit attempt, commit, monitor, statistics, plane, snapshot, warning and terminal event order.
R14 One statistic implementation per backend, after-finish sampling, exact coincident-result reuse and resident bulk row delivery.
R15 Added aggregate non-regression and structural return counters alongside the per-case 5% trigger. Measurement windows must actually contain the relevant events.
R16 Preserved explicit Helpers ownership and its test, including MLX's installed/no-native distinction.
R17 Corrected dependencies, Adams timing attribution, SEP 46 status and interface descriptions. Clarified that public backend opacity already exists, while the native layout prohibition and shared-march coupling are real constraints being removed.

Validation passed: git diff --check, codespell on changed documentation, 342 SEP tests passed / 3 expected skips, and poe docs with Sphinx warnings treated as errors. Also verified that the reviewer's original text remains unchanged and all R1–R17 points are answered.

This remains a documentation-only Draft, In review. No native build, numerical qualification or fresh benchmark is claimed. The requested maintainer arbitrations are resolved; the revised whole proposal is ready for the next review before acceptance.

jtravs and others added 2 commits November 9, 2026 12:18
Record that R1-R17 are answered and consistent with the maintainer's
decisions, and add six follow-ups (A1-A6): checkpoint representation,
device scope of ported host-march cases, SEP 46 Final updates, ledger
readings, structural counters and the SEP 0 exception's bookkeeping.
Part I is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPtLzcRHFYzZjsv6qNk

jtravs commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Addressed A1–A6 from the Opus 5.5 re-review in 2c11c68. The individual responses are in Part III; both original reviews and the first response are preserved unchanged.

The re-review accepts the R1–R17 responses and says none of these follow-ups reopens a maintainer decision. I checked the remaining claims against the checkpoint writer/reader, field resume, native admission/host-engine paths, device capability table, SEP 46's final report and the benchmark ledger.

  • A1 — Checkpoints: §§5.4–5.5 now explicitly preserve today's field-checkpoint representation and rounding-level modal normalization inverse. The Python record uses output amplitudes in the internal FFT convention; HDF5 stores the same optics convention as regular saved fields. Running checkpoints reuse the existing record/datasets plus optional metadata. A landed checkpoint and coincident plane share one export and output conversion; a dense plane at a different position needs its own field payload. The optional Adams raw-state/history snapshot remains a separate existing record. No new raw-field checkpoint format is implied.
  • A2 — Devices: every §6.3 row now has an explicit CUDA/HIP disposition. Existing production coupled cases—including off-eigenbasis and more than 16 channels—groups, and low-storage replay are session ports with real CUDA/HIP qualification required before Final. CUDA low-storage is currently admitted but not yet qualified; its qualification must be demonstrated. Existing carrier-frame/general-z-table/CF4/coupled-group refusals remain. No additional production device capability is dropped.
  • A3 — SEP 46 / chunks: corrected the SEP 46: calibrating the native backend to its machine #582 and The node chunk is sized at translation, whatever the device, and each chunk is a kernel launch #361/node_budget still counts 20 column-equivalents per node, about three times the live group buffers #349 descriptions. SEP 46 retained no calibrated device setting and landed no chunk default. The node chunk is sized at translation, whatever the device, and each chunk is a kernel launch #361 may land independently on the current seam; phase 3 then moves its qualified rule into backend lowering without changing that default.
  • A4 — Timing attribution: the 290/75 µs figures are explicitly attributed to producing the statistics row, rather than the return. Added the snapshot-forced observation proxy: M5 39.5 → 15.2 µs/trial and EPYC approximately 58 µs/native step. These do not measure snapshot persistence or copying.
  • A5 — Structural counters: added transverse-edge reads, scheduled native-group steps, and snapshots enabled with no pending request to the zero-avoidable-return requirements.
  • A6 — Process bookkeeping: indexed the approved exception in SEP 0's History. SEP 47 §11 specifies that the Implementation header retains every successive milestone branch and linked PR, in delivery order, while Part II holds commits and qualification.

Validation passed: git diff --check, codespell, 342 SEP tests / 3 expected skips, and poe docs with warnings treated as errors. Verified both review texts and the first response are unchanged, and every A1–A6 point is answered. No runtime changes, fresh benchmark or device qualification are claimed.

No new maintainer arbitration is needed. The proposal remains Draft / In review, pending human review and formal acceptance under SEP 0.

jtravs commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Updated in 2a66238 following John Travers's scope decisions. The SEP remains Draft / In review, pending his reading and explicit acceptance.

The roadmap, current proposal and tracking issue agree. Earlier reviews and responses remain unchanged in Part III, followed by a record of the superseding scope decisions.

Checks passed: diff/spelling, SEP validation (342 passed; 3 expected skips), and Sphinx with warnings as errors. No implementation or fresh benchmark was performed.

@codecov

codecov Bot commented Nov 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

John Travers accepted SEP 47 with five decisions taken before implementation:
device capabilities keep their per-vendor status (CUDA low-storage is the only
new qualification); every periodic statistic is computed natively; the native
session owns the monitor's decision state and records, with exact identity on
scalar decision logic; the reference session is built independently in phase
2; and #584 is delivered separately by PR #598.

Part I also now specifies running checkpoint contents, monitor restoration,
the calibration timing probe, groups as sequences of member problems and the
A/B/C attribution. The roadmap entry moves to Accepted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPwVEY6NQKGaDDptaj9
@jtravs jtravs changed the title docs: draft SEP 47 for backend-owned execution sessions docs: accept SEP 47, backend-owned execution sessions Nov 9, 2026
@jtravs
jtravs merged commit 930ddcb into main Nov 9, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: native The native backend, threading and FFT planning documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants