Skip to content

Record stage-specific concurrency and execution scheduling follow-up - #1002

Closed
helly25 wants to merge 1 commit into
mainfrom
docs/stage-specific-job-control
Closed

helly25 wants to merge 1 commit into
mainfrom
docs/stage-specific-job-control

Conversation

@helly25

@helly25 helly25 commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Record the concurrency follow-up from the #997 review: directory discovery, evaluation and external execution need separate scheduling controls. This preserves #997's scope and current CLI behavior while making the design questions and qualification work explicit.

AG;DR

Scope and reasoning

  • Add a future-work insight to docs/design-parallel.md and linked JC01-JC06 entries in TODO.md.
  • Explore a --jobs-* family for directory I/O, matching/comparison and external execution. The listed spellings are illustrative, not implemented or stable options.
  • Leave the meaning of --jobs in mixed-stage configurations open: fallback versus aggregate ceiling, caller-inclusive accounting, defaults, all, overrides/repetition and INI behavior need a design decision.
  • Distinguish one-at-a-time queued execution from synchronous expression continuation: serializing a child queue alone does not preserve exit-status-dependent truth, short-circuiting or chained effects.
  • Record traversal/effect barriers for prune, quit, post-order, captures, reductions, mutations and ordered publication. Speculative directory reads must not become unauthorized visits/actions.
  • Compare role-aware command-owned execution with coordinated persistent stage pools. Reuse workers, prevent blocked I/O from exhausting every CPU/execution slot, avoid nested pool waits, and bound queued work, retained bytes and outstanding children separately.
  • Require filesystem/hardware-dependent directory admission. The user's roughly three-reader local and 50-core network examples are motivating hypotheses, not measured XFF defaults; explicit I/O concurrency may exceed detected CPU count.
  • Track Linux/macOS and local/slow/network qualification, actual thread/child counts, output correctness, cancellation, useful task size, first-output latency, throughput and memory.

Changes deliberately excluded

  • No edits to Bound and cancel directory read-ahead on the shared executor #997 or its bounded-frontier implementation.
  • No new flags, registry/help entries, changed defaults, runtime scheduling, build dependencies or toolchains.
  • No decision about provisional spellings; implementation with unsettled spelling must use the repository's unstable gate and ship its documentation/tests together.
  • No engine adoption or performance acceptance claim.

Validation and portability

  • Based on main 4ea895f63106f05df7c90b629d20433ae1e03016.
  • Commit cc7b0dec173a68251c216e06b131fbbf18634061 changes only the two Markdown files (99 insertions).
  • git diff --check and the repository Markdown-table alignment check passed.
  • pre-commit run --files TODO.md docs/design-parallel.md passed; irrelevant source hooks reported skips.
  • The normal commit and push hooks, including Trunk formatting/checks, passed without bypasses.
  • There is no executable/build change to affect macOS behavior; cross-platform performance measurement is future work, not validation claimed by this documentation PR.

Known limitations

This records insight and actionable work, not a settled API or implemented pipeline. Hardware examples do not establish a universal concurrency limit. Strict dependent expressions and arbitrary child commands require semantic analysis before concurrency can be expanded safely.

Consolidation into #999

This exact documentation commit is now included in PR #999 at head 2e3e789693f38a5f6e7fee4ffe209b6b2f7710ad, rebased onto merged main 88d4facd15ebd00acf749222b3028de2e8916970. The original two-file documentation was preserved; #999 additionally carries JC07 tuning and JC08 opt-in, super-low-overhead analysis work requested in follow-up, the fail-closed compile-source query, and a cross-platform native benchmark launcher staging fix. None of these changes implement engine scheduling behavior.

The combined change passes the five affected local tooling targets, live repository tooling, normal focused/commit/push hooks, and canonical compile-database preparation (294 expected sources, none missing). Hosted validation for the new combined head completed successfully in Test run 38050820172, including both platforms and Snyk; earlier green original-head runs do not establish combined-head acceptance. The user approved CI-gated squash auto-merge for #999, which is enabled. PR #999 merged on 2026-10-10 at 12:42:02 UTC as bbc0bf9376c7fc41fe75094998c3c7e74bd02e94; this PR is closed as superseded after the merged source/doc preservation check succeeded. Its original branch is retained; no separate push or duplicate CI run was triggered here.

Verified merged preservation

  • The actual squash commit is bbc0bf9376c7fc41fe75094998c3c7e74bd02e94, with parent 88d4facd15ebd00acf749222b3028de2e8916970 and tree 38d5ecc9e1789f0f0a187960c24daee1f7884da0.
  • The merged tree is exactly identical to the locally validated combined Fix benchmark launcher staging and consolidate audit preparation #999 head 2e3e789693f38a5f6e7fee4ffe209b6b2f7710ad; git diff --exit-code found no source or documentation differences.
  • The final TODO.md and docs/design-parallel.md bytes exactly match the preserved follow-up documentation in the original Record stage-specific concurrency and execution scheduling follow-up #1002 worktree (SHA-256 a798981a7a1e54149d83022bc875840146829986055b1b342bf12d9a442af2a4 and a522015a0d2fc5feb52135bd2a6e56c9ae6b8e53d5b55652e8abf6f5dd6879f5).
  • All required exact-head CI checks succeeded, with four adaptive extra shards skipped as designed. Closing this superseded PR does not delete its original branch or alter any frozen driver, measurement artifact, experimental scheduler worktree or uncommitted documentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant