Skip to content

Reuse typed directory task slots with atomic completion - #1048

Merged
helly25 merged 1 commit into
mainfrom
perf/directory-task-slots
Nov 2, 2026
Merged

helly25 merged 1 commit into
mainfrom
perf/directory-task-slots

Conversation

@helly25

@helly25 helly25 commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

Replace per-directory future/callback allocations with reusable typed task records. A per-task atomic handoff improves directory-heavy Linux scans while preserving worker policy, ordered errors, traversal and early-stop draining; macOS remains near baseline.

AG;DR

  • O08 from docs/performance-analysis.md / TODO.md. Based on current main after Reuse budgeted native comparison cursor buffers #1047 (c14d39cf281c084f8833bc877fca6999e2f9f810). Exact measurement parent d2ff56bc444be383de44b3e3b5dd349700abac87 has the same complete tree; preserve its original identity in all data.
  • Replace shared packaged tasks, future states and erased callbacks with deque-stable typed records and move-only requests. Required storage/pool observers are references; optional task observers use OptionalRef. This changes handoff, not the existing persistent thread pool or admission.
  • Keep two-directory admission, 128-entry stat chunks, the 512-entry stat gate, source/archive restrictions, four-listings-per-worker frontier, requested worker counts and coordinator visitation/error order. Directory paths remain owned; stat slices remain borrowed only until all results complete. No nested worker submissions/waits.
  • Publish payloads through a release/acquire Pending/Ready/Abandoned atomic handoff. Exactly one consumer owns a request; the second participant disposes of an abandoned result. Queue admission/recycling retain the existing mutex; completion no longer takes it. Standard atomic wait blocks the consumer. Late notifications are harmless because wait rechecks the value and slot addresses remain valid until workers join.
  • Move discarded payloads into a local value so allocation cleanup runs after unlocking. Drain already submitted jobs on stop without publishing unvisited errors. Slot metadata retains its high-water capacity until walker destruction; the per-level request frontier is not a global byte bound.
  • No new CLI flags, dependencies, exception wrappers, shared task ownership, timed gates or executor hierarchy. Do not describe the former packaged tasks as creating a thread per file.

Decision evidence:

  • docs/benchmarks/directory-task-slots-20261102.md and its lossless bundle retain all initial/refined/targeted populations, source patches, binary identities, fixture recipes and raw samples. The 1.4 MB compressed bundle has an exact-path large-file exception alongside the existing lossless performance reports; no general file-size rule is relaxed.
  • Three-way initial controls: old implementation, lazy old queue, typed records with global-lock completion. The global-lock candidate repeatedly regresses important Linux cells about 11-14% and is rejected. A second three-way matrix isolates atomic completion against fresh baseline/global-lock controls.
  • 28,800 native invocations across both hosts including warmups; three sessions, nine interleaved rounds, fastest seven, independent complete output/digest oracles. Eight tree shapes include flat 1,000/10,000-file directories and 1,024 one-file directories so both job types are exercised. Requested 1/3/10 workers; Linux tmpfs/affinity, macOS warm APFS/unpinned. No builds or diagnostics run during timing.
  • Refined Linux 1,024-directory listing improves 48.27% with three workers and 8.03% with ten; three-worker metadata improves 23.30%. macOS stays mostly near baseline. Worst native paired medians remain +1.16% macOS / +3.40% Linux; no universal speedup claim.
  • Preserve noisy unpinned Linux rg-files readings (+26/-3% and +20/-2% across the two full-kernel sessions). Supplementary paired single-case measurements use three sessions/eight controls; Linux explicitly pins its requested 1/4 CPUs. The 1,000/10,000-file four-worker medians remain small regressions (+2.79%/+1.40%), not erased or normalized away. Keep separate contracts and raw data.
  • Coordinator-only ASan counts at 1,024 directories fall 4,145 -> 1,056 macOS and 4,137 -> 1,056 Linux; requested bytes roughly 617/625 kB -> 333 kB. Worker allocations/frees are excluded, so this is neither all-process traffic nor net live-byte/leak accounting. No sanitizer timing is used.
  • Build-output executables shrink 18,384 bytes macOS and 14,192 bytes Linux before stripping/compression.

Validation:

  • Initial and atomic implementations both pass 305/305 complete release/extension tests on macOS/Linux, with working FUSE required on Linux. Focused ASan/TSan and all allocation diagnostics pass. Current-main synchronization preserves the measured source; focused synchronized tests are recorded before publication.
  • Added mixed stat/directory slot reuse above the actual 512-entry gate, odd tail chunks, exact read counts and coordinator order, ordered read errors, and abandoned-error suppression after early stop. Existing prune/depth, source ownership, archives, race suppression and frontier bounds remain covered.
  • The first actual-worker allocator diagnostic exposed macOS TLS bootstrap recursion in the old hook. Reject non-coordinator callbacks before touching POD counters and keep a worker-start regression. This is test-only; production timing binaries are uninstrumented.
  • Scoped local clang-tidy passes all three modified TUs; source hashes match the refined selected manifests. Normal commit/push hooks and current-head CI must pass before merge.

@helly25
helly25 merged commit 7cf2a52 into main Nov 2, 2026
32 checks passed
@helly25
helly25 deleted the perf/directory-task-slots branch November 2, 2026 06:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant