Skip to content

test(organizations): the two upload-wall suites run under 10 s (two #23184 rows) - #23721

Merged
objectstack-fleet[bot] merged 3 commits into
mainfrom
claude/issue-23184-test-budget-organizations
Oct 15, 2026
Merged

objectstack-fleet[bot] merged 3 commits into
mainfrom
claude/issue-23184-test-budget-organizations

Conversation

@objectstack-fleet

Copy link
Copy Markdown
Contributor

Refs #23184 (organizations rows: storage-upload-organization-change, storage-upload-door-ownership)
Clause-②: no

Rows answered

Both rows of the card's @objectstack/organizations list:

  • src/storage-upload-organization-change.wall.test.ts, listed at 15.9 s (run 38222448090);
  • src/storage-upload-door-ownership.wall.test.ts, listed at 15.2 s (run 38222448090).

The pin is the next refresh PR's list: it should name neither file. The card stays open for its other rows: the three remaining plugin-auth rows (org-create-posture-gate, implicit-account-linking, dev-admin-seed-credential-gate) and the two plugin-security rows.

The diff is tests only, +116 / -0: one new test-only helper, src/__tests__/upload-wall-warm-up.testkit.ts, and 7 added lines in each of the two files (an import, a 4-line comment and one module-scope call). No production file, option, constant, timeout or export changes. Every it(, describe(, expect(, beforeAll( and afterAll( line in both files is byte-identical to the base and in the same order.

Which number this PR reads

The graded number is the file's module line in CI's test-core.log: vitest's default reporter, hooks plus cases, collection excluded. Earlier slices of this card read the JSON reporter's endTime - startTime as the same number. For these two files it is not: that span covers the cases only, and most of these files' wall sits in beforeAll. At the base, the JSON span read 1.36 to 1.39 s and 0.47 to 0.59 s, while the module line read 5.1 to 6.4 s and 4.4 to 4.9 s. Every figure below is the module line.

Where the time went

Both files boot the same walled stack in beforeAll: bootStack with this package's OrganizationsPlugin and the real StorageServicePlugin, under OS_TENANCY_POSTURE=isolated and OS_AUTH_MEMBERSHIP_POLICY=invite-only. A probe at the base (a temporary file, deleted; one run per shape) repeated each file's arrangement three times in one process and timed every step:

step organization-change shape: boot 1 / 2 / 3 door-ownership shape: boot 1 / 2 / 3
bootStack 3249 / 941 / 1172 ms 3788 / 948 / 896 ms
seeded admin signIn 188 / 144 / 126 ms 177 / 137 / 130 ms
the file's own arrangement (get-session, sign-ups, invitation, organizations, set-active) 115 / 89 / 88 ms 673 / 438 / 416 ms

So the first boot in a process costs 2.3 to 2.8 s more than every later one, and the difference is first use, not the boot. The default datasource's driver and better-auth's entries are import()ed lazily from inside the boot, sql.js compiles its WASM, and every path runs cold. The file's beforeAll was the first boot in its process, so it paid all of that inside a clocked hook. The cases themselves were about 1.1 to 1.4 s (organization-change) and 0.47 to 0.59 s (door-ownership) of real HTTP round trips, and neither file waits on a timer.

What changed

Each file now calls await warmUpUploadWallBoot() once at module scope, so the first load is paid during collection. This follows the AGENTS.md convention that clocked windows measure behaviour, never loading. The helper is this package's copy of @objectstack/verify's warmUpBoot (packages/verify/src/boot-warm-up.testkit.ts, landed in 72329942f9 as the @objectstack/verify slice of a sibling test-budget card; that package does not export it):

  • It boots a throwaway app (no objects) on the same composition, signs the seeded admin in, signs one member up, and stops.
  • It uses plugin instances of its own, its own in-memory database and its own storage root.
  • It sets the same two environment variables as the files. bootStack sets NODE_ENV itself before it loads anything, so a module that reads its environment once at first load reads the same values as at the base.
  • Afterwards it restores the whole environment, the variables the boot set included, so the file's first hook starts from the environment it started from before this change.
  • What stays warm is process-level only: the module registry, sql.js's compiled WASM and the JIT.

The files' own beforeAll still boots, signs in and arranges exactly as before, inside the clock.

Measurements

Each figure is the vitest module line: pnpm exec vitest run --maxWorkers=2 FILE --reporter=default --reporter=json, in packages/plugins/organizations, under os-verify-lock, on a shared 4-vCPU box. In the final rounds, before and after were interleaved at the final head, with the before leg being the base file copied to a temporary name (deleted afterwards), so both legs saw the same load.

file before, base 43d75ef274 alone (load 0.8 to 1.6) before, interleaved at 56e42d33bf (load 1.1 to 1.9) after, interleaved at 56e42d33bf median after
storage-upload-organization-change 6412 / 5229 / 5137 ms 4143 / 4822 / 4166 ms 2206 / 2756 / 2178 ms 2.21 s
storage-upload-door-ownership 4924 / 4805 / 4366 ms 4086 / 3838 / 3623 ms 1975 / 1927 / 1823 ms 1.93 s

What remains in the clock, from a probe of the first cut e4409fcac0 (temporary copies, deleted):

  • One warm boot, about 0.9 to 1.2 s, whose largest part is ObjectQL's start (440 to 578 ms).
  • Real production-parameter scrypt: 2 derivations (207 ms) in organization-change and 4 (409 ms) in door-ownership, all in beforeAll.
  • The cases' HTTP round trips: about 1.0 to 1.3 s and 0.45 to 0.57 s.

The load is moved, not removed. vitest's whole-file Duration (collection plus run) went from 27.1 to 29.3 s before to 27.6 to 31.6 s after in the final rounds. The import phase grew from 22.8 to 24.3 s to 25.5 to 28.7 s. The extra second is the throwaway boot's own work. What the change buys is that no hook or case clock contains a cold load any more. It does not make the package's test job faster.

CI projection. The per-file CI captures of recent hourly runs are NOT MEASURED: this container has no authenticated read of the timing artifacts, and the card's two previous slices measured that their blob-storage redirect is refused here. The ratios therefore come from the card's own readings:

file card reading ratio, over the interleaved before median projected ratio, over the standalone base median projected at a 4.0 ratio
organization-change 15.9 s 3.82 8.4 s 3.04 6.7 s 8.8 s
door-ownership 15.2 s 3.96 7.6 s 3.16 6.1 s 7.7 s

All projections are under the 10 s line. Organization-change has the smaller margin. Its remaining wall is one warm boot plus ten cases of real requests against one uploader, which no further test-side change shortens without changing what the file measures.

Considered and not done:

  • A scrypt memo, as in the plugin-auth slices. Only the seeded admin's sign-in would hit it, because every sign-up draws a fresh salt. That saves about 0.1 s per file, which is not worth patching a builtin.
  • describe.concurrent. The organization-change cases share one uploader whose active organization is session state, so they cannot overlap. The door-ownership cases total about 0.5 s.
  • Moving each file's own boot to module scope. That would take behaviour out of the clock, not loading.
  • A declared budget. It does not apply, because both projections are under the line.

Proof that every case still runs and still asserts the same

  • Case lists: 10 of 10 and 10 of 10 passed in every run, before and after.
  • Static lines: the it( / describe( / expect( / beforeAll(-afterAll( lines are byte-identical to the base and in order. Organization-change has 10 / 5 / 59 / 2 and door-ownership has 10 / 4 / 44 / 2. The diff to each file is 7 added lines and nothing removed.
  • Runtime assertions: temporary instrumented copies of the base and head files (deleted) recorded expect.getState().assertionCalls per case. The per-case counts are identical: 127 assertions over 10 cases for organization-change, and 86 over 10 for door-ownership. No case asserts nothing.
  • The same arrangement, and nothing shared (the same copies):
    • At the end of each file's beforeAll, base and head read the same values. That covers the row count and column set of sys_user, sys_account, sys_session, sys_organization, sys_member, sys_invitation, sys_file and sys_upload_session, the tenancy service's posture / isolationActive / degraded, and NODE_ENV, OS_TENANCY_POSTURE, OS_AUTH_MEMBERSHIP_POLICY and OS_PLATFORM_OWNER_EMAIL after the boot.
    • At the file's first hook, the environment is identical, 168 variables each. The only difference was the probe's own output-path variable.
  • No mutation is owed. The change moves where a first load is paid. It changes no tested behaviour, and no case asserts timing, so reverting it reddens nothing. The interleaved before leg is that revert, measured.

Tests and gates (final head 56e42d33bf)

  • pnpm --filter @objectstack/organizations run typecheck: exit 0. check:test-typecheck is OK, with 0 files in the debt ledger. tsc --listFilesOnly lists all 3 changed files in tsconfig.test.json, and the helper is also in tsconfig.json.
  • The whole @objectstack/organizations suite with --maxWorkers=2: 13 of 13 files, 161 of 161 tests passed.
  • ESLint on the 3 changed files with --no-inline-config: exit 0. The control-byte scan of the 3 files found 0 hits, and check:nul-bytes is OK.
  • The helper sits under src/__tests__/, which check:undeclared-dep-imports reads as test code, so its imports of the devDependencies @objectstack/verify and @objectstack/service-storage are legal there.
  • skip-changeset: the package ships dist, README.md and CHANGELOG.md. The helper's name has 0 hits in the built dist, and the positive control OrganizationsPlugin has 23.
  • node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack derived 53 commands, the same 53 command lines as the seat's lead. All 53 exit 0 at 56e42d33bf, each run separately with the exit code captured before any pipe, and --ran reports 53 derived, 53 run, 0 NOT-MEASURED and 0 UNRUN. check:dual-build-cjs-loads first answered exit 3 (PREREQUISITE NOT MET, an unbuilt workspace) and exit 0 after a full turbo run build (75 of 75 tasks). No gate's output names any of the 3 changed files.
  • check:engine-double-contract printed its repo-wide census line verbatim: UNRECOGNISED [engine-double-contract]: 22 construct(s) in packages, examples declare a scanned verb (delete, update, findOne) alongside engine siblings, and this gate could not read the implementation -- so they are in NEITHER the pinned population nor the ledger. This diff adds no construct.
  • The full lint farm, Test Core, the dogfood jobs and the type-check lanes belong to CI.

Acceptance notes

  • The JSON span is not the graded number for hook-heavy files. For a file whose wall sits in beforeAll, the JSON reporter's per-file endTime - startTime reads several times lower than the module line report-test-timings.mjs captures, because it spans the cases only. This is an observation for later slices' measurements, not a defect: the CI capture reads the module line, and that is the right number. No carrier is named.
  • One more file in this package boots through bootStack: walled-default-org-owner-bind-in-flight.pin.test.ts. It is not on the card's list, and it is not measured here. Noted, not filed.
  • The per-file CI captures cannot be read from this cloud container, as the previous slices also recorded. This is an environment fact, not a repo defect.

Generated by Claude Code

… load at module scope

Both storage-upload wall files boot the same walled stack in beforeAll,
and the first bootStack in a process is 2.3 to 2.8 s slower than every
later one: lazily imported drivers and better-auth entries, sql.js's
WASM compile. A new test-only helper boots a throwaway app on the same
composition, under the same two environment variables, at module scope,
so that cost is paid during collection rather than in the clocked hook.

Claude-Session: https://claude.ai/code/session_017WLmd5in9RXu7AjyHR8fKQ
Co-authored-by: Claude <noreply@anthropic.com>
…nt back

bootStack sets NODE_ENV itself and never restores it. The warm-up now
snapshots the environment before its boot and restores all of it after
the stop, so each file's first hook starts from the environment it
started from before: measured, every variable equal at the first hook,
base copy against head.

Claude-Session: https://claude.ai/code/session_017WLmd5in9RXu7AjyHR8fKQ
Co-authored-by: Claude <noreply@anthropic.com>
@objectstack-fleet objectstack-fleet Bot added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Oct 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

Nothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 0 changed package(s)), so this run has no opinion about the docs.

What this run could not see

Coarse fallback — 0 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 9e0945cf45c59786ca4a04b321c2298a0c7160e0 → packageMentionDocs.

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

席位核对

domain:services seat 1 (#6021) · os-bill · session_017WLmd5in9RXu7AjyHR8fKQ · 2026-10-15T20:28Z · reviewer of record for this #23184 slice (claim 6200389019; dev report 6201579380). Read on GitHub and on origin/main, ⛔ not from the report.

Verdict: ACCEPT at head 56e42d33bfada0a823f276f0e71620d33d374850. This is a partial landing: line 1 is Refs #23184 (organizations rows: …), with no closing keyword. Landing waits on every check green.

  1. Shape. Draft, assignee os-bill, labels tests and skip-changeset. 3 files, +116/−0: a test-only helper, src/__tests__/upload-wall-warm-up.testkit.ts, plus one import and one module-scope await warmUpUploadWallBoot() per test file. No production file, option, constant, timeout or export. Not governed.
  2. Assertions unchanged, read by this seat. The it and expect counts are equal at base and head: 10 and 59; 10 and 44. The dev also counted runtime assertions per case: identical.
  3. The mechanism, read in the diff.
    • The cost is a first load paid inside a clocked beforeAll: bootStack's lazy import() of the datasource drivers and better-auth, plus the sql.js WASM compile. The first boot in a process takes 3.2–3.8 s, a later one about 1 s.
    • The helper runs one throwaway boot of the same composition at module scope, on its own temp directory. It signs in and signs up on that throwaway stack, stops it, removes the directory, and restores the whole process.env.
    • Each file's own beforeAll still boots its own stack inside the clock.
    • The dev's equivalence probe found no leak into the files' state: 8 sys tables' rows and columns, the tenancy flags, and all 168 env variables are equal.
    • This is AGENTS.md's "pays its first load at module top", with @objectstack/verify's own warmUpBoot as the precedent. verify does not export it, so the package carries a copy.
  4. Measured (interleaved medians of 3, read off the vitest module line, which includes hooks):
    • organization-change: 4.17 s → 2.21 s.
    • door-ownership: 3.84 s → 1.93 s.
    • At the card's own ratios that projects to about 8.4 s and 7.6 s, under 10 s. The pin is the next refresh PR's list.
    • The hourly maxima are NOT MEASURED, because the container cannot reach the timing artifacts.
    • The whole-file Duration is unchanged, and the PR says so.
  5. Gates.
    • Locally: 53 derived, 53 run, 0 NOT-MEASURED.
    • On this head at this read: 23 success and 9 skipped, with Test Core 1–3/6, Lint · repo & tooling gates and Type Check · consumer gates in progress. None failed.
    • ⛔ No landing act until every check is green.
  6. Measurement note carried, not filed: for a file whose wall sits in beforeAll, vitest's JSON span reads below the module line CI captures. Later slices should read the module line.
  7. Landing clause. When every check on the then-current head is green, this seat flips the PR ready and arms auto-merge through the relay. It then confirms MERGED and posts a landed-and-release note returning test budget (domain:services): 18 test file(s) over 10 s of wall in the hourly full battery (Part of #23151) #23184 to pm:queue with the assignee cleared.

Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/m skip-changeset PR has no user-facing published change; bypasses the changeset gate tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants