Skip to content

objectql: the lifecycle sweep runs 60 s after every kernel boot, even on a database created seconds earlier, so a fresh tenant kernel pays 130 statements in its first minute #29918

Open

Description

@objectstack-fleet

repo:cloud#2 · os-zhuang · session_01QUs3BtyTpTkZhBZdWY1Xun

Filing gate ①: a product defect with a named locus and a measured reach. Left over from cloud#3413 after objectstack#28383 (PR #28403) landed; this is item 5 of #28383, which #28403 put out of scope.

维护者速读

一个全新环境第一次打开,内核启动 60 秒后就会跑一遍 objectql 生命周期清扫(保留期读取、表计数和悬空引用审计),在 cloud 的托管内核上是 130 条语句。可这时数据库才建好一分钟,没有任何东西需要清理。cloud 的免费环境每次访问只驻留 30 分钟,清扫间隔却是一小时,所以每次访问都付一遍首轮清扫。它现在是空闲内核第一分钟 160 条里的 130 条,cloud#3413 的目标是第一分钟不超过 10 条。

Reach (measured)

Measured on cloud's hosted tenant composition (ArtifactKernelFactory with HOSTED_FORCED_REQUIRES, production TursoDriver in remote mode over a local libSQL file, every statement counted at the client), framework pin 118bee4d, with scripts/dev-local/bootstrap-curve.mjs --hosted --objects 0 --first-boot --hold 1800 --hold-slice 60 --hold-shapes in objectstack-ai/cloud:

Run First minute What
default 160 130 in the 50–60 s slice (this card), 20 delivery-dispatcher ramp, 10 DbQueueAdapter ramp
OS_LIFECYCLE_DISABLED=1, same flags, --hold 120 30 the two ramps only

A 10-second slicing (--hold 120 --hold-slice 10) puts all 130 in one burst at the sweep's initial delay (DEFAULT_LIFECYCLE_INITIAL_DELAY_MS, 60 s). By shape: 17 retention reads (SELECT * FROM sys_… WHERE … < ?), 16 SELECT COUNT(*), the read-only dangling-reference audit over every system table with lookups, and a handful of settings reads on the same clock. At pin 99a6a781 (2026-10-31) the same burst was 94; it grows with the system objects the audit walks.

The sweep interval is an hour (DEFAULT_LIFECYCLE_SWEEP_MS); cloud evicts a free kernel after 30 idle minutes (OS_KERNEL_FREE_IDLE_MS), so a hosted kernel only ever runs the first sweep, and runs it on every boot. On a 2,000-organization launch day with each environment opened once, that is 0.26 million statements of the 0.61 million an idle visit now costs in all.

Expected

A sweep runs when the data can need one, not once per process start:

  1. Persist a last-swept watermark (one row, or a sys_ setting) and, at the initial delay, read it and sweep only when now − watermark ≥ sweepIntervalMs. A rebuilt or re-booted kernel then pays one read, not 130.
  2. Seed the watermark when the tables are created. A database whose system tables were created by this boot has no row older than any retention window, so the first sweep is due one interval later. Without this step item 1 alone does not help the first boot, which is the case measured here.

Cloud could instead push initialDelayMs past the 30-minute residency, but then a tenant that visits briefly and often would never be swept, so the fix belongs here.

Done when

The same reading at the new pin shows no lifecycle-sweep statements in a fresh kernel's first minute (at most the watermark read), a kernel whose watermark is older than the interval still sweeps, and the lifecycle tests still pass. cloud then re-measures under cloud#3413.

Business reason

  • Who benefits: every hosted environment's database reads (Turso quota and bill) and runtime CPU on launch day; self-hosted deployments that restart often the same.
  • If not done: the first minute of every visit stays above 100 statements whatever the dispatchers cost, and cloud#3413's target (at most about 10 a minute) cannot be met.

Reader who acts

objectstack triage routes it to the lane owning packages/objectql/src/lifecycle/ (lifecycle-service.ts, and the audit in integrity/dangling-reference-audit.ts that rides its clock).

Dedup

Searched objectstack issues, open and closed: "lifecycle sweep initial delay" (6 hits) and "lifecycle sweep watermark" (7 hits). The only relevant one is #28383 (closed), whose item 5 named this and whose PR #28403 left it out of scope. The sibling dispatcher-ramp card is filed alongside this one.

Activity

  1. added
    bugSomething isn't working
    domain:engineRetired 2026-10-22, alias of domain:server (engine surface); in-flight cards keep it until they land
    on Nov 3, 2026
  2. objectstack-fleet commented on Nov 3, 2026

    @objectstack-fleet
    ContributorAuthor

    triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U

    Path: none

    Triage: first-touch audit · p2 kept · domain:engine → domain:server · pm:queue · 2026-11-03T12:54Z

    ⛔ Not a claim.

    • Read fresh. Measured on cloud's hosted composition: the objectql lifecycle sweep runs 60 s after every boot, even on a database created seconds earlier. It costs 130 of the idle kernel's 160 first-minute statements, against cloud#3413's target of 10. p2 kept.
    • domain:engine is retired (an alias of domain:server) and this is a new card, so it gets the live label. It is pure objectstack work that the cloud PM split out, stating the cloud facts, so an objectstack seat takes it.

    Generated by Claude Code

  3. added
    domain:serverDomain lane — seat card indexed by label:pm:seat
    and removed
    domain:engineRetired 2026-10-22, alias of domain:server (engine surface); in-flight cards keep it until they land
    on Nov 3, 2026
  4. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    domain:server#4 · os-bill · session_01Utp6CZMkHvuDeVYgTD73qF

    Claim: PM loop round 1
    Session: session_01Utp6CZMkHvuDeVYgTD73qF (seat) · dev card thread session_01Utp6Ca4cwEoenneeTMpCvy
    Account: os-bill
    Branch: claude/project-thread-u71k2j (the card thread's designated push branch, the recorded platform exception to claude/issue-<n>-<slug>)
    Worktree: objectstack-issue-29918
    Domain: domain:server
    Seat: domain:server#4
    File surface: packages/objectql/src/lifecycle/** (lifecycle-service.ts, lifecycle-settings.ts, their tests), the gate on packages/objectql/src/integrity/dangling-reference-audit.ts only where it rides the same clock, one changeset. ⛔ packages/spec/**, packages/platform-objects/** (a new sys_* object is a fork), packages/core/src/dispatch-loop.ts, service-messaging, service-queue (sibling #29919), docs/adr/**, skills/**, .claude/**, AGENTS.md (stop on breach; explain in the report)
    Container & model: M, mode:thread, model: default judgment tier (opus) — dispatch-gates --tier: no path-derived mandate, effort: high
    Clause-②: yes (widening) — the exported LifecycleEngineLike gains the optional member wasDatastoreCreatedFromEmpty?(): boolean (an engine without it keeps the old schedule); no accept set changes. Precedent PR #30015. Amended by this seat at review; the earlier no is superseded.
    Metadata-protocol: none
    Responsibility: objectql's LifecycleService schedules the first sweep at a fixed 60 s after every boot (DEFAULT_LIFECYCLE_INITIAL_DELAY_MS) with no persisted last-swept reading | none: OS_LIFECYCLE_DISABLED only turns the sweep off, and a larger initialDelayMs stops brief, frequent tenants from ever being swept (card) | every kernel boot reaches it; measured by cloud on hosted tenant kernels (130 of 160 first-minute statements), self-hosted restarts likewise
    Thread-read: 6604322143
    Premise: holds
    Serial constraints cleared: none — the lane's in-flight claims (#30108 plugin-security, #30106 cli/runtime packages, #30009 trigger-api, #29997 plugin-auth, #30062 metadata-protocol) do not touch packages/objectql/src/lifecycle/ or the integrity audit; no open PR touches them; sibling #29919 is a separate surface and unclaimed.

    Dispatch order: /mnt/project-files/server-seat4/dispatch-29918.md (project shared folder). Ruling section: no maintainer ruling on this card; triage first-touch audit 6604322143. Premise re-read on origin/main b7075d99: DEFAULT_LIFECYCLE_INITIAL_DELAY_MS = 60_000, DEFAULT_LIFECYCLE_SWEEP_MS = 3_600_000, no watermark read under packages/objectql/src/lifecycle/.

    Posted 2026-11-05T00:58Z.

  5. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    domain:server#4 · os-bill · session_01Utp6CZMkHvuDeVYgTD73qF
    os-dev-report

    Status: needs_decision — the dispatch's fork condition is reached before any code. Item 2 has an existing signal; item 1 (the persisted watermark) has no existing persistence seam that needs no new public config key and no new sys_* object. No branch commits, no PR.

    Readings on objectstack-ai/objectstack b7075d99:

    • packages/objectql/src/lifecycle/lifecycle-service.ts:81,85: DEFAULT_LIFECYCLE_SWEEP_MS = 3_600_000, DEFAULT_LIFECYCLE_INITIAL_DELAY_MS = 60_000; start() (507-521) arms a bare setTimeout(initial) then setInterval(interval). git grep -ciE 'watermark|last_swept|lastSwept' -- packages/objectql/src/lifecycle/ = 0 hits (exit 1); control word DEFAULT_LIFECYCLE_INITIAL_DELAY_MS = 3 hits in the same tree. Premise holds.
    • The dangling-reference audit rides the same clock with no timer of its own: sweep() calls auditReferences() at line 808, and the only other caller of inspectDanglingReferences is the CLI (packages/cli/src/utils/orphan-detail-sweep.ts:185). A gate in front of sweep()'s scheduled call therefore gates the audit, the governance settings reads (loadGovernance, 875-898) and the reaps together.
    • The settings seam is read-only to this service: LifecycleSettingsLike (158-164) declares get only. SettingsService.setMany refuses an undeclared key (packages/services/service-settings/src/settings-service.ts:2382, UnknownKeyError, SETTINGS_UNKNOWN_KEY), and the lifecycle manifest (lifecycle-settings.ts) declares five operator keys and no state key. A watermark there is a new manifest specifier, i.e. a new config key (settings REST, the Setup "Data Lifecycle" panel, and an OS_LIFECYCLE_* env override that locks it). The spec's only visibility control is a visible expression, not an internal-key flag.
    • sys_setting / sys_platform_setting are managedBy: 'engine-owned' with SettingsService.setMany as the only writer, so a direct engine write beside the resolver is not an existing seam either.
    • sys_migration (DATA_MIGRATION_FLAG_OBJECT) is the ADR-0104 data-migration ledger; a non-migration row there would be read by every migration gate. sys_job.last_run_at exists only where service-job is composed; moving the sweep onto the job service is a re-architecture, not this card.
    • Item 2 needs no new surface: ObjectQL.wasDatastoreCreatedFromEmpty() (packages/objectql/src/engine.ts:5867, body in engine-migration-flags.ts:281-295) answers from every driver's getSchemaSyncStats() (Turso remote face included, turso-driver.ts:4309), in process, with zero statements. LifecycleEngineLike would gain it as an optional member.
    {
      "issue": 29918,
      "status": "needs_decision",
      "branch": "claude/project-thread-u71k2j",
      "pr": null,
      "session": "session_01Utp6Ca4cwEoenneeTMpCvy",
      "premise_still_valid": true,
      "summary": "Premise holds at b7075d99: fixed 60 s first sweep, no watermark, and the reference audit rides sweep() with no timer of its own. Item 2 (seed on create) has an existing zero-statement signal, ObjectQL.wasDatastoreCreatedFromEmpty(). Item 1 (persist a last-swept watermark) has no existing read-and-write seam: the lifecycle settings seam is get-only, SettingsService refuses undeclared keys, and the remaining candidates are a ledger with another meaning or a service that is not always composed. Stopped at the fork condition with no code written.",
      "tests": "none run: the fork condition was reached before any code; readings only (commands and counts above)",
      "gates": "none: no diff",
      "dogfood_battery": "n/a: no PR",
      "mcp_calls": "0",
      "api_writes": "1: this os-dev-report comment on #29918 through post-stamped (relay)",
      "open_questions": [
        {
          "question": "Where does the last-swept watermark (card Expected item 1) persist?",
          "options": [
            "A: a global-scope key in the lifecycle settings manifest (lifecycle-settings.ts), written by the service after each completed sweep through an optional set on LifecycleSettingsLike. Cost: a new operator-visible config key and env override (pinning it to a future instant silently stops every sweep), one sys_platform_setting upsert plus one sys_platform_setting_audit row per sweep, which re-feeds an audit table against ADR-0057 3.3's one-log-line trace. Kernels without a settings service keep today's schedule. Size M.",
            "B: a one-row engine-owned state object (for example sys_lifecycle_state) in packages/platform-objects, read once at the initial delay and upserted after each sweep. Cost: a new sys_* object on every deployment, with its translations and registry rows; no operator knob, no audit row. Size M.",
            "C: item 2 only, in process: at the initial delay, when wasDatastoreCreatedFromEmpty() is true, skip the first sweep so it runs one interval later. Zero new surface and zero statements. Meets Done-when clause 1 for the measured first-boot reading; does not meet clause 2, because a re-boot on an existing store still sweeps at 60 s. The PR says Part of #29918 and item 1 stays open. Size S."
          ],
          "recommendation": "C now as a Part-of PR, because it removes the measured 130 first-boot statements with no new surface and needs no decision. Then B for item 1, because the watermark is internal state, not an operator setting: under A an env or settings value can disable sweeps with nothing in any log, and every sweep writes an audit row. The dispatch did not carry the decision-axes framework, so this recommendation is not graded against those axes; the seat supplies them if wanted."
        }
      ],
      "security_findings": "none",
      "out_of_scope_findings": []
    }

    Report taken 2026-11-05T01:02Z.


    Generated by Claude Code

  6. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    domain:server#4 · os-bill · session_01Utp6CZMkHvuDeVYgTD73qF

    Seat note · 2026-11-05T01:04Z · the dev's needs_decision (6646684881) splits this card in two.

    • Item 2 (seed on create) proceeds now, at the seat's discretion (task order, no new surface, no product contract change): a store this process created from empty (ObjectQL.wasDatastoreCreatedFromEmpty(), measured at engine-migration-flags.ts) skips the first sweep and the dangling-reference audit that rides sweep(), so the first sweep is due one interval later. Same claim, same branch. The PR carries Refs #29918 (item 2), ⛔ not Fixes.
    • Item 1 (persist a last-swept watermark) goes to a decision. Measured: no existing read/write seam (LifecycleSettingsLike is get-only; SettingsService refuses undeclared keys with UnknownKeyError). Options on the table: A, a declared last_swept_at lifecycle setting (a new public config key with an env override); B, a new single-row sys_* state object. Both add surface, so neither is the seat's to pick. Analysis and the maintainer's ruling will be posted here.
    • After item 2 lands, this card leaves pm:dispatched in one stroke with a Release: line naming item 2 as landed and item 1 as the remainder, into the decision state.
  7. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    domain:server#4 · os-bill · session_01Utp6CZMkHvuDeVYgTD73qF
    os-dev-report

    Status: done for item 2 under the seat's note 6646713437 (option C). Draft PR #30214, head 07cef192, Refs #29918 (item 2), no closing keyword. Item 1 stays with the decision.

    {
      "issue": 29918,
      "status": "done",
      "branch": "claude/project-thread-u71k2j",
      "pr": "https://github.com/objectstack-ai/objectstack/pull/30214",
      "session": "session_01Utp6Ca4cwEoenneeTMpCvy",
      "premise_still_valid": true,
      "summary": "Item 2 only: LifecycleService.start() asks, at the initial delay, whether this process created the datastore from empty (ObjectQL.wasDatastoreCreatedFromEmpty(), answered from driver schema-sync stats with no statement); when it did, the scheduled first sweep is skipped and the sweep runs on the interval, one interval later. The reference audit is a leg of sweep() with no timer of its own, so it is deferred with that tick. LifecycleEngineLike gains the optional member; an engine without it, or whose answer throws, keeps the old schedule. Direct sweep() calls always sweep. Item 1 (persisted watermark) is not done and waits on the decision.",
      "tests": "head 07cef192: vitest src/lifecycle/ Test Files 5 passed (5), Tests 135 passed (135); pnpm --filter @objectstack/objectql test Test Files 548 passed (548), Tests 10867 passed (10867), lock VERDICT command-exit 0; objectql typecheck (tsc --noEmit and check:test-typecheck) VERDICT command-exit 0. Ablation on committed 027c6689 via scripts/ablation-replace.mjs: gate line replaced by an unconditional sweep (anchor 1 to 0, blob 026c59ebf9a8 to f36281d0bba1), -t 29918 gave 1 failed (the fresh-store negative pin) and 3 passed; restore blob equals HEAD and git diff HEAD is empty. The suite imports the service from source, so no build sits between mutation and run.",
      "gates": "head 07cef192, dispatch-gates --commands: 59 families. 58 exit 0, including pnpm lint exit 0. pnpm check:dual-build-cjs-loads exit 3, PREREQUISITE NOT MET (needs the whole workspace built): NOT MEASURED, left to CI. check-plugin-teardown-shape --self-test exit 3 on the shallow clone, exit 0 (48 cases) after fetching 621a4876. --ran: 59 derived families accounted for, 58 run, 1 NOT-MEASURED. PR check-runs: in_progress at report time. Records: /mnt/project-files/server-seat4/gates-29918-ran.txt",
      "dogfood_battery": "n/a: no bang in the PR title and no (narrowing) arm",
      "mcp_calls": "0",
      "api_writes": "4: os-dev-report comment 6646684881 (needs_decision, relay), git push (not REST), pr_create #30214 (relay), this os-dev-report comment (relay)",
      "open_questions": [],
      "security_findings": "none",
      "out_of_scope_findings": []
    }

    Deviations: the seat's note placed the gate at the entry of sweep(). It sits on the scheduler's first tick instead, because a gate at the entry would stop the direct sweep() callers (the dogfood growth gate, db:clean-style tooling, and more than a dozen suites in objectql, service-knowledge, service-queue and service-messaging) from sweeping on a fresh store. The audit is still deferred, since it runs only inside sweep(). The merge commit 07cef192 carries no trailer pair (git's default merge message). The session-context assignee and reviewer step on the PR was skipped under os-dev rule 2 and the order's no-MCP-write line; the seat sets them at hand-over.

    Report taken 2026-11-05T02:10Z.


    Generated by Claude Code

  8. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    domain:server#4 · os-bill · session_01Utp6CZMkHvuDeVYgTD73qF

    Release: claim 6646620870 (session_01Utp6CZMkHvuDeVYgTD73qF, domain:server#4) · reason: item 2 landed, item 1 awaits the maintainer's ruling · destination: needs-user-decision (item 1, watermark storage A vs B)
    Resume-from: none (item 2 is merged; item 1 starts fresh from main once ruled)

    Landed · item 2 · PR #30214

    Reading Value
    PR #30214 MERGED 2026-11-05T03:14Z, merge commit 314ad9557, an ancestor of origin/main 6f095b9c
    main source packages/objectql/src/lifecycle/lifecycle-service.ts:529 if (this.firstSweepDueAtInitialDelay()) void this.sweep(); and :559 the gate reading wasDatastoreCreatedFromEmpty
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions