Skip to content

feat(assets): delete asset revisions nothing can still serve, and finish the #327 follow-ups - #345

Draft
oglassdev wants to merge 5 commits into
mainfrom
feat/assets-followups
Draft

oglassdev wants to merge 5 commits into
mainfrom
feat/assets-followups

Conversation

@oglassdev

Copy link
Copy Markdown
Contributor

Closes #328. Since asset revisions landed (#327), nothing is ever deleted. Every pushed revision and blob stays in management's store, and every revision core or a JVM downloaded stays on its disk. This PR adds retention, plus the other #328 follow-ups.

Retention

Management collects on every SetAssetHead, so every push triggers it. A revision is kept while any of these holds:

  • it's the project's head;
  • it's one of the 10 newest READY revisions;
  • an upload protects it: every UploadAssets sets protected_until to now + 1 h, whether the revision is UPLOADING or READY;
  • an ACTIVE, PENDING or IN_PROGRESS deployment pins it, or one of the environment's 10 newest deployments does (so rollback works);
  • a SUPERSEDED deployment that a delayed ACTIVE report could still activate pins it;
  • a deployment core last reported as resident in its backend pins it. Core now sends resident_deployment_ids and resident_deployments_known on ReportStatus. Until a report has actually read that set, which includes every new lease, all of the environment's deployments count as resident.

Concurrency:

  • Lock: a per-project advisory lock is held exclusively by collection, and shared by upload, completion, head moves, deployment creation and activation.
  • Two-phase delete: collection first deletes the rows and records the orphaned blobs in asset_deletions. In a second locked transaction it deletes those blobs from the store, skipping any a new upload references again. A failed delete stays recorded and is retried on the next collection.

Core prunes revisions, apps/<rev> directories and blobs that no resident or loading deployment pins. chunk-jvm keeps only the revision its launch runs.

Other follow-ups

  • Dashboard: deployment rows and the rollback dialog show their asset revision. A project panel lists revisions with head marker, size and age. The protobuf TS was regenerated; it was missing assets_pb.ts.
  • Downloads: core, chunk-jvm and chunk assets pull fetch 4 blobs at once. Size and SHA-256 checks and staging cleanup are unchanged.
  • chunk assets push: it builds the revision directly, without Gradle.
  • chunk dev: new --pack-bind and --pack-url options. --pack-url is required when packs bind to all addresses.

Known limits

  • Rollback window: rolling back to a deployment older than the 10 newest fails once its revision is collected. Restoring a fork whose deployment's revision is gone fails the same way.
  • Store delete race: a store delete cut off by a management crash or a lost database connection can still land after the same content was re-uploaded. Pushing again repairs it; the README says so.
  • Collection timing: collection only runs on SetAssetHead, and the limits are constants.

Checks

  • Management: assets, deployments and environments tests against Postgres 17 (31). Migrations 0004 and 0005 were also applied to databases with existing rows.
  • Typecheck: management and dashboard are clean.
  • Dashboard tests: 8.
  • Rust: cargo test on chunk-build, chunk-jvm, chunk-environment and chunk-cli, plus clippy and fmt.
  • Proto: buf lint proto and buf format.
  • Lint and format: oxfmt and oxlint on touched files.
  • Review: five rounds. The last one found nothing.

🤖 Generated with Claude Code

…ush without Gradle and serve dev packs off localhost

- management: each SetAssetHead deletes the project's revisions that are not the
  head, one of its 10 newest ready revisions, under an hour old, or pinned by an
  environment's active, unfinished or 10 newest deployments, then every blob no
  remaining revision holds from the database and the release store. A
  per-project advisory lock, shared by uploads and deployments, keeps
  collection from racing them. ReleaseStore gains delete.
- core removes asset revisions and blobs no resident or loading deployment
  pins; chunk-jvm keeps only the revision it launches.
- core, chunk-jvm and chunk assets pull download four blobs at once.
- chunk assets push builds only the assets revision, without Gradle.
- chunk dev takes --pack-bind and --pack-url for the pack server.
- dashboard: deployments show their asset revision, and projects list their
  revisions.

Closes #328
…o phases and refresh protection on repeated uploads

- management: collection also keeps revisions pinned by a superseded deployment
  within its environment's drain deadline.
- SetAssetHead checks the revision and moves the head under the shared asset
  lock, in one transaction.
- Collection deletes rows and records blobs in asset_deletions in one
  transaction, then deletes the store objects of unreferenced keys in a second,
  retrying earlier failures.
- uploadAssets sets protected_until on the revision it declares again, ready or
  not, replacing the create_time rule.
- core records the kept set only after a prune fully succeeds, and invalidates it
  whenever a load finishes.

Refs #328
…evisions for the deadline they were drained under, and count loads once their inserts finish

- completeAssetUpload runs in one transaction holding the shared project lock, so collection can't delete the
  revision or its blobs between verification and the READY transition. eachLimited now waits for every started run
  before it rejects.
- A deployment records drain_until when it is superseded; collection keeps its revision until the later of that and
  update_time plus the environment's current deadline, so shortening the setting removes nothing early.
- Migration 0004 protects existing revisions for an hour, and 0005 backfills drain_until.
- core's asset load accounting waits for the blocking inserts a cancelled or failed load left running.
- management README notes the remaining window of a store delete outliving its lock.

Refs #328
…set, wait out store deletes and register blobs in digest order

- core reports every deployment resident in its backend, including one loading, in
  ReportStatusRequest.resident_deployment_ids; management stores the set of each
  accepted report on the environment.
- collection keeps the revisions of superseded deployments that can still be
  activated and of deployments in the reported resident set, replacing the drain
  deadline rules and drain_until (migration 0005 now adds the resident set).
- activateDeployment takes the shared asset lock, so activation and collection
  don't interleave.
- the S3 store's delete waits for both of its deletes to settle before it fails.
- completions verify blobs concurrently and register them in one insert sorted by
  digest, so concurrent completions can't deadlock.

Refs #328
…esident set

An unknown resident set looked like an empty one: a fresh core whose first read timed out reported nothing, and the next collection deleted the revision of a draining deployment.

- ReportStatusRequest.resident_deployments_known is true only when the observation read the resident set. Core no longer repeats a cached set.
- Management stores the IDs of reports that read them and records the flag on the environment, resetting it on each new lease.
- Collection keeps the revisions of all of an environment's deployments while the flag is false.

Refs #328

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

assets: follow-ups after asset revisions (#327)

1 participant