Skip to content

Leaked kv.list() objects can cause confusing database deadlocks #233

Description

@thoughtpolice

I have a system I'm building on celld that uses Workflows extensively, which ran into this (a flaw in the KV layer). The below MWE is from Fable 5.1/Opus 5.5, the diagnosis is rewritten in my own prose.

This has been tested against celld 0.6.0 but it also exists in earlier versions, too.


ctx.storage.kv.list() gives you a key/value iterator, backed by a SQLite prepared statement object. At some point in time, the LTX replicator will checkpoint the database backing the iterator, after which the statement object becomes invalid. However, this then breaks everything and causes the DO itself to become unusable; all kv.put() or storage calls or SQL statement calls after this point will fail. The only way this can be undone is by opening a new kv.list() iterator in order to finalize/collect the old one so it can go away.

Workflows can hit this quite reliably. step.waitForEvent returns from inside a kv.list() loop when it consumes an event, so a Workflow that waits for events and writes about 4 MB afterwards starts failing status() and sendEvent() with storage.transaction: database is locked.


MWE: wrangler.jsonc & index.js

{
  "name": "kv-list-cursor-lock",
  "main": "index.js",
  "no_bundle": true,
  "compatibility_date": "2026-08-20",
  "durable_objects": { "bindings": [{ "name": "REPRO", "class_name": "Repro" }] },
  "migrations": [{ "tag": "v1", "new_sqlite_classes": ["Repro"] }]
}
import { DurableObject } from "cloudflare:workers";

export class Repro extends DurableObject {
  leak() {
    const kv = this.ctx.storage.kv;
    kv.put("a", 1);
    kv.put("b", 2);

    // leak the cursor, then commit about ~3 MiB, going past the 1000-page threshold & forcing a checkpoint
    for (const _ of kv.list()) break;
    for (let i = 0; i < 200; i++) kv.put("blob" + i, "x".repeat(16384));
  }
  put() {
    try {
      this.ctx.storage.kv.put("c", 3);
      return "ok";
    } catch (e) {
      return e.message;
    }
  }
  drain() {
    for (const _ of this.ctx.storage.kv.list());
  }
}

export default {
  async fetch(request, env) {
    const repro = env.REPRO.getByName(new URL(request.url).pathname);
    await repro.leak();
    const before = await repro.put(); // this will fail with "database is locked"
    await repro.drain(); // a new kv.list finalizes the old cursor and removes it
    const after = await repro.put(); // this will succeed
    return Response.json({ before, after });
  },
};

Actual output

$ celld dev . --host 127.0.0.1 --port 8787 --no-watch &
$ curl http://127.0.0.1:8787/run1
{"before":"storage.put: database is locked","after":"ok"}

Expected output

I expect both before and after to return ok and not violate any other consistency guarantees, etc.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions