Partition the Telegram outbox and LLM request events by day

[iamwavecut]

Summary

Retention of llm_request_events deleted raw rows in batches, and the Telegram outbox had no retention at all (about a million delivered rows since July). Both tables now use daily partitions, and retention drops whole days.

llm_request_events

• Parent PARTITION BY LIST (is_rollup): llm_request_events_rollup holds the long-lived rollups; llm_request_events_raw holds raw events, PARTITION BY RANGE (created_at) with one partition per UTC day.
• The cleanup worker keeps its order: roll up raw events older than the retention cutoff, then drop the raw partitions before the cutoff day (drop_daily_partitions_before). The batched DELETE is gone.
• The rollup's ON CONFLICT DO NOTHING uses the unique rollup key on the rollup partition. The primary key on id becomes a plain index; ids still come from the same sequence.

telegram_outbox and telegram_outbox_attempts

• Daily RANGE partitions on created_at / claimed_at.
• A partitioned table cannot keep UNIQUE (operation_id) or UNIQUE (batch_id, part_index), so enqueue changes:
  • It takes pg_advisory_xact_lock on the batch id.
  • It reuses existing parts whose immutable fields match.
  • It inserts only the missing parts, in part order (claiming orders a chat's sends by id).
  • A mismatching existing part still yields TelegramOutboxIdempotencyConflict.
• Attempts lose the foreign key to the outbox (it cannot reference a partitioned parent); their partitions age out on the same schedule.

Partition maintenance

• New hourly worker daily-partition-maintenance keeps partitions for today plus 7 days for all three partitioned tables and drops outbox and attempt days older than TELEGRAM_OUTBOX_RETENTION_DAYS (default 7; 0 keeps history). An outbox day that still has pending, leased or retry_wait rows is kept.
• Partition DDL runs with lock_timeout = 3s and retries on the next tick, so it never queues behind a long reader.

Migrations

• 204: ensure_daily_partition(parent, day) and drop_daily_partitions_before(parent, cutoff_day, keep_when).
• 205: rename, create the partitioned table and partitions from the oldest raw day to today + 7, copy all rows, re-own the sequence, drop the old table, rebuild indexes. Down converts back to one table with the original keys and indexes.
• 206: same for the outbox and attempts, copying only the last 7 days and every in-flight row (with their attempts). Down converts back with the original keys and the cascading foreign key.
• One-way for data: 206 drops outbox history older than 7 days, and dropped partitions are not restored by down migrations.

Deployment risk and verification plan

• Risk: migrations 205/206 copy about 1.4 GB of LLM events and about 0.2 GB of outbox rows inside the startup migration, so the bot starts later than usual (expected a few minutes; the deploy waits up to 10). The old tables are dropped afterwards, freeing about 8 GB.
• Risk: an insert whose timestamp has no partition fails. Partitions exist 7 days ahead and the worker refills them hourly; event and outbox timestamps are the current time.
• Plan after deploy: confirm migrations 204–206 and the partition layout; check dialog replies are delivered and telegram_outbox claims progress; check LLM events are written to today's raw partition; read the daily_partition_maintenance readiness entry; check table sizes.

Verification

• cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace.
• Live Postgres (pgvector/pg17), serially: all live_ storage tests and the outbox round trip (idempotency, ordering, fencing, recovery-scan index use) pass on the partitioned schema; app maintenance live tests pass.
• New live tests: a second enqueue of a batch waits for an uncommitted first enqueue and then reuses its parts (this test fails when the advisory lock is removed); partitions are created ahead and an expired outbox day with an in-flight row is kept.
• The previously ignored LLM retention test (run against an empty database) checks rollups are kept, recent and cutoff-day events survive, an archive failure drops nothing, and a retry drops the recovered day.
• Migrations 204–206 applied to a database seeded with old and recent raw events, rollups, delivered, old and in-flight outbox rows; then down 206→204 and up again.

[github-actions]

	Messages
📖	Danger checked 14 changed file(s).

Generated by 🚫 dangerJS against 301fba8

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 5 🔵🔵🔵🔵🔵

🔒 No security concerns identified

⚡ Recommended focus areas for review Lost uniqueness enforcement on operation_id Previously the UNIQUE (operation_id) constraint made the database reject any duplicate operation_id, even across different batches. With partitioning, that constraint is gone and the advisory lock only serializes enqueues that share the same batch_id. If two enqueues with different batch_id values ever produce the same operation_id (e.g., a caller reusing operation ids across batch retries with a regenerated batch id), both inserts now succeed and duplicate outbox rows are created silently, where the old code surfaced a conflict. This is only reachable if operation_id is not strictly derived from batch_id + part_index; if that invariant is guaranteed elsewhere, the issue does not apply, but nothing in this diff (no remaining unique index, lock keyed only on batch id) enforces it. const SQL_LOCK_OUTBOX_BATCH: &str = "SELECT pg_advisory_xact_lock(hashtextextended('telegram_outbox:' || $1, 0))"; const SQL_ENQUEUE_OPERATIONS_PREFIX: &str = "WITH input (\ operation_id, batch_id, part_index, bot_id, chat_id, thread_id, ordering_key, \ causation_update_id, dialog_job_id, trigger_message_id, method_kind, payload_version, \ original_payload, payload, blob_id, delivery_policy, protected, priority, available_at, expires_at) AS ("; const SQL_ENQUEUE_OPERATIONS_SUFFIX: &str = "), existing AS ( \ SELECT outbox.id, outbox.operation_id, outbox.part_index, outbox.state, \ (outbox.batch_id = input.batch_id \ AND outbox.part_index = input.part_index \ AND outbox.bot_id = input.bot_id \ AND outbox.chat_id IS NOT DISTINCT FROM input.chat_id \ AND outbox.thread_id IS NOT DISTINCT FROM input.thread_id \ AND outbox.ordering_key = input.ordering_key \ AND outbox.causation_update_id IS NOT DISTINCT FROM input.causation_update_id \ AND outbox.dialog_job_id IS NOT DISTINCT FROM input.dialog_job_id \ AND outbox.trigger_message_id IS NOT DISTINCT FROM input.trigger_message_id \ AND outbox.method_kind = input.method_kind \ AND outbox.payload_version = input.payload_version \ AND outbox.original_payload = input.original_payload \ AND outbox.blob_id IS NOT DISTINCT FROM input.blob_id \ AND outbox.delivery_policy = input.delivery_policy \ AND outbox.protected = input.protected \ AND outbox.priority = input.priority \ AND outbox.expires_at IS NOT DISTINCT FROM input.expires_at) AS same \ FROM input JOIN telegram_outbox AS outbox ON outbox.operation_id = input.operation_id \ ), inserted AS ( \ INSERT INTO telegram_outbox (\ operation_id, batch_id, part_index, bot_id, chat_id, thread_id, ordering_key, \ causation_update_id, dialog_job_id, trigger_message_id, method_kind, payload_version, \ original_payload, payload, blob_id, delivery_policy, protected, priority, available_at, expires_at) \ SELECT operation_id, batch_id, part_index, bot_id, chat_id, thread_id, ordering_key, \ causation_update_id, dialog_job_id, trigger_message_id, method_kind, payload_version, \ original_payload, payload, blob_id, delivery_policy, protected, priority, available_at, expires_at \ FROM input \ WHERE NOT EXISTS (SELECT 1 FROM existing WHERE existing.operation_id = input.operation_id) \ ORDER BY input.part_index \ RETURNING id, operation_id, part_index, state \ ) \ SELECT id, operation_id, part_index, state, FALSE AS inserted FROM existing WHERE same \ UNION ALL \ SELECT id, operation_id, part_index, state, TRUE AS inserted FROM inserted";

[iamwavecut]

PR-Agent review guide, lost uniqueness on operation_id: the invariant holds by construction. Callers cannot supply an operation id: prepare_parts always derives it inside storage as deterministic_telegram_operation_id(batch_id, method_kind, part_index) (SHA-256 of the batch coordinates). Different batch ids therefore never produce the same operation id, and enqueues of the same batch id are serialized by the advisory lock, so a duplicate row can only arise from a SHA-256 collision. The same predicate also compares batch_id, so a part from another batch could never be reused silently.