Repository navigation
Description
Summary
pgmq's insert notifications are implemented by a trigger (pgmq.notify_queue_listeners()). On every inserted row it runs:
UPDATE pgmq.notify_insert_throttle
SET last_notified_at = clock_timestamp()
WHERE queue_name = … AND (throttle_interval_ms = 0 OR clock_timestamp() - last_notified_at >= throttle);There is one row per queue. Once a transaction has updated it, the row stays locked until that transaction ends, and every other transaction inserting into the same queue blocks on it. Even when the throttle window has not elapsed, the UPDATE must wait for the lock holder in order to re-check the row. As a result:
- Producers to one queue are serialized for the remainder of the first producer's transaction.
- Deadlocks: two transactions publishing to two queues in opposite order lock the two throttle rows crosswise.
pgmq.send_topic()orders queues within one call, butthesis_message_bus.pgmq_dispatch_v1()sends all envelopes of a unit of work in handler order, so the order across transactions is arbitrary.
Observed
- All
deadlock detectederrors in our consumer logs (6, then 5 after the other fixes) are inUPDATE pgmq.notify_insert_throttlecalled fromnotify_queue_listeners()←pgmq.send/pgmq.send_topic←thesis_message_bus.pgmq_dispatch_v1. They are retried successfully, but they cost retries and get more frequent as throughput grows. - Sampling
pg_stat_activityduring a load test: 26 of 36 lock waits wereselect thesis_message_bus.pgmq_dispatch_v1(…)waiting ontransactionidbehind this row.
Reproduction (deterministic)
Runs in the thesis/pgmq dev stack: docker compose run --rm php php var/repro-throttle.php
declare(strict_types=1);
require __DIR__ . '/../vendor/autoload.php';
use Amp\Postgres\PostgresConfig;
use Amp\Postgres\PostgresConnectionPool;
use Amp\Postgres\PostgresLink;
use function Amp\async;
use function Amp\delay;
$pg = new PostgresConnectionPool(PostgresConfig::fromString('host=pgmq user=postgres password=postgres'));
$pg->query('create extension if not exists pgmq');
foreach (['repro_a', 'repro_b'] as $queue) {
$pg->query("select pgmq.drop_queue(queue_name) from pgmq.list_queues() where queue_name = '{$queue}'");
$pg->query("select pgmq.create('{$queue}')");
$pg->query("select pgmq.enable_notify_insert('{$queue}', 30)");
}
$send = static fn(PostgresLink $tx, string $queue) => $tx->query("select pgmq.send('{$queue}', '{}'::jsonb)");
// 1. Two producers sending to the SAME queue: the second one waits until the first one commits.
$first = $pg->beginTransaction();
$send($first, 'repro_a');
$startedAt = microtime(true);
$second = async(static function () use ($pg, $send): void {
$tx = $pg->beginTransaction();
$send($tx, 'repro_a');
$tx->commit();
});
delay(1); // the first producer keeps working inside its transaction
$first->commit();
$second->await();
printf("1. second producer waited %.2f s - exactly until the first one committed\n", microtime(true) - $startedAt);
delay(0.1); // let the 30 ms throttle elapse so the next inserts update the rows again
// 2. Two producers sending to TWO queues in opposite order (e.g. two handlers publishing events).
$a = $pg->beginTransaction();
$b = $pg->beginTransaction();
$send($a, 'repro_a');
$send($b, 'repro_b');
$aSendsB = async(static fn() => $send($a, 'repro_b')); // waits for B's throttle row
delay(0.1);
$bSendsA = async(static fn() => $send($b, 'repro_a')); // waits for A's throttle row -> cycle
foreach (['A -> repro_b' => $aSendsB, 'B -> repro_a' => $bSendsA] as $name => $future) {
try {
$future->await();
echo "2. {$name}: ok\n";
} catch (\Throwable $e) {
echo "2. {$name}: ", strtok($e->getMessage(), "\n"), "\n";
}
}
foreach ([$a, $b] as $tx) {
if ($tx->isActive()) {
$tx->rollback();
}
}
foreach (['repro_a', 'repro_b'] as $queue) {
$pg->query("select pgmq.drop_queue('{$queue}')");
}Output (pgmq 1.11.1, the image from thesis/pgmq's compose):
1. second producer waited 1.02 s - exactly until the first one committed
2. A -> repro_b: ERROR: deadlock detected
2. B -> repro_a: ok
Related: no trailing notification inside the throttle window
The throttle notifies on the leading edge only. When several separate transactions insert into one queue within 30 ms, only the first insert notifies. If a consumer wakes up and drains the queue before the later transactions commit, nothing wakes it again: the last messages wait for the poll timer (pollInterval, 5 s by default).
This is exactly what happens when many handlers dispatch to the same queue, each in its own unit of work. Measured with a stress harness: a backlog of 500 plus 300 more messages sent in bursts of 10 single-message transactions, 2 workers, pollInterval 10 s. All 800 arrive, but in every run the last 2–4 messages of the final burst arrive only after the 10 s timer, with both thesis/message-bus and thesis/pgmq's own consumer. With the throttle interval set to 0, all arrive within 5 s.
NOTIFY from pgmq_dispatch_v1() (one per committing transaction and queue, no throttle) removes this gap as well.
Proposal
The trigger is pgmq's design, but thesis/message-bus can avoid it, because every message it sends goes through its own thesis_message_bus.pgmq_dispatch_v1():
- in
pgmq_dispatch_v1(), callpg_notify('pgmq.q_' || queue || '.INSERT', NULL)for every queue a message was written to. Forsend, that is the address. Forpublish, it is the same setpgmq.send_topic()routes to (SELECT DISTINCT queue_name FROM pgmq.topic_bindings WHERE routing_key ~ compiled_regex); - do not use pgmq's insert trigger:
createQueue()disables it instead of enabling it, and the consumer keeps listening on the same channel name.
Why this is better:
NOTIFYis transactional (delivered on commit, nothing on rollback) and takes no row lock while the transaction runs.- Postgres folds duplicate notifications within one transaction, so a batch of 1000 messages to one queue produces one notification.
- The consumer-side throttle becomes unnecessary: a consumer wakes once per committed transaction rather than once per row.
Trade-offs to decide:
- Messages inserted by other means (raw
pgmq.send) would be picked up by the poll timer only. - Delayed messages still need the timer, as today.
- At commit, notifying transactions take Postgres' global notify-queue lock briefly. That is cheap at this scale, but it is a global point.
We are happy to send a PR if this direction is acceptable.
Activity
iGrog commentedon Oct 7, 2026
Added a related finding to the description: the throttle notifies on the leading edge only. When separate transactions insert into one queue within 30 ms, the last of them get no notification. If a consumer has already drained the queue, they wait for the poll timer. Measured with a stress harness: the last 2–4 messages of a burst of single-message transactions arrive only after
pollInterval, with boththesis/message-busandthesis/pgmq; with a throttle of 0 they arrive at once. NOTIFY frompgmq_dispatch_v1()would close this gap as well.iGrog commentedon Nov 9, 2026
Quick follow-up: we've now hit this deadlock in a second service built on the bus, so it's no longer something you only see under a load test.
I've implemented the proposal from the description in #9.
What changes
pgmq_dispatch_v1()callspg_notify()itself, on the same channel the trigger used (pgmq.q_<queue>.INSERT), so listeners stay as they are. Forpublishit notifies every queuesend_topic()routed the message to.createQueue()undoes an earlier setup without stalling a live queue. It deletes the throttle row first (without it the trigger locks nothing), then drops the trigger with a 2 slock_timeout. If the queue is busy, the trigger is left for the next setup and a warning is logged. When there's nothing to undo, it only reads the catalog.Trade-offs
pgmq.send(), bypassing the bus, no longer wakes the consumer and waits for the next poll. If that's a real use case, the trigger could stay as an opt-in.This replaces the trigger handling from #7, so I'll close that one. Unlike #7, it doesn't depend on a new
thesis/pgmqrelease.Tests are in #10, on top of the integration-test setup in #8: two producers on one queue don't wait for each other, crosswise sends to two queues don't deadlock,
publishwakes the subscribed consumer only after commit, andcreateQueue()removes an old trigger without stalling a busy queue. On the current 0.5.x the first two fail with a lock wait anddeadlock detected.