Workers — Concurrent Event Processing

What workers do

By default, the Processor uses one database connection and processes one event at a time. With --workers N, it opens N connections and spreads the pending list of events across all of them at once, every tick.

For most deployments, one worker is enough. An event on channel A does not block an event on channel B — they simply queue up and are processed one after another. Workers start to matter when:

• Many events arrive per second and one worker cannot keep up.
• Your actions take more than a few milliseconds each (for example, calling an external HTTP service or running a slow query), and you need genuine parallelism to keep throughput up.

How dispatch works

When the Processor fetches the list of pending IDs, it puts all of them into a buffered channel (an internal Go concurrency primitive — not to be confused with a pg_relay channel). Each of the N worker goroutines reads from that channel, calling pgrelay.process_one() on its own dedicated database connection. Once every worker has finished, the Processor checks whether anything was actually processed, and decides whether to drain immediately or wait for the next tick.

Each worker uses one connection exclusively — there is no sharing. This means no locking or serialisation happens inside the Go program itself. pgrelay.process_one() uses _queue_claim() with FOR UPDATE SKIP LOCKED, so if two workers ask for the same row at the same time, exactly one wins and the other gets a skipped result.

Per-channel dispatch concurrency (concurrency_mode)

(v1.1) FOR UPDATE SKIP LOCKED only stops two workers claiming the same row twice. It does nothing to stop two different rows for the same channel — or the same channel and payload — running at the same instant on two different connections. With --workers N > 1 (or more than one Processor instance), that is a real scenario, and some actions must never run twice at once for the same logical thing.

Each channel declares its own policy through pgrelay.actions.concurrency_mode:

Mode	Exclusivity group	Guarantee
concurrent (default)	each row is its own group	none — identical to the original 1.0 dispatch behaviour
channel	lower(channel)	one in-flight event per channel, strictly in queued order
channel_payload	lower(channel) + exact payload	one in-flight event per (channel, payload) pair, strictly in queued order per pair

How it is enforced. The restriction lives inside pgrelay.queue_pending_ids() — the function that decides which rows are even offered to the worker pool in the first place. A window function (a SQL feature for ranking or comparing rows against others in the same group) ranks the eligible rows within each exclusivity group by (run_at, queued_at), and only the top row per group is offered. Two consequences follow from this:

1. No same-group race is possible within a batch — only one row per group is ever in the candidate list, so there is nothing for two workers (or two Processors) to race over.
2. The cross-batch case needs no extra machinery. While a claimed row is still executing, it remains marked as not-yet-dispatched and is still row-locked, so the ranking still selects it as the group's top row on the next poll — and the final FOR UPDATE SKIP LOCKED step fails to lock it, yielding zero candidates for that group. The next row in the group only surfaces once the current one is genuinely marked done.

NULL payloads under channel_payload mode group per channel (NULL is treated as equal to NULL, matching notify()'s own deduplication rules), never across different channels, and a NULL payload is treated as a different group from an empty-string payload. Channel comparison is case-insensitive; payload comparison is exact.

Waiting is silent and free. A row held back because its group is currently occupied was never offered, never claimed, and never attempted: its retry count is untouched, its dispatched time stays unset, and no pgrelay.log row is written for the wait itself. It is simply picked up on a later poll — no different from an ordinary double-claim skip.

Ordering caveats.

• Within a group, execution is strictly oldest-first (run_at, then queued_at) and never concurrent. When a serialised event finishes mid-drain, the group's next event is typically handed out on the very next poll — often within a second — but the actual guarantee is only "on a subsequent poll", so do not design around a guaranteed sub-second hand-off.
• Ordering across different groups (or across different run_order tiers) remains advisory, exactly as in the original design: a later-queued event handed to an idle worker can finish before an earlier-queued event stuck on a busy worker, if the two belong to different groups.
• A retry row carries a future run_at (its backoff delay), so under channel/channel_payload mode, a waiting retry does not block the rest of its group — other events in the group proceed, and the retry rejoins the ordering once its run_at arrives.

Because all of this is enforced in the database at the point candidates are selected, the guarantee holds no matter how many workers or Processor instances you are running — the Processor program itself needed no changes at all for this feature.

When to increase workers

The Performance Testing chapter describes how to measure your own throughput. As a rule of thumb:

• If the test shows a breakeven below 200 events per second with --workers 1, try --workers 2 or --workers 3.
• If your actions are fast SQL (sub-millisecond), the bottleneck is usually network round-trip time, and more workers help.
• If your actions call external services (HTTP, SMTP), workers let those calls happen in parallel.
• There is diminishing return beyond 4–5 workers for most workloads — the overhead of managing more connections eventually outweighs the extra parallelism.

Multiple Processors vs multiple workers

--workers increases parallelism within a single Processor instance. Running several separate Processor programs (each with --workers 1) achieves similar throughput, but adds more moving parts to operate. The practical guidance:

• For one server and one database, prefer --workers N on a single Processor.
• If there is any chance the Processor could be stopped, a second instance on a different host gives you redundancy. It does not need the same number of workers as the first — though testing shows two instances of two workers each tends to give the fastest throughput on 8-vCPU servers.
• For active/active high availability with failover, run one Processor per host, each with --workers 1 or more. See High Availability.

---

Continue to Connection Poolers to see which poolers the Processor can and cannot work with.