rdq | Blog

Introducing rdq: retry & dead-letter queues for any broker, any storage, any language

Mon, 27 Jul 2026 00:00:00 GMT

Every event-driven system eventually runs into the same question: a message handler failed — now what?

Today, teams answer it the hard way. You hand-roll a retry topic per Kafka consumer. You wire up broker-specific dead-letter queues — SQS redrive policies, RabbitMQ dead-letter exchanges — each with its own quirks. Or you reach for a heavyweight durable-execution platform and take on a new programming model and a dedicated cluster to solve one narrow problem. The result is duplicated, broker-locked, language-locked plumbing. And when a message finally does land in a DLQ, the on-call engineer opens it to find a bare payload: no error, no stack trace, no history, and no safe way to replay it once the bug is fixed.

rdq is the tool that should have existed for that job. It’s a durable retry engine with first-class dead-letter queues, and its entire promise is small enough to say in one breath: hand it a failed unit of work, and it guarantees one of exactly two outcomes.

The call eventually succeeds, retried on your configured backoff schedule, or
it lands in a dead-letter queue with its full failure history — every attempt’s error, stack trace, and timestamp — where it can be inspected, fixed, and redriven.

That’s it. rdq is not a broker, not a workflow engine, and it never sits on the hot path of your successful messages.

The problem is more general than messaging

The framing that unlocked the design: a function was called with some arguments, and it failed. A Kafka handler, an SQS consumer, a plain func(args) — they’re all the same shape. What you actually need is something that durably remembers the function reference and its arguments, re-invokes the call on a policy, and — if it never succeeds — parks it somewhere inspectable and replayable.

Once you frame it that way, the broker stops mattering. rdq never talks to Kafka or SQS. It accepts failures from any source and returns one of two terminal outcomes. That’s what lets one tool serve Kafka, SQS, Redpanda, RabbitMQ, AutoMQ — or no broker at all.

Four decisions that define rdq

1. Bring your own storage

rdq adds no new stateful infrastructure. Retry queues and DLQs live in a datastore you already operate. PostgreSQL is the v1 reference backend; Redis, MongoDB, and others plug in through a documented storage SPI that ships with a public compliance test-kit, so third parties can build and verify their own plugins.

This is the difference between “adopt a new database to get retries” and “point rdq at the Postgres you’re already running.” The second one is a Tuesday afternoon.

2. Broker-agnostic, off the hot path

rdq never sits between your producer and consumer. It receives work only after something has already failed. Your throughput on the happy path is untouched; rdq scales with your failure rate, which is (hopefully) a much smaller number.

3. Polyglot from day one

There is one language-neutral wire envelope — JSON, with the payload as opaque bytes plus a content type — shared by both SDKs, the server API, and every storage plugin. v1 ships a Go SDK, a Java SDK, and a REST/gRPC API for everything else. A task submitted from Go can be redriven and executed through the server by a totally different service. No language-specific serialization is allowed in the core.

4. The DLQ is a product, not a graveyard

This is the part most tools get wrong. In rdq, the complete failure history travels with the task into the DLQ. You can browse and filter by queue, error type, handler, and time range. You can redrive a single task or bulk-redrive by filter — “every payments.charge task that failed with TimeoutException after 14:00” — with the retry policy reset. Single-task redrive can even edit the payload first. And every mutation is audit-logged: who, when, what filter. The DLQ analysis & redrive guide walks through it.

One core, two hosts

The engine — task model, retry policies, storage SPI, DLQ semantics, wire format — is a single core shipped in two form factors, both in v1:

Embedded SDK (Go and Java): the core runs inside your application process; handlers are in-process functions. Zero extra infrastructure beyond your storage.
Standalone service (rdq-server, in Go): the same core behind REST/gRPC intake; handlers are remote callbacks (HTTP or gRPC) registered per queue. A central retry hub any language can use over the wire.

And they compose. Because the client and worker artifacts are split, you can submit here and execute there — enqueue from a lightweight producer, let a fleet of workers (or the server) drain the queue. A callback that times out is just another failed attempt; it flows through the identical retry/DLQ path as an in-process error. The full picture is in Architecture.

Correctness without a coordinator

rdq has no leader election and no cluster membership. Every process — embedded workers and rdq-server nodes alike — is stateless, and the storage backend is the only coordination point. Workers claim due tasks atomically (Postgres uses FOR UPDATE SKIP LOCKED), and every claim carries a lease. If a worker dies mid-task — kill -9, a crash, a network partition — the lease expires and any other worker reclaims the task. Nothing is lost, and no two workers ever run the same task concurrently.

That’s the whole scaling story: add a node to go faster; node death is a non-event. Your availability inherits your storage backend’s HA — Postgres replicas, Redis Sentinel, Mongo replica sets — because rdq adds no stateful component of its own. The task lifecycle has the details.

The honest limits

rdq is deliberately narrow, and the non-goals are part of the design:

At-least-once, not exactly-once. Your handlers must be idempotent. We say this loudly and often because it’s the one contract you can’t ignore.
Not a workflow engine. One task equals one function call. No sagas, no signals, no multi-step orchestration.
No ordering guarantee by default. Retrying out-of-band inherently breaks partition ordering; per-key ordered retry is a post-v1 opt-in.

If you need durable multi-step orchestration, you want Temporal. If you need retries and a DLQ you can actually operate, you want rdq.

Where it’s going

v1 is the core engine, the Go and Java SDKs, rdq-server, the PostgreSQL plugin, the rdq CLI, Prometheus metrics, and DLQ browse/redrive/audit. On the roadmap: a web UI for DLQ analysis, Redis and MongoDB plugins, Python and TypeScript SDKs, per-key ordered retry, and broker-native intake adapters that dead-letter on behalf of legacy apps with zero code change.

Try it

The fastest path is the Quickstart: submit a task, run a worker, watch it retry and then dead-letter, then redrive it after a fix. If you just want the mental model first, start with What is rdq?.

rdq is open source under Apache-2.0. The code — and the design docs behind every decision above — live on GitHub.

rdq — Retry & Dead-letter Queues for any broker, any storage, any language.

No two workers, one task: how rdq claims work — and the UNIQUE collision redrive taught us

Mon, 27 Jul 2026 00:00:00 GMT

rdq runs a fleet of workers against a shared datastore with no leader election and no cluster membership. There’s no coordinator process to ask “whose task is this?” — and that’s on purpose. The entire correctness argument rests on two primitives, both borrowed from the storage backend: an atomic claim and a lease. Get those two right and the rest of the system — “add a node to scale,” “a kill -9 is a non-event” — falls out for free.

This post is the detail behind the task lifecycle: how a claim works, why the lease is the whole fault-tolerance story, and a subtle bug that only surfaced when a redriven task’s lease expired.

The one rule

Everything reduces to a single correctness bar:

No two workers may ever claim the same task.

Not “rarely.” Not “we detect and reconcile.” Never. If two workers can both believe they own a task, you get concurrent double-execution on top of the at-least-once contract, and no amount of idempotency guidance saves you from the surprise.

rdq refuses to solve this with a coordinator, because a coordinator is a stateful component with its own HA story — exactly the thing rdq promises not to add. Instead, the claim is delegated to whatever the storage backend already does atomically.

The atomic claim

A worker’s inner loop is: find due tasks, take one, run it. “Find due” is easy — status = PENDING AND next_attempt_at <= now(). The hard part is “take one” without racing every other worker doing the same query at the same millisecond.

Each storage plugin owns its own atomic-claim mechanics, and this is a hard requirement of the storage SPI:

PostgreSQL — SELECT … FOR UPDATE SKIP LOCKED. The row lock makes the claim atomic; SKIP LOCKED means a worker steps over rows another worker has already locked instead of blocking on them. Ten workers hit the same query and walk away with ten different tasks.
Redis — an atomic sorted-set pop, scored by next_attempt_at, in a Lua script.
MongoDB — findAndModify, which claims and updates in one atomic step.

In Postgres the claim looks roughly like this:

UPDATE tasks
SET status = 'IN_FLIGHT',
    lease_expires_at = now() + $lease
WHERE id IN (
    SELECT id FROM tasks
    WHERE queue = $1 AND status = 'PENDING' AND next_attempt_at <= now()
    ORDER BY next_attempt_at
    FOR UPDATE SKIP LOCKED
    LIMIT $batch
)
RETURNING *;

The SELECT … FOR UPDATE SKIP LOCKED does the claiming; the surrounding UPDATE flips the task to IN_FLIGHT and stamps a lease in the same transaction. There is no window between “I saw it” and “I own it.”

The lease is the fault-tolerance story

An atomic claim stops two live workers from grabbing the same task. But what about a worker that claims a task and then dies — OOM-killed, crashed, partitioned off the network — while the task sits IN_FLIGHT? Without more, that task is stuck forever, owned by a ghost.

That’s what the lease is for. Every claim stamps lease_expires_at. A worker is expected to finish (or heartbeat) before the lease runs out. If it doesn’t, the task becomes reclaimable:

-- reclaim: an IN_FLIGHT task whose lease has expired is fair game again
WHERE status = 'IN_FLIGHT' AND lease_expires_at <= now()

The reclaiming worker treats the expiry as a failed attempt — the handler exceeded its visibility timeout — and reschedules per policy. A kill -9 mid-task doesn’t lose work and doesn’t hang the queue; it just costs one lease interval. This is why rdq can say node death is a non-event: the lease turns a crash into an ordinary retry.

Where it got subtle: LEASE_EXPIRED meets redrive

Here’s the part that only shows up in production.

rdq keeps a full attempt history — one row per execution attempt, and that history is what travels into the DLQ. Attempts are keyed by UNIQUE(task_id, attempt_no): attempt 1, attempt 2, and so on, one row each. A lease expiry isn’t invisible — it records a synthetic LEASE_EXPIRED attempt in that same history, so the DLQ shows why a task took an extra lap.

Now stack two features that were each correct in isolation:

Redrive takes a dead task and re-enqueues it with its policy reset — it starts attempting again.
Lease expiry records a LEASE_EXPIRED attempt using the task’s attempt number.

The bug: when we recorded the LEASE_EXPIRED attempt, we derived attempt_no from the task’s live attempt_count. For a fresh task that’s fine. But a redriven task has already accumulated attempt rows from its first life. Reset the counter on redrive, let the next attempt’s lease expire, and the LEASE_EXPIRED write computes an attempt_no that already exists in the history from before the redrive — and slams straight into the UNIQUE(task_id, attempt_no) constraint. The reclaim transaction aborts. The task can’t be reclaimed. The one path that’s supposed to rescue stuck work was the path that jammed.

It’s a lovely little bug because both features are individually correct; only their composition — a redriven task whose subsequent lease expires — reveals the shared assumption that attempt numbers are monotonic within a single logical life.

The fix: derive the number from history, not from a counter

The fix was to stop trusting the live counter and instead compute the next attempt_no from the attempt history itself — MAX(attempt_no) + 1 for that task — so a redriven task’s LEASE_EXPIRED row always lands after everything already recorded, never on top of it. History is the source of truth for “what number comes next,” because history is the thing the constraint is actually protecting.

The general lesson has outlived the specific bug: when a monotonic sequence has a UNIQUE constraint, derive the next value from what’s already persisted, not from a mutable counter that a feature like redrive is allowed to reset. The counter and the history disagreed exactly at the seam between two features, and the database — correctly — refused to let them.

Why this is the architecture, not a detail

It would be easy to file all of this under “implementation.” It isn’t. The claim-and-lease model is the reason rdq can make its headline promises:

Stateless workers — because the claim lives in storage, a worker holds no authoritative state. Any worker can pick up any task.
Horizontal scaling — because SKIP LOCKED (and its Redis/Mongo equivalents) hands concurrent workers disjoint tasks, “scale up” is “start another process.”
Fault tolerance for free — because the lease turns any crash into a reclaimable task, node death needs no special handling.
HA inherited, not invented — because the only coordination point is the backend, rdq’s availability is your Postgres/Redis/Mongo availability, and rdq adds nothing stateful of its own.

Every storage plugin has to clear the same bar — no double-claims, ever — which is why the SPI ships with a compliance test-kit that includes a multi-worker chaos test: kill -9 a worker mid-processing and assert the task is reclaimed after its lease expires, with no double-execution. A plugin that can’t pass that isn’t a plugin.

Read on

Tasks, attempts & the lifecycle — the full state machine.
Storage SPI & compliance kit — what every backend must guarantee.
Architecture — one core, two hosts — where claims fit in.

rdq is open source under Apache-2.0 — github.com/srjn45/rdq.