Transactional dispatch
What happens when you publish a job inside a database transaction. pg-boss joins it, every other adapter commits an intent to a ledger table and a relay publishes after the commit, and both routes have to survive the process dying in between.
This is the one place where swapping the adapter changes behavior you can
observe rather than just configuration you can read. publish() looks the same
either way. What backs it does not.
The two routes
pg-boss with useApplicationTransaction: true, its default, implements
publishInTransaction and inserts the job through the same Drizzle transaction
your business write is using. One commit, both rows, nothing in between.
BullMQ, Cloudflare Queues and any adapter without that method cannot see your
transaction. QUESTPIE writes an intent row into questpie_queue_dispatch inside
your transaction instead, and a leased relay publishes it to the broker after
the commit lands.
| Situation | Result |
|---|---|
| Transaction rolls back | Neither a job nor an intent exists. |
| Crash after commit, before the broker accepted | Recovered on the next execution opportunity. |
| Crash after acceptance, before the receipt saved | Another physical delivery is possible under the same dispatchId. |
No transaction, no idempotencyKey | The publish awaits adapter acceptance directly. |
No transaction, with idempotencyKey | The ledger runs, so repeated calls resolve to one dispatch. |
Handlers have to be idempotent
The stable dispatchId identifies retries. It does not make a downstream side
effect happen once. Pass it to the provider's own idempotency facility, or
keep a processed-dispatch record your handler checks first.
The ledger table
questpie_queue_dispatch is Queue-owned and separate from the realtime outbox
on purpose. It is added to the schema unconditionally, even when the adapter can
publish directly, so changing adapters never generates a destructive
DROP TABLE migration and never drops recovery state you still needed.
idempotencyKey and singletonKey cannot be combined. A publish suppressed by
native singleton dedup has no new logical identity, so it cannot produce a
trustworthy receipt for a dispatch that was never accepted.
Relay bounds
Publication recovery is finite. A row gets 25 attempts with exponential backoff
capped at one hour, and then stays failed. queue.drain() counts it under
terminal and logs a structured error whose fields never carry the payload.
A terminal row is not retried again. Fix whatever the adapter was rejecting and
publish a new logical attempt under a new idempotencyKey, because the original
key stays bound to its terminal receipt. A terminal row that never carried a
secret keeps its payload so you can see what broke.
Recovery needs somewhere to run
The framework does not start a second process to drain the ledger. Something has to give it a turn.
Long-running workers
listen() drains on startup and then ticks every five seconds, each tick
processing up to ten batches. You get recovery for free and never call drain()
yourself.
Serverless and push
runOnce() relays before and after its own batch, and a push consumer drains on
each delivery. Both only run when something invokes them, so a committed intent
sitting behind a queue that receives no new traffic waits. On Cloudflare, add a
platform Cron Trigger that calls app.queue.drain().
queue.drain()
Call it directly when you need a bounded relay pass on your own schedule.
import { app } from "#questpie";
const { claimed, accepted, failed, terminal } = await app.queue.drain({
batchSize: 100,
maxBatches: 5,
});| Option | Default | Notes |
|---|---|---|
batchSize | 100 | Rows claimed per batch, and the page size for secret inspection. |
maxBatches | 1 | Consecutive batches in this pass. An integer from 1 to 100. |
concurrency | 8 | Rows relayed in parallel within a batch. |
It returns { claimed, accepted, failed, terminal }. Concurrent calls collapse
into the one already running rather than stacking, so a cron that fires while
the previous pass is still going will not pile up.
Where each topic lives
| Topic | Page |
|---|---|
| Which adapters can publish in a transaction | Queue |
Implementing publishInTransaction yourself | Writing an adapter |
idempotencyKey, singletonKey, PublishOptions | Jobs |
| Reading a dispatch receipt | Secret payloads |
| Starting a worker in each model | Running a worker |
Direct-to-storage uploads
Mint a presigned upload in a route, hand it to the browser, and let the bytes travel straight to S3 or R2 while your server handles only the metadata.
Writing an adapter
The obligations behind each optional method on QueueAdapter, what the framework guarantees you in return, and why declaring a capability you cannot actually honor is the one mistake that costs data.