An outbox that has never been written to, and an ordering guarantee it could not keep

Contribution Date
Contribution Project
Contribution Details
An outbox that has never been written to, and an ordering guarantee it could not keep The webhook schema has been in migration 0004 since the start — subscriptions, a transactional outbox, per-asset ordering, retry, dead-lettering, auto-disable — and nothing has ever written a row to it. dam-connect/src/lib.rs is a one-line doc comment. This is the db layer. The schema's own comments make the arguments, and each one rules out the obvious implementation. enqueue takes a connection, not a pool, so the row lands in the same transaction as the change: emitting after a commit loses events on a crash, and emitting before one announces changes that may roll back. claim hands out at most one delivery per (subscription, asset), because an asset.version_created delivered after asset.expired republishes an asset whose rights were withdrawn. FOR UPDATE SKIP LOCKED is what makes that safe under concurrency — the check for an in-flight sibling and the claim are one statement, so a second worker cannot slip between another's check and its write. Then the test found that 0004's ordering guarantee could not hold with the column it gave us to order by. created_at defaults to now(), and now() is the transaction timestamp — identical for every statement in the transaction. So two events enqueued together for one asset tie, and the tie-break falls to gen_random_uuid(): random order, on the table whose entire purpose is order. Not a corner case either, since an outbox row is written in the transaction that made the change, so "publish this version and expire the old one" is exactly one transaction with two events for one asset — 0004's own example. Migration 0035 adds a sequence and re-cuts the ordering index onto it. clock_timestamp() would have broken the tie without removing the dependency: it moves backwards on an NTP correction. What a sequence does not promise is stated in the migration rather than left to be discovered — it is allocated at INSERT, not at COMMIT, so ordering is exact within a transaction and best-effort across concurrent ones. That is the right trade, because a cross-transaction race over one asset has no correct order to preserve. Smaller decisions, each one a failure mode with a name. A row stuck in `delivering` blocks every later event for its asset, so a killed worker silently halts one asset's stream — reclaim_stalled returns it and keeps the attempt, since the request may well have arrived. release drops a claim without charging an attempt, so a rolling restart does not look like a failing endpoint. The failure count moves only when a delivery dies, so one payload the endpoint chokes on cannot disable an endpoint that is otherwise healthy. A success forgives the count, because "consecutive" is the whole meaning of that column. revive resets attempts rather than incrementing, because an operator retrying a dead letter has just fixed the endpoint and one attempt would make the button useless. The log withholds payloads: it is the largest column, on the query a UI runs most often, and returning them would be the cheapest way to read a tenant's whole change history in one request. 15 cases, including two workers claiming concurrently, a null-asset stream ordered by IS NOT DISTINCT FROM rather than =, and the same-transaction ordering as an explicit regression.
Contribution Author
Bassam Ismail
Files count
0
Patches count
1