The archive nothing ever moved to, and the restore nobody could ask for

Contribution Date
Contribution Project
Contribution Details
The archive nothing ever moved to, and the restore nobody could ask for Asked what was still missing for archival. The answer turned out to be: the wiring, and only the wiring. Everything else was built and good. The planner in `dam_store::lifecycle` with its dry-run default and its per-candidate skip reasons. The restore arithmetic in `dam_core::restore`, refusing Expedited on Deep Archive rather than silently answering a request for minutes with twelve hours. `transition` and `restore` on the store trait, conformance-tested against real AWS every night, asserting the two properties that matter — `GLACIER_IR` readable without a restore, `DEEP_ARCHIVE` not. The bookkeeping in `dam_db::restores`, coalescing duplicate requests so two people asking do not pay twice. The schema for all of it, down to an index built for a poll query. None of it was called from anywhere. So every asset stayed in the class it was uploaded to, permanently; `restore_requests` was a table with no writer; and `restore_requests_poll_idx` had been sitting there since the first migration with no query behind it. ## Two jobs `tier_sweep` plans every enabled policy and executes what is not a dry run. `restore_poll` issues what is queued, checks what is in flight, and expires what has lapsed. Separate kinds because they fail differently: a sweep can skip a day with no consequence, while a restore is a person waiting, and sharing one kind would let the daily pass's backoff decide how long that person waits. Both re-queue themselves, following M5c's pattern — and *not* under a dedupe key, which is the subtlety `requeue_backfill_collect` documents at length: the running job still holds the key, so a shared one resolves to the job doing the enqueueing and the chain ends silently the moment it completes. There is a case for each kind asserting exactly one successor is left behind. A backfill that stops is a library that stays undescribed; a sweep that stops is a tenant paying Standard rates on a cold archive forever, with a policy row that says `enabled`. ## `pinned` was never written by anything `object_placements.pinned` exists, the schema comment says it covers legal-hold assets and `pin_hot` collections, and there is an index over it for the candidate scan. Nothing has ever set it. Trusting it would mean archiving assets under legal hold — the single worst thing this engine can do, since a legal hold means litigation and Deep Archive means the bytes are unreadable for up to 48 hours. So the scan derives pinning from the facts that mean it and ORs the column on top, where it stays useful as a manual override rather than as the only thing standing between a policy typo and a legal problem. The interface derivatives needed no such treatment: `Key::is_tier_exempt` already reads the key's namespace, so it holds "even for an object whose placement row is missing or stale" — strictly better than anything the scan could re-derive from `derivatives.role`, and a second implementation would only be a second thing to keep in step. ## The module could not be called, which is why it never was `dam_db::restores` took `&sqlx::PgPool`. Its tables are tenant tables, a tenant table needs the tenant's `search_path`, and that lives on a connection — so from the worker, which holds a global pool, every one of those functions was unreachable. The signature is why the code was dead. It takes `&mut PgConnection` now and the claim shares the caller's transaction, which is where the rest of a poll's writes are anyway. ## Choosing at-most-once, and paying for it honestly The poll claims in one transaction and issues outside it. A worker that dies in between leaves a row saying `requested` with nothing asked for; holding the transaction across the vendor call would trade that for re-issuing every restore in the batch on the same crash, and each of those is a real charge. So the cheap failure is the one that was chosen, and the poll reconciles against S3's own state to close it: an object with no restore in progress under a row that claims one is a call that never landed, and it is re-issued. Without that, a crash at the wrong instant leaves somebody watching a spinner since Tuesday. ## Per-tier retrieval prices `storage_pools` had one `cost_per_gb_retrieval`, and §6.5's whole argument for showing an estimate before somebody confirms is that Expedited against Bulk is roughly 10× on price. One column cannot express that: the screen asking a user to choose would have shown the same number three times. Two nullable columns now, falling back to the single one — not derived from AWS's published ratios, because a hardcoded multiplier inside a figure somebody approves a spend against is wrong on any other provider and says nothing about it. Deliberately unseeded: the dev stack is SeaweedFS, where retrieval is free. Worth recording from writing the tests: at eight kilobytes all three tiers quote the same single cent, because the per-1000-requests charge dominates and the per-GB term rounds away. Arithmetically right, and it means the tier chooser is genuinely meaningless for small objects — worth knowing before somebody reports the identical prices as a bug. Seventeen cases against `FakeS3Store`, which models the minimum billable duration and the restore timeline; eight mutations over the paths where being wrong is expensive or unlawful, all caught, including both halves of the pinning derivation.
Contribution Author
Bassam Ismail
Files count
0
Patches count
1