Contribution Date
Contribution Project
Contribution Details
The archive nothing ever moved to, and the restore nobody could ask for
Asked what was still missing for archival. The answer turned out to be: the wiring, and
only the wiring.
Everything else was built and good. The planner in `dam_store::lifecycle` with its
dry-run default and its per-candidate skip reasons. The restore arithmetic in
`dam_core::restore`, refusing Expedited on Deep Archive rather than silently answering a
request for minutes with twelve hours. `transition` and `restore` on the store trait,
conformance-tested against real AWS every night, asserting the two properties that
matter — `GLACIER_IR` readable without a restore, `DEEP_ARCHIVE` not. The bookkeeping in
`dam_db::restores`, coalescing duplicate requests so two people asking do not pay twice.
The schema for all of it, down to an index built for a poll query.
None of it was called from anywhere. So every asset stayed in the class it was uploaded
to, permanently; `restore_requests` was a table with no writer; and
`restore_requests_poll_idx` had been sitting there since the first migration with no
query behind it.
## Two jobs
`tier_sweep` plans every enabled policy and executes what is not a dry run. `restore_poll`
issues what is queued, checks what is in flight, and expires what has lapsed. Separate
kinds because they fail differently: a sweep can skip a day with no consequence, while a
restore is a person waiting, and sharing one kind would let the daily pass's backoff
decide how long that person waits.
Both re-queue themselves, following M5c's pattern — and *not* under a dedupe key, which
is the subtlety `requeue_backfill_collect` documents at length: the running job still
holds the key, so a shared one resolves to the job doing the enqueueing and the chain
ends silently the moment it completes. There is a case for each kind asserting exactly
one successor is left behind. A backfill that stops is a library that stays undescribed;
a sweep that stops is a tenant paying Standard rates on a cold archive forever, with a
policy row that says `enabled`.
## `pinned` was never written by anything
`object_placements.pinned` exists, the schema comment says it covers legal-hold assets and
`pin_hot` collections, and there is an index over it for the candidate scan. Nothing has
ever set it.
Trusting it would mean archiving assets under legal hold — the single worst thing this
engine can do, since a legal hold means litigation and Deep Archive means the bytes are
unreadable for up to 48 hours. So the scan derives pinning from the facts that mean it and
ORs the column on top, where it stays useful as a manual override rather than as the only
thing standing between a policy typo and a legal problem.
The interface derivatives needed no such treatment: `Key::is_tier_exempt` already reads the
key's namespace, so it holds "even for an object whose placement row is missing or stale" —
strictly better than anything the scan could re-derive from `derivatives.role`, and a second
implementation would only be a second thing to keep in step.
## The module could not be called, which is why it never was
`dam_db::restores` took `&sqlx::PgPool`. Its tables are tenant tables, a tenant table needs
the tenant's `search_path`, and that lives on a connection — so from the worker, which holds
a global pool, every one of those functions was unreachable. The signature is why the code
was dead. It takes `&mut PgConnection` now and the claim shares the caller's transaction,
which is where the rest of a poll's writes are anyway.
## Choosing at-most-once, and paying for it honestly
The poll claims in one transaction and issues outside it. A worker that dies in between
leaves a row saying `requested` with nothing asked for; holding the transaction across the
vendor call would trade that for re-issuing every restore in the batch on the same crash,
and each of those is a real charge.
So the cheap failure is the one that was chosen, and the poll reconciles against S3's own
state to close it: an object with no restore in progress under a row that claims one is a
call that never landed, and it is re-issued. Without that, a crash at the wrong instant
leaves somebody watching a spinner since Tuesday.
## Per-tier retrieval prices
`storage_pools` had one `cost_per_gb_retrieval`, and §6.5's whole argument for showing an
estimate before somebody confirms is that Expedited against Bulk is roughly 10× on price.
One column cannot express that: the screen asking a user to choose would have shown the
same number three times. Two nullable columns now, falling back to the single one — not
derived from AWS's published ratios, because a hardcoded multiplier inside a figure
somebody approves a spend against is wrong on any other provider and says nothing about
it. Deliberately unseeded: the dev stack is SeaweedFS, where retrieval is free.
Worth recording from writing the tests: at eight kilobytes all three tiers quote the same
single cent, because the per-1000-requests charge dominates and the per-GB term rounds
away. Arithmetically right, and it means the tier chooser is genuinely meaningless for
small objects — worth knowing before somebody reports the identical prices as a bug.
Seventeen cases against `FakeS3Store`, which models the minimum billable duration and the
restore timeline; eight mutations over the paths where being wrong is expensive or
unlawful, all caught, including both halves of the pinning derivation.
Contribution Issue Link
Files count
0
Patches count
1