Contribution Date
Contribution Project
Contribution Details
The library said the bytes were there and only the download disagreed
`object_placements.state` has carried `missing` and `corrupt` in its CHECK since the
first migration. `remote_checksum` and `last_verified_at` have been columns just as
long. Nothing has ever written any of them, and `PlacementState` documents `Corrupt`
as the state that "needs a scrub" — of a scrub that did not exist. The whole
verification surface was schema and vocabulary with no code behind it.
Found by losing objects. A load run filled the disk under a single-node SeaweedFS,
the container was killed and restarted, and the writes from its last three minutes
did not survive: 608 gone outright, around 80 listed at full size and serving
nothing. Postgres had flushed every row and kept them all. The API went on reporting
those assets `active` at their recorded size and answered a download with a 404, and
no report anywhere said otherwise — which is the part that is the product's fault
rather than the disk's.
**A `HEAD` per placement, and one byte.** `ObjectState::checksum` exists so the
scrub can "verify without paying egress to re-download", so size and the backend's
own checksum are what it compares — the content hash would mean pulling the library
through the network every pass, and a check that costs that gets turned off. But a
metadata pass calls a full-size-and-serves-nothing object healthy, and SeaweedFS
reports no checksum on `HEAD`, so there is nothing else in the response to disagree
with. Hence the first-byte probe.
**A store that cannot be reached is not a finding**, and that rule is what bounds
what this catches. Only a probe that succeeds and returns *nothing* is corruption —
the one answer no working backend can give. A probe that errors is weather, because
one failed read cannot be told from a blip, and on SeaweedFS that is exactly how
those 80 fail: this pass does not flag them. Telling a damaged object from a flaky
one needs the same probe failing across several passes, which is a column and a
decision this slice does not have. So: missing is detected reliably, unreadable only
when the backend answers rather than errors, and the docs say so in those words.
Verified against the database that produced the problem: five chains started on
worker boot, 2,558 placements checked, 609 recorded `missing` — matching the count
the dead derive jobs had independently implied — and acme, which was not ingesting
when the disk filled, came back clean at 136 verified.
Contribution Issue Link
Files count
0
Patches count
1