The nightly that never ran, and three defects only real Glacier could find

The nightly that never ran, and three defects only real Glacier could find Asked whether we could switch to AWS and test it. The first answer was that the thing supposed to be testing it had never run. ## A green nightly that executed nothing `nightly-aws.yml` invoked `cargo test -p dam-store --features aws-conformance`. There was no such feature. It also exited early every night because the credential secret was unset, so cargo never got far enough to say so — green forever, having run nothing, for months. Worse than the gap itself: both skip messages in the shared conformance suite point at that nightly as the thing covering them. "driver has no RestoreObject — covered by FakeS3Store and the AWS nightly" was a cheque written against a file that did not exist, and I repeated the claim in this session before checking it. The feature and an `#[ignore]`d target now exist. Against a real bucket the suite reports **20 passed, 0 skipped** — the first time `transition` and `RestoreObject` have executed against actual Glacier. The workflow's skip is now a `::warning::` naming what did not run, and it invokes the named target rather than `-- --ignored` across the package, which is how a missing feature hid behind nine "0 tests" lines. Missing configuration is a failure inside the test rather than a skip, because a silent skip is exactly what went wrong. ## Then the whole stack against the same bucket, which found three **A deployment against real AWS was inexpressible.** `storage.endpoint` defaulted to the dev SeaweedFS; `S3Store::aws` is chosen only when it is `None`; and neither an environment variable nor a TOML file can put an `Option` back to `None` — env vars only set, and TOML has no null. So every reachable configuration produced *some* endpoint and real S3 could not be configured at all. The default is now `None`, which is the production shape and fails in the honest direction: `damd` with no storage config asks AWS for a bucket and says so, rather than quietly talking to a SeaweedFS that may not be there. An empty string reads as "no endpoint" for the case where something upstream already set one. **Pinned rows consumed the run cap.** A tenant with 136 pinned placements and `max_objects_per_run = 1` planned nothing, run after run: the scan fetched cap-plus-one rows in key order and both were pinned. Two attempts to order the unmovable rows last each missed a case — first collection pins, then cold-to-warm — because each was an incomplete reimplementation of the planner in SQL. The fix is to stop trying: the cap bounds what *moves*, because that is what costs money, and a separate much wider window bounds what is *read*, which is one indexed scan. Conflating two different budgets is what produced a policy that looked configured and silently did nothing forever. **Credential refresh timed out at five seconds.** A worker up for twenty minutes failed every `CopyObject` with `ConnectorError { TimedOutError(5s) }`, while the identical copy took 0.58 s from the CLI and a restarted worker succeeded immediately. The five was `DEFAULT_LOAD_TIMEOUT` in the SDK's identity cache, not anything about the request — an SSO exchange or an IMDS call behind a refresh does not always fit in it. Now thirty seconds and stated rather than defaulted, alongside the retry and timeout config that is already explicit for the same reason. This would have hit any long-lived deployment an hour after deploy, wearing a network error's clothes. ## What ran, end to end, in ap-south-1 Ten objects uploaded to real S3 and probed; one original transitioned `STANDARD → GLACIER` by the sweep with AWS agreeing and the 90-day counter written forward to November; a quote priced from the pool's real retrieval rates; an Expedited restore issued, observed in AWS as `ongoing-request="true"`, and completing in 1 m 26 s; the tier reading `restored`; and the original fetched back through the signed-URL chokepoint byte-identical to what went up. Synthetic plates rather than real photographs, deliberately: nothing about this test needs personal images and they would have ended up in a company bucket. One number kept rather than corrected: our `expires_at` for a restored copy is seven days from availability, while AWS reported `expiry-date` a day later — it rounds to a day boundary. Ours is the conservative side, so delivery stops before the bytes do. The bucket is emptied and kept: empty buckets are free, and the nightly needs one to point at once its secrets exist.