Contribution Date
Contribution Project
Contribution Details
Add the filesystem source connector and transfer (G7·2)
A migration needs bytes, and bytes come from a source. dam_pipeline::source
holds the Source trait and its first implementation, the filesystem: a folder
of files with each record naming its own by relative path. Paths are checked
twice — a component walk that refuses `..` and absolute paths before touching
the disk, and a canonical-prefix check afterwards that catches a symlink
pointing out of the export. The records come from somebody else's system, so
`../../etc/passwd` is a failed record rather than an asset.
The filesystem first, not a vendor API: an API cannot be reached from here, so
it would only ever be verified against its own fake, and a migration verified
against a fake is discovered to be wrong while somebody's library is half
moved. That was the right call — most of what this commit fixes was found by
running it against real files.
dam_pipeline::transfer moves one record. It has no ingest of its own: it opens
a session, streams the file in through the resumable engine, and calls
finalise, which is where content addressing, deduplication, scanning and
derivation already live. source_id is the idempotency key, so a run that died
half way is resumed by running it again.
Three things the first real run found:
- finalise does not queue the follow-on work; its one production caller, the
worker's finalise handler, does. Calling finalise and stopping produced five
assets with a placement each and nothing queued: no proxy, no thumbnail,
nothing indexed. The library looked full and searched empty — the drift the
no-second-ingest rule exists to prevent, arriving through the gap between
finalise and the thing that calls it.
- A migration must not queue its renders in the interactive band. Priority 40
assumes somebody is watching the grid for a thumbnail, which is false for the
four hundred thousandth asset of a transfer and would put it in front of every
real upload for as long as it ran. Split out enqueue_derive_at.
- A store outage is not a bad record. Every failure used to mark the record
failed; the store went away mid-run and all seven were branded failed,
permanently, for a connection refused. Error::is_transient already drew that
line — now a transient error stops the run and leaves the record pending, and
only Permanent is written against it.
damctl gains `import transfer`, which refuses a job that has not been dry-run,
requires --confirm, and stops at the job's batch size so §G7's QA gate between
batches stays a human decision.
Verified by migrating: five files became five assets with the type sniffed from
the bytes, dimensions probed and the crosswalked metadata landed; a missing file
and a traversal attempt each failed alone while the run continued; re-running
skipped what had arrived; a real worker then drained the queue into fifteen
derivatives and five index jobs.
Contribution Issue Link
Files count
0
Patches count
1