Add the filesystem source connector and transfer (G7·2)

Contribution Date
Contribution Project
Contribution Details
Add the filesystem source connector and transfer (G7·2) A migration needs bytes, and bytes come from a source. dam_pipeline::source holds the Source trait and its first implementation, the filesystem: a folder of files with each record naming its own by relative path. Paths are checked twice — a component walk that refuses `..` and absolute paths before touching the disk, and a canonical-prefix check afterwards that catches a symlink pointing out of the export. The records come from somebody else's system, so `../../etc/passwd` is a failed record rather than an asset. The filesystem first, not a vendor API: an API cannot be reached from here, so it would only ever be verified against its own fake, and a migration verified against a fake is discovered to be wrong while somebody's library is half moved. That was the right call — most of what this commit fixes was found by running it against real files. dam_pipeline::transfer moves one record. It has no ingest of its own: it opens a session, streams the file in through the resumable engine, and calls finalise, which is where content addressing, deduplication, scanning and derivation already live. source_id is the idempotency key, so a run that died half way is resumed by running it again. Three things the first real run found: - finalise does not queue the follow-on work; its one production caller, the worker's finalise handler, does. Calling finalise and stopping produced five assets with a placement each and nothing queued: no proxy, no thumbnail, nothing indexed. The library looked full and searched empty — the drift the no-second-ingest rule exists to prevent, arriving through the gap between finalise and the thing that calls it. - A migration must not queue its renders in the interactive band. Priority 40 assumes somebody is watching the grid for a thumbnail, which is false for the four hundred thousandth asset of a transfer and would put it in front of every real upload for as long as it ran. Split out enqueue_derive_at. - A store outage is not a bad record. Every failure used to mark the record failed; the store went away mid-run and all seven were branded failed, permanently, for a connection refused. Error::is_transient already drew that line — now a transient error stops the run and leaves the record pending, and only Permanent is written against it. damctl gains `import transfer`, which refuses a job that has not been dry-run, requires --confirm, and stops at the job's batch size so §G7's QA gate between batches stays a human decision. Verified by migrating: five files became five assets with the type sniffed from the bytes, dimensions probed and the crosswalked metadata landed; a missing file and a traversal attempt each failed alone while the run continued; re-running skipped what had arrived; a real worker then drained the queue into fifteen derivatives and five index jobs.
Contribution Author
Bassam Ismail
Files count
0
Patches count
1