An upload that never becomes an asset, and the outage that does not wave things through

Contribution Date
Contribution Project
Contribution Details
An upload that never becomes an asset, and the outage that does not wave things through Go-live item 5. M1 listed a virus scan; nothing existed. ## Talking to clamd rather than shelling out `dam_media::antivirus` speaks `clamd`'s `INSTREAM` over a socket. The protocol is four lines of framing — `zINSTREAM\0`, length-prefixed chunks, a zero terminator, one reply — which is why there is no dependency for it. `clamscan` was the alternative and it is worse twice over: it reloads the entire signature database on every invocation, seconds of CPU and hundreds of megabytes per file, and it takes a path — meaning the bytes would have to be written to a shared filesystem before we have decided to accept them. ## Scanned before promotion Infected bytes never reach a content-addressed key and never become an asset. Quarantining the asset row afterwards was the alternative, and it leaves the object sitting in the library's own namespace while turning "is this asset safe" into a question about a column. The already-promoted retry path skips the scan deliberately: those bytes were scanned by the attempt that promoted them, and re-reading a 200 GB master to re-scan it on every transient failure would make an outage expensive. ## Three outcomes, three kinds of failure **Infected** is permanent, with the signature in the log — a signature does not change on a retry. **Unreachable** is transient. The upload waits in staging and finalises when `clamd` returns. This is fail-closed, and the recovery is what makes that acceptable rather than merely strict: verified by pointing the worker at a dead port, watching the job re-queue with no asset created, then pointing it back and watching that same upload finalise itself. A configurable fail-open was the alternative and it is a footgun — the setting that lets ingest continue during an outage is the setting still switched on a year later. **Unintelligible** is permanent and deliberately not read as clean. A parser that fell through to `Clean` would turn any future protocol change into a silent bypass of the only thing standing between an upload and the library, so `parse` errors on anything it does not recognise and there is a case asserting that for four different replies. ## Large files are not scanned, and this says so Past `security.max_scan_bytes` — 100 MB by default, matching `clamd`'s own `StreamMaxLength` — an upload is **accepted unscanned** with a warning naming the size. A DAM whose purpose is video masters and layered PSDs cannot refuse every large file. That is a real hole and `docker/DEPLOY.md` states it in those words rather than implying coverage that does not exist. What actually protects those files is elsewhere: nothing is executed, derivatives render in a sandbox with the environment cleared, and delivery is a signed redirect rather than a served file. ## Verified against a real ClamAV, not only a fake Eight unit cases, including a fake `clamd` that asserts the framing from the server's side — the zero-length terminator is the mistake that presents as "the scanner is down", so it is checked rather than assumed, with a payload larger than one chunk to exercise the loop. Then the real thing (arm64 needed `--platform linux/amd64`; there is no native image): EICAR refused permanently as `Eicar-Test-Signature` with no asset created, a clean PNG through to three derivatives, and the outage-and-recovery sequence above. Off by default, because requiring `clamd` on every developer machine to accept an upload would make the dev stack a three-container affair and a security control that blocks the inner loop is one somebody disables permanently. The cost is that a deployment which never sets it never scans, so it is listed as required in `docker/DEPLOY.md` and the worker logs `virus scanning enabled` at startup — the line to check after a deploy. Not built: re-scanning existing assets when signatures update. A scan happens once, at ingest.
Contribution Author
Bassam Ismail
Files count
0
Patches count
1