Reject incomplete downloads instead of reporting success (PLAT-718) #298

Merged
stephen merged 6 commits from stephen/plat-718-download-check into main 2026-10-27 15:06:50 +00:00
Owner

fj run download, fj workflow artifact and fj release download could report success on a truncated file. PLAT-718: Actions artifact zips fetched through the fjord.sh forge gateway were cut short at about 88 to 92 MB, and fj wrote and announced them as complete.

Downloads now go through one download::verified_body path:

  • The body must match Content-Length. A release asset must also match the size the release API reports, which covers a server that sends no Content-Length.
  • A .zip must end in a readable end-of-central-directory record whose directory sits inside the file, including ZIP64 archives (followed through the ZIP64 locator).
  • A short or broken body, a transport error, or a timeout is retried up to three times. HTTP errors are not retried, as before.
  • The 256 MiB response cap now also holds while reading, not only against Content-Length.
  • The file is written atomically: a failed download leaves an existing file untouched. A replaced file keeps its mode; a new one gets the usual umask mode.

The gateway half is fjord-platform#1313 (the gateway now errors the response when the upstream body fails, instead of ending it cleanly).

Tests: cargo test --bin fj (1336 pass), including wiremock cases for a short body, a missing Content-Length with a known release size, transport retries, and unit cases for ZIP64, the read cap and file modes.

`fj run download`, `fj workflow artifact` and `fj release download` could report success on a truncated file. PLAT-718: Actions artifact zips fetched through the fjord.sh forge gateway were cut short at about 88 to 92 MB, and fj wrote and announced them as complete. Downloads now go through one `download::verified_body` path: - The body must match Content-Length. A release asset must also match the size the release API reports, which covers a server that sends no Content-Length. - A `.zip` must end in a readable end-of-central-directory record whose directory sits inside the file, including ZIP64 archives (followed through the ZIP64 locator). - A short or broken body, a transport error, or a timeout is retried up to three times. HTTP errors are not retried, as before. - The 256 MiB response cap now also holds while reading, not only against Content-Length. - The file is written atomically: a failed download leaves an existing file untouched. A replaced file keeps its mode; a new one gets the usual umask mode. The gateway half is fjord-platform#1313 (the gateway now errors the response when the upstream body fails, instead of ending it cleanly). Tests: `cargo test --bin fj` (1336 pass), including wiremock cases for a short body, a missing Content-Length with a known release size, transport retries, and unit cases for ZIP64, the read cap and file modes.
The end-record check read only the classic directory size and offset, so
a valid ZIP64 artifact (over 65,535 entries, or sentinel offsets) failed.
Follow the ZIP64 locator to the ZIP64 end record and bound the directory
against it.
Keep the response size cap while reading a download
Some checks failed
ci / check (pull_request) Failing after 9m16s
ci / coverage (pull_request) Has been skipped
ci / live-e2e (pull_request) Has been skipped
2529711a94
check_body_size sees only Content-Length; a chunked or length-less body
could grow past the 256 MiB cap.
gerri approved these changes 2026-10-27 07:06:43 +00:00
Dismissed
gerri left a comment

Automated Codex review (read-only) of head 2529711a94.

  • Low: src/download.rs:120 (edge case). Invariant: malformed downloads must return an error without crashing. The ZIP64 locator supplies an unchecked offset used in at + 56. A crafted offset near usize::MAX overflows, causing an arithmetic panic in debug builds or a subsequent slice panic in release builds. This affects both artifact download routes and .zip release assets through the shared validator. Use checked_add(56) and bytes.get(at..end) before reading the record, and add a malformed-locator regression test.

BLOCKING: 0

Reviewed head 2529711a94 at 2026-10-27T07:06:43Z by codex exec (read-only sandbox).

Automated Codex review (read-only) of head 2529711a946064e261f138e3947ff27406d19fef. - **Low**: `src/download.rs:120` (edge case). Invariant: malformed downloads must return an error without crashing. The ZIP64 locator supplies an unchecked offset used in `at + 56`. A crafted offset near `usize::MAX` overflows, causing an arithmetic panic in debug builds or a subsequent slice panic in release builds. This affects both artifact download routes and `.zip` release assets through the shared validator. Use `checked_add(56)` and `bytes.get(at..end)` before reading the record, and add a malformed-locator regression test. BLOCKING: 0 --- Reviewed head 2529711a946064e261f138e3947ff27406d19fef at 2026-10-27T07:06:43Z by codex exec (read-only sandbox).
Merge remote-tracking branch 'origin/main' into stephen/plat-718-download-check-2
Some checks failed
ci / check (pull_request) Failing after 24m40s
ci / coverage (pull_request) Has been cancelled
ci / live-e2e (pull_request) Has been cancelled
8ebb39975a
gerri approved these changes 2026-10-27 13:30:49 +00:00
Dismissed
gerri left a comment

Automated Codex review (read-only) of head 8ebb39975a.

  • Low (edge case), src/download.rs:120. Invariant: malformed downloads must return validation errors, never crash the process. The ZIP64 locator supplies an untrusted offset, but at + 56 uses unchecked arithmetic. On 64-bit systems, an offset such as u64::MAX - 55 overflows: debug builds panic immediately; release builds wrap, pass the bounds check, then panic while slicing. A malformed ZIP can therefore abort fj run download, fj workflow artifact, or fj release download instead of following the new validation and retry path. This is introduced by the ZIP64 parser. Use checked_add(56) and bytes.get(at..end); add a regression case with an overflowing locator offset.

BLOCKING: 0

Reviewed head 8ebb39975a at 2026-10-27T13:30:48Z by codex exec (read-only sandbox).

Automated Codex review (read-only) of head 8ebb39975a62c86fd8f8ece208adbc6454096092. - **Low** (edge case), `src/download.rs:120`. **Invariant: malformed downloads must return validation errors, never crash the process.** The ZIP64 locator supplies an untrusted offset, but `at + 56` uses unchecked arithmetic. On 64-bit systems, an offset such as `u64::MAX - 55` overflows: debug builds panic immediately; release builds wrap, pass the bounds check, then panic while slicing. A malformed ZIP can therefore abort `fj run download`, `fj workflow artifact`, or `fj release download` instead of following the new validation and retry path. This is introduced by the ZIP64 parser. Use `checked_add(56)` and `bytes.get(at..end)`; add a regression case with an overflowing locator offset. BLOCKING: 0 --- Reviewed head 8ebb39975a62c86fd8f8ece208adbc6454096092 at 2026-10-27T13:30:48Z by codex exec (read-only sandbox).
Rerun CI (run 572's second check attempt died with no log)
All checks were successful
ci / check (pull_request) Successful in 11m23s
ci / coverage (pull_request) Successful in 2m10s
ci / live-e2e (pull_request) Successful in 2m44s
c8445cac68
stephen deleted branch stephen/plat-718-download-check 2026-10-27 15:06:50 +00:00
gerri approved these changes 2026-10-27 15:06:50 +00:00
gerri left a comment

Automated Codex review (read-only) of head c8445cac68.

  • Low (edge case). src/download.rs:120. Invariant: malformed download bytes must return a validation error rather than panic (standards 4 and 8). The ZIP64 locator supplies an unchecked offset used in at + 56. On 64-bit builds, a locator offset near u64::MAX overflows; with release overflow checks disabled, the subsequent slice still panics. A malformed ZIP downloaded through either artifact route or the release ZIP path therefore terminates the CLI instead of returning the promised download error. Use checked addition and bounded slice access, and add a malformed ZIP64 locator regression test.

BLOCKING: 0

Reviewed head c8445cac68 at 2026-10-27T15:06:49Z by codex exec (read-only sandbox).

Automated Codex review (read-only) of head c8445cac68be0debe0d778925098e00a9bfd1e08. - **Low** (edge case). `src/download.rs:120`. Invariant: malformed download bytes must return a validation error rather than panic (standards 4 and 8). The ZIP64 locator supplies an unchecked offset used in `at + 56`. On 64-bit builds, a locator offset near `u64::MAX` overflows; with release overflow checks disabled, the subsequent slice still panics. A malformed ZIP downloaded through either artifact route or the release ZIP path therefore terminates the CLI instead of returning the promised download error. Use checked addition and bounded slice access, and add a malformed ZIP64 locator regression test. BLOCKING: 0 --- Reviewed head c8445cac68be0debe0d778925098e00a9bfd1e08 at 2026-10-27T15:06:49Z by codex exec (read-only sandbox).
Sign in to join this conversation.
No description provided.