← back to the counter

Mounted-bundle cutover silently dropped every seed file with a space in its name

Summary

PR #10013 flipped the default TAIGA run path from baking problem data into a per-run Docker image to one shared framework image plus problem bundles loaded at runtime ("mounted bundles"). The flip routed problem seed files through TAIGA's presigned-upload endpoint for the first time — which silently rewrites ASCII spaces to underscores in file paths at signing time. Bundles were staged with underscored names while the generated initialize.py still referenced the original space-named paths, so at env init those files simply didn't exist. Agents started tasks with empty Desktops and missing email attachments, built deliverables from scratch, and scores were materially depressed for about two days until the transport was rewritten to tarballs (HAL-7070, PR #10218).

Impact

• 289 of 488 prod bundles (314 runs across 149 envs) affected on day one — any bundle containing at least one file with a space in its name.
• Affected runs booted with missing Desktop seeds, missing email attachments (emails delivered with "attachments": [] while the body said "here is the attachment"), and "Source file not found" at env init.
• Agents improvised deliverables from nothing, so runs completed with quietly wrong, depressed scores — the worst failure shape for a grading platform, and part of why detection took until the next morning.
• Not subsystem-specific: 186 affected bundles had Nextcloud assets, 116 desktop assets, 97 email; only 39 were email-only.

Timeline (PT)

How it happened

1. Why were files missing at env init? They were staged under underscored names (Sample_TEV_Bridge.pptx) while initialize.py referenced the canonical ones (Sample TEV Bridge.pptx).
2. Why were they renamed? TAIGA's presigned-upload-url endpoint sanitizes file_path by replacing ASCII space with _ — only 0x20; tabs, unicode, and punctuation pass through. It's undocumented, and the upload succeeds silently at the renamed path.
3. Why did this only break now? The rename had existed all along (pre-flip supporting_files already showed underscored names), but the mounted-bundle flip newly routed problem seed files through this upload path — per-file, at staging time.
4. Why wasn't the loss caught at staging? stageProblemBundleForTaiga never verified staged names against the bundle manifest — a silent-loss gap. The manifest declared the canonical paths the whole time.
5. Why did it take a day to diagnose? Runs didn't fail — they succeeded with bad scores. The signal was "agents seem worse," which looks like a model problem, a prompt problem, or a grading problem before it looks like a filesystem problem.

Root cause: a lossy, undocumented transform in an upstream API, combined with a staging pipeline that never verified what it uploaded against what the manifest declared. Either documentation or verification would have caught it in minutes instead of days.

What surprised us

• An upload API that renames your files and tells no one. The endpoint returns success; the object just lands at a different key. TAIGA later confirmed it's intentional and permanent.
• Only spaces. Tabs, commas, parens, unicode — all preserved. A sanitizer that handles exactly one character is almost harder to find than one that handles everything.
• a b.txt and a_b.txt collide post-sanitization — the transform isn't just lossy, it's many-to-one.
• The failure mode was silence at every layer: upload succeeded, staging succeeded, env init logged a missing file and moved on, agents adapted, runs completed. Five layers each decided this wasn't worth failing loudly over.

What went well / what went poorly

Went well

• The evidence chain was airtight once suspicion landed: direct API probe, production state inspection, and a controlled 2×2 repro isolating spaces as the sole variable.
• The tarball fix was validated end-to-end (including collision pairs, unicode, and zero-byte files) before shipping, and let us delete the older zero-byte-materialization workaround.

Went poorly

• ~314 runs of writer and agent time burned before detection.
• Two days of degraded prod for a bug that a staging-vs-manifest check would have caught on the first bundle.
• The flip shipped a transport-layer change to the default prod path without a filename-edge-case fixture (spaces are common in real deliverable names — this corpus is full of 809 Keyser Ave - Aerial Annotation Rationale.docx).

Action items

Written up 2026-07-28 from the investigation records.