← back to the counter

Horizon's new OIDC runtime role could read an object but not a version of it, so every job with seed files failed at startup

Summary

HAL-10076 moved the Horizon Vercel deployment off the wyatt IAM user's access key and onto the horizon-vercel-runtime-production role through Vercel OIDC. The new role granted s3:GetObject on taiga-horizon but not s3:GetObjectVersion. The build-context step copies each problem seed file with a versioned CopySource (?versionId=), and S3 authorizes that copy as s3:GetObjectVersion. Every job on a problem with seed files failed at startup. Adding the one action (#13250) and running pulumi up fixed it.

Impact

• 37 job runs across 16 environments failed at startup, 16:58:36 – 17:31:21 PDT. They stay failed; owners re-ran them.
• 4 internal QA evals failed at kickoff with the same error (3 post-job, 1 pre-job). 75-project-bravo-deck #43 was re-run and completed at 17:35; arr-waterfall #5 (post-job) and kestrel-accretion-dilution-rerun #1 (pre-job) needed a manual re-run. A failed post-job eval also blocks the auto post-job QA cron for that run: its anti-join on qa_evals matches any status, so a failed kickoff never retries by itself.
• Jobs on problems without seed files, and runs already past startup, were unaffected.
• No other denied action: a scan of production runtime logs for not authorized / AccessDenied from 16:50 to 17:47 found only s3:GetObjectVersion, and no other run or QA table recorded a denial.

Timeline

How it happened

• The wyatt key had broad S3 access, so nothing in the app ever had to declare the exact S3 actions it uses. The role policy was written from the obvious verbs (get, put, delete, multipart), not from a census of the calls the app makes.
• docker-build.ts pins each seed file to the version it fingerprinted (HeadObject → VersionId), then copies with ?versionId=. That is the right design — it stops a mid-build edit from mixing versions — but it is the one call path that needs a second read action.
• taiga-horizon has versioning on, so every seed file has a VersionId and every copy takes the versioned path. There was no "unversioned" case to hide behind.
• No job on a problem with seed files ran under the role before the production switch.

What surprised us

• s3:GetObject does not cover reading a specific version. S3 treats "read the latest" and "read version X" as separate permissions, even when version X is the latest. A CopyObject with a versioned source checks s3:GetObjectVersion on the source, not s3:GetObject.
• The failure looked like the infra-cleanup PR that merged 13 minutes later, but the trigger was the env var flip. The error arrived before the PR everyone suspected.

What went well / what went poorly

Went well:

• The error message named the role, the action, and the object. Victoria's report named the fix outright.
• One-line fix, a pulumi preview showing exactly two policy changes, and recovery 14 minutes after the first report.
• The runtime-log scan confirmed this was the only missing permission.

Went poorly:

• The role was cut over in production before one end-to-end job ran under it.
• Failed runs do not retry once the cause is fixed; 37 runs needed a manual re-run.
• The auto post-job QA cron treats a failed eval as done, so a transient kickoff failure silently skips QA for that run.

Action items