← back to the counter

Legitimate Harbor traffic tripped AWS fraud detection, which suspended CodeBuild account-wide

Summary

Harbor run volume ramped hard over Aug 2–4, pushing CodeBuild spend to $9,835.86 in three days (peak $7,928/day, ~98% of it BUILD_GENERAL1_XLARGE minutes — the tier multi-trial Harbor runs require for memory). AWS's fraud systems read the spike as stolen-credential abuse ("your AWS Account may have been inappropriately accessed by a third-party") and suspended the account's CodeBuild access: in-flight builds were killed with CLIENT_ERROR: access suspended, and new StartBuild calls bounced with AccountLimitExceededException: Cannot have more than 0 builds in queue for the account — the suspension is literally implemented as a quota set to zero. Access was restored automatically ~3.5 hours later, before we had done anything. A security review found no evidence of actual compromise: no unexpected IAM users, keys, roles, or policy changes in CloudTrail; every build initiated by our own automation principal; root MFA enabled.

Impact

• 24 builds killed mid-run with the suspension error (15:54–17:01 PDT); all StartBuild calls rejected 18:43–19:02 PDT (observed window — the gap likely extends earlier). Every Harbor problem run, GT test, and environment image build in the window failed or could not start.
• Victoria's and Rob's Harbor runs surfaced as unexplained failures in Horizon, with no hint that the cause was account-level.
• ~$9.8k of CodeBuild spend Aug 2–4 — legitimate, but nobody had sized or approved the ramp, and the anomaly alert that would have flagged it day-one went unread.
• An urgent AWS case (178588806900392) now requires a written security-confirmation response by 2026-08-09, else AWS may suspend the entire account and terminate resources.

Timeline

How it happened

1. Harbor runs execute on CodeBuild (horizon-gt-test project), overriding compute per build: LARGE for single-trial runs, XLARGE for multi-trial runs, which need >16 GiB (harbor-runner.ts). Run volume grew ~27× in spend terms over a weekend.
2. AWS's abuse heuristics are tuned for exactly this shape — a quiet account suddenly burning thousands of dollars a day of big Linux build minutes is the canonical stolen-key crypto-mining signature. With no usage history or account-team context to say otherwise, AWS assumed compromise and pulled the brake.
3. The early warning existed and fired on time: Cost Anomaly Detection emailed a day before the suspension. But billing alerts go to a single inbox with no routing to Slack or on-call, so the first human signal was two engineers' runs failing.
4. The structural cause: nothing in our process treats "spend profile changed dramatically" as an event anyone — us or AWS — should be told about in advance.

What surprised us

• AWS suspends first and emails second. In-flight builds died ~1h before the "[Action Required]" email was sent.
• A suspension is a quota. There's no distinct "suspended" API state — enforcement is your concurrency quota set to 0, so it presents as AccountLimitExceededException, which reads like a self-inflicted scaling problem, not an account-security action. (First guess was exactly that: "did we hit a quota limit?" — mechanically yes, but the quota was 0 because Trust & Safety set it there.)
• It un-suspended itself. Access was restored ~3.5h later with no action on our side — while the case still demands a response on a 5-day fuse.
• The XLARGE split is structural, not a bug: 98.5% of anomaly spend was XLARGE because multi-trial runs are the common case and CodeBuild has no on-demand tier between 16 and 72 GiB.

What went well / what went poorly

Well

• Diagnosis was fast once a human looked: build phase errors named the suspension outright, and the AWS emails + CloudTrail confirmed cause, window, and (absence of) compromise within ~20 minutes.
• The security review came back clean, with evidence: no IAM mutations since Aug 1, no unknown principals, all 100 recent builds initiated by our own automation identity, root MFA already enabled.
• AWS restored service quickly and on their own.

Poorly

• Two AWS emails (anomaly day-one, then the urgent case) produced zero operational response; users were our monitoring.
• Horizon surfaced the failures as generic run errors — nothing distinguished "your run broke" from "the account is suspended."
• We had no relationship or prior note with AWS that would have let their systems classify the ramp as expected.

Action items