← back to the counter

A string prompt closed the Agent SDK control channel, silently scoring every agentic verifier 0.0

Summary

The HAL-7441 parallel-grading work (#10531, merged and deployed to verifier-service at 05:05 UTC) made two Claude Agent SDK control-channel features unconditional in the agentic verifier runner: an in-process SDK MCP render tool and a PreToolUse Bash guard hook. But the prompt was still handed to query() as a plain string. On the string path the SDK calls end_input() the moment it sends the message, closing the CLI's stdin — which is the return leg those two features answer over. With an MCP server configured, the connect reply can never be written back, so the run dies with a CLIConnectionError wrapped in an anyio ExceptionGroup. Every agentic verifier (verify_agent, verify_slides_match_agentic) returned 0.00 for ~14 hours. Worse, is_infra_exception() did not unwrap ExceptionGroup, so the crash was scored as a genuine zero rather than flagged as infrastructure failure and discarded — the outage looked like a wall of clean, legitimate 0s. Fixed by streaming the prompt as a one-message async iterator (#10563), which holds the control channel open for the whole run.

Impact

• Every agentic / LLM-judge verifier scored 0.0 from 05:05 UTC onward. These are the workhorse verifiers of the DD benchmark (the big rubrics are 80–107 verify_agent checks each), so any benchmark or QA run graded during the window has agentic subscores that are meaningless zeros, not model performance.
• Silent. Because is_infra_exception() didn't recognize the wrapped crash, the pipeline recorded these as real 0.0 scores instead of discarding the episode. No error surfaced to the grading dashboards; a run's rubric just came back all zeros with an opaque ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) buried in the logs.
• No user-facing site downtime; the damage is corrupted grades, which is in some ways worse than an outage because nothing announced it.

Timeline (UTC)

How it happened

1. Why did agentic verifiers score 0? The Agent SDK run raised a CLIConnectionError ("Tool permission stream closed before response received"), the verifier caught it, and returned 0.0.
2. Why did the SDK connection fail? The in-process SDK MCP render server and the PreToolUse hook answer the CLI over a bidirectional control protocol whose return leg is the CLI's stdin. Stdin was already closed.
3. Why was stdin closed? query() was called with a plain string prompt. On the string path the SDK sends the one message and immediately calls end_input(), closing stdin — fine when nothing needs to talk back, fatal once an MCP server or hook does.
4. Why did we add MCP-server + hook without switching to streaming input? They were introduced by the HAL-7441 concurrency-hardening work (the render tool to serialize LibreOffice, the Bash guard to block raw soffice) and made unconditional, but the prompt-passing call was never revisited — the string path had always worked because nothing used the control channel before.
5. Why was it invisible for 14 hours? is_infra_exception() didn't unwrap ExceptionGroup, so a control-channel crash was classified as a normal verifier result and scored 0.0 instead of discarding the episode as broken infrastructure. The failure mode was indistinguishable from a model genuinely earning zero.

Root cause: new SDK control-channel features (MCP server + hook) were made mandatory on an input path (query(prompt=<str>)) that closes the control channel before they can respond — and the grading pipeline scored the resulting crash as a real 0 rather than an infra failure, so nothing surfaced it.

What surprised us

• Adding a feature broke an unrelated one via a shared, invisible channel. The render tool and the Bash guard both silently rely on the SDK's control channel staying open; passing the prompt as a string (which had always been correct) now closed it out from under them. Neither feature's code changed — the string-vs-stream prompt did.
• A crash scored as a grade. We assumed an exception during grading would be loud — a failed run, a discarded episode. Instead it was laundered into a clean, plausible 0.0. The only tell was an ExceptionGroup line in the logs, and even that hid the real cause behind "1 sub-exception."
• Silence is worse than downtime. A hard outage would have paged us in minutes. This ran for 14 hours producing confident, wrong numbers, and the benchmark sheet would have absorbed them as real model scores.

What went well / what went poorly

Went well

• Once someone looked at a uniformly-zero rubric, the ExceptionGroup + "stream closed before response received" led straight to the string-prompt path; the fix is a true root-cause change (stream the prompt), not a workaround, and it restored the render tool + Bash guard rather than ripping them out.
• The same PR hardened diagnostics: is_infra_exception() now unwraps ExceptionGroup, so the next control-channel crash gets discarded, not scored.

Went poorly

• No canary. We deployed a change to how every agentic verifier talks to the SDK with no post-deploy smoke run that would have caught "they all crash now."
• No alert on the signature: an entire rubric returning exactly 0.0, or a spike in ExceptionGroup, is a bright red flag we didn't watch for.
• Error laundering: the pipeline turned a connection crash into a legitimate score. Infra failures and real zeros must never be indistinguishable.
• The regression rode in on a large, multi-commit PR (parallelization + render tool + hook) where the interaction between "make the hook mandatory" and "prompt is a string" wasn't obvious in review.

Action items

Evidence: verifier-service logs (ExceptionGroup + "Tool permission stream closed before response received"), _agent_runner.py prompt-passing diff across #10531 → #10563, HAL-7441 deploy at 05:05 UTC (7d4c803), HAL-7501 fix at ~19:25 UTC (fdd2f73).