Testing agents beyond the examples

This is a very small sample dataset of five tasks we built. We asked AI agents to build programs from expert-written requirements and examples. We then tested their code on unseen cases to find where it worked—and where it broke.

On this deliberately difficult set, none of the 25 submitted programs passed every check. In follow-up tests across three tasks, all 15 programs improved when the same test records were reworded to resemble the examples, without asking the AI to fix its programs.

Five experts contributed one executable task each: reproduce a CNC interpreter, reconcile pharmacy records, reduce river field books, certify a fare engine and audit expense claims.

These are red-team evaluations of reliability. They target unfamiliar wording, rare software behavior and interactions between rules. The task files include instructions, examples, environments and graders.

The supplied Claude Opus 5.5 submissions made substantial progress. The closest CNC programs matched 590 of 600 hidden programs. Every fare suite caught 30 of 31 faulty builds and accepted all five correct builds. Yet none of the 25 main submissions earned a full solve.

That result reflects a deliberately difficult selection: solved candidates were dropped during development. Several submissions were regraded after task revisions. The scores describe these failure cases, rather than an average success rate across professional work.

When familiar wording matters

One pharmacy log says “quantity charted 2 in error, actual 1.” Three submitted parsers kept the original quantity; two treated the correction as a void. All five matched the ten public example shifts. The strongest reconciled only five of 48 hidden shifts completely.

To examine that gap, we regraded saved programs on the same hidden events, rewritten in the phrasing used by the public examples. Expected answers stayed fixed. The agent didn’t get another attempt to repair its solver.

Four stream-gauging programs then passed all 60 hidden books; the remaining program passed 58. One expense program passed all 56 hidden claims. Pharmacy improved to 39–42 of 48 shifts.

Our experts noticed that the programs often depended on familiar phrasing. Code that handled the examples could still miss the same kinds of information written differently. In this small sample, rewording the test records helped the same programs get more answers right.

The pharmacy diagnostics preceded a policy clarification about converting drug amounts to units. Six remaining misses involved that form, so they don’t establish failure under the clarified rule.

A different way to present the same work

The pharmacy database control presents the same shifts and answers as structured data. All three fresh attempts solved it. The successful attempts show that these rules were solvable through a structured interface. The log and database versions used separate programs, so the comparison cannot isolate wording from every other difference.

The practical question is which parts of a workflow survive a change in its inputs. The examples below show a specific failure from each task, alongside the required behavior. A fresh evaluation on a frozen release would test whether new attempts reproduce these failures.