Abstract
Abstract
Background.
Code that passes a coding agent's visible tests may still break what was asked for. In Warrant (https://github.com/mv2a/warrant), an open-source protocol and tool, a person states plain-language promises, hidden checks tell the agent only how many passed and which promises they found unmet, and a deterministic gate grants actions such as merge only on evidence tied to an exact code snapshot.
Method.
The study uses 20 small Python command-line tasks, each with deliberately incomplete visible checks, hidden checks, an audit suite never shown to the builder, a reference and broken implementations. Part 1 judged the 160 broken implementations under visible-only acceptance (A) and Warrant's gate (B). In Part 2, a coding agent running the open-weights model gpt-oss:20b built each task once per rule, scored by the audit suite. The evaluation, by Warrant's author, is exploratory and not pre-registered.
Results.
A accepted 93 (58.1%) and B none, but the same generation step produced each task's visible and hidden checks and its defects, so this describes only these suites. In Part 2, A shipped code the audit suite finds broken in 12 of its 18 accepted runs. B had fewer false passes (1 against 12; p = 0.001) and escaped defects (1 against 28; p = 0.003) and more correct deliveries (12 against 6; p = 0.031), all exploratory and largely by construction (review extended the hidden checks toward the audit suite), by chance and because B shipped less. B took about twice the wall-clock time and 2.5 times the tokens.