Evidence
Evidence for the AI coding framework experiment
This page publishes the run counts, condition-level results, and limits behind the numbers in the AI coding framework comparison.
What was run
Two tasks were repeated under several Codex conditions. Claude Code was not run here; its section is kept separate as an interpretation of official documentation and published research.
- Valid runs
- 18
- Excluded blocks
- C
Results by condition
| Task | Condition | Implementation success | Median runtime |
|---|---|---|---|
| Ambiguous interpretation | Superpowers | 2/3 | 549.4s |
| Ambiguous interpretation | Vanilla Codex | 0/3 | 141.2s |
| Ambiguous interpretation | Vanilla Codex + concise instruction | 1/3 | 89.4s |
| Clear maintenance | Ponytail | 3/3 | 73.7s |
| Clear maintenance | Vanilla Codex | 3/3 | 92.3s |
| Clear maintenance | Vanilla Codex + concise instruction | 3/3 | 68.5s |
What this evidence cannot show
- Each cell has three runs on one task; no statistical superiority or causal effect is claimed.
- Block C was excluded after its executable validator bytes failed the preregistered commitment.
- Post-run scoring corrections are append-only amendments in the public protocol.
- Claude Code was not executed; Claude-related statements require external research.
Raw logs and workspaces contain local paths and controller-only material, so they are withheld. The JSON below retains the run-level results needed to recompute the article's tables and its stated limits.