When do AI coding frameworks actually help?

Installing an AI coding framework feels like adding discipline. It can ask for requirements, produce a plan, enforce tests, and send reviews to subagents. Vanilla Codex or Claude Code looks much simpler by comparison.

The repeated Codex experiment and published research in this article point to a narrower conclusion: a framework helps when it adds a constraint that the task genuinely needs. On a clear maintenance task, vanilla Codex, vanilla with a concise instruction, and Ponytail all succeeded in three runs. On a task that required interpreting an ambiguous request, the Superpowers condition succeeded more often, but it also received more information and several additional interactions.

Heavier process did not reliably produce better work. Sometimes success improved. Sometimes the result stayed the same while overhead increased. With only three repetitions per condition, these results cannot rank tools in general. They can, however, show what a useful evaluation must measure.

The practical question is not “Which framework is number one?” It is:

What failure am I trying to reduce, and does preventing it cost less than the task itself?

What we compared and how

The work had two stages. An exploratory pilot first applied a broad set of tools to a local pytest bug and a multilingual-tag feature in a static blog. The repeated controlled experiment then used separate public-repository tasks selected after the pilot.

• Clear maintenance: reject an empty Flask blueprint name
• Ambiguous interpretation: explain safe recursive use of p-limit through documentation and an API example
• Excluded complex task: keep blog routes, listings, and SEO metadata consistent

The conditions included vanilla Codex, vanilla Codex with a concise explicit instruction, and the tools below. These projects solve different problems, so I interpreted them as one-shot implementation helpers, conversational approval workflows, long-running artifact systems, or orchestration systems—not as interchangeable competitors.

I no longer had a Claude Code subscription, so direct experiments used Codex only. The Claude Code section relies on official documentation and published research.

The evidence also supports different kinds of claims.

Controlled experiment: clear vs ambiguous tasks

After the pilot, I froze two tasks and ran each condition three times. Ponytail and Superpowers were not tested head to head on the same task. Each was compared with vanilla and concise-instruction baselines on a task aligned with the problem it claims to solve.

• Clear maintenance: vanilla, vanilla + concise instruction, Ponytail
• Ambiguous documentation and API task: vanilla, vanilla + concise instruction, Superpowers
• Each condition used an isolated workspace, a separated run order, and a cumulative 20-minute model-time cap.
• A run counted as an implementation success only if it passed both public tests and hidden validation.
• A separate final response delivered field recorded whether the agent reached and returned its final response. This kept a missing final report separate from a wrong implementation.
• Framework-created subagents were allowed and measured, but no subagent was used to control or score the experiment. No subagent appeared in the valid runs.

The protocol, condition matrix, and analysis constraints were frozen before execution. When runner or scoring problems appeared, I recorded the reason and impact instead of silently overwriting the result.

Raw logs contained local paths and are not published. The per-run values needed to recompute the success counts and medians appear later in this article.

All three conditions solved the Flask maintenance task

Implementation success was identical. The concise instruction had the lowest median runtime, but one run took 9 minutes 20 seconds. One vanilla run took 19 minutes 57 seconds. With only three runs, the median does not establish that the concise instruction is generally faster.

The defensible conclusion is narrower: this task showed no implementation-quality difference that required Ponytail. Ponytail’s missing final report in one run was not an implementation failure, but it was still execution friction worth recording.

On the p-limit task, success and cost rose together

The vanilla conditions generally declared completion while missing at least one hidden requirement. Whenever a condition asked the predefined clarification question, it received the same prepared answer once. The two successful Superpowers runs then received three or four additional prepared approval responses. Their total extra interactions reached four and five. The Superpowers run that never entered the question flow ended after 49 seconds without changing code.

The three Superpowers runs show what the median hides.

The short IDs map to the runId prefixes in the CSV below. attempt: 2 means that the valid run followed an invalid infrastructure attempt.

The exploration indicator was a post-hoc interpretation aid, not a preregistered primary metric. It counts appearances of strings such as rg, find, git status, and ls, minus the first occurrence. It is not a tool-call count or a reliable file-reread count. The 35 and 38 in successful runs versus 8 in the failed run only show that exploration-related logging rose with the longer workflow; they do not prove why the runs succeeded.

These results do not establish that Superpowers is better than vanilla. The treatment combined Superpowers, additional information, and more opportunities to continue. Its condition succeeded in two of three runs while using a median of roughly nine minutes, 4.9 million cumulative input tokens, and four extra interactions.

Comparing medians, Superpowers used 3.9 times the runtime, 20.6 times the cumulative input, and 8.9 times the output tokens of vanilla on this task. Those ratios include the framework, clarification turns, and follow-up work. They are not general estimates of Superpowers overhead. Cumulative input is the sum reported across conversation turns; it is not the context-window size or the billable-token total.

Task selection, prepared answers, and hidden validation were designed by the author. The hidden tests accepted multiple valid forms, but that does not remove designer bias. Each cell also contains only three runs.

Per-run data for all 18 valid runs

The CSV below is the minimum data needed to recompute the tables. durationMs is model runtime. inputTokens is cumulative input reported across turns. success means implementation success, while operational records whether the final response was delivered.

runId,task,condition,success,operational,durationMs,inputTokens,outputTokens,interactions
r1__ambiguous-issue-2__superpowers,ambiguous-issue-2,superpowers,true,true,1015284,4906389,37064,4
r1__ambiguous-issue-2__vanilla,ambiguous-issue-2,vanilla,false,true,141238,216519,4148,1
r1__ambiguous-issue-2__vanilla-explicit,ambiguous-issue-2,vanilla-explicit,false,true,85013,165015,2349,1
r1__benchmark-2__ponytail,benchmark-2,ponytail,true,true,69974,241446,1928,0
r1__benchmark-2__vanilla,benchmark-2,vanilla,true,true,92272,347851,2676,0
r1__benchmark-2__vanilla-explicit,benchmark-2,vanilla-explicit,true,true,559697,183325,1971,0
r2__ambiguous-issue-2__superpowers,ambiguous-issue-2,superpowers,false,false,48672,156560,1707,0
r2__ambiguous-issue-2__vanilla,ambiguous-issue-2,vanilla,false,true,629383,421349,5205,1
r2__ambiguous-issue-2__vanilla-explicit,ambiguous-issue-2,vanilla-explicit,true,true,144004,264380,3992,1
r2__benchmark-2__ponytail,benchmark-2,ponytail,true,false,73670,233617,1975,0
r2__benchmark-2__vanilla,benchmark-2,vanilla,true,true,1196553,330715,2884,0
r2__benchmark-2__vanilla-explicit,benchmark-2,vanilla-explicit,true,true,68522,182503,1702,0
r3__ambiguous-issue-2__superpowers,ambiguous-issue-2,superpowers,true,true,549419,5539327,54029,5
r3__ambiguous-issue-2__vanilla,ambiguous-issue-2,vanilla,false,true,121979,238709,3911,1
r3__ambiguous-issue-2__vanilla-explicit,ambiguous-issue-2,vanilla-explicit,false,true,89419,187127,2507,1
r3__benchmark-2__ponytail,benchmark-2,ponytail,true,true,103841,293447,2183,0
r3__benchmark-2__vanilla,benchmark-2,vanilla,true,true,75441,216617,2006,0
r3__benchmark-2__vanilla-explicit,benchmark-2,vanilla-explicit,true,true,62555,157939,1775,0

Why the third task was excluded

The original design included a third block for the long-running artifact system GSD. Before scoring, however, the executable hidden validator did not match its preregistered SHA-256, and the frozen original bytes could not be restored. Refreezing a validator after seeing results would let the designer choose a favorable scorer, so I excluded the entire block.

Two invalid attempts remain in the research record but are not part of the 18 valid runs. There is therefore no repeated controlled result for GSD. The final scope is two tasks × three conditions × three repetitions. It does not establish statistical superiority or a ranking of complete frameworks.

The harness can fail before the agent does

Several failures had nothing to do with model quality. A missing lockfile stopped setup before a model call. A shallow clone omitted the registered base commit. A scorer expected JSON from pytest and interpreted normal text output as failure. An interactive runner reset the 20-minute timeout on every turn and stopped after the first approval.

I did not count those failures as model outcomes. Pre-model failures became preflight failures. When a post-run scoring bug appeared, the original result stayed intact and the correction used a derived input with recorded hashes and amendments. The invalid GSD block was excluded rather than rerun under a newly selected validator.

This does not make the design perfect. It makes the boundary between model behavior and measurement failure traceable.

What the exploratory pilot revealed

Before the controlled experiment, I applied vanilla, concise instructions, Ponytail, Superpowers, BMAD, GSD, Spec Kit, and Oh My Codex once each to a small bug and a medium feature. The goal was to discover execution patterns and failure modes, not calculate a ranking.

Three observations mattered:

1. Vanilla, concise instructions, and Ponytail fixed the small bug in roughly a minute. GSD and Spec Kit produced plans and specifications while cumulative input exceeded one million tokens.
2. Superpowers and BMAD stopped for questions or approval. That was not necessarily a quality failure; it was a mismatch between a conversational design and an unattended one-shot runner.
3. Conditions that implemented the medium feature passed many self-tests but interpreted the same ambiguous requirement differently. More tests did not automatically make requirement interpretation correct.

Each pilot condition ran only once, and order was not fully randomized. The private acceptance criteria also reflected the designer’s interpretation. After seeing those problems, I rebuilt and froze the tasks and validators used in the 18-run experiment.

Why published studies disagree

The pilot has one run per condition, and the controlled experiment has only three runs per cell. Broader research is necessary, but published results do not point in one direction either.

SkillsBench v4 evaluated 87 tasks across eight domains, 18 model–harness combinations, and three runs per condition. Curated skills raised mean solve rate from 33.9% to 50.5%, a gain of 16.6 percentage points. Effects varied substantially by task and model, and the benchmark is not limited to software engineering. SkillsBench v4

SWE-Skills-Bench focused on software development. It paired 49 skills with roughly 565 tasks using Claude Haiku 4.5 and Claude Code. Average improvement was 1.2%, and 39 of 49 skills did not improve pass rate. Seven well-matched specialist skills improved results by as much as 30%, while three version-conflicting skills reduced them by as much as 10%. Some conditions used up to 451% more tokens without changing pass rate. This was a preliminary, single-model, single-harness study with one run per pair. SWE-Skills-Bench

Both are preprints. They are useful evidence, not guarantees for every product or repository.

Persistent repository instructions show a similar split. A study covering 300 SWE-bench Lite tasks and 138 tasks from repositories with developer-written context files found no overall success improvement and more than 20% higher reasoning cost. Agents followed the instructions, explored more broadly, and sometimes performed irrelevant requirements. Evaluating AGENTS.md

A paired Codex experiment reconstructing 124 small merged pull requests found the opposite operational pattern. With AGENTS.md, median runtime fell from 98.57 to 70.34 seconds, or 28.64%, and median output tokens fell from 2,925 to 2,440, or 16.58%. The study did not score full functional correctness for every patch and manually checked only minimum non-empty relevance for 50 outputs. It is early evidence about efficiency, not improved success. AGENTS.md efficiency study

These findings are not necessarily contradictory. A skill can help substantially when it supplies procedural knowledge that the task needs. It can add cost when it repeats what the model already knows or conflicts with the project’s version. The useful question is whether this skill fills a real information gap in this task.

When frameworks help

Good frameworks do not magically raise model intelligence. They narrow paths where the model is likely to make mistakes.

They preserve recurring team rules

Test commands, file locations, security rules, and completion criteria are worth packaging when they recur. OpenAI describes Codex plugins as a way to bundle skills and apps for repeated work. Codex plugin documentation

The gain here is fewer omissions and more team consistency, not stronger reasoning in the abstract.

Model-friendly tools can improve success

In a 300-task SWE-bench Lite subset using GPT-4 Turbo, SWE-agent solved 10.7 percentage points more tasks with model-friendly search, editing, and feedback interfaces than with a plain Linux shell. SWE-agent paper

Clear search tools, concise error feedback, and safer editing commands can matter. That is different from claiming that more planning documents always help.

Long tasks need external state

Work that spans sessions benefits from explicit artifacts describing the plan, completed items, and next action. Anthropic’s long-running-agent research similarly separates initialization from implementation and leaves structured artifacts between sessions. Effective harnesses for long-running agents

This is the value GSD, Spec Kit, and Ralph Loop are trying to create. On a multi-day feature, documentation can reduce the cost of restarting the next session—if somebody actually reuses it.

Deterministic rules belong in hooks and tests

“Do not modify environment files” is interpreted language. A hook that rejects the change is enforcement. Claude Code’s official comparison likewise recommends hooks for guardrails that must happen every time. Claude Code feature overview

For critical rules, a small deterministic mechanism is often stronger than a long prompt.

When frameworks get in the way

Context is an attention budget, not just capacity

Skill descriptions, agent roles, MCP tools, plans, and test logs all become input to later reasoning. Fitting inside the context window does not mean every token receives equal attention. Lost in the Middle found substantial performance differences based on where relevant information appeared in long contexts. Paper

Anthropic similarly frames context as a finite resource whose token utility must be optimized. Context engineering for AI agents

In the small-bug pilot, cumulative input was 187k for vanilla, 1.047 million for GSD, and 1.149 million for Spec Kit. In the ambiguous repeated task, the Superpowers median was 4.906 million versus 239k for vanilla. These totals sum input reported across turns and include substantial cached input. They are not one context-window size or a direct bill. They do show how much information and interaction the run processed to produce its result.

For a deeper treatment of persistent instructions, see Does AGENTS.md actually help AI coding?.

External skills are a supply chain

A skill may contain not only documentation but instructions and executable code. That creates another prompt-injection path. Skill-Inject evaluated 202 injection–task pairs across 12 models and Codex CLI, Claude Code, and Gemini CLI. Under deliberately malicious skill conditions, attack success reached as high as 80%, including data exfiltration and destructive behavior. Skill-Inject v3

That number is not the malicious-skill rate in real marketplaces. It measures vulnerability to controlled attack skills. It still supports reviewing the pinned commit, instructions, hooks, scripts, permissions, and external connections before installation—and testing at project scope before user-global scope.

Human approval workflows conflict with unattended runs

In the pilot, Superpowers asked about test scope and tag-conflict policy. BMAD produced an implementation specification and waited for approval. In the controlled experiment, continuing the Superpowers approval flow produced two successes in three runs.

That can be safer in an interactive session. In CI or overnight automation, it may produce no code at all. Unattended use therefore needs one of the following:

• put approval criteria in the initial prompt
• define a recommended default when questions arise
• separate planning and implementation into different runs
• avoid a conversational framework for unattended work

Orchestration can become larger than the task

If planners, implementers, reviewers, and validators each reread the repository, you pay repeatedly for the same context. That can reduce risk on long work and become pure duplication on a small edit.

Subagents help when they isolate independent research and return a compact result. They add coordination cost when several agents reread the same files and share the same decisions. Claude Code’s official guidance similarly recommends subagents for isolated work and agent teams for independent work that needs communication. Each teammate is a separate instance and therefore costs more tokens. Claude Code feature overview

A real case where review consumed the work

During this research, review loops continued for 6 hours 36 minutes after the first task-pool draft. At least 14 traceable subagent sessions ran. The design was declared frozen four times. Related documents and code grew to 4,876 lines. The main experiment had still run zero times.

Each reviewer found a local problem, but nobody owned the total time budget or maximum number of review passes. Every subtask could look successful while the overall effort failed. Subagents need time caps, re-delegation limits, process cleanup, and an explicit completion artifact—not just a task description.

Simple baselines are easy to underrate

Agentless used a simple locate → edit → validate pipeline and still produced competitive results on SWE-bench Lite at the time. The authors also categorized weaknesses in benchmark descriptions and reference-patch bias. Agentless paper

In a separate comparison, METR found no statistically significant evidence that Claude Code and Codex outperformed the ReAct and Triframe baselines used in that study. This does not mean the products are poor. It means that a sophisticated dedicated harness is not guaranteed to win every evaluation. METR research note

What this means for Claude Code

I did not run Claude Code, so the Codex results cannot be copied directly onto it. Its official documentation does, however, support a similar selection rule.

Claude Code says its built-in file editing, search, execution, web, and code-intelligence tools handle most coding tasks. It recommends starting with CLAUDE.md for project rules and adding extensions when a concrete need appears. Claude Code feature overview

The documentation does not recommend subagents for everything. It favors the main conversation for fast targeted edits, tightly coupled multi-step work, frequent interaction, and latency-sensitive tasks. Subagents fit bulk log analysis, independent investigation, and work where only a summary needs to return. Agent teams run separate Claude instances and cost more tokens. Claude Code subagent documentation

This is also a useful general principle for Codex, although context transfer, permissions, and stopping behavior differ by product. The purpose of a subagent is not to maximize agent count. It is to isolate independent work or noisy intermediate output from the main context.

Claude Code usage allowance and one session’s context capacity are different limits. See Claude Code usage limits vs context limits for that distinction.

How to choose by task

Start with vanilla Codex or Claude Code when

• the likely edit location is visible
• the feature stays within one or two modules
• tests define completion clearly
• a person will review the result immediately
• speed and a small diff matter

Most such prompts need only four things:

1. the behavior you want
2. boundaries that must not change
3. the test that proves completion
4. whether to ask when blocked or continue with a reasonable default

Add one lightweight skill when

• the same review checklist recurs
• the team has a specific API, security, or migration rule
• one type of mistake keeps recurring
• formatting, tests, or forbidden paths have a narrow scope

A task-specific skill is often better than an entire general framework. Disable unused tool descriptions, start with explicit invocation, pin the commit, and review instructions, hooks, scripts, and permissions together.

Consider GSD, Spec Kit, BMAD, or Ralph-style workflows when

• work spans several days or sessions
• requirements must remain traceable to implementation
• teammates genuinely review the specification and plan
• work can be divided into independent units
• generated plans, PRDs, and checklists will be reused

Artifact reuse is the deciding factor. A spec.md or plan.md that nobody reads is context waste, not a quality mechanism.

Pilot it on 10 tasks before adopting

Do not decide from GitHub stars or a demo. Use recent work from your own repository.

1. Select five small bugs and five medium features.
2. Compare vanilla with one candidate framework. Do not stack several at once.
3. Mix execution order and keep permissions and time limits equal.
4. Record success, human interventions, wall-clock time, total and cached tokens, and diff size.
5. One week later, check whether anybody reused the plans or review artifacts.

One run per task is not enough to claim superiority. Repeat across days, mix external public tasks with your own, and freeze scoring rules before implementation.

The 18 valid runs in this experiment used about 85 minutes 27 seconds of model runtime in total. That excludes installation, calibration, invalid attempts, failure recovery, scorer fixes, and manual review. Applying the same average to 66 runs gives roughly 5 hours 13 minutes of model runtime alone.

Interactive conditions and 20-minute caps also create high variance, so real project time can easily exceed half a day. Repetition counts need a feasibility budget and stopping rules, not only statistical ambition.

When to turn the framework off

Reduce or remove the framework when:

• plans and review files grow but no source diff appears
• the same question or validation repeats
• subagents duplicate repository exploration
• a small task uses several times the vanilla input and interaction
• an unattended run waits at a human approval gate
• framework installation, authentication, or runtime debugging takes longer than the task
• self-tests grow while the same real acceptance condition still fails

“Keep going until done” is especially risky. It can be interpreted as permission to repeat, not a termination condition. Pair it with a time cap, maximum iteration count, a stop when no implementation diff appears, and a condition for returning control to a person.

Frequently asked questions

Should I use Superpowers or Spec Kit for a small bug?

If the edit location and failing test are visible, start with vanilla and a concise acceptance criterion. Add a narrow skill only when the bug needs design clarification or the same failure keeps recurring. A full specification workflow is hard to justify if its artifacts will not be reused.

Is installing more skills always better?

No. Loading and invocation scope matter more than installation count. Persistent instructions, automatically selected skills, and exposed tools can increase exploration and validation when they do not apply. Start with explicit invocation and automate only skills that repeatedly help.

When is a subagent better than the main agent?

Use one for independent work such as bulk-log analysis or reading several documents and returning a short synthesis. Keep tightly coupled planning, implementation, and testing in the main context. Check dependencies and duplicate exploration cost before parallelizing.

How should I measure framework value in my repository?

Mix recent small bugs and medium features. Freeze order, permissions, time limits, and scoring before execution. Record human intervention, cumulative input, runtime, diff size, and artifact reuse as well as success. Look for a recurring failure that consistently decreases, not a single win.

Conclusion

AI coding skills and frameworks are not quality bonuses added on top of a model. They are control mechanisms that spend time, context, and interaction to reduce a specific failure.

For small and clear work, start with vanilla Codex or Claude Code plus a concise acceptance criterion. Add one narrow skill when a team rule or failure repeats. Consider specification, planning, and subagent frameworks first for work that spans sessions and produces artifacts people will reuse.

The best setup is not the one with the most features. It is the smallest setup that prevents a failure you first observed with vanilla.

Sources and further reading

The sources cited throughout the article are grouped here for easier verification.

Research and benchmarks

• SkillsBench v4 — average effects of curated skills across multiple domains
• SWE-Skills-Bench — software-development skill success and token cost
• Evaluating AGENTS.md — repository context files, success, and reasoning cost
• AGENTS.md efficiency study — runtime and output tokens in reconstructed pull requests
• SWE-agent — model-friendly agent–computer interfaces
• Lost in the Middle — information position in long contexts
• Skill-Inject v3 — prompt-injection risk through external agent skills
• Agentless — a simple repository repair pipeline
• METR research note — Claude Code, Codex, and existing harness baselines

Official documentation

• OpenAI Codex plugin documentation
• Anthropic: Effective harnesses for long-running agents
• Anthropic: Effective context engineering for AI agents
• Claude Code feature overview
• Claude Code subagent documentation

Tools compared

• Ponytail
• Superpowers
• BMAD
• GSD
• Spec Kit
• Ouroboros
• Everything Claude Code
• Oh My Codex
• Ralph Loop