Do AGENTS.md Files Actually Make Coding Agents Better? I Tested the Research

AGENTS.md is becoming a small ritual in software repositories. Put one in the root, explain the layout, list the test commands, add a few rules, and the coding agent should stop behaving like a talented contractor dropped into a building with no map.

I like the idea. A repository has habits that do not fit inside a type system: generated directories that should stay untouched, a test suite with a strange entrypoint, a module that looks replaceable but is public API. Giving an agent that information ought to save time.

Then I read a paper from ETH Zurich's SRI Lab with a less comfortable result. Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? found no general improvement in task success from repository context files, while inference cost rose by more than 20% on average. The paper was presented at the MemAgents workshop at ICLR 2026, where it received an oral presentation and runner-up Best Paper recognition.

The result is not that context is bad. It is that an agent can be very good at obeying context that was never necessary for the job in front of it.

I wanted to see what that looked like in a small, inspectable experiment. The first version of my reproduction taught me an annoying lesson before it taught me anything about AGENTS files: testing the context is harder than adding one.

The question

For a small, well-specified bug fix, does repository context change whether an agent finishes the work? If the outcome stays the same, what changes instead: time, token use, testing behaviour, or the amount of process the agent decides to perform?

I used two historical fixes with regression tests from public repositories:

• openai/openai-agents-python PR #4419: non-finite PCM audio rates such as NaN and inf raised when code called int(rate). The desired behaviour was to take the existing 24 kHz fallback path.
• PrefectHQ/fastmcp PR #4829: a StatefulProxyClient left a nesting reference behind after a persistent session failed, preventing a later reconnection.

For each task, I checked out the pre-fix commit and applied the regression test introduced by the original pull request. The untouched checkout failed the target test. An agent's job was to make the smallest defensible source change until that test passed.

I ran the same agent in three conditions:

Two repetitions per task and condition: 2 tasks x 3 conditions x 2 repetitions, 12 runs total. The order was shuffled with a recorded seed (20260816) so no condition always ran first. Each trial had its own checkout and a five-minute timeout. I recorded whether the target test passed, elapsed time, input and output tokens, shell commands, and test commands.

The part I had to throw away

My first attempt was not clean enough.

Removing AGENTS.md and CLAUDE.md looked like a no-context condition. It was not. One repository also had a .github/copilot-instructions.md; another had a .agents/skills directory full of workflow instructions. The agent found those files. Calling that a no-context run would have been misleading.

So I discarded those batches.

The final harness removes named instruction files and repository-local agent workflow directories before every run: AGENTS.md, CLAUDE.md, Copilot instructions, .agents, .codex, .cursor, and .claude. It then injects exactly one root AGENTS.md for the human and LLM conditions, or nothing for the no-context condition. I also generated the LLM file from a scrubbed copy, so it could not quietly borrow the human document while writing its own.

That cleanup is not a footnote. It changed the experiment from three slightly different repositories into a useful comparison.

One wrinkle remained. The human-written file sometimes told the agent to consult workflow material under .agents/skills. Those files had been removed by design, so the agent occasionally tried the path and moved on. The text alone was enough to send it towards process that was irrelevant to the fix.

What happened

All 12 runs passed their targeted regression tests.

That is the clearest result. On these two small bug fixes, neither kind of repository context improved task completion. The agent could solve both problems without a context file.

The rest of the measurements were less neat, which is why I do not think this supports a grand claim about AGENTS.md.

Input tokens = regular + cached. Cost is illustrative: GPT-5-style pricing applied to codex-default usage, cached tokens charged at full rate. Read it as a relative comparison, not an invoice.

The human context was slowest in every useful sense: about 52% slower than no context on average, with more commands and more test commands. It did not fail. It simply did more.

One OpenAI Agents SDK run shows the gap clearly. The no-context agent found the function, ran the regression test, patched the finite-number check, and finished in 72 seconds. The human-context runs took 159 and 174 seconds. They searched for skills, attempted broader verification, invoked make format, and looked for pull-request material. None of that changed the targeted outcome.

The LLM-generated file produced a different shape. Deliberately short: repository map, test guidance, and constraints around public API and generated documentation. Slower than no context, but fewer input tokens and the lowest pricing-scenario estimate. That might mean a short orientation note is helpful. It might also be ordinary run-to-run variation. Four runs per condition is not enough to settle it.

FastMCP makes the caution obvious. Its no-context runs used roughly 354,000 and 621,000 input tokens. Same task, same condition, different exploration paths. The averages describe these runs; they are not a confidence interval.

Why the human file cost more

The human files were long: roughly 210 lines in FastMCP and 310 lines in the Agents SDK repository. They contained real knowledge about tests, compatibility, release practice, documentation, and repository conventions. A contributor working on a serious change would benefit from much of it.

The problem was fit.

For a bug that needed one guarded int() call or one reference-counter decrement, the agent treated broader engineering procedure as part of the assignment. It inspected extra material, attempted more checks, and prepared for a fuller change process. Sensible obedience, but expensive obedience.

This is where the ETH paper's result feels practical rather than surprising. The researchers evaluated both LLM-generated context and developer-committed context across multiple agents and models. Their finding was not that agents ignored instructions. They usually followed them. The question is whether the instruction earns its place in the task.

What I would put in AGENTS.md

I have not come away thinking teams should delete these files. I would write them with more suspicion of my own enthusiasm.

A useful file should remove ambiguity the agent cannot cheaply resolve for itself: where runtime source lives, where the relevant tests live, which generated directory must not be edited, and one command that establishes a credible local check. If a repository has one strange but important invariant, include that too.

Keep release procedure, pull-request templates, full contributor guidance, and optional review ceremony somewhere else. Those things may matter. They just do not belong in the first context window for every one-line fix.

The test I would apply is blunt: if an instruction does not change what the agent should do on an ordinary patch, move it out of AGENTS.md.

What this experiment does and does not show

It is a small reproduction, not a replication of the paper. Two Python bugs do not represent a large feature, a security incident, a migration, or a repository the agent has never seen before. The task descriptions also named the failing behaviour clearly, making them better regression exercises than open-ended debugging work.

Still, the result is useful because it is concrete. I gave an agent the same two jobs twelve times. Every run succeeded. The large human-written documents bought no extra correctness on those jobs, and they usually bought extra work.

Context is valuable when it answers a question the agent would otherwise answer badly. Everything else has to justify the tokens.

Experiment artifacts

The full harness, task definitions, run schedules, and raw results are in the GitHub repository:

References

• Gloaguen et al. — Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? (arXiv:2602.11988)
• ETH Zurich SRI Lab — MemAgents @ ICLR 2026
• Full experiment repository — Mide6x/AGENTS.md-Research