Delete a Section and Watch the Agent

Which lines in an agent's instruction file earn their place? Delete one and rerun.

9 minute read

A developer at a desk with a long markdown file open on one monitor and a terminal running a set of agent tasks on the other.

Most agent instruction files grow the way a legacy README grows with each line or section going in after something goes wrong (a “lesson”). However, almost nobody goes back to ask whether it’s still valid. My own agent setup had the same problem and I’d guess yours does too.

The usual advice is to give an agent more context, including architecture notes and conventions. Thoughtworks put the other side of that on its Technology Radar this spring. The April 2026 edition lists Context engineering under Adopt and Agent instruction bloat under Caution. The bloat entry describes how files grow by accretion, with each addition sensible on its own while the total gets long, contradictory and easy to skim past.

First the research, then a method I’d use.

Can a longer file make things worse?

A long instruction file has at least three hidden costs.

The first is position. Nelson Liu and colleagues tested language models on questions that depended on information buried in long inputs. Their Lost in the Middle paper, published in TACL in 2024, found a U-shaped curve: accuracy was highest when the relevant text sat at the start or end of the input and dropped when it sat in the middle. The effect showed up even in models built for long contexts. The scale is worth noting, because “long” in that study is nowhere near what today’s model cards advertise. The question-answering tests used 10, 20, or 30 Wikipedia passages of roughly 100 tokens each, so the inputs ran a few thousand tokens, and the longest-context model in the test handled 100K. Nothing here says a 256K or 1M-token session collapses. It says the middle of an input is where attention weakens, and an instruction file lives entirely inside that middle. Those were question-answering tests, not coding tasks, so I’d read the result as a mechanism to watch for, not a measurement of your repo.

The second is distraction. Chroma’s Context Rot report, published in July 2025, tested 18 models, including several Claude models. The authors report that performance consistently degrades as input length grows. A single distractor passage, related to the topic but not the answer, lowered accuracy compared with the relevant passage alone, and four distractors compounded the drop. The authors also found that focused prompts of a few hundred tokens beat full prompts near 113K tokens on a conversational-memory benchmark. Claude models showed the lowest hallucination rates and tended to abstain, while GPT models hallucinated the most. In an instruction file, the everyday version is a section of retired deployment notes that nothing points at anymore. Every run still pays to read it.

The third is cost without benefit. A February 2026 paper, Evaluating AGENTS.md by Gloaguen and colleagues, asked whether repository context files help coding agents on real tasks. The answer was that context files do not generally improve task success rates, and they raise inference cost by over 20% on average. The agents followed specific instructions well. Repository overviews, the kind model providers often recommend, did not help. The files were useful for non-standard practices the agent could not infer from the code. The purest case is a long tour of a conventional folder layout the agent could reconstruct by listing the directory. It passes no test, and it’s paid for on every run. Keep the lines that tell the agent something the code can’t show it. Everything else is up for testing.

20%+
average inference cost increase from repository context files
with no general improvement in task success rates
Evaluating AGENTS.md, Gloaguen et al., February 2026

Running the Deletion Test

Hamel Husain’s AI Evals FAQ, last updated in September 2026, is written about LLM judges, but the method transfers directly. Asked how much context a judge should get, he answers:

“Give each judge only the parts of the trace it needs for its failure mode.” He also warns that “Extra context can cause context rot and make the judge worse.”

He calls the technique an ablation, machine learning’s term for removing one component and remeasuring. Strip the vocabulary and it’s a deletion test: take one section out, rerun the work, and see whether the checks notice. If they hold steady or improve, the section can likely go.

Applied to an instruction file, the framework has three parts: a fixed task set, a small harness that runs it, and a few rules that keep the results honest.

The task set. Pick ten to twenty tasks that represent the work the agent actually does on this repo, and give each one a check that returns pass or fail. That might be a test suite, a grep for a required convention, a rubric applied to the diff, or your own label. On a service codebase, that could look like:

  • “Add a DELETE endpoint to the orders service following the existing handler pattern,” checked by the route tests plus a grep for the error-envelope helper.
  • “Explain where request retries live,” checked by whether the answer names the right module without reading five files first. A human label is fine here.
  • “Rename a configuration key across the repo,” checked by the full test run.

Weight the set toward the tasks where your rules actually matter. A rule that guards against a rare failure or an edge case will never show its value on tasks where that failure can’t happen, so the set has to include some of those on purpose. Most of the value sits in this step.

The harness. The loop is boring on purpose, and a couple hundred lines of script is enough:

  1. Start from the full file and record a baseline: every task, run several times, recording pass or fail plus tokens for each run.
  2. Remove one section (match on its heading), write the trimmed file, and rerun the same task set the same way.
  3. Restore the section, move to the next one, and repeat until every section has had a turn.
  4. Point a simple report at the results: score change per section, token cost per section.

Most agent CLIs have a headless mode (claude -p, codex exec), so the harness can shell out to the same binary you use by hand. The one rule that matters most: run every task in a fresh context session. A session that already saw the full file, or the previous task’s answer, will leak that context into the result, and you’ll credit or blame a section for something the conversation history did. New session, same prompt, every time, including the baseline.

The rules. Keep the task set and the checks frozen for the whole run. Several runs per task per variant, and record the spread rather than one number. One section out at a time, restored before the next. Log the tokens each run consumes along with the pass or fail, because the AGENTS.md study was about cost as much as success. A section that adds nothing to the score but adds tokens to every run still costs you each time.

Section Deletion Test Decision FlowFlowchart showing how removing an instruction section determines whether it is kept, deleted, or audited.1. Fixed Task Suite10-20 checkable runs2. Single Section CutDelete 1 heading & rerunScore DropsLoad-bearing: keep it and condense wordingScore Flat (No Change)Bloat candidate: delete to cut tokens & rotScore ImprovesConflicting rule: audit against evaluation tasks

Two limits are worth keeping in mind. One-at-a-time deletion misses redundancy. If two sections say the same thing, deleting either one alone shows no change, so each looks dispensable. Test pairs of sections that cover the same topic together. The second is that results belong to the models and tasks you tested. A section that did nothing on an earlier model may matter after a model change, so rerun the set when the model changes or you move between models (e.g. Claude to GPT to GLM).

Managing the memory around the agent

The same logic applies to everything the agent reads at the start of a session, not just the main file. Stale lines are harder to find than bloat. A rule that was true last iteration or with a previous feature set can contradict the rule you wrote yesterday, and the agent will pick one, not necessarily the current one.

Here’s a quick rundown of how I tend to structure my projects.

  • AGENTS.md is the canonical project policy.
  • Skills live in one folder with an index at .\agents\skills. A task loads the one skill it needs, plus the reference files that skill points to, and nothing else.
  • Roles (subagents) live in .\agents\roles with an index.md in the root that explains each agent role and responsibility.
  • Project memory lives in its own folder at .\agents\memory with a MEMORY.md index. Each entry is a short file with a date and a reason. The index holds one rule that matters most. When a lesson is stable, fold it into the skill that owns it and delete the memory entry.
  • The generated tracking files run past 100KB. The rule is to grep them by keyword and never load them whole.

Anthropic’s documentation points the same way. The Claude Code memory docs recommend keeping each CLAUDE.md under 200 lines, note that longer files use more context and reduce adherence, and say that Claude treats these files as context rather than enforced configuration. Auto memory loads only the first 200 lines or 25KB of its notes.

I’d hold any agent setup to these practices:

  • One canonical source. Whatever else reads instructions imports or symlinks the same AGENTS.md. Don’t copy it, because two copies drift.
  • Dated entries with the reason attached. The date tells you when to doubt a rule. The reason tells you whether the situation behind it still exists.
  • Fold, then delete. A lesson that lives in memory forever becomes a second source of truth.
  • Index first, body on demand. The agent reads a one-line pointer and opens only the matching file.
  • Prune on a schedule and after every model change. Use the same deletion test as above, not your gut.

The AGENTS.md format is now an open standard stewarded by the Agentic AI Foundation under the Linux Foundation, which makes it portable across tools.

I wrote about the loop around the agent in Building the Loop Around the Agent, and about where instructions sit in the wider setup in The Harness vs. the Interface. A loop without a pass-or-fail signal can’t tell you whether a rule helped.

Where to start

An instruction file is a set of bets about what the agent needs. Each line is a bet, and you can test it by deleting the line and rerunning the work that was supposed to depend on it. If a line loses nothing when you delete it, it’s costing tokens and attention on every run.

This works best where the work has checks you trust. Without one, the deletion test is a vibe test, and you’d be deleting lines based on how a single run felt. Where the output can’t be checked, I’d keep the file short and treat every new line as a cost it has to justify.

I’m still unsure how many runs per task are enough, and whether section-level deletion is the right grain for every file. If you run this on your own files, I’d like to hear which sections turned out to matter.