TOWARD
INSTRUCTIONS
AS CODE.

148 GitHub projects, 15,549 agentic pull requests, one question: does writing an AGENTS.md or CLAUDE.md file actually make Copilot, Claude, and Cursor produce better pull requests? The answer is not a clean yes. 27.7% of projects saw their merge rate jump by 20% or more after adding instruction files and 26.35% saw it drop by just as much. The difference wasn't whether a file existed, it was how much was in it: projects whose merge rate improved had instruction files nearly twice as long, with far deeper header structure, than projects whose merge rate got worse.

Authors Ali Arabat, Mohammed Sayagh

Affiliation École de Technologie Supérieure

Published MSR '26 arXiv 2606.13449

Core Finding Verbosity & Structure, Not Existence

Agentic Pull Requests◆ AGENTS.md◆ CLAUDE.md◆ Copilot Instructions◆ Merge Rate◆ Code Churn◆ Instructions-as-Code◆ Mining Software Repositories◆ AIDev Dataset◆ Agentic Pull Requests◆ AGENTS.md◆ CLAUDE.md◆ Copilot Instructions◆ Merge Rate◆ Code Churn◆ Instructions-as-Code◆ Mining Software Repositories◆ AIDev Dataset◆

§ 3.0 Four Questions, 148 Projects

Every project in the dataset got compared against itself agentic PRs before its instruction file existed versus agentic PRs after across success, complexity, and effort.

RQ1

Does merge rate improve after instruction files?

27.7% of projects increased merge rate by ≥20% after the first instruction file; 26.35% decreased by the same margin. With every instruction file counted, increases rise to 31.93%, decreases to 27.73%. Roughly equal odds of helping or hurting.

RQ2

Does task complexity increase?

35.35% of projects significantly increased PR description length, 10.10% increased code churn, 12.12% increased commit count outnumbering the projects that saw each metric decrease (4.04%, 7.07%, 7.07% respectively).

RQ3

Does merging get easier or harder?

13.13% of projects saw merge time significantly increase and 15.15% saw discussion volume increase, while 8.8% saw both metrics drop instruction files can add friction as often as they remove it.

RQ4

What separates the projects that improved?

Instruction files in merge-rate-increasing projects ran a median of 976 words versus 569 in decreasing projects and were structured with roughly 9 H3 subsections versus just 1. Depth and structure, not just presence, predicted the outcome.

§ 2.2 Every Agent Reads a Different File

There's no single standard yet each coding agent looks for its own filename and location, which is exactly why a project with multiple agents in play often ends up with multiple instruction files stacked on top of each other.

Agent	Where it looks
Devin	**/PULL_REQUEST_TEMPLATE/DEVIN_PR_TEMPLATE.md
Cursor	.cursor/, .cursorrules, /.mdc
Copilot	.github/copilot-instructions.md, .github/instructions/*
Claude	.claude/, CLAUDE.md, .github/workflows/claude.yml
Codex	/AGENTS.override.md, /TEAM_GUIDE.md, **/.agents.md
Common	**/AGENTS.md the closest thing to a cross-agent standard

§ 3.1 Four Real Repositories, Two Outcomes

Aggregate percentages hide how dramatic individual swings can be in either direction.

Project	Before	After	Change
NethermindEth/nethermind	20.00%	69.23%	+246.15%
Significant-Gravitas/AutoGPT	42.85%	65.30%	+52.38%
theopenco/llmgateway	89.00%	66.66%	−25.10%
Azure/adx-mon	75.00%	35.35%	−52.87%

Nethermind's instruction files turned a one-in-five merge rate into better than two-in-three. Azure/adx-mon's instruction files cut a strong 75% merge rate by more than half. Both projects did the same thing added an instruction file and got opposite results, which is the entire reason this paper exists.

§ 4.0 What Actually Separates Them

Comparing only the projects with a clear ≥20% merge-rate swing in either direction isolates the one variable that actually correlated with the outcome: how much was written, and how it was organized.

Merge Rate Increased ≥20%

976 words

Median instruction file length. Structured with a median of 3.5 H1, 7.5 H2, and 9 H3 sections fine-grained, deeply nested guidance.

Merge Rate Decreased ≥20%

569 words

Median instruction file length nearly half as long. Structured with a median of just 1 H1, 5 H2, and 1 H3 section shallow, underspecified.

Statistical confirmation Mann-Whitney U test word count: p = 0.0029 // significant difference in H1, H2, and H3 header counts too, all with small effect size

§ 5.0 Instructions Are Code Now

If thin, unstructured instruction files correlate with worse outcomes and thick, well-organized ones correlate with better outcomes, the implication is direct: writing AGENTS.md shouldn't be a five-minute afterthought. It needs the same review, iteration, and quality bar as the code it's meant to guide versioned, structured, and treated as a first-class engineering artifact rather than a one-time README nobody revisits.

TOWARD INSTRUCTIONS AS CODE.

TOWARD
INSTRUCTIONS
AS CODE.