A note on this one. This essay is a summary of my preprint, Cross-Repository Guidance for Coding Agents: A Single-Trial Replication Addendum (PDF), and the summary was largely written by AI. My other essays aren’t written this way; for examples of those, see A Model’s A Priori Is Frozen A Posteriori and Emergent Misalignment is Co-occurrence Density Made Visible.
In Probe-and-Refine Tuning of Repository Guidance for Coding Agents, a coding agent gets a short AGENTS.md file that has been refined against the repository it is about to work in. The procedure generates synthetic bug-fix probes, watches the agent attempt them, and edits the guidance based on what went wrong. It works, but it assumes you can run that loop on the target repository first.
This follow-up asks a narrower question. Can you get comparable guidance without ever tuning on the target, by borrowing what the loop learned on other, structurally similar codebases? If most of what probe-and-refine teaches is general operating discipline (reproduce the bug first, gather evidence before patching, navigate by tracing rather than symbol search), it should transfer.
The pipeline
- Mine. Run the unchanged probe-and-refine procedure on 50 active Python repositories that are not in SWE-bench, spread across eight domains from web frameworks to scientific computing. Keep every edit it merges into the guidance.
- Filter. Of 760 distinct edits, drop the ones that name a file path, a project-specific symbol, or a project-specific command. A regex pass catches the obvious cases and an LLM judge handles the rest, leaving 496 generic edits. A sympy-specific rule would actively mislead an agent working on Django.
- Match. Describe every repository with eight simple categorical features (test framework, linters, size, docstring style, import style, type-hint coverage, hub density, test-directory layout), weight them, and pick the ten most similar source repositories for each target.
- Assemble. Start from the same structural scaffold the original paper uses, then add generic edits from the most similar sources until the file reaches the same 3,000-character budget.
The target repository is never seen by the tuning loop. The mining step is the expensive one, and it is reusable: a repository mined once can supply edits to any later target.
What happened
On SWE-bench Verified (500 instances, Qwen3.5-35B-A3B, 200 steps, one trial), the three conditions resolved:
- No guidance: 198 / 500 (39.6%)
- Cross-repo guidance: 227 / 500 (45.4%)
- Per-target probe-refined guidance: 240 / 500 (48.0%)
Cross-repo guidance beats no guidance clearly (paired McNemar p = 6.4×10−4), and the gap is larger than the one the static knowledge base produced in the original paper, even though none of these edits ever touched the target. The gap to probe-refined guidance, 2.6 points, is not significant at this sample size (p = 0.118).
The mechanism matches the original paper too. Per-patch precision barely moves across conditions (52–54%). What changes is how often the agent produces a patch that can be evaluated at all: 380, 420, and 441 of 500. Malformed diffs drop from 51 with no guidance to 13 with cross-repo guidance, and some of the transferred edits literally tell the agent to check its diff syntax and reject empty patches. Guidance helps the agent finish, more than it helps the agent be right.
The per-repository breakdown suggests where the remaining gap lives. On Django, which is nearly half the benchmark, cross-repo and probe-refined guidance tie. The probe-refined advantage is concentrated in sympy, sphinx, astropy, and xarray, the scientific and numerical codebases with unusual internal layouts. That is the one thing cross-repo guidance can’t provide by construction: instructions that name where to look in this codebase. If that is the cause, adding more source repositories won’t close the gap.
What it doesn’t show
This is one trial. It supports “cross-repo guidance reliably beats no guidance.” It does not support “cross-repo guidance is as good as tuning on the target.” A 13-instance gap in the expected direction is consistent with both a real difference and no difference, and settling it would take roughly three or four more paired trials.
The other limits:
- Everything is specific to one model and one scaffold. The original paper showed that guidance tuned for one model can hurt another, and this guidance was mined with the same model.
- The similarity score was chosen to be readable, not tuned. It may be that test-framework match does all the work, or that any rough “modern Python library” match would do. A feature ablation is the obvious next experiment.
- Half of the assembled files draw from a single source repository, because the most similar source tends to fill the budget first. Whether forcing diversity helps is untested.
The practical takeaway is modest but useful: a few thousand characters of operating advice, borrowed from codebases the agent will never see, is enough to help. The pipeline is documented so the multi-trial replication that would settle the harder question is straightforward to run.