Does a Test Perspective Library Help a Coding Agent? Measuring It at Bug Detection, Implementation, and Design

In the previous piece, I used controlled experiments to find the conditions under which accumulated decision records (ADRs) help a coding agent. My repository has one more body of knowledge I’ve been growing. It’s a test perspective library (TPL): failure patterns that could recur, generalized from past bugs and stored one perspective per file.

The TPL and the ADRs have separate jobs. An ADR is a record of the past: “this is what we decided.” A TPL entry is an instruction for the future: “this is what you should verify.” The library has grown to 70 entries, and I’ve kept up the practice of consulting it when designing new features and when designing acceptance tests.

So does this accumulation actually change what a coding agent produces? I measured it the same way as last time. I cloned the repository and removed from one side the 70 TPL entries along with the paths that lead to them. Those paths are more than 200 files: the relevant sections of the agent’s standing rules, the PR template, cross-links from the specs, and so on. Then I gave both sides the same request, over and over¹.

Summary

The TPL raises the completeness of specs at the design stage. Perspectives distilled from past failures fill “spec gaps that are written down neither in the code nor in general knowledge yet.” That is the core of the value of keeping a TPL.
It does not help bug detection in code review. Removing it doesn’t lower the detection rate. The agent re-derives the perspective on its own.
Nor does it help cover routine implementation. The existing code already enumerates “where things need to be filled in,” and the agent copies that enumeration as-is.

Below, I walk through the experiments that led to those three lines. I’ll show the two nulls first. The mechanism behind them is what explains why the difference in the third experiment is real.

Experiment 1: plant a bug and ask for a review

The purpose of the TPL is to prevent recurrence. The most direct measure, then, is to plant a fresh bug of the same shape as a failure pattern the TPL is supposed to guard against, and see whether review catches it.

The perspective I aimed at was: “when the same vocabulary is held in several representations, updating only one of them makes them drift silently.” Concretely, the appearance of each node type (kind) is enumerated independently in two distant places, the SVG renderer and the draw.io exporter. I prepared a change that adds a new kind and dressed it up as a “feature PR” in which only the exporter side was forgotten. Type checking, lint, and the roughly 3,700 existing tests all pass. Since no machine check can catch it, the only thing that could detect it is knowledge of the perspective. That was the intent of the design.

The request was just “review whether this commit can be merged,” with no hint pointing at the perspective.

Every trial detected the bug, with or without the TPL. Reading the transcripts, the agents in both conditions did the same thing. They searched the whole repository for the new kind’s string, listed every place the vocabulary appears, and found the gap in the exporter’s enumeration.

You could object that a missing entry is bound to be found by search. So in round 2 I switched to a bug that looks “wired up” to a search: the entry exists, but its value was copied over from another kind and is wrong. This too was detected in all 5 trials of each condition. The agents didn’t stop at finding the representations; they compared the values. The TPL’s checklist says “run every representation through the same input and compare the results,” and the agents in the condition that never read it did exactly that on their own.

The interesting part is that the agents in the condition with the TPL went to read it in every trial. The paths leading to it work. They read it, and still detected at the same rate as those who didn’t. The record exists, gets read, and still doesn’t change the results. This is a different kind of null from last time.

Experiment 2: measure implementation coverage

If review doesn’t show it, what about prevention? I stopped planting bugs and switched to an implementation request: “add a new kind and finish it to a mergeable standard.” There are 15 correct places to wire it into, and I compared, with and without the TPL, whether the agent would fill in the distant exporter and the Visual Studio Code extension on its own.

No difference here either. Every trial wired it through to the exporter even without the TPL. The transcripts show the reason plainly. Before starting, the agent searched the whole repository for where existing kinds appear, and in one trial it found a past “add a kind” commit in the Git history and read it as a blueprint. The existing code is itself “a complete list of where to fill things in,” so the TPL has nothing to add.

The two nulls reduce to one mechanism. For problems whose answer is enumerated in the code, a thorough agent re-derives the perspective on its own. Detection and coverage were both within reach of habits the agent practices without being told: searching for vocabulary and mirroring precedent.

Had I stopped the experiment here, the conclusion would have been “the TPL isn’t needed.” But the moments I had felt the TPL pay off weren’t in review or implementation. They were at the design stage, when I closed a spec gap that would have become a bug before it did. When strengthening feature A ripples into feature B, that ripple isn’t written anywhere in the code yet. In a place where the code holds no answer, a different measure might show a difference.

Experiment 3: have it write a design doc

I changed the request to “Don’t implement. Write a design doc and acceptance criteria.” The subjects were features that have new interactions with existing features and no precedent in the repository to mirror. The first was a display feature: “expand just one container inline within the diagram.” The second was session restore: “when the app is reopened, restore the previous state.”

Scoring was a blind judgment, on 6 items drawn from the relevant TPL checklists, of whether the design doc lays down “a rule the implementer won’t have to guess at”².

This time a difference appeared. On the first subject, the 6-item average was 5.2 vs. 4.2. On the second, 5.7 vs. 4.0, again in favor of the side with the TPL.

More telling than the averages is which items split and which didn’t.

Items like “set an upper limit on how many can be expanded at once” and “define the operation to collapse it again” were written by every trial in both conditions. A good designer writes those from common sense. What split was the item “put into the acceptance criteria that an element being expanded is neither drawn twice nor dropped.” It appeared in 5 of 5 trials with the TPL and 0 of 5 without³. This perspective was distilled from a layout bug that actually happened, and it is written only in the TPL. Every trial with the TPL read the relevant entry, and the perspective it read showed up directly in the spec.

As the footnote says, the scale is small, but the second subject reproduces the same split. In the restore feature, the items that differed came from past incidents, such as “distinguish corrupted saved data from no saved data (don’t overwrite corrupted data as if it were empty).” And here I got one observation that shows what the boundary really is.

The restore feature has a well-known trap: “the app’s initialization rolls back the state you just restored.” This had been written up in detail as a code comment when a past bug was fixed. Every trial, including the condition without the TPL, reached that comment, and in both conditions everyone put this item into the spec. Within the same round, the perspective written in the code was picked up without the TPL, and only the perspectives not written in the code split on whether the TPL was present.

Translating the boundary into practice

The boundary running through all three experiments can be stated like this:

A perspective finishes its job as a TPL entry the moment it is written into the code. The TPL helps only while a lesson exists neither in the code nor in general knowledge.

That sounds like it narrows the TPL’s value, but in fact it lightens how you run it.

First, you don’t need to maintain the TPL for the sake of thorough agent reviews. The agent carries detection and implementation coverage through without the perspectives. Building in a practice of “check the TPL before review” brought no return, at least not in the results.

That applies, though, only when the reviewer is an agent. Detection without the perspective worked through brute force no human has: searching the whole repository for new vocabulary and listing every representation in a few minutes. You can’t extrapolate this null to a human reviewer who mostly reads the diff. The TPL’s value as a checklist for people wasn’t measured here, and it’s more natural to assume it remains. The same goes for the writer’s side: the act of generalizing a bug into a perspective makes you understand the failure as a structure, and that value is also outside this experiment.

The TPL belongs at the entrance to the design phase. My repository’s standing rules include a path, “check the relevant perspectives when writing a design doc,” and in the experiment every trial with the TPL reached the perspectives through that path. In the ADR experiment I wrote that “a single sentence that embeds the check in the procedure is the biggest lever,” and the TPL has the same structure.

And once a perspective has settled into the code as a test or a comment, its TPL entry has retired as a breakwater. More precisely, what draws the boundary isn’t “is it written somewhere in the code” but “is it written within what the agent reads for this task.” The initialization trap was caught in both conditions because it was written in a hook that anyone designing a restore feature is sure to read. For the next feature of the same shape, where the settled lesson lives (the previous feature’s tests and docs) is far from the design target and less likely to fall within what gets read. So even after a perspective settles into the code, the TPL still helps the design of the next feature of that shape. But the worry that “a thin TPL leaves reviews full of holes” was not supported by the experiment, as long as the reviewer is an agent. That’s no reason to stop accumulating. It’s a reason to aim the accumulation at “being recalled at design time.”

Limitations

As before, this is one repository from a personal project, one model, and 3 to 5 trials per cell. Read it as a check on the consistency of how the conditions split, not as an effect size.

It also assumes the reviewer is a thorough frontier model. With time-boxed CI review, review formats that see only the diff text, or a human reviewer, the nulls of experiments 1 and 2 may not hold as they are. The failure patterns I aimed at are also biased toward structural ones that can be tracked down by reading code. Perspectives you can’t notice without knowing the reproduction steps, like timing or environment differences, weren’t measured. In my repository I also consult the TPL when writing acceptance test scenarios, and raise e2e tests from those. That chain (whether a perspective, by way of a scenario, becomes defensive strength in regression tests) is outside this experiment, and I plan to cover it in a separate piece.

Takeaways

I closed the previous experiment with “a decision record helps only when the agent is made to retrieve it at the moment of conflict.” This experiment adds the following to it.

The value of accumulated perspectives should fade as agents get smarter, starting with the parts that can be re-derived from the code. What remains to the end is its value as a place for lessons not yet written anywhere, and the moments where that pays off are concentrated at the entrance to design.

The karasu repository I used as the test bed is public, so you can read the actual TPL (docs/test-perspectives/). The environment, condition definitions, request prompts, and result tables for every round are collected in the notes.

Every run used headless Claude Code with the model fixed to Claude Opus 4.8. Each trial resets the repository to its initial state, and in the condition without the TPL I also scrubbed its traces from Git history (a lesson from last time, when the agent dug a deletion out of the history). 16 runs for bug detection, 6 for implementation, and 16 for design: 38 in all. ↩
The judge was an LLM with the condition hidden, asked for the 6 criteria and a supporting quote for each. Every discriminating item was mechanically checked against the original text. What was scored was the concatenation of all responses, not just the final one. Scoring only the final response missed the substance in one case during this experiment, and I corrected the misjudgment (details in the notes). ↩
5/5 vs. 0/5 gives p ≈ 0.004 (one-sided) by Fisher’s exact test. The scale is still small, so look at the consistency of the split rather than an effect size. ↩