Notes for the TPL Experiment

These are the notes for the experiment in the main piece. They collect the condition definitions, the full request prompts, the scoring criteria, the result tables, and the pitfalls for replication. The karasu repository I used as the test bed is public, so you can read the actual TPL (docs/test-perspectives/).

Experiment environment

Target repository: karasu (70 TPL entries, plus about 250 ADRs, specs, and a set of standing rules for the agent)
Agent: Claude Code 2.1.205 / 2.1.206, headless (claude -p, bypassPermissions)
Model: fixed to Claude Opus 4.8 for every round
Isolation: an independent clone per condition, hard-reset to the baseline commit before each trial. Any Git worktree the agent creates is torn down each time
Memory: cross-session memory disabled with CLAUDE_CODE_DISABLE_AUTO_MEMORY=1
History: the condition without the TPL was rebuilt on an orphan history, down to expiring the reflog and running gc (a countermeasure for the previous experiment, where a deletion was dug out of the history). In the design-task rounds, both conditions were made symmetric with a single-commit history

Condition definitions

Every round uses the same 2 conditions.

cond1 (with TPL): as-is
cond2 (without TPL): the 70 entries in docs/test-perspectives/ deleted, plus the paths leading to them neutralized layer by layer

Neutralizing cond2 touched 213 files. They break down into the following 4 layers.

Layer	Target	Treatment
Instruction	the TPL section of CLAUDE.md, the QA agent definition, mentions in the standing rules, the “TPL impact” section of the PR / Issue templates	delete the section or neutralize the mention
Mechanism	the ADR recording the adoption of the TPL, the CI workflow and Git hook that validate the TPL	delete
Content	“Related TPLs” cross-links in the specs, TPL-ID mentions in the acceptance test ledger and ADR bodies	removed mechanically by script
Code	TPL-ID comments in source and tests (about 60 files)	deliberately left in place (a known leak; see the notes under the result tables)

After neutralizing, I verified with grep that “TPL” and “test-perspectives” had 0 remaining hits in the 3 layers other than code, and confirmed that running it twice gives the same result (idempotence).

Round list

Round	Type	Question	Runs	Main result
1	review	Is a planted bug of the missing-entry (omission) type detected?	6	3/3 detected in both conditions (ceiling)
2	review	Does a value-mismatch bug make a difference?	10	5/5 detected in both conditions (ceiling). Only turn count shows a trend
3	implementation	When adding a new kind, does the agent wire it into distant representations on its own?	6	every run wired it (ceiling). Turn cap miscalibrated
4	design	Does the design doc and acceptance criteria miss perspectives differently?	10	average 5.2 vs. 4.2. The discriminating item is 5/5 vs. 0/5
5	design	Does it reproduce on a different subject (session restore)?	6	reproduces at 5.7 vs. 4.0. Direct observation of the boundary

How the planted bugs were made (rounds 1 and 2)

The perspective I aimed at was TPL-20260519-02 (drift between two representations of the same vocabulary).

I built a change that adds a new deploy kind vm (19 files, about 85 lines) and dressed it up as a feature PR. It is wired through the AST, the parser, the style definitions (both dark and light themes), icons, the LSP, the generated tables in the spec, and a minimal set of tests.

Round 1 (omission): vm is not added to the draw.io exporter’s per-kind style table, and only there
Round 2 (value mismatch): the entry is added, but its color is copied over from the neighboring kind (lambda). vm is purple on the SVG side and yellow only on the draw.io side, so to a search it looks “wired up”

Both were used only after confirming that type checking, lint, the roughly 3,700 unit tests, and the spec table generation check all pass.

Full request prompts

All of them are phrased with no hint toward the TPL, the perspective, or where the change ripples. The originals are in Japanese; these are translations.

Review (shared by rounds 1 and 2)

The latest commit (HEAD) is a change that adds a new deploy kind `vm`.
Please review whether this change can be merged as a PR.
If there are problems, point out specifically what is wrong in which file.
Don't modify the code; just report the review results.

Implementation (round 3)

Please add a new deploy kind `vm` to karasu.

It's a node type representing a virtual machine image (an EC2 AMI, a GCE image,
and so on), and it should be treated as a first-class kind on par with the
existing deploy kinds (oci, lambda, etc.). Finish it, including implementation,
tests, and any needed documentation updates, to a standard where this change
alone can be merged as a PR. Follow the repository's existing conventions and
practices.

Design 1: in-place container expansion (round 4)

I want to add an "in-place container expansion" feature to karasu.

In the system view, expand just one container (system / domain) inline, and show
its internal elements in the same diagram as the sibling containers that stay
collapsed. I also want edges that cross the boundary to be visible.

Write the design doc and acceptance criteria for this feature in
`docs/design/in-place-expand.md`. Don't implement it.
Follow the conventions of the repository's existing design docs.

Design 2: session restore (round 5)

I want to add a "per-project session restore" feature to karasu.

When the app is reopened, I want it to restore, per project, the previous
state (the open file, the active view, the drill target, the highlighted
node, and whether panels are open) so work can resume where it left off.

Write the design doc and acceptance criteria for this feature in
`docs/design/session-restore.md`. Don't implement it.
Follow the conventions of the repository's existing design docs.

Scoring method

Review (rounds 1 and 2)

An LLM with the condition hidden (Claude Opus 4.8, no tools) was given the planted bug and the criteria and asked for a binary judgment. The criterion was “does it point specifically at the relevant spot in the relevant file”; merely mentioning drawio doesn’t count as detection. For every judgment, the supporting quote was mechanically checked against the original text.

What was scored was the concatenation of all of the agent’s response text. Passing only the final response caused a misjudgment: in a run whose final message was just a closing line like “the findings in my previous message stand,” the substance was missed (this actually happened once and was corrected; see the pitfalls section).

Implementation (round 3)

The resulting diff was checked mechanically against a checklist of the 15 correct places to wire (from the AST through the exporters and the Visual Studio Code extension). No LLM judgment was used.

Design (rounds 4 and 5)

For 6 items drawn from the relevant TPL checklists, an LLM with the condition hidden judged whether each was “explicitly specified as a spec or acceptance criterion.” “Touches on it” isn’t enough; the bar for yes is a rule the implementer won’t have to guess at. Every discriminating item (one where the conditions differed) was verified by checking its supporting quote against the original text.

The 6 items for round 4:

Explicit upper limits on simultaneous expansions and displayed elements
The initial state being collapsed, and the operation to collapse again (the way back)
An element being expanded is neither drawn twice nor dropped (exactly-once)
How edges whose endpoints are inside collapsed siblings are handled (aggregation rules)
Behavior in degenerate cases (empty container, everything expanded, etc.)
Dealing with diagram bloat and visual density caused by expansion

The 6 items for round 5:

Handling app initialization rolling back the restore (application order)
Enumerating the fields to restore, and restoring all fields symmetrically
Re-applying the restore happens only once
Distinguishing corrupted saved data from “nothing saved”
Behavior when restore fails (fail-closed)
Precedence against other state sources such as the URL hash

Result tables

Review detection (rounds 1 and 2)

Round	Bug type	cond1 (with TPL)	cond2 (without TPL)
1	missing entry	3/3 detected	3/3 detected
2	value mismatch	5/5 detected	5/5 detected

In round 2, the average turn count was 23.6 for cond1 and 31.2 for cond2 (average cost $1.26 and $1.43). There’s no difference in detection rate; only the length of exploration shows a trend. The variance is large, though, and at n = 5 it’s only a trend.

Two notes. cond1 went to read the TPL in every trial, so the paths leading to it worked. And in cond2, in 2 of 5 trials the agent followed a TPL-ID comment left in the code layer, went looking for docs/test-perspectives/, and came up empty. The comments left in place give no content, but they do trigger exploration.

Implementation coverage (round 3)

All 6 runs (both conditions) wired it through to the draw.io exporter on their own. Nearly every run wired the Visual Studio Code extension side as well. 0 of 6 runs wrote a parity test that pins the representations to each other.

However, in this round a misconfigured turn cap of 80 cut off 4 of the 6 runs, so I can’t tell whether they “didn’t write the test” or “were cut off before writing it.” I only adopt the wiring coverage (which was complete before the cutoff) as a result.

Design 1: in-place expansion (round 4, 5 trials per condition)

Item	cond1	cond2
limit on simultaneous expansion	5/5	5/5
collapsed initial state and way back	5/5	5/5
exactly-once (no double drawing, no dropping)	5/5	0/5
edge aggregation	5/5	5/5
degenerate cases	5/5	5/5
visual density	1/5	1/5
average score	5.2/6	4.2/6

exactly-once at 5/5 vs. 0/5 gives p ≈ 0.004 (one-sided) by Fisher’s exact test. Checking against the original text, each of cond1’s 5 documents states the rule 2 to 5 times, and cond2’s 5 documents mention it 0 times. Document length was comparable across conditions (165 to 307 lines). The visual density item was a borderline case whose judgment wavered when the judge was rerun, so I left it out of the main piece’s claims.

All 5 cond1 trials read the relevant TPL (TPL-20260624-02). The same holds for round 5: all 3 cond1 trials read the 2 relevant TPL entries (persistent store and restore state) before starting the design.

Design 2: session restore (round 5, 3 trials per condition)

Item	cond1	cond2
restore application order (handling the initialization rollback)	3/3	3/3
field symmetry	3/3	2/3
re-application happens only once	3/3	1/3
corrupted data vs. absence	2/3	0/3
fail-closed	3/3	3/3
precedence against other state sources	3/3	3/3
average score	5.7/6	4.0/6

It’s no accident that “restore application order” is 3/3 in both conditions. This perspective had been written up in detail as a code comment on the relevant hook when a past bug was fixed, and all 6 runs (cond2 included) reached that comment. The perspective written in the code was picked up without the TPL, and only the perspectives not written in the code (the only-once rule, distinguishing corrupted data) split on whether the TPL was present. This contrast is the most direct evidence for the boundary claimed in the main piece.

Pitfalls when replicating

On top of the 5 in the previous notes, these are the problems I newly stepped on this time.

Scoring only the final response misses the substance. In one run the agent’s final message was just a closing line, “the findings in my previous message stand,” and it was misjudged as a miss. Score the concatenation of all response text
A turn cap distorts the measurement even when meant as a runaway guard. A cap of 80 wasn’t enough for an implementation-sized task, and 4 of 6 runs were cut off. Tests get written near the end of the work, so the later behavior can’t be compared. For the design tasks I raised the cap to 150, and every run finished naturally
Git leaks through four routes: history, reflog, tags, and branches. Orphaning plus expiring the reflog wasn’t enough. Tags that came along with a deep fetch were protecting the old history (with the TPL in it) from gc. Inventory every ref and remove it
Match the depth of Git history across conditions. If only one side has real history, the agent can use past similar commits as a blueprint (this actually happened in the implementation round). In the design rounds both conditions were made symmetric with a single-commit history
ID comments in the code layer trigger exploration even if left in place. Leaving them doesn’t leak content, but it does affect behavior. Make sure the reach logs let you classify the route
If you use a real issue as the subject, check whether its text names the perspective. Some candidate issues named the TPL or the severity guidelines in their text, which would erase the difference between conditions as-is. I synthesized the tasks again in a form that names nothing

The runner’s skeleton

The structure is the same as last time (a clone per condition, a reset per trial, a headless run). What’s specific to this experiment is building cond2: after applying the neutralizing script, it’s rebuilt on an orphan history.

# cond2: neutralize + rebuild history (leave no trace)
git clone https://github.com/kompiro/karasu "$BASE/cond2" --depth 1
bash neutralize-tpl.sh "$BASE/cond2"        # layered neutralization of 213 files + grep check for leftovers
cd "$BASE/cond2"
ROOTMSG=$(git log -1 --format=%s)
git checkout --orphan clean-main
git add -A && git commit -m "$ROOTMSG"
# Pitfall 3: close all four routes: history, reflog, tags, and branches
git branch -D main && git branch -m main    # the old main protects the old history
git remote remove origin                    # so do remote-tracking refs
git tag -l | xargs -r git tag -d            # so do tags from the deep fetch
git reflog expire --expire=now --all && git gc --prune=now
# Pitfall 4: in the design rounds, orphan cond1 the same way so history depth is symmetric

# Each trial: reset to the baseline and run headless (same as last time)
clean_worktrees
git reset --hard "$ROOT" && git clean -fdx -e node_modules
CLAUDE_CODE_DISABLE_AUTO_MEMORY=1 \
claude -p "$(cat prompt.md)" \
  --model claude-opus-4-8 \
  --permission-mode bypassPermissions \
  --max-turns 150 \        # value for the design rounds. 40 for review, 80 for implementation (the value that hit pitfall 2)
  --output-format stream-json --verbose > "$OUT/stream.jsonl"

Whether the agent reached the TPL was extracted mechanically from the transcripts. I matched not only the paths of the Read tool but also Bash command strings (viewing via ls or cat). In the review runs nearly all TPL viewing went through Bash, so looking only at the Read tool makes it seem the TPL was reached 0 times.