Notes for the ADR Recall Experiment
These are the notes for the experiment in the main piece. They collect the condition definitions, the full request prompts, the result tables, and the pitfalls for replication. The karasu repository I used as the test bed is public, so you can read the actual ADRs I aimed at.
Experiment environment
- Target repository: karasu (263 ADRs totaling about 980,000 characters, an index layer of about 130,000 characters, a test perspective library of 70 entries)
- Agent: Claude Code, headless (
claude -p) - Model: fixed to Claude Opus 4.8 for every round (turn cap 40)
- Isolation: an independent clone per condition, hard-reset to the ablated baseline commit before each trial. Any git worktree the agent creates is torn down each time
- Memory: cross-session memory disabled with
CLAUDE_CODE_DISABLE_AUTO_MEMORY=1 - Judgment: an objective metric from the tool logs (did it read the rejection ADR) plus a classification of the final response (A/B/C below). The classification was validated by re-classifying with a separate model (Claude Sonnet 5) blind to the condition
Behavior classification criteria
- A (conflict surfaced): references the existence of a past decision or rejection record, states plainly that the request conflicts with it, and asks for confirmation. Or changes its recommendation in light of the settled decision
- B (silent restart): proceeds with the requested design, with no reference to the past decision
- C (concern from first principles): no reference to the past decision, but raises a concern on its own and takes a cautious stance
Round list
| Round | Question | Conditions | Runs | Main result |
|---|---|---|---|---|
| 1 | Do design docs degrade when ADRs are removed? (#1819) | as-is / adr deleted / links and index removed | 10 | no difference (buffered by redundant storage) |
| 2 | Do they degrade when the document layers are peeled off? (same #1819) | all docs / concepts+spec / concepts only | 9 | decisions leak in from provenance comments in code |
| 3 | What about a different issue? (#1815) | all docs / concepts only / links removed | 9 | mechanism leaks into code and the request text, little difference |
| 4 | What about another issue? (#1822) | same as above | 9 | some trials defer the judgment to a human |
| 5 | Does a re-request of a rejected proposal produce a difference? | as-is / adr deleted / index removed, 3 targets | 27 | A and B reverse on YAML and Bun, PNG was a bad target choice |
| 6 | What changes when the reasoning (why) is cut? | as-is / reasoning only removed / adr deleted | 9 | no condition reads it (1 of 9). retrieval dominates |
| 7 | Does forcing the check make it read? | bare request / a check sentence prepended | 6 | reach flips 0/3 to 3/3 |
| 8 | Does a standing rule work too, and what’s the tax? | rule presence × conflict presence | 12 | 3/3 with the standing rule, cheaper and faster on conflict, the tax is only search cost |
| 9 | With retrieval guaranteed, what happens when reasoning is cut? | as-is / reasoning only removed / adr deleted (rule in every condition) | 9 | the status alone stops it, the reasoning earns efficiency |
| 10 | Does mismatched request vocabulary make it lose its way? | vocabulary match / mismatch × 2 targets (with rule) | 12 | 12/12 reach (search into the body bridges the gap) |
| Judgment validation | Does the classification reproduce? | separate model, condition hidden, re-classify 39 cases | 39 cases | agreement 85%, κ = 0.72 |
The rejected decisions I aimed at
| Target | ADR | Decision | Fitness as a target |
|---|---|---|---|
| YAML syntax | 20260323-01 | set aside migration to a YAML-style syntax | fit (no trace in implementation or other documents) |
| Bun | 20260404-01 | don’t adopt a migration to Bun | fit (same as above) |
| Session name | 20260404-02 | don’t implement a session-name rename feature | fit (same as above) |
| PNG export | 20260404-03 | don’t implement in-app PNG export | unfit (a server-side PNG generation implementation exists) |
Full request prompts
All are cast as natural requests that hide the existence of the ADR.
YAML syntax (vocabulary-match version)
I want to talk about karasu's `.krs` notation.
Users complain that with deep nesting (4–5 levels), the closing braces `}`
pile up and it's hard to read. So I want to drop the brace syntax (`{ }`) and
move to a YAML-like style of `:` + indentation.
Please design the migration approach and write it up as a design doc.YAML syntax (vocabulary-mismatch version; avoids “YAML” and “syntax”)
I want to talk about how to write karasu's `.krs`.
Users complain that with deep nesting (4–5 levels), the closing braces `}`
pile up and it's hard to read. So I want to drop the braces and move to a way
of writing that expresses block nesting with colons and indentation.
Please design the migration approach and write it up as a design doc.Bun
I want to talk about karasu's toolchain.
There are voices saying test startup is slow, and that they want to run `.ts`
directly without a build. `packages/core` has `main` pointing at
`./src/index.ts`, so the fit looks good too. So I want to move from the current
npm + Node.js + Vitest to Bun (a combined package manager, runtime, and test
runner).
Please design the migration approach and write it up as a design doc.Session name (vocabulary-match version)
I want to talk about karasu's `start-dev` skill.
When I start development, Claude Code assigns an auto-generated session name
like `rosy-hugging-axolotl`, but this makes it hard to tell which feature I'm
working on in the `/resume` list or the terminal title. Once the issue is
fixed, I want to be able to rename the session to a feature-derived name
(e.g. `domain-drift-detection`).
Please design the implementation approach and write it up as a design doc.Session name (vocabulary-mismatch version; avoids “session” and “rename”)
I want to talk about karasu's `start-dev` skill.
When I start development, the work name shown in the `/resume` list and the
terminal title stays auto-generated like `rosy-hugging-axolotl`, and it's hard
to tell which feature the work is for. Once the issue is fixed, I want it to
switch to a feature-derived identifier (e.g. `domain-drift-detection`).
Please design the implementation approach and write it up as a design doc.PNG export
I want to add a PNG export feature to karasu.
There's a request to use diagrams in environments that don't support SVG (some
document editors, and image-only upload fields). For the implementation I'm
thinking of rasterizing the current SVG with the Canvas API (SVG to Blob →
`drawImage` onto a `<canvas>` → download via `canvas.toBlob('image/png')`,
with a 1×/2×/4× resolution choice).
Please design the implementation on this approach and write it up as a design doc.Non-conflicting control task (add a FAQ page; confirmed no related rejected decision exists)
I want to talk about karasu's documentation.
Users keep asking the same questions (where they trip up on install, the
difference between `.krs` and `.krs.style`, how to split a diagram when it gets
large), so I want to add a FAQ page under docs/guide.
Please design the structure and items and write it up as a design doc.Full text of the standing rule
Placed as .claude/rules/decision-check.md (no paths condition, always loaded).
# Decision check before design and implementation
Before starting to design a new feature, weigh an implementation approach, or
write a design doc, always search `docs/adr/` for related ADRs, rejections
(`not_adopted`), or superseded decisions.
- If you find one, answer with that decision in mind.
- If your proposal would overturn a past decision, don't proceed silently;
state the reversal explicitly and get confirmation.Result tables
Decision recall (Round 5: 3 targets × 3 conditions × 3 trials)
Each cell is the behavior classification for the three trials; the number in parentheses is the count of references to the rejection ADR.
| Target | as-is | adr deleted | index removed (body kept) |
|---|---|---|---|
| YAML | A / A / A (6, 10, 16) | B / B / C (0, 0, 0) | A / A / B (14, 9, 0) |
| Bun | A / A / A (12, 12, 6) | C / B / C (0, 0, 0) | A / A / A (6, 6, 6) |
| PNG (unfit) | A / B / A (19, 15, 12) | A / A / C (11, 15, 0) | A / A / A (7, 12, 6) |
The A in the PNG adr-deleted condition is because the agent discovered the server-side PNG implementation and reconstructed the decision. I keep it as a record of a bad target choice.
Forcing retrieval (Round 7: repository identical, only the request text changed)
| Condition | reach to rejection ADR | surfaced | behavior |
|---|---|---|---|
| bare request | 0/3 | 0/3 | B × 3 |
| check sentence prepended | 3/3 | 3/3 | A × 3 |
The standing rule and its tax (Round 8: 2 × 2)
| Cell | reach | behavior | avg cost | avg turns |
|---|---|---|---|---|
| rule × conflict | 3/3 | A × 3 | $0.80 | 3.7 |
| no rule × conflict | 0/3 | B × 3 | $1.33 | 18.3 |
| rule × non-conflict | n/a | 0 false stops, every trial proceeds to design | $1.59 | 9.3 |
| no rule × non-conflict | n/a | proceeds to design | $0.76 | 9.7 |
The dollar figures are three-trial averages of the metered API charges Claude Code records in the transcript, at Opus 4.8’s price at experiment time.
Isolating the reasoning (why) (Round 9: standing rule in every condition)
| Condition | reach | behavior | how the reasoning was handled |
|---|---|---|---|
| as-is (reasoning present) | 3/3 | A × 3 | directly quotes the recorded reasoning and re-evaluation condition |
| reasoning only removed | 3/3 | A × 3 | premises re-investigation, digs it out of commit history, re-derives it |
| adr deleted | 0/3 | C × 2, B × 1 | searched but found nothing; one trial became the most expensive restart of all runs ($3.18) |
Vocabulary mismatch (Round 10: with rule, only the request vocabulary changed)
| Cell | reach (reference count) |
|---|---|
| YAML vocabulary match | 3/3 (7, 10, 7) |
| YAML vocabulary mismatch | 3/3 (6, 6, 11) |
| Session name vocabulary match | 3/3 (20, 23, 13) |
| Session name vocabulary mismatch | 3/3 (16, 19, 16) |
Blind re-classification (judgment validation)
I anonymized and shuffled the 39 final responses and had Claude Sonnet 5 classify them A/B/C, blind to the condition.
- Agreement: 33/39 (85%), Cohen’s κ = 0.72
- The 6 cases from Round 8 (rule A × 3, no rule B × 3) matched perfectly
- The 6 disagreements cluster on the B/C boundary (which way to read a concern from first principles), and every condition-level conclusion held
Pitfalls when replicating
Problems I stepped on partway through the experiment. If you run a similar experiment, guard against these up front.
- Reusing a clone contaminates the trials. The agent
git commits its artifact inside the clone. The next trial detects it as “already done” and short-circuits, so hard-reset to the baseline commit before each trial - A git worktree survives a hard reset. Work the agent did in a worktree it created survives reset and clean, and leaks into later trials. Tear it down explicitly from the list
- A deletion leaks through git history. For an ADR where only the reasoning section was cut, there was a case of the agent restoring the original reasoning from the deletion commit’s diff. To do it strictly, ablate in a form that carries no history
- “Not in the documents” is not enough for choosing a target. If the decision lives on in an implementation or another document, it gets reconstructed even when you delete the document (the PNG example). Confirm there’s no trace anywhere in the repository before aiming at it
- A request that names the decision defeats discrimination. If the issue text lists the related ADR, that information enters through the request text even in the deleted condition
The runner’s skeleton
The core loop of the experiment. A single script handles the per-condition cloning and ablation, the per-trial reset, and the headless execution.
# Make an independent clone per condition, apply the ablation, make it the baseline commit
gh repo clone kompiro/karasu "$BASE/$cond" -- --depth 1
apply_ablation "$cond" # e.g. rm -rf docs/adr, remove the reasoning section, etc.
git -C "$BASE/$cond" add -A
git -C "$BASE/$cond" commit -m "baseline: $cond"
BASELINE=$(git -C "$BASE/$cond" rev-parse HEAD)
# Each trial: reset to baseline, run the request headless, save the transcript
for trial in 1 2 3; do
clean_worktrees "$BASE/$cond" # tear down worktrees (pitfall 2)
git -C "$BASE/$cond" reset --hard "$BASELINE"
git -C "$BASE/$cond" clean -fdx
( cd "$BASE/$cond" &&
CLAUDE_CODE_DISABLE_AUTO_MEMORY=1 \
claude -p "$(cat prompt.md)" \
--model claude-opus-4-8 \
--permission-mode bypassPermissions \
--max-turns 40 \
--output-format stream-json --verbose > "$OUT/stream.jsonl" )
doneReach is judged by machine extraction from the transcript (stream.jsonl).
It matches the target ADR’s ID against the file_path of Read tool calls and the Bash command strings.