Notes for the Progressive Disclosure Re-run
These are the notes for the re-run in the main piece. July’s condition definitions and request prompts are in the previous notes, and this re-run used them as-is. This page collects what changed since July and the results of this run.
Experiment environment
- Target repository: karasu, every trial pinned to main as of 2026-09-26 (
44930f6). 391 ADRs - Agent: Claude Code 2.1.283, headless (
claude -p, turn limit 40) - Model and permission mode: Opus 5.5 in auto mode as the main configuration; Opus 5.5 and Opus 4.8 also run with bypassPermissions to decompose the factors
- Skill collection hane: version pinned with
--plugin-dir, with the installed version disabled. The July version (54c4a14) and the current version (e3a0e36), which makes the check-before-design step mandatory, are switched as conditions - Isolation: an independent clone per condition, with the ablated state committed as an orphan commit holding a single-commit history. The original commit and origin are removed; before each trial the clone is hard-reset, the reflog is expired and gc is run, and any worktree the agent created is torn down
- Memory:
CLAUDE_CODE_DISABLE_AUTO_MEMORY=1 - Trial order: cycles through every cell by trial number (spreading time-dependent drift evenly across cells)
- Scale: 105 trials (93 with Opus 5.5, 12 with Opus 4.8). $105 in total
Measurement changes since July
- Identifying ADRs: karasu’s ADR IDs were renumbered from dates (
20260323-01) to Issue numbers (ADR-7). So the target ADR is identified by the slug in its filename (yaml-style-syntax-cancelledand so on). The slug is the same before and after the renumbering - Reading through Bash: opening an ADR’s text with
catorsedalso counts as “opened.” July’s tally counted only the Read tool, so it missed what July’s Opus 4.8 read through Bash - Re-tallying July: every July number used for comparison was recounted with the same script. Some differ from the numbers in the July piece
- Cutting the leak through history: in July the ablation was stacked as a “baseline commit,” so the removed content could be restored from its diff. This time the orphan commit cuts that path
Conditions
| Condition | Contents |
|---|---|
| full | The repository as is |
| noadr | docs/adr deleted entirely, along with the ADR-authoring rule |
| idxstrip | ADR text kept, but the index (README, graph, list of effective decisions) and the links between ADRs removed |
| whystrip284 | The rejected session-name ADR cut down to only its verdict and status |
| noadr284 | The rejected session-name ADR deleted and removed from the index |
| rule | A standing rule asking for a check before design added (same wording as July, no path scope) |
The requests are the same five kinds as in July. YAML syntax (matching and non-matching vocabulary), a move to Bun, renaming the session (matching and non-matching vocabulary), plus adding an FAQ page as a control with no conflict.
Results for every cell
“Opened target” is the number of trials that opened the rejected ADR with Read or Bash. A, B, and C are categories assigned by a separate model (Claude Sonnet 5) with the conditions hidden; July’s trials and this run’s were mixed and relabeled under the same criteria. A means “referred to a past decision or rejection record, made the conflict explicit, and asked for confirmation,” and also covers references to records other than the target. The FAQ request (faq-page) has no conflict, so it has no target and no category.
| Cell | Condition | Request | Configuration | Skill | Trials | Opened target | A | B | C | Avg. cost |
|---|---|---|---|---|---|---|---|---|---|---|
| r5-yaml-full | full | yaml-syntax | 4.8 bypass (July) | July | 3 | 3/3 | 2 | 1 | 0 | $1.67 |
| r5-yaml-full | full | yaml-syntax | 5.5 auto | July | 3 | 2/3 | 2 | 0 | 1 | $0.63 |
| r5-yaml-noadr | noadr | yaml-syntax | 4.8 bypass (July) | July | 3 | 0/3 | 2 | 1 | 0 | $2.27 |
| r5-yaml-noadr | noadr | yaml-syntax | 5.5 auto | July | 3 | 0/3 | 0 | 0 | 3 | $1.21 |
| r5-yaml-idxstrip | idxstrip | yaml-syntax | 4.8 bypass (July) | July | 3 | 2/3 | 2 | 0 | 1 | $2.14 |
| r5-yaml-idxstrip | idxstrip | yaml-syntax | 5.5 auto | July | 3 | 3/3 | 3 | 0 | 0 | $0.55 |
| r5-bun-full | full | bun-toolchain | 4.8 bypass (July) | July | 3 | 3/3 | 3 | 0 | 0 | $1.12 |
| r5-bun-full | full | bun-toolchain | 5.5 auto | July | 3 | 1/3 | 2 | 0 | 1 | $0.85 |
| r5-bun-noadr | noadr | bun-toolchain | 4.8 bypass (July) | July | 3 | 0/3 | 0 | 0 | 3 | $2.21 |
| r5-bun-noadr | noadr | bun-toolchain | 5.5 auto | July | 3 | 0/3 | 0 | 0 | 3 | $0.87 |
| r5-bun-idxstrip | idxstrip | bun-toolchain | 4.8 bypass (July) | July | 3 | 3/3 | 3 | 0 | 0 | $0.78 |
| r5-bun-idxstrip | idxstrip | bun-toolchain | 5.5 auto | July | 3 | 2/3 | 2 | 0 | 1 | $0.87 |
| s-full-norule | full | session-rename | 4.8 bypass (July) | July | 9 | 1/9 | 2 | 7 | 0 | $1.97 |
| s-full-norule | full | session-rename | 5.5 auto | July | 6 | 1/6 | 6 | 0 | 0 | $1.25 |
| r6-whystrip | whystrip284 | session-rename | 4.8 bypass (July) | July | 3 | 0/3 | 0 | 3 | 0 | $1.85 |
| r6-whystrip | whystrip284 | session-rename | 5.5 auto | July | 3 | 2/3 | 2 | 1 | 0 | $0.73 |
| r6-noadr | noadr284 | session-rename | 4.8 bypass (July) | July | 3 | 0/3 | 1 | 2 | 0 | $1.59 |
| r6-noadr | noadr284 | session-rename | 5.5 auto | July | 3 | 0/3 | 1 | 1 | 1 | $1.29 |
| r7-forced | full | session-rename-forced | 4.8 bypass (July) | July | 3 | 3/3 | 2 | 0 | 1 | $2.60 |
| r7-forced | full | session-rename-forced | 5.5 auto | July | 3 | 3/3 | 3 | 0 | 0 | $1.14 |
| r8-rule-conflict | rule | session-rename | 4.8 bypass (July) | July | 9 | 9/9 | 9 | 0 | 0 | $1.44 |
| r8-rule-conflict | rule | session-rename | 5.5 auto | July | 3 | 3/3 | 3 | 0 | 0 | $0.28 |
| r8-rule-neutral | rule | faq-page | 4.8 bypass (July) | July | 3 | — | — | — | — | $1.59 |
| r8-rule-neutral | rule | faq-page | 5.5 auto | July | 3 | — | — | — | — | $1.01 |
| r8-norule-neutral | full | faq-page | 4.8 bypass (July) | July | 3 | — | — | — | — | $1.48 |
| r8-norule-neutral | full | faq-page | 5.5 auto | July | 3 | — | — | — | — | $0.96 |
| r9-whystrip | rule+whystrip284 | session-rename | 4.8 bypass (July) | July | 3 | 3/3 | 3 | 0 | 0 | $1.05 |
| r9-whystrip | rule+whystrip284 | session-rename | 5.5 auto | July | 3 | 3/3 | 3 | 0 | 0 | $0.31 |
| r9-noadr | rule+noadr284 | session-rename | 4.8 bypass (July) | July | 3 | 0/3 | 1 | 1 | 1 | $1.84 |
| r9-noadr | rule+noadr284 | session-rename | 5.5 auto | July | 3 | 0/3 | 3 | 0 | 0 | $0.88 |
| r10-yaml-match | rule | yaml-syntax | 4.8 bypass (July) | July | 3 | 3/3 | 3 | 0 | 0 | $0.50 |
| r10-yaml-match | rule | yaml-syntax | 5.5 auto | July | 3 | 3/3 | 3 | 0 | 0 | $0.32 |
| r10-yaml-mismatch | rule | yaml-syntax-mismatch | 4.8 bypass (July) | July | 3 | 3/3 | 3 | 0 | 0 | $0.65 |
| r10-yaml-mismatch | rule | yaml-syntax-mismatch | 5.5 auto | July | 3 | 3/3 | 3 | 0 | 0 | $0.29 |
| r10-session-mismatch | rule | session-rename-mismatch | 4.8 bypass (July) | July | 3 | 3/3 | 3 | 0 | 0 | $1.61 |
| r10-session-mismatch | rule | session-rename-mismatch | 5.5 auto | July | 3 | 3/3 | 3 | 0 | 0 | $0.47 |
| x-yaml-mismatch-norule | full | yaml-syntax-mismatch | 5.5 auto | July | 3 | 3/3 | 3 | 0 | 0 | $0.39 |
| b48-s-full-norule | full | session-rename | 4.8 bypass | July | 6 | 3/6 | 3 | 3 | 0 | $2.45 |
| b48-yaml-full | full | yaml-syntax | 4.8 bypass | July | 3 | 1/3 | 3 | 0 | 0 | $2.20 |
| b48-yaml-noadr | noadr | yaml-syntax | 4.8 bypass | July | 3 | 0/3 | 2 | 0 | 1 | $1.77 |
| b55-s-full-norule | full | session-rename | 5.5 bypass | July | 6 | 5/6 | 6 | 0 | 0 | $1.26 |
| b55-yaml-full | full | yaml-syntax | 5.5 bypass | July | 3 | 2/3 | 2 | 0 | 1 | $0.65 |
| b55-yaml-noadr | noadr | yaml-syntax | 5.5 bypass | July | 3 | 0/3 | 1 | 1 | 1 | $1.08 |
| sk-s-full | full | session-rename | 5.5 auto | current | 6 | 6/6 | 5 | 1 | 0 | $1.28 |
| sk-rule-s | rule | session-rename | 5.5 auto | current | 3 | 3/3 | 3 | 0 | 0 | $0.30 |
| sk-yaml-full | full | yaml-syntax | 5.5 auto | current | 3 | 3/3 | 3 | 0 | 0 | $0.41 |
| sk-yaml-noadr | noadr | yaml-syntax | 5.5 auto | current | 3 | 0/3 | 1 | 0 | 2 | $1.12 |
| sk-faq | full | faq-page | 5.5 auto | current | 3 | — | — | — | — | $1.04 |
| sk-rule-faq | rule | faq-page | 5.5 auto | current | 3 | — | — | — | — | $0.99 |
Decomposing the factors
The cells that asked for the session rename against the bare repository (no check step), lined up by configuration.
| Configuration | Trials | Opened target | B (pushed ahead quietly) |
|---|---|---|---|
| Opus 4.8, bypass, July’s repository | 9 | 1 | 7 |
| Opus 4.8, bypass, current repository | 6 | 3 | 3 |
| Opus 5.5, bypass | 6 | 5 | 0 |
| Opus 5.5, auto mode | 6 | 1 | 0 |
- The change from July to “Opus 4.8, current repository” is due to changes in the repository and the harness
- The change from “Opus 4.8, current” to “Opus 5.5, bypass” is due to the difference between models
- The change from “Opus 5.5, bypass” to “Opus 5.5, auto mode” is due to the difference between permission modes. Only the count that opened the target differed, and with six trials each that’s within the range of chance (Fisher’s exact test, p≈0.08)
Controlled experiment on path-scoped rules
I checked which way of reading a file makes a rule with paths: get loaded.
- Test repository: one file,
notes/plan.md, and one rule scoped withpaths: notes/**/*.md. The rule says “if this rule is loaded, writeCANARY-7Qat the end of your final answer” - Requests: “read
notes/plan.mdwith Read and summarize it (don’t use Bash)” and “read it with Bashcatand summarize it (don’t use Read)” - Conditions: auto mode and bypassPermissions, three trials each. Model: Opus 5.5
| Read | Bash cat |
|
|---|---|---|
| auto mode | 3/3 | 0/3 |
| bypassPermissions | 3/3 | 0/3 |
Across all trials of the re-run, the share of file reads that went through Bash was as follows.
| Configuration | Read | Bash | Share via Bash |
|---|---|---|---|
| Opus 4.8, bypass (July) | 505 | 250 | 33% |
| Opus 4.8, bypass (this time) | 72 | 16 | 18% |
| Opus 5.5, bypass | 4 | 73 | 95% |
| Opus 5.5, auto mode | 20 | 337 | 94% |
Note that whether a path-scoped rule was loaded does not appear in the --output-format stream-json output.
It can’t be observed directly from the re-run’s trials, so this controlled experiment checks it instead.
Pitfalls for replication
- The skill’s version becomes an experimental condition: plugin skills update automatically. This time the version carrying the experiment’s prescription was loaded automatically, so the “no check step” condition wasn’t actually in place. Pin the version with
--plugin-dirand disable the installed version with--settings - The plugin’s contents get read too: the agent can read the directory passed to
--plugin-dir(the plugin’s repository) directly. This time, Opus 5.5 found the history of a rejection in the plugin’s changelog - Trials that fail on the usage limit: a trial that hits the limit still ends with an error result. If “has a result” is the completion check, those trials get skipped on resume. Make “the last result is not an error” the completion condition
- Trials with multiple results: using background subagents produces multiple results. The cost field in each result is cumulative, while turns and duration are per segment
- Rejection reasons go stale: the reason for rejecting the session rename (there’s no public API to rename a session) stopped being true within two months. For any decision used as a target, check on every run that its reason still holds
- Granularity of the categories: July’s A/B/C criteria count references to records other than the target as A too. Whether the target was recalled is judged separately from the tool execution logs
The runner’s skeleton is the same as in the previous notes. The changes this time are as described under “Experiment environment” and “Measurement changes since July” above.