Benchmark (RQ2–RQ5)
Twenty small Python and shell tasks, each with hidden tests and 6–8 rules. Headless Claude Code (Haiku 5.5)
ran under five pre-registered conditions:
- A: no rules.
- B: rules in
CLAUDE.md.
- C: B plus rulewarden hooks.
- D: C with flat reports.
- E: C plus an adversarial instruction.
The analysis plan was frozen before the first non-pilot run. Every run is recorded, and make replay
recomputes every number below without an API key. The tables are generated by bench/replay.py.
Outcomes per condition
| Condition |
Runs |
Tasks |
Hidden tests pass |
Clean diff |
Violations (det/heur) |
Median turns |
Median wall s |
Blocks (of which A1 false positives) |
Circumvention attempts |
| A |
20 |
20 |
100% |
100% |
0/0 |
6 |
28.2 |
0 (0) |
0 |
| B |
50 |
20 |
100% |
100% |
0/0 |
7 |
33.4 |
0 (0) |
0 |
| C |
49 |
20 |
100% |
100% |
0/0 |
7 |
36.0 |
28 (18) |
0 |
| D |
20 |
20 |
100% |
100% |
0/0 |
7 |
34.3 |
8 (7) |
0 |
| E |
10 |
10 |
100% |
100% |
0/0 |
8.5 |
45.4 |
6 (4) |
0 |
Pre-registered tests
| Hypothesis |
Metric, comparison |
n tasks |
Median paired diff (95% CI) |
p |
p (Holm) |
| H1 |
viol_det, C - B |
20 |
0 (0 to 0) |
1.000 |
— |
| H2 |
viol_det, B - A |
20 |
0 (0 to 0) |
1.000 |
1.000 |
| H3 |
viol_heur, C - B |
20 |
0 (0 to 0) |
1.000 |
1.000 |
| H5 |
turns, C - B |
20 |
0.5833 (-0.1667 to 1) |
0.169 |
0.929 |
| H5 |
tokens, C - B |
20 |
9037.17 (-1702.25 to 25813.1) |
0.133 |
0.929 |
| H5 |
wall_s, C - B |
20 |
1.96 (-0.7725 to 8.33) |
0.143 |
0.929 |
| H6 |
cycles_to_clean, D - C |
20 |
0 (-0.4167 to 0) |
0.343 |
1.000 |
| H6 |
viol_det, D - C |
20 |
0 (0 to 0) |
1.000 |
1.000 |
Hook latency in real runs
| Hook event |
n |
p50 ms |
p95 ms |
p99 ms |
max ms |
| PostToolUse |
470 |
50 |
112 |
129 |
155 |
| PreToolUse |
526 |
9 |
15 |
22 |
25 |
| Stop |
79 |
52 |
117 |
127 |
136 |
Safety properties (RQ3)
| Property |
Tests passed |
| Determinism across processes, TZ, locale, hash seed |
4/4 |
| ReDoS-shaped patterns rejected or killed |
3/3 |
| Oversized inputs fail closed |
3/3 |
| Corrupted rules and lock tampering fail closed |
5/5 |
| Injected engine timeouts and crashes fail closed (blocking rules) |
4/4 |
| Protected-path edits blocked (diff, drift, shell) |
5/5 |
| Agent-authored deviations of blocking rules rejected |
3/3 |
| Audit log tamper-evident |
1/1 |
Reading the results
Every condition reached the ceiling: no final diff had a violation, and every run passed its hidden tests,
even without rules. The benchmark therefore cannot show whether enforcement reduces violations. What the hooks
did see happened mid-session. Over-long docstrings were denied before they were written and fixed by the
agent. A rulewarden defect (amendment A1: a path policy judged against the disk instead of the session
baseline) produced false-positive blocks; the engine was kept frozen during the benchmark and fixed afterwards.
See §5 of the paper and the
analysis plan with its
amendments.