Skip to content

Corpus study (RQ1)

How much of what developers write into CLAUDE.md, AGENTS.md, .cursorrules, GEMINI.md and .windsurfrules can a deterministic tool enforce?

Every line was labelled twice: by a deterministic regex rubric (corpus/label.py) and by a model, in two independent passes with a fixed prompt. No human labelled the corpus. Both labellings are reported, because they disagree, and the disagreement is itself a finding. The tables below are generated by corpus/analyse.py.

Decidability

Tier Rubric share (95% CI) n Model pass 1 share (95% CI) n
deterministic 34.1% (31.3%–37.2%) 1312 19.3% (17.2%–21.7%) 742
heuristic 5.5% (4.5%–6.5%) 211 41.1% (37.8%–44.4%) 1581
undecidable 60.4% (57.3%–63.3%) 2321 39.6% (35.4%–43.6%) 1521

Where the rubric and the model disagree

Rows are rubric labels, columns are model pass 1.

Rubric \ Model pass 1 deterministic heuristic undecidable
deterministic 351 658 303
heuristic 5 68 138
undecidable 386 855 1080

Categories

Category n share deterministic heuristic undecidable
style 700 18.2% 35.1% 10.3% 54.6%
tooling 421 11.0% 43.5% 1.2% 55.3%
scope 40 1.0% 52.5% 10.0% 37.5%
process 469 12.2% 32.6% 4.7% 62.7%
testing 505 13.1% 30.7% 3.0% 66.3%
security 130 3.4% 57.7% 3.1% 39.2%
voice 123 3.2% 24.4% 29.3% 46.3%
other 1456 37.9% 30.8% 3.6% 65.5%

Method

(fixed before collection)

  • Source: GitHub code search REST API (gh api search/code), authenticated, 10 requests/min limit honoured.
  • File types (strata) and queries, one stratum each, 60 files per stratum, total cap CORPUS_MAX_FILES=300:
Stratum Query
CLAUDE.md filename:CLAUDE.md
AGENTS.md filename:AGENTS.md
.cursorrules filename:.cursorrules
GEMINI.md filename:GEMINI.md
.windsurfrules filename:.windsurfrules
  • Order and selection: results are taken in API order, page by page (100 per page), skipping (a) forks, (b) a second file from a repo already sampled in any stratum, (c) files > 200 KB, and (d) repos whose licence is not OSI-approved (see licensing below), until the stratum reaches 60 or results run out. API ordering is not random. This is a convenience sample and the paper says so.
  • Recorded per file: stratum, repo full name, path, file blob SHA, repo default-branch commit SHA at collection time, SPDX licence id, stars, collection timestamp. → corpus/data/files.csv.

Licensing (fail-closed)

  • Verbatim rule lines are stored only for repos whose SPDX licence is on the OSI-approved allowlist in corpus/collect.py. Other repos are skipped at collection time, so their text is never stored.
  • Every stored line is passed through the rulewarden secrets pack. A line that matches is replaced by its SHA-256 and labels only. E-mail addresses and @handles are masked.

Extraction

corpus/extract.py: a line (or bullet item) is rule-like if, after stripping markdown, it contains an imperative or modal cue from a fixed lexicon. The lexicon is _CUE_RE in src/rulewarden/compile/extract.py, the same extractor rulewarden compile uses, so the study measures what the tool sees. Code blocks, headings, tables and HTML comments are excluded. The extractor is deliberately broad, so it also catches descriptive lines (for example "Tests use .mjs"); the model labels flag those with is_rule (below). Output: corpus/data/rules.csv (file id, line number, text or hash).

Labelling

  1. Deterministic rubric (corpus/label.py, primary labels): ordered regex rules map each line to a category (style, tooling, scope, process, testing, security, voice, other; first match wins) and a decidability tier (P3), judged against a checker that sees only the diff and the commands:
  2. deterministic: a concrete token (code span, path, file extension, tool name, number with a unit, named syntax such as console.log or camelCase) together with a prohibition ("never", "do not", "avoid", a leading "No"), or together with a prescription (a modal, or an imperative such as "Use"/"Keep" at the start) that is not an obligation to act;
  3. obligations to act ("Run …", "Check …", "Ask …") are undecidable from a diff unless phrased as a prohibition, because a diff cannot show that something was done;
  4. heuristic: an approximate-quality cue (short, concise, duplication, nesting, redundant, …);
  5. undecidable: everything else (intent, judgement, vague quality words). The rubric also tags whether a shipped compile template matches the line (compilable, i.e. rulewarden compile would produce a rule for it). tests/test_corpus_rubric.py pins the documented examples. The rubric was revised once, before any model labels existed: a 40-line sample of the first 120 files showed descriptive lines ("Tests use .mjs extension") labelled deterministic because of a mid-sentence "use".
  6. Model-assisted labels (secondary): the same lines are labelled by claude -p --model haiku (Haiku 5.5), with no tools, no settings files, no MCP servers, in an empty directory, using the fixed prompt corpus/prompts/label_v1.txt, in batches of 80 lines, in two independent passes (pass 2 sees each batch in reverse order). The model also returns is_rule (does the line instruct the agent, or only describe the project?). The CLI exposes no temperature control, so we do not claim temperature 0. Cohen's κ is reported for pass 1 vs pass 2 and for rubric vs pass 1. Raw responses are kept in corpus/data/model_raw/.
  7. Shares are reported for all extracted lines and for the rules-only subset (both passes say is_rule), with 95% bootstrap CIs that resample files (2,000 resamples, fixed seed), because lines are nested in files.
  8. The paper states that labels are rubric-based and model-assisted, and reports both.

Outputs

corpus/data/files.csv, rules.csv, labels.csv, labels.parquet, summary.json; figures in paper/figures/. Rebuild the analysis offline: make corpus (needs only corpus/data/).