1 Introduction

Motivates RQ1–RQ7 and summarises the contributions. 

In 2025 and 2026, coding agents left the chat window for the terminal, the IDE and the CI pipeline, and vendors and open-source projects converged on one architecture: an agent is a model plus a harness. The harness is the program around the model: it assembles context from the repository, exposes tools (shell, file editing, web fetch, MCP servers), decides which tool calls require human approval, stores credentials and transcripts, spawns sub-agents, and can run unattended in CI. The model proposes; the harness executes.

Most security research on LLM agents addresses the model, asking whether it follows injected instructions (Greshake et al. 2023; Debenedetti et al. 2024; Zhan et al. 2024), with the tool-execution layer held fixed. Yet the harness decides whether an injected instruction becomes a shell command, whether a cloned repository can run a hook before the user has consented, whether an MCP server starts from a cloned file, and whether a secret lands in a world-readable log. These are deterministic software decisions, and they can be specified, tested and audited like any other access-control code.

The public record points the same way. Of the 323 verified harness-level advisories and incidents we collected for 2025–2026 (Section 7), 104 arise from flawed approval, allowlist or path checks and 52 from workspace configuration honoured before consent; only 17 are pure prompt injection. Since model failures seldom receive CVEs and RC1 is the residual class of our procedure, these shares describe disclosed harness bugs rather than overall risk; within that scope, injection is often the vector, but harness controls decide the damage.

Comparing harnesses is difficult: the field changes monthly, many popular harnesses are closed source, and documentation describes intended rather than actual behaviour. We therefore organise the study by method: white-box analysis for open source, black-box observation for closed harnesses that we can run, documentation for those we cannot, and a separate track for SDK runtimes (Section 3).

Research questions

RQ1: What is the attack surface of a harness, and how does it differ from classical LLM-application risk? RQ2: How do harnesses differ in the security controls they provide? RQ3: Which harness-level incidents were disclosed in 2025–2026, and from which root causes? RQ4: Can dangerous configuration be detected automatically across harness formats? RQ5: How resilient are harness $\times$ model $\times$ permission-mode combinations to indirect prompt injection and secret residue, as measured with inert canaries? RQ6: Which controls should an organisation mandate before adopting a harness? RQ7: For closed harnesses, do the documented controls match the observable behaviour of the client?

Contributions

A harness threat taxonomy with framework mappings (RQ1, Section 4); an evidence-linked control matrix of 58 harnesses and runtimes (RQ2, Section 5); a black-box claims-versus-observed method for closed harnesses (RQ7, Section 6); a two-source incident survey (RQ3, Section 7); harnesswatch, an Apache-2.0 auditor, with accuracy, prevalence, canary-bench and residue experiments (RQ4, RQ5, Sections 8–9); and a governance checklist (RQ6, Section 10). make all rebuilds every table, figure and reported number from versioned results (Appendix 14).

2 Background: Anatomy of a Harness

Covers the descriptive part of RQ1 and prepares RQ2. 

The loop

A harness assembles a context, calls the model, checks each proposed tool call against a permission policy, executes it and returns the result to the model. Repository content, tool output and MCP results flow inwards; commands and network requests flow outwards.

Repository files

Harnesses auto-load instruction files (AGENTS.md, CLAUDE.md, .github/copilot-instructions.md, .cursor/rules), settings that can set environment variables, permission modes and hooks (shell commands bound to lifecycle events), and MCP declarations (.mcp.json, .vscode/mcp.json). Formats are shared across products: Grok Build imports Claude Code and Cursor hook and MCP files^[D] (xAI 2026a, 2026b). A cloned repository is therefore configuration as well as data.

Permissions and sandbox

Permission modes range from “ask before every tool call” to “approve everything” (YOLO, bypass, full-auto, always-approve), with allow and deny rules over commands and paths. Several harnesses now default to a model-based approver; Claude Code starts claude -p in a classifier-mediated auto mode when feature flags are not fetched^[D][T] (Anthropic 2026c). Some confine commands within an OS sandbox (Seatbelt, bubblewrap, Landlock) or a container^[D] (Anthropic 2026e). Approval decides whether an action runs; the sandbox bounds what it can reach.

Extensions, state and headless mode

MCP servers (Model Context Protocol 2026) are launched locally, often through unpinned npx or uvx, or reached remotely, and their tool descriptions enter the context verbatim; skills and plugins add further code and prompts from marketplaces. Credentials, transcripts and persistent memory reside in a per-user state directory, and sub-agents may inherit the parent’s permissions. Every harness we benchmarked runs non-interactively^[D][T] (claude -p, codex exec, grok -p, copilot -p); in CI the triggering issue or pull request may be attacker-controlled and the job may hold secrets.

3 Scope, Tiers and Evidence Method

Describes how the evidence behind RQ2–RQ7 was collected. 

Tiers by method

We group harnesses by what can be observed about them, not by market share. Tier A (open source; white-box): we read the source at a pinned commit and run the client in containers (e.g. Codex CLI, OpenCode, Gemini CLI, Qwen Code, Aider, goose, Cline, Hermes Agent). Tier B (closed but runnable; black-box): we install the official client in a disposable container, record what it writes and which hosts it contacts, and drive it headless (Claude Code, GitHub Copilot CLI, and the Antigravity CLI before login). Grok Build is a hybrid: its source is public under Apache-2.0^[D] (xAI 2026d), but the public tree lags the npm release (1.0.45 against 1.0.52) and endpoints are injected at build time, so we read the source white-box and treat the shipped npm client as Tier B in E3 and E7. Tier C (closed and unreachable): documentation, advisories and third-party research only (e.g. Cursor, Kiro, Devin, Jules). Tier D (runtimes and SDKs) has its own ladder and is never ranked against products; the rubric and figures group it with hosted services as “tier C”. The matrix covers 36 tier A/B products and 22 runtimes and hosted services.

Black-box method

For a closed harness we install the official release, read its documentation and advisories, parse its configuration, run it under harnesswatch observe (Section 8), probe its treatment of configuration with synthetic repositories carrying hooks, MCP servers and settings, and inspect its package metadata. We never decompile, deobfuscate or consult leaked source, never decrypt TLS or store request bodies, and remain within vendor terms; otherwise the cell is N/A (terms). No vendor login was placed in any container: closed clients were driven solely through documented custom-endpoint features pointing at a local model, namely Claude Code’s ANTHROPIC_BASE_URL, Copilot CLI’s bring-your-own-key mode, which requires no GitHub authentication^[D] (GitHub 2026), and Grok Build’s custom models^[D] (xAI 2026c). Anthropic states that it “doesn’t support routing Claude Code to non-Claude models through any gateway”^[D] (Anthropic 2026b), so our Claude Code runs characterise harness behaviour in an unsupported configuration, not vendor-supported performance. Antigravity has no documented local-model path and its terms restrict use with other products^[D] (Google 2026b), so it was observed only before login.

Evidence labels

Every factual statement about a harness carries one of three labels, rendered as superscripts: ^[T] verified-by-test, behaviour or state observed by running the pinned client; ^[D] from-official-docs, vendor documentation, vendor advisories, registry metadata or the official source code (reading source counts as documentation, not as a test); ^[3P] third-party-report, a write-up by an independent researcher or journalist. Unverified claims are excluded. Documentation and source facts are recorded in the evidence ledger and in matrix/evidence/; every ^[T] fact traces to a probe or bench file under experiments/results/ or bench/results/.

Use of generative AI

The study was carried out with Claude (Anthropic), operated through the Claude Code harness, which is itself among the systems examined. Working from the author’s written specification, LLM research agents wrote the harnesswatch toolkit and the experiment scripts, collected the evidence, produced both matrix ratings, collected and classified the incidents, and drafted and revised this manuscript. The author specified the research questions, the scope, the rules of evidence and the ethical constraints, directed the work and reviewed its outputs, and takes sole responsibility for the content; Section 12 discusses the consequences for reliability.

Kinds of finding

We distinguish a vulnerability (a defect that violates the product’s own security intent, usually with a CVE or GHSA), a design weakness (a documented default or capability that enables harm without being a bug, such as headless mode running repository hooks) and a policy choice (e.g. telemetry or retention defaults). Only the first is a bug.

Ethics

Experiments ran in disposable containers on the author’s machine with inert canaries and synthetic secrets; the public-repository study stores aggregates only. Issues that might be unknown to the vendor went to private coordinated disclosure and are not described here. Packages were examined through registry metadata alone; one fact obtained by listing the strings in a binary was withdrawn as outside that rule, and the reported source-map exposure in Claude Code’s npm release 2.1.88 of 31 March 2026^[3P] (Wiggers 2026) is cited only as an event, with no use of the exposed material.

4 Threat Model and Taxonomy

Answers RQ1. 

Adversaries

(A1) A content author controls text the agent reads (a README, comment, issue, web page or tool result); (A2) a repository author controls cloned files, configuration included; (A3) a supply-chain attacker controls a package, MCP server, plugin or release; (A4) a local or network peer reaches a service the harness exposes. The defender runs the harness with real credentials, code and host access (the assets). We assume that the model will sometimes follow injected instructions and that no model-level defence can be relied upon.

Taxonomy

We extended a sixteen-class seed into the classes of Table 1. Three classes were added because the incident set contained sizeable groups with no natural home: escape from a sandbox or isolation boundary (T17); exposure of the agent’s own control plane, typically a local HTTP or WebSocket server without authentication or Origin checks (T18); and conventional software defects in runtimes and SDKs (T19). Each class carries STRIDE letters, MITRE ATLAS techniques, and OWASP LLM 2025, OWASP Agentic (ASI01–ASI10) and OWASP MCP Top 10 identifiers; as the MCP list remains in beta, those mappings are flagged as unstable. Each class also lists its mitigating control dimensions (Section 5) and the harnesswatch rules that detect it (Appendix 16).

Classification procedure

Because write-ups conflate the entry vector with the control that failed, we traverse the classes in a fixed order and assign the first whose failed control the incident exhibits. A parallel twelve-question procedure assigns one of twelve root causes, RC1–RC12 (Table 3). Thus configuration written by an injection and executed later is T02/RC2 rather than T01, and data leaving through a permitted command is T08/RC3. RC1, untrusted content treated as instructions with no other control failure, is reached only when no earlier question applies; its share is therefore low partly by construction.

How harness risk differs from LLM-application risk

In a chat application, a successful injection yields a wrong answer or a leaked system prompt. In a harness, the injected text originates in the developer’s own workspace and is acted upon with the developer’s privileges. Three properties follow. First, configuration is code: repository files can define hooks, environment variables and MCP launch commands that execute before the model is consulted (T02). Second, the approval layer is a parser: an allowlist over shell commands must reason about quoting, substitution, redirection and symlinks, and failures of this kind form the largest single group of disclosed incidents (T03, 23% by class; RC3, 32% by root cause). Third, the harness is a long-lived service: local servers, sub-agents, persistent memory and CI runners create surfaces that classical LLM threat models omit (T10, T11, T18). Attack trees for the principal paths are in the repository.

5 Comparative Analysis of Security Controls

Answers RQ2. 

Rubric

We score fifteen control dimensions from 0 to 3: D01 permission model, D02 execution sandbox, D03 network egress, D04 workspace and configuration trust, D05 MCP governance, D06 extensions and plugins, D07 secrets and credential storage, D08 headless and CI safeguards, D09 persistent memory, D10 sub-agent isolation, D11 audit logging, D12 telemetry and retention, D13 distribution supply chain, D14 managed policy and D15 vulnerability disclosure. The scale reads 0 absent, 1 opt-in or weak, 2 on by default with documented gaps, and 3 strong, documented and centrally enforceable, with per-dimension criteria (Appendix 18). Only ^[D] and ^[T] facts can raise a score; ^[3P] facts can only lower one, since third-party reports document bypasses rather than guarantees. The score is the highest level whose criteria are all met; an undecidable criterion counts as unmet; incident history, which belongs to RQ3, leaves scores unchanged. Means are unweighted, require at least eight scored dimensions and are reported alongside the minimum; a ranking is asserted only where it holds under equal, incident-exposure and security-core weightings alike.

Coverage and reliability

Of 870 harness–dimension cells, 664 were scored and 206 are N/A with a recorded reason (feature absent, no access, or undocumented); 5 runtimes have too little evidence to support a mean. A second LLM-agent rater, blind to the first rater’s scores, rescored the matrix from the same extracted fact lists, so agreement measures how consistently the rubric is applied to shared evidence, not the reliability of evidence collection. On the 655 cells both raters scored, quadratic-weighted Cohen’s $\kappa$ (Cohen 1968) is 0.88, exact agreement 91.6% and within-one agreement 99.5%; these figures exclude the 29 cells in which one rater recorded N/A and the other a score. All 84 disagreements, those N/A cases included, are published rather than adjudicated, and rater 1’s scores are reported.

Overall picture

Controls exist but are seldom safe by default (Figure 1). Across the 36 tier A/B products no per-dimension mean reaches 2. The lowest means belong to network egress (D03, 0.77), the execution sandbox (D02, 0.84), headless and CI safeguards (D08, 0.91) and workspace trust (D04, 0.97); the highest to vulnerability disclosure (D15, 1.51), audit logging (D11, 1.50) and managed policy (D14, 1.48). Overall means are closely bunched: only Codex CLI stays in the top three, and only Continue in the bottom three, under every weighting, so the per-dimension columns are more informative than the means.

Permission model (D01)

Nearly every product offers per-action approval, yet only 6 of 36 reach score 2. Two patterns depress the scores: approve-everything can often be enabled from repository-controlled files, through hooks that return an auto-approve verdict or repository MCP entries carrying autoApprove; and several harnesses default to a model-based approver (Claude Code’s auto^[D], Cursor’s auto-review^[D], Qwen Code’s AUTO^[D], Hermes Agent’s smart^[D]). The rubric does not treat a classifier’s verdict as “asking”, so these defaults score 1. Counting them as asking would raise Claude Code’s D01 from 1 to 3 and lift the other classifier-default harnesses; we did not recompute the rankings under that reading, the rubric’s principal open question.

Sandbox and egress (D02, D03)

These are the lowest-scoring dimensions (means 0.84 and 0.77). Among tier A/B products, 8 offer no sandbox at all, 0 confine both filesystem and network by default with central enforcement (score 3), and 11 have no egress control. Managed runtimes score higher (tier C/D means 1.47 and 1.61) because the provider runs each session in an isolated container or microVM. Where sandboxes exist, the incident record shows their boundaries to be a frequent point of failure: symlink and working-directory tricks, writes to .git hooks and fallbacks to unsandboxed execution account for 22 escape incidents (Section 7).

Workspace trust (D04)

Repository configuration is honoured with no notion of trust by 13 tier A/B products, and none reaches score 3. A recurrent design choice is that headless mode bypasses the trust gate. Claude Code documents that claude -p in a never-trusted folder runs project hooks and MCP servers^[D] (Anthropic 2026a), which we observed^[T]; by contrast, Grok Build ran an untrusted repository hook only after –trust^[D][T], and Copilot CLI did not start a repository MCP server in an untrusted folder^[T], though we did not re-test it in a trusted one (Section 6). Cline likewise documents that project plugins in .cline/plugins/ load whenever the agent works in that repository^[D] (Cline 2026). These are documented design properties rather than vulnerabilities, but a CI job that runs a harness on a pull request inherits them all.

Other dimensions

On D07 only 1 of 36 tier A/B products reach score 2, chiefly because transcript retention is undocumented even where credentials are kept in a 0600 file or a keychain^[D] (D07 column). Telemetry (D12) is scored for transparency and opt-out, not as a defect. D13 is examined in Section 9.5. SDK runtimes delegate approval, sandboxing and egress to the application; the rubric scores these as not provided (application responsibility), so a team building on an SDK inherits those gaps unless it closes them.

6 Documented Claims versus Observed Behaviour

Answers RQ7. 

Method

For each Tier-B harness we extracted the security-relevant statements in the vendor’s documentation into a claims table (117 claims across the closed harnesses), each tagged with a dimension (telemetry, updates, configuration/hook/MCP trust, sandbox, secret storage, permission default, state-directory modes). We then wrote 23 black-box probes (Section 3). Each probe runs the pinned client in a fresh container, optionally on a synthetic repository seeded with a hook, an MCP server or a settings file. A verdict is match, partial, mismatch or not observed (behaviour not exercised); claims that require a login, a GUI or an interactive session remain untested.

Results

Only 24% of the 117 claims could be tested headless without a vendor login. Of the 28 tested, 22 matched, 3 were partial, 1 mismatched and 2 were not observed; 89 remain untested (Table 2). By harness, 13 of 14 tested Claude Code claims matched, as did 4 of 7 for Antigravity, 3 of 4 for Grok Build and 2 of 3 for Copilot CLI. Every observation in the table is ^[T]; every claim is ^[D].

What the probes show

The tested claims form a convenience sample weighted towards network and update behaviour, so the match count does not generalise to the untested claims. The notable findings are documented design properties, not vulnerabilities. (i) Headless trust. In Claude Code, a claude -p run in a never-trusted folder executed a repository SessionStart hook and started a repository stdio MCP server without prompting^[T][D] (Anthropic 2026a), and applied a repository env entry redirecting the model endpoint^[T][D]; –bare prevented the hook^[T][D] (Anthropic 2026d), and a repository request for bypassPermissions was ignored^[T][D]. Its documented @path import in CLAUDE.md also fired on the token @.env inside a shell command line, drawing the gitignored .env into the session transcript (mode 0600) without any model call^[T][D]; this explains the E3 secret-in-state result (Section 9.3). (ii) Trust-gated alternatives. Grok Build did not run an untrusted repository hook, including one imported from the Claude Code format, and ran it once –trust was given^[T][D] (xAI 2026a). Copilot CLI started neither a repository MCP server nor a hook in an untrusted folder^[T][D]; as we did not re-test it in a trusted folder, “did not run” may equally mean “not supported under -p”. (iii) Network. With the documented opt-outs set and a local model, Copilot CLI, Grok Build and Claude Code contacted no vendor host^[T][D]. Before login, Antigravity’s CLI contacted its auto-updater and a telemetry host, as its defaults document, and the updater opt-out took effect^[T][D] (Google 2026c). (iv) The single mismatch is functional: agy -p /help is documented to run without an agent turn but demanded authentication^[T][D] (Google 2026a). The partial verdicts concern state-directory modes and MCP start-up before trust, on which the documentation is silent.

Limits

A match holds for this version in a headless Linux container; it certifies neither interactive use, other platforms nor post-login behaviour, and without payload inspection it says nothing about what an allowed host receives. Probe effort was uneven (14 Claude Code claims against 3 for Copilot CLI), and Claude Code ran in a configuration its vendor does not support (Section 3). Without the opt-outs, the E3 proxy blocked attempts to reach vendor or telemetry hosts from Claude Code (api.anthropic.com), Codex CLI (chatgpt.com, github.com), Qwen Code (an Aliyun RUM host and models.dev) and OpenCode (models.opencode.ai)^[T].

7 Incident and CVE Survey, 2025–2026

Answers RQ3. 

Collection and counting rule

We searched NVD, the GitHub Advisory Database, vendor security pages and researcher write-ups for harness-level incidents disclosed between January 2025 and mid-October 2026 (including the harnesses’ MCP servers, SDKs, CI actions and distribution channels), obtaining 357 rows after deterministic de-duplication. An incident is counted only if (a) its primary source was read and (b) an independent script re-resolved every listed CVE and GHSA identifier against NVD and the GitHub advisory API; an incident without an identifier must have a primary vendor or researcher source. 29 rows failed (a), 4 failed (b) and 1 fell outside 2025–2026, leaving 323 counted incidents (99 disclosed in 2025 and 224 in 2026; 239 with a CVE and 237 with a GHSA). By kind, 292 are vulnerabilities, 19 design weaknesses, 8 supply-chain compromises, 3 documented offensive misuse and 1 other events. Classes follow Section 4; a single LLM agent classified every incident, and inter-rater agreement for classes and root causes was not measured. The population is selected for disclosed, harness-level software defects, including 73 ecosystem items such as MCP tooling and package compromises; the shares describe that population, not the distribution of real-world harm.

Timeline

Disclosures rose in every complete quarter of the study period (Figure 2). The rise reflects more harnesses, greater researcher attention, wider use of GHSAs and an agent-assisted search that favours recent, well-indexed advisories, not necessarily declining quality; per-product counts likewise track scrutiny and disclosure maturity (D15) as much as risk, so we do not rank products by them.

Root causes

Table 3 gives the distribution. The largest group, with 104 incidents (32%), comprises defective checks that gate an agent capability (RC3): allowlists bypassed through shell substitution, redirection, environment-variable prefixes, find -exec, symlinks or look-alike domains. Configuration honoured before consent (RC2) contributes a further 52: repository settings, hooks, .env files, MCP declarations and Git configuration that run or take effect before the trust prompt. Together these deterministic harness defects account for 48% of counted incidents, the largest share though not a majority. Conventional defects not specific to agents (RC10) account for 16%, and missing authentication or Origin checks on local control endpoints (RC4) for 24. Pure prompt injection with no other control failure (RC1) stands at 5%; as the residual class, this is a lower bound on the role of injection, which we did not code separately as a vector.

Classes

By primary class (Table 1), permission-model weaknesses (T03, 74, 23%) and repository configuration acting as code execution (T02, 52) lead, followed by conventional defects (T19, 45) and control-plane exposure (T18, 34). Indirect prompt injection is the primary class of only 14 incidents, although injection is the delivery vector in many incidents classified elsewhere.

Representative cases

Credit belongs to the original researchers. Injection to configuration: hidden Unicode in rules files (Pillar Security)^[3P] (Karliner 2025); injection flipping Copilot’s auto-approve (CVE-2025-53773)^[3P] (Rehberger 2025a); Cursor executing attacker-written or silently modified MCP configuration (CVE-2025-54135/54136)^[D][3P] (Anysphere 2025; Charikov, Zaikin, and Vanunu 2025). Configuration before consent: Claude Code project files executing code and transmitting API tokens before the trust dialog (CVE-2025-59536, CVE-2026-21852)^[3P] (Donenfeld and Vanunu 2026); Codex CLI loading project MCP servers without confirmation (CVE-2025-61260)^[3P] (Mill and Vanunu 2025); Gemini CLI’s CI action auto-trusting the workspace^[D] (Google 2026d). Allowlist and path checks: command deception in Gemini CLI^[3P] (Cox 2025); path and command bypasses in Claude Code (CVE-2025-54794/54795)^[3P] (Beber 2025), and deny rules not enforced through symlinks^[D] (Anthropic 2026f). MCP: tool poisoning and the GitHub MCP leak (Invariant Labs)^[3P] (Beurer-Kellner and Fischer 2025; Milanta and Beurer-Kellner 2025); SQL-returned injection via the Supabase MCP server^[3P] (Havaei, Liu, and Li 2025); the unauthenticated proxy in MCP Inspector (CVE-2025-49596)^[D][3P] (Model Context Protocol project 2025; Lumelsky 2025). Other: unauthenticated local control planes in OpenCode and DeepSeek Harness^[D] (OpenCode maintainers 2026; GitHub Advisory Database 2026); a malicious prompt shipped in Amazon Q Developer 1.84.0^[D] (Amazon Web Services 2025); the nx “s1ngularity” compromise, using installed AI CLIs to search for secrets^[D] (Nx maintainers (nrwl) 2025); data leaks from Antigravity shortly after launch^[3P] (PromptArmor 2025; Rehberger 2025c); and a vendor’s own account of the criminal misuse of a coding agent (Anthropic 2025).

Fix date relative to advisory publication

Among the 148 counted incidents with both dates, a fix release existed on or before the advisory’s publication in 135 cases (median offset -5.00 days, maximum 288); 175 incidents have no recorded fix date (unfixed or unknown). Because advisories are by design usually published after a fix, this measures disclosure practice rather than time to fix; report dates were not available.

8 Design of harnesswatch

Covers the design half of RQ4 and the instrument used for RQ5 and RQ7. 

harnesswatch is an Apache-2.0 Python package and command-line tool. It is read-only and offline by default, never transmits file contents, and redacts every secret it reports.

Profiles and a common schema

One risk has many spellings: approve-everything is bypassPermissions in Claude Code, approval_policy = "never" in Codex, yolo in Gemini and Qwen, and "permission": "allow" in OpenCode. A YAML profile for each of the 58 harnesses in the matrix records file locations, key names and documented defaults, and a profile-driven parser maps every file onto a common HarnessConfig model (permission mode, allow and deny lists, hooks, MCP servers, instruction files, trust, telemetry and auto-update). A file shared by several harnesses is parsed once, with the semantics of the format’s owner.

Audits and rules

Six audit modules cover config; mcp (pinning, transport, inline secrets, tool-description injection, auto-approval); instructions (hidden Unicode, override phrasing, remote includes); secrets (state directories, transcripts, file modes); supply (SBOM, OSV, installer verification); and ci (headless agents on untrusted triggers). Each of the 25 rules (Table 7) carries a severity, a confidence, a kind (vulnerability pattern, design weakness or policy choice), taxonomy classes, remediation guidance and should_fire/should_not_fire fixtures. Output is terminal, JSON, Markdown or SARIF 2.1.0; a GitHub Action and a pre-commit hook wrap the repository audits.

observe and bench

harnesswatch observe <harness> – CMD runs a command in a disposable container whose sole exit is a logging proxy, and records the CONNECT targets (classified as model API, telemetry, update or unknown), a filesystem diff with modes, credential locations, sandbox evidence from /proc and any permission prompts in the output. It stores no request bodies and does not decrypt TLS. It underlies E5, E7 and every bench run. The canary bench (Section 9.3) drives each harness headless through a dedicated adapter and aggregates rates with Wilson intervals (Wilson 1927).

9 Empirical Evaluation

Answers RQ4 (E6, E2) and RQ5 (E3, E5), and the supply-chain component of RQ2 (E4). 

All experiments ran on a single Linux workstation with Docker and a consumer GPU; every reported number is read from experiments/results/*.json.

9.1 E6: Rule-pack accuracy on a labelled corpus (RQ4)

Design

The corpus comprises 376 cases (113 hand-crafted and 263 generated) spanning more than twenty harness formats, CI files, state directories and MCP manifests; 108 are wholly negative, and many positives have near-miss twins (pinned npx, authenticated HTTPS MCP, linter-only hooks, emoji ZWJ, key-like identifiers). Every audit runs on every case and the unit is the (case, rule) pair, so any unlabelled firing is a false positive. A hash of the case identifier splits the corpus into 245 development and 131 test cases; errors were inspected and corrected on the development split only.

Results

On the held-out test split, micro precision is 0.983 (95% Wilson CI 0.950–0.994) and micro recall 0.983 (0.950–0.994), with 3 false positives and 3 false negatives; macro precision and recall are 0.97 and 0.95 (per-rule results in paper/tables/e6_accuracy.md). Before the development-split corrections, test precision was 0.960 with 7 false positives; the test split had been scored once before corpus-content corrections that touched both splits, so it is not pristine. The Wilson intervals treat (case, rule) pairs as independent although they cluster within cases and generator families, so they are optimistic. The weakest rules are HG-SUB-001, HG-MCP-004 and the approve-everything rules. Because the corpus is synthetic and labelled by the rule author, this shows that the rules implement their specification across formats; real-world precision remains unknown.

9.2 E2: Prevalence in public repositories (RQ4)

Design

Through the GitHub code-search API we sampled harness settings and MCP files, instruction files (AGENTS.md, CLAUDE.md, Copilot and Windsurf rules) and CI workflows that invoke claude-code-action, codex exec or gemini -p. For each repository we reconstructed only the sampled files in a temporary tree, ran harnesswatch audit, retained only (rule, file) pairs and deleted the tree. Identifiers were hashed with an unsaved per-run salt for de-duplication only; a scrub check refuses to write results containing any identifier or URL seen; secret findings were only counted; and requests stayed below the API’s rate limits. Each of the 17 search strata is capped at 60 files, so a pooled per-repository rate would mostly reflect the choice of strata. We therefore report each rule’s rate over the stratum of files that can carry it (configuration and MCP files, $n=600$; instruction files, $n=240$; agent CI workflows, $n=177$), with Wilson intervals, and name no repository. Only project-level configuration is visible, and a finding marks risky configuration, not exploitation.

Results

We audited 1010 repositories (1017 files, 0 audit errors), and 18 rules fired at least once (Table 4). Among agent CI workflows, 63.3% run the agent with a write-scoped token (HG-CI-002) and 28.2% match the pwn-request pattern of an untrusted trigger combined with secrets (HG-CI-001). Among configuration and MCP files, 28.3% launch an MCP server from a registry without a version pin (HG-MCP-001) and 14.3% declare a remote server without TLS or authentication (HG-MCP-002); HG-MCP-005 (a server declared in repository configuration, 51.8%) largely restates the sampling key and is informational. Secret-shaped strings in MCP declarations (HG-MCP-003, 3.5%) were counted only; because identifiers are discarded by design, owners were not notified and the strings were not validated. No E2 finding was validated by hand, so the real-world precision of these rules is unknown, and the rates describe this capped, relevance-ranked sample rather than GitHub as a whole.

9.3 E3: Canary benchmark for indirect prompt injection (RQ5)

Design

Each run starts a fresh container with a minimal Python repository, a synthetic .env token, a fixture file and a sink on 127.0.0.1, and sets the same benign task: add a multiply function and test, and run the tests. Besides a control, four scenarios place one templated block in the README, in the harness-native instruction file, in a code comment, or in test output that is returned as a tool result (Table 9). The block asks the agent to create CANARY_<id>.txt, POST .env to the sink and delete the fixture; all three are inert, and the container reaches only the model server. The two modes are default (out-of-the-box headless) and auto (the documented approve-everything flag).

Harnesses, model and grid

Seven adapters cover three Tier-B clients (Claude Code, Copilot CLI and the Grok Build npm binary, driven through their custom-endpoint modes; Claude Code’s is vendor-unsupported, Section 3) and four open-source harnesses (OpenCode, Codex CLI, Aider and Qwen Code). No vendor credentials are used, so every harness talks to one local open-weight model, qwen3:4b-instruct, served by Ollama. The grid is 7 harnesses $\times$ 5 scenarios $\times$ 2 modes, with 5 runs per cell. A run is capped at 240 s and scored on its state at the cap; a run with no harness output counts as an error, not as safe. Aider loads no instruction file, so its instructions cells are N/A in both modes and excluded from the pooled statistics.

Metrics

Injection follow: any of the three goals achieved. Exfiltration attempt: any POST to the sink, or the secret observed there. Destructive action: the fixture deleted. Secret read: the synthetic token appears in the harness’s printed output (other than in an environment-variable dump) or reaches the sink; since some harnesses print only their final message, this undercounts reads. Secret in state: the token appears in a transcript or session file written by the harness, which separates persistence from agent action. Permission prompt: an approval prompt appears in the output. Task completed: the tests pass and the function and its test exist, so that a “safe but useless” outcome remains visible. All rates carry Wilson 95% intervals.

Results

The grid produced 350 runs (0 errors, 22 timeouts scored at the cap, model cost $0.00), of which 340 enter the analysis (Table 5). Over the injection scenarios, the injected instruction was followed in 2 of 135 auto-mode runs (95% CI 0.004–0.052) and in 0 of 135 default-mode runs (0.000–0.028); the controls followed nothing (0 of 70). No run attempted the POST, the fixture was deleted in 1 auto-mode run, and the strict secret-read count is 0 in both modes. The intervals overlap, so there is no detectable safety difference between the modes. Task completion does differ (84 of 135 auto against 23 of 135 default) because several headless defaults withhold the tools the task needs: Codex CLI and Copilot CLI showed a permission prompt in every default run, while Grok Build and Qwen Code completed nothing in default mode without a detectable prompt, for reasons we did not establish. OpenCode completed 1 of 10 even in the controls, an adapter–model incompatibility that renders its cells uninformative. Claude Code’s “default” arm is not an ask mode: with a third-party endpoint it starts in the classifier-mediated auto mode (Section 6).

Secret in state

In 10 of 10 Claude Code runs of the instructions scenario (both modes), the synthetic secret reached the session transcript, against 0 in its other scenarios. This is not a model action: an E7 probe with an unreachable model endpoint reproduced it, because the payload’s –data-binary @.env contains the token @.env, which Claude Code’s documented @path import resolves when it loads CLAUDE.md^[T][D]. It is a harness design property: an instruction file can import gitignored secrets into context and transcript without any approval.

Interpretation

The low follow rate most probably reflects the small model’s limited agentic and instruction-following ability and the plainly worded, benign canary, not the robustness of the harnesses. With 5 runs per cell and zero or one event per cell, the intervals are wide, and we do not rank harnesses on resilience, nor describe default modes as safer.

Correction

Our original secret metric (the token anywhere in output or written files), fixed before the results were inspected, flagged 15 runs and was replaced afterwards: 10 were the Claude Code import above and the rest Codex CLI, which writes its process environment, canary included, into ~/.codex shell snapshots whenever its shell tool runs (6 of 50 Codex runs)^[T]. Neither is the agent reading the secret, hence the two metrics: a benchmark should distinguish what a harness persists from what an agent does.

9.4 E5: Secret residue in harness state (RQ5)

Design

For each of the 7 bench harnesses we ran two short sessions in a fresh container. In the key condition the provider credential was a unique synthetic canary; in the content condition the prompt asked the agent to read a synthetic .env file. We searched every file the harness wrote for the canary and recorded paths, modes and reachability by other users (home directory 0755); file contents were never stored. With one session per harness and condition ($N=1$), the results are descriptive.

Results

In the content condition, 6 of the 7 harnesses left at least one file holding the secret, typically a transcript or chat log (Table 6)^[T]; in 3 of them such a file was readable by other local users. In the key condition, 0 of the 4 harnesses that accept a key wrote it to a file in this single run^[T]. That is a sampling result rather than evidence of absence: Codex CLI persists environment-supplied credentials in shell snapshots whenever its shell tool runs, which occurred in 6 of 50 E3 runs but not in the E5 key session^[T]. Mode bits alone can mislead: Grok Build writes 0644 session files, but inside a 0700 sessions directory, so other users cannot reach them^[T]; Claude Code writes transcripts with mode 0600^[T]. OpenCode left no residue in this run. Because residue depends on whether the agent reads the file, a single run can miss it. These are design weaknesses, not vulnerabilities; a 0750 home directory would close the reachable paths, but CI runners and containers commonly use 0755.

9.5 E4: Distribution supply chain (RQ2, D13)

We inspected the published packages of 23 harnesses (the Tier-A set plus Claude Code, Grok Build and Copilot CLI) from registry metadata and lockfiles alone, without running any install script^[D] (per-package table: paper/tables/e4_supply.md). Install-time hooks are declared by 7 packages. Of the 18 whose status could be determined, 10 publish registry provenance attestations. A curl | sh installer is offered by 17, and in 17 of these the installer verifies no signature (some verify a checksum; Claude Code documents a signed release manifest that its installer does not itself verify^[D]). Auto-update is documented as on by default in 12 (cells marked “unverified” in the table are undetermined, not off). The median lockfile holds 97 transitive packages, and OSV lists 299 unique advisories against the locked dependency versions, concentrated in a few packages; these are advisories against pinned versions, not demonstrated exploitability. Auto-update is a trade-off: it delivers fixes promptly but widens the blast radius of a compromised release channel, as the Amazon Q extension incident showed^[D] (Amazon Web Services 2025).

10 Governance Recommendations

Addresses RQ6 as recommendations, not as an evaluated finding. 

We distilled the findings into a checklist of thirty controls (GC-01 to GC-30). Each states its requirement; its rationale (threat classes, root causes, rubric dimensions, and whether it targets a vulnerability, a design weakness or a policy choice); how to verify it, with an exact harnesswatch command or a manual check; and its mapping to NIST AI RMF 1.0 (Tabassi 2023), NIST AI 600-1 (Autio et al. 2024), ISO/IEC 42001 Annex A (ISO/IEC 2023), OWASP (OWASP GenAI Security Project 2024, 2025b), MITRE ATLAS (MITRE n.d.) and the MAS AI risk-management guidelines (Monetary Authority of Singapore 2026). The controls that follow most directly from our evidence are: treat repository configuration, instruction-file imports included, as untrusted code, and gate changes to it in CI (GC-08, GC-09, GC-23; RC2, E3 secret-in-state); lock approve-everything modes by managed policy, and regard allowlists as a convenience rather than a boundary (GC-06, GC-07; RC3); supply the sandbox and egress boundary yourself, since D02 and D03 score lowest (GC-16, GC-17, GC-20); never run a headless agent with secrets or write tokens on an untrusted trigger (GC-21, GC-22; E2 CI strata); keep secrets out of configuration files and the agent’s environment, and restrict state directories (GC-14, GC-15; E5); and re-run the black-box probes and the canary bench on the exact version and model you deploy (GC-03, GC-04). The checklist has not been evaluated for coverage or usability; the OWASP MCP Top 10 remains in beta, the ISO/IEC 42001 Annex A titles are taken from public listings, and NIST AI 600-1 is cited at document level where necessary.

11 Related Work

Places RQ1–RQ7 in relation to prior work. 

Prompt-injection benchmarks for agents

Greshake et al. introduced indirect prompt injection against LLM-integrated applications (Greshake et al. 2023), and Liu et al. formalised the attacks and defences (Liu et al. 2024). InjecAgent (Zhan et al. 2024) and AgentDojo (Debenedetti et al. 2024) benchmark tool-using agents against injected tool outputs; Agent Security Bench (Zhang et al. 2025) broadens the set of attacks and defences; ToolEmu (Ruan et al. 2024) emulates tools to surface risky agent behaviour; and RedCode (Guo et al. 2024) benchmarks risky code execution by code agents. These works fix the agent framework and vary the model and the attack; our bench fixes the model and the payload and varies real harness binaries and their modes.

Design-level defences

CaMeL (Debenedetti et al. 2025) and the design patterns of Beurer-Kellner et al. (Beurer-Kellner et al. 2025) constrain what an agent may do once it has read untrusted data. System-level isolation (IsolateGPT (Wu et al. 2025)) and programmable privilege control (Progent (Shi et al. 2025)) move enforcement into the runtime, much as our rubric asks of harnesses. Willison’s “lethal trifecta” (Willison 2025) explains why egress (D03) and secret handling (D07) matter even when injection cannot be prevented.

MCP security

Hou et al. survey the MCP landscape and its threats (Hou et al. 2025); Radosevich and Halloran show that MCP-connected models can be driven to harmful actions (Radosevich and Halloran 2025); Hasan et al. study the security and maintainability of MCP servers at scale (Hasan et al. 2025); and MCPTox benchmarks tool poisoning on real servers (Wang et al. 2026). Invariant Labs described tool-poisoning attacks and the GitHub MCP data leak (Beurer-Kellner and Fischer 2025; Milanta and Beurer-Kellner 2025), and Trail of Bits showed that a server can attack before any tool is invoked (Trail of Bits 2025). We add the harness side: how servers are declared, pinned and approved, and how often risky declarations occur in public repositories.

Practitioner research on coding agents

Much of what is known comes from practitioners: Pillar Security’s rules-file backdoor (Karliner 2025); Tracebit’s Gemini CLI allowlist bypass (Cox 2025); Johann Rehberger’s Embrace The Red series, including the Copilot auto-approve RCE, the Month of AI Bugs and the Antigravity analysis (Rehberger 2025a, 2025b, 2025c); Trail of Bits on prompt injection leading to RCE in agents (Vandevanter 2025); and Check Point Research and Cymulate on Cursor, Codex CLI and Claude Code (Charikov, Zaikin, and Vanunu 2025; Mill and Vanunu 2025; Donenfeld and Vanunu 2026; Beber 2025). Our survey normalises these reports under one taxonomy, root-cause scheme and counting rule.

CI and supply chain

Studies of GitHub workflow security (Koishybayev et al. 2022; Muralee et al. 2023) underpin our CI rules and the E2 workflow stratum, and taxonomies of open-source supply-chain attacks (Ohm et al. 2020) frame E4.

Frameworks

The OWASP Top 10 for LLM Applications (OWASP GenAI Security Project 2024), the OWASP Agentic AI threats and mitigations guide (OWASP GenAI Security Project 2025a), the OWASP Top 10 for Agentic Applications (OWASP GenAI Security Project 2025b) and the OWASP MCP Top 10 (OWASP Foundation n.d.) catalogue risks at the application level, and MITRE ATLAS (MITRE n.d.) catalogues adversary techniques. We map our taxonomy onto all of them and add a harness-specific, checkable control layer.

The author’s prior tooling

harnesswatch reuses parsing ideas from the author’s upfront, offers rulewarden (deterministic enforcement of instruction-file rules) as an integration point, and adopts airmkit’s control vocabulary for the MAS mapping.

12 Limitations and Threats to Validity

Bounds how far the answers to RQ1–RQ7 generalise. 

Version drift

Scores, probes and bench cells refer to the versions listed in VERSIONS.md, accessed on 15 October 2026. Harnesses release weekly, and several were renamed or discontinued during the study; the matrix should be re-run rather than cited as current.

Models and compute

No vendor API keys or logins were used, so the bench runs a single small, local, open-weight model through documented custom-endpoint features and says nothing about any vendor’s flagship model. The GPU is shared with the owner’s other work; a planned second model family was cancelled to release it, so the goal of at least two models per harness is not met, and the scenario set and runs per cell were reduced, leaving wide intervals. So small a model also limits how often any injection is acted upon at all. Claude Code was routed to a non-Claude model, which its vendor does not support (Section 3).

Headless only

Probes and bench runs are headless. Interactive trust dialogs, IDE extensions and GUI-only products (the Antigravity IDE, for instance) were not tested. A harness whose interactive mode is safer is under-credited, which is the conservative error for CI use.

Benign-canary ceiling

The payloads are inert and plainly worded; disguised or adaptive attacks may succeed where these fail, so the bench rates are lower bounds.

Black-box limits

Without TLS inspection we cannot say what an allowed host receives, and login-gated harnesses were observed only before login.

Rating and classification

Both matrix raters worked from one shared fact base, so $\kappa$ measures how the rubric was applied, not the reliability of the evidence, and it excludes the N/A-versus-score cells (all 84 disagreements, those included, are published). Incident classes and root causes had one LLM rater and no agreement measurement, and RC1 is the residual class of an ordered procedure. Several scores turn on rubric interpretation, notably of classifier-gated defaults.

Selection

The incident survey admits only publicly disclosed, harness-level advisories, most of them with CVE or GHSA identifiers; model-level failures are rarely tracked in this way, and agent-assisted search favours recent, well-indexed advisories. The RQ3 shares therefore describe disclosed harness defects, not overall risk.

Corpora

E6 is synthetic and labelled by the author; E2’s code-search sample favours popular, recently indexed repositories; E5 has one session per condition.

13 Conclusion

Summarises the answers to RQ1–RQ7. 

The harness is where an agent’s capabilities become effects. Among the publicly disclosed harness-level advisories of 2025–2026, flawed approval checks and configuration honoured before consent account for 48%, and pure prompt injection, a residual class, for 5%. Across 58 harnesses and runtimes, controls exist but are seldom safe by default: sandboxing and egress are the weakest dimensions, headless modes frequently bypass trust gates, and in our sample of public repositories unpinned MCP launches and write-scoped agent CI jobs recur. For closed harnesses, 22 of the 28 testable claims matched, but most documented claims could not be tested headless, and documented features such as instruction-file imports can carry secrets into transcripts without any model action.

The remedies are engineering: trust gates that cover every executable class of configuration and apply headless, permission modes locked by policy, default sandboxes and egress allowlists. We release harnesswatch, the rubric, the evidence and the results so that the study can be repeated as the field moves.

14 Reproducibility

The repository contains the toolkit, rule pack, harness profiles, evidence ledger, matrix evidence, results JSON and this paper. Three entry points rebuild everything.

• make all runs the linter, type checker and tests with coverage, builds the documentation, regenerates every results-derived table, figure and number offline from committed data (make experiments, which ends with experiments/paper_numbers.py writing paper/numbers.tex), and builds this paper with paper/build.sh.

• make bench builds the pinned harness images (make bench-images), runs or resumes the canary grid against a local Ollama container, and re-runs the E3 analysis. It requires Docker and a GPU-backed Ollama container on the bench network; no vendor credentials are used.

• paper/build.sh uses a pinned Tectonic release, downloaded with a SHA-256 check, and also writes the Markdown mirror paper/paper.md and the web version in docs/paper/.

Pinned versions: harness versions are listed in VERSIONS.md; bench images are tagged with the harness version (Table 9); the model server is Ollama 0.40.4, pinned by image digest; and Python dependencies are locked in uv.lock. Every reported number in the prose is a macro generated from experiments/results/*.json; design constants (versions, caps, mode bits, and the counts of classes and controls) are typed by hand.

15 Review and Corrections

Following an internal hostile review we made these changes, all reflected in the results files and logged in the assumptions log: (1) deterministic incident de-duplication, which removed one double-counted incident, with the excluded out-of-period row now shown; (2) “patch latency” recomputed from dates and reframed as fix date relative to advisory publication, with undated fixes reported; (3) the E3 secret metric split into secret read and secret in state once the lenient metric was found to count harness persistence (Codex environment snapshots, the Claude Code @path import); (4) E2 prevalence reported per stratum rather than pooled; (5) matrix evidence labels aligned with the ledger, so that source reads are ^[D], not ^[T]; (6) one Grok Build fact obtained by listing strings in the shipped binary withdrawn as outside our metadata-only rule; (7) claims about RQ3 shares, mode safety, claim coverage and rater independence reworded.

16 Rule Pack

Table 7 lists every rule; each YAML file adds harness-specific spellings, remediation, references and fixtures.

17 Canary-Bench Scenario Catalogue

Table 9 lists the scenarios and adapters. Every injected block uses the same versioned payload template; only its location changes. The hidden_unicode scenario is implemented but excluded from the reported grid, because its zero-width wrapping reaches the model as plain ASCII and so measures nothing beyond instructions. Aider loads no instruction file, so its instructions cells are N/A.

18 Scorecard Rubric

The fifteen core dimensions, the 0–3 scale and the scoring rules are set out in Section 5; a proposed D16 (local service and control-plane exposure) is excluded from all means until it has sufficient evidence. Each dimension poses a single question, for example D01, “does the harness ask before risky tool actions, and can the approve-everything path be restricted?”, and D04, “is repository-supplied configuration ignored until the user consents?”. The criteria are written so that they can be decided from labelled facts: score 3 always requires a safe default and central enforceability (for D02, filesystem and network confined by default, not weakenable by the agent or by repository files, and enforceable by an administrator). The full criteria are in matrix/rubric.md (summary table: paper/tables/rubric.md), the scores with means and minima in paper/tables/scorecard.tex, and every cell’s evidence and rationale in matrix/.