Security Evals & Benchmarks
The full 534-case benchmark report (309 attacks, 225 benign) — 100% detection, 0 false positives, 0 open evasion gaps. Download the raw report (JSON) →
These figures are measured against a curated, continuously-maintained corpus — a regression bar, not a guarantee against every possible or novel adversarial input. We disclose the exact cases and add new ones as evasions are found.
Reproduce it yourself. The labeled corpus and harness are public at github.com/omertt27/mergen-benchmark — browse every case, or re-run the measurement. The published npm package also ships the dataset and runner, running against the same built gate the package enforces with:
npm install mergen-server
node node_modules/mergen-server/scripts/benchmark/run.mjs
# → benchmark-report.json + scorecard, written inside the package directoryOur release pipeline installs the packed tarball into a clean directory, runs this exact command, and refuses to publish unless the result matches the numbers on this page.
534-Case Regression Harness
We do not claim 100% security. Mergen is evaluated against a labeled adversarial dataset (309 attacks, 225 benign) run through the exported gate decision function the live interceptor itself calls — not a test-only mirror. Our results disclose exact failure categories, open evasion gaps, and known false positives so you can calibrate policy thresholds according to your risk tolerance.
Breakdown covers the 309 attack fixtures. The 225 benign fixtures are scored separately for false positives (0 flagged) under a strict definition: any verdict other than PASS on a benign command — including a conservative HOLD — counts as a false positive. Numbers are imported from the measured run of 2026-07-27 (server/benchmark-report.json); regenerate with npm run benchmark, publish with npm run sync-benchmark. Detection latency, measured in-process around the gate decision function: p50 0.871ms · p95 1.513ms · p99 2.671ms over 2,670 evaluations.
Disclosed Vulnerabilities & Failure Logs
Below are actual cases where Mergen failed to block, blocked incorrectly, or conservatively holds a benign command. We document and test these continuously to narrow detection bounds. Cases marked CLOSED were open gaps that are now caught.
Zero-width space injection inside "terraform" command
Fixed: U+200B (zero-width space) is now in INVISIBLE_SIGNAL_RE. The obfuscation pre-check fires before the policy engine, routing this to HOLD for human review. Previously missed; now caught.
Hex escape sequence for 'rm' — literal \xHH sequences now detected
Fixed: Bare \xHH sequences outside $'...' ANSI-C quoting are now detected by detectObfuscationSignal as "hex-encoded characters" and routed to HOLD before the policy engine runs. Previously passed through; now caught as suspicious.
Natural-language destructive framing — now caught via semantic expansion
Fixed: SEMANTIC_EQUIVALENTS now maps /\bdrop\b...\btable\b/ to a canonical nl-drop-table marker. The hold_nl_destructive_intent rule fires on this marker, routing the instruction to HOLD for human review. No LLM in the critical path — purely deterministic pattern expansion.
Deleting a single completed pod is held for one-click approval — by design
Tradeoff: Removing one finished worker pod is routine cleanup, but the local policy does not run an LLM to parse intent, so it holds any targeted kubectl deleteunder the same ruleset that blocks namespace and deployment deletion — a human resumes it with one click. This is a deliberate conservative HOLD. Under the strict scoring above a benign HOLD would count as a false positive, so the corpus labels this case as an expected HOLD on the attack side — it is disclosed here instead of being quietly scored as a pass. Tighten the rule with a scoped allow if this friction outweighs the safety margin for your team.
Developer friction: actively tracked and fixed
Every false positive the gate has ever produced is listed here. The table grows when a new FP is found — and shrinks to zero every time we ship a fix. A non-empty table means the feedback loop is working.
The ( character in the grep pattern matched the word-boundary separator for the destroy verb, causing a source-code search to trigger the destructive command rule.
terraform destroy prod → still BLOCK ✓
The word-boundary regex fired at the -/d boundary in -destroy, so a Terraform flag value matched the standalone destroy command pattern.
terraform destroy prod → still BLOCK ✓
The s3 rm rule held every object removal, including a single build artifact. The rule is now scoped to recursive and wildcard operations, where the blast radius actually lives.
aws s3 rm s3://prod-analytics/ --recursive → still BLOCK ✓
The blast-radius escalation scanned raw text, so SQL quoted inside an echo argument (and "truncate" inside a git commit -m message) fired the data-destructive rule. It now scans the shell-AST haystack the policy engine itself matches against: inert argument text is data, not action — while a command substitution hidden inside that text is still extracted and evaluated as an executed command.
DROP TABLE users (executed, not quoted) → still stopped ✓
All fixed cases are locked as PASS assertions in the eval corpus and test suite — they cannot silently regress.