Skip to content

Animus Stack — 10/10 Adversarial Review (2026-06-03)

Method: 7 parallel reviewers adversarially probed each scorecard dimension against the post-Session-6 code; every high/critical finding independently re-verified; synthesized below. 19 agents, ~1.3M tokens. Reviewers ran targeted tests (not the full suite).

Headline: NOT 10/10. Real, well-tested cores with bypassable edges. 7 confirmed HIGH issues (several are bugs introduced in Sessions 1-6 whose tests passed only because they mocked/centered the buggy path).

Verdict

The stack is not 10/10 and is not fully functional in its deployed state. The headline claims survive in their narrow forms but fail at the boundaries that matter most: cost enforcement is real for raw tokens but bypassable on the effective-token axis, content-aware egress DLP is wired on four of six cloud providers and blind to tool-block payloads, the integrity gate is in a guaranteed-boot-fail state on the live machine right now, the code-execution metric scores crashes as perfect passes, and the one "real experiment runner" that grounds auto-promotion silently drops the score it was built to surface. Several of these are masked by green test suites that mock or center exactly the path where the bug lives, so a passing suite is not evidence the claim holds. The strongest area is D6 (StabilityScorer), which the roadmap itself honestly scores 7/10, not 10. Net: real, well-tested cores with bypassable edges; multiple confirmed high-severity issues must close before any 10/10 claim is defensible.

Dimension Claimed Review verdict Note
D1 Cost discipline 10/10 Not 10/10 Raw-token allocate/release is atomic and well-tested. ET ceiling not enforced at admission (medium); single-source pricing claim false, dead method (low).
D3 Egress + DLP 10/10 Not 10/10 4/6 providers enforce. Bedrock/Vertex unenforced, Azure streaming bypass, tool-block content unscanned (all high).
D3 Integrity baseline 10/10 Not 10/10 Checker code mostly sound, but deployed baseline stale (daemon won't boot) and forge-side enforcement files untracked (both high).
D4 Judge + sandbox 10/10 Not 10/10 Judge raise-paths and RLIMIT bounding genuinely hold. CodeExecution scores crashes 1.0 (high); regex precedence no-op (medium).
D4 Power advisor + calibration 10/10 Not 10/10 Judge calibration solid. Equivalence uses CI half-width not bounds, mislabels off-center intervals (medium).
D5 Evolution loop + auto-promote 10/10 Not 10/10 Staging/approval separation and guards hold. Experiment runner reads nonexistent avg_score, drops the score (high).
D6 StabilityScorer protocol 7/10 (roadmap) Verified with caveats Narrow B7 claim true and byte-identical. Falsy-scorer drop, duplicated formula, missed CANM.md flip (all low/medium). Area does not assert 10/10.

Confirmed issues (prioritized)

Critical/High (independently confirmed)

  1. HIGH — Bedrock and Vertex have no egress enforcement at all. Both are registered, reachable cloud providers (manager.py:46-47) that send prompt/system_prompt/messages straight to AWS/GCP with no DLP scan, no tier check, not even the ANIMUS_OFFLINE kill-switch. bedrock_provider.py complete() calls invoke_model (~:264) directly; vertex_provider.py complete() calls model.generate_content (~:177) directly. A CONFIDENTIAL/SECRET request, or a credential-bearing PUBLIC body, egresses unconditionally. Fix: add self._check_request_egress(request) plus the ANIMUS_OFFLINE init gate to both providers' complete/complete_async/complete_stream/complete_stream_async. (Listed critical in the raw issue set; lead reviewer verdict downgraded to high because exploitation requires these providers to be configured and explicitly selected, never auto-defaulted.)

  2. HIGH — Azure streaming path bypasses the egress check. azure_openai_provider.py complete()/complete_async() call _check_request_egress at :168/:245, but the overridden complete_stream (:305) and complete_stream_async (:344) call self._client.chat.completions.create(stream=True) at :329/:368 with no egress check. Azure is the only gated provider that overrides streaming (OpenAI/Anthropic/OpenRouter inherit the base streaming impl that routes through complete()), so it is the only one that drops the check. supports_streaming returns True, so the path is reachable. Fix: call _check_request_egress(request) at the top of both streaming overrides.

  3. HIGH — scannable_text() ignores tool definitions, tool_result blocks, and tool_use args. providers/base.py:109-126 collects only prompt, system_prompt, top-level string content, and blocks keyed "text". A secret placed in request.tools, in a tool_result block (keyed "content"), or in a tool_use "input" dict yields scannable_text()=='hi' (the prompt only) — verified empirically. These fields reach the wire verbatim (anthropic_provider.py:235 sets kwargs['tools']; openrouter _build_messages:339-348 forwards messages). The DLP fails open on exactly the payload shape (tool I/O in agentic loops) most likely to carry live credentials. The docstring claim "All outbound text in this request" is false. Fix: extend scannable_text() to recurse into tool definitions, tool_result content (incl. nested text blocks), and tool_use input args.

  4. HIGH — Deployed integrity baseline is stale; the live daemon refuses to boot. ~/.config/animus/integrity-baseline.json (generated 2026-05-27T10:56) tracks only 4 keys; checker.py:36-59 now tracks 10 plus the self-hash and module:* keys. Live verify_or_raise(default_baseline_dir()) raises IntegrityMismatchError with 9 drifts (6 new-key "not in baseline" plus real content drift on redaction.py/egress.py/mcp_server.py). Running the exact systemd ExecStart command produces exit status 2 right now. The only escape is ANIMUS_INTEGRITY_OVERRIDE=1, which disables the gate entirely (checker.py:212-217). The currently-running PID predates the drift and is not re-checked, so the failure is latent until the next restart but reproducible on demand. The gate has provided zero tampering coverage since May 27. Fix: regenerate the baseline from an attested-clean tree and wire regeneration into the deploy step so the tracked set and on-disk manifest cannot diverge.

  5. HIGH — Forge-side egress/DLP enforcement files are untracked; gate bypassable by patching them. The baseline tracks the policy primitive (animus_types.egress/.secrets) but NOT the call sites that invoke it: animus_forge.providers.base (assert_egress_allowed, base.py:129-153), the per-provider _check_egress methods, animus_forge.network.egress, and providers/router.py (TierRouter). A daemon-write attacker (the exact in-scope threat per integrity/__init__.py:3-6) patches base.py:assert_egress_allowed to return or a provider _check_egress to pass, dead-ending all tier+DLP enforcement while the baseline stays green. Even the one tracked forge provider (openrouter) delegates enforcement to the untracked base.py. Fix: add the forge enforcement call-site modules to the tracked set.

  6. HIGH — CodeExecutionMetric scores 1.0 for crashing/memory-bombing code when expected is None. metrics.py:329-334 returns 1.0 with no check on subprocess success; _execute_python (:379-426) runs subprocess.run(check=False) and never inspects result.returncode (returncode appears nowhere in the file). Verified: raise RuntimeError, 1/0, sys.exit(7), and a bytearray(2_000_000_000) memory bomb all score 1.0; even the caught "TIMEOUT" sentinel scores 1.0 on the expected=None branch. The RLIMITs correctly bound the host but never feed the score, so in a "does this generated code run?" eval every crashing snippet passes, inflating scores and masking regressions. Fix: capture result.returncode and return 0.0 on non-zero exit; treat the TIMEOUT sentinel as 0.0 on the expected=None branch.

  7. HIGH — eval_experiment_runner reads nonexistent avg_score on the real SuiteResult; score silently dropped. evolution_loop.py:551 reads getattr(result, "avg_score", None), but SuiteResult's field is total_score (runner.py:46, set at :329); no avg_score alias exists. Verified against a real SuiteResult(total_score=0.873): the report string reads avg_score=n/a and the measured 0.873 never reaches the LLM evaluator that emits the keep/discard verdict. The test passes only because it feeds a MagicMock with a fabricated avg_score (test_evolution_loop.py:565-579) — a passing test that does not test the real type. This is the auto-promotion decision path; the one quantitative "better" signal the runner exists to surface is discarded. Fix: read total_score (with avg_score fallback) and rebuild the test on a real SuiteResult.

Notable mediums

  • D1 — Effective-Tokens ceiling bypassable at admission. _can_allocate_unlocked (manager.py:341-373) and allocate() (:388-407) consult only raw tokens/reservations, never _total_effective/effective_ceiling. ET is enforced post-hoc via status==EXCEEDED checked before the next step, so a single output-heavy/opus step can overshoot the ET ceiling 100x (verified: 50k opus output tokens = 1,000,000 ET vs a 10,000 ceiling), and a terminal expensive step is never gated. Downgraded from high because the post-hoc design is documented and the raw-token axis has the identical single-step-overshoot property; both rely on the optional per_step_limit. Fix: fold an ET estimate for the pending step into _can_allocate_unlocked.

  • D4 — Regex precedence silently no-ops configured patterns. metrics.py:117/:152 compute self._pattern or str(expected) if expected else None, which parses as (self._pattern or str(expected)) if expected else None; a falsy expected nulls a configured self._pattern. Verified: RegexAbsenceMetric(pattern='SECRET').score('my SECRET key', None, case) returns 1.0 despite the forbidden token being present. The only production suite (benchgoblins-ask.yaml) gives every case a truthy expected, so blast radius is zero today, but the defect arms itself the moment a future case omits expected for a global invariant. Tests pin the bug as intended behavior. Fix: pattern = self._pattern if self._pattern is not None else (str(expected) if expected else None), and correct the two pinning tests.

  • D4 — Equivalence vs underpowered uses CI half-width, not bounds. compare.py:88-100 decides equivalence by ci_halfwidth > MEANINGFUL_EFFECT rather than both bounds inside ±0.05. Verified: CI [0.0, +0.099] is labeled "effectively equivalent within ±0.05" though the upper bound is nearly double the meaningful effect; the printed "within ±0.05" is factually false. Downgraded from high because auto_promote gates strictly on comparison.significant (correctly bounds-derived), so no bad auto-action results; impact is operator misinformation in an advisory. Fix: not (lo > -MEANINGFUL and hi < MEANINGFUL) and derive the displayed wording from actual bounds.

  • D5 — Dry-run not stamped on the structured verdict. The dry-run flag lives on the loop object and in the experiment_summary text, but the IterationRecord and audit JSONL outcome field carry a clean keep/discard with no is_dry_run field (evolution_loop.py:342-349, :479-489). A consumer reading outcome sees an unmarked keep for a dry run unless it string-parses the summary. Fix: add is_dry_run to the record and audit entry.

(Lower-severity items — the dead BudgetManager.estimate_cost / stale CostTracker pricing table whose high-severity "governs live spend" framing was refuted to low, the in-process self-hash defeat, .local/missing-credential-pattern egress gaps, the falsy-scorer drop, duplicated stability formula, and the cosmetic CLI/log mislabels — are real but not gating for a 10/10 claim. Full list in the corpus all_issues.)

What actually holds up

These claims survived adversarial probing and are the credibility floor of the stack:

  • D1 raw-token reservation is genuinely atomic and thread-safe. The lock wraps check-and-reserve plus the pending increment (manager.py:401-407); the 20-thread barrier test asserts exactly 10/20 granted with no overspend or leak. The parallel executor reserves once per sub-step and releases in a finally, so reservations do not leak on the exception path or re-reserve per retry. This is a real, well-tested fix, not a stub.

  • D3 pattern source is unified with no drift. Core redaction imports CREDENTIAL_PATTERNS from animus_types.secrets (redaction.py:25), so egress DLP and redaction share one canonical set. The four wired providers fail closed before the cloud client is invoked and never echo the secret value (only pattern names). OpenRouter correctly adds the strictest PUBLIC-only gate. Loopback userinfo-spoofing (localhost@evil.com) is correctly resolved to evil.com and denied.

  • D3 integrity checker design tracks the real cross-package primitive. It baselines module:animus_types.egress + module:animus_types.secrets via importlib (not just the core shim), self-hashes the checker, and verify_or_raise genuinely raises and converts to sys.exit — a hard boot gate, not log-only. The 14-test suite is honest about the in-tree code. (The gap is coverage and the stale deployment, not the verification mechanism.)

  • D4 judge failure handling is correct end-to-end. All three judge paths (no provider, provider exception, unparseable) raise JudgeError; the evaluator converts to an ERROR result and the classifier buckets it provider_error — no silent 0.5, verified by test and by trace. RLIMIT bounding is real and kernel-enforced (512MB RLIMIT_AS triggers MemoryError; RLIMIT_CPU kills infinite loops at 10s), and the subprocess hardening (sys.executable, -I, scrubbed env, isolated tempdir) is sound. The sandbox docstring is honest that it is not a full syscall sandbox.

  • D4 judge calibration is statistically sound. A raising judge is counted as an error and excluded from agreement; calibrated is gated on errors == 0; the Pearson guards (n<2, zero-variance) are correct; bootstrap_ci_delta handles empty inputs, is deterministic under seed, and uses strict lo>0/hi<0 for significance.

  • D5 staging and approval separation holds. propose_patch only stages {workflow_id}.pending.yaml and never applies; approve is a distinct human-invoked CLI command with no auto-chain; auto_promote_on_improvement has zero production callers and correctly rejects not-significant, regression, and underpowered cases; _validate_patch enforces version increment, snake_case step types, a code-injection blocklist, and token budget. Dry-run emits a loud warning and surfaces via is_dry_run/status().

  • D6 B7 protocol extraction is provably byte-identical. The git diff (631c840) shows DefaultStabilityScorer.score() is a line-for-line copy of the old inline logic; all 13 production callers invoke compute_stability() with no args and hit the unchanged default path; tests assert exact numeric equality, custom-scorer override, and protocol runtime-checkability. The Session-6 except Exception: counts the failure as an error per contract, not a swallow. The roadmap honestly scores this 7/10.

Must fix before claiming 10/10:

  1. Regenerate the deployed integrity baseline from an attested-clean tree and wire regeneration into the deploy step (D3 issue 4) — the gate is non-functional in production right now.
  2. Add _check_request_egress + ANIMUS_OFFLINE init gate to Bedrock and Vertex on all four entrypoints (D3 issue 1).
  3. Add the egress check to Azure complete_stream/complete_stream_async (D3 issue 2).
  4. Extend scannable_text() to cover tool definitions, tool_result content, and tool_use args (D3 issue 3).
  5. Add the forge-side enforcement call-site modules (providers/base, per-provider _check_egress, network.egress, router) to the integrity tracked set (D3 issue 5).
  6. Inspect result.returncode in CodeExecutionMetric and score 0.0 on non-zero exit / TIMEOUT (D4 issue 6).
  7. Read total_score in eval_experiment_runner and rebuild its test on a real SuiteResult (D5 issue 7).

Should fix:

  1. Fold an ET estimate into _can_allocate_unlocked so the effective-token ceiling is enforced at admission (D1).
  2. Fix the regex operator-precedence no-op and correct the two pinning tests (D4).
  3. Replace the half-width equivalence test with bounds-based logic and derive the displayed wording from actual CI bounds (D4).
  4. Stamp is_dry_run onto the IterationRecord and audit JSONL entry (D5).
  5. Resolve the dead BudgetManager.estimate_cost / stale CostTracker pricing-table divergence and remove the false "single source of truth" docstring (D1).
  6. Close the lower-severity integrity gaps (in-process self-hash defeat via external pre-exec check, .local/missing credential patterns) and the D6 falsy-scorer drop and duplicated stability formula.