The evaluation protocol.
The model is only one part of an agent. Sanre tests the complete operational system: model + harness + tools + environment + state + permissions.
Before any evaluation begins, we define the exact boundary conditions under which the agent will operate.
- ·Deterministic Agent Definition: Model weights, temperature, prompt templates, and tool interfaces.
- ·Task Suite Scoping: Defining ground truth tasks, starting repository state snapshots, and expected artifacts.
- ·Safety & Resource Boundaries: Establishing filesystem access policies, network proxy constraints, and timeout limits (e.g. 300s ceiling).
Every rollout executes inside a hermetic, ephemeral Linux container to prevent state carryover between runs.
- ·Zero State Carryover: Container filesystems are provisioned from a clean git commit snapshot and discarded immediately after each run.
- ·Network & Process Barriers: Restricting unauthorized external calls to prevent side-effect pollution or data contamination.
- ·Clean Initial Snapshot: Pre-run environment validation confirms zero residual lockfiles or cache artifacts.
We run 5 identical rollouts per task to expose stochastic variance that single-run evaluations fail to detect.
- ·Identical Initial State: 5 independent rollouts executed under identical starting conditions.
- ·Trajectory Divergence Analysis: Tracking where agent reasoning branches, file edits, and tool calls diverge.
- ·Loop Trapping: Detecting stochastic failure traps where agents repeat failing commands until hitting timeouts.
Completion claims are never trusted directly from the agent. Independent oracles verify actual system state.
- ·Hidden Test Assertions: Out-of-band evaluation tests that are strictly unreachable by the agent during execution.
- ·Filesystem AST Diffs: Checking code syntax correctness, logical diff validity, and detecting unintended side-effect modifications.
- ·Integrity Audits: Ensuring tool safety rules, permission barriers, and filesystem constraints were respected.
Findings are organized into structured dimensional scorecards, root-cause failure catalogs, and raw trace bundles.
- ·Multi-Dimensional Scorecard: Disaggregated metrics for capability, integrity, execution latency, and consistency.
- ·Failure Taxonomy: Categorization into recursive loop traps, context window degradation, tool argument errors, or syntax faults.
- ·Raw Evidence Bundle: Structured JSON execution traces, timestamped tool logs, and git diffs for internal team replay.
Need an evaluation adapted to your agent's harness?
We customize task suites and invariants for your production environment.