EMPIRICAL EVIDENCE //
Repeated-run evaluation matrix.
Across four 25-rollout experiments, Sanre evaluated autonomous coding agents on the same 5-task benchmark suite, producing 100 total rollouts (Experiments 004–007).
AGENT EVALUATION SELECTORN = 5 ROLLOUTS / TASK
Selected Agent System
Gemini 3.7 Flash
PASS RATIO20/25
TIMEOUTS0
AVG LATENCY69.9s
Task Performance Distribution (5 Independent Runs Per Task)
T1
5/5
Repo Understanding
T2
5/5
Multi-Bug Debugging
T3
5/5
Regression Safe
T4
5/5
Rate Limiter Integration
T5
0/5
Ambiguous HTTP Debug
Full Empirical Dataset Matrix
| Model / Agent | Experiment | Pass Rate | Timeouts | Avg Duration |
|---|---|---|---|---|
| Gemini 3.7 Flash | EXP 007 | 80.0% (20/25) | 0 Timeouts | 69.9s |
| Gemini 3.1 Pro | EXP 004 | 64.0% (16/25) | 9 Timeouts | 140.3s |
| Gemini 3.8 Flash | EXP 006 | 52.0% (13/25) | 11 Timeouts | 186.8s |
| Gemini 3.1 Flash-Lite | EXP 005 | 44.0% (11/25 (75% valid)) | 0 Timeouts · 9 Violations | 86.8s |
Methodology & Constraints
· 300s timeout ceiling strictly enforced per rollout.
· Zero state leakage: every rollout runs in a fresh container instance from a clean git commit.
· Independent oracle evaluation: hidden assertions verify completion out-of-band.