Benchmark task suite.

Real software engineering tasks designed to isolate stochastic failure modes, tool-loop trapping, and invariant violations across repeated rollouts.

Evaluation Task Invariants & Multi-Run Dispersion

TASK 01Repo Understanding & Dependency Retrieval

Code Retrieval

Target Invariants: Identify all 4 interacting configuration dependencies across a multi-package monorepo without context degradation.

Observed 5× Rollouts: Gemini 3.7 Flash: 5/5 · Pro 3.1: 5/5 · Flash 3.8: 5/5 · Flash-Lite 3.1: 1/1 (4 violations)

TASK 02Multi-Bug Root Cause Analysis & Debugging

Diagnostic Logic

Target Invariants: Fix 3 interacting async race conditions without breaking existing passing regression suites.

Observed 5× Rollouts: Gemini 3.7 Flash: 5/5 · Pro 3.1: 4/5 · Flash 3.8: 3/5 · Flash-Lite 3.1: 5/5

TASK 03Regression-Safe Code Modification

Refactoring Safety

Target Invariants: Refactor serialization cache keys while preserving legacy binary compatibility and AST invariants.

Observed 5× Rollouts: Gemini 3.7 Flash: 5/5 · Pro 3.1: 2/5 · Flash 3.8: 4/5 · Flash-Lite 3.1: 0/0 (5 violations)

TASK 04Distributed Rate Limiter Integration

System Architecture

Target Invariants: Implement thread-safe token bucket algorithm with Redis backend and mutex locking under concurrency.

Observed 5× Rollouts: Gemini 3.7 Flash: 5/5 · Pro 3.1: 3/5 · Flash 3.8: 1/5 · Flash-Lite 3.1: 5/5

TASK 05Ambiguous HTTP Trajectory Debugging

Edge Case Debugging

Target Invariants: Identify intermittent connection reset root cause in chunked transfer encoding stream.

Observed 5× Rollouts: Gemini 3.7 Flash: 0/5 · Pro 3.1: 2/5 · Flash 3.8: 0/5 · Flash-Lite 3.1: 0/5

We scope tasks reflecting your production environment.