Benchmarks · Chanto docs

The benchmark suite lives in its own repository, chanto-sh/benchmarks: coding tasks in three tiers, each with an automated check: core (small exercises in several languages, plus trap tasks), medium (hand-written bug fixes, a feature, a cross-file refactor, test writing checked with mutation testing, codebase Q&A, a CSV CLI) and hard (real GitHub issues from SWE-bench Lite). Clone it next to this repository and run it against several agents on the same local model, or on a hosted model through the benchmark proxy:

cargo build --release                    # in this repository
git clone https://github.com/chanto-sh/benchmarks.git ../benchmarks
cd ../benchmarks
python3 run.py --agents chanto,opencode,claude --model qwen3.8-bench
./proxy.sh deepseek/deepseek-flash       # in another terminal
python3 run.py --api http://localhost:4000 --agents chanto,opencode,claude,codex --tier hard

--tier picks tiers (default all), --tasks narrows by name, and --estimate prints the projected cost and time without running anything.

See RESULTS.md for the latest results, per-task numbers, methodology and caveats.

To measure the agent’s own CPU and memory overhead with an instant mock model, run the published harness from the benchmark repository:

python3 reproduce.py --agents chanto --no-install --chanto ../chanto/target/release/chanto

The default workload is 40 tool turns reading 40 KB files, repeated 10 times. It reports median process-tree CPU time per model request and median peak process-tree resident memory sampled every 60 ms; the final answer is also a model request. Results record versions, machine details and every run. Component microbenchmarks are separate measurements and do not establish a whole-agent CPU or memory improvement.

Benchmark variants turn on the experimental optimizations below: chanto+concise, chanto+quiet, chanto+provenance, chanto+all, or any combination joined with +. They run the same command as chanto with CHANTO_OPTIMIZE set, and each result and the run's metadata record which switches were on. --chanto-effort LEVEL passes --effort LEVEL to every Chanto agent in the run.