BharatCode

Reference

Eval Suites

bharatcode eval is a task-suite benchmark harness. It runs built-in fixture tasks — real, small coding problems — against a scripted stub provider, entirely offline, and reports how the agent did: which tasks passed, how many steps each took, and how often it had to recover from a mistake. No API keys and no network access are required, so it is safe to wire straight into CI as a regression check on the agent loop itself.

What it measures

Each suite is a set of fixture tasks that exercise common editing patterns — a syntax error to fix, a missing function to stub, a failing test to make pass. For every task the harness spins up an agent with a scripted stub provider (no real model, no credentials, no network), runs the task to completion, and records:

  • Pass / fail — whether the task's success check was met.
  • Steps — how many agent steps the task took.
  • Recoveries — error tool results and repeated identical calls, a proxy for how often the agent had to dig itself out of a mistake.

Per-task results roll up into a suite report with the total passed, the pass percentage, the average step count, and the total recoveries.

Deterministic and offline by design

Because the provider is a deterministic stub rather than a live model, eval results are reproducible and free — the harness benchmarks the agent loop (tool wiring, recovery behavior), not a model's quality. It runs the same way on a laptop and in an air-gapped CI runner.

Running the suites

With no arguments, bharatcode eval runs every built-in suite and prints a human-readable report per suite:

terminalbash
bharatcode eval

A report looks like this:

bharatcode eval (illustrative)text
Suite: go-fix  Passed:       5 / 5  (100.0%)  Avg steps:    4.2  Recoveries:   1  Duration:     842ms   TASK              PASS   STEPS   RECOVERIES   REASON  syntax-error      PASS   3       0  missing-function  PASS   5       1  update-comment    PASS   2       0  add-return-value  PASS   6       0  fix-off-by-one    PASS   5       0

Numbers above are illustrative

Step counts, recoveries, and durations depend on the harness build; the shape of the report — the per-suite summary and the per-task table — is what to expect. The suite shipped today is go-fix; run with --list to see the current set.

Flags

  • --list — enumerate the available suites (name, task count, description) without running them.
  • --suite <name> — run a single suite by name instead of all of them.
  • --json — emit newline-delimited JSON, one object per suite, for CI ingestion.
  • --max-steps <n> — cap agent steps per task; 0 uses the harness default of 20.
terminalbash
bharatcode eval --list              # enumerate suites without runningbharatcode eval --suite go-fix     # run one suite by namebharatcode eval --json             # newline-delimited JSON for CIbharatcode eval --max-steps 30     # cap agent steps per task (0 = default 20)

Wiring it into CI

bharatcode eval exits non-zero when any task fails, so it behaves like any other test command in a pipeline — a regression in the agent loop turns the build red. Pair the exit code with --json to capture machine-readable results for a dashboard:

cibash
bharatcode eval --json > eval-results.jsonl

Each JSON line is a full suite report — suite name, timestamps, aggregate pass/fail counts, pass percentage, and the per-task array — ready to archive or feed into other tooling. Because the harness is fully offline, it composes cleanly with offline mode and air-gapped runners.

Related

For one-off agent runs from a script (as opposed to benchmarking the loop), see bharatcode run on the CLI Reference.