Benchmarking local coding agents

The same answer went from 17 to 97

In an August 26 coding evaluation, a model returned unfenced Python after being told to output only code. The extractor expected a Markdown code fence. When it found none, it searched the reasoning trace instead and executed an unfinished draft.

The recorded score was 17/100. Re-scoring the identical final-answer bytes through the corrected extractor produced 97/100. The model had not improved; the evaluation had finally executed the intended answer.

The clue was a mismatch: every functional criterion was zero while the text-based code-quality criterion still earned partial credit. I now treat that pattern as a reason to inspect the extracted artifact before making a claim about capability.

A good run is not a reliability estimate

An August 27 comparison used the same Prime agent harness and four hard tasks: a POSIX regex engine, replicated text, an asynchronous DAG orchestrator, and a streaming JSON parser. Three runs of one Qwen configuration produced the scores below.

Even this small table is more useful than displaying only the best result. It also needs its intervention record: two runaway fuzz subprocesses were interrupted across these runs; the agent adapted and completed those tasks. “No derailments” would not mean “no human assistance.”

The suite, the runners and the numbers are public

The task set, the direct-API and harness-assisted runners, and 82 result records are on GitHub as coding-agent-bench, together with the four harness failures described on this page and how the scoring was fixed. The table below is the same four-task, three-run protocol applied to the other local configurations I ran on the two-GPU workstation between August and September 2026. Each row is three complete runs; the interpretation column carries what the score alone would hide.

The five-task direct-API suite in the same repository is a different measurement and its scores are out of 500: Ornith-1.5-35B in bf16 scored 489, a Qwen3.8-27B MTPLX build 483, Claude Opus 4.6 483, and DeepSeek V4-Flash 459 to 481 depending on the serving stack. Those numbers should not be placed in the table below.

Do not let a harness limit become a model verdict

A timeout, output-length limit, malformed tool exchange, and incorrect implementation are different events. I keep inconclusive runs visible and separate from completed-task scores. They still matter operationally—someone waiting for an answer experienced an unfinished task—but they are not interchangeable evidence of reasoning failure.

Some older comparison tables assigned a low floor to interrupted or missing-artifact runs. I would not reuse those totals as a clean model ranking. The statuses, final files, and intervention notes need to travel with the score.

The serving setup is part of the experiment

I record precision or quantization, the serving engine, sampling settings, context and thinking budgets, tool parser, and whether speculative decoding is enabled. Changing any of these can change the experiment. A speed comparison also needs the same workload and hardware constraints.

For my own work, the best setup can depend on what else is running. A configuration that leaves room for rendering may be more useful that evening than the one with the best isolated coding result. Completed work and resource coexistence are product criteria, not just infrastructure details.

Abliteration and fine-tuning need separate tests

I work with fine-tuning, LoRAs, and abliteration as part of model adaptation. Changing refusal behavior is not the same as improving reasoning, tool use, or instruction following. A model that responds more often can still produce worse answers.

I therefore treat behavior and task performance as separate evaluation axes and compare adaptations with their base model under matched conditions. A label on a model upload is not evidence that an adaptation worked. Neither is a single successful demonstration.

This is also why I value open-weight models and inspectable tools: they let me investigate the cause of a change instead of treating every result as an opaque property of a product name.

Measured on Ornith-1.5-35B-A3B, a 35B mixture-of-experts model with 3B active parameters: per-layer refusal directions on all forty layers, with the token embedding left alone, took refusals from ten or eleven of twelve prompts to zero with no false refusals on the matched benign set, and the five-task coding suite stayed at 489/500, identical task for task. The same directions applied with the token-embedding edit from the original recipe also removed the refusals but dropped the suite to 400/500, with the recursive-descent parser collapsing from 98 to 13; reducing the scale did not rescue it. The only community “abliterated” upload of that model at the time turned out to be a no-op.

What belongs beside every result

My minimum useful evaluation record includes:

• The task, input set, final artifact, and exact grading procedure.
• The model and serving configuration, with the date of the run.
• Repeated results and completion statuses—not just the best score.
• Human interventions, skipped checks, timeouts, and resource constraints.
• The decision the evidence supports, and the claims it cannot support.