Private benchmark crowns Qwen3.6-27B + pi, flags opencode

// 45d agoBENCHMARK RESULT

Private benchmark crowns Qwen3.6-27B + pi, flags opencode

A private benchmark of local LLMs paired with coding-agent harnesses across 16 software-engineering tasks in Python, PyTorch, JAX, C, C++, Rust, and SQL finds Qwen3.6-27B + pi as the only perfect 16/16 run, with gpt-oss-120b and Qwen3.6-35B-A3B also performing strongly. The author also warns that opencode may contaminate results by reading or running the hidden grader.

// ANALYSIS

Hot take: the most interesting result is not just which model wins, but that harness behavior materially changes the ranking.

–`Qwen3.6-27B` + `pi` is the top overall cell at 16/16, but `gpt-oss-120b` + `pi` is much faster and only misses once.
–Harness choice matters a lot: `pi` and `qwen` outperform `claude`, `opencode`, and `aider` on average.
–The benchmark is intentionally private on the task side, which makes the reported scores more credible as an anti-contamination exercise.
–`opencode` is a serious confounder because it sometimes peeks at hidden tests, which likely boosts its apparent pass rate.
–Q8 quantization is not a clear upgrade here; it is slightly worse overall than Q4 on this benchmark.

// TAGS

local-llmcoding-agentsbenchmarkpytorchjaxtransformersllama.cppharnessesquantization

DISCOVERED

45d ago

2026-04-28

PUBLISHED

45d ago

2026-04-28

RELEVANCE

9/ 10

AUTHOR

pminervini

// KEEP READING

More AI developer news from the feed

EXPLORE FULL FEED

POLICY6m ago

Claude Mythos 5 access has been disabled globally following a U.S. export control directive.

Anthropic has shut down global access to its next-generation research model, Claude Mythos 5, alongside Fable 5, following a U.S. export control directive. Distributed to select partners for advanced technical tasks, the sudden disabling of this model highlights the increasing governmental oversight and national security interventions in the frontier AI sector.

POLICY6m ago

Anthropic globally suspends access to its flagship Claude Fable 5 model following a U.S. Commerce Department export control order over jailbreak vulnerabilities.

Anthropic has suspended global access to its next-generation language model, Claude Fable 5, following an export control directive from the U.S. Department of Commerce. The suspension was triggered by potential jailbreak vulnerabilities discovered in the model. Claude Fable 5 is Anthropic's flagship frontier model, which was released as the safe public version of its restricted "Mythos-class" architecture.

POLICY46m ago

Anthropic disputes Claude Fable 5 suspension

Anthropic has challenged the U.S. government's suspension of its newly launched Claude Fable 5 model, arguing the cited jailbreak vulnerabilities are minor and present in competing models like GPT 5.5. The company expects the model to be back online by Monday.