TurboQuant restores Qwen3.6 speculative decoding

// 45d agoINFRASTRUCTURE

TurboQuant restores Qwen3.6 speculative decoding

A PR to TheTom's llama-cpp-turboquant fork syncs upstream llama.cpp speculative checkpointing so hybrid MoE+SSM models like Qwen3.6-35B-A3B can actually use speculative decoding. The fix targets a silent fallback where recurrent state rollback failed, disabling the speedup path without an obvious error.

// ANALYSIS

This is niche infrastructure work, but it matters because local inference performance often dies in exactly these edge-case compatibility gaps.

–Qwen3.6's hybrid recurrent layers need checkpoint-style state restore when draft tokens are rejected
–The PR pulls in upstream llama.cpp work, including save/restore support for recurrent speculative decoding
–Benchmarks in the PR show no meaningful regression on M2 Pro, with identical Wikitext-2 perplexity
–Draft/main tokenizer mismatch remains a practical speed limit, especially for Qwen3.5-to-Qwen3.6 pairings
–For regular llama.cpp users this is already handled; the value is specific to TurboQuant fork deployments

// TAGS

llama-cpp-turboquantqwen3.6inferenceopen-sourcellmbenchmark

DISCOVERED

45d ago

2026-04-22

PUBLISHED

45d ago

2026-04-21

RELEVANCE

8/ 10

AUTHOR

dangerousdotnet

// KEEP READING

More AI developer news from the feed

EXPLORE FULL FEED

UPDATE4h ago

Hermes Desktop adds Simplified Chinese support

Hermes Desktop, the cross-platform native application for running and managing Nous Research's open-source Hermes AI Agent, has released a complete localization update for Simplified Chinese. The update translates all user interface components including the chat window, sidebar, settings, command center, cron schedules, messages, user profiles, skills, and agents, making the local agent platform more accessible to Chinese-speaking users.

UPDATE4h ago

NVIDIA SkillSpector Secures Claude Code Templates

NVIDIA's open-source security scanner, SkillSpector, has been integrated into the Claude Code Templates repository to scan and protect new AI agent skill additions. SkillSpector detects potential vulnerabilities, prompt injections, and agentic risks by analyzing instruction sets and tool definitions prior to execution, ensuring that third-party contributions do not introduce malicious behaviors or security flaws into development environments.

MODEL6h ago

MAI-Image-2.5 generates its own launch images

Microsoft AI has released MAI-Image-2.5, a high-performance image generation and editing model whose promotional launch images were entirely self-generated by the model itself. Available via Azure AI Foundry and OpenRouter, it targets professional workflows with advanced text rendering, precise image-to-image editing, and improved visual reasoning.