NVIDIA Brings Tool Calls to Duplex Speech
NVIDIA researchers introduced a modular frontend-backend architecture that enables full-duplex speech models to execute tool calls without sacrificing real-time conversational fluency. By having a lightweight duplex frontend emit delegation tokens to a text LLM backend and reinject results via a prefill-and-repeat mechanism, the system achieves up to 97.2% tool-call recall while maintaining natural turn-taking.
Pure end-to-end monolithic voice models are hitting a wall on complex reasoning—decoupling real-time conversational duplex streaming from text-based agentic reasoning is the practical architecture conversational AI needs.
* **Pragmatism beats end-to-end purity:** Instead of burdening streaming speech networks with complex API schemas and symbolic reasoning, routing decisions to frontier text backends recovers the massive ~30-50% drop in task completion.
* **Preserves conversational dynamics:** By restricting frontend modifications to a lightweight delegation token and a prefill-and-repeat mechanism, the system maintains natural interruption handling and low-latency duplex turn-taking.
* **Plug-and-play modularity:** Decoupling allows developers to independently optimize low-latency speech frontends while swapping backend LLMs to fit domain-specific tool-calling and reasoning budgets.
DISCOVERED
1h ago
2026-09-20
PUBLISHED
1h ago
2026-09-20
RELEVANCE
AUTHOR
omarsar0