Peer-to-peer LLM inference hits bandwidth wall

Feasible technically, but only if you’re honest about the tradeoffs: p2p inference is an infrastructure trick, not a free scaling law.

• –Petals has already demonstrated decentralized inference and fine-tuning over the internet, including claims of running large models with interactive latency better than simple offloading.
• –LocalAI now supports p2p/federated inference for llama.cpp-compatible models, but its docs make the constraints clear: one model, workers need to be present up front, and the system is still tightly scoped.
• –Exo pushes the idea further with automatic discovery and dynamic partitioning across heterogeneous devices, which makes it a strong proof of concept for cooperative clusters.
• –The real bottleneck is communication overhead per token; once layers, KV cache, and activations need to move between nodes, latency quickly dominates compute savings. That is why this tends to work better on LANs, homelabs, or curated networks than on the open internet.
• –Desirable? Yes, for community compute, redundancy, and lowering the hardware bar. No, as the default serving model for most products, where a single box or a proper GPU cluster is simpler, faster, and easier to secure.