OPEN LINK ↗
// 1h agoOPENSOURCE RELEASE
Garlic LLM runs Qwen 9B on PS4
Garlic LLM is an open-source homebrew runtime that executes Alibaba's Qwen3.5-9B fully offline on a stock PlayStation 4 GPU at up to 43 tokens per second. It combines custom 3-bit and 4-bit quantization, speculative decoding via native MTP heads, and direct PM4 hardware command buffers to fit inside the console's memory budget.
// ANALYSIS
Running a 9B-parameter LLM at interactive speeds on an 11-year-old console's unified 8GB GDDR5 memory proves how much headroom edge quantization and bare-metal GPU dispatch still offer.
- –Bare-metal efficiency: By bypassing high-level compute APIs and writing raw PM4 packets directly to the PS4's AMD GCN GPU, the runtime maximizes memory bandwidth over unified GDDR5 memory.
- –Mixed-precision squeeze: Allocating 4-bit weights for attention heads and 3-bit weights for MLP projections compresses the 9B model into 4.5GB, fitting within the ~5GB userland memory budget alongside a 16K context window.
- –Speculative decoding boost: Leveraging Qwen3.5's native Multi-Token Prediction (MTP) head delivers 31–39 tokens/sec for text and 43 tokens/sec for code generation without cloud dependencies.
- –Full edge agent stack: Packaging local STT (Parakeet TDT), TTS (Chatterbox), and embedded MicroPython code execution into a single 5.7GB PKG turns aging consumer hardware into an autonomous local AI node.
// TAGS
garlic-llmqwenllmquantizationinferenceedge-aiopen-source
DISCOVERED
1h ago
2026-11-14
PUBLISHED
2h ago
2026-11-14
RELEVANCE
8/ 10
AUTHOR
IntCyberDigest