TileRT optimizes Large Language Model execution on GPUs by using persistent kernels to minimize microsecond-scale execution gaps and enable ultra-low-latency serving.

Eliminating operator launch overheads via persistent kernels on commodity hardware shifts the LLM serving bottleneck from computation limits to memory and execution efficiency.

• –Persistent kernels reduce CPU-GPU communication overhead, which is critical for low-batch, high-speed interactive generation.
• –Co-development with Xiaomi indicates a high priority on optimizing LLMs for edge-adjacent or consumer-grade hardware architectures.
• –Moving from per-operator optimization to tile-level runtime orchestration represents a mature step in deep learning compilation.