Luma AI unveils MARBLE attention

MARBLE’s strongest idea is making parallelism mask-aware instead of forcing every multimodal workload into one communication pattern.

• –Per-layer strategy selection adapts to locality, density, sliding windows, and attention-head limits.
• –FP32 softmax accumulation preserves single-GPU numerics across ring steps, addressing a major ring-attention weakness.
• –Sparse all-to-all avoids 55–100% of unnecessary remote key transfers on localized masks.
• –PTRR reduces ring-step imbalance to within 0.1% and roughly halves layer time.
• –The main tradeoff is CPU planning overhead, which must be hidden through data-loader parallelism.