SPLICE benchmark exposes VLMs' video reasoning failures

VLMs aren't watching video — they're reading descriptions of video, and SPLICE makes that embarrassingly clear.

• –Best-performing model (Gemini 2.0 Flash) hit 51% vs. 85% human baseline — a 34-point gap that can't be explained away by task difficulty
• –Adding human-written text descriptions boosted model scores significantly but didn't affect human performance at all — definitive proof that models are leaning on language, not vision
• –Models predicted visually similar clips were adjacent 57% of the time; humans did so only 2.5% of the time, and random chance is 27% — models are running image similarity, not event reasoning
• –Scaling the language model doesn't fix the vision bottleneck: Qwen2-VL-7B matched the 72B variant on pure visual reasoning, meaning bigger LMs can't compensate for weak vision encoders
• –Open-source models fared worst: LLaVA-OneVision-72B barely cleared random guessing in vision-only mode, and Qwen2-VL-72B outperformed Gemini on text-only — a damning inversion