Fisher-Rao Monitor Targets LLM Drift

This is the kind of monitoring idea that feels more production-relevant than yet another embedding-based drift metric, because it watches the model’s actual predictive behavior instead of proxy features.

• –Output-distribution monitoring is a stronger fit for deployed LLMs than input-embedding checks when the failure mode is gradual policy, domain, or prompt mix shift
• –Fisher-Rao distance gives the pitch a real theoretical spine, but the adoption hurdle will be implementation complexity versus simpler entropy, KL, or confidence-based baselines
• –The reported OpenAI logprobs result is the interesting bit: catching drift in 7 steps with no warmup false alarms is exactly the kind of early-warning behavior ops teams want
• –Reviewers will likely care about robustness across temperatures, prompts, and models, plus whether the signal still wins once you compare against lightweight production heuristics
• –This is research, not a shipped product, so the main value right now is methodological: it could become a useful monitoring primitive if the broader evaluation holds up