HydRAG benchmark finds no RAG winner

The real story here is not that CRAG “fails,” but that retrieval optimization is brutally corpus-specific. A pipeline can look excellent on a familiar codebase and fall apart the moment the domain shifts.

• –BM25 still looks like the most reliable cheap baseline: sub-ms on the fast path and good enough to justify staying in the stack.
• –CRAG behaves like a high-variance bet: when the uncertainty gate is right it helps, but when it fires unnecessarily it turns latency into the product problem.
• –The external corpus drop on CPython and Kubernetes reads like domain shift plus query mismatch, not just model weakness.
• –Reciprocal Rank Fusion smooths over disagreements between heads, but it does not eliminate the underlying dependence on corpus familiarity.
• –The open-sourced harness is the most interesting part for the broader community, because this kind of benchmark is exactly what separates “works in my repo” from a reusable retrieval strategy.