RAG retrieval quality faces production test

The key lesson is that retrieval quality is a systems problem, not a single score. Teams that get real value combine labeled evals, online monitoring, and retrieval-stack fixes instead of treating embedding similarity as the answer.

• –Build a real golden set from production queries and score recall@k and MRR against it.
• –Use LLM-as-judge for scalable relevance checks, but calibrate it to human labels and keep the rubric simple.
• –Hybrid search, query expansion, and metadata filters often outperform more embedding tuning.
• –Watch reformulations, follow-up questions, and thumbs down as the best production canaries.