LocalLLaMA proposes crowdsourced LLM hallucination registry

Static benchmarks are failing to keep up with the rapid pace of open-source model releases and quantization-driven regressions.

• –Real-world performance of models like Qwen 3.5 9B and Gemma 4 26B often differs from synthetic benchmark scores, especially on niche tasks like git wiki editing.
• –A crowdsourced registry would allow users to submit "hallucination triplets" (prompt, model, result) to track factual reliability across different quantization levels (Q4, Q5, etc.).
• –This could evolve into a living leaderboard, complementing existing AI-Omniscience and HLE indexes by focusing on the local-first community's specific edge cases.
• –The success of such a database depends on automated verification of submissions, potentially using stronger models as "judges" to maintain data quality.