OPEN LINK ↗
// 67d agoOPENSOURCE RELEASE
PageGod Narrative Scanner finds continuity errors without LLMs
A local, zero-API-cost deterministic engine that identifies narrative continuity errors in long-form prose, such as characters in impossible locations or objects being used after custody breaks. Developed by Joshua Kirchner, the project includes a scanner, evaluation harness, and a research paper that audits the reliability of existing LLM-as-judge benchmarks, finding that many "expected findings" in those benchmarks were actually false ground truths.
// ANALYSIS
PageGod proves that deterministic state-tracking can outperform expensive LLM-based judges for specialized QA tasks while exposing the fragility of current AI evaluation datasets.
- –Tracks character movement, object custody, barrier states, and timeline drift without non-deterministic LLM calls.
- –Achieves a 0.75 F1 score on authored benchmarks, providing a reliable alternative to high-cost API pipelines.
- –Exposes significant reliability issues in LLM-judged benchmarks like ConStory, where 37.5% of findings were false ground truths upon manual inspection.
- –Ideal for local/offline writing QA and as a deterministic validator for synthetic narrative generation.
// TAGS
llmtestingcode-reviewopen-sourcedevtoolresearchpagegod-narrative-scanner
DISCOVERED
67d ago
2026-03-22
PUBLISHED
67d ago
2026-03-22
RELEVANCE
8/ 10
AUTHOR
Glass_Offer5140