REDDIT · REDDIT// 21d agoOPENSOURCE RELEASE

PageGod Narrative Scanner finds continuity errors without LLMs

A local, zero-API-cost deterministic engine that identifies narrative continuity errors in long-form prose, such as characters in impossible locations or objects being used after custody breaks. Developed by Joshua Kirchner, the project includes a scanner, evaluation harness, and a research paper that audits the reliability of existing LLM-as-judge benchmarks, finding that many "expected findings" in those benchmarks were actually false ground truths.

// ANALYSIS

PageGod proves that deterministic state-tracking can outperform expensive LLM-based judges for specialized QA tasks while exposing the fragility of current AI evaluation datasets.

–Tracks character movement, object custody, barrier states, and timeline drift without non-deterministic LLM calls.
–Achieves a 0.75 F1 score on authored benchmarks, providing a reliable alternative to high-cost API pipelines.
–Exposes significant reliability issues in LLM-judged benchmarks like ConStory, where 37.5% of findings were false ground truths upon manual inspection.
–Ideal for local/offline writing QA and as a deterministic validator for synthetic narrative generation.

// TAGS

llmtestingcode-reviewopen-sourcedevtoolresearchpagegod-narrative-scanner

DISCOVERED

21d ago

2026-03-22

PUBLISHED

21d ago

2026-03-22

RELEVANCE

8/ 10

AUTHOR

Glass_Offer5140