Prisma garage model beats GPT-2 on 30B tokens

A one-person garage model with genuine architectural novelty outperforming established baselines on a fraction of compute — if G²LU and WoRPE scale, this could quietly influence how mainstream architectures handle feature learning.

• –G²LU (Gated-Gated Linear Unit) replaces SwiGLU with nested gates, enabling 100x higher learning rates and creating saddle-surface decision boundaries that resist memorization
• –Mirrored weight sharing achieves 2N virtual layers from N unique parameter sets — a parameter-efficient trick preserving representational depth without added cost
• –WoRPE encodes word-boundary position geometrically in attention heads, surfacing information already latent in BPE tokenization without a new tokenizer
• –Beats GPT-2 Medium (40B tokens) on 5/8 benchmarks with only 30B tokens, with particular strength on reasoning tasks (ARC-Easy +11pp, BoolQ 0.620)
• –Key caveats: untested beyond ~350M scale, single-developer maintenance, and minimal community traction (2 Reddit upvotes, 230 HF downloads/month)