NCP-ArchPreview cuts pretraining tokens, outperforms OLMo-3-7B

Next Concept Prediction directly attacks the compute inefficiency of token-by-token autoregression, potentially disrupting pretraining economics if scaling laws hold at larger capacities. Achieving comparable pretraining loss with ~51% of training tokens represents a massive reduction in the compute budget required to train competitive models, while downstream gains like +5.99 on GSM8K indicate that higher-level conceptual structures enhance reasoning. Open-sourcing weights, checkpoints, and recipes allows the community to validate stability and test concept bottlenecks, though open questions remain around inference latency, codebook collapse risks, and multimodal adaptation.