Unified Training Rewrites Foundation Model Training

The compelling idea is not eliminating training stages, but reframing them as a data-mixture scheduling problem. The results are promising, though the experiments remain far smaller than frontier-model scale.

• –A shared weighted cross-entropy trainer can handle static data, demonstrations, and reward-weighted model generations.
• –Unified schedules outperformed staged training on controlled reasoning tasks, including larger unseen problems.
• –Mixing midtraining data helped preserve broader capabilities and reduce forgetting during later RL.
• –Data proportions and timing create real tradeoffs: retention and coverage can improve while task-specific pass@1 declines.
• –The infrastructure is simpler conceptually, but reward verification, asynchronous generation, and schedule design remain substantial engineering challenges.