Llama 1B continued pretraining wipes chat skills

At 2e-4 on a tiny corpus, this looks more like overfitting and distribution drift than a broken model. Continued pretraining can add domain knowledge, but it will not preserve chat behavior unless you explicitly protect that behavior during training.

• –If you started from an instruct-tuned checkpoint, raw-text CPT likely stripped the alignment layer; the safer pattern is base CPT first, then a fresh instruction-tuning pass.
• –Meta’s Llama 3.2 model card says the chat-oriented variants are built with post-training on top of the pretrained base, so raw CPT alone is not supposed to carry assistant behavior.
• –Apple’s 2025 work on forgetting says even a small replay mix of pretraining data can help shield the model, and continual-pretraining papers make that general-vs-domain tradeoff explicit.
• –The simplest guardrail is a mixture: keep some general/instruction data in the run, clean out markup or boilerplate, and stop early before the model locks onto template noise.