SmolLM2-360M RL loops stress M4 Macs

This looks less like a "unified memory" win turning into a lie and more like Mac training hitting the messy edge of allocator limits, backend quirks, and weak reward design all at once.

• –bfloat16 is the right instinct for stability here; fp16 can be brittle in small RL loops and can amplify NaN problems fast
• –Unified memory does not guarantee training headroom on Apple Silicon, especially once rollout count, activations, and context length stack up
• –The reward-hacking behavior is classic: if format gets rewarded more reliably than correctness, a tiny model will learn the shortcut every time
• –SmolLM2-360M is small enough that RL can easily reinforce surface-form compliance before any real reasoning capacity emerges
• –If the goal is local experimentation, the next bottleneck is usually backend choice and reward shaping, not just squeezing more context into the same setup