Qwen2.5 trained for Reddit summarization via GRPO

This experiment is a masterclass in "smol" distributed training, proving that GRPO—the algorithm behind DeepSeek-R1—is viable on consumer-grade hardware for specialized tasks.

• –Using ROUGE-L as a quality reward alongside length penalties is critical; without it, the model tends to "game" the length constraint by outputting repetitive gibberish.
• –The 3x Mac Mini setup (1 master for training, 2 workers for vLLM rollouts) showcases the growing maturity of distributed MLX-based training ecosystems.
• –LLM-as-a-Judge (DeepEval) remains the gold standard for evaluating subjective qualities like clarity and faithfulness where traditional metrics fail.
• –The project highlights a common pitfall in reward engineering: confusing character counts with token counts can lead to unexpected model collapse.
• –While the absolute scores (2.5/4) are modest, the p-value of 0.0042 confirms the statistical validity of the reward pairing strategy.