Nemotron 3 Nano Challenges Fine-Tuning Playbook

Hybrid Mamba-MoE models are the efficiency endgame, but their fine-tuning mechanics are currently undocumented "war zones" for solo developers.

• –Router Risk: Standard LoRA often targets all linear layers, but modifying MoE routers without careful weighting usually leads to expert collapse or degraded routing logic; keeping them frozen is often the safer baseline.
• –Mamba Recurrence: The selective SSM state in Mamba-2 is more fragile than attention weights; low-rank perturbation in the projection matrices can cause state drift or instability over long sequences.
• –Task Isolation: Multi-task imbalance in sparse models is a feature, not a bug—aggressive auxiliary load-balancing loss can force the model to homogenize experts that should have specialized for distinct tasks.
• –Evaluation Granularity: Aggregate metrics are deceptive in MoE; per-task expert activation tracking is required to ensure that specific capabilities aren't quietly "hollowed out" during training.