RWKV v6 Trainer Says Bigger Batches Win

The hot take: if your custom LM run looks stuck, the bottleneck may be optimization noise, not model capacity or one more learning-rate sweep.

• –The poster saw almost no progress at effective batch 8, then a sharp perplexity drop after scaling batch size up, which suggests the training regime was underpowered for the setup.
• –RWKV’s own ecosystem treats batch size and gradient accumulation as first-class training knobs, so this is consistent with the broader tooling around the project.
• –The result is useful for single-GPU trainers: gradient accumulation is often the cheapest lever to test before spending days on hyperparameter churn.
• –It is not a universal rule; effective batch interacts with dataset size, sequence length, optimizer choice, and decay settings, so this is a strong signal, not a guarantee.
• –Best read: for small custom LLM runs, try larger effective batches earlier than you think.