UC Berkeley releases Continual Learning Bench

While LLM agents excel at isolated, stateless tasks, true autonomy requires learning and adapting online over time. CL-Bench exposes a critical flaw in current frontier models: they cannot learn continuously without overfitting or failing to transfer knowledge.

• –The shift from stateless evaluations to stateful, sequential testing is a necessary step towards evaluating real-world agents (like coding assistants or database administrators) that interact with the same environment over time.
• –Introducing a "gain metric" is a clever way to isolate online learning performance from the model's baseline pre-trained capabilities.
• –Current frontier models struggle immensely with continual learning, showing that we cannot just scale context size or pre-training data; we need fundamental algorithmic improvements in memory and online optimization.