8B Model Outperforms LoRA in Continual Learning Experiments

A recent experiment involving the Apertus-8B model has demonstrated that certain approaches to continual learning can significantly outperform traditional LoRA (Low-Rank Adaptation) methods when absorbing new information. The study highlights a critical challenge in AI development: how a model can learn new facts without losing its ability to recall previous knowledge or perform general reasoning tasks.

What was the continual learning experiment?

According to a report shared on Reddit ML, the experiment focused on how well an 8B parameter model could integrate and retain real-world news data over a four-day period. The methodology involved training the model on specific news from four consecutive days (November 2, 4, 5, and 6).

Each night, the model's weights were updated with the day's news. The following morning, the model was tested on its ability to recall that news without any of the information being provided in the prompt. To ensure scientific rigor, the researcher noted that the passing benchmarks were established before the questions were actually posed. The study compared the performance of the Apertus-8B model against a version using a plain LoRA trained nightly on the same practice questions.

How did the 8B model compare to LoRA?

Results indicated a stark difference in how the two methods handled information retention. After four nights of continuous learning, the Apertus-8B model maintained a 90.7% accuracy rate regarding the news from Day 1. In contrast, the LoRA-trained model dropped to just 26.7% accuracy for that same initial day of news.

While the LoRA method showed high performance on immediate practice questions—reaching between 93% and 96% the morning after training—it struggled to maintain that knowledge as more data was introduced. This phenomenon is often referred to in machine learning as catastrophic forgetting, where new training overrides or interferes with previously acquired information.

Did the learning impact general reasoning skills?

One of the most significant findings in the experiment was the impact on the model's general reasoning capabilities, measured by the GSM8K benchmark. The Apertus-8B model finished the four-day period with a GSM8K score of 69.2%, which is nearly identical to its original release score of 69.6%.

The LoRA approach, however, saw a massive degradation in reasoning abilities. According to the source, the LoRA-trained model's performance on GSM8K plummeted to 4.4% after the four days of news training. This suggests that while LoRA can be effective for rapid, short-term adaptation to specific tasks, it may be highly unstable for long-term continual learning where general intelligence must be preserved.

What were the results for the subsequent days of news?

As the experiment progressed beyond the first day, the retention of newer information also showed fluctuations. The researcher reported that for Days 2 and 3, the accuracy for stories that had been previously introduced dipped to approximately 70%. This indicates that while the 8B model is much more robust than LoRA, maintaining perfect recall across a continuous stream of shifting data remains a complex hurdle for even advanced architectures.

How can researchers apply these findings?

For developers and researchers working on autonomous agents or long-term AI assistants, these results emphasize the importance of selecting training architectures that prioritize stability. The ability to ingest daily updates—such as news, logs, or user preferences—without breaking the model's underlying logic is essential for moving toward truly adaptive AI. The experiment suggests that weight-based updates in specific models may offer a more sustainable path for continual learning than standard LoRA fine-tuning in certain high-stakes or high-volume data environments.

As the industry moves toward models that learn in real-time, understanding the trade-off between rapid adaptation and long-term stability will be a primary focus for AI architecture design.

For those looking to analyze similar research papers or technical documentation regarding model architectures and training methodologies, Grafy Reader provides an environment to upload, search, and question complex PDFs and academic papers using AI.

Frequently asked questions

How did the Apertus-8B model perform on Day 1 news after four days?

The Apertus-8B model achieved a 90.7% accuracy rate on the news from the first day of the experiment. This was measured after the model had undergone four consecutive nights of learning additional news data, demonstrating high levels of information retention compared to other methods.

What happened to the LoRA model's reasoning abilities?

The LoRA model experienced a significant decline in its general reasoning capabilities. While the original model's GSM8K score was much higher, the version trained nightly with LoRA fell to a 4.4% score on the GSM8K benchmark after four days of news integration.

Why did the LoRA model fail to retain Day 1 news?

The LoRA model failed to retain the Day 1 news because it only achieved 26.7% accuracy after four nights. While it was highly effective at learning immediate practice questions, the method proved unable to maintain older information as new data was introduced nightly.

---

Source: Reddit ML — Continual learning on 4 days of real news: an 8B model keeps day 1 at 90.7% vs 26.7% for LoRA, GSM8K unchanged (sealed test) [R]