Local Support Learning
What it is
This paper proposes Local Support Learning (LSL), a framework to prevent catastrophic forgetting in large pre-trained models. It works by pairing a standard weight adapter with a Gaussian Mixture Model (GMM) gate that restricts updates to inputs from the current training distribution, thereby preserving prior capabilities without needing access to old data.
The problem it solves
Catastrophic forgetting in large pre-trained models, where fine-tuning on new data causes the loss of previously learned capabilities.
What you could use it for
- Continual learning for Large Language Models (LLMs)
- Multi-phase fine-tuning where previous capabilities must be retained
- Post-training adaptation of 7B+ parameter models
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.