Catastrophic Forgetting
When fine-tuning an AI model on new data causes it to lose previously learned capabilities and knowledge.
Definition
Catastrophic forgetting occurs when a neural network, upon being trained on new information, rapidly loses the ability to perform tasks it previously handled well. The new training overwrites the weight configurations that encoded earlier knowledge, effectively erasing what the model once knew.
This is a fundamental challenge in AI development because it means models cannot simply be continuously updated with new data. Special techniques like elastic weight consolidation, replay buffers, or progressive training are required to add new capabilities without degrading existing ones.
Why It Matters
Organizations that fine-tune AI models for specific business needs risk degrading the model's general capabilities. A model customized to write in your brand voice might lose its ability to reason about numbers or follow complex instructions if fine-tuned carelessly.
Understanding catastrophic forgetting helps teams make informed decisions about when to fine-tune versus when to use prompt engineering, and how to structure training data to preserve critical base capabilities.
Examples in Practice
A company fine-tunes a language model extensively on customer support transcripts. The model becomes excellent at support conversations but loses its ability to write coherent long-form content or perform summarization tasks it previously handled well.
A news organization updates their content classifier with recent articles but finds it can no longer accurately categorize older article types that were well-represented in the original training but absent from the update data.