EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory
What it is
This paper proposes EngramEdit, a method for updating factual knowledge in Large Language Models (LLMs) by modifying conditional memory embeddings rather than the Transformer backbone. It works by computing target memory representations for multiple expressions of a fact and jointly updating shared n-gram embeddings to match these targets, while penalizing changes to frequently reused embeddings to prevent collateral damage to unrelated knowledge.
The problem it solves
Traditional model editing methods often fail to generalize across different expressions of a fact or cause unintended changes to unrelated knowledge when updating shared model parameters.
What you could use it for
- Updating outdated factual knowledge in deployed LLMs without retraining the entire model
- Maintaining model consistency across different phrasings of the same fact
- Performing multi-hop reasoning with updated knowledge without degrading general capabilities
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.