In-context Learning of Single-index Targets: Comparing Kernel and Feature Learners
What it is
This paper analyzes the generalization performance of two one-layer attention architectures for nonlinear in-context learning tasks. It uses the replica method to derive theoretical predictions for memorization and generalization errors, comparing a kernel learner (fixed feature map) against a feature learner (learned readout). The results are validated against numerical experiments and used to generate phase diagrams showing when each architecture is advantageous.
The problem it solves
Existing theories for in-context learning primarily focus on linear target functions, leaving a gap in understanding how attention mechanisms handle nonlinear tasks.
What you could use it for
- Theoretical analysis of transformer attention mechanisms
- Understanding the impact of pretraining data size on in-context learning
- Designing attention architectures for specific task distributions
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.