Attention Sinks in VLMs Are Borrowed, Not Seen
What it is
This paper investigates visual attention sinks in vision-language models (VLMs), identifying that they are image tokens whose keys align with the language model's text-sink key. It proposes a training-free method to detect and mitigate these sinks by redirecting their attention to the text-sink and inserting a cropped, cleaned attention region into the reasoning trace.
The problem it solves
Visual attention sinks in VLMs absorb disproportionate attention regardless of task, misleading methods that rely on attention maps to understand where the model is 'looking' and reducing performance on fine-detail tasks.
What you could use it for
- Improving accuracy on fine-detail visual question answering (VQA) tasks
- Mitigating misleading attention maps in VLMs for better interpretability
- Enhancing reasoning traces in vision-language models without additional training
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.