PositionalHidden
What it is
An implementation of a method to mitigate position bias in Large Language Models by scaling a single dimension of the hidden states. It provides scripts to search for positional dimensions and evaluate performance on long-context benchmarks like NQ and LongBench.
The problem it solves
LLMs suffer from position bias, particularly in long-context scenarios, leading to degraded performance when relevant information is located in the middle of the input sequence.
What you could use it for
- Improving LLM performance on long-context tasks like multi-document QA
- Mitigating the 'lost in the middle' phenomenon in LLMs
- Evaluating the impact of positional encoding modifications on model accuracy
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.