AGDe-Judge
What it is
This is the official code implementation for the EMNLP 2025 paper on Assistant-Guided Mitigation of Teacher Preference Bias in LLM-as-a-Judge. It provides a pipeline to generate refined training data by using an assistant model to critique and refine the feedback of a stronger 'teacher' model, specifically targeting bias in pairwise evaluations.
The problem it solves
LLM-as-a-Judge systems often inherit or amplify the preference biases of the teacher model used to generate training data, leading to unfair or skewed evaluations.
What you could use it for
- Training LLM judges to reduce preference bias towards specific teacher models
- Generating high-quality DPO and SFT datasets for evaluation tasks
- Benchmarking judge models on bias metrics like OffsetBias and MT-Bench
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.