AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation
What it is
This paper introduces AdviSD, a method for training small advisor models to steer frozen large language model executors. It combines outcome-based reinforcement learning with selective self-distillation, where the advisor uses the difference in executor responses with and without its advice to determine which corrections to learn from.
The problem it solves
Standard reinforcement learning for advisors can be hindered by 'plausible' corrections that do not actually change the executor's behavior, leading to suboptimal learning signals and reduced final performance.
What you could use it for
- Improving the performance of frozen LLMs on tool-use benchmarks like BFCL-v3
- Enhancing agent performance on complex environments like EnvScaler
- Training lightweight advisors that can transfer across different executor model families
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.