Harness Compilation: Which Decisions Should a Small Vision-Language Model Keep?
What it is
This paper introduces Harness Compilation (HC), an offline procedure that optimizes the division of labor between a frozen small vision-language model (VLM) and its external tool harness. A large teacher model analyzes student execution traces to revise reusable content and control logic, while a validation set selects the final deployed harness configuration.
The problem it solves
Small vision-language models often struggle to effectively obtain and utilize external evidence, leading to suboptimal performance compared to larger models or those with optimized tool-use strategies.
What you could use it for
- Improving the performance of small VLMs on visual question-answering tasks without fine-tuning weights
- Optimizing agent tool-use policies for bounded choice and evidence integration tasks
- Reducing the computational cost of deploying small VLMs by offloading complex reasoning to a static harness
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.