ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skills
What it is
ViSkill is a framework for Vision-Language Model (VLM) agents that encodes successful interaction trajectories as composite visual skill cards rather than text. It creates a closed feedback loop where these visual skills guide inference and reward shaping, while new successes are distilled back into the library to improve the policy.
The problem it solves
Existing skill-augmented agents are text-centric, which linearizes spatial layouts and loses critical geometric structure, leading to suboptimal policy learning and slower convergence.
What you could use it for
- Training VLM agents for grid-based games like Sokoban and FrozenLake
- Improving sample efficiency in reinforcement learning for visual agents
- Developing reusable visual strategies for robotic manipulation or navigation
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.