OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning
What it is
OmniSeek is an agentic framework that enables Omni-LLMs to dynamically acquire sparse audio and visual evidence over specific temporal windows during multi-turn reasoning. It uses a synthesized dataset (OmniTraj-170K) for supervised learning and two-stage reinforcement learning with an Audio-Visual Necessity objective to ensure the model relies on both modalities rather than taking single-modality shortcuts.
The problem it solves
Standard Omni-LLMs passively process entire audio-visual sequences in a single forward pass, failing to efficiently retrieve sparse but critical evidence across long contexts and often relying on single-modality shortcuts.
What you could use it for
- Long-form video understanding requiring specific temporal evidence retrieval
- Audio-visual question answering where critical information is sparse in time
- Multi-hop reasoning tasks involving interleaved audio and visual contexts
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.