Falcon: Efficient Streaming Audio-Visual Target Speaker Extraction
What it is
This paper introduces Falcon, a lightweight causal model for real-time audio-visual target speaker extraction, and RealSSA, a new benchmark for realistic acoustic scenes. Falcon uses a single-pass multi-scale time-frequency separator and a speaker contrastive loss to achieve state-of-the-art performance with significantly reduced parameters and latency compared to existing baselines.
The problem it solves
Existing causal audio-visual target speaker extraction systems are either too computationally heavy for edge deployment or rely on inefficient repeated separation calls, and current benchmarks fail to capture the complexity of real-world acoustic scenes with multiple dynamic speakers.
What you could use it for
- Real-time speaker extraction in noisy, multi-speaker environments
- Edge deployment of audio-visual speech enhancement systems
- Benchmarking speaker extraction models against realistic near/far-field interference
Why it matters
Enables high-quality, low-latency speaker extraction on edge devices by reducing computational overhead by over 20x, making real-time audio-visual processing feasible for latency-sensitive applications.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.