UniPRO: Progressive Multimodal Representation Learning for Cross-modal Retrieval
What it is
This paper introduces UniPRO, a framework for analyzing how the order of fusing visual, acoustic, and speech modalities affects multimodal video retrieval performance. It demonstrates that fusion order is task-dependent, showing that schedules introducing the query-matching modality last generally perform better, though this can reverse under specific modality missingness conditions.
The problem it solves
The lack of clear guidelines on how the order of fusing different modalities (visual, audio, speech) should be determined based on data availability and query type in multimodal retrieval.
What you could use it for
- Designing fusion strategies for text-to-video retrieval systems
- Optimizing multimodal model architectures for scenarios with missing audio or speech data
- Benchmarking the impact of modality integration order on retrieval accuracy
Why it matters
Provides a principled method for selecting optimal fusion orders in multimodal systems, potentially improving retrieval accuracy by up to significant margins on standard benchmarks like MSR-VTT-9k.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.