CrossWeave: Cross-Modal Scene and Instance Retrieval
What it is
CrossWeave is a framework for cross-modal retrieval that aligns sparse 2D image patches with 3D points to retrieve corresponding scenes or objects. It uses an optimal transport-based aggregator to cluster redundant regions, allowing a single model to handle both scene-level and instance-level retrieval tasks effectively.
The problem it solves
Existing cross-modal retrieval methods only learn alignment at the whole-scene or whole-object level, lacking supervision for fine-grained local structures and requiring separate models for scene versus instance tasks.
What you could use it for
- Retrieving 3D environments from 2D image queries for robotics navigation
- Finding specific 3D objects within a scene using 2D visual cues for AR/VR applications
- Spatial reasoning tasks requiring mapping between 2D visual data and 3D geometric structures
Why it matters
Enables a single unified model to perform both scene and instance retrieval, reducing the need for separate specialized models and improving the efficiency of spatial reasoning in robotics and augmented reality.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.