MOSAIC: Benchmarking Theory of Mind in Vision-Language Models
What it is
MOSAIC is a controlled benchmark that evaluates how well vision-language models (VLMs) translate theory of mind (ToM) inferences into coordinated social actions. It tests 13 models across 200 trials per model in scenarios requiring the integration of verbal, spatial, gaze, and facial cues, revealing that most VLMs fail to produce consistent nonverbal signals or interpret others' behaviors despite ToM constraints.
The problem it solves
Existing benchmarks evaluate theory of mind reasoning and embodied behavior in isolation, failing to measure the critical gap between inferring mental states and executing coordinated social actions.
What you could use it for
- Evaluating the social reasoning capabilities of new vision-language models
- Identifying bottlenecks in the translation of mental state inferences to physical behavior
- Benchmarking embodied agents in cooperative and competitive multi-agent settings
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.