As the Story Unfolds: Watching a Film and Identifying Characters as a Human Does
What it is
This paper introduces a method for identifying and naming principal characters in films directly from video and dialogue, without relying on external cast lists or actor photos. It proposes a 'character naming' task that resolves names using cinematic cues like shot-reverse-shot structure and gaze, and provides a progressive framework to build a character bank as the movie plays.
The problem it solves
Existing video understanding models struggle to identify who is who in a film without external metadata, failing to leverage the cinematic and dialogue cues humans use to resolve character identities.
What you could use it for
- Automated movie indexing and metadata generation
- Real-time character tracking for accessibility features (e.g., subtitles with speaker names)
- Evaluating multimodal LLMs on long-form video understanding
Why it matters
Enables AI systems to understand narrative structure and character relationships in real-time, moving closer to human-like comprehension of visual media without needing pre-existing knowledge of the cast.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.