Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models
What it is
Co-Design Bench is a reference-free benchmark consisting of 273 tasks derived from real user coding sessions to evaluate agentic models on multimodal collaborative coding. It features two tracks, Multimodal Editing and Repository Editing, and uses a judge-then-refute protocol with hidden expert-authored acceptance criteria to grade model performance without requiring gold code or target screenshots.
The problem it solves
Existing evaluations often lack realistic, multimodal, and repository-level tasks that reflect actual user coding sessions, making it difficult to assess whether agents can fully satisfy complex, multi-faceted user requests.
What you could use it for
- Evaluating the ability of LLMs to translate multimodal user requests (image, audio, video, text) into working code revisions
- Benchmarking agentic models on their capacity to perform coordinated repository-level edits
- Studying the gap between partial requirement satisfaction and complete task success in human-AI co-design workflows
Why it matters
Provides a rigorous, realistic testbed for measuring the practical capabilities of coding agents in collaborative design scenarios, helping identify specific weaknesses in grounded understanding, UI transformation, and functional implementation.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.