arXiv AI
CogArena: Benchmarking Multimodal Agents on Interactive Behavioral Experiments
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
Hugging Face Trending Papers
5d ago
CogArena: Benchmarking Multimodal Agents on Interactive Behavioral Experiments
Claims about the cognitive capabilities of large language models are typically tested by translating behavioral paradigms into text prompts, bypassing the visual interfaces and time pressure through w...
arXiv AI
Sep 21
CogGym: Towards Large-Scale Comparative Evaluation of Human and Machine Cognition
CogGym is a scalable, unified framework that standardizes diverse cognitive experiments into a task‑agnostic Experiment Markup Language (EML) for systematic comparison of human and AI behavior. The initial release curates 258 experiments from 100 papers focused on human commonsense reasoning and evaluates 50 large language models, revealing a scaling trend where larger models better reproduce human judgments but still lag far behind human split‑half reliability. The framework aims to continually incorporate new cognitive science experiments to track where model behavior aligns with or diverges from human cognition as models evolve.
arXiv AI
5d ago
Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models
Co-Design Bench is a new, reference‑free benchmark comprising 273 multimodal coding tasks derived from 26,314 real user sessions. It features two tracks—Multimodal Editing and Repository Editing—that test agents on localized edits with optional image, audio, video, or document inputs and coordinated repository changes across three capability dimensions: Grounded Understanding, UI Transformation, and Functional Implementation. The benchmark uses a judge‑then‑refute protocol with hidden expert‑authored acceptance criteria, and the best of ten model endpoints achieves an average of 69.9% criterion completion but fully satisfies only 26.4% of tasks, especially struggling with repository edits involving multiple change types.
arXiv AI
Oct 15
GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?
arXiv:2610.15129v1 Announce Type: new Abstract: Human social perception is incremental: we infer intent before an utterance is complete, revise our interpretation as new cues arrive, and attend to sp...
arXiv AI
Aug 7
Unified Agent: Managing Interactions across Devices
arXiv:2608. 05729v1 Announce Type: new Abstract: As capabilities rapidly increase, AI agents can move from running inside one app to acting across a user's devices over time.
arXiv AI
Jul 1
MECoBench: A Systematic Study of Multimodal Agent Collaboration in Embodied Environments
arXiv:2606. 31966v1 Announce Type: cross Abstract: Recent multimodal large language models (MLLMs) have strong potential as embodied agents, but their ability to collaborate in visually grounded environments remains underexplored.