arXiv:2610.36484v1 Announce Type: new Abstract: Claims about the cognitive capabilities of large language models are typically tested by translating behavioral paradigms into text prompts, bypassing...
CogGym is a scalable, unified framework that standardizes diverse cognitive experiments into a task‑agnostic Experiment Markup Language (EML) for systematic comparison of human and AI behavior. The initial release curates 258 experiments from 100 papers focused on human commonsense reasoning and evaluates 50 large language models, revealing a scaling trend where larger models better reproduce human judgments but still lag far behind human split‑half reliability. The framework aims to continually incorporate new cognitive science experiments to track where model behavior aligns with or diverges from human cognition as models evolve.
Co-Design Bench is a new, reference‑free benchmark comprising 273 multimodal coding tasks derived from 26,314 real user sessions. It features two tracks—Multimodal Editing and Repository Editing—that test agents on localized edits with optional image, audio, video, or document inputs and coordinated repository changes across three capability dimensions: Grounded Understanding, UI Transformation, and Functional Implementation. The benchmark uses a judge‑then‑refute protocol with hidden expert‑authored acceptance criteria, and the best of ten model endpoints achieves an average of 69.9% criterion completion but fully satisfies only 26.4% of tasks, especially struggling with repository edits involving multiple change types.
arXiv:2610.15129v1 Announce Type: new Abstract: Human social perception is incremental: we infer intent before an utterance is complete, revise our interpretation as new cues arrive, and attend to sp...
arXiv:2606. 26348v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) can process diverse inputs, e.
arXiv:2508.13186v2 Announce Type: replace-cross Abstract: AI agents with advanced reasoning and tool-use capabilities have demonstrated impressive performance in web browsing for deep search. However...
arXiv:2608. 05729v1 Announce Type: new Abstract: As capabilities rapidly increase, AI agents can move from running inside one app to acting across a user's devices over time.
DREAM (Dynamic Resolution Assignment For Multimodal Multi-Agent Debate) enhances multimodal multi-agent debate by dynamically assigning each agent the optimal visual resolution through a zero-shot probe round that measures uncertainty with Average Normalized Log-Likelihood. It also introduces Uncertainty-Guided Rollback Aggregation to mitigate groupthink by tracking and restoring low-uncertainty answers that are overridden by group pressure. Across six multimodal datasets, DREAM improves the accuracy-token trade-off by 1.5–3.2% over existing baselines without requiring dataset-specific tuning.
arXiv:2601. 04203v2 Announce Type: replace-cross Abstract: We present FronTalk, a benchmark for front-end code generation that pioneers the study of a unique interaction dynamic: conversational code generation with multi-modal feedback.
The paper introduces Test-Time Adaptation through Human‑Agent Interaction (TAHI), a method that uses iterative human feedback to adapt AI agents to individual users’ criteria. By integrating cross‑session interaction data into agent context and weights, and building an evolving rubric module, the authors demonstrate that agents can improve task success by 4.5–20.9% after only a few interactions. The evolving rubric also serves as a scalable annotation tool, detecting 16.0–22.3% more failures than language models or humans alone, and personalized agents can even generalize improvements up to 8.8% across users.
arXiv:2606. 31966v1 Announce Type: cross Abstract: Recent multimodal large language models (MLLMs) have strong potential as embodied agents, but their ability to collaborate in visually grounded environments remains underexplored.
Multimodal Language Models as Text-to-Image Model Evaluators presents MT2IE, a framework where a multimodal large language model generates evaluation prompts and scores images, achieving higher correlation with human judgment than prior metrics. MT2IE recovers official T2I model rankings using only 20 prompts—far fewer than traditional benchmarks—and adapts prompts to each model’s performance, maintaining informative scoring ranges. The approach demonstrates that dynamic, interactive evaluation can replace static benchmarks as T2I models improve.