arXiv AI
ArtifactArena: Evaluating Models by What They Build in the Physical World
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 17
LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition
The paper introduces LM Fight Arena, a new benchmark that pits large multimodal models against each other in the fighting game Mortal Kombat II to evaluate real‑time visual understanding and sequential decision‑making. Six leading open‑ and closed‑source models were tested in a controlled tournament where each controlled the same character, ensuring a fair comparison. The framework offers a fully automated, reproducible, and objective assessment of an LMM’s strategic reasoning in a dynamic setting.
arXiv AI
Jun 30
StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valley
arXiv:2507. 07445v3 Announce Type: replace Abstract: Autonomous agents navigating human society must master both production activities and social interactions, yet existing benchmarks rarely evaluate these skills simultaneously.
arXiv AI
Jun 12
From Digital to Physical: Digital Agents as Autonomous Coaches for Physical Intelligence
arXiv:2601. 21570v2 Announce Type: replace Abstract: The field of Embodied AI is witnessing a rapid evolution toward general-purpose robotic systems, fueled by high-fidelity simulation and large-scale data collection.
arXiv AI
Sep 28
SciHorizon-eLab: An Agentic Protocol-to-Task Compiler for Scalable Benchmarking of Scientific Embodied Agents
arXiv:2609. 30971v1 Announce Type: new Abstract: Embodied agents offer a promising route to automating scientific experimentation, yet their progress is constrained by the lack of reliable and systematic evaluation environments.
arXiv AI
Oct 15
Words to World: Benchmarking LLMs in Interactive Physical Environments
arXiv:2610.15228v1 Announce Type: new Abstract: Words are increasingly becoming an interface for action: large language models (LLMs) can now invoke tools, operate devices, and make decisions that af...
arXiv AI
Oct 31
Gym-Anything for Robotics: Scaling physical simulations with digital agents
Gym-Anything for Robotics is a framework that uses coding agents to automatically create realistic robotics simulations, addressing the scalability bottleneck of expert-driven simulation design. By selecting tasks based on U.S. GDP data and employing an adversarial creation-audit loop, the authors built RoboWorld, a benchmark of 25 simulations with 25 test and 175 training RL environments. The paper also introduces new techniques—such as in-context learning, privileged teachers, self-assisted learning, and code policies—to improve frontier models, all of which significantly boost performance on the benchmark.