ShapeY is a benchmarking framework that evaluates shape-based recognition in object recognition systems using 68,200 grayscale images of 200 3D objects rendered from multiple viewpoints, with optional appearance changes. It employs a nearest‑neighbor matching task to probe how well an embedding space clusters views by 3D shape similarity across varying viewpoints and non‑shape alterations, providing quantitative and qualitative performance readouts such as error rates, viewpoint tuning curves, and matching score histograms. Testing 321 pre‑trained networks shows that even state‑of‑the‑art models struggle to generalize robustly across 3D viewpoint and appearance changes, highlighting gaps in current shape recognition capabilities.
By Jong Woo Nam, Amanda S. Rios, Bartlett W. Mel
BabelFake is a multilingual audio‑visual DeepFake benchmark featuring 399,000 clips from 496 consenting participants across five languages (English, German, Italian, French, Spanish). The dataset pairs 11 modern video manipulation methods with 4 voice‑cloning engines, covering both visual‑only and joint audio‑visual attacks. Experiments show that detection difficulty varies with the audio‑visual pairing, with significant performance drops when authentic audio is retained, and that cross‑language and demographic factors influence detector performance.
By Carlotta Segna, Joel Tschesche, Anna Rohrbach
The paper investigates when to evolve a harness versus training model weights for long‑horizon LLM agents. By labeling failed trajectories with the first signal that fires, the authors separate process failures from content failures, showing that harness evolution fixes the former while weight training addresses the latter. Experiments on DeepPlanning and WebArena‑Lite demonstrate significant performance gains, with LoRA adapters trained on evolved harness trajectories further improving held‑out scores and reducing content failures.
By Yuan Tian, Bing Hu, Hao Wang, Binghang Lu, Fang Wu
The paper introduces two inference‑time designs—factor‑compiled admission (FC) and permission‑semantic admission (PS)—to govern persistent memory in large language model agents without retraining. FC evaluates entire memory entries, while PS decomposes entries into typed units, both using deterministic policies to decide what recalled information enters the working context and how it is expressed. Experiments on a development suite and an external benchmark show that FC and PS reduce failure rates and cross‑domain leakage compared to verbatim injection, though they increase personalization failures and do not meet all preregistered criteria.
By Chang Liu, Deliang Ding
Local Timescale Gating (LT‑Gate) is a neuron model that tracks fast and slow information in parallel and uses a learned gate to adjust their influence, enabling spiking neurons to preserve long‑term context while responding quickly. The approach includes a variance‑tracking regularization inspired by biological homeostasis and operates with local updates, making it compatible with neuromorphic hardware such as Intel’s Loihi chip. Empirical results show LT‑Gate improves accuracy and retention on sequential learning tasks, achieving about 51 % final accuracy versus 46 % for a recent Hebbian baseline and outperforming prior SNN methods.
By Devansh Deep, Ansh Tiwari, Ayush Chauhan
AgentHorizon is a benchmark comprising 1,373 computer‑use tasks with instruction‑trajectory pairs collected from 166 hours of human‑recorded activity across three operating systems. The benchmark uses a paired design that swaps instructions to create negative tasks, testing judges’ ability to distinguish successful trajectories from those that completed a similar but incompatible request. Eleven judges were evaluated, with GPT‑5.5 achieving the highest balanced accuracy of 80.9% on the full trajectory subset, while tool‑use benefits some models but harms others, revealing large variability in judges’ performance.
By Xing Han L\`u, Dheeraj Vattikonda, Sina Hajimiri, Fatemeh Pesaran Zadeh, Parishad BehnamGhader, Ghazwa Darwiche, Amirhossein Kazemnejad, Christopher Pal, Alexandre Drouin, Siva Reddy
StoreBench is a live‑commerce environment where an autonomous agent runs a mid‑size online apparel store on a production‑grade backend, facing continuous customer orders, supplier price changes, and market shocks. The agent uses the same 29 merchant tools as a human operator, with a windowed operation budget that ties simulated time to actions, ensuring that model latency does not affect the simulation. Seven large language models were evaluated across 11 scenarios of 30–45 days and a full simulated year, but none matched the scripted smart‑triage policy, and human experts outperformed all models.
By Daksh Raghuvanshi, Ved Vedere, Yifan Wang
The paper introduces Keyframe Mnemonics, a self‑supervised approach that identifies critical observations (mnemonics) to aid behavior cloning in non‑Markovian settings. By training a policy conditioned on these keyframes, the method achieves perfect success rates on synthetic memory tasks and improves performance on a real‑robot manipulation benchmark, maintaining high success even at horizons far beyond training. The approach offers theoretical context‑retention guarantees and practical scalability to long‑horizon decision making.
By Prabin Kumar Rath, Omkar Patil, Nakul Gopalan
The paper introduces E‑Ledger, a multi‑agent harness that ensures safe and persistent execution of large language model agents in partially observable environments. It adds a code approval layer to enforce policy compliance and maintains a world ledger of verified hidden rules and dynamic state. To discover unknown rules, the authors propose WorldAbduct, an abductive framework that diagnoses execution trajectories, hypothesizes latent rules, and validates them before integrating them into the ledger, improving safe task completion on enterprise and scientific benchmarks.
By Yisen Gao, Yue Guo, Qing Zong, Yiwen Guo, Yangqiu Song
OpenProblemBench is a new benchmark that presents 82 unresolved problems from mathematics and theoretical physics, each with detailed context, assumptions, and prior progress. The benchmark evaluates AI submissions on correctness, completeness, and progress using four independent evaluator models, without providing reference solutions. GPT‑6‑Astra achieved the highest mean judged solve rate of 14.0%, outperforming other evaluated models, and the study links stronger outcomes to improved problem representation, general reasoning, and proof steps.
By Zhiyi Li, Sihan Hu, Tianning Xiao, Xiansheng Cai, Xiaojun Tan, Youjin Deng, Kun Chen
ReCast is a new method for learning step representations in large language model (LLM) based agent systems, aimed at improving failure attribution by transforming hidden states into attribution‑oriented representations. The approach selects relevant layers, builds pattern and deviation features, and trains an encoder with contrastive and ranking objectives. In experiments on four benchmarks, ReCast outperforms the strongest baseline, achieving higher Hit@1 scores on the Who&When Algorithm and Handcrafted datasets.
By Weilin Jin, Mingyu Wang, Taiyu Zhu, Ziqi Zhou, Wenbo Li, Haoyang Huang, Nan Duan, Yifan Wu, Ying Li, Zhonghai Wu
The paper introduces grasp, a neurosymbolic framework that learns probabilistic logic programs by combining functional gradient boosting with large language models (LLMs). In grasp, the weak learner is a first‑order rule and the combinatorial search for new rules is delegated to an LLM proposal oracle, allowing systematic inductive reasoning. Experiments on four relational benchmarks—including molecular toxicity prediction, mutagenesis, and citation matching—show that grasp outperforms purely symbolic, neural, and LLM‑based baselines while producing interpretable weighted rule ensembles.
By Saurabh Mathur, Sahil Sidheekh, Bhavan Vasu, Farbod Tavakkoli, Prasad Tadepalli, Kristian Kersting, Sriraam Natarajan
The paper introduces SED-MCTS, a Monte Carlo tree search method that enhances PDE solution discovery by distilling structural experience from evaluated expressions. Unlike previous approaches that rely on a single terminal score, SED-MCTS uses counterfactual subtree interventions to estimate local structural contributions, guiding the search toward reliable construction edges and preserving useful components in a structural archive. The method extends naturally to coupled multiphysics systems and demonstrates improved performance, efficiency, and robustness across diverse PDE benchmarks, especially under noisy or scarce observations.
By Yunpeng Gong, Huolong Wu, Can Yang, Min Jiang
MemoWM proposes a memory allocation framework that leverages world models to compress what long‑term agents retain. By using shared predictions, it reconstructs omitted content and balances reconstruction error against storage cost, keeping only information that adds downstream value beyond the predictive prior. On five long‑term agent‑memory benchmarks, MemoWM improves average answer accuracy by 42.42% and cuts experience‑specific storage by 53.9% compared to the most storage‑efficient baseline.
By Bingfan Zeng, Zhisheng Chen, Chenbo Sang, Zhengwei Xie, Jinpeng Wang, Xiangchen Guan, Rui Qian, Zheng Lu, Jingwei Song
The paper investigates whether the HELM Safety benchmark’s HarmBench dataset measures a single attribute—harmful refusal—using psychometric methods. Multidimensional item response theory indicates HarmBench does not capture a single attribute, and differential item functioning reveals that models from different developers score items differently, suggesting domain‑specific effects. The authors conclude that safety scores should only be used for attribute comparison after confirming single‑attribute validity, warning that aggregated scores can mask saturation and conflated behaviors.
By Christopher M. Stewart, Preston Botter, Natalie Sarabosing, Muye Zhang, Rachel Phinnemore, Shalini Ghosh, Hong Shen, Hoda Heidari
The paper introduces ALERT‑BENCH, a benchmark that simulates enterprise security telemetry and evaluates how large‑language‑model agents triage alerts. Five common reasoning strategies—single‑pass tool use, iterative retrieval, sampled investigations, self‑review, and explicit verification—were tested on 1,247 alerts from a multi‑stage attack scenario, each missing at least 40.4% of attack‑related alerts. Building on these findings, the authors propose AIDA, a multi‑agent framework that enforces explicit decision proposals, independent challenges, and stronger evidence requirements, achieving an F1 score of 0.958 and reducing false‑negative alerts from 40.4% to 3.1% while escalating 18.4% to analysts.
By Saimon Amanuel Tsegai (Daphne), Alex Kantchelian (Daphne), Danfeng (Daphne), Yao, Peng Gao
WorldBench is a benchmark and automated judge for evaluating large language models that generate interactive 3D worlds using Three.js. It combines code inspection with dynamic visual analysis, requiring the model’s code to produce observable features such as biomes, physics, and seasonal cycles. The study shows that relying on either code or visual output alone can misjudge a model’s performance, with a mutation test revealing that code-only judging overestimates success by a third.
By Krish Bakshi
The paper introduces CoCo, a conjoint framework for deep graph clustering that emphasizes compactness and consistency in node representations. CoCo uses graph convolutional filters to learn robust embeddings from both local and global views, then encodes them into low‑rank compact embeddings to reduce redundancy and noise. A consistency learning strategy further enriches node semantics, enabling knowledge transfer between perspectives and yielding superior performance over state‑of‑the‑art methods on multiple datasets.
By Wei Ju, Siyu Yi, Kangjie Zheng, Yifan Wang, Ziyue Qiao, Li Shen, Yongdao Zhou, Xiaochun Cao, Jiancheng Lv
SWE-Journey is a new benchmark designed to evaluate coding assistants in more realistic settings, focusing on long-horizon tasks and multi-turn interactions. It introduces a weak-to-strong synthesis pipeline to create extended coding challenges and a user-simulation agent based on four real user personas to mimic authentic code-assistance dialogues. The benchmark reveals that while models succeed on over 75% of tests with software architects, they manage less than 25% with non-coders, highlighting a significant gap in current assistant capabilities.
By Hexuan Deng, Yue Wang, Wenyu Jiang, Cheng Yang, Haolin Yang, Zhaohua Zhang, Chenchen Zhao, Beiduo Chen, Muxi Chen, Sa Zhu, Geyuan Zhu, Jianhuan Zhuo, Qiuyong Xiao, Tianwen Jiang, Jihong Zhang, Xuebo Liu
The paper compares classical motion planners with learning-based methods for real-time navigation in environments with dynamic hazards, using a unified benchmark that includes identical settings and metrics. In deterministic scenarios, classical planners achieve near-perfect success and high-quality paths, though they can incur significant planning time. When obstacle dynamics are stochastic, classical planners become highly budget-sensitive, whereas PPO-trained policies consistently deliver better latency, success rates, and path quality.
By Eran Iceland, Alexander Tuisov, Oren Gal, Ariel Barel, Alfred M. Bruckstein