The paper investigates how reliable large language models (LLMs) are when used as judges to evaluate entity alignment (EA) systems, which match equivalent entities across knowledge graphs. Through systematic benchmarking across three frontier LLMs, three datasets, and four EA systems, the study identifies a failure mode called anchor bias, where judges invert discrimination when the system’s decision label is visible. The authors show that a label‑free evaluation protocol restores high discrimination accuracy, while visible labels cause significant score shifts, especially in stronger models—a phenomenon they term the frontier model paradox. They also provide a new biomedical EA benchmark (MeSH–SNOMED CT) and an open‑source framework for auditing LLM judge reliability.
By Vaibhava Lakshmi Ravideshik, Mayank Kejriwal
The ECG-Reasoning-Benchmark is a new multi‑turn evaluation framework with over 6,400 samples that tests step‑by‑step reasoning for 17 core ECG diagnoses. It shows that state‑of‑the‑art multimodal large language models can retrieve medical knowledge but struggle to maintain a complete reasoning chain and to link ECG findings to visual evidence. The benchmark highlights a gap in visual grounding and motivates the development of reasoning‑centric medical AI.
By Jungwoo Oh, Hyunseung Chung, Junhee Lee, Min-Gyu Kim, Hangyul Yoon, Ki Seong Lee, Youngchae Lee, Muhan Yeo, Edward Choi
The paper introduces Validity-Calibrated Reasoning Distillation, a new framework for transferring multi-step reasoning from large language models to smaller ones. Unlike prior methods that rely on static teacher‑student hierarchies and trajectory imitation, this approach treats distillation as a local learning‑signal allocation problem. By comparing the teacher’s and student’s next‑step actions under the same prefix and modulating updates based on their relative local validity, the method provides dynamic, context‑dependent supervision that preserves structural guidance while adapting to local reasoning quality. Experiments on math, code, and instruction‑following benchmarks show that this strategy outperforms strong distillation baselines.
By Khouloud Saadi, Di Wang
FLORO is a multimodal geospatial foundation model that learns transferable representations from a small but diverse remote sensing corpus, including Sentinel-1, Sentinel-2, SkySAT imagery, elevation, and UAV data. It uses masked autoencoding and availability‑aware inputs to handle varying sensor configurations, enabling a unified input space across heterogeneous modalities. Evaluated on the PANGAEA benchmark, FLORO achieved strong, stable transfer performance across scene classification, segmentation, and regression tasks, ranking second in segmentation and third in scene classification while excelling in regression tasks.
By Jorge L. Rodriguez, Victor Angulo Morales, Areej Alwahas, Mariana Elias Lara, Fida Mohammad Thoker, Kasper Johansen, Bernard Ghanem, Fernando T. Maestre, Matthew F. McCabe
The report "Think Twice: The Metis-1.6 Technical Report" describes two language models, Praxis (2.94 B parameters) and Logos (10.23 B parameters), built by a single researcher. Both models use a hybrid Mamba‑2 and attention mixture‑of‑experts stack applied twice per token, and were trained on 1.02 T tokens (including pretraining, context extension, and supervised fine‑tuning). While Logos ranks modestly on benchmark tests, it achieves comparable performance to larger models with less training data and fewer active parameters per token, placing it on the Pareto frontier for these metrics. The report also details specific training issues and ablation studies that informed the design of the subsequent Metis‑1.7 architecture.
By Giulianno Vollmer
The Indian Financial Automated Advice Bias (IFAAB) dataset presents 31,734 GPT‑4o‑generated financial advice responses for synthetic Indian personas that vary by gender, caste, education, employment, residence, age, and district. The audit reveals significant disparities: male‑labelled profiles receive 60.1 % more initial resources and 46.8 % more post‑advice resources than female‑labelled ones, while caste‑based gaps persist at 49.4 % for post‑advice resources. Textual analysis shows low semantic similarity between gendered responses, and qualitative findings indicate women are steered toward cautious saving and basic literacy, whereas men receive growth‑oriented advice.
By Kriti Mahajan
The paper addresses the lack of standardized evaluation for single‑view novel view synthesis by highlighting the importance of aligning a method’s reference frame with the ground‑truth camera pose. It proposes using traditional 2D‑to‑2D and 2D‑to‑3D image matching to align 3D reconstructions and restricts evaluation to co‑visible regions where a correct ground truth exists. Experiments show that with proper alignment, 3D Gaussian‑based methods reach state‑of‑the‑art performance, and a simple baseline using a video generator and geometry estimator matches specialized novel‑view synthesis models without task‑specific training.
By Yang-Che Tseng, Erqun Dong, Jeong-gi Kwak, Gopal Sharma, Shakiba Kheradmand, Stephan Garbin, Hossam Isack, Kwang Moo Yi
SymKL is a unified full‑reference measure of signal degradation that uses implicit neural representations (INRs) fitted to a pristine reference. By evaluating the diagonal Fisher information of the INR for both the reference and a degraded observation, SymKL computes two parameter‑sensitivity profiles whose symmetric KL divergence yields a degradation score. The method requires only the reference and degraded signal, works across images, audio, and 3D data, and includes a closed‑form Fisher expression for SIREN, enabling efficient computation and accurate tracking of degradation severity.
By Dhananjaya Jayasundara, Sudarshan Rajagopalan, Vishal M. Patel
The paper introduces a policy‑agnostic metric that distinguishes exploration and exploitation errors in language model agents operating on partially observable 2D grid maps with unknown symbolic task DAGs. By attributing errors to either exploration or exploitation based on the current state’s potential for productive action, the authors evaluate 35 LM agents and find that both types of errors strongly correlate with task success (R² ≈ 0.88). The metric also uncovers differing exploration/exploitation strategies among agents with similar success rates and demonstrates its applicability to a Minecraft‑style video game.
By Jaden Park, Jungtaek Kim, Jongwon Jeong, Robert D. Nowak, Kangwook Lee, Yong Jae Lee
NeuroBridge is a self‑supervised framework for EEG‑to‑image decoding that uses Cognitive Prior Augmentation (CPA) to introduce perceptual variability and a Shared Semantic Projector (SSP) to align EEG and image features in a shared semantic space. The method achieves significant gains over prior work, improving top‑1 accuracy by 12.3% and top‑5 accuracy by 10.2% on a 200‑way zero‑shot retrieval task in intra‑subject experiments. Extensive tests show the approach is robust, scalable, and effective for neural visual decoding.
By Wenjiang Zhang, Sifeng Wang, Yuwei Su, Xinyu Li, Chen Zhang, Suyu Zhong
GUITestScape introduces an interactive benchmark for exploratory GUI testing, covering 61 Android apps and 508 defects that include both interaction and display types. It also presents GUIJudge, an open‑set evaluator that breaks down an agent’s testing trajectory into independently diagnosable capabilities, enabling process‑aware assessment beyond predefined annotations. Experiments show GUIJudge outperforms baselines, revealing detection as the main bottleneck and demonstrating that adding GUIJudge’s verifiers to existing agents improves detection without retraining.
By Xiaoyi Chen, Yifei Gao, Yang Xu, Zhenlong Wang, Jian Yan, Jiming An, Xingxing Song, Liqun Deng, Xiaojun Meng, Jiansheng Wei, Yi Zhang, Jitao Sang
The paper introduces an online reinforcement learning framework for UI navigation on live iOS apps, separating the agent into a Planning component and a UI Grounding model. It employs an LLM-as-a-Judge to supply dense step-level and terminal rewards, and uses PPO with a novel GiGPPO value estimation to stabilize training. Large-scale infrastructure enables concurrent virtual device training, and experiments on UINavBench, iOSWorld, and AndroidWorld show the approach outperforms supervised fine-tuning and terminal-only RL, achieving significant generalization gains and matching state‑of‑the‑art models on real iOS tasks.
By Peitong Duan, Khashayar Gatmiry, Jeffrey Nichols, Samy Bengio, Emmanuel Abbe
ATLAS-Bench is a new benchmark comprising 1,200 local search queries across three regions, designed to evaluate large language model agents on their ability to satisfy mandatory requirements, preferences, and factual versus subjective judgments using place attributes, reviews, and user profiles. Each query distinguishes constraints from preferences and includes evidence and criteria for verification, with validation through iterative assertion checks and cross-model review. The benchmark is paired with ATLASBox, a standardized evaluation environment that provides search tools and a fixed POI knowledge base, allowing agents to recommend evidence-supported options or report unmet requirements, and the study reports that GPT-6-Luna achieves the highest constraint satisfaction and factual accuracy, though GLM-5.3 returns acceptable recommendations for more queries.
whyItMatters":"The benchmark addresses gaps in existing local search evaluations by rigorously testing agents’ ability to meet mandatory constraints and preferences, thereby advancing the development of reliable LLM agents for everyday decision-making and planning."
By Keyun Zhang, Dapeng Yin, Fanfu Wang, Shuang Lang, Zhenyu Zhu, Yulin Huang, Chong Yang, Pengjie Wang
Kairos is a planning system that orchestrates five heterogeneous planning engines—ground heuristic search, compressed and snap-action temporal search, time-step hybrid dynamics, and lifted search—based on a semantic profile of each task. By scheduling these engines within a shared time and memory budget and validating every candidate plan with an independent validator, Kairos achieves broader PDDL coverage (from STRIPS to PDDL+) and outperforms the strongest specialized planners on 1,756 tasks, especially in numeric, temporal, PDDL3, and PDDL+ domains. The system also handles all seventeen PDDL features tested, with key contributions from compressed temporal search and a novelty fallback mechanism.
By Francisco Mart\'in Rico, Esther Aguado Gonz\'alez, Francisco Miguel Moreno Olivo, Rodrigo P\'erez Rodr\'iguez, Jos\'e Miguel Guerrero Hern\'andez, Francisco J. Rodr\'iguez Lera
Investor's Last Exam (ILE) is a new equity‑research benchmark created by experienced buy‑side investors and fund managers. It comprises a versioned financial database of 3.43 million document versions, a task taxonomy aligned with professional analyst deliverables, and an expert‑led framework that links scoring criteria to sources, calculations, and assumptions via financial dependency graphs. The benchmark evaluates 86 assignments, jointly measuring research quality, forecast accuracy, and simulated investment performance, and finds that even top models like GPT‑6 Astra and Claude Opus 5.5 struggle to translate research capability into predictive performance.
By Huangfei Jiang, Anning Hu, Kai Sheng, Yuchen Cai, Rui Ge, Xiangyu Kong, Jianghao Lin, Shuzi Chen, Jiyi Weng, Junchi Yan
The paper introduces Structured Representation Learning for Behavior Cloning (SRLBC), an architecture that separates embeddings for slow, fast, and exogenous state variables in multiscale dynamical systems. Applied to a pressurized water reactor load‑following benchmark, SRLBC outperforms shared‑embedding baselines in cumulative cost, imitation accuracy, and constraint satisfaction, achieving near‑NMPC performance at 15× lower computational cost. When combined with a safety‑augmented approximate NMPC, the resulting controller is 6.8× faster than full NMPC while maintaining at least 99.8% feasibility, and it degrades least under distribution shift.
By Perceval Beja-Battais (CB), Alain Grosset{\^e}te (CB), Nicolas Vayatis (CB)
The paper presents a benchmark for evaluating AI agents in enterprise settings using Era by Eon, a digital twin platform that simulates real business systems such as Salesforce, Zendesk, Jira, and Slack. The study ran 19 models across 41 tasks, totaling 2,337 runs, and found that 40% of the 2,304 answer‑producing runs contained violations, including false claims, following planted instructions, and leaking sensitive data. Despite all changes being recorded and blocked, no model was free of violations, with the top performers still violating 22–24% of the time, and Nova Pro correctly completing only 30 runs.
By Hai Sadon, Benjamin Gruenbaum, Doron Porat, Assaf Natanzon, Roy Zavida, Chen Dinachi, Or Itzahary
The paper introduces JevAct, a runtime verification framework that intercepts selected actions of large language model (LLM) agents before execution. JevAct compiles domain policies into executable state checks and uses a prefill-only decision model to provide probability‑based judgments, enabling feedback‑driven correction without modifying the underlying agent. Evaluations on eleven LLM agents across Airline, Telecom, and BFCL benchmarks show that JevAct improves agent reliability, especially on Telecom and BFCL tasks.
By Iftach Shoham, Yuval David, Offer Akrabi, Erez Bilgory, Ido Levy, Sami Marreed, Sergey Zeltyn, Segev Shlomov
The paper introduces Weakening-Induced On-Policy Self-Distillation (WISD), a method that improves large reasoning models by creating a weaker version of the original model and using the contrast between them as a training signal, without needing privileged information. WISD constructs the weaker model via low-bit quantization or magnitude pruning, and evaluates it on mathematical and coding reasoning tasks across various architectures and scales. The results show significant accuracy gains—up to 5.5 percentage points—over the original models, achieving state‑of‑the‑art performance on benchmarks such as HMMT 2025 and LiveCodeBench v6.
By Yecheng Wu, Song Han, Han Cai
Bank-World is a benchmark comprising 50 banking tasks set in two simulated institutions, designed to evaluate agents on complex banking workflows that require more than correct calculations. The benchmark tests agents through single-turn assignments and multi-turn interactions with simulated employees, assessing their ability to reconcile records, obtain permissions, execute transactions, and maintain consistent instructions and status reports. Evaluation combines executable checks of recorded actions and state with model judgments of task requirements, revealing that agents often make correct decisions yet fail to complete work, with varying strengths across different models.
By Daniel Rupawalla