Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv AI
Oct 26

SemGeo-Gen: Unsupervised Generation of Approximate Cross-Instance Semantic-Geometric Correspondences

SemGeo-Gen is an unsupervised approach that generates approximate cross‑instance semantic‑geometric correspondences from weakly aligned 3D object collections. It uses coarse category‑level rotation alignment, lifts multi‑view DINOv2 features to 3D, applies continuous piecewise‑affine registration, and prunes matches with semantic consistency to produce 3D correspondences that can be projected into 2D views for scalable supervision. The method achieves high accuracy against human annotations and improves state‑of‑the‑art semantic correspondence models after synthetic pretraining.

By Roy Amoyal, Shira Ifergane, Oren Freifeld
arXiv AI
Oct 26

Yalda-Embedding: Open Development of a Competitive Persian Retrieval Model

Yalda-Embedding is a 595.8‑million‑parameter Persian retrieval model built by adapting Qwen3‑Embedding‑0.6B and trained on over one million query‑document pairs. It achieves the highest macro‑average score (0.5488) among 0.6B models on eight FaMTEB Persian retrieval tasks, outperforming Qwen3‑Embedding‑0.6B and F2LLM‑v2‑0.6B on every task and approaching the performance of the larger Qwen3‑Embedding‑4B. The authors also conduct an ablation study on contrastive batch size, optimization stabilization, and relational knowledge distillation, showing that the distilled configuration improves all tasks without increasing inference‑time model size and includes hard‑negative evaluation gains despite no explicit hard‑negative mining.

By Yalda Taheri, Mohammad Hassan Heydari, Armon Rasooli
arXiv AI
Oct 26

Symbolic Guidance for LLM Agents in Distributed Multiagent Coordination

The paper introduces the Symbolic Guidance Taxonomy (SGT) to regulate the autonomy of large language model agents in distributed multi‑agent coordination. By providing symbolic guidance—ranging from open‑ended reasoning to fully prescriptive algorithmic steps—the authors find that intermediate levels of autonomy consistently outperform both unguided agents and fully prescriptive specifications. These results suggest that carefully controlling agent autonomy is a crucial design principle for effective LLM‑based coordination.

By Ben Rachmut, Ning Zhang, Yevgeniy Vorobeychik, William Yeoh
arXiv AI
Oct 26

Bridging Exploration and Persona: Information-Directed Dueling for Test-Time Agent Personalization

The paper introduces ID‑TAP, a neural dueling‑bandit framework that personalizes large‑language‑model agents at test time by selecting personas through binary comparisons of candidate responses. It combines a query‑conditioned neural utility model with a lightweight Bradley‑Terry explorer, and uses similarity‑weighted warm starts to transfer evidence across related queries. Experiments on seven user‑conditioned generation benchmarks show that ID‑TAP outperforms baselines within a ten‑round comparison budget, demonstrating the effectiveness of information‑directed challenger selection for persona adaptation.

By Jie Mao, Pingchen Lu, Xiangyi Wang, Hengquan Huang, Shuang Qiu, Min Zhang, Zhongxiang Dai
arXiv AI
Oct 26

MedAction: Towards Active Multi-turn Clinical Diagnostic LLMs

The paper introduces MedAction, a tree‑structured distillation pipeline that generates high‑quality multi‑turn diagnostic trajectories by interacting an LLM with an environment. It identifies three failure modes in current LLMs—ungrounded test ordering, unreliable diagnostic updates, and degraded multi‑turn coherence—and proposes two knowledge‑graph‑grounded metrics, Disease Trajectory Consistency and Reasoning‑Action Consistency, to filter trajectory quality. Using this pipeline, the authors create MedAction‑32K, a dataset of 32,681 trajectories from 2,896 PMC cases, and fine‑tune an 8B model that achieves state‑of‑the‑art performance on MedR‑Bench and the MedAction‑300‑Hard benchmark among open‑source medical LLMs.

By Hsin-Ling Hsu, Zizheng Wang, Donghua Zhang, Nai-Chia Chen, Jerry Wang, Jun-En Ding, Chia-Hsuan Hsu, Guoan Wang, Feng Liu, Fang-Ming Hung, Chenwei Wu, Liyue Shen
arXiv Computer Vision
Oct 26

Tempo-Zero: Recursive Self-Play in Executable Worlds for Video Reasoning

Tempo‑Zero is a self‑improvement framework for video reasoning that uses recursive self‑play in executable environments. It closes a loop of environment execution, self‑play verification, and environment evolution, allowing a single policy to act as both questioner and solver and to receive execution‑grounded rewards. The approach trains only on simple programs, without external video data or supervised fine‑tuning, and achieves superior performance on twelve real‑world video benchmarks, especially on temporal grounding tasks.

By Jingmin Zhu, Yanyan Zou, Xiuyu Li, James Bailey, Dinh Phung, Tao Yu, Yunyun Hou, Jun Liu, Jun Zhou, Sirui Wang, Nan Duan, Jiachen Liu, Qiuhong Ke
arXiv Computer Vision
Oct 26

OmniSpace: Efficient Geometry Awareness for Autonomous Vehicles MLLMs

OmniSpace is a plug‑and‑play framework that enhances multimodal large language models (MLLMs) with geometry awareness using only 2D observations. It introduces a Camera Pose Injector, a Multi‑view Epipolar Attention module, and a 3D Geometric Distillation objective to improve cross‑view correspondence and depth estimation. Experiments demonstrate that OmniSpace outperforms existing methods on planning, risk detection, language, and generalization benchmarks for autonomous vehicles.

By Hao Vo, Phu Loc Nguyen, Khoa Vo, Sieu Tran, Duc Minh Nguyen, Ngo Xuan Cuong, Nghi D. Q. Bui, Anh Nguyen, Duy Minh Ho Nguyen, Ngan Le
arXiv AI
Oct 26

POSEIDON: Scaling Training Environments for Scientific Agents from Code Repositories

POSEIDON is an automated pipeline that generates diverse, verifiable training environments from scientific code repositories, scaling up the number of environments by nearly tenfold compared to previous methods. It decomposes scientific workflows into four task types—Reproduction, Subflow, Completion, and Correction—to target distinct scientific research capabilities, and uses a recursive improvement loop to refine and recover environments. Using Poseidon, the authors created Pearl, a collection of 97,659 executable and verifiable environments from 7,110 public repositories across ten scientific disciplines, the largest multidisciplinary set to date, which enabled Qwen3.5 models to achieve consistent gains on six scientific-agent benchmarks.

By Qifan Zhang, Yuhan Li, Nuo Chen, Bowen Liu, Xiao Hu, Miao Peng, Chenyi Zi, Lixin Liu, Lydell Huang, Jing Tang, Jia Li
arXiv AI
Oct 26

GeoDiv KAN Solver for Time Fractional Navier Stokes Equations on Complex Domains to Realistic CFD Reconstruction

The paper introduces GeoDiv KAN, a geometry‑gated, divergence‑free Gaussian radial basis function Kolmogorov Arnold network designed to solve time fractional Navier–Stokes equations on complex domains. By incorporating geometry‑dependent feature gating and a streamfunction formulation, the architecture preserves incompressibility and achieves higher predictive accuracy than existing PINN, PIKAN, RBF KAN, and DF RBFKAN models on square, circular, and star‑shaped benchmarks. GeoDiv KAN is further validated on two data‑driven CFD cases: unsteady flow past a circular cylinder and turbulent RANS flow through a complex U‑bend.

By Yatin Kumar, Shivani Saini, Ramesh Kumar Vats
arXiv Computer Vision
Oct 26

VTRL-Med: Progressive Visual Refinement with Topology-Guided Regularization for Medical Image Clustering

VTRL-Med is a progressive visual‑structural learning framework that incorporates topology‑guided regularization for medical image clustering. It adapts the Segment Anything Model to preserve structural cues, generates pseudo‑labels via prototype selection and k‑NN consistency, and refines visual features while an auxiliary topology branch—built from persistent homology descriptors—provides geometric regularization. The method achieves state‑of‑the‑art clustering accuracy, improving average performance by 4.48% over prior approaches and further boosting accuracy by 1.3–4.3% with topology guidance.

By Guangyu Meng, Pengfei Gu, Peixian Liang, John P. Lalor, Erin Wolf Chambers, Danny Z. Chen
arXiv AI
Oct 26

Beyond Post-Hoc Rewards: Privileged Reasoning Exploration for Multimodal Embeddings

The paper introduces PILOT, a framework that treats reasoning as a latent variable to enhance multimodal embeddings. By modeling prior and posterior reasoning modes with a shared large language model, PILOT uses relational supervision during training to guide exploration of retrieval‑relevant reasoning traces. On the MMEB‑V2 benchmark, PILOT outperforms baselines across all task groups, notably improving visual‑document retrieval scores by over six points.

By Tianyu Yang, Yijun Tian, Lars Benedikt Kaesberg, Jan Philip Wahle, Bela Gipp, Terry Ruas, Daniel Kurzawe
arXiv AI
Oct 26

Do "True" and "False" Add Up to One? Complement Consistency in LLM-Agent Opinion Simulations

The paper investigates whether large language model (LLM) agents maintain complementary consistency when reporting probabilities for a claim and its negation in opinion simulations. Using three small open models (Qwen3‑4B, Llama‑3.2‑3B, Gemma‑3‑4B), the authors find that paired reports often sum to more than one, even on real fact‑checking claims, and that this inconsistency persists across different simulation settings. Averaging a report with its complement restores consistency but only marginally improves predictive accuracy for one model.

By Anouar Imel, Jin Qi
arXiv AI
Oct 26

Closing the Modality Gap via Simplex-Constrained Representations

The paper proposes replacing the unit-sphere embedding space used by multimodal contrastive models with the probability simplex. By adding lightweight softmax adapters and measuring similarity with Total Variation, the authors reduce the modality gap by 97–99% on five retrieval benchmarks, showing that simplicial geometry can align modalities as well as or better than unconstrained models.

By Shubham Gupta, Siva Reddy, Perouz Taslakian, Valentina Zantedeschi, Cem Subakan
arXiv Computer Vision
Oct 26

SCDM-HTD: Spectral-Wise Conditional Diffusion Model for Hyperspectral Target Detection

The paper introduces SCDM-HTD, a spectral‑wise conditional diffusion model for hyperspectral target detection that reframes detection as generating a confidence map. It treats background and target confidence scores as training samples, guiding the diffusion process with pixel spectra to learn distributional differences between background and target. A lightweight MLP denoising network and a deterministic reverse chain accelerate inference, while a composite loss combining constrained energy minimization and binary cross‑entropy improves background‑target separation, yielding superior results on four benchmark datasets.

By Dunbin Shen, Wenfeng Kong, Jianjun Liu, Xiaorui Ma, Hongyu Wang
arXiv AI
Oct 26

Spectra: Multi-Scale Implicit Decomposition for Deepfake Detection

Spectra is a deepfake detection framework that uses implicit neural representations (INRs) to analyze how an image is reconstructed, producing multi-level reconstruction heatmaps. These heatmaps are fused with semantic visual features via a bidirectional mechanism, enabling the model to detect forgery cues that are less tied to specific generation pipelines. Experiments on several deepfake benchmarks show that Spectra outperforms detectors relying only on RGB appearance or semantic representations, and it also extends to face presentation attack detection.

By Dhananjaya Jayasundara, Kartik Narayan, Vishal M. Patel
arXiv AI
Oct 26

Cycle-of-Thought: Self-Evaluative Iterative Reasoning for Table Visual Question Answering in Document Images

The paper introduces Self-RefineVQA, a training‑free, inference‑time framework that lets a single Vision‑Language Model iteratively generate, critique, and refine its own answers for table visual question answering. By forming a closed self‑feedback loop guided by prompts, the method corrects common reasoning errors such as spatial misalignment and logical hallucinations without requiring additional labeled data or model updates. Experiments on tabular VQA benchmarks show consistent accuracy gains across monolingual and cross‑lingual settings, especially for lightweight models, while keeping inference cost linear and eliminating the need for expensive supervised fine‑tuning.

By Somraj Gautam, Gaurav Harit
arXiv AI
Oct 26

Mitigating Saliency Collapse: Robust Saliency-Aware Long-Text Image-Text Alignment

The paper introduces RoSA, a saliency-aware fine‑tuning framework for long‑text image‑text alignment that addresses the failure mode of saliency collapse. RoSA decomposes image‑text pairs into salient and contextual views using a modality‑independent strategy that combines visual saliency estimation with LLM‑based text decomposition, and employs auxiliary objectives to align these views. The method improves long‑text retrieval performance and robustness to saliency collapse, and its auxiliary objectives can be added to existing tuning methods for consistent gains.

By Qiuyu Kong, Zanxi Ruan, Marco Cristani, Yiming Wang
arXiv AI
Oct 26

Do AI-for-Science Agents Preserve Model Safety?

AI-for-Science agents, which combine large language models (LLMs) with domain-specific harnesses to formulate hypotheses, run experiments, and interpret evidence, may compromise model safety. The study evaluated five harnesses on 50 harmful chemistry requests, finding that harnesses can increase harmful compliance even with direct prompting and that adaptive attacks that fail against the base LLM succeed once the harness is added. These results indicate that the safety behavior of the underlying LLM is not durable when integrated into an agent harness.

By Michael Yew Chuan Lee, Manya Sriram, N. Benjamin Erichson