CompLit is a new literature search benchmark covering all eight arXiv domains, designed to evaluate agents on three diagnostic settings: implicit, cumulative, and unmet needs. The benchmark shows that top agents like Codex and Claude Code perform well on standard queries (86‑87%) but drop to 66‑68% on implicit queries, lose additional accuracy when requirements accumulate, and abstain on up to 53% of unmet queries. The study finds that agents often locate the correct or near‑correct paper but misjudge it, indicating a need for more reliable search capabilities.
By Dayoon Ko, Jihyuk Kim, Soyeong Jeong, Young-jun Lee, Dahyun Lee, Juyeon Kim, Gunhee Kim, Moontae Lee, Kyungjae Lee
Kongzi is a historical large language model that enhances factual accuracy through continued pre‑training on historical corpora, supervised fine‑tuning, and fact‑enhanced reinforcement learning. Its reward system compares entities and relations in generated responses with historical references via knowledge‑graph similarity. Evaluations on 0.5B and 7B models show that Kongzi‑7B achieves the highest historical‑accuracy ratings among compared models and that reinforcement learning improves answer scores and CMMLU‑Chinese‑History accuracy, though it still falls short of the strongest baselines.
By Jiashu Yang, Ningning Wang, Yian Zhao, Chaoran Feng, Junjia Du, Hao Pang, Miao Yu, Xuxin Cheng
LoopLook introduces a lightweight encoder-side recurrent perception framework that equips pretrained Vision‑Language Models with adaptive visual depth. By repeatedly revisiting a middle‑to‑late window of the vision encoder and injecting question representations at each loop, it refines visual evidence in a task‑directed manner. The learned stopping mechanism and token‑wise Mixture‑of‑Looks allow each visual token to selectively integrate information across loops, yielding consistent performance gains across multiple backbones and benchmarks.
By Wencheng Ye, Anning Hu, Hongye Fang, Tianyi Wang, Junchi Yan
The paper introduces Video Attribute-Based Complexity (VideoABC), a non‑parametric framework that estimates the likelihood of failure for video‑LLMs on video‑question pairs. VideoABC projects reference videos into a space defined by a pre‑defined vocabulary of video attributes (e.g., scene complexity, event speed), quantizes this space, and computes expected complexity values for each quantization cell. To handle small reference datasets, it combines a k‑means quantizer for in‑distribution samples with a universal lattice quantizer for out‑of‑distribution generalization, and uses a synthetic video generation procedure to populate lattice cells during training. Experiments demonstrate that VideoABC outperforms methods like ‘video‑LLM as judge’ while requiring fewer resources, and its attribute‑based scores provide interpretable insights into benchmark complexity.
By Aditya Sarkar, Yi Li, Zihao Wang, Hao Yuan, Sai Vidyaranya Nuthalapati, Aashu Singh, Jiacheng Cheng, Ang Li, Shlok Kumar Mishra, David Jacobs, Nuno Vasconcelos
DeepLook is a training‑free decoding framework that uses segment‑level confidence to guide branching during inference. It only expands the search tree at points of high uncertainty and ranks candidate continuations by Average Lookahead Confidence (ALC), a mean confidence over fixed‑horizon rollouts, before finalizing and voting on a pruned subset. Experiments on four mathematics benchmarks show that DeepLook matches or surpasses DeepConf‑low in 12 of 16 settings while reducing token usage by 87.3%, though its advantage disappears on the knowledge‑intensive GPQA‑Diamond task due to hallucinations receiving high ALC.
By Tingxin Yang, Zefeng Wang, Mengyue Wang, Jinhe Bi, Xingcheng Zhou, Volker Tresp, Yunpu Ma
The paper investigates how compact an encoder can be for closed‑set decision tasks in language‑model pipelines, comparing joint versus per‑option scoring. In a 32‑million‑parameter study, per‑option scoring outperforms joint scoring by 5.8 chance‑corrected skill points on a 37‑benchmark suite, and two external benchmarks corroborate this advantage. Across six released joint encoders ranging from 17 million to 1 billion parameters, the 68‑ and 150‑million‑parameter models surpass a 435‑million‑parameter DeBERTa‑v3 zero‑shot classifier on 209 overlapping tasks and match it on 145 unseen tasks, while the smallest model processes a median request in 14 ms versus 728 ms for the classifier.
By Tardelli Ronan Coelho Stekel
The paper investigates whether large language models’ refusal behavior in text transfers to tool‑calling contexts. A benchmark of 2,304 prompts across four domains compares text‑only, standard tool‑enabled, and adversarially framed tool‑enabled settings, revealing that 10–52 % of harmful requests refused in chat still lead to unsafe tool calls. The authors identify a refusal‑associated direction in the models’ residual stream, showing that lower activation along this direction predicts unsafe calls, but restoring the signal only partially mitigates the issue and often suppresses benign tool use.
By Ekagra Gupta, Raj Saha, Aniekan-Abasi Inyang, Adam Al Gharib
The paper introduces CiteResolve, a benchmark of 423 real-world hallucinated citations annotated for source recoverability, and CiteAnchor, a source-aware grounding framework that adapts retrieval and field-level verification to each source type. CiteAnchor determines citation validity and whether a unique source can be recovered, achieving 0.841 F1 on an existing detection benchmark and 74.7% end-to-end success on CiteResolve, outperforming baselines by significant margins. The authors provide code and data at their GitHub repository.
By Chenyi Tong, Ping Nie, Yu Zhang
The paper introduces Random Remask-Denoise (R$^2$D) inference, a test‑time, finetuning‑free method for masked diffusion models that randomly re‑introduces masks and denoises to form a Markov chain on clean sequences. It also proposes Trajectory Correction Training (TCT), which trains the denoiser by exposing it to its own errors during generation. Experiments show that R$^2$D with TCT achieves near‑log‑linear scaling, achieving over 97% accuracy on hard Sudoku puzzles, over 99% on 3‑SAT, and surpasses iterative correction baselines on language tasks such as HumanEval+.
By Zhaoyi Zhou, Frank Wu, Andrea Zanette
The paper introduces a benchmark that demonstrates that large language models (LLMs) fail to consistently retrieve the same chemical facts when presented with different co‑referring identifiers such as trivial names, IUPAC names, or SMILES. Across models ranging from 7B to 72B parameters, including chemistry‑specialized ones, the authors find that knowledge is not uniformly bound to each identifier, a failure they call cross‑identifier binding failure. They further analyze internal representations and propose a mitigation called Binding Self‑Distillation that transfers knowledge from name‑based to structure‑based routes, enabling direct factual retrieval from SMILES without retraining the model.
By Khouloud Saadi, Di Wang
arXiv:2505. 11109v2 Announce Type: replace-cross Abstract: We present the first large-scale open-set benchmark for multilingual audio-video deepfake detection.
By Florinel-Alin Croitoru, Vlad Hondru, Marius Popescu, Radu Tudor Ionescu, Fahad Shahbaz Khan, Mubarak Shah
MIG‑COW is a new adversarial attack framework that uses momentum‑integrated gradients and a consensus‑orthogonal decomposition to create highly generalizable, visually imperceptible perturbations against deepfake detectors. It achieves a 99.96% white‑box attack success rate on the AADD‑2025 Challenge benchmarks while maintaining high perceptual similarity (SSIM). In black‑box settings, its success rate drops to 7.16%, and adding low‑performing but diverse models to the ensemble can reduce attack effectiveness, highlighting the importance of careful surrogate model selection.
By Wonjune Seo, Joonhyuk Baek, Yeseong Jung, Saerom Park
The paper introduces Diverse Parallel Thinking (DPT), a reinforcement learning approach that encourages path diversity in parallel reasoning for large language models. DPT penalizes prefix similarity among parallel paths, fostering distinct reasoning strategies and improving exploration of the reasoning space. Experiments on multiple benchmarks show that DPT outperforms prior parallel thinking methods, achieving state‑of‑the‑art results.
By Jiaru Qian, Yihong Dong, Xue Jiang, Chun-Wah Hsu, Zhi Jin, Ge Li
WhatIf-Bench is a semi‑synthetic benchmark that tests how well time‑series forecasters use covariates to predict a target’s response to future events. It generates 37,665 tasks from mechanistic simulators of data centers, power grids, road traffic, and water networks, varying covariate numbers, historical events, and event length. Evaluation shows large‑language models outperform the best time‑series foundation models, with all models performing well on replayed events but struggling on novel covariate settings, highlighting generalization as the main challenge.
By Arjun Ashok, Mihir Parmar, Palash Goyal, Bhavana Dalvi Mishra, Chun-Liang Li, Irina Rish, Alexandre Drouin, Jinsung Yoon, Tomas Pfister
The paper introduces TWICE (TWIn-world Credit Estimation), a method for agentic reinforcement learning that uses twin-world counterfactuals to assign credit to individual decisions within an episode. By re‑running episodes with the same sampling noise but different decisions, TWICE produces unbiased turn‑level advantages for policy‑gradient updates, reducing the number of rollouts needed compared to independent counterfactuals. Experiments on seven question‑answering benchmarks show that TWICE improves exact match scores for GRPO by 5.3 and 4.5 points on Qwen2.5‑3B and 7B, outperforms tree‑search and counterfactual‑branching baselines, and requires 68% fewer auxiliary tokens than VinePPO.
By Zeropy Surio, Tobias Engel, Mei-Lin Chou, Anika Raman, Samuel Okafor
The article reports on a study comparing Jev, a decision language model that provides structured judgments, with large language model (LLM) judges for evaluating clinical AI outputs. Across multiple use cases—including pediatric answer quality, binary HealthBench criteria, MedHELM open‑ended tasks, and HealthBench Professional—Jev matched or exceeded the performance of several frontier and open‑weight LLM judges. Moreover, Jev was significantly cheaper and faster, costing 73–294 times less and taking 7–57 times less time than frontier judges, suggesting it could enable efficient, scalable clinical AI evaluation.
By Rotem Gershon, Seffi Cohen, Dean Ariel, Havatzelet Bilavsky-Yarden, Amiel Sberro, Shai Yitzhaki, Osnat Tausky, Naama Golan, Maayan Rabinovitch, Ariel Avraham Hasidim, Itay Arianu, Maya Dagan, Ran Balicer, Noa Dagan
StyleStream 2.0 is a fast, controllable streaming voice style conversion system that reduces end‑to‑end latency to about 520 ms on a consumer GPU, halving the delay of its predecessor while preserving conversion quality. It replaces mel‑spectrogram context with a compact style embedding, enabling a unified text‑conditioned flow model that supports both text‑based voice design and instruction‑based voice editing. Experiments demonstrate that StyleStream 2.0 achieves superior target style fidelity, voice design, and voice editing performance while maintaining intelligibility and attribute preservation.
By Yisi Liu, Nicholas Lee, Gopala K. Anumanchipalli
The paper introduces MeMoDecode, a decoding framework that enforces both concrete and abstract syntax as well as well‑formedness constraints for domain‑specific languages (DSLs) during large language model generation. By mapping generated text to partial metamodel instances and retrying from the last consistent snapshot, MeMoDecode guarantees consistency without sacrificing alignment metrics. Experiments on four DSLs and six open‑weight LLMs show a 23% average improvement over unconstrained decoding and an 8% improvement over grammar‑constrained decoding, with modest runtime overhead.
By Boqi Chen, Aren A. Babikian, Jos\'e Antonio Hern\'andez L\'opez
The paper reports the first empirical, phase‑decomposed benchmark of handshake overhead in three major AI agent communication protocols—MCP (2025 and 2026 versions), A2A, and ACP—using their official SDKs. It measures round‑trip counts, per‑phase latency, wire bytes, and token cost across cold‑start, warm‑repeat, capability‑mismatch, and degraded‑metadata scenarios, finding that round‑trip count dominates cost (e.g., MCP’s legacy handshake requires three round trips and 230 ms at 50 ms RTT, while A2A needs one round trip and 61 ms). The study also shows that handshake cost scales with the counterpart’s capability inventory, with a 200‑capability counterpart consuming 11 % of a 200k‑token context window, and documents protocol evolution that reduces MCP’s handshake from three to zero round trips on warm reconnects.
By Pranav Varshney
KinoUAV introduces a kinodynamic benchmark for UAV vision-language navigation, featuring a dataset and an agent that generate trajectories under velocity, acceleration, and jerk constraints. The benchmark incorporates directional free-space distance labels to guide safe motion decisions and a Cruise-Seek task that eases target search. Experiments show the KinoUAV-Agent achieves higher success rates with lower yaw and collision rates than existing methods.
By Chenyao Yang, Peijin Wang, Yingchao Feng, Wenhui Diao