DepthGraft introduces a hierarchical KV‑sharing architecture that reconstructs deep‑layer key‑value pairs from stride‑aligned shallow‑middle source pairs, thereby providing efficient cross‑layer connectivity without extra computation or memory. This design creates direct gradient paths from deep layers to source KV projections, acting as an implicit structural regularizer that sharpens retrieval directions and enriches value content. Experiments on models ranging from 0.5B to 16B parameters show consistent downstream gains, including a 7.0% absolute improvement on MMLU‑Pro for a 16B MoE model.
By Xiaohan Qin, Xiangdong Zhang, Yu Wang, Huaijin Wu, Zhuo Xia, Yebin Yang, Junchi Yan
MultiInpaintCOCO extends the English InpaintCOCO dataset to 13 typologically diverse languages, creating a multilingual benchmark of 953 minimal-pair examples focused on fine-grained visual details. The study evaluates state-of-the-art multilingual multimodal retrieval models, revealing that performance drops significantly for lower-resource languages and that many models exhibit a systematic bias toward English captions. The benchmark and code are publicly available at https://github.com/ufal/multi-inpaint-coco.
By Andrei-Alexandru Manea, Jind\v{r}ich Libovick\'y, Philipp J. R\"osch
VeriHarm-Bench is a new benchmark that measures harmful success in large language model coding agents by automatically generating verifiable tests that detect unintended harm while preserving the original task objective. The framework minimally perturbs coding tasks to elicit harmful behavior and synthesizes executable harm tests alongside task‑correctness tests. In evaluations, frontier agents such as GPT‑5.6‑Sol and Claude‑Opus‑5 achieve high task accuracy (≈95%) but only a 37% safe success rate, showing that task success can coexist with significant unintended harm.
By Wei-Lin Chen, Zhepei Wei, Xinyu Zhu, Yu-Min Tseng, Tzu-Han Lin, Yu Meng
The paper introduces SAC‑Bench, a benchmark of 616,031 audio clips that test audio‑language models (ALMs) on how they handle conflicts between spoken claims and acoustic evidence. It covers 188 sound‑event classes across seven semantic, temporal, and spatial tasks, pairing environmental recordings with either compatible or contradictory synthesized spoken claims and including audio‑only controls. Evaluation of five open‑weight ALMs shows that high audio‑only accuracy often coexists with significant drops in performance under conflicting claims, revealing gaps in conflict robustness and the mixed effects of prompt interventions.
By Wenbin Wang, Yuheng Li, Koichiro Yoshino
The paper introduces TAILOR, a framework that uses Conditional Value at Risk (CVaR) to guide large language model (LLM) synthesis of heuristics for computer systems. By optimizing the paired CVaR of cost differences relative to a baseline, TAILOR ensures improvements in both average performance and tail optimality gaps without needing to compute the unknown optimum. Experiments on Azure GPU batch admission, Azure VM packing, and server placement demonstrate that TAILOR achieves the best mean–tail trade‑off and outperforms robust, human‑designed alternatives in the tail while maintaining comparable average gains.
By Pantea Karimi, Roozbeh Bostandoost, Dany Rouhana, Siva Kesava Reddy Kakarla, Pooria Namyar, Mohammad Alizadeh, Venkat Arun, Behnaz Arzani
The paper introduces Orthogonal Drift Correction (ODC), an inference-time guidance method that mitigates semantic drift in text-to-image generation without retraining or extra user input. ODC operates in two stages: first it detects drift by comparing the generated image to the prompt in a shared vision-language embedding space, isolates the orthogonal component of the drift, and converts it back into text; second it refines the text conditioning for a second generation pass using an adaptive rank-reduced concept removal module. Experiments show ODC improves prompt-image alignment, raising CLIP scores by up to 5.4% on COALBench-300 and PartiPrompts, and boosting the 3-in-1 score by 4.5% on T2I-CompBench’s complex subset.
By Kasra Kamali, Basim Azam, Tuan Duong Trinh, Jean Honorio, Naveed Akhtar
The paper introduces Zep Agent Skills, a production service that autonomously compiles and curates procedural memory for large language model agents by converting verified execution traces into versioned skills. It records each task attempt as a trace, uses a compiler with Reflector and Curator roles to transform verified traces into skills, and applies admission gates and search ranking based on verified outcomes. The open Agent Skills format, originally from Anthropic, allows any host to load compiled skills, and evaluation on several benchmarks shows significant performance gains for weaker executor models without additional training.
By Daniel Chalef, Peter Evans, Pavlo Paliychuk, Mehul Patel, Preston Rasmussen, Jack Ryan
The paper introduces Online Meta-Control of Experience (OMCE), a framework that updates an agent’s experience memory and learns a meta-control policy during test-time adaptation. OMCE uses a lightweight contextual-bandit meta-controller to choose among four cognitive actions—DIRECT, EPISODE, GUIDE, and DISTILL—based on compact retrieval signals, thereby deciding when to invoke experience, which memory format to use, and when to consolidate new guidance. Experiments on four online adaptation benchmarks across four backbones show that OMCE can improve first-attempt success by up to 20.5 percentage points over the strongest baseline while reducing per-task token use by a factor of 5.7 to 62.5 relative to ACE with GPT‑5.2, and controlled ablations confirm the benefits of learned routing and ongoing policy adaptation.
By Zeyuan Zhao, Yuanyuan Shi, Peng Yu, Lingyue Fu, Shao Zhang, Ying Wen
PixelBench is a benchmark that evaluates large language models on their ability to generate and understand images pixel by pixel, without relying on learned image tokenizers or decoders. The benchmark includes text-to-pixel (T2P) and image-to-pixel (I2P) generation tasks across multiple resolutions, as well as a pixel-understanding evaluation. Models produce an RGB palette followed by a grid of indices that map each pixel to a palette color, and the study shows that strong models can generate recognizable images from text and outperform specialized pixelization algorithms at low resolutions, with performance correlating with pixel-understanding ability and improving up to a point before declining at high resolutions. Additionally, Pixel Reconstruction Tuning (PRT) augments visual instruction tuning with caption‑palette‑grid trajectories, and post‑training Molmo2‑4B with PRT data improves performance on 16 visual‑centric benchmarks, demonstrating the utility of explicit pixel reconstruction as auxiliary supervision.
By Haoquan Zhang, Feiyang Liu, Juanxi Tian, Wanhua Li, Haodong Duan, Weiyang Liu
InteractionBench is a benchmark that evaluates real‑time voice systems on four dimensions—conversational timing, spoken‑task accuracy, whispering and volume, and groundedness—using the same recorded user audio across thirteen scenarios. It tests seven systems (GPT‑Live, GPT‑Realtime, Gemini Live, Moshi, PersonaPlex, Freeze‑Omni, and a turn‑based cascade) over 1,519 runs, revealing correlations between timing and task accuracy and highlighting differences in whispering, grammar correction, and groundedness performance. The study also notes protocol variations and the impact of network delay on timing scores.
By Richard Yucheng He, Chen Xu, Yihang Liu, Tairan Chen, Baodong Cao
DR Pixel introduces a page‑native deep research agent that uses rendered document page images as the sole unit for retrieval and reasoning, bypassing OCR. The open‑source system operates on 1.06 M pages and, without training, outperforms OCR‑based baselines on GPT‑5.4, Gemini‑3.5‑Flash, and Qwen3.6‑27B while requiring fewer tool calls. Training with counterfactual credit reweighting further boosts retrieval recall by ~15 points and accuracy by >10 points on the DRPixelBench benchmark, and brings the model within 1.8 points of an oracle on the ViDoRe‑V3 out‑of‑distribution test.
By Tiansheng Hu, Yichuan Wang, James V. Talwar, Nicholas Tomlin, Chen Zhao, Sewon Min
arXiv:2611. 03196v1 Announce Type: cross Abstract: Dh\={a}r\={a} Bench is a retrieval benchmark for Indian Knowledge Systems (IKS), with 325 questions in 12 categories and 60,121 text chunks from 15 source groups.
By Avinash Holenarasipur Nagendra, Anand Kaniyaar Sriranga Prasad, Kousik Sarathy Sridharan, Mohan Raghavan
The paper investigates how the scoring rules used in multimodal large‑language‑model (LLM) benchmarks can dramatically alter reported accuracy. By reproducing official scorers for eight benchmarks and comparing them to a blinded human‑plus‑LLM adjudication, the authors show that the same model outputs can yield accuracy differences of up to 22 points depending on the rule. They argue that no single scoring rule is reliable and recommend that benchmarks report non‑compliance rates and use at least two distinct scoring rules.
By Ahmad Jarjis Hasan, Jonathan Lwowski
The paper introduces Mr Dre, an evaluation suite that tests Deep Research Agents (DRAs) on multi-turn report revision, a process more reflective of human research practices. It defines a unified long‑form report protocol covering comprehensiveness, factuality, and presentation, and includes a human‑verified feedback simulation pipeline. Analysis of five DRAs shows that while agents can address most user feedback, they regress on 16‑28% of previously covered content and citation quality, and over successive turns they disrupt unrelated content and fail to preserve earlier edits, with prompt engineering and dedicated subagents offering limited improvement.
By Bingsen Chen, Boyan Li, Ping Nie, Yuyu Zhang, Xi Ye, Chen Zhao
The paper introduces EcoTab, a stepwise routing framework that improves table reasoning efficiency by separately estimating uncertainties for table tokens and text tokens. It shows that these token types have distinct uncertainty distributions correlated with next-step error risk, and that existing methods do not model them separately, leading to suboptimal routing. Experiments on several benchmarks demonstrate that EcoTab outperforms strong baselines while achieving a better accuracy‑efficiency balance.
By Shenghao Ye, Yuxiang Wang, Yu Guo, Tao Zhang, Shuangwu Chen, Zijian Wen, Qihui Zhu, Mengjie Zhang, Dong Jin, Wang Luo, Jian Yang
SpeechEQ is a new benchmark that evaluates the sociolinguistic reasoning of Speech‑Language Models (SLMs) by using a dataset of 2,265 dialogues covering 15 Emotional Quotient (EQ) subscales from EQ‑i 2.0 theory. The framework introduces a multi‑turn evaluation protocol and a Spoken EQ (SEQ) score inspired by human EQ assessments. Experiments reveal that current multimodal models suffer from a text‑reliant shortcut, an alignment‑induced safety trap, and contextual amnesia, limiting their emotional awareness in spoken interaction.
By Liang-Yuan Wu, Zih-Ching Chen, Tongshuang Wu, Chao-Han Huck Yang, Hua Shen
AgentHop is a diagnostic benchmark comprising 1,011 multiple‑choice questions that test agentic multi‑hop question answering over scientific citation graphs. It provides a controlled seven‑tool sandbox with fixed token, turn, and tool‑call limits, allowing researchers to dissect model performance across four operational axes: retrieval, synthesis, tool‑call, and resource management. Analysis of 19 models shows distinct family‑specific behaviors, such as GPT models committing early, Anthropic and GLM checkpoints verifying before committing, DeepSeek and Kimi over‑searching, and Gemini‑3 Pro maintaining balance, while within‑family differences are revealed in retrieval versus synthesis strategies.
By Chanhee Park, Jeongho Yoon, Sungbin Han, Hyeonseok Moon, Heuiseok Lim
DARE introduces a distribution‑aware retrieval framework that enhances Large Language Model agents’ ability to select appropriate R functions for data science tasks. The system builds on a curated R function knowledge base (RPKB), a new retrieval method (DARE), an R‑oriented agent (RCodingAgent), and a benchmark (RStatBench). Experiments show significant gains in retrieval quality (up to 32.65 pp improvement in NDCG@10) and a near‑doubling of downstream analysis accuracy on RStatBench.
By Maojun Sun, Yue Wu, Yifei Xie, Ruijian Han, Binyan Jiang, Defeng Sun, Yancheng Yuan, Jian Huang
The paper introduces a multi-reward reinforcement learning from internal feedback (RLIF) framework that splits the training signal into an answer-level reward from cluster voting and a completion-level reward from token-wise self-certainty. It uses GDPO-based normalization to balance reward scales and adds KL-Cov regularization to maintain exploration and avoid entropy collapse. Experiments on mathematical reasoning and code-generation tasks show the method improves stability and robustness compared to prior unsupervised RL approaches, approaching the performance of supervised RLVR methods.
By Shourov Joarder, Diganta Sikdar, Ahsan Habib Akash, Binod Bhattarai, Prashnna Gyawali
The paper presents a multimodal house‑price prediction framework that combines structured attributes, textual descriptions, and property images. It uses a large language model to extract spatio‑temporal and contextual features from text and an unsupervised vision‑language method to cluster images into categories such as indoor, outdoor, mixed, or floorplan without manual labeling. The fused representation is fed to an optimized regressor, and experiments on a large real‑world dataset show improved accuracy over traditional and existing multimodal baselines, with ablation studies highlighting the contribution of the new textual and visual features.
By Khandokar Md. Rahat Hossain, Souvik Das, Tahsina Hashem, Mohammed Eunus Ali, Tanzima Hashem