Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Machine Learning

Sep 22

TAC-Time: Texts as Channels For Multimodal Time Series Forecasting

arXiv:2609.24156v1 Announce Type: cross Abstract: Most existing time series forecasting methods rely solely on numerical observations, overlooking rich contextual information from auxiliary texts. Re...

By Jiayi Liang, Xiaotian Gu, Xinyu Xie, Yuanbin Wu, Xiaoling Wang

llms multimodal benchmarks safety

arXiv Computation and Language

Sep 22

Taramandal-GPT: Enhancing Astrodynamics Problem-Solving with Knowledge Retrieval and Structured Thinking

arXiv:2609.24246v1 Announce Type: new Abstract: Large language models (LLMs) have shown remarkable progress in natural language understanding, yet their effectiveness in specialized fields like astro...

By Akhil Sharma, Jatin Gupta, Ali Imam Abidi

llms rag robotics benchmarks safety

arXiv Machine Learning

Sep 22

MemCalib: Benchmarking and Optimizing Memory Use in LLM Agents

arXiv:2609.24259v1 Announce Type: new Abstract: The effectiveness of agent memory ultimately depends on whether the underlying LLM gives each memory in context an appropriate degree of influence over...

By Ruike Cao, Fanyu Zhao, Fugen Yao, Liang Dong, Jian Xu, Guanjun Jiang, Yifei Zhao, Han Zhang, Li Xiao

llms agents reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Sep 22

KV-COBRA: KV Cache Compression via Co-Optimized Bit-Rank Allocation

arXiv:2609.24298v1 Announce Type: new Abstract: What limits KV-cache compression at extreme bit-rates? We argue that it is not the choice of compression scheme, but how its budget is allocated across...

By Sihyeon Ha, Jaeho Lee, Yo-Seb Jeon

efficiency benchmarks

arXiv Machine Learning

Sep 22

Information-Time Proximal Policy Optimization

arXiv:2609.24380v1 Announce Type: new Abstract: RLVR has substantially improved the reasoning capabilities of LLMs. However, existing methods typically parameterize temporal progression in the Markov...

By Yongcheng Zeng, Xinyu Cui, Yan Song, Guoqing Liu, Hongsheng Xin, Kaike Zhang, Cheng Deng, Kun Zhan, Jian Ying, Jian Zhao, Haifeng Zhang, Jun Wang

llms reinforcement-learning benchmarks

arXiv Machine Learning

Sep 22

ARM: Attention with Routed-Memory for Learnable Sparse Control

arXiv:2609.24417v1 Announce Type: new Abstract: Despite advances in long-context inference, large language models (LLMs) remain fundamentally limited by the key-value (KV) caching mechanisms that are...

By Qiuhao Zeng, Jerry Huang, Peng Lu, Ruiyi Fang, Gezheng Xu, Zihao Jing, Yufei Cui, Charles Ling, Gang Niu, Boyu Wang

llms efficiency benchmarks

arXiv Computer Vision

Sep 22

Estimating Accurate Hand Pose in Camera Space with Vision Transformer

arXiv:2609.24424v1 Announce Type: new Abstract: Monocular RGB-based hand pose estimation has emerged as a critical research frontier in computer vision. The local hand pose estimation methods predict...

By Kaiwen Ren, Yiran Jiang, Yongjing Ye, Shihong Xia

llms rag computer-vision benchmarks

arXiv Computer Vision

Sep 22

VPRune: Efficient Training-free Pre-LLM Visual Token Pruning

arXiv:2609.24485v1 Announce Type: new Abstract: Visual token pruning is a promising approach to reducing the inference cost of large vision-language models (LVLMs), yet aggressive token reduction oft...

By Guangchuan Lv, Dianxing Shi, Dingjie FU

llms efficiency multimodal benchmarks safety

arXiv Computer Vision

Sep 22

AgentSTAR: Agentic Shape Tracking and Reconstruction from Monocular Videos

arXiv:2609.24487v1 Announce Type: new Abstract: In this work, we present a method for shape reconstruction and tracking from video via agentic analysis-by-synthesis. Unlike prior methods which first...

By Kirill Mazur, Nikita Karaev, Matthew Chang, Jitendra Malik, Nur Muhammad "Mahi'' Shafiullah

llms agents multimodal benchmarks

arXiv Machine Learning

Sep 22

$t_0$: A Time-Series Foundation Model for Forecasting with Context

arXiv:2609.24559v1 Announce Type: new Abstract: We present $t_0$, a family of open-weights foundation models for forecasting with multivariate context. We release its first two members: $\texttt{t0-a...

By Lucas Meyer, Claudio Sole, Huikan Xiang, Nicolas Li, Lucas Franceschino, Arnau Quera-Bofarull, Maarten P. Scholl, Joachim Fainberg, Geoffrey N\'egiar

llms benchmarks

arXiv Machine Learning

Sep 22

iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs

arXiv:2609.24646v1 Announce Type: new Abstract: On-policy self-distillation fine-tuning (SDFT) learns new skills from demonstrations while reducing forgetting, but it always distils toward the full d...

By Ahmed Khaled Khamis, Xiaotong Ji, Hassan Jaber, Rasul Tutunov, Matthieu Zimmer, Jun Wang, Haitham Bou-Ammar

llms fine-tuning efficiency benchmarks

arXiv Machine Learning

Sep 22

Enhancing Transformer Representations of Symbolic ODE Expressions

arXiv:2609.24746v1 Announce Type: new Abstract: Existing approaches to solving differential equations, such as symbolic regression, physics informed neural networks, and neural operators, typically f...

By Xiyue Fan, Adam Prugel-Bennett, Stuart E. Middleton

llms rag benchmarks

arXiv Computer Vision

Sep 22

PrismGPT: Proxy-Guided Learning for Region-Aware Photo Editing with Self-Synthesized Reasoning

PrismGPT is a Vision‑Language Model that generates structured, region‑aware photo‑editing plans from a single image, without relying on commercial black‑box tools. It learns to diagnose aesthetic issues globally and locally while predicting precise editing parameters, using proxy‑guided learning with operation decomposition and region‑aware aesthetic ranking to bootstrap the model. A competence‑based dynamic scheduler shifts training focus from proxy tasks to the main editing task as skills improve, and all reasoning traces for fine‑tuning are self‑synthesized by the model itself. Experiments on MIT‑Adobe FiveK and a new professionally retouched benchmark, SPIRE, show PrismGPT achieves state‑of‑the‑art results using only about 6% of the training data required by previous methods.

By Ke Zhao, Hue Nguyen, Abhijith Punnappurath, Zhongling Wang, Iqbal Mohomed, Michael S. Brown

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Sep 22

Mobile Imaging Solutions for Medical Diagnosis: Trends and Applications

arXiv:2609.24814v1 Announce Type: cross Abstract: Advances in processing power, camera technologies, and mobile image analysis have made smartphones and other mobile devices, such as laptops, increas...

By Syed Muhammad Ibne Zulfiker, Tanzima Hashem, Fariha Tabassum Islam, Md Sultanul Arifin, Khandker Aftarul Islam, Nishat Anjum Bristy, Faria Huq, Priyeta Saha, Syeda Nahida Akter, Arpita Saha

benchmarks

arXiv Machine Learning

Sep 22

OSWorld-Pro: Process-based Evaluation for Computer Use Agents

arXiv:2609.24890v1 Announce Type: cross Abstract: Evaluation of Computer-Use Agents (CUAs) is often limited to the final deliverables they create (at the end of hundreds of steps) and assessed with f...

By Zhilin Wang, Shaokun Zhang, Yifan Zhang, Hao Zhang, Jin Xu, Binfeng Xu, Jian Hu, Yunheng Zou, Karan Sapra, Andrew Tao, Jan Kautz, Yi Dong

llms agents benchmarks

arXiv Machine Learning

Sep 22

Subgoal Search For Complex Reasoning Tasks

arXiv:2108.11204v4 Announce Type: replace-cross Abstract: Humans excel in solving complex reasoning tasks through a mental process of moving from one idea to a related one. Inspired by this, we propo...

By Konrad Czechowski, Tomasz Odrzyg\'o\'zd\'z, Marek Zbysi\'nski, Micha{\l} Zawalski, Krzysztof Olejnik, Yuhuai Wu, {\L}ukasz Kuci\'nski, Piotr Mi{\l}o\'s

llms benchmarks

arXiv Machine Learning

Sep 22

Reflective Policy Optimization

arXiv:2406.03678v2 Announce Type: replace Abstract: On-policy reinforcement learning methods, like Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO), often demand extensi...

By Yaozhong Gan, Renye Yan, Zhe Wu, Junliang Xing

agents reinforcement-learning benchmarks

arXiv Machine Learning

Sep 22

Streaming Deep Reinforcement Learning Finally Works

arXiv:2410.14606v3 Announce Type: replace Abstract: Learning from a stream of experience as it arrives, also known as streaming learning, is a core part of natural learning. However, reliable streami...

By Mohamed Elsayed, Elena Sorina Lupu, Gautham Vasan, A. Rupam Mahmood

reinforcement-learning robotics benchmarks

arXiv Computation and Language

Sep 22

Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning

Lingshu is a medical‑specialized multimodal large language model that addresses key limitations of existing medical MLLMs, such as narrow knowledge coverage, hallucinations, and weak reasoning. The authors curate a comprehensive dataset combining medical imaging, texts, and general‑domain data, then train Lingshu in multiple stages to embed medical expertise and improve task performance. They also introduce MedEvalKit, a unified evaluation framework, and demonstrate that Lingshu outperforms current open‑source multimodal models on multimodal QA, text‑based QA, and medical report generation.

By Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, Yu Sun, Junao Shen, Chaojun Wang, Jie Tan, Deli Zhao, Tingyang Xu, Hao Zhang, Yu Rong

llms nlp reinforcement-learning multimodal benchmarks safety

arXiv Machine Learning

Sep 22

Unlocking Pretrained Vision Transformers for Time Series Classification

arXiv:2506.08641v3 Announce Type: replace Abstract: Adapting vision models for time series analysis is compelling, yet all existing approaches are falling short of dedicated time series foundation mo...

By Simon Roschmann, Quentin Bouniot, Vasilii Feofanov, Ievgen Redko, Zeynep Akata

llms computer-vision fine-tuning benchmarks safety

Get the daily briefing