Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

23,288 stories · RSS feed

arXiv AI

Jun 6

Do More Agents Help? Controlled and Protocol-Aligned Evaluation of LLM Agent Workflows

arXiv:2606. 05670v1 Announce Type: new Abstract: Does adding more agents help an LLM workflow once compared systems share the same benchmark loader, tool access, answer contract, usage accounting, and trajectory logging?

By Yuhang Fu, Ruishan Fang, Jiaqi Shao, Huiyu Zheng, Zhengtao Zhu, Bing Luo, Tao Lin

llms agents benchmarks

arXiv AI

Jun 6

Toward Culturally Aligned LLMs through Ontology-Guided Multi-Agent Reasoning

arXiv:2601. 21700v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) increasingly support culturally sensitive decision making, yet often exhibit misalignment due to skewed pretraining data and the absence of structured value representations.

By Wonduk Seo, Wonseok Choi, Junseo Koh, Juhyeon Lee, Hyunjin An, Minhyeong Yu, Jian Park, Qingshan Zhou, Seunghyun Lee, Yi Bu

llms agents benchmarks safety

arXiv AI

Jun 6

I Know What You Meme, Even If it Emerged Today: Understanding Evolving Memes through Open-World Knowledge Acquisition

arXiv:2606. 05316v1 Announce Type: new Abstract: Multimodal memes are dynamic and often require up to date background knowledge for interpretation.

By Shanhong Liu, Rui Cao, Pai Chet Ng, De Wen Soh

multimodal benchmarks

arXiv AI

Jun 6

Emotion-Aware Image Generation from Korean Diary Text via LLM-based Prompt Translation and LoRA Fine-Tuning

arXiv:2606. 05816v1 Announce Type: cross Abstract: T2I models cannot effectively capture sentiment from various types of text, including diaries, as they primarily focus on visual object-related patterns rather than contextual emotional understanding.

By Jihun Cho, Soo-Yeon Jeong, Sun-Young Ihm

llms diffusion fine-tuning benchmarks

arXiv AI

Jun 6

LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video

arXiv:2606. 05677v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have advanced image and video understanding and can increasingly handle longer visual inputs.

By Shiqiang Lang, Jing Liu, Haoyang He, Peiwen Sun, Yuanteng Chen, Tao Liu, Lan Yang, Longteng Guo, Honggang Zhang

llms agents robotics multimodal benchmarks

arXiv AI

Jun 6

Beyond Waveform Robustness: Robust Feature-Vocoder Adversarial Attacks on Automatic Speech Recognition

arXiv:2606. 05678v1 Announce Type: cross Abstract: Automatic speech recognition (ASR) systems have become widely used for multilingual speech-to-text transcription.

By Yifan Liao, Zongmin Zhang, Zhen Sun, Yuhui Sun, Xinhu Zheng, Xinlei He

multimodal benchmarks safety

arXiv AI

Jun 6

Whose Alignment? Comparing LLM Process Alignment Across Diverse Organizational Decision Contexts

arXiv:2605. 25256v2 Announce Type: replace Abstract: Steerable pluralism requires a model to faithfully represent one specified perspective.

By Niklas Weller, Emilio Barkett

llms benchmarks safety

arXiv AI

Jun 6

Controllable and Verifiable Process Data Synthesis for Process Reward Models

arXiv:2605. 02395v2 Announce Type: replace Abstract: Process reward models (PRMs) rely on high-quality process supervision data, yet existing construction methods often provide limited control over error location, error type, and trajectory consistency.

By Yinghui Chi, Lucien Wang

rag benchmarks

arXiv AI

Jun 6

SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations

arXiv:2606. 05563v1 Announce Type: new Abstract: Evaluating LLM mediators remains challenging, as mediation unfolds as a real-time trajectory shaped by disputants' shifting emotions, intentions, and context.

By Taewon Yun, Hyeonseong Park, Jeonghwan Choi, Hayoon Park, Yeeun Choi, Hwanjun Song

llms agents benchmarks safety

arXiv AI

Jun 6

Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments

arXiv:2606. 05661v1 Announce Type: new Abstract: Continual learning, the ability of AI systems to improve through sequential experience, has attracted substantial interest, but no high-quality benchmark exists to evaluate it.

By Parth Asawa, Christopher M. Glaze, Gabriel Orlanski, Ramya Ramakrishnan, Benji Xu, Asim Biswal, Vincent Sunn Chen, Frederic Sala, Matei Zaharia, Joseph E. Gonzalez

llms agents benchmarks

arXiv AI

Jun 6

Reducing Hallucinations in Complex Question Answering using Simple Graph-based Retrieval-Augmented Generation (long version)

arXiv:2606. 05901v1 Announce Type: cross Abstract: Large language models (LLMs) have fundamentally transformed the landscape of Natural Language Processing.

By Christopher J. Wedge, Joshua Stutter, Danny Dixon, Jacek Ca{\l}a

llms rag agents nlp fine-tuning benchmarks safety

arXiv AI

Jun 6

Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluation

arXiv:2606. 05241v1 Announce Type: cross Abstract: Public benchmarks enable fair and reproducible evaluation of LLM reasoning, but they become fragile for deep research agents that actively search the web during inference.

By Yongjie Wang, Xinyue Zhang, Kunhong Yao, Zhiwei Zeng, Kaisong Song, Jun Lin, Zhiqi Shen

llms agents benchmarks

arXiv AI

Jun 6

ReTreVal: Reasoning Tree with Validation and Cross-Problem Memory for Large Language Models

arXiv:2601. 02880v2 Announce Type: replace Abstract: Every existing inference-time reasoning framework discards all failure context at problem boundaries, leaving a model solving problem 500 no wiser than it was on problem 1.

By Abhishek HS, Pavan C Shekar, Arpit Jain, Ashwanth Krishnan

llms fine-tuning benchmarks

arXiv AI

Jun 6

ITP-STDP: An Intrinsic-Timing Power-of-Two Learning Engine for On-Chip SNN Training

arXiv:2606. 06159v1 Announce Type: cross Abstract: Spiking neural networks (SNNs) have the potential to emerge as the third generation of neural networks and have attracted increasing attention across a wide range of applications.

By Haihang Xia, Xinyu Zhao, Xuecheng Wang, John Goodenough, Charith Abhayaratne, Panagiotis A. Panagiotou, Chunyi Song, Tiantai Deng

benchmarks

arXiv AI

Jun 6

MAVEN A Multi-Agent Framework for Multicultural Text-to-Video Generation

arXiv:2605. 16716v4 Announce Type: replace-cross Abstract: Text-to-video (T2V) generation has rapidly progressed in visual fidelity, yet its ability to faithfully represent multiple cultures within a single prompt remains underexplored.

By Shuowei Li, Yuming Zhao, Parth Bhalerao, Oana Ignat

agents diffusion multimodal benchmarks

arXiv AI

Jun 6

Agent-Orchestrated Adaptive RAG: A Comparative Study on Structured and Multi-Hop Retrieval

arXiv:2606. 05658v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by grounding their responses in external knowledge, but conventional pipelines rely on static, single-step retrieval that limits performance on complex queries.

By Anuj Maharjan, Devinder Kaur, Richard Molyet

llms rag agents benchmarks

arXiv AI

Jun 6

The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?

arXiv:2504. 10020v4 Announce Type: replace-cross Abstract: Contrastive decoding strategies are widely used to reduce object hallucinations in multimodal large language models (MLLMs).

By Hao Yin, Guangzong Si, Zilei Wang

llms multimodal benchmarks safety

arXiv AI

Jun 6

Noise-Aware Visual Representation Learning for Medical Visual Question Answering

arXiv:2606. 05535v1 Announce Type: cross Abstract: Medical visual question answering (Med-VQA) has strong potential for clinical decision support by enabling AI models to interpret medical images and answer clinically relevant queries.

By I Putu Adi Pratama, Bahadorreza Ofoghi, Atul Sajjanhar, Shang Gao

llms rag nlp fine-tuning benchmarks

arXiv AI

Jun 6

When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents

arXiv:2606. 05806v1 Announce Type: new Abstract: Existing benchmarks evaluate Tool-Integrated Reasoning (TIR) in LLMs on idealized ''happy paths'', largely overlooking real-world tool failures.

By Dongsheng Zhu, Xuchen Ma, Yucheng Shen, Xiang Li, Yukun Zhao, Shuaiqiang Wang, Lingyong Yan, Dawei Yin

llms agents benchmarks

arXiv AI

Jun 6

SciVisAgentSkills: Design and Evaluation of Agent Skills for Scientific Data Analysis and Visualization

arXiv:2606. 05525v1 Announce Type: new Abstract: Recent advances in agentic visualization have enabled the translation of natural language into executable scientific visualization (SciVis) workflows.

By Kuangshi Ai, Haichao Miao, Kaiyuan Tang, Shusen Liu, Chaoli Wang

llms agents benchmarks

Get the daily briefing