Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

23,288 stories · RSS feed

arXiv AI

Jun 6

Agent-Orchestrated Adaptive RAG: A Comparative Study on Structured and Multi-Hop Retrieval

arXiv:2606. 05658v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by grounding their responses in external knowledge, but conventional pipelines rely on static, single-step retrieval that limits performance on complex queries.

By Anuj Maharjan, Devinder Kaur, Richard Molyet

llms rag agents benchmarks

arXiv AI

Jun 6

CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectives

arXiv:2504. 10823v4 Announce Type: replace-cross Abstract: Navigating dilemmas involving conflicting values is challenging even for humans in high-stakes domains, let alone for AI, yet prior work has been limited to everyday scenarios.

By Ayoung Lee, Ryan Sungmo Kwon, Peter Railton, Lu Wang

llms benchmarks safety

arXiv AI

Jun 6

Image Generators are Generalist Vision Learners

arXiv:2604. 20329v3 Announce Type: replace-cross Abstract: Recent works show that image and video generators exhibit zero-shot visual understanding behaviors, in a way reminiscent of how LLMs develop emergent capabilities of language understanding and reasoning from generative pretraining.

By Valentin Gabeur, Shangbang Long, Songyou Peng, Paul Voigtlaender, Shuyang Sun, Yanan Bao, Karen Truong, Zhicheng Wang, Wenlei Zhou, Jonathan T. Barron, Kyle Genova, Nithish Kannen, Sherry Ben, Yandong Li, Mandy Guo, Suhas Yogin, Yiming Gu, Huizhong Chen, Oliver Wang, Saining Xie, Howard Zhou, Kaiming He, Thomas Funkhouser, Jean-Baptiste Alayrac, Radu Soricut

llms diffusion computer-vision benchmarks

arXiv AI

Jun 6

Whose Alignment? Comparing LLM Process Alignment Across Diverse Organizational Decision Contexts

arXiv:2605. 25256v2 Announce Type: replace Abstract: Steerable pluralism requires a model to faithfully represent one specified perspective.

By Niklas Weller, Emilio Barkett

llms benchmarks safety

arXiv AI

Jun 6

ProfiliTable: Profiling-Driven Tabular Data Processing via Agentic Workflows

arXiv:2605. 12376v2 Announce Type: replace Abstract: Table processing-including cleaning, transformation, augmentation, and matching-is a foundational yet error-prone stage in real-world data pipelines.

By Wei Liu, Yang Gu, Xi Yan, Zihan Nan, Beicheng Xu, Keyao Ding, Bin Cui, Wentao Zhang

llms agents benchmarks

arXiv AI

Jun 6

MAVEN A Multi-Agent Framework for Multicultural Text-to-Video Generation

arXiv:2605. 16716v4 Announce Type: replace-cross Abstract: Text-to-video (T2V) generation has rapidly progressed in visual fidelity, yet its ability to faithfully represent multiple cultures within a single prompt remains underexplored.

By Shuowei Li, Yuming Zhao, Parth Bhalerao, Oana Ignat

agents diffusion multimodal benchmarks

arXiv AI

Jun 6

Controllable and Verifiable Process Data Synthesis for Process Reward Models

arXiv:2605. 02395v2 Announce Type: replace Abstract: Process reward models (PRMs) rely on high-quality process supervision data, yet existing construction methods often provide limited control over error location, error type, and trajectory consistency.

By Yinghui Chi, Lucien Wang

rag benchmarks

arXiv AI

Jun 6

ReTreVal: Reasoning Tree with Validation and Cross-Problem Memory for Large Language Models

arXiv:2601. 02880v2 Announce Type: replace Abstract: Every existing inference-time reasoning framework discards all failure context at problem boundaries, leaving a model solving problem 500 no wiser than it was on problem 1.

By Abhishek HS, Pavan C Shekar, Arpit Jain, Ashwanth Krishnan

llms fine-tuning benchmarks

arXiv AI

Jun 6

Reducing Hallucinations in Complex Question Answering using Simple Graph-based Retrieval-Augmented Generation (long version)

arXiv:2606. 05901v1 Announce Type: cross Abstract: Large language models (LLMs) have fundamentally transformed the landscape of Natural Language Processing.

By Christopher J. Wedge, Joshua Stutter, Danny Dixon, Jacek Ca{\l}a

llms rag agents nlp fine-tuning benchmarks safety

arXiv AI

Jun 6

Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judges

arXiv:2606. 05384v1 Announce Type: new Abstract: LLM-as-judge evaluation is widely used in benchmarking pipelines, where model outputs are compared and ranked using automated evaluators.

By Srimonti Dutta, Akshata Kishore Moharir

llms benchmarks

arXiv AI

Jun 6

SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations

arXiv:2606. 05563v1 Announce Type: new Abstract: Evaluating LLM mediators remains challenging, as mediation unfolds as a real-time trajectory shaped by disputants' shifting emotions, intentions, and context.

By Taewon Yun, Hyeonseong Park, Jeonghwan Choi, Hayoon Park, Yeeun Choi, Hwanjun Song

llms agents benchmarks safety

arXiv AI

Jun 6

ITP-STDP: An Intrinsic-Timing Power-of-Two Learning Engine for On-Chip SNN Training

arXiv:2606. 06159v1 Announce Type: cross Abstract: Spiking neural networks (SNNs) have the potential to emerge as the third generation of neural networks and have attracted increasing attention across a wide range of applications.

By Haihang Xia, Xinyu Zhao, Xuecheng Wang, John Goodenough, Charith Abhayaratne, Panagiotis A. Panagiotou, Chunyi Song, Tiantai Deng

benchmarks

arXiv AI

Jun 6

The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?

arXiv:2504. 10020v4 Announce Type: replace-cross Abstract: Contrastive decoding strategies are widely used to reduce object hallucinations in multimodal large language models (MLLMs).

By Hao Yin, Guangzong Si, Zilei Wang

llms multimodal benchmarks safety

arXiv AI

Jun 6

Emotion-Aware Image Generation from Korean Diary Text via LLM-based Prompt Translation and LoRA Fine-Tuning

arXiv:2606. 05816v1 Announce Type: cross Abstract: T2I models cannot effectively capture sentiment from various types of text, including diaries, as they primarily focus on visual object-related patterns rather than contextual emotional understanding.

By Jihun Cho, Soo-Yeon Jeong, Sun-Young Ihm

llms diffusion fine-tuning benchmarks

arXiv AI

Jun 6

LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video

arXiv:2606. 05677v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have advanced image and video understanding and can increasingly handle longer visual inputs.

By Shiqiang Lang, Jing Liu, Haoyang He, Peiwen Sun, Yuanteng Chen, Tao Liu, Lan Yang, Longteng Guo, Honggang Zhang

llms agents robotics multimodal benchmarks

arXiv AI

Jun 6

Beyond Waveform Robustness: Robust Feature-Vocoder Adversarial Attacks on Automatic Speech Recognition

arXiv:2606. 05678v1 Announce Type: cross Abstract: Automatic speech recognition (ASR) systems have become widely used for multilingual speech-to-text transcription.

By Yifan Liao, Zongmin Zhang, Zhen Sun, Yuhui Sun, Xinhu Zheng, Xinlei He

multimodal benchmarks safety

arXiv AI

Jun 6

Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments

arXiv:2606. 05661v1 Announce Type: new Abstract: Continual learning, the ability of AI systems to improve through sequential experience, has attracted substantial interest, but no high-quality benchmark exists to evaluate it.

By Parth Asawa, Christopher M. Glaze, Gabriel Orlanski, Ramya Ramakrishnan, Benji Xu, Asim Biswal, Vincent Sunn Chen, Frederic Sala, Matei Zaharia, Joseph E. Gonzalez

llms agents benchmarks

arXiv AI

Jun 6

HypRAG: Hyperbolic Dense Retrieval for Retrieval Augmented Generation

arXiv:2602. 07739v2 Announce Type: replace-cross Abstract: Embedding geometry plays a fundamental role in retrieval quality, yet dense retrievers for retrieval-augmented generation (RAG) remain largely confined to Euclidean space.

By Hiren Madhu, Ngoc Bui, Ali Maatouk, Leandros Tassiulas, Smita Krishnaswamy, Menglin Yang, Sukanta Ganguly, Kiran Srinivasan, Rex Ying

llms rag benchmarks safety

arXiv AI

Jun 6

InfoShield: Privacy-Preserving Speech Representations for Mental Health Screening via Information-Theoretic Optimization

arXiv:2606. 05561v1 Announce Type: cross Abstract: Speech-based mental health screening offers scalable depression detection, yet clinical deployment faces a significant barrier: users' privacy concerns about demographic information exposure.

By Xueyang Wu, Siyuan Liu, Kezhuo Yang, Guang Ling

rag multimodal benchmarks safety

arXiv AI

Jun 6

HiDe: Rethinking The Zoom-IN method in High Resolution MLLMs via Hierarchical Decoupling

arXiv:2510. 00054v3 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) have made significant strides in visual understanding tasks.

By Xianjie Liu, Yiman Hu, Yixiong Zou, Liang Wu, Jian Xu, Bo Zheng

llms reinforcement-learning multimodal benchmarks safety

Get the daily briefing