arXiv Computation and Language

IFC-MCQ: Automatically Generating Multiple-Choice Benchmarks for Evaluating LLM Agents on IFC Data Retrieval

IFC-MCQ is a method that automatically creates multiple-choice benchmarks from Industry Foundation Class (IFC) files, enabling exact option-set grading for large language model (LLM) queries in architecture, engineering, and construction (AEC). The taxonomy spans 15 categories and 36 subtypes across four groups, with answer keys derived from the IFC file and distractors generated from file-derived values or predefined rules. Experiments on IFC models of varying sizes and characteristics revealed category-specific failure modes and highlighted how performance can differ based on the source IFC model and retrieval interface.

arXiv AI
Oct 27

The Hitchhiker's Guide to System 1 Models: Fast Decision-Making Beyond LLMS

The paper introduces System 1 models—fast, single‑pass decision engines that map inputs directly to structured outputs, addressing the latency, cost, and calibration issues of large language models. It presents a five‑family taxonomy (encoder‑based classifiers, reward models, non‑autoregressive engines, contrastive models, and extraction models) and discusses training, deployment trade‑offs, and calibration theory, including proper scoring rules and RLCD. The guide also covers integration into agentic workflows, System 1+System 2 cascades, 18 application domains, an open‑source ecosystem, and practical resources such as code, a model zoo, and assessment questions.

By Haggai Roitman
arXiv Computation and Language
Sep 28

Large Language Model Selection with Limited Annotations

arXiv:2605.24981v2 Announce Type: replace Abstract: Choosing a Large Language Model (LLM) for a given task requires comparing many strong candidates, yet standard evaluation relies on costly annotati...

By Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch, Torsten Hoefler, Nezihe Merve G\"urel
arXiv Machine Learning
Oct 22

Code Comprehensor: Hierarchical Documentation and Cross-Modal Retrieval for Large Repositories

Code Comprehensor is a hybrid retrieval‑reasoning framework designed to aid understanding of large codebases. It combines a code dependency callgraph, CodeScribe—which generates hierarchical documentation using code‑aware LLMs—and CodeContrast, a CLIP‑style dual‑encoder that learns aligned text‑code embeddings for fast, semantically precise retrieval. Experiments on OpenCV and a 5M‑LOC proprietary repository show that CodeScribe‑derived supervision improves cross‑modal retrieval accuracy, and LLM‑based evaluation reports over 90% faithfulness.

By Sujoy Roychowdhury, Joseph H. R. Isaac, Jiangning Gao, Martin Klitte, Diarmuid Corcoran
Hugging Face Trending Papers
Aug 6

Benchmarking and Enhancing LLMs for Rule-Intensive Review of National Standard Documents

Large language models (LLMs) increasingly support complex professional tasks, yet their capabilities in rule-intensive document review remain insufficiently evaluated. National standard documents, such as China GB/T standards, offer a representative testbed: they are lengthy, highly structured, and governed by explicit rules for scope, terminology, normative wording, and cross-section consistency.

arXiv Machine Learning
Sep 22

Toollery: Scaling LLM Agents to Thousands of Skills and Tools

Toollery is a training‑free framework that compresses candidate lists for large language model agents, enabling efficient selection from thousands of skills and tools. It generates user‑intent queries from each skill or tool specification, builds a retrieval index, and limits online selection to a compact top‑k set before the LLM makes its final decision. Evaluations on the SkillRouter benchmark, BFCL‑V4, and a proprietary smart‑cockpit dataset show that Toollery improves recall and end‑to‑end selection while keeping selection costs bounded.

By Xiangxi Tian, Ran Guan
arXiv AI
Jul 17

OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios

arXiv:2607. 14989v1 Announce Type: cross Abstract: Large language models are increasingly evolving from text generators into general agents capable of understanding user requests, invoking external tools, and completing complex tasks through interaction.

By Chengyu Shen, Yujie Fu, Gangtao Xin, Yanheng Hou, Wenlong Fei, Guojie Zhu, Jiawei Li, Hongcheng Gao, Runming He, Zhen Hao Wong, Meiyi Qiang, Hao Liang, Zhao Cao, Hao Jiang, Chong Chen, Wentao Zhang
arXiv Computation and Language
Oct 19

Chinese Decision: A Family of Language Models for Single-Pass Probabilistic Decision-Making

The paper introduces Chinese Decision, a family of Chinese-language models designed for single-pass probabilistic decision-making. The models generate logits for candidate choices in one forward pass and normalize them without autoregressive generation, supporting Boolean, categorical, and ordinal decisions. A 0.8B model trained on 13.048 million decisions achieves 72.24% general-domain accuracy and 76.98% average accuracy across medicine, law, and finance, while a 2B model shows comparable performance after partial training.

By Qiang Yan (Institute of Computing Technology, Chinese Academy of Sciences)