Large language models

Model releases, architecture work and prompting research on large language models — from frontier-lab announcements to the arXiv papers behind them.

32,584 stories · RSS feed

arXiv Machine Learning
Sep 17

Behavior2Value: Benchmarking and Empowering LLMs for Consumer Value Measurement from E-commerce Behaviors

The paper introduces the Behavior-to-Value (B2V) task, which seeks to identify consumer values from e-commerce behavioral trajectories. It presents the E-commerce Consumption Value Taxonomy (ECVT) and the B2V-Bench dataset, derived from anonymized Taobao logs and covering 25 purchase behaviors with associated value orientations. A new model, B2V-Verifier, is proposed to improve value measurement accuracy, achieving a 34% boost in multi-label classification over strong LLM baselines.

By Peixuan Hou, Bin Chen, Li He, Jian Xu, Bo Zheng, Xiuli Ma, Guojie Song
arXiv Machine Learning
Sep 17

Fallacy Benchmarks Measure Scheme Recognition, Not Fallacy Detection

The paper argues that current fallacy-detection benchmarks are flawed because they pair fallacy classes with a broad "valid" or "none" class that includes many correct arguments, leading to misleadingly low false‑positive rates. By constructing scheme‑matched negatives—correct arguments that use the same argumentation scheme as the fallacy—the authors show that false‑positive rates rise dramatically, indicating that classifiers are learning to recognize schemes rather than detecting fallacies. The study releases these scheme‑matched examples as Scheme Foils and cautions that reported false‑positive rates should not be trusted until the valid class is audited for scheme coverage.

By Navyansh Singh, Animesh Pathak, Aarav Singh
arXiv Computation and Language
Sep 17

From Pixels to Pairs: A Comprehensive Benchmark of LLM-Based Key-Value Extraction in Noisy Document Settings

The paper introduces a benchmark for evaluating open‑source instruction‑tuned large language models (LLMs) on key‑value pair extraction from documents under both clean‑text and noisy OCR conditions. It tests decoder‑only models such as Gemma, Mistral, Qwen2.5, LLaMA 3, and DeepSeek on FUNSD, CORD, and SROIE datasets, using OCR outputs from PaddleOCR, EasyOCR, and Tesseract. Results show that while modern LLMs perform well on high‑quality text, their performance drops sharply with OCR noise, and the main determinants of success are semantic reasoning and textual fidelity, with larger models offering diminishing returns under noisy inputs.

By Zahra Anvari, Vassilis Athitsos
arXiv Computation and Language
Sep 17

Relation Before Entity: Deferred Commitment in Language Model Factual Recall

The paper investigates when relation-type and entity-specific information become active during factual recall in decoder-only language models. Using causal diagnostics across multiple models and prompts, it finds that relation information becomes generation-controlling earlier than entity information, with a temporal gap of 10–16 layers. Despite entity information being present early, its influence on generation is deferred until later layers.

By Divyansh Agarwal
arXiv Machine Learning
Sep 17

Interpretable Retinal Disease Prediction Using Biology-Informed Heterogeneous Graph Representations

The paper introduces a biology-informed heterogeneous graph representation that models retinal vessel segments, intercapillary areas, and the foveal avascular zone to predict diabetic retinopathy stages from OCTA images. This graph-based approach reframes staging as a graph-level classification task solved with a graph neural network, achieving AUC-ROC values up to 84% and outperforming biomarker-based classifiers, CNNs, and vision transformers. The method also provides detailed, interpretable explanations by precisely localizing abnormal vessels and non-perfusion areas.

By Laurin Lux, Alexander H. Berger, Maria Romeo Tricas, Richard Rosen, Alaa E. Fayed, Sobha Sivaprasada, Linus Kreitner, Jonas Weidner, Martin J. Menten, Daniel Rueckert, Johannes C. Paetzold
arXiv AI
Sep 17

Hypothesis-Driven Autonomous Materials Synthesis with Multimodal LLM Agents

SynAgent is a framework that uses large language model agents to run autonomous experiments while building an explicit, revisable understanding of the synthesis process. Unlike traditional black‑box optimizers, SynAgent generates analysis skills on the fly and reasons multimodally over data such as X‑ray diffraction patterns and electron micrographs. In an 18‑experiment campaign on LiCoO₂ thin‑film deposition, SynAgent produced highly crystalline films and uncovered a sharp temperature threshold and optimal growth window (650–690 °C) for crystallization.

By Izumi Takahara, Kazunori Nishio, Akira Aiba, Shigeru Kobayashi, Takao Nakajima, Taro Hitosugi, Teruyasu Mizoguchi
arXiv AI
Sep 17

Bypassing the Rationale: Causal Auditing of Implicit Reasoning in Language Models

The paper introduces a causal, layerwise audit method called the CoT Mediation Index (CMI) to evaluate whether chain-of-thought (CoT) prompting truly influences a language model’s internal computation. By comparing performance degradation from patching CoT-token hidden states against matched control patches, the authors find that CoT influence is often confined to narrow reasoning windows and can be nearly absent even when the model produces fluent rationales. The study shows that models explicitly tuned for reasoning exhibit stronger mediation, while Mixture-of-Experts models display more distributed mediation, indicating that CoT faithfulness varies across models and tasks.

By Anish Sathyanarayanan, Aditya Nagarsekar, Aarush Rathore
arXiv AI
Sep 17

PACT-WAM: Predicting Actions and Visual Foresight with Compact Temporal Encoding for Robot Manipulation

PACT‑WAM is a world‑action model that simultaneously predicts a 16‑step action trajectory and its corresponding visual forecast for robot manipulation. It uses a hierarchical history encoder that compresses past observations into fewer tokens, reducing processing cost by 75% compared to dense encoding. The model’s shared flow module updates action and visual states jointly, and a TiTok‑VAE decoder reconstructs multi‑view future images, which are then used by a vision‑language component (Proposal Review) to improve execution‑prefix selection and proposal rejection, boosting success rates on several benchmarks.

By Yushan Liu, Jingjing Fan, Shoujie Li, Yifan Xie, Xiao-Ping Zhang, Wenbo Ding
arXiv AI
Sep 17

CzechTopic: A Benchmark for Zero-Shot Topic Localization in Historical Czech Documents

The paper introduces CzechTopic, a human‑annotated benchmark for zero‑shot topic localization in historical Czech documents. It provides topics with manually annotated spans and evaluates models at both document and word levels, using human agreement as the reference. Experiments show wide performance differences among large language models, with the best models approaching human agreement while smaller distilled token‑embedding models remain competitive.

By Martin Kosteln\'ik, Michal Hradi\v{s}, Martin Do\v{c}ekal
arXiv AI
Sep 17

AMIGO: Agentic Multi-Image Grounding Oracle Benchmark

AMIGO (Agentic Multi-Image Grounding Oracle Benchmark) is a long-horizon evaluation framework for vision‑language models that tests hidden‑target identification across galleries of visually similar images. The benchmark requires a model to ask a sequence of attribute‑focused Yes/No questions, receiving Yes/No/Unsure feedback and penalizing invalid actions with Skip, thereby stressing question selection under uncertainty, constraint tracking, and fine‑grained discrimination. Using the Guess My Preferred Dress task, the study shows that final‑answer accuracy alone overstates performance, as models may guess correctly without verified evidence, waste turns, or violate the protocol, highlighting the need for combined visual discrimination, informative questioning, and robust protocol adherence.

By Min Wang, Ata Mahjoubfar
arXiv AI
Sep 17

HINTBench: Horizon-agent Intrinsic Non-attack Trajectory Benchmark

HINTBench is a new benchmark for evaluating agents’ intrinsic risk, comprising 596 trajectories (400 synthetic risky, 136 synthetic safe, 30 real risky, 30 real safe) with an average length of 24 steps. It supports three tasks—risk detection, risk-step localization, and intrinsic failure-type identification—using a unified five-constraint taxonomy. Experiments show a large performance gap: while large language models can detect risky trajectories, they score below 37 on strict-F1 for risk-step localization, and existing guard models transfer poorly to this setting.

By Jiacheng Wang, Jinchang Hou, Fabian Wang, Ping Jian, Chenfu Bao, Zhonghou Lv
arXiv AI
Sep 17

Schema-Key Wording as an Instruction Channel in Structured Generation under Constrained Decoding

The paper investigates how the wording of schema‑key tokens can serve as an implicit instruction channel in constrained decoding for structured generation. By treating structured generation as a multi‑channel instruction problem, the authors analyze when an instructional advantage is preserved after grammar projection and conduct experiments on GSM8K and Math500 across seven language models. Results show that changing only the schema‑key wording can significantly alter accuracy, with both positive and negative effects, and that prompt‑level and schema‑level instructions interact non‑additively.

By Yifan Le
arXiv AI
Sep 17

Multimodal Conditioning of Fine-Tuned Stable Diffusion XL for Controllable and Culturally Faithful Ulos Motif Generation

The paper introduces a multimodal generative framework that fine‑tunes Stable Diffusion XL with LoRA and couples it to a LLaMA 1.5‑7B language model to produce culturally faithful Batak Ulos motifs. Four conditioning mechanisms—text, image, representation, and semantic map—are jointly used to steer the generation, and an ablation study shows that combining all four actually degrades performance compared to selective combinations. A web prototype and user studies with weavers and the public demonstrate the system’s practical value for preserving and innovating Ulos design.

By Humasak Simanjuntak, Tamara Yunika Sianipar, Bronson T. M Siallagan, Difya Laurensya Ambarita, Arlinta Barus
arXiv AI
Sep 17

AeroWeaver: An Embodied-Agent Harness for Weaving Aerial Skills into Distributed, Adaptive Swarm Execution

AeroWeaver is a new embodied‑agent harness that integrates large language model (LLM) decision making with the executable skills of individual UAVs, enabling distributed, adaptive swarm execution. It connects semantic mission decisions to governed skills, organizes role‑conditioned local agents for coordination, and refines skill selection online using role‑indexed state‑action‑reward experience. Experiments demonstrate that AeroWeaver maintains valid skill execution without a central joint‑action generator and supports reward‑guided, training‑free adaptive learning from accumulated execution experience.

By Jiabin Lou, Yirong Yang, Haopeng Wang, Xuxin Lv, Xinyu Liu, Diyuan Hou, Xuehong Liu, Rongye Shi, Wenjun Wu
arXiv Machine Learning
Sep 17

Attention Dispersion as a Diagnostic Signal for Hallucination in Large Language Models

The paper proposes using the temporal volatility of internal attention mechanisms—measured by an unsupervised attention dispersion metric—as a diagnostic signal for hallucinations in large language models. It demonstrates that spikes in attention entropy within intermediate layers correlate with reasoning breakdowns, and shows statistically significant AUC improvements of up to +0.076 over output-based baselines on GSM8K and MATH-500 benchmarks using the Qwen2.5 model family.

By Shardul P. More, Tanuja S. Pawar