Large language models

Model releases, architecture work and prompting research on large language models — from frontier-lab announcements to the arXiv papers behind them.

29,760 stories · RSS feed

arXiv Computation and Language
Sep 25

Benchmarking Argumentative Behaviour of LLMs: A Study of Defences Against Character Attacks

The paper investigates how well large language models (LLMs) can handle character attacks—ad hominem arguments—in political debates. By analyzing natural political dialogues and comparing LLM-generated responses to a corpus of U.S. presidential debates, the study finds that most LLMs favor logical defenses and rarely use ethos-based counterattacks. The authors suggest that safety fine‑tuning limits LLMs’ strategic options, preventing them from fully engaging in realistic political discourse.

By Ewelina Gajewska, Katarzyna Budzynska, Jaroslaw Chudziak
arXiv AI
Sep 25

Canopy: Exploiting Piecewise Smooth Tree Priors for Multi-Fidelity Bandits

CANOPY is a multi‑fidelity tree bandit algorithm that learns where a piecewise‑smooth prior holds instead of assuming global smoothness. It uses cheap random‑path probes to certify local aggregation bias and then focuses expensive leaf evaluations on cells where smoothness is violated. The method achieves provable fixed‑budget and regret guarantees that scale with the number of discontinuities, matching smooth‑tree rates when no violations exist and approaching structure‑blind search when violations are dense.

By Michael Jerge, Suman Jana
arXiv AI
Sep 25

The Troy Moment: How LLM Agents Adjudicate the Decision Point Under Impossible Tasks, Claimed Authority, and Peer Information

The paper investigates how large language model agents decide whether to persist, stop, or escalate when faced with impossible software‑repair tasks that also involve conflicting test requirements. Using ImpossibleBench tasks and models such as GPT‑5.6 Sol, Claude Fable 5.1, and Gemini 3.8 Flash, the study varies peer precedent, forged authority claims, instruction wording, and tool friction to observe differing adjudication policies. The authors propose a conflict adjudication framework that maps information to interpretation to action, arguing it better captures agent alignment under competing pressures.

By Ivy Zhang
arXiv AI
Sep 25

TabSieve: Explicit In-Table Evidence Selection for Tabular Prediction

TabSieve is a select‑then‑predict framework that explicitly chooses a small set of informative rows from a table as evidence before predicting a missing target. The authors build a large synthetic dataset, TabSieve‑SFT‑40K, and introduce a reinforcement learning method, TAB‑GRPO, to jointly optimize evidence selection and prediction. Experiments on 75 classification and 52 regression tables show consistent performance gains, with TabSieve improving classification by 2.92% and regression by 4.45% over the best baseline while enhancing robustness to noisy context.

By Yongyao Wang, Ziqi Miao, Lu Yang, Haonan Jia, Wenting Yan, Chen Qian, Lijun Li
arXiv Computation and Language
Sep 25

CONSISTRE: A Unified Consistency-Aware Framework for Document-Level Relation Extraction with Large Language Models

CONSISTRE is a consistency‑aware framework for document‑level relation extraction that tackles contradictions in large language model predictions. It offers two tracks: an inference‑time track that refines black‑box LLM outputs through constraint‑aware prompting, verification, and self‑reflection, and a training‑time track that distills consistency knowledge into smaller open‑source models via supervised fine‑tuning and reinforcement learning. Experiments on DocRED show both tracks outperform baselines, with the inference‑time track matching competitive F1 scores and the training‑time track narrowing the performance gap to proprietary LLMs while reducing inference cost.

By Mingxuan Sun
arXiv Computer Vision
Sep 25

CinematicVQA: Benchmarking Film-Grammar Reasoning in Large Vision-Language Models

CinematicVQA is a new benchmark for evaluating large vision‑language models on film‑grammar reasoning. It introduces the Cinematic Scene Graph, a structured representation linking filming techniques to perceptual effects and narrative functions, and tests models on tasks beyond low‑level technique recognition. The study finds a semantic gap where models excel at describing visuals but struggle to identify underlying techniques, and shows that fine‑tuning improves performance on narrative function and multi‑hop reasoning.

By Shuo Xing, Pooja Verlani, Balu Adsumilli, Zhengzhong Tu
arXiv AI
Sep 25

A Living Benchmark for Information Retrieval from Electronic Health Records

The paper introduces BRIE, a continuously maintainable benchmark for evaluating large language models (LLMs) in electronic health record (EHR) information retrieval. It presents a scalable framework that automatically generates question–answer pairs from longitudinal EHR notes, validated by nineteen clinicians. The benchmark allows assessment of multiple inference strategies and highlights that state‑of‑the‑art LLMs often miss clinically important information, especially when synthesis across documents is required.

By Jordan L. Cahoon, Chloe O. Stanwyck, Sulaiman Somani, Philip Chung, Kevin R Keet, Kameron C. Black, Andrea T. Fisher, Sarita Khemani, Jerry Liu, Stephen Ma, Saloni K. Maharaj, Rita M. Pandya, Eduardo Perez-Guerrero, Priyanka Pillai, Lisa Shieh, David J. H. Wu, James Xie, James C. McAvoy, Teresa Nguyen, Jessica Tran, Lucy Yin, Bridget Lin, Alison Callahan, Jason A. Fries, Nigam H. Shah, Emily Alsentzer
arXiv AI
Sep 25

DrGait: Biomechanically Grounded Visual Reasoning for Interpretable Clinical Gait Analysis

DrGait is a training‑free framework that transforms Vision‑Language Models into clinical planners for gait analysis. It separates semantic reasoning from geometric perception using a Triage‑Verification‑Synthesis workflow, where hypotheses are generated, verified with deterministic biomechanical tools, and refined in a closed‑loop. This approach reduces hallucinations and produces transparent, audit‑ready clinical reports with competitive diagnostic accuracy.

By Xiangyu Yin, Shiqi Wang, Abrar Alamri, Yasir Aljohani, Weichen Liu, Goeran Fiedler, Wei Gao
arXiv Computation and Language
Sep 25

Measuring Brand and Source Discovery under Repeated LLM Queries: A Finite-Sample Audit

The study audits large language model (LLM) outputs by measuring how well repeated queries recover a collected set of responses versus the full set of possible outputs. Using sample-based rarefaction on 4,500 responses from 50 buying questions across six configurations, the authors find historical-dictionary median recovery rates between 92.6% and 95.2%, which drop to 89.5%–94.7% after re‑adjudicating all candidate strings. Additional analyses with Gemini 3.1 Pro annotations and matched roster data confirm that recovery percentages vary with extraction methods, question selection, and the finite reference collection, underscoring the need for explicit measurement definitions and sensitivity analyses in LLM audits.

By Dmitrij \.Zatuchin
arXiv AI
Sep 25

AI in Science: Early Insights

The paper "AI in Science: Early Insights" analyzes AI’s impact on scientific work using data from 15 million Gemini interactions, 2,600 specialized AI models, and a survey of 600 scientists. It finds widespread AI adoption, complementary use of large language models and specialized tools, significant productivity gains of about seven hours per week, and a shift in research bottlenecks toward hypothesis backlog and verification needs. The study suggests AI can boost scientific productivity but its full effect depends on addressing new downstream challenges.

By Mihai Codreanu, Alex Imas, Juan Mateos-Garcia, Joseph Emmens, Evalyne Muiruri, Arthur Turrell, Julian Jacobs, Atoosa Kasirzadeh, Ana Trisovic, Yiyuan Chen, Tanya Rodchenko, Catherine Pollard, Scott Strand, Daniel Rock, Zanna Iscenko, Fabien Curto Millet, Neil Thompson, James Manyika
arXiv AI
Sep 25

Rufus-Air: An Open LLM Post-Training Recipe

Rufus-Air is an open, reproducible post‑training recipe for the GLM‑4.5‑Air‑Base model, structured as a serial pipeline of eight stages: SFT, Reasoning RL, Coding RL, Instruction‑Following RL, General Agent, Coding Agent, Search Agent, and RLHF. The paper documents the data, reward design, infrastructure, stage order, and stagewise results required to reproduce the recipe, emphasizing that stages progress from basic to advanced capabilities and from hard, verifiable rewards to softer judge‑based signals. Key findings highlight the importance of diverse, high‑quality SFT, difficulty filtering, reward reliability for stage ordering, and the role of infrastructure and engineering choices.

By Chia-Yuan Chang, Renyuan Cheng, Rui Feng, Xiaotian Han, Yuan He, Hongye Jin, Linwei Li, Shiyang Li, Fenglin Liu, Xin Liu, Priyanka Nigam, Haoyang Wen, Zhenghao Xu, Zhuocheng Xu, Bing Yin, Qingyu Yin, Chao Zhang, Rongzhi Zhang, Zhihan Zhang, Zixuan Zhang, Zixuan Zhang, Tuo Zhao
arXiv AI
Sep 25

Detecting Glaucoma Across Multi-ethnic Myopic and Non-Myopic Populations Using an Uncertainty-Aware Vision Transformer: A Multicentre Model Development and Validation Study

The study developed a Vision Transformer-based deep learning model with uncertainty estimation to detect glaucoma from colour fundus photographs across multi‑ethnic populations, including those with high myopia. Using 56,483 images for training, the model achieved an internal AUROC of 98.7% and maintained high performance (AUROC 86.4–99.6%) on 16 external datasets from eight countries. In high‑myopia eyes, the model outperformed ophthalmologists and matched specialists when full clinical data were available.

By Raghavan Lavanya, Yangqin Feng, Ten Cheer Quek, Quan V. Hoang, Linda Yi-Chieh Poon, Jost B. Jonas, Ya Xing Wang, Vinay Nangia, Jin Wook Jeoung, Sehie Park, SoYeon Kim, Benjamin Y Xu, Sreenidhi Iyengar Munimadugu, Paul Mitchell, Gerald Liew, Yanin Suwan, Jirayu Hong-amata, Sahil Thakur, Monisha E Nongipur, Tina Wong, Rahat Husain, Ng Si Rui, Yamon Syn, Phey Feng Lo, Nicholas Tan Yi Qiang, Shaista Hussain, Xiaofeng Lei, Zhi Da Soh, Marco Yu, Haslina Hamzah, Zizhou Wang, Yan Wang, Liangli Zhen, Xinxing Xu, Tien-Yin Wong, Tin Aung, Rachel S Chong, Yong Liu, Ching-Yu Cheng
arXiv Machine Learning
Sep 25

Return or Revise? Learning When Revision Helps Retrieval-Augmented QA

The paper investigates when it is better to return an existing draft answer or revise it using retrieved evidence in retrieval‑augmented QA systems. By grading both the draft and its candidate revision with the same correctness judge, the authors define a paired effect called recoverability and train policies to predict it before revision. Experiments on 25,870 open‑domain questions show that a recoverability‑based scorer outperforms a draft‑correctness scorer across multiple Llama setups, improving accuracy–revision trade‑offs and closing a significant portion of the oracle gap, though it still applies harmful revisions in a substantial fraction of cases.

By Nicholas Kashani Motlagh, Tim Anderson, Jeremy Gwinnup, Grant Erdmann
arXiv Computation and Language
Sep 25

Do not be greedy, Think Twice: Sampling and Selection for Document-level Information Extraction

The paper introduces ThinkTwice, a framework that uses sampling and selection to improve document‑level information extraction. Instead of relying on greedy decoding, the method generates multiple candidate templates and selects the best one via an unsupervised agreement‑based approach or a supervised reward model. It also proposes a rejection‑sampling technique to create silver training data for reasoning traces, and experimental results show that ThinkTwice consistently outperforms greedy baselines and the current supervised state‑of‑the‑art.

By Mikel Zubillaga, Oscar Sainz, Oier Lopez de Lacalle, Eneko Agirre
arXiv Computation and Language
Sep 25

Persona Prompting in Multimodal Urban Perception: Descriptive Convergence and Interpretive Variation

The paper investigates how persona prompting influences the language produced by two multimodal large language models—Qwen3‑VL and Gemma4—when describing urban scenes. Outputs are categorized into descriptive grounding (captions), perception tags, and interpretive framing (justifications). Results show captions largely converge across persona profiles, while justifications differ markedly, especially along economic status, political orientation, and personality dimensions, with economic status producing the greatest variation. Perception tags also reflect attribute similarities, and exploratory topic analysis indicates persona‑specific evaluative emphasis. Overall, persona prompting has a stronger effect on interpretive framing than on descriptive grounding.

By Neemias da Silva, Matt Ratto, Myriam Delgado, Rodrigo Minetto, Daniel Silver, Thiago H Silva
arXiv AI
Sep 25

Planning Takes More Than Token Prediction: Causal Plan for Benchmarking and Building Physically Grounded Embodied Reasoners

The paper argues that current embodied vision‑language planning benchmarks favor linguistic next‑token prediction over physically grounded next‑state reasoning, leading models to rely on language priors rather than true causal dependencies. To address this, the authors introduce Causal‑Plan‑Bench, a diagnostic suite covering four causal dimensions, and Causal‑Plan‑1M, a million‑scale corpus of explicit causal reasoning traces extracted from egocentric videos. Extensive experiments show that existing models perform poorly on these tasks, while a new model trained with a tailored recipe—Causal Planner based on Qwen3‑VL‑8B—achieves significant gains, demonstrating the feasibility of physically grounded causal reasoning.

By Zheng Lu, Mingqi Gao, Qinlei Xie, Wanqi Zhong, Hanwen Cui, Zirui Song, Lijie Wang, Chong Luo, Bei Liu, Yiming Li
arXiv Machine Learning
Sep 25

RLVR landscapes for iterated multiplications can be benign: Insights from spin-glass theory

The paper investigates the optimization landscape of reinforcement learning with verifiable rewards (RLVR) on algorithmic tasks such as iterated group and quasigroup multiplication. By mapping entropy‑regularized RLVR with myopic tabular policies to a spin‑glass energy model, the authors prove that for a broad class of models with uncorrelated inputs the landscape is benign—there are no local minima that can trap training. They identify practical obstacles like diffusive barriers and gradient‑estimation errors, show these can be mitigated by choosing an appropriate entropy regulator, and demonstrate that a transformer trained from scratch with only last‑token rewards can learn an algorithmic chain of thought for iterated non‑Abelian group multiplications.

By Noa Rubin, Zohar Ringel
arXiv Machine Learning
Sep 25

An Exposition of GPT Astra's Proof of Lower Bound on DP Continual Counting

The note provides a detailed proof of Astra’s lower bound for differentially private continual counting, building on recent work by Harrison and Leeman. It discusses earlier results, including a Ω(√{3}√{log(n)}) bound by Bairaktari and Larsen and their subsequent Ω(log^2(n)) bound for pure differential privacy. The authors aim to offer a more natural and accessible proof, hoping to aid further research in the area.

By Jalaj Upadhyay
arXiv Computation and Language
Sep 25

LLM surprisal is necessary but not sufficient to capture English garden-path effects: Evidence from joint latent modeling of reading paradigms

The study introduces a latent‑process multinomial processing tree (MPT) model to analyze human reading and comprehension of garden‑path sentences across four reading paradigms (eye tracking, uni‑ and bidirectional self‑paced reading, Maze). The model separates the likelihood of an incorrect initial analysis, the cost of encountering an incompatible continuation, and the cost of syntactic reanalysis, yielding more realistic parameter estimates when inattentive trials are considered. Cross‑validation shows that this MPT model predicts human reading patterns and end‑of‑trial judgments better than a model relying solely on large‑language‑model (LLM) surprisal, and that incorporating surprisal as an additional predictor further improves fit.

By Dario Paape, Tal Linzen, Shravan Vasishth
arXiv Computation and Language
Sep 25

Combating Instruction Conflict via Energy-Driven Latent Conflict Detection

The paper introduces ELCD, a latent conflict detector that verifies LLM outputs after generation to catch instruction conflicts that static input checks miss. ELCD builds a hidden-state representation from the final-token embedding and the mean-pooled response embedding, then trains a pairwise margin ranking objective to distinguish compliant from drifting responses. Experiments on five large language models show ELCD outperforms baselines, boosting PR-AUC for Llama‑2‑7B by ~30 percentage points and cutting FPR95 for Mistral‑7B to 2.67%.

By Mingyu Ma, Yuxin Wu, Jingbo Wang, Tianxiao Huang, Leixin Sun, Xiaochuan Shi