arXiv AI September 12, 2026 By Qibai Chen, Zeming Liu

SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics

SemVerBench is a benchmark that evaluates how well large language models (LLMs) understand and apply version-constraint resolution semantics, such as determining whether a version satisfies constraints like ^1.2.3 or >=2.0. The study finds that many models struggle with certain corner cases, with GPT‑5.1 performing poorly while Claude and Opus perform much better. The authors suggest that the failures stem from an activation/application gap rather than a lack of knowledge, and recommend that coding agents delegate version resolution to a dedicated resolver tool.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Aug 14

Large Language Models Can Follow Instructions, But Not Many at Once: Phase Transitions in Compositional Constraint Satisfaction

arXiv:2608. 12426v1 Announce Type: new Abstract: Large language models are increasingly deployed in settings that require simultaneous adherence to multiple explicit constraints - reasoning structure, safety boundaries, output schemas.

By Mariya I. Vasileva

llms benchmarks safety

arXiv AI

Jun 2

Don't Ask the LLM to Track Freshness: A Deterministic Recipe for Memory Conflict Resolution

arXiv:2606. 01435v1 Announce Type: new Abstract: LLM-based memory systems increasingly maintain facts that evolve over time, where a recurring failure is conflict resolution: when a fact has multiple contradictory values, which should the agent return?

By Vikas Reddy, Sumanth Challaram

llms rag agents

arXiv AI

Jul 8

RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications

arXiv:2607. 06411v1 Announce Type: cross Abstract: Developers increasingly delegate real maintenance work to product-grade coding agents, and many state tasks in their native language, in the style of a customer request rather than a curated English issue.

By Evgeny Shilov (Independent Researcher)

llms agents benchmarks

arXiv AI

Aug 6

Diagnosing Tool-Selection Reasoning in LLM Agents with Canary Tools

arXiv:2608. 04719v1 Announce Type: new Abstract: Agent evaluations tell us that a model picked the wrong tool, but rarely why.

By Atul Anand, Sourav Chattaraj

llms agents benchmarks safety

arXiv AI

Jul 28

TLA$^{+}$-Bench: An Execution-Grounded Benchmark and Dataset for Natural-Language to TLA+ Specification Generation

arXiv:2607. 23425v1 Announce Type: cross Abstract: Large language models increasingly write TLA$^{+}$ formal specifications from natural-language descriptions, but progress is hard to measure: existing resources grade by resemblance to a reference or by whether the output parses, neither of which shows correctness.

By Arslan Bisharat, Eric Spencer, Brian Ortiz, Khushboo Bhadauria, Mujtaba Nazari, Beatriz Santos, Anisa Ramos, TaiNing Wang, George K. Thiruvathukal, Konstantin L\"aufer, Mohammed Abuhamad

llms benchmarks

arXiv AI

Jul 28

Confidently Wrong: Exception Chain Collapse in Frontier LLM Rule Evaluation

arXiv:2607. 23386v1 Announce Type: new Abstract: We document a failure class in frontier large language models -- exception chain collapse -- observed in eligibility evaluation under nested conditional rules of the form "A is required UNLESS B applies, UNLESS C overrides B".

By Paul Simpson, John Kozak, Lisa Doake

llms benchmarks safety

One email a morning, machine-written

Related stories

Large Language Models Can Follow Instructions, But Not Many at Once: Phase Transitions in Compositional Constraint Satisfaction

Don't Ask the LLM to Track Freshness: A Deterministic Recipe for Memory Conflict Resolution

RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications

Diagnosing Tool-Selection Reasoning in LLM Agents with Canary Tools

TLA$^{+}$-Bench: An Execution-Grounded Benchmark and Dataset for Natural-Language to TLA+ Specification Generation

Confidently Wrong: Exception Chain Collapse in Frontier LLM Rule Evaluation