arXiv AI

Reasoning Shortcuts and Value Symmetries: What Symmetry Permits, Architecture Realizes, and Optimization Selects

arXiv:2608. 10420v1 Announce Type: new Abstract: Reasoning shortcuts are solutions of a neurosymbolic system's rules that produce correct predictions through unintended concepts.

arXiv AI
Sep 24

Not What You Meant: Can LLMs Follow a Specified Negation Semantics?

The paper investigates how large language models (LLMs) interpret negation across different logical semantics—open‑world vs. closed‑world, two‑ vs. three‑valued, and credulous vs. skeptical reasoning. Using the newly introduced NAFBench, a procedural generator that creates solver‑certified logic programs and their natural‑language verbalizations, the authors evaluate LLMs on four semantic viewpoints (SLDNF, well‑founded semantics, and stable‑model semantics). Results show a persistent gap: even the strongest models achieve only 59–74% accuracy, with many models sensitive to rule ordering and prone to overcommitment on undefined cases, though some frontier models reach near‑perfect performance on a fixed‑complexity set. "whyItMatters":"The study highlights that current LLMs struggle to reliably follow explicitly specified negation semantics, underscoring a limitation in their logical reasoning capabilities."

By Qiming Bao, Agnieszka Mensfelt, Michael J. Witbrock, Kostas Stathis
arXiv AI
Sep 12

SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics

SemVerBench is a benchmark that evaluates how well large language models (LLMs) understand and apply version-constraint resolution semantics, such as determining whether a version satisfies constraints like ^1.2.3 or >=2.0. The study finds that many models struggle with certain corner cases, with GPT‑5.1 performing poorly while Claude and Opus perform much better. The authors suggest that the failures stem from an activation/application gap rather than a lack of knowledge, and recommend that coding agents delegate version resolution to a dedicated resolver tool.

By Qibai Chen, Zeming Liu