The paper evaluates two large language models, Claude Sonnet 4.5 and Claude Opus 5, on the bidirectional English Resource Grammar (ERG) tasks of generating English from Minimal Recursion Semantics (MRS) and parsing English into MRS. In generation, Opus achieves 76.3 BLEU—surpassing a 72k‑pair trained system and matching a million‑pair system—while Sonnet scores 65.7 BLEU, rising to 69.6 when selecting from ACE’s candidates. In parsing, both models lag behind ACE, attaining only 57.2 and 65.5 F₁ respectively, with exact‑match on about 1 % of sentences, highlighting that high generation scores do not guarantee accurate semantic parsing.
By Soham Dan
arXiv:2604. 26157v4 Announce Type: replace-cross Abstract: Structural generalization in semantic parsing requires systems to apply learned compositional rules to novel structural combinations.
By Zichao Wei
The paper introduces domain-specific languages (DSLs) as an explicit representation layer to bridge the gap between high-level natural-language abstractions and concrete actions. Experiments on the visually-grounded “Hexagons” task and the “Text2CAD” benchmark show that code generation using a human-curated DSL yields dramatic performance gains, especially at higher abstraction levels, while automatically induced DSLs only partially recover these gains. The study highlights that the right representation can close much of the performance gap caused by abstraction, but that automatically learning such representations remains a challenge.
By Amit Lisha, Reut Tsarfaty
arXiv:2604.16656v2 Announce Type: replace
Abstract: All languages are equal; when it comes to tokenization, some are more equal than others. Tokens are the hidden currency that dictates the cost and...
By Maitrey Mehta, Nishant Subramani, Zhichao Xu, Ashim Gupta, Vivek Srikumar
arXiv:2606. 25987v1 Announce Type: cross Abstract: Large language models (LLMs) attain remarkable surface fluency on code, yet they neither formally guarantee the syntactic validity of their output nor leverage the hierarchical structure defining the target language.
By Alexandre Bouayad
The paper investigates natural language autoencoders (NLAs) as tools for verbalizing activation differences in large language models. It examines whether NLAs capture compositional structure, generalize to fine‑tuned models, recover fine‑tuning objectives, and can be specialized for model diffing. Experiments across four models show that NLAs largely succeed in these tasks, though they do not yet match state‑of‑the‑art diffing methods on emergent misalignment detection.
By Andrea Blasi N\'u\~nez, Gianluca Barmina, Federico Torrielli, Thiusius Rajeeth Savarimuthu, Peter Schneider-Kamp, Lukas Galke Poech
The paper introduces a drill‑down evaluation framework that breaks down Text‑to‑SQL queries into simpler executable sub‑queries and natural‑language questions, enabling analysis of hallucination patterns along the generation path. It distinguishes recurrent hallucinations—those appearing in both early and final steps—from emergent hallucinations that only surface at the end. Experiments on five large language models and an expanded BIRD‑mini benchmark reveal varying recurrence and emergence probabilities across categories, with conversational history generally increasing recurrence and decreasing emergence.
By Hayden Moore, Abhinav Verma, Anshuman Dutt
arXiv:2608. 10137v1 Announce Type: cross Abstract: Grammar Constrained Decoding (GCD) forces Language Models (LMs) to produce syntactically valid outputs by masking out non-conforming tokens at each step.
By I\c{s}{\i}l \"Ozg\"u, Yaoxuan Wu, Guy Van den Broeck, Miryung Kim
arXiv:2609.23742v1 Announce Type: new
Abstract: Small open-source large language models (LLMs) in the 0.6B-4B parameter range are increasingly deployed for structured output generation (JSON, functio...
By Akash Chavan
arXiv:2511. 14967v3 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown great promise in generating structured diagrams from natural language descriptions, particularly Mermaid sequence diagrams for software engineering.
By Basel Shbita, Farhan Ahmed, Chad DeLuca
The paper investigates natural language autoencoders (NLAs) that convert large language model activations into readable text. It examines whether NLAs capture compositional structure, generalize to fine‑tuned models, recover fine‑tuning objectives, and can be specialized for model diffing. Experiments across multiple models show positive results: NLAs preserve compositionality, transfer to lightweight fine‑tunes, recover hidden objectives, and outperform a black‑box baseline on diffing tasks, though they lag behind state‑of‑the‑art methods for misalignment detection.
arXiv:2601. 22588v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are widely used as reference-free evaluators via prompting, but this "LLM-as-a-Judge" paradigm is costly, opaque, and sensitive to prompt design.
By Zhuochun Li, Yong Zhang, Ming Li, Yuelyu Ji, Yiming Zeng, Ning Cheng, Yun Zhu, Yanmeng Wang, Shaojun Wang, Jing Xiao, Daqing He