← Back to all news
arXiv Computation and Language By Zehao Lin, Qianfeng Shen, Xixuan Hao, Shaobo Cui, Chunyu Li, Kai Chen, Bo Tang, Zhenjun Wang, Jin Yan, Zhiyu Li, Feiyu Xiong

Compiling Autoregressive LLM Control into Policy-Exact Parallel Inference for Hierarchical RAG

Read the original on arXiv Computation and Language →

The paper presents a method for compiling autoregressive large language model (LLM) control into a policy‑exact parallel inference scheme for Hierarchical Retrieval‑Augmented Generation (RAG). By constructing a shallow action‑prefix tree and using a single root prefill plus tree‑causal probes, the authors eliminate the need for serial decoding while preserving the exact control policy in exact arithmetic. Experiments on MetaX devices show that the compiled approach matches the serial policy on 84–89 % of questions and reduces control latency by 35–47 % compared to token‑wise serial decoding.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Aug 25

Withholding the Completing Chunk: Exact Release-Boundary Equivalence for Production Streaming Guardrails

The paper presents a production streaming guardrail system that ensures exact release‑boundary equivalence for language‑model outputs. It compiles regular‑language predicates into persistent NFAs, distinguishes stable from provisional states, and applies document‑order priority to decide before each chunk release. Evaluations on over 200,000 partitioned cases show zero mismatches and demonstrate that incremental matching can outperform native regex at larger chunk sizes while remaining competitive at smaller ones.

By Christopher M. Frost
safety
More like this →
arXiv Computation and Language
3d ago

When Copying Is Hard: Copy-Constrained Decoding for Exact Span Reproduction

The paper introduces CopyGen, a lightweight copy-constrained decoding plugin for frozen large language models that improves exact span reproduction. CopyGen decomposes copying into three decisions—when to copy, how far to copy, and when to stop—using adaptive logit adjustments during verification to emit the longest valid prefix. Across three LLM backbones and four benchmark tasks, it boosts average exact match by 35% relative without finetuning and further improves by 4% with SFT, while also accelerating generation.

By Jinghui Zhang, Lang Gao, Zongfang Liu, Ruihong Zeng, Zirui Song, Rui Yan, Kentaro Inui, Xiuying Chen
llmsragnlpfine-tuningbenchmarks
More like this →
arXiv AI
Jul 22

Decode-Time Grammars: Constrained LLM Generation over a Refinement Order of Grammar Fragments

arXiv:2607. 18357v1 Announce Type: cross Abstract: Large language models now write a growing share of the world's code, increasingly inside agents and serving systems that compile, execute, or dispatch generated code without line-by-line review.

By Shuoming Zhang, Ruiyuan Xu, Haofeng Li, Qiuchu Yu, Yangyu Zhang, Chunwei Xia, Xiaobing Feng, Chenxi Wang, Huimin Cui, Jiacheng Zhao
llmsagentsdata-engineering
More like this →
arXiv AI
Jul 15

GRID: Grammar-Railed Decoding for Enterprise SQL Generation

arXiv:2607. 11951v1 Announce Type: new Abstract: Large language models can write SQL, but enterprise deployment demands more than plausible text: outputs must be syntactically valid, must respect per-role and per-schema policy, must carry provable (not best-effort) guarantees, must not slow down as generations grow, and must leave a compliance-grade record of every decision.

By Mohsen Arjmandi
llmsbenchmarksdata-engineering
More like this →
arXiv AI
Aug 5

Approximate Speculative Decoding

arXiv:2608. 03447v1 Announce Type: cross Abstract: Speculative decoding accelerates autoregressive generation by verifying a draft block with a target model in parallel.

By Yuannuo Feng, Zegang Peng, Yuxin Xie, Yubing Ye, Yizhe Chen, Wenshuai Yao, Wenyong Zhou, Wang Kang
fine-tuningefficiency
More like this →