arXiv AI
HARISSA: Inference-Time Self-Checks for Efficient and Safe Local Language Model Deployment
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Sep 22
this-that-model-1.0: A typed decision model that decides in 30 ms, for a millionth of a cent
arXiv:2609.23886v1 Announce Type: new Abstract: Software delegates more of its branches to models every year: which queue a ticket enters, whether a command is safe to run, whether a claim clears wit...
arXiv AI
Aug 17
Not All Tokens Are Equal: Inflation-Aware Routing for Agentic LLM Systems
arXiv:2608. 13571v1 Announce Type: cross Abstract: When a language model fails to answer a query on the first attempt, an agentic system retries, consuming additional tokens each time.
arXiv Machine Learning
Sep 15
One Size Does Not Fit All: Setting Inference Depth from the Questions a Deployment Actually Asks
arXiv:2609.14144v1 Announce Type: cross Abstract: A transformer language model is trained to respond to any prompt, but each deployment asks only a narrow range of questions: a support assistant sees...
arXiv AI
Aug 25
More Accurate or More Efficient? Evaluating Locally Deployed Compact Open-Weight Language Models for Mathematical Reasoning
arXiv:2608.22048v1 Announce Type: new Abstract: Large language models are increasingly deployed on local hardware for privacy, cost, and accessibility reasons. Yet many evaluations emphasize accuracy...
arXiv Computation and Language
Aug 27
Beam Search, Self-Consistency, and the Limits of Inference-Time Scaling for Grammar-Constrained Text-to-SQL in Small Language Models
The paper investigates how increasing inference-time computation—via wider beam search or sample‑plus‑vote—affects performance on grammar‑constrained text‑to‑SQL tasks for small language models. Using the Qwen2.5‑Instruct family (0.5B–7B parameters) on the Spider benchmark, the authors find that larger models consistently outperform higher inference compute on the same model size, and that beam search yields better accuracy than sample‑plus‑vote under matched budgets. These results suggest that, unlike unconstrained settings, scaling inference compute does not compensate for smaller model size when strict grammar constraints are applied.
arXiv AI
2d ago
Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost
The paper introduces Galahad, a memory layer that stores a transformer language model’s key‑value state for blocks of text, allowing subsequent requests to reuse previously computed attention rather than recomputing it. On seven real‑world datasets, 98.7% of prompt tokens were already read, and with Galahad the model could attend to an entire 97,000‑token corpus, achieving 98–100% recall on a 100‑fact test while reducing inference time and energy consumption dramatically. The approach was validated across 30 models and all runtimes, demonstrating that stateful inference can replace stateless serving without loss of accuracy.