The paper introduces LocalIntent, a structured generation framework that routes smart building automation requests to either ordinary or persistent-rule branches using locally deployed large language models. LocalIntent improves outcome-specific prompting, yielding higher accuracy in intent understanding, device identification, and action/parameter correctness compared to generic two-stage prompting, especially for rule-based requests. Experiments on Gemma and Qwen models (2B‑9B) across 101 non‑persistent and 54 persistent‑rule records demonstrate significant performance gains in rule-mode correctness and overall intent handling.
arXiv:2607. 18034v1 Announce Type: new Abstract: Smart home assistants interpret a wide range of user commands, from explicit device control to underspecified and preference dependent requests.
arXiv:2607. 26922v1 Announce Type: new Abstract: Multi-agent LLM pipeline systems break down the task among multiple roles for better reasoning, but are benchmarked mainly with large-scale commercial models.
arXiv:2606. 01099v1 Announce Type: cross Abstract: Command understanding systems in smart home ecosystems can automate device control and substantially improve user experience.
arXiv:2505.13546v2 Announce Type: replace-cross Abstract: Automatic prompt generation plays a crucial role in enabling general-purpose multi-agent systems to perform diverse tasks autonomously. Exist...
LOGIC is a benchmark and evaluation framework that tests how language models can ground engineering requests in a deterministic inventory of candidate changes before propagating selected changes through an electrical traceability graph. The benchmark includes 168 scenarios—144 for selection and 24 for abstention—and evaluates three 7–8B models against intent‑agnostic, lexical, and structured‑evidence methods. Results show that structured evidence can achieve perfect candidate F1 on anchored cases, while large language models perform better on relational‑paraphrase cases; however, grounding accuracy drops as candidate inventories grow, and strict evidence gating reduces false positives but may also remove correct selections.
arXiv:2606. 18636v1 Announce Type: cross Abstract: Recent advancements in Large Language Models (LLMs) have empowered home assistants with natural language interaction capabilities.
The paper introduces STEPGATE, an uncertainty‑aware handoff framework that evaluates each step of a small language model (SLM) agent and selectively escalates difficult steps to a stronger model. On a 52‑task single‑step benchmark, the Qwen2.5‑1.5B/7B pair achieved 82.7% task success with only 30.8% escalation, outperforming local‑only and random escalation baselines. In multi‑turn tests, STEPGATE reached 69.0% trajectory success and 84.0% action success while using only 30.0% cloud actions, demonstrating that step‑level escalation can close much of the performance gap to a stronger backend with fewer remote tokens.
The paper evaluates the planning abilities of large language model (LLM) agents by comparing their performance against formal semantics defined in hidden PDDL models. Across five domains, four LLMs, six prompting strategies, and three agent architectures, the best setup solved 78% of solvable tasks offline but only 40% online, with limited recognition of unsolvable tasks and no clear benefit from more elaborate prompting. The study concludes that claims of LLM planning competence require validated action traces rather than end‑to‑end success rates.
The paper introduces a 202-scenario benchmark to evaluate how large language models (LLMs) handle safety-critical authorization decisions for vehicle voice commands. It tests two local open-weight models and three API-based LLMs, finding alignment scores ranging from 40.1% to 89.1% and noting persistent false execution errors. The study concludes that structured LLM decisions alone are insufficient for safety, recommending an independent enforcement layer to verify tool permissions and vehicle-state constraints before any vehicle function is invoked.
arXiv:2606. 02965v2 Announce Type: replace Abstract: As large language models gain tool access and are deployed as autonomous agents capable of editing records, executing transactions, and modifying infrastructure, we still evaluate them based on the sole metric of task completion.
arXiv:2606. 29116v1 Announce Type: new Abstract: Large Language Models (LLMs) are rapidly being adopted in low-code and no-code automation platforms, where non-expert users design workflows that combine natural language understanding with external services and APIs.
The paper "Can You Check That? The Checkability Boundary for Local LLM Network Automation" proposes a method called Touchstone that uses local small language models (SLMs) to generate network‑automation candidates and applies task‑specific intrinsic checks to reject incorrect outputs before escalating to a larger frontier LLM. By defining a task as checkable when a cheap, deterministic test can reject outputs violating a necessary correctness condition, the authors demonstrate that Touchstone achieves high end‑to‑end accuracy (98.6% on conflict detection and 93.8% on intent translation) while escalating only a small fraction of inputs. The study shows that local inference is preferable when precise, low‑cost checks are available, whereas tasks lacking such checks should rely on the frontier model.