The paper introduces a 202-scenario benchmark to evaluate how large language models (LLMs) handle safety-critical authorization decisions for vehicle voice commands. It tests two local open-weight models and three API-based LLMs, finding alignment scores ranging from 40.1% to 89.1% and noting persistent false execution errors. The study concludes that structured LLM decisions alone are insufficient for safety, recommending an independent enforcement layer to verify tool permissions and vehicle-state constraints before any vehicle function is invoked.
By Diba Afroze, Xingli Zhang, Yazhou Tu, Xiali Hei
arXiv:2609.25373v1 Announce Type: new
Abstract: On-device assistants require function-calling models that map natural language to local system actions, but existing resources emphasize web APIs or na...
By Ali Rezagholizadeh, Soheila Samiee
arXiv:2608. 10692v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed as mobile assistants, where a key challenge is leveraging personal information scattered across multiple applications (apps) to complete user instructions.
By Junjie Ye, Zhuohui Sheng, Shaofan Liu, Yulun Zhu, Wenjie Fu, Dingwei Zhu, Ming Zhang, Yujiong Shen, Weichao Wang, Xin Zhao, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang, Pluto Zhou
arXiv:2608.23078v1 Announce Type: new
Abstract: Large language models increasingly operate over large collections of tools, functions, APIs, and specialized agents. As the candidate action space grow...
By Saurav Singla, Aarav Singla, Advik Gupta, Parnika Gupta
The paper introduces CRAFT, a data‑centric fine‑tuning approach that aligns small language models (SLMs) for pre‑hoc reasoning in AI‑native 6G radio access networks (RAN). By automatically generating verified (input, trace, label) triplets and fine‑tuning with low‑rank adaptation, CRAFT achieves high accuracy and F1 scores on TRACTOR and IC xApp datasets while avoiding parse failures that plague RL methods like GRPO. It also reduces energy consumption by 59% compared to GRPO baselines, offering a more sustainable path to auditable AI in 6G RAN.
By Pranshav Gajjar, Vijay K Shah
The next generation of mobile networks is envisioned as fully AI-native, with AI-RAN architectures embedding small language models (SLMs) to perform reasoning over real-time telemetry. The state-of-th...
arXiv:2608.22472v1 Announce Type: new
Abstract: Function calling represents the core capability of agentic large language models (LLMs). Existing research has focused on enhancing LLMs function-calli...
By Yalda Taheri, Mohammad Hassan Heydari, Erfan Naaman, Afsaneh Fatemi
arXiv:2610.17436v1 Announce Type: new
Abstract: Large language models (LLMs) on mobile devices can act as agents that call external software tools to complete complex tasks, but the major challenge o...
By Boyuan Yang, Wei Gao
The paper introduces Evidence‑Grounded Local Behavioral Prediction (ELBP), a four‑layer architecture that uses deterministic interaction processing, dual‑view behavioral representation, small language model (SLM) prediction, and evidence verification with abstention to forecast future assistance use. Evaluated on three instruction‑tuned SLMs, the best model (Qwen2.5‑3B) achieved an F1‑score of 0.80 and PR‑AUC of 0.84, outperforming raw‑event prompting and XGBoost by significant margins. With evidence verification and selective abstention, the system’s selective F1 rose to 0.84 while maintaining 91% coverage and 96% evidence support, demonstrating that structured behavioral representation and evidence‑verified prediction enhance reliability in on‑device SLM forecasting.
By Adeyemi Adeseye, Aisvarya Adeseye, Hannu Tenhunen, Jouni Isoaho
Agent Seer is a pipeline that automatically synthesizes realistic evaluation scenarios for AI agents that use external tools, using only the tool’s specification (function names, natural‑language descriptions, and typed parameter schemas). Starting from a single Model Context Protocol (MCP) specification, it enriches raw schemas, generates graded scenarios with synthetic tool outputs, and expands them into mock‑data‑grounded multi‑turn dialogues that demonstrate strong tool‑calling correctness and conversational coherence. Across seven diverse MCP specifications, the pipeline achieves high quality, with parameter‑schema complexity emerging as the main driver of quality variation and argument‑value accuracy identified as the dominant failure mode.
By Harish Karumuri, Mahesh Vemula, David Lopes Pegna
The paper presents a unified evaluation of seven open reasoning language models across four benchmarks (ARC-Challenge, GSM8K, MATH levels 1–3, and TruthfulQA MC1) using a consistent 238-example subset and three prompting strategies (zero-shot, chain-of-thought, few-shot CoT). It reports not only accuracy but also Wilson confidence intervals, latency, VRAM usage, weighted aggregate performance, Pareto-efficient points, prompt-sensitivity, and compatibility diagnostics, revealing that Gemma-4-26B-A4B tops the weighted score while Gemma-4-E4B offers a strong practical trade-off. The study emphasizes that model rankings shift with prompting strategy and that deployment trade-offs remain crucial, advocating for a deployment-aware, multi-objective evaluation framework rather than a single-score leaderboard.
By Md Motaleb Hossen Manik, Ge Wang
The paper introduces PTC-Decoder, a training‑free, plug‑and‑play decoder designed to improve the reliability of small language models (SLMs) on offline, resource‑constrained edge devices such as remote sensing satellites. PTC-Decoder combines a Plan‑to‑Act paradigm that forces the model to invoke a planning tool at the first inference step with a deterministic finite automaton (TC‑Decoder) that imposes token‑level hard constraints on tool names while preserving the model’s reasoning capability. Evaluated on 200 real satellite tasks across seven SLMs, the approach achieves a statistically significant mean overall score gain of +1.21, with the TC‑Decoder component identified as the primary contributor to performance gains.
By Minghui Yu, Ke Mu, Gang Wu