Tool use, function calling, orchestration and the protocols that let models act rather than only answer.
arXiv:2609.24470v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are increasingly explored as interfaces for scientific image analysis, where a visual question-answering (VQA)...
arXiv:2609.24526v1 Announce Type: new Abstract: Physical AI requires models to ground visual and linguistic understanding in real-world environments while accounting for environmental constraints and...
arXiv:2609.24187v1 Announce Type: cross Abstract: Autonomous endoscopic navigation requires the policy model to predict actions from texture-poor monocular observations, make safe control decisions,...
arXiv:2609.24576v1 Announce Type: cross Abstract: Modern Vision-Language Navigation (VLN) models rely mostly on pre-trained large Vision-Language Models (VLMs) to predict navigation actions. While th...
Contrastive World Models propose a new method for learning latent dynamics without pixel reconstruction. By replacing observation reconstruction with a Deep InfoMax-like objective that maximizes mutual information between state-action sequences and local patch features of future observations, the approach encourages state representations to retain predictive information while ignoring visually irrelevant details. Experiments show that this method matches existing baselines in simple settings and significantly outperforms them when distractors or natural video backgrounds are present, while also training more efficiently by eliminating the pixel decoder.
CTRL is a new framework for time‑series forecasting that separates semantic reasoning from quantitative prediction. It uses a frozen backbone to produce base forecasts, while LLM agents act as controllers that analyze prediction errors by decomposing them into trend, seasonal, and irregular components. The agents generate compact control signals that a lightweight residual decoder uses to correct the forecasts, and the system can adapt at test time to distribution shifts with only a few LLM calls.
The paper introduces the Situated Identity Test (SIT), a framework that assesses whether a language model’s behavior can be traced to a specific developmental lineage rather than merely imitating a persona. SIT requires agents to possess accurate knowledge of their recorded experiences while appropriately ignoring ungrounded information, and it demonstrates that policies based only on compressed profiles are limited in distinguishing between colliding life histories. The authors present SITBench, an evaluation suite with 25 profile‑collision pairs and 10,000 probes across nine model architectures, and provide open‑source tools and pilot results on state‑of‑the‑art foundation models.
The paper introduces UniK, a universal knowledge perception platform designed to serve both digital AI—such as chatbots and agent workflows—and physical AI, which controls robots and autonomous systems. UniK handles the entire knowledge lifecycle—ingestion, enrichment, indexing, retrieval, and continuous evaluation—across diverse modalities including text, video, molecular data, and sensor telemetry, without task‑specific fine‑tuning. In five digital AI domains, UniK paired with a 70‑billion‑parameter model matches or surpasses larger proprietary LLMs, achieving high retrieval‑augmented generation accuracy on government data, medical QA, and chemistry tasks, and it also addresses similar data challenges in physical AI world‑model training.
The paper investigates using offline reinforcement learning to improve line‑selective tripping in distribution grids. A convolutional Q‑network trained with conservative Q‑learning (CQL) processes voltage‑current phasor and impedance data, optionally with raw waveforms, to predict faulted lines. On a realistic CIGRE medium‑voltage network, the best model achieved high per‑timestep precision, recall, and F1‑score, and correctly identified the first trip action in over 98% of fault episodes, though it mis‑tripped in a notable fraction of non‑fault cases.
The paper reviews the evolution of multi‑agent unmanned systems from isolated sensing to collaborative intelligence, where agents share compact features to overcome local observation limits such as occlusions and sensor range. It introduces a five‑dimensional taxonomy (collaboration stage, communication paradigm, fusion architecture, learning strategy, application domain) and three cognitive synergy conditions (Semantic Disambiguation, Pragmatic Information Exchange, Proactive Informational Foraging) to unify existing research. The authors survey architectures, neural‑communication co‑design, embodied action‑perception loops, and resilience mechanisms, map advances onto operational domains (V2X, UAV, logistics, smart cities), and propose the GCI‑Bench scoring protocol to standardize evaluation across studies.
CoaG introduces a minimal geometric interface— a ground plane grid and a single cylinder per person—to control the placement and movement of people and the camera in generated videos. Users draw the grid, position cylinders, and animate them over 81 frames; the model then renders a photorealistic video that follows these layouts and camera paths, with appearance guided by a text prompt and a background image. To train the system, the authors automatically generate 2,000 captioned clips, reconstruct geometry from each clip, and fine‑tune a LoRA on Wan2.2‑Fun‑Control, enabling the model to respect user‑drawn layouts and camera motions on unseen clips.
Relationally Grounded Latent World Models for Autonomous Driving proposes using traffic scene graphs as privileged semantic supervision for latent world representations. The approach builds actor‑centric scene graphs from nuScenes 3D annotations, encodes their relational structure with a frozen text embedding model, and aligns visual latent representations to this semantic target during training. At inference the supervision branch is removed, requiring no scene graphs or 3D annotations and adding no extra computation, while achieving a 5.9% reduction in average trajectory L2 error and a 52.4% drop in collision rate compared to the LAW baseline.
The paper presents a two-step method to correct machine‑learning based perception for safety in autonomous systems. First, it uses offline computation to characterize uncertainties from the ML module via preimages of perception contracts. Then, at runtime, a risk heuristic selects specific states from these uncertain estimates to guide control decisions, reducing safety violations in adaptive cruise control scenarios while adding minimal delay.
The paper investigates using reservoir computing, specifically an echo state network, to predict nonlinear dynamics in a jumping quarter‑car model. By training on limited time‑series data, the network can reconstruct bifurcation diagrams, phase‑space attractors, and time trajectories across both periodic and chaotic regimes, reproducing the period‑doubling route to chaos. These findings show that reservoir computing can serve as a data‑driven predictor for nonlinear, non‑smooth vehicle dynamics.
arXiv:2609. 22833v1 Announce Type: new Abstract: We study personalized federated reinforcement learning, in which $n$ agents, each acting in its own Markov decision process, collaborate through a server to learn a shared MAML-style policy initialization that becomes effective for an individual agent once that agent adapts it with a single local policy-gradient step.
The paper introduces Calibrated Clipping, a dynamic method to align FP8 quantization bounds with high‑precision BF16 distributions, thereby mitigating training instability in full‑pipeline FP8 reinforcement learning for large language models. It identifies that compounded FP8 noise distorts importance ratios, causing entropy surges and garbled outputs. Experiments across GRPO and DAPO algorithms on 8B‑32B models show the technique restores performance to BF16 levels.
The paper audits LLM-based explainers attached to an Active Inference agent that manages German grid demand, testing three large‑language‑model backends (GPT‑4o, Claude‑3‑Opus, Gemini). By injecting corrupted observations and attacker‑controlled text, the study finds that the explainers fail to flag errors, produce fluent but incorrect rationalizations for wrong actions, and can be steered to exfiltrate data. The authors propose mitigations but do not evaluate them, emphasizing that explanations are never verified for truth before operators rely on them.
The paper proposes a hybrid PID–Deep Reinforcement Learning (DRL) controller for industrial processes, addressing the limitations of traditional PID controllers in complex, non‑linear, multi‑input environments. Using the Industrial Benchmark (IB) to test DRL, the authors develop a multi‑objective reward function and employ a TD3 agent to discover optimal settings for the IB’s ‘Gain’ and ‘Shift’ parameters. These parameters are then fed into a tuned PID controller, yielding a system that combines the optimal performance and efficiency of DRL with the reliability of classical control.
The paper introduces Strategy Accumulation and Guided Execution (SAGE), a two-stage framework that makes automated fine-tuning of large language models cumulative. In the first stage, a multi-agent pipeline uses Monte Carlo Tree Search to explore training strategies while a Distillation Agent records task-specific insights and cross-task confidence scores into a structured repository. In the second stage, SAGE retrieves relevant experience from this repository to guide training on new tasks, achieving a 12.4‑percentage‑point improvement over a baseline pipeline without accumulated experience on nine unseen tasks.
arXiv:2609.24146v1 Announce Type: new Abstract: Language model agents are increasingly used to simulate social interaction, and the resulting transcripts read as though the agents understand one anot...