Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.
arXiv:2610.08791v1 Announce Type: new Abstract: Video world models can produce visually convincing yet physically inconsistent sequences, raising concerns about their reliability for prediction and p...
arXiv:2609.34178v2 Announce Type: replace Abstract: Video text editing aims to replace or add text in a video while keeping the rest of the video unchanged, which requires the edited text to be corre...
arXiv:2610.05552v2 Announce Type: replace Abstract: Three-dimensional gait analysis guides rehabilitation after spinal cord injury but depends on marker-based motion capture and force plates, which f...
The paper investigates how to extend music annotation schemas when new attributes need to be added to commercial music catalogs. It compares zero‑shot prediction using audio‑language models, learning from pretrained representations, and supervised adaptation on existing annotations, using a benchmark built on the MGPHot dataset. The findings show that supervised adaptation outperforms zero‑shot prediction even with limited annotation budgets, while reusing frozen representations remains the best option for very modest budgets.
The paper introduces SAGA, a framework that enables large language model agents to evolve by abstracting experiences into reusable principles, procedures, and episodic descriptions. SAGA transforms interaction trajectories into hierarchical knowledge with explicit applicability conditions, linking them back to execution evidence. Experiments on ScienceWorld and ALFWorld show that this execution–abstraction feedback loop improves task performance, and ablation studies confirm the importance of contextual instantiation and action regulation.
EPOCH is an evidence‑governed architecture for AI research agents that improves how candidate discoveries are evaluated and validated. By combining explicit task contracts, typed memory, active falsification, admission checks, and independent replay, EPOCH ensures that each candidate is judged against the strength and scope of its claim. The system achieves state‑of‑the‑art performance on several benchmarks, including AlgoTune, Math14, and AgentHPO, and demonstrates significant advances in executable constructions, algorithm optimization, counterexamples, and proof‑supported results across ten discovery problems.
The paper introduces a cost‑aware hybrid approach that uses small, open‑source language models to classify smart data models (SDMs) in Internet of Things (IoT) environments. It benchmarks general‑purpose, reasoning‑specialized, and code‑specialized models across domain‑specific datasets, highlighting their suitability for edge devices with limited resources. The study also compares these lightweight models to large language models and near‑zero‑cost baselines such as TF‑IDF and a lightweight sentence encoder to demonstrate practical performance gains.
The paper introduces SPECTRUM, a method for Loop Self‑Distillation that repeatedly generates and learns from a model’s own outputs while maintaining a fixed information budget. By re‑estimating loss‑sensitive key/value geometry from a reference anchor and applying full‑rank proximal spectral modulation, SPECTRUM preserves a high proportion of the initial model’s correct implementation diversity across multiple rounds of training. Experiments on MBPP, HumanEval+, and APPS Intro show that SPECTRUM retains more correct‑solution breadth than vanilla self‑distillation and subspace‑projection controls, and the resulting student model achieves higher matched‑correct richness without additional training.
MemMux is a local runtime designed to provide runtime verification and honest resource attribution for fleets of parallel coding agents. It emits observable signals that track per‑agent memory usage, ensure complete reclamation of terminated agents, detect escaped child processes, and keep the system from exceeding a bounded memory footprint. In benchmarks against tmux and a raw‑process baseline, MemMux keeps a fleet under a 7.5 GiB budget with zero swap, while ungoverned tools exceed the budget and spill into swap, and it achieves 100 % attribution with low overhead.
The paper investigates how large language model agents can over-rely on agentic memory, a phenomenon where retrieved memories distort inference even when they are correctly stored and retrieved. It shows that memory is helpful when past experience fully transfers to the current task but becomes misleading under partial query-memory overlap, a pattern confirmed by controlled experiments. To address this, the authors propose MEMTRIM, a plug‑and‑play framework that indexes memory evidence at write time and limits its reuse at read time, thereby reducing over-reliance without retraining and preserving useful memory benefits across models and memory architectures.
Rationale-Guided Policy Optimization (RGPO) is a reinforcement‑learning framework that adaptively uses ground‑truth rationale information to scaffold a language model’s reasoning process. Instead of treating reference solutions as fixed imitation targets, RGPO temporarily incorporates rationales to help the model generate better responses, then reverts to unguided learning with higher‑reward, model‑generated solutions. Experiments in both language‑only and vision‑language tasks show that RGPO consistently outperforms RLVR baselines, with ablation studies confirming that adaptive rationale guidance is a key factor in its success.
The paper investigates whether semantic entropy—a measure of disagreement among a language model’s sampled answers—can serve as a cheap signal for deciding when to route a query from a small to a larger language model. Experiments on GSM8K and other benchmarks show that semantic entropy can distinguish small‑model mistakes and improve routed accuracy, but the authors also reveal that a simple question‑difficulty rule can mimic its performance and that other factors (definition of success, benchmark design, live sampling cost) can undermine its effectiveness. They propose a checklist of checks to validate escalation signals and demonstrate how to predict when a cached‑outcome approach will fail. "whyItMatters":"The study highlights the importance of rigorous evaluation of escalation signals, showing that seemingly promising metrics can be misleading without proper controls and that practical routing decisions must account for cost and benchmark design."
The paper introduces 2D‑FET‑Bench V2, a benchmark of 128 field‑effect transistor layout tasks derived from real microscopy‑derived flake contours, including complex shapes and multi‑flake scenarios. Each task provides a textual device specification and contour coordinates; agents generate GDSII operations that are verified for geometric, structural, and contour‑integrity compliance. Six models and seven GPT‑based workflow variants were evaluated, with GPT‑5.6‑Luna using ReAct‑3 achieving the highest performance, passing 62.3 % of attempts and solving 80.5 % of tasks at least once.
arXiv:2610.07570v1 Announce Type: new Abstract: In clinical practice, agreement among independent experts is treated as evidence of reliability, and multi-round consensus has become a core mechanism...
LOGIC is a benchmark and evaluation framework that tests how language models can ground engineering requests in a deterministic inventory of candidate changes before propagating selected changes through an electrical traceability graph. The benchmark includes 168 scenarios—144 for selection and 24 for abstention—and evaluates three 7–8B models against intent‑agnostic, lexical, and structured‑evidence methods. Results show that structured evidence can achieve perfect candidate F1 on anchored cases, while large language models perform better on relational‑paraphrase cases; however, grounding accuracy drops as candidate inventories grow, and strict evidence gating reduces false positives but may also remove correct selections.
The paper introduces Personal-Agent Mediated Recommendation, a new paradigm where a personal LLM agent uses cross‑platform user history to adjust a platform’s recommendation ranking. It presents MediateRec, a benchmark for evaluating this mediation, and proposes Personal Attribution Mediation Optimization (PAMO) to balance beneficial rescues against harmful overrides. Experiments show that PAMO improves over outcome‑only reinforcement learning, achieving a better rescue‑harm trade‑off on both synthetic and real cross‑platform tests.
BioStudyBench is a new benchmark that tests AI agents on 25 long‑horizon analysis tasks derived from biomedical studies published between July and September 2026, after the models’ knowledge cutoffs. Each task gives the agent a neutral research question but no data files, requiring it to locate and download relevant public data, search literature using tools limited to pre‑cutoff records, and perform data analysis to report findings. The benchmark shows that providing data and tools improves model performance by an average of 47 percentage points, with closed‑weight models outperforming open‑weight ones (best open‑weight 81.3% vs. best closed‑weight 94.7%).
The paper presents a symbolic sub‑policy model for chess that incorporates domain knowledge to improve interpretability. By adapting patterns learned through inductive logic programming (PAL), the authors evaluate the model using a divergence metric against a random baseline and show that it can suggest moves comparable to a human beginner. They also propose a computational evaluation scheme that augments an existing engine with their model.
AgentMemGate is a write-time gate for conversational AI assistants that prevents speculation contamination in long‑term memory by classifying extracted statements as speculation, completed event, correction, or other. Speculations are kept outside memory until conditions allow promotion or deletion. The approach eliminates observed contamination and improves task accuracy from 65% to 95% on core benchmarks, with further gains on a harder held‑out set.
ST-Bench is a new benchmark that tests whether multi‑agent systems (MAS) outperform single agents on complex scientific data analysis tasks. It includes 100 Earth‑science data‑science tasks expanded into 2,067 queries, validated by domain experts. Evaluations show that most MAS configurations beat the cheapest single‑agent baseline, with the best reaching nearly three times its score, though at higher inference cost.