The study evaluates large language model (LLM) graders on two computer‑science exams, testing 171 configurations of closed‑ and open‑weights models. While the best LLM configuration achieved a mean absolute error of 1.64/35—better than the 2.61/35 error between two human graders—its performance was highly sensitive to the prompt. A short "strict grader" preamble caused most open‑weight models to exceed acceptable error thresholds or stop grading entirely, whereas fine‑tuning with a single LoRA adapter restored parity with human graders and reduced sensitivity to harsh prompts.
By Ali Habibullah, Yazan Alshoibi, Mohammad Alshiekh, Salman Khan, Naeemullah Khan
arXiv:2609.05435v1 Announce Type: new
Abstract: Modern language agents are expected to operate over long horizons: they ask follow-up questions, reuse worked examples, handle tool feedback, and adapt...
By Zerui Cheng, Jiawei Xu, Huacan Chai, Jiayang Sun, Pramod Viswanath, Maxm Pan
PROOF-Gen is a method that improves distillation of tool‑calling models by recovering successful trajectories from teacher failures. It uses per‑scenario prompt optimization to generate corrective guidance that steers the teacher to a passing trajectory, then removes this guidance before training so the student learns from clean demonstrations. On τ2‑bench, PROOF-Gen recovers 93% of failed scenarios, boosting Qwen3‑4B‑Instruct‑2507’s Pass^1 from 0.132 to 0.529 and improving Gemma 4 E4B‑it by 7.2pp on BFCL v4 multi‑turn, while also raising deployed on‑device model performance by up to 5.0pp across response‑quality metrics.
By Anh Ta, Junjie Zhu, Shahin Shayandeh
arXiv:2609.05435v2 Announce Type: replace
Abstract: Can language agents continually learn from experience, turning earlier interactions into reusable capabilities? AhaBench evaluates this ability thr...
By Zerui Cheng, Jiawei Xu, Huacan Chai, Jiayang Sun, Pramod Viswanath, Maxm Pan
UpgradeBench is a decision‑centric longitudinal benchmark that evaluates how fine‑tuned language‑model specialists should be handled when new base‑model releases occur. It covers four consecutive Qwen releases, a continuation checkpoint, six tasks, two model sizes, and OLMo checkpoints with known training lineage, and examines whether retraining, adapter transfer, or other recovery strategies improve specialist performance. The benchmark reveals that upgrade gains vary by task and release interval, that direct adapter copying is sensitive to pretraining distance, and that teacher relabeling can recover specialists without new annotations.
"whyItMatters":"The study provides actionable insights into the cost‑effective management of specialist models across model releases, showing how to balance retraining effort with performance gains."
By Ye Chen, Weining Zhang
The paper investigates how curriculum difficulty progression affects training in Group Relative Policy Optimization (GRPO). It finds that continually selecting medium‑difficulty samples is not optimal; a staged curriculum that gradually shifts toward harder samples improves performance, reaching a peak accuracy of 0.6591. However, progressing too far to very hard samples degrades accuracy, indicating a non‑monotonic benefit of difficulty progression.
By Ronghan Li, Fei Chen, Mingyu Lu
GEAR is a two‑stage framework that distills tabular foundation models into lightweight MLP or tree‑based predictors for efficient CPU deployment. In the first stage, synthetic covariates are used as teacher‑query locations to train the student on soft TFM targets, expanding coverage beyond observed rows. The second stage re‑anchors the student to the target distribution using real labels and out‑of‑fold teacher predictions, preventing self‑labeling leakage and improving performance. Experiments on TALENT and TabArena show that GEAR‑distilled MLPs outperform supervised MLPs by up to 2.00 AUC points on binary tasks and 1.35 on multiclass tasks, and also outperform CatBoost, while dramatically reducing inference time and memory usage.
By Qi Qin, Jiajie Zhu, Dali Chen, Yuzhao Zhang, Jia-Xing Han, Yu Su, Peng Zhang, Ying Yan, Yifan Sun
The paper presents a causal analysis of a compressed VLA policy that performs well in offline tests but fails in closed‑loop execution on a simulated pick‑and‑place task. An 8‑layer distillation of Octo‑Base retains most parameters and passes all offline metrics, yet collapses during deployment, with early stages degrading gradually and final transport failing entirely. The failure is traced to a negative, late‑heavy residual in the action trace, and standard remedies (continued training, offline data, command‑level compensation, clamping) do not restore performance; only a minimal‑pair intervention that mixes deployment‑distribution rollouts with teacher data restores parity with the teacher.
whyItMatters":"The study demonstrates that offline validation metrics alone are insufficient to guarantee closed‑loop success for compressed policies, highlighting the need for targeted deployment‑time testing and interventions."
By Fengze Jia (The Ohio State University)
arXiv:2606. 29280v1 Announce Type: cross Abstract: We identify intervention bias as a previously unquantified failure mode of zero-shot large-language-model (LLM) educational advisory agents: without task-specific training, they recommend action when a hindsight-optimal oracle policy mandates inaction.
By Craig Atkinson
The paper introduces Group‑Calibrated On‑Policy Distillation (GC‑OPD), a method that aligns token‑level teacher guidance with trajectory‑level verifier rewards for long‑context reasoning tasks. GC‑OPD normalizes rewards within rollout groups, uses the signed teacher‑verifier disagreement as a residual, and distributes this residual across tokens via Relative‑Advantage‑Based Credit Assignment (RACA). Experiments on five long‑context benchmarks show that GC‑OPD improves Qwen3‑4B and Qwen3‑8B checkpoints from 29.08/35.12 to 40.47/44.65, outperforming vanilla OPD and demonstrating the effectiveness of group‑relative residual calibration.
By Zhu Zhang, Jixun Wang, Xiaoang Xu, Xiaorong Wang, Zihan Zhou, Zhiyuan Wang, Shuo Wang, Chaojun Xiao, Yuezhi Zhou
MetroLLM-Bench is a 955‑case benchmark designed to evaluate language models as the policy layer of transit kiosks across six real metro systems, covering routing, fare calculation, disruptions, accessibility, and adversarial input. The benchmark includes 14 deterministic scoring components (Tier 1) and 8 semantic‑quality components (Tier 2), with a 75/25 split for training‑data generation and held‑out evaluation. Twenty‑six models from six vendors were tested, and a 4B Qwen 3.5 student fine‑tuned via PEFT outperformed GPT‑5.6 on Tier 1 and matched GPT‑5.4 on the combined score, while larger models offered no further improvement.
By Remco Hendriks (Continker)
arXiv:2608. 15565v1 Announce Type: new Abstract: Experience-learning agents for optimization modeling improve by storing verified skills, but existing learners admit knowledge by checking against known answers, which real ticket streams do not provide.
By Junbo Jacob Lian, Huiling Chen, Hanzhang Qin, Chung-Piaw Teo