arXiv AI
2d ago

Rules, Traces and Routing Statistics: Training and Calibration Data for Open Models on ARC-AGI-2

The paper reports on training and calibration experiments for three open models on the ARC‑AGI‑2 benchmark, using offline runs on four L4 GPUs for up to 12 hours. For the NVARC Qwen3‑4B grid model, a curated corpus of 12,000 executable rules and 95,993 tasks was built, and fine‑tuning on 1,920 real task records raised the semi‑private score from 28.96 to 33.06, with no synthetic‑task checkpoint outperforming the real‑task model. For the Qwen3.8‑27B reasoning model, fine‑tuning on 1,036 of its own reasoning traces and a specialized inference procedure increased the score from 26.81 to 44.03, while the Qwen3.8‑Flash‑Next mixture‑of‑experts model achieved a best score of 56.67 after expert pruning, routing statistics adaptation, and an FP8 indexer cache.

By Nguyen Minh Thang, Trinh Thi Phuong
arXiv Computation and Language
3d ago

Learning by Doing: Autonomously Compiling and Curating Agent Skills from Verified Traces

The paper introduces Zep Agent Skills, a production service that autonomously compiles and curates procedural memory for large language model agents by converting verified execution traces into versioned skills. It records each task attempt as a trace, uses a compiler with Reflector and Curator roles to transform verified traces into skills, and applies admission gates and search ranking based on verified outcomes. The open Agent Skills format, originally from Anthropic, allows any host to load compiled skills, and evaluation on several benchmarks shows significant performance gains for weaker executor models without additional training.

By Daniel Chalef, Peter Evans, Pavlo Paliychuk, Mehul Patel, Preston Rasmussen, Jack Ryan
arXiv Machine Learning
Jul 1

Calibration, Not Compilation: Detecting and Repairing Misspecified Probabilistic Programs Written by Language Models

arXiv:2606. 31630v1 Announce Type: new Abstract: Language models increasingly write probabilistic programs (in NumPyro, Stan, or Pyro), but a program that compiles, runs, and passes every unit test can still be \emph{statistically} wrong -- a Gaussian likelihood for heavy-tailed data, a Poisson for over-dispersed counts, an invalid prior support, or a pathological parameterization.

By Jian Xu, Delu Zeng, John Paisley, Qibin Zhao
arXiv Machine Learning
Aug 12

MERA: Model Evolution and Routing with Skill Adaptation for Agentic Systems at Scale

arXiv:2608. 10333v1 Announce Type: new Abstract: LLM agents execute heterogeneous sequences of model calls within a single task: some invocations require careful reasoning, while others are structured steps such as formatting or tool-argument construction.

By Yuhang Yao, Zeyu Wang, Wanyi Chen, Tongyun Yang, Yuhang Han, Jie Xiao, Chengke Bao, Tianyi Zhao, Lynn Ai, Eric Yang, Tianyu Shi