Instella‑MoE is a fully open Mixture‑of‑Experts language model with 16 billion total parameters and 2.8 billion active parameters per token, trained from scratch on AMD Instinct GPUs. It incorporates a sparsely activated MoE design with Gated Multi‑head Latent Attention and FarSkip‑Collective connectivity, and follows a multi‑stage pipeline that includes pre‑training, long‑context extension, supervised fine‑tuning, direct preference optimization, and reinforcement learning with Multi‑Teacher On‑Policy Distillation. The model achieves an average score of 76.7 on pre‑training benchmarks and 73.2 on instruction‑following, reasoning, math, coding, and chat benchmarks, outperforming comparable fully open and open‑weight models, and its full training pipeline, weights, and code are released for reproducibility.
By Jiang Liu, Sudhanshu Ranjan, Prakamya Mishra, Yonatan Dukler, Gowtham Ramesh, Jialian Wu, Ximeng Sun, Wen Xie, Chaojun Hou, Vikram Appia, Zhenyu Gu, Zicheng Liu, Emad Barsoum
The study evaluates large language models for assessing suicide risk in Arabic crisis helpline calls, comparing Arabic and English models. Using de‑identified transcripts from Lebanon’s National Lifeline, the researchers fine‑tuned instruction‑tuned LLMs and transformer encoders, achieving a macro‑F1 of 81.19 and ROC‑AUC of 90.61 for high‑risk calls in Arabic, and 85.00/92.59 in English. The results show that high‑risk calls are more distinguishable than at‑risk calls, and translating to English does not degrade performance, indicating potential for operator‑facing tools.
By Linhai Ma, Rita El Hachem, Mahatab El Hajj, Lilian Ghandour, Samah Fodeh
The paper introduces a scalable framework that uses finetuned large language models (LLMs) to translate unstructured textual resources into structured event logs for process mining. By creating a new text-to-log dataset and finetuning LLMs on it, the authors demonstrate that the resulting models produce high‑fidelity event logs, outperforming few‑shot or zero‑shot prompting methods. This approach enables previously unused organizational data, such as incident tickets and manuals, to be incorporated into process mining workflows.
By Maximilian Seeth, Gabriel Marques Tavares, Daniel Schuster
The paper investigates how different training strategies affect the prompt sensitivity of large language models. It reproduces and compares methods such as refined data construction and robustness objectives, finding that while robustness fine‑tuning improves over standard fine‑tuning and in‑context learning, the prompt gap remains large (40–57%). Notably, newer techniques like CoIN and PPCL often underperform a simple data‑construction approach that uses one template per batch, and diagnostics suggest that mixed‑template batches force the optimizer to reconcile conflicting updates rather than learn a prompt‑agnostic representation.
By Frederic Sadrieh, Michal \v{S}tef\'anik
arXiv:2510.01030v2 Announce Type: replace
Abstract: The human ability to translate diverse perceptual and linguistic inputs into structured behavior has been thought to rest on learning robust repres...
By Zach Studdiford, Timothy T. Rogers, Kushin Mukherjee, Siddharth Suresh
arXiv:2609.01573v1 Announce Type: cross
Abstract: How to divide a fixed annotation budget between supervised fine-tuning (SFT) and reinforcement learning (RL) during LLM post-training remains an open...
By Jingtan Wang, Arun Verma, Xiaoqiang Lin, Zhengyuan Liu, Nancy F. Chen, Daniela Rus, Bryan Kian Hsiang Low
arXiv:2609.01564v1 Announce Type: cross
Abstract: Large language models (LLMs) struggle to classify text into taxonomies with many semantically similar labels, as the distinctions are domain-specific...
By Manish Gupta, Chaitanya Giri, Jayasimha Talur
The paper introduces a new consistency criterion for auditing decision systems that combines ensemble margin with local prediction variability to address predictive multiplicity, or the Rashomon effect. It shows that finite ensembles converge to the expected model’s consistency score as ensemble size and sample count grow, and demonstrates that ensembling models from the Rashomon set reduces unchecked incorrect predictions while keeping diversions moderate. Experiments on transformer and fine‑tuned language models for NLP and tabular classification confirm the method’s effectiveness and stronger alignment with existing multiplicity metrics.
By Sinjini Banerjee, Tim Marrinan, Anand D. Sarwate
The paper proposes CANOPY, a minimalist reinforcement learning protocol that addresses two common pitfalls—signal starvation and policy drift—in outcome‑only RL for long‑horizon interactive tasks. By scaling same‑task exploration, keeping updates on‑policy, and anchoring updates with KL divergence, CANOPY enables a Qwen3‑14B agent to achieve top leaderboard results on the AppWorld coding benchmark without auxiliary supervision or elaborate scaffolding. The approach also improves performance on SWE‑bench for a Qwen3.5‑9B model.
By Liming Pu, Xiaoxia Li, Yifu Liu, Teng Cao, Bin Yang
ViTAMINS is a method that incorporates synthetic hard negatives into unsupervised vision transformer pretraining to enhance representation quality. The approach is evaluated on ImageNet and a range of downstream tasks—including transfer learning, image retrieval, copy detection, and image/video segmentation—showing significant performance gains. The synthetic negatives also lead to emergent properties, such as representations that encode explicit semantic information and act as strong classifiers, improving over baselines by up to 11.3%.
By Nikos Giakoumoglou, Andreas Floros, Kleanthis-Marios Papadopoulos, Tania Stathaki
The paper introduces a cross‑modal pseudo‑labeling pipeline for unsupervised domain adaptation in semantic segmentation, particularly for waste sorting. It combines SAM for class‑agnostic region proposals with EVA‑CLIP to assign semantic labels via region‑text similarity, applying confidence filtering to ensure reliable pseudo‑labels for self‑training. An optional BLIP‑based language‑grounded verification further refines ambiguous regions, and the method shows consistent improvements over source‑only baselines on synthetic‑to‑real driving and lab‑to‑factory waste sorting shifts.
By Udo Schlegel, Shubhangi, Gabriel Dax, Sai Rahul Kaminwar, Florian Karl, Thomas Seidl
The paper proposes a fine‑tuning‑free listwise selector for Text‑to‑SQL systems that replaces traditional learning objectives with inference‑time strategies. It introduces reusable structured memories (MaP‑SQL) that encode mappings from natural language to schema elements, SQL operations, and expected outputs, and uses these memories to evaluate candidate queries. To reduce positional bias, the method aggregates rankings across multiple input permutations, optimizing inference cost through execution results and pointwise scoring. The approach achieves higher selection accuracy, fewer unnecessary comparisons, and outperforms the prior state‑of‑the‑art R^3‑SQL on the BIRD‑dev benchmark while using fewer tokens.
By Yeonseok Jeong, Soyoung Yoon, Seongjun Lee, Seung-won Hwang
TUTTI is a new pre‑training framework for audio‑to‑score transcription that uses a large, fully synthetic multi‑instrument dataset generated by a symbolic music model. The approach trains a standard Transformer encoder‑decoder on these synthetic audio‑score pairs, producing a stronger foundational representation than single‑instrument training. When fine‑tuned on real datasets, TUTTI surpasses prior methods, achieving state‑of‑the‑art results and demonstrating strong cross‑instrument transferability.
By Jianhuai Hu, Yashan Wang, Shangda Wu, Zhancheng Guo, Shijie Liang, Wuna Meng, Chuanqi Yang, Xiaobing Li, Feng Yu, Maosong Sun
The paper introduces FedRoRA, a federated learning framework that combines Low‑Rank Adaptation (LoRA) with rank‑heterogeneous personalization. It separates model adaptation into shared global directions and client‑specific rank‑wise magnitudes, using SVD on the server to extract a global subspace and a personalized projection with top‑k selection for each client. Experiments on natural language understanding and generation tasks show that FedRoRA outperforms existing state‑of‑the‑art methods.
By Lei Wang, Jieming Bian, Letian Zhang, Jie Xu
EEG-VID is a task‑guided latent predictive pretraining framework designed to improve EEG decoding across session and subject shifts. It predicts future latent EEG states from recent history using an exponential‑moving‑average target encoder and weak task guidance, then fine‑tunes with supervised learning. The method achieves significant accuracy gains on VIG‑48 and BCI Competition datasets, and demonstrates effective assistive target selection in a robot‑scene study.
By Guanzhong Sun, Junyi Ma, Yuxuan Wu, Yanzi Miao
The paper introduces WHALE, a method that alternates between updating a language model’s weights and searching for a better harness (the code that manages context and control flow). By iteratively fine‑tuning the model under the current harness and then optimizing the harness under the updated model, WHALE improves performance across search QA, math reasoning, and chess puzzles, outperforming weight‑only, harness‑only, and Fast‑Slow Training by 4.15–24.38 percentage points in mean@8 accuracy. The approach uses either fixed phase lengths or an adaptive patience rule to decide when to switch phases, and the authors provide code on GitHub.
By Haechan Kim, Yoonho Lee, Gisang Lee, Chelsea Finn, Kangwook Lee
The paper introduces CoLT-Drive, a 3,536-sample counterfactual long‑tail benchmark for evaluating decision‑level driving affordance prediction, which tests whether models can infer how rare objects affect an ego vehicle’s high‑level actions. It also proposes KPA, a knowledge‑preserving adaptation framework that combines structured prompting, expert merging, and a regime‑aware LoRA mixture‑of‑experts module to improve small VLMs on driving tasks. Experiments show KPA achieves 60.8% pair accuracy on CoLT‑Drive, outperforming the Qwen3‑VL‑2B baseline and LoRA SFT while keeping competitive in‑domain performance.
By Zhengxu Tang, Guofeng Cui, Ziyu Gong, Xiaozhou Zhang, Ruifeng Deng, Chengzhi Qi, Ke Chen, Sachin Patil, Tianjun Xiao, Langechuan Liu, Pichao Wang
RW-LoRA introduces a random‑walk approach to fine‑tune LoRA models in a decentralized setting, using a single model token that moves through the network and updates locally. This eliminates the need for global synchronization and reduces communication and computation costs compared to centralized or gossip‑based methods. The authors provide convergence guarantees for non‑convex objectives and demonstrate competitive performance on NLP tasks across various graph topologies.
By Xingran Chen, Rohit Bhagat, Ghadir Ayache, Rawad Bitar, Yanmin Gong, Salim El Rouayheb
Pix2Rep-v2 is a self‑supervised learning framework that learns pixel‑ and voxel‑level representations for dense medical imaging tasks, using a redundancy‑reduction objective and equivariance principles to scale to 3D and wide field‑of‑view data. The method is evaluated on four datasets across multiple modalities, tasks, and backbones, demonstrating higher data‑efficiency in few‑shot scenarios and competitive performance, such as a +9.3 Dice point improvement in one‑shot segmentation on the M&Ms‑2 dataset. An in‑context dense prototype approach is also proposed, eliminating the need for downstream training.
By S. Sifaoui, E. Angelini, S. Toupin, T. Pezel, L. Le Folgoc
The paper presents a cumulative turn‑based risk assessment framework for detecting financial scams targeting older adults, which aggregates conversational turns and updates risk estimates at each step. A multi‑turn dialogue dataset covering investment, charity, and tech support scams is created, with annotations for risk level, score, rationale, and safety recommendation at every cumulative stage. Four small language models (Phi‑4, LLaMA‑3.2, DeepSeek‑R1, Qwen3) are fine‑tuned; Phi‑4 and LLaMA‑3.2 outperform others in turn‑aware risk estimation, demonstrating that compact models can effectively support incremental scam detection in resource‑constrained, privacy‑aware deployments.
By Parviz Ghafariasl, Weimin Fu, Xiaolong Guo, Shing I. Chang