GFlowPO introduces a probabilistic framework for optimizing language model prompts by treating prompt search as posterior inference over latent prompts, regularized by a meta-prompted reference-LM prior. The method first fine‑tunes a lightweight prompt‑LM using an off‑policy Generative Flow Network objective that reuses past prompt evaluations for sample‑efficient exploration. It then applies a training‑free Dynamic Memory Update to refine the meta‑prompt with diverse and top‑performing prompts, leading to consistent improvements over recent discrete prompt optimization baselines on few‑shot classification, instruction induction, and question‑answering tasks.
By Junmo Cho, Suhan Kim, Sangjune An, Minsu Kim, Dong Bok Lee, Heejun Lee, Sung Ju Hwang, Hae Beom Lee
TutorCaseBench is a new benchmark designed to evaluate large language model (LLM) tutors on their ability to support students exhibiting challenging behaviors. It is built from qualitative case studies in pedagogy, educational psychology, and special education, converting documented student categories into 923 case-specific dialogues and rubrics. The benchmark tests 11 state‑of‑the‑art LLMs on open‑ended response generation and multiple‑choice selection, finding that even the best model meets rubric criteria only 55.8% of the time for open‑ended cases, highlighting a gap between recognizing interventions and adapting to context.
By Yeon Su Park, Chanyoung Park, Tae Soo Kim, Seungju Kim, Hyunwoo Kim, Daho Jung, Jinsung Yoon, Juho Kim
SIM-UT is a multi‑turn, user‑simulation evaluation framework designed to uncover realistic failure modes in large‑language‑model (LLM) medical triage chatbots. The study applied SIM‑UT to three WhatsApp‑based health chatbots in India, South Africa, and Kenya, comparing a medically fine‑tuned model (MedGemma) with general‑purpose models (ChatGPT and Claude) across more than 5,000 multilingual simulated conversations. A prominent failure identified was de‑escalation risk, where chatbots incorrectly down‑rated serious symptoms after users minimized them, potentially leading to dangerous clinical outcomes.
By Deepali Mittal, Dheeraj Mundhra, Yulia Sidorova, Tersia de Kock, Oli Munns, Sofie S Meyer
IFC-MCQ is a method that automatically creates multiple-choice benchmarks from Industry Foundation Class (IFC) files, enabling exact option-set grading for large language model (LLM) queries in architecture, engineering, and construction (AEC). The taxonomy spans 15 categories and 36 subtypes across four groups, with answer keys derived from the IFC file and distractors generated from file-derived values or predefined rules. Experiments on IFC models of varying sizes and characteristics revealed category-specific failure modes and highlighted how performance can differ based on the source IFC model and retrieval interface.
By Rabindra Lamsal, Sisi Zlatanova, Xuesong Shen
The paper presents a clinician‑led safety guardrail for mental‑health chatbots that replaces typical prompt‑engineering or Constitutional AI methods. Two clinicians design, finalize, and adjudicate a rule base of four safety checks, which is formally verified for internal consistency and applied to LLM outputs. The verification layer’s verdicts are recomputable from a named rule and sixteen facts, and it consistently avoids false crisis‑resource demands that a matched LLM judge frequently raises.
By Chenhan Lyu, Kit Wislocki, Sarah Elizabeth Stoeckl, Maitreyi Sinha, Amir Rahmani
Group Relative Policy Optimisation (GRPO) is applied to a non‑LLM encoder‑decoder ASR model, Whisper‑large‑v3, to improve multilingual speech recognition across 51 African languages. The method mitigates exposure bias from teacher forcing, especially benefiting low‑resource and noisy audio scenarios. The resulting model, SunflowerASR, sets new macro‑averaged WER records on AfriVox‑v2 (0.227) and SimbaBench (0.372).
By Tim Wenjie Hu, Evelyn Nafula Ouma, Benjamin Akera, John A. Quinn
The paper introduces a new benchmark of 13,044 spontaneous Nepali‑English code‑switched podcast utterances, totaling 15.45 h, and evaluates a two‑stage synthetic‑to‑real adaptation framework for ASR. Stage 1 uses supervised fine‑tuning on 0.95 h of synthetic speech, reducing Mixed Error Rate from 23.40 % to 21.52 % and switch‑point error rate from 35.89 % to 28.77 %. Stage 2 applies Group Relative Policy Optimization with script and transcription rewards, yielding modest but statistically significant further improvements, particularly in English script confusion and switch‑point accuracy. The study also shows that an auxiliary script reward term offers no benefit beyond MER‑only optimization in this setting, and that the synthetic SFT provides the main cross‑domain transfer to spontaneous human speech.
By Firoj Paudel
RetiJev is a family of 0.8B‑9B decision models designed for ophthalmic clinical decision support, offering fast, accurate, and privacy‑preserving inference without autoregressive decoding. The authors created RetiJev‑Data, a source‑grounded corpus of 78,158 clinical‑decision and diagnosis questions, 50,074 guideline‑grading questions, and a 24,662‑question evaluation benchmark. RetiJev outperforms a 27B LLM and the commercial Jev 1.13 model on guideline grading and clinical decisions, achieving the highest accuracy while answering a case in 158 ms on a single GPU—3.5× faster than LLM generation and 470× faster than reasoning.
By Bo Liu, Lianyu Wang, Meng Wang, Ching-Yu Cheng, Huazhu Fu
IndoNutriQA is a newly released Indonesian clinical nutrition question‑answer benchmark comprising 911 expert‑curated question‑answer pairs drawn from real clinical‑nutrition reports. Each pair is grounded in a clinically relevant context and enriched with structured metadata such as anthropometric assessment, medical diagnosis, and document provenance. The dataset supports generative, extractive, and retrieval‑augmented QA, and includes fine‑tuned baselines and zero‑shot evaluations of contemporary large language models.
By Tirana Noor Fatyanosa, Ummu Ditya Erliana, Mahendra Data, Muhammad Nurwegiono, Eleazar Tadeo Eman, Alham Fikri Aji
The paper introduces the Bashkir Web Corpus, a 71,567‑document collection (46.9 M tokens) and three labeled datasets for text classification, and benchmarks eight large language models (from 82 M to 8 B parameters) on binary, multiclass, and multilabel tasks. It evaluates three adaptation strategies—full fine‑tuning, QLoRA, and in‑context learning—alongside continual pretraining, finding that QLoRA on 3.8–8 B models outperforms encoder‑only fine‑tuning for binary classification, while continual pretraining’s impact varies by model and task. The study also reports tokeniser fertility differences, shows that Cyrillic tokenisers pretrained on Russian perform best, and demonstrates that LLM‑generated ratings correlate with native judgments for grammar and coherence but not for language correctness, underscoring the need for native evaluation.
By Mullosharaf K. Arabov, Svetlana S. Khaybullina
The study introduces an evidence-aware evaluation protocol for a retrieval‑augmented generation system focused on breast lesion management literature. Using an 18‑document corpus and 18 queries, the authors compared dense embeddings, BM25, and cross‑encoder reranking across various chunk sizes and overlaps, finding that a hybrid retriever with 384‑token chunks and 20% overlap achieved top‑three retrieval success for 17 of 18 queries and high recall and precision metrics. Expanding ground‑truth relevance annotations increased recognized evidence spans and accepted chunk targets, though retrieval rankings remained unchanged, underscoring the sensitivity of evaluation to annotation coverage.
By Miguel Mochizuki Silva, Samuel Ismael, Mateus C. Dias, Tha\'is Gaud\^encio, Yuri Malheiros, Barufaldi
The paper introduces MemPABench, an interactive environment designed to evaluate how personal agents remember and act on user interaction preferences. The benchmark features long‑horizon tasks where agents must decide when to clarify, confirm, or act autonomously, using data from four simulated users who provided 99 sessions over four months and five preference changes each. Experiments across four memory systems and LLM backbones show that current memory approaches offer limited improvement in preference selection, with retrieval‑based methods performing no better than no memory and file‑based memory sometimes applying preferences in irrelevant contexts.
By Jialin Li, Zhenhao Chen, Hanjun Luo, Hanan Salam
ColaBench is a synthetic benchmark designed to evaluate how personal agents use long‑term interaction history to acquire and apply external knowledge. It contains 349 tasks in health and daily living, 10 user profiles, and a 13,033‑document knowledge base, with an Evidence Graph linking personal conditions, disclosed information, prior document exposure, and answer requirements. The paper also introduces MemKB, a system that uses state summaries and memory‑document links to retrieve relevant knowledge, achieving 86.25% accuracy with an average of 4.07 interaction rounds per task.
By Wenjie Zhang, Zhichao Shi, Xuhui Jiang, Xiaojun Wu, Cehao Yang, Chengjin Xu, Xiaoning Song
SCOPE compiles natural‑language policies into neurosymbolic predicate programs that combine small neural models, deterministic extractors, and logical composition. The system is jointly optimized with adversarially generated counterexamples, achieving higher accuracy than prompt‑optimization, task‑decomposition, and workflow‑generation baselines on real‑world policy benchmarks. Programs trained with the weakest executor transfer to other models, enabling deployment on inexpensive models with stable, interpretable predictions.
By Arnon Mazza, Elad Levi
The paper introduces Video‑Centered Deep Research Report Generation, a new benchmark that evaluates agents on their ability to generate long, interactive research reports from a source video and user request. It presents the OmniDR‑Bench dataset with 50 tasks in Chinese and English, 19 subcategories, and 6,414 criteria, and proposes Omni Atomic Rubric Trees (Omni‑ARTs) for evaluation across audiovisual grounding, multimodal evidence use, and interactive presentation. The authors also detail a skill‑based workflow, an automated rubric construction process, and report that current agents achieve a best score of 63.3/100, highlighting gaps in content coverage, causal analysis, and citation accuracy.
By Jiahao Meng, Qize Yang, Yuxuan Wang, Yunfei Chu, Wensi Dai, Jin Xu, Yunhai Tong
Foundation4D is a unified feed‑forward model that performs geometry estimation, 3D detection, semantic occupancy, and Gaussian reconstruction from multi‑camera driving video. It uses a shared backbone with factorized spatiotemporal attention to merge observations across cameras and time, and a video inference pipeline that refines camera trajectories via pose graph optimization and assembles object tracks. The authors also provide a data annotation pipeline and release the PAI‑Foundation4D benchmark, achieving state‑of‑the‑art results on depth, detection, and occupancy metrics and demonstrating that co‑training detection with geometry and occupancy improves performance.
By NVIDIA, :, S\'ergio Agostinho, Maximilian Apodaca, Emmanuel Attia, Guillem Braso, George Dong, Ilya Fedorov, Michele Fenzi, Gang Huang, Jiahui Huang, Shengyu Huang, Bingxin Ke, Laura Leal-Taix\'e, Kuan-Ting Lee, Mingxuan Liu, Tian Liu, Tim Meinhardt, Aljo\v{s}a O\v{s}ep, Naveen Kumar Rai, Kefei Ren, Cristiano Saltori, Keqiang Wu, Ning Xu, Fei Xue, Jin-Bey Yu, Lei Zhang, Qunjie Zhou
The paper introduces ProCoRe, a contrastive reinforcement learning approach that improves visual grounding in multimodal large language models (MLLMs) for in-context learning (ICL). By generating captions for contrastive image pairs, decomposing them into propositions, and optimizing contrastive rewards, ProCoRe enhances alignment with relevant visual evidence without requiring ICL-formatted training data. The method achieves significant gains in multimodal ICL classification accuracy and captioning ROUGE-L over the Qwen-VL-3-8B baseline and outperforms state‑of‑the‑art ICL‑data‑dependent methods across multiple benchmarks. Additionally, the authors present PICL, a personalized multimodal ICL captioning benchmark designed to test genuine understanding of visual demonstrations.
By Minhyuk Seo, Minjae Lee, Chaeeun Lee, Wei Lin, Muhammad Jehanzeb Mirza, Tinne Tuytelaars, Jonghyun Choi
The paper introduces CODI, a method that enhances self‑supervised Vision Transformers by mixing a small portion of the teacher’s class‑token distribution into each teacher patch‑token target. This cross distillation approach gives every masked patch an attention‑independent semantic correction without altering the architecture. Experiments demonstrate that CODI improves performance on both dense prediction and image‑level tasks under aligned pretraining settings.
By Siran Dai, Qianqian Xu, Peisong Wen, Yang Liu, Xiaochun Cao, Qingming Huang
The paper introduces FETA, a Feedback‑Efficient Test‑Time Adaptation method for Vision‑and‑Language Navigation. FETA uses a hierarchical sample selection strategy to identify informative trajectories and critical actions, querying only a single binary feedback per action. The method updates the policy at the queried step with a discrepancy‑aware entropy objective, achieving state‑of‑the‑art performance while reducing human interventions and maintaining robustness to noisy feedback.
By Zixu Wang, Huaquan Zhang, Shoukai Xu, Jinwu Hu, Zhiquan Wen, Mingkui Tan
PhyEvolve is a multi‑agent framework that improves physics‑aware video generation by enhancing natural‑language prompts through structured state‑evolution planning. It employs three agents—a Director, a Physical Evolver, and a Composer—to ground scenes, identify physical mechanisms, and construct causally linked state transitions that guide existing generators without retraining. Experiments on four major benchmarks show that PhyEvolve outperforms the prior agentic baseline by an average of 7.9 percentage points, demonstrating that explicit causal planning boosts physical consistency and state continuity in generated videos.
By Yang-Hao Zhou, Lixing Tan, Zhiqi Li, Guang Yang, Xiao Yang