Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.
arXiv:2610. 27106v1 Announce Type: cross Abstract: We present the Sparse Particle-interaction Transformer Architecture (SParTA), a modular, scalable, and efficient transformer for jet tagging at high-energy colliders.
The paper shows that in text‑to‑image evaluation, human preference for attractive faces can outweigh objective image quality, leading to biased rankings and model outputs. A user study revealed that annotators consistently rate more attractive faces as higher quality, even when comparing real photographs of equal fidelity. This bias can invert leaderboard rankings, dominate preference model scores, and cause generated faces to shift toward younger, more feminine features, undermining user control.
The paper introduces Hybrid Belief Reinforcement Learning (HBRL), a two‑phase framework for coordinating multiple autonomous agents in spatial exploration tasks. In phase one, agents build spatial beliefs via a Log‑Gaussian Cox Process and plan information‑driven trajectories using a Pathwise Mutual Information planner; in phase two, a Soft Actor‑Critic policy is warm‑started through dual‑channel knowledge transfer that incorporates belief states and replay‑buffer demonstrations. Experiments on UAV wireless service provisioning and NYC taxi demand sensing show significant performance gains, with the dual‑channel transfer outperforming single‑channel approaches.
TreeWalker is a method for efficiently evaluating tree‑ensemble models on groups of rows that share many feature values. By treating constant features as static and varying features as dynamic, it walks each tree only once per group, partitions row bitmasks at varying splits, and skips empty subtrees, achieving significant speedups over row‑independent traversal. Experiments on Intel and Arm CPUs show up to 9× speedup for large group sizes, with outputs matching existing libraries to within round‑off limits.
The paper introduces CETBench, a benchmark for assessing large language models (LLMs) on code-equivalence checking by generating program pairs through random code transformations. It demonstrates that even simple transformations significantly degrade LLM performance, especially in cross-lingual scenarios, and proposes a fine-tuning approach to improve results. The study provides detailed ablations on program difficulty and transformation types, offering insights into LLMs’ limited semantic understanding of code.
The paper introduces HASTEN, a hierarchical semantic tree anchoring framework that enhances CLIP-based class‑incremental learning by incorporating external hierarchical knowledge into hyperbolic space. By treating classes as part of a structured hierarchy rather than isolated labels, HASTEN stabilizes visual and textual feature mappings, reduces feature drift, and mitigates catastrophic forgetting across incremental tasks. Experiments on nine benchmarks demonstrate that HASTEN consistently outperforms existing methods.
The paper introduces Adaptive Dual-Teacher Distillation with Subnetwork Rectification (DDSR), a method for black-box domain adaptation that fuses predictions from a black-box source model and a vision‑language model to generate reliable pseudo‑labels for the target domain. DDSR uses adaptive prediction fusion, subnetwork regularization to prevent overfitting, iterative refinement of pseudo‑labels and prompts, and class‑wise prototypes for self‑training. Experiments on several benchmark datasets show that DDSR outperforms existing state‑of‑the‑art approaches, even those that have access to source data or source model parameters.
The paper investigates whether machine learning can predict chaotic quantum many-body dynamics by treating it as a time-series forecasting problem. Using a few-qubit PXP chain as a benchmark, the authors compare a complex Transformer model with a simple linear DLinear model across regimes from ergodic behavior to quantum many-body scarring. They find that while the Transformer works well in ergodic regimes, its performance degrades near scarred states, whereas DLinear remains accurate across all initial states, indicating that complex quantum dynamics can be forecasted with a simple linear mapping.
The paper introduces CERTA, a decision‑level statistical certificate that decides when a quantized clinical language model can replace its native BF16 reference and escalates uncertain decisions to the reference model. Across eleven instruction‑tuned models (7‑72 B parameters) and 2,500 clinical‑verification decisions from 2,484 de‑identified patients, 8‑bit quantization achieved 99.8‑100% coverage, while 4‑bit configurations cut deployment costs to roughly 26–28% of the BF16 baseline with peak memory below a 48 GB accelerator. On external named‑entity benchmarks, the chosen NF4 settings showed only a 0.045 mean absolute F1 change, indicating minimal performance loss. whyItMatters":"CERTA enables risk‑controlled deployment of smaller, cheaper models for clinical decision support, potentially making advanced language models more accessible in resource‑constrained healthcare settings."
The paper introduces CultureLens, a benchmark with 9,261 scenarios from 75 countries that tests how large language models (LLMs) incorporate cultural information. Experiments on Llama‑3.1‑8B, Gemma‑3‑12B, and Qwen‑3.5‑27B show that presenting a contradictory cultural norm reduces aligned answers by 32‑47 percentage points, even when the model knows the correct country. Fine‑tuning with contradictory‑norm examples improves alignment to 85‑93% under contradictory norms while only slightly affecting performance under true norms.
The paper introduces SHAM-Bench, a unified benchmark for detecting span-level hallucinations across six natural language generation tasks, compiled from 6,461 human-annotated examples under a common schema. It evaluates existing detection methods, finding that fine‑tuned detectors perform well in‑domain but falter under distribution shift, while prompt‑based methods with reasoning show greater robustness. The study also highlights challenges such as subtle unsupported content, ambiguous span boundaries, and noisy labels, and releases the dataset, evaluation code, and a public leaderboard for future research.
Self-Decomposing Diffusion for Code (SDDC) is a new diffusion language model framework that first generates a contract outlining program decomposition and inter‑unit dependencies, then uses this contract to guide coordinated parallel implementation of complex programs. By planning decomposition before generation, SDDC balances the trade‑off between aggressive parallel decoding and maintaining program correctness. Across five benchmarks and two 7B diffusion backbones, SDDC improves pass@1 by up to 31.1 percentage points while decoding 7.5–13.7× faster than base models, and outperforms a fine‑tuning baseline by up to 23.5 points with lower latency.
The study investigates how progressive neuronal loss affects language models by systematically ablating computational units in LSTM and Transformer architectures. LSTMs show a gradual decline in syntactic and semantic abilities, while Transformers exhibit a threshold-like collapse under MLP ablation and a smoother sigmoid-like decline under attention-head ablation. These distinct failure modes mirror different aphasic syndromes, suggesting that architecture-dependent degradation patterns can provide computational insights into neurodegenerative diseases.
The Asymmetric Looped Transformer (AsymLoop) separates context encoding from state refinement by using a single-pass causal encoder to produce fixed global key-value caches and a weight-tied recurrent decoder to generate local key-values from its evolving state. Joint attention over both sources allows each update to combine persistent contextual information with the current intermediate state. Experiments on FineWeb-Edu across models from 60M to 1B show lower validation perplexity and higher downstream accuracy than full-loop models, with a 1B model improving zero-shot commonsense reasoning accuracy from 49.58% to 50.85%.
SINT-Flow is a schema integration framework that uses five large language model (LLM) operators to automatically produce a unified schema from multiple input tables, even when those tables are denormalized and contain attributes for several entity types. The framework decomposes such tables into entity‑specific relations during integration. Evaluation on the newly introduced SINT‑Bench benchmark, using GPT‑5.2 and Qwen‑3.6‑27B, shows high performance with at least 97% completeness in entity type detection, 81% attribute detection, and an 84% F1‑score for schema mapping, while ablation studies confirm the benefits of self‑consistency and a review loop.
The paper investigates how dimensionality reduction and quantization together affect text embedding compression across four task families in the Massive Text Embedding Benchmark using five pretrained models. It finds that combining these techniques can preserve near-original performance while drastically reducing storage, with some classification models retaining at least 99% of their score using less than 0.1% of the original storage. The optimal trade‑off between dimensionality and bit‑width varies by task, with retrieval tasks being more sensitive to compression.
SpatialScout is a benchmark that tests vision‑language model agents on active, object‑level 3D mapping. It includes 80 indoor scenes with 3,140 target objects, where agents must control a camera, gather RGB observations, and estimate object centers in a shared coordinate frame. Performance is measured by localization accuracy and target coverage using object‑scale overlap scores, revealing that current VLMs, including Claude Opus 5.5, struggle to build precise metric maps.
The paper introduces TAME, a Text‑Anchored Modality Experts framework that treats language as a fixed, always‑active expert and routes audio‑textual and video‑textual cross‑modal branches as dynamic auxiliary experts. It incorporates lightweight Prophet Heads to estimate sample‑level uncertainty for each modality, guiding a reliability‑aware routing mechanism for expert aggregation. Experiments on CMU‑MOSI, CMU‑MOSEI, and CH‑SIMS show that TAME achieves state‑of‑the‑art performance while reducing parameters by up to 78.7% and FLOPs by up to 41.2% compared to a mixture‑of‑experts baseline.
The paper introduces OTFS, a principled optimal transport framework for selecting frames in long-video understanding. It formulates frame selection as a structured evidence-allocation problem, using a semi‑unbalanced entropic optimal transport objective and an efficient OTFS‑Sinkhorn solver, followed by a dynamic‑programming extraction step. Experiments on three benchmarks show OTFS consistently outperforms strong training‑free baselines.
The paper introduces RoboJudge, a multimodal language model trained to evaluate embodied video generation. It presents RoboJudge-Bench, a benchmark of 800 human‑annotated videos assessing Physical Adherence and Instruction Alignment, and RoboJudge-Train, a 12K‑video dataset with human‑verified scores and rationales. Fine‑tuned on this data, RoboJudge‑9B achieves the highest agreement with human judgments, outperforming existing models and improving performance on external physics benchmarks.