Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

Hugging Face Trending Papers
Oct 5

MS-Exam-Gen: Source-Grounded Benchmark Construction for Evaluating LLMs on Textual Multiple Sclerosis MRI Knowledge

MS-Exam-Gen is a reproducible framework that builds a source‑grounded multiple‑choice question benchmark for evaluating large language models on knowledge about multiple sclerosis MRI. The pipeline uses expert‑indexed sources, topic induction, evidence‑grounded MCQ generation, automated quality audits, and consistency checks to produce a 3,058‑item benchmark covering 16 topics and 53 subtopics. Evaluation of 12 LLM endpoints on this benchmark revealed a wide accuracy range (89.7% to 46.9%) and identified items frequently missed by models, while audits showed reduced answer cues and position‑sensitivity in scoring.

Hugging Face Trending Papers
Oct 5

TIGER: Time-Series Classification with In-Context-Learning Gated Ensemble of Representations

TIGER is a time‑series classification method that uses a small set of three generic classifiers applied to four different representation families, producing twelve base learners. The predictions are stacked into a meta‑feature matrix and an adaptive meta‑classifier—choosing between a weighted hard majority vote and the pretrained TabICLv2 model—selects the best rule per dataset based on training sample size. On a 142‑dataset UCR benchmark, TIGER achieves the highest mean accuracy, balanced accuracy, and F1‑score among six compared algorithms, outperforming each constituent method and demonstrating strong generalization with a single hyperparameter.

Hugging Face Trending Papers
Oct 5

Reinforcement Learning-Based Optimization of Workload-Aware Power Delivery Networks

The paper introduces a reinforcement learning framework for optimizing workload‑aware power delivery networks (PDNs) in VLSI chips. By mapping system‑level power traces to spatial power density distributions, the method adaptively allocates PDN resources and uses a Deep Q‑Network to perform wire‑width optimization, reducing PDN area while meeting electromigration and IR‑drop constraints. Experiments on multiprocessor floorplans with PARSEC and SPLASH‑2 workloads show a 47% area reduction and a 26× speedup over simulated annealing.

Hugging Face Trending Papers
Oct 5

Benchmarking CLIP for Zero-Shot Face and Periocular Gender Estimation

The paper evaluates CLIP’s zero‑shot gender estimation on full‑face and periocular images from the Adience dataset. Using image‑text similarity with male/female prompts, CLIP achieves 95.54% accuracy on full faces without task‑specific training. Periocular predictions are initially biased toward males, but threshold alignment improves accuracy to 85.29%, and a linear SVM on CLIP features yields a modest 86.17% accuracy, slightly better than prior Adience results.

Hugging Face Trending Papers
Oct 5

StagQ: Constraint-Driven Multi-Precision Weight Quantization for LLMs

StagQ is a multi‑precision weight format for large language models that uses a 2‑bit group‑wise affine base followed by optional 1‑bit refinement planes. Each supported precision can be read as a prefix of the main stream, decoded via a shared affine map without per‑weight lookups, and a sparse side record stores the few weights that the grid handles poorly. Experiments show that StagQ outperforms baseline multi‑precision schemes on Llama‑3.1‑8B, Phi‑4, and OLMo‑2‑7B across various bit‑widths, and its GPU kernel is faster than baseline kernels for most shape‑precision combinations.

Hugging Face Trending Papers
Oct 5

Adaptive-Shot Hybrid Quantum Anomaly Detection for Tactile Internet Security: Reliability-Aware Measurement Allocation Under Resource Constraints

The paper introduces the Adaptive‑Shot Variational Quantum Circuit (AS‑VQC) policy for finite‑shot hybrid quantum anomaly detection in Tactile Internet security. Starting with 128 shots per record, the policy escalates to 256, 512, and 1024 shots only when the anomaly score is near a validation‑selected threshold, thereby reducing computational load while maintaining decision accuracy. Experiments on a 4,875‑record benchmark show that the AS‑VQC‑95 policy averages roughly 130–280 shots per record, saving 73–87% of the resources compared to a fixed 1024‑shot baseline, with decision disagreement below 1%. whyItMatters":"The study demonstrates that adaptive quantum measurement allocation can significantly cut resource usage in security analytics without compromising reliability, offering a practical trade‑off for constrained Tactile Internet environments."

Hugging Face Trending Papers
Oct 5

Nash Equilibrium Text: A Game-Theoretic Decoding Framework for Text Generation

The paper introduces a game-theoretic approach to text revision, treating token positions as players and vocabulary items as actions, with utilities based on a language model’s log conditional probability. It shows that Nash equilibria can yield exponentially higher likelihoods than autoregressive outputs as sequence length increases, and proposes Nash decoding, an algorithm that finds an ε-Nash equilibrium in O(1/ε) time. Experiments on CLAPNQ, PubMedQA, and CoQA demonstrate that equilibria derived from masked language models achieve higher F1 and ROUGE scores than autoregressive models, up to 18× larger, without fine-tuning, though with extra test-time computation.

Hugging Face Trending Papers
Oct 5

Adaptive Utilization of Low-Rank Adaptation via Conditioned Gating

The paper introduces U-LoRA, an adaptive version of Low‑Rank Adaptation that uses conditioned gating to generate token‑specific utilization coefficients for the low‑rank subspace. By coordinating these coefficients with sequence‑level context and applying a bias‑corrected EMA prior, U‑LoRA achieves more consistent adaptive patterns and reduces noise from batch fluctuations. Experiments on mathematical reasoning and natural language understanding tasks show that U‑LoRA matches or surpasses strong LoRA baselines while keeping the same parameter budget.

arXiv Machine Learning
Oct 5

Learning Jazz Pianist Style with Cross-Attention Conditioning

The paper demonstrates that a pretrained symbolic music transformer already encodes jazz pianist identity sufficiently for accurate classification across two benchmarks. By adding cross‑attention over learned pianist embeddings, the model can generate music conditioned on a specific artist’s style, and evaluation protocols confirm that the generated continuations are correctly attributed to the intended pianist. Additionally, the classifier is repurposed to identify the most characteristic moments in a performance, revealing the musical gestures that distinguish each pianist’s voice.

By Drew Edwards, Akira Maezawa, Simon Dixon
arXiv AI
Oct 5

Towards Unified Music Emotion Recognition across Dimensional and Categorical Models

The paper introduces a unified multitask learning framework for Music Emotion Recognition that simultaneously handles categorical and dimensional emotion labels, enabling training across multiple datasets. It leverages musical features such as key and chords, MERT embeddings, and employs knowledge distillation from teacher models to a student model to improve generalization. Experiments on MTG‑Jamendo, DEAM, PMEmo, and EmoMusic show that this approach outperforms state‑of‑the‑art models, including the top MediaEval 2021 entry.

By Jaeyong Kang, Dorien Herremans
arXiv AI
Oct 5

Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Experts

The paper introduces a hardware-software co‑design framework that compresses Mixture‑of‑Experts (MoE) model weights into low‑precision, hardware‑native sparse representations, enabling efficient execution on Sparse Tensor Cores (SpTCs). By relaxing discrete support selection through continuous reparameterization, the method jointly optimizes quantized weights and supports a router‑weighted reconstruction objective, achieving up to 4.35 percentage‑point gains in joint sparse‑quantization accuracy while retaining 96.09% of the original model’s performance. A custom grouped sparse GEMM kernel further boosts inference speed, outperforming NVIDIA’s baseline by up to 1.65× and reducing latency by up to 4.03× on B200 GPUs.

By Kwanhee Lee, Namhoon Lee, Dan Alistarh
arXiv Machine Learning
Oct 5

Identify then Realize: Contrastive Learning of Latent Port-Hamiltonian Dynamics from Partial Observations

The paper introduces CIPHER, a two‑stage identify‑then‑realize framework that learns latent port‑Hamiltonian models from partial, high‑dimensional observations. In the first stage, a contrastive teacher jointly trains an encoder and a neural ODE to produce predictive state representations. The second stage trains a student to learn an invertible coordinate transformation and port‑Hamiltonian dynamics that match encoded future observations, with theoretical guarantees for recovering the physical state up to a diffeomorphism. Experiments on ten datasets show that CIPHER matches or surpasses state‑of‑the‑art baselines, especially in long‑horizon forecasting, and remains robust to over‑specified latent dimensions while benefiting from port‑Hamiltonian parameterization.

By Peilun Li, Kaiyuan Tan, Daniel Moyer, Thomas Beckers
arXiv Computer Vision
Oct 5

The Shape of Speech: A Geometric Measure of Coarticulation for Speech-Driven 3D Facial Animation

The paper introduces a geometric measure of coarticulation for speech‑driven 3D facial animation, comparing lip‑path length to the shortest route through vowel, consonant, and vowel positions. Using only forced alignment, the measure evaluates four state‑of‑the‑art animation methods, revealing that all produce flatter lip trajectories than captured speech and that some methods lose 15–60% of the fast articulatory component. A pre‑registered viewer study confirms that damping real motion lowers perceived quality while exaggeration is not penalized, and viewers prefer real speech in 73.4% of sentence comparisons.

By Danzel Serrano, Przemyslaw Musialski
arXiv Computer Vision
Oct 5

Uncertainty Estimation in Pathology Foundation Models via Deep Mutual Learning

The paper introduces DICE, a plug‑and‑play framework that ensembles multiple frozen pathology foundation models (PFMs) and estimates uncertainty from their consensus. By aligning ensemble members through deep mutual learning, DICE theoretically bounds epistemic uncertainty and localizes abnormalities at the patch level without explicit supervision. Experiments on three whole‑slide image benchmarks show that DICE provides reliable uncertainty estimates, accurately flags failure‑prone cases in both in‑ and out‑of‑distribution settings, and matches or surpasses state‑of‑the‑art baselines in classification, calibration, and localization.

By Gb\`egninougbo Aurel Davy Tchokponhoue, Sevda \"O\u{g}\"ut, Ali Idri, Dorina Thanou, Pascal Frossard
arXiv Computer Vision
Oct 5

Active Sampling for Ultra-Low-Bit-Rate Video Compression via Conditional Controlled Diffusion

ActDiff-VC is a diffusion-based video compression framework designed for ultra‑low‑bitrate scenarios. It partitions videos into variable‑length segments, transmits keyframes only when necessary, and encodes temporal dynamics with a compact set of tracked point trajectories. Conditioned on these sparse signals, a conditional diffusion decoder reconstructs the remaining frames, achieving significant bitrate reductions and perceptual quality gains compared to strong learned codecs.

By Amirhosein Javadi, Shirin Saeedi Bidokhti, Tara Javidi
arXiv AI
Oct 5

IntentCoding: Amplifying User Intent in Code Generation

IntentCoding is a decoding strategy that amplifies user intent in large language model code generation by masking the intent and applying a multi‑strength ensemble mechanism. It is model‑agnostic, requires no extra training, and integrates with existing decoding procedures. Experiments on the new CodeConstraints benchmark and other datasets show significant improvements in constraint satisfaction and functional correctness, with up to 71.0% relative gains on CodeConstraints and 29.3% on HumanEval and LiveCodeBench compared to greedy decoding.

By Zheng Fang, Yihong Dong, Lili Mou, Dongming Jin, Zhi Jin, Ge Li
arXiv Computation and Language
Oct 5

Investigating the Role of Reasoning-Language Alignment in Monolingual Retrieval-Augmented Generation

The study explores how the language used for reasoning affects retrieval‑augmented generation (RAG) in a monolingual German setting. Using a German RAG question‑answering testbed based on the tabletop game The Dark Eye, the authors show that aligning the reasoning language with the query and retrieved documents improves performance, with German reasoning outperforming French reasoning. However, German reasoning still does not surpass the model’s native English reasoning, indicating that native multilingual reasoning is necessary for optimal results.

By Oliver Hauck, Mario Sanz-Guerrero, Katharina von der Wense
arXiv Computer Vision
Oct 5

CrowdOcc: Monocular Semantic Scene Completion for Quadruped Robots in Crowded Indoor Environments

CrowdOcc introduces a new RGB‑D dataset and monocular semantic scene completion framework tailored for quadruped robots navigating crowded indoor spaces. The dataset comprises 25.1K frames across 11 scenes, with annotations derived via static‑dynamic decoupling. The framework employs Normal Guided Scene Geometry Fusion and Human‑Centric Sparse Interaction to robustly model occluded geometry and human‑scene relationships, achieving state‑of‑the‑art performance on the dataset’s test set.

By Feiyang Chen, Jincheng Hu, Yiduo Chen, Jihao Li, Yue Liang, Bingzhao Gao, Yanjun Huang, Yuanjian Zhang
arXiv AI
Oct 5

Interpreting at Write Time: A Policy Ablation for Multi-Goal Agent Memory

The paper investigates how a long-running assistant should summarize its memory when it must serve multiple user goals simultaneously. Three summarization policies are compared: a neutral summary with no goal in mind, a single summary covering all goals, and separate per‑goal summaries. Experiments show that per‑goal summaries outperform the others in relevance, completeness, and accuracy, while the all‑goal summary performs poorly even compared to a neutral summary with a smaller budget.

By Albert Sadowski, Jaros{\l}aw A. Chudziak
arXiv Computer Vision
Oct 5

LightLoc++: Sensor-Robust Representation Learning for Efficient Outdoor LiDAR Localization

LightLoc++ introduces a sensor‑robust framework for outdoor LiDAR localization that decouples a pretrained backbone from lightweight scene‑specific heads, enabling efficient new‑scene training. The authors present SULID, a synchronized urban multi‑LiDAR dataset featuring 32‑, 64‑, and 128‑beam sensors, which is used to pretrain a backbone via cross‑sensor consistency learning. Additional techniques such as sample classification guidance and redundant sample downsampling reduce regression ambiguity and computational load, achieving state‑of‑the‑art performance with minimal new‑scene training cost.

By Wen Li, Shangshu Yu, Dunqiang Liu, Shaoyang Chen, Qiming Xia, Sheng Ao, Siqi Shen, Chenglu Wen, Cheng Wang