Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Computer Vision
Sep 22

Beyond the Survey: A Systematic Empirical Study of Detection and Association in Visual MOT

This paper conducts a systematic empirical study of multi‑object tracking (MOT) algorithms, focusing on how detection and association components affect overall performance. By evaluating state‑of‑the‑art methods on benchmarks such as MOT16/17/20, SportsMOT, DanceTrack, and CrowdTrack, the authors find that detection quality has a far greater impact than association strategies, and that transformer‑based end‑to‑end models are more robust to detection variations but computationally expensive. The study provides a unified pipeline diagram and practical guidance for researchers and practitioners in selecting and designing MOT systems.

By Linh Van Ma, Juhua Hu, Wei Cheng, Unse Fatima, Moongu Jeon
arXiv Computer Vision
Sep 22

Human-Level Accuracy, Non-Human Strategies: Revealing Model-Human Divergence in Video Physical Reasoning

Video foundation models now match human accuracy on physical‑reasoning benchmarks, but a new distributional evaluation framework shows that their predictions diverge markedly from human judgments. On the Physion benchmark, ViT‑L models such as V‑JEPA2, VideoMAE‑v2, and DINOv2 achieve near‑human accuracy yet exhibit a 26.4% model‑human disagreement, far above the 4.8% human‑human disagreement, and lower agreement (kappa ~0.48 vs. 0.91). The divergence varies by task: models excel at geometric reasoning but lag on gravitational dynamics and causal chains, indicating they rely on statistical regularities rather than explicit forward simulation.

By Fanhong Li, Shurui Zheng, Zi Yin, Junbo Cui, Lei Ji, Jia Liu
arXiv Computer Vision
Sep 22

Semi-automated reconstruction of indoor geometry from 360-degree video for CFD-based airflow analysis in classrooms

The paper presents a semi‑automated pipeline that transforms a single 360‑degree video of a classroom into editable, simulation‑ready 3D geometry for CFD analysis. Using NeRF for dense point clouds, SAM‑based 2D instance masks lifted to 3D, and octree‑based instance separation, the workflow produces object‑level assets that can be reconfigured in a browser editor. The resulting geometry is validated with OpenFOAM simulations and benchmarked against IEA Annex 20, demonstrating that per‑room geometry acquisition is essential for accurate airflow predictions.

By Dhruv Gamdha, James Afful, Shambhavi Joshi, Ulrike Passe, Adarsh Krishnamurthy, Baskar Ganapathysubramanian
arXiv Computer Vision
Sep 22

VGG16-MCA UNet: Whole-Tumor Segmentation in 2D FLAIR MRI with Decoder-Side Channel Attention

VGG16-MCA UNet is a hybrid neural network that combines an ImageNet‑pretrained VGG16 encoder with a decoder enhanced by a Multi‑Channel Attention module, trained using Focal Tversky loss to address class imbalance. The model was evaluated as a 2‑D, FLAIR‑only whole‑tumor segmenter on BraTS 2020 and LGG datasets, achieving a pixel‑level Dice of 95.10 % on BraTS and 88.32 % on LGG in a 5‑fold cross‑validation setting. Inference time is 66.32 ms per 256×256 slice on a single RTX 2060, only slightly slower than a VGG16‑UNet without attention. whyItMatters":"The study provides a reproducible 2‑D FLAIR baseline for whole‑tumor segmentation, demonstrating high Dice scores and detailed reporting of training and evaluation protocols."

By Shubham Gajjar, Deep Joshi, Avi Poptani, Vishal Barot
arXiv Computer Vision
Sep 22

ShearFuse-UNet: Hadamard, DCT, and Shearlet Transform Fusion for Next-Day Wildfire Spread Prediction

ShearFuse-UNet is a lightweight deep learning model designed for next‑day wildfire spread prediction using multi‑modal satellite data. It incorporates three transform‑domain branches—Fast Walsh‑Hadamard Transform, Discrete Cosine Transform, and a cone‑adapted digital Shearlet residual—within each encoder block of a U‑Net backbone, fusing spectral representations via a learned SpectralFusion gate and adding Shearlet reconstruction as a residual. On the WildfireSpreadTS dataset, the model achieves an F1 score of 0.596 with only 267k parameters, outperforming a larger ResNet18‑based U‑Net and demonstrating a favorable accuracy‑efficiency trade‑off, with additional validation on the Google Next‑Day Wildfire Spread dataset.

By Ene Meco, Yingyi Luo, Emadeldeen Hamdan, Adam Watts, Ahmet Enis Cetin
arXiv Machine Learning
Sep 22

Adaptive Physics-Informed Neural Networks for the Blasius Boundary-Layer Problem

The paper presents an adaptive physics-informed neural network (PINN) framework for solving the Blasius boundary‑layer equation. By combining gradient‑norm‑based loss weighting, nonuniform residual‑based collocation, and a sequential Adam–L‑BFGS optimization, the authors achieve a highly accurate prediction of the wall‑shear coefficient, with an absolute error of $1.896 imes10^{-5}$ for $f''(0)$. A comparative study of network architectures shows that a two‑hidden‑layer model yields the lowest wall‑shear error, while deeper networks reduce the weighted loss but increase physical error. "whyItMatters":"The adaptive framework demonstrates that coordinated adjustments to loss weighting, collocation strategy, and optimization can substantially improve the physical accuracy of PINNs for classical fluid dynamics problems."

By Mehari Fentahun Endalew, Xiaoming John Zhang
arXiv Machine Learning
Sep 22

Joint Domain-Class Modeling for Federated Learning Under Feature Skew

The paper introduces Joint Domain-Class Federated Learning (JDFL), a lightweight, optimizer‑agnostic extension designed to address feature skew in federated learning. JDFL infers pseudo‑domains from local update signals and expands the classifier head to output joint domain‑class logits, enabling the model to capture domain‑conditioned appearance while sharing a backbone. Two supervision strategies—similarity‑aware soft‑labeling and per‑sample randomized target assignment—are proposed to train the expanded head, and experiments on domain‑shifted image benchmarks show consistent improvements in global test accuracy over standard FL methods.

By Sina Najafi, Mostafa Tavassolipour, Seyed Pooya Shariatpanahi
arXiv Computation and Language
Sep 22

Extracting Arguments, Not Just Classifying Them: Instruction-Tuned LLMs for Generative Component Detection

arXiv:2609.24855v1 Announce Type: cross Abstract: Argumentative component detection (ACD) is a core subtask of Argument(ation) Mining (AM) and one of its most challenging aspects, as it requires join...

By Sofiane Elguendouze (UniCA, I3S, MARIANNE), Erwan Hain (UniCA, MARIANNE), Elena Cabrio (MARIANNE, UniCA), Serena Villata (CNRS, MARIANNE)
arXiv Machine Learning
Sep 22

Can Coding Agents Reproduce Official Statistics? Metadata, Retry Budget and the Limits of Execution Feedback in a Controlled Eurostat Benchmark

The paper investigates whether large language models can reliably reproduce official Eurostat statistics by generating executable code. It evaluates a coding agent across four experimental conditions—task only, task plus metadata, metadata with a repair loop using execution feedback, and metadata with a retry budget but no diagnostics—using 30 natural‑language tasks spanning seven domains and datasets. Results show that success depends on semantic validation against frozen specifications, a fully specified output contract, and a retry budget, rather than on execution diagnostics alone.

By Sabina-Cristiana Necula
arXiv Computation and Language
Sep 22

Knowledge Graph-Augmented Ambient AI for Clinical Note Generation

arXiv:2609.22239v1 Announce Type: new Abstract: Ambient AI is increasingly adopted in healthcare to automatically generate clinical notes from patient-clinician conversations, with the potential to s...

By Jakir Hossain, Yi-Fei Zhao, Hongjian Wang, Minmei Shih, Katie Leigh Mullen, Ahmad P. Tafti, Leming Zhou, Manoj Purohit, William Hogan, Jay Zeng, Elizabeth Skidmore, Yanshan Wang
arXiv Machine Learning
Sep 22

CALM: A Calibrated LLM Choice Network Framework for Activity-Based Traveler Simulation

CALM is a reproducible hybrid framework that combines an optional large language model (LLM) activity planner with calibrated stochastic choice, shared network feedback, memory and habit, typed feasibility checks, and deterministic offline replay. It executes a closed traveler‑day loop and evaluates each generative module against an empirical, reproducible baseline, using the 2024 New York City Citywide Mobility Survey data. The framework demonstrates significant improvements in mode‑choice accuracy, quantifies trade‑offs through live‑LLM ablation, and supports controlled stress testing and deterministic replay of downstream simulations.

By Yezhou Cheng
arXiv Machine Learning
Sep 22

Resist, Update, Reject: Preference Optimization Installs a Prior-Dependent Reliability Switch

The paper demonstrates that a preference‑optimization objective can learn to distinguish reliable from unreliable sources by installing a prior‑dependent reliability switch. By training on data where a source’s stated reliability is paired with its answer, the model learns to flip its response only when the stated reliability exceeds a threshold that grows with the model’s prior. Experiments on Qwen2.5‑7B‑Instruct and Llama‑3.1‑8B show that this switch generalizes to unseen reliability values and follows stated reliability over role prestige, whereas supervised imitation fails to learn it.

By Sen Yang, Yuen-Hei Yeung