arXiv AI
Bias Robustness of Genetic Multi-Criteria Sorting versus Learned Rankers under Proxy Skew and Label Contamination
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
6d ago
Rank-Variance Scores for Importance Sampling and Data Pruning under Label Contamination
The paper introduces Rank‑Variance Scores (TRV) as a new metric for importance sampling and data pruning in the presence of label contamination. It shows that traditional loss‑based methods fail because corrupted examples often have high loss, and proves that a useful score must be non‑monotonic in loss. By ranking examples across multiple independently trained proxy models, TRV captures both between‑proxy disagreement and within‑proxy variance, enabling Disagreement‑Regularized Importance Sampling (DR‑IS) and TRV‑based pruning to outperform random selection on noisy CIFAR‑10/100 datasets.
arXiv AI
Jun 3
CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks
arXiv:2606. 03650v1 Announce Type: cross Abstract: Choosing or ranking language models for a specific application is hardest when no task-specific labeled data exists, and standard public benchmarks cannot be trusted, their items having likely leaked into pretraining, so scores reflect memorization rather than fitness.
arXiv AI
Sep 25
Baszta: Data-Centric Fine-Tuning of a Polish Multi-Label Safety Classifier
The paper presents Baszta, a Polish multi‑label content‑safety classifier trained by fine‑tuning the 124M‑parameter allegro/herbert‑base‑cased model on five categories (hate, vulgarity, sexual content, crime, self‑harm) using a Focal + R‑Drop objective. In out‑of‑distribution evaluation on the Gadzi Język benchmark, Baszta achieves a small but statistically significant improvement in micro‑F1 over the Bielik Guard system, though the macro‑F1 advantage disappears when both models are properly tuned. The study also explores calibration techniques, showing that per‑category temperature scaling can recover performance lost by Platt scaling or isotonic regression, and discusses the trade‑offs between robust calibration and adversarial recall.
arXiv AI
Aug 20
Pairwise Ranking Outperforms Single-Action RL for Offline Explanation Selection: A Practical Lesson
The paper presents a cost‑effective approach for industrial explainable‑recommendation systems by decoupling explanation generation from selection. Candidate explanations are pre‑generated using six prompt styles and two commodity LLMs, then a lightweight CPU‑resident selector (e.g., LambdaRank) chooses the best one at request time, achieving sub‑100 ms latency without GPUs. Experiments on a 2,958‑pair Google Local subset and a 300‑pair MovieLens‑1M split show that pairwise ranking methods outperform single‑action RL baselines, while KG‑path selectors achieve near‑perfect user satisfaction scores.
arXiv Computation and Language
Sep 17
When Audit Quality Fails to Predict Downstream Utility: A Counterfactual Study of Synthetic-Data Selectors for Low-Resource African NLP
arXiv:2609. 18960v1 Announce Type: new Abstract: Quality-aware synthetic-data selection rests on a proxy: examples that an LLM judge rates as good should also help a downstream model learn.
Hugging Face Trending Papers
Aug 19
Pairwise Ranking Outperforms Single-Action RL for Offline Explanation Selection: A Practical Lesson
The paper presents a cost‑effective approach for industrial explainable‑recommendation systems by decoupling explanation generation from selection. Explanations are pre‑generated using six prompt styles and two commodity LLMs, then a lightweight CPU‑resident selector (e.g., LambdaRank) chooses the best one at request time, achieving sub‑100 ms latency without GPUs. Experiments on a 2,958‑pair Google Local subset and a 300‑pair MovieLens‑1M split show that pairwise ranking outperforms single‑action RL methods, while KG‑path selectors achieve near‑perfect unique‑output rates, and the overall end‑to‑end build cost is around $15 on commodity hardware.