Hugging Face Trending Papers
Does task decomposition improve automatic NLG evaluation?
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv Computation and Language
Sep 2
Does task decomposition improve automatic NLG evaluation?
The paper evaluates whether breaking down evaluation tasks into simpler sub‑tasks improves the LLM-as-a-judge (LLMaJ) framework for reference‑free NLG assessment. Systematic comparisons across multiple datasets show no performance advantage for LLMaJ methods that use task decomposition over a fair baseline that does not. The authors attribute previously reported gains to the use of human labels for training rather than to decomposition itself, and note that LLMaJ without decomposition can match human annotators when such labels are available.
arXiv Computation and Language
Aug 31
Select, Label, Evaluate: Active Testing in NLP
The paper introduces Active Testing, a framework that selects the most informative test samples for annotation in NLP, aiming to reduce human effort while accurately estimating model performance. Experiments across 18 datasets and 4 embedding strategies show up to 95% annotation savings with less than 1% loss in performance estimation accuracy. The authors also propose an adaptive stopping criterion to determine the optimal number of samples without a predefined budget.
arXiv AI
4d ago
Lost-in-the-Middle in Long-Text Generation: Synthetic Dataset, Evaluation Framework, and Mitigation
arXiv:2503.06868v2 Announce Type: replace-cross Abstract: Existing long-text generation methods produce lengthy outputs from short inputs, leaving long-input-to-long-output generation underexplored....
Hugging Face Trending Papers
Aug 19
Assessing Quality of Experience in Natural Language Generation of German Text
The paper introduces TextQ‑German, a dataset suite for evaluating German natural language generation (NLG) from a Quality of Experience (QoE) perspective, covering tasks such as summarization and machine translation. Human ratings collected via crowdsourcing identify perceptual quality dimensions, and the authors develop automatic QoE prediction models—including transformer‑based, linguistic feature‑based, and hybrid approaches—showing that hybrid models outperform pure transformers and that linguistic features alone can rival fine‑tuned language models. The dataset is further enriched with large language model (LLM) outputs annotated with overall QoE scores, and validation on held‑out data demonstrates generalization to unseen data. "whyItMatters":"The resource provides a publicly accessible benchmark and baseline models for human‑centered NLG evaluation, enabling the development of systems that better align with human quality perception."
arXiv AI
Jul 21
BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation
arXiv:2604. 09497v2 Announce Type: replace-cross Abstract: Accurate evaluation is central to the large language model (LLM) ecosystem, guiding model selection and downstream adoption across diverse use cases.
arXiv AI
Jun 3
Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling
arXiv:2606. 02837v1 Announce Type: cross Abstract: Accurate translation from Natural Language to First-Order Logic (NL-to-FOL) underpins neurosymbolic AI systems and Natural Language Inference (NLI), making the quality of NL-to-FOL benchmarks essential -- yet these datasets have never been rigorously audited.