arXiv Computation and Language September 4, 2026 By Chihiro Yano, Ryohei Sasano

FrameBench:A Language Understanding Benchmark Based on Frame Semantics

Read the original on arXiv Computation and Language →

FrameBench is a new benchmark that evaluates language models on their ability to distinguish semantic frames evoked by the same verb in different contexts, using multiple-choice questions grounded in FrameNet-style resources for English and Japanese. The dataset is generated and verified through a pipeline that incorporates native-speaker judgments, and the authors provide both the data and the code for construction and evaluation. Experiments show that small models struggle with this task, while several large models outperform human reference scores.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

Aug 25

The Multilingual FrameNet Corpus

The paper presents the Multilingual FrameNet Corpus (mFNC), a resource that expands the English Berkeley FrameNet by integrating and harmonizing language‑specific corpora in nine additional languages: Brazilian Portuguese, Chinese, Dutch, French, German, Italian, Korean, Latvian, and Swedish. Experiments with various model architectures on mFNC consistently surpass existing state‑of‑the‑art Frame Semantic Parsers in both multilingual and cross‑lingual scenarios, highlighting the value of multilingual training data. The mFNC and the trained Frame Semantic Parser models are publicly released on GitHub.

By Beatrice Fiuman\`o, Nicolas Lazzari, Simone Paolo Ponzetto, Valentina Presutti

benchmarks

arXiv AI

Jul 21

BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation

arXiv:2604. 09497v2 Announce Type: replace-cross Abstract: Accurate evaluation is central to the large language model (LLM) ecosystem, guiding model selection and downstream adoption across diverse use cases.

By Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Emmanuel Malherbe, C\'eline Hudelot, Pierre Colombo

llms

arXiv Computation and Language

Sep 2

ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs

arXiv:2607.00171v2 Announce Type: replace Abstract: Text embeddings are standard for semantic similarity tasks, yet their evaluation remains an open challenge. Current benchmarks are static, cover on...

By Andrianos Michail, Stylianos Psychias, Michelle Wastl, Simon Clematide, Rico Sennrich, Juri Opitz

rag nlp benchmarks

arXiv Computation and Language

Sep 1

Learning to Reason and Use Tools through Unsupervised Fine-Tuning in Task-Oriented Dialog Systems

arXiv:2608.30426v1 Announce Type: new Abstract: Current dialogue systems struggle with dynamic information retrieval, often leading to hallucinations and lower response accuracy. We address this by a...

By Markel Ferro, Oier Lopez de Lacalle

llms agents fine-tuning safety

arXiv AI

Jun 19

DeFrame: Debiasing Large Language Models Against Framing Effects

arXiv:2602. 04306v2 Announce Type: replace-cross Abstract: As large language models (LLMs) are increasingly deployed in real-world applications, ensuring their fair responses across demographics has become crucial.

By Kahee Lim, Soyeon Kim, Steven Euijong Whang

llms benchmarks safety

arXiv Computation and Language

1d ago

The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness

arXiv:2605.28190v2 Announce Type: replace Abstract: Embedding benchmarks like MTEB report a single score per model, implicitly treating robustness as a static, scalar property. We argue that embeddin...

By Manuel Frank, Haithem Afli

llms rag benchmarks

One email a morning, machine-written

Related stories

The Multilingual FrameNet Corpus

BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation

ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs

Learning to Reason and Use Tools through Unsupervised Fine-Tuning in Task-Oriented Dialog Systems

DeFrame: Debiasing Large Language Models Against Framing Effects

The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness