← Back to all news
arXiv Computation and Language By Juliette Woodrow, Chris Piech

PromptNCE: Conditional Probabilities and PMI Using Only LLMs and Contrastive Estimation Prompts

Read the original on arXiv Computation and Language →

The paper introduces PromptNCE, a zero‑shot method that uses large language models to estimate pointwise mutual information (PMI) by framing conditional probability estimation as a contrastive task with an explicit OTHER category. The authors benchmark PromptNCE against four other prompting‑based estimators on three human‑annotated datasets, finding that PromptNCE achieves the best conditional probability estimates and Spearman correlations up to 0.78 for full PMI. A case study demonstrates the method’s utility for scoring student knowledge summaries in low‑data settings, and the authors release code and prompts for reproducibility.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Sep 3

Benchmarking Language Models for Statistical Problem Formulation

Large language models are increasingly used to assist with statistical and data science tasks, but current evaluations assume the analysis goal is already defined. This paper formalizes the upstream step of Statistical Problem Formulation into two subtasks—classification of the statistical problem and identification of relevant variables—and introduces StatFormBench, a benchmark comprising 1,013 samples from five statistics textbooks and a data science case library. Across 14 open- and closed‑source LLMs, the best zero‑shot models achieve only 72.0% fine‑grained classification accuracy and 63.2% variable set overlap, with no model consistently excelling in both subtasks and limited gains from enhanced prompting strategies.

By Chen Wang, Junzhe Zhao, Xin Cong, Wanlu Deng, Ke Deng
llmsbenchmarks
More like this →
arXiv Machine Learning
Sep 14

Limits of LLM Text Detectors in Education

The paper "Limits of LLM Text Detectors in Education" argues that existing LLM‑generated text detectors assume a binary human/LLM distinction, which fails to capture realistic student‑AI collaboration. It introduces a contribution‑aware evaluation framework with eight student contribution levels and presents GEDE, a benchmark of over 900 human‑written and 12,500 generated essays across 886 tasks. Using GEDE, the authors evaluate four detection methods and find that most detectors perform poorly on intermediate contribution levels, especially LLM‑assisted revisions, raising concerns about false accusations.

By Lukas Gehring, Benjamin Paa{\ss}en
llmsbenchmarks
More like this →
arXiv AI
Sep 3

Task-Level Natural Language Priors as Learning Signals for Low-Resource LLM Training

Task-Level Natural Language Priors as Learning Signals for Low-Resource LLM Training proposes Prior-Guided Tuning (PGT), a training approach that treats natural-language priors as auxiliary learning signals rather than just input context. The method introduces Contrastive Prior Steering (CPS), which adds positive and negative prior-conditioned auxiliary losses while preserving the original supervised objective. Experiments on AmbiMath, Jigsaw, and MNLI/HANS demonstrate that CPS consistently outperforms plain and prompt fine-tuning, achieving high accuracy and significant gains with limited training data.

By Jian Gao, Xiao Zhang, Xun Zhu, Miao Li, Ji Wu
llmsfine-tuning
More like this →