arXiv Computation and Language
Sep 1

WildSEEK: Evaluating Language Models for Information-Seeking

WildSEEK is a new dataset of 3,000 real user information‑seeking queries, manually annotated for risk‑sensitive domains and whether the query is factoid or analytical. The accompanying evaluation framework tests LLM responses against four failure criteria—sycophantic behavior, overreliance, a default US‑centric perspective, and poor handling of vulnerable populations—finding higher failure rates for analytical queries. The authors also train classifiers on WildSEEK to analyze over 1.8 million realistic queries, revealing that more than a third are high‑risk and often analytical.

By Tanise Ceron, Joachim Baumann, Elisa Bassignana, Berat Cabuk, Dirk Hovy, Debora Nozza
arXiv AI
Jul 24

Benchmarking the Personalization Capabilities of Large Language Models

arXiv:2607. 20471v1 Announce Type: new Abstract: Personalization, the act of varying a message to induce action from a specific receiver while keeping sender, channel, and time fixed, has a long tradition in psychology and marketing as a two-party problem in which sender and receiver have independent objectives.

By Ashutosh Srivastava, Siddharth Yedlapati, Vinay Aggarwal, Yaman Kumar Singla, Shashwat Dixit, Jitendra Ajmera, Balaji Krishnamurthy
arXiv Machine Learning
Aug 28

Token-Level Advertising

The paper introduces the Latent Advertiser Mixture Auction (LAMA), a token‑level advertising framework that integrates advertiser influence directly into the text generation process. Advertisers provide local continuation values that shape next‑token policies, and the platform decodes these through a latent mixture while updating an allocation posterior. LAMA is shown to satisfy Markov DSIC and IR, achieve near‑optimal KL‑regularized welfare, and, in proof‑of‑concept experiments on commercial‑search queries, improve platform welfare and revenue without compromising user‑facing response quality.

By Hanbing Liu, Bowei Zhang, Changyuan Yu, Yinyu Ye, Qi Qi