LifeVal is a new benchmark that evaluates AI model performance on personalized agentic consumer tasks, covering over 1.7 trillion US dollars in annual willingness-to-pay across 148 task categories such as managing finances and finding childcare. The benchmark was built using a survey of more than 3,800 consumers and cross‑referenced with national time‑use studies. Task creators, who met strict expertise thresholds, designed AI evaluation tasks personalized with real ChatGPT conversations, memories, reference files, and personal context, and all tasks underwent at least three rounds of human quality control before inclusion.
We introduce LifeVal, a benchmark evaluating AI model performance on personalized agentic consumer tasks. LifeVal covers the most valuable life tasks that a personalized digital superassistant could a...
arXiv:2608. 13833v1 Announce Type: cross Abstract: Conversational advertising aims to deliver useful ads within multi-turn assistant interactions.
arXiv:2608. 10875v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly deployed as personal assistants.
Large language model (LLM) agents are increasingly deployed as personal assistants. Existing evaluations, however, mostly use short, self-contained requests in static environments.
The paper presents a systematic study of token consumption in AI agents performing coding tasks. It finds that agentic tasks are far more expensive—about 1000 times more tokens than code reasoning or chat—primarily due to input tokens, and that token usage varies wildly, with accuracy peaking at moderate costs. Models differ significantly in efficiency, and current frontier models cannot reliably predict their own token usage, often underestimating it.
The paper introduces Test-Time Adaptation through Human‑Agent Interaction (TAHI), a method that uses iterative human feedback to adapt AI agents to individual users’ criteria. By integrating cross‑session interaction data into agent context and weights, and building an evolving rubric module, the authors demonstrate that agents can improve task success by 4.5–20.9% after only a few interactions. The evolving rubric also serves as a scalable annotation tool, detecting 16.0–22.3% more failures than language models or humans alone, and personalized agents can even generalize improvements up to 8.8% across users.
The paper introduces Active Preference Acquisition (APA), a method for web agents to decide which unknown user preferences to ask about by evaluating the Prospective Personalization Value (PPV)—the expected cumulative benefit across future tasks minus the cost of asking. APA calculates counterfactual gains for each candidate preference using task prototypes and aggregates them by expected reuse, enabling agents to prioritize questions that may not help immediately but improve long‑term performance. Experiments on the ActiveWebPref benchmark show APA can acquire a large share of valuable preferences and achieve higher net utility than baseline strategies, demonstrating that considering future reuse is crucial for effective personalization.
The paper introduces Active Preference Acquisition (APA) for long‑term personalized web agents, defining the Prospective Personalization Value (PPV) of a missing preference as its expected cumulative utility over future tasks minus the cost of asking. APA selects questions within a per‑interaction budget, prioritizing preferences that may not benefit the current task but can be reused across many future tasks. Experiments on the ActiveWebPref benchmark show APA acquiring 80.3% of amortization‑dependent preferences and achieving higher net utility than baseline methods, while limiting the number of questions asked.
arXiv:2608. 04205v1 Announce Type: new Abstract: Human evaluation of AI systems and digital products is costly, slow, and difficult to scale.
PersonaMem-v3 is a benchmark and evaluation harness designed to assess omni-platform personal intelligence for AI agents. It is built from over one million anonymized real-world engagement histories, covering social media, chatbots, calendars, and AI companions, and tracks user preferences and habits over time. The benchmark tests agents on personalization, LLM-powered recommendation, proactiveness, agentic tool use, and geo-temporal reasoning, evaluating their ability to infer holistic user understanding, personalize responses, rerank recommendations, follow user steering, and avoid inappropriate personalization.
arXiv:2606. 12871v1 Announce Type: new Abstract: Search Agents (SAs) typically leverage large language models (LLMs) to support complex information-seeking tasks by autonomously exploring web sources and synthesizing information into comprehensive responses.
arXiv:2608. 13940v1 Announce Type: new Abstract: AI research agents (AIRA) can now propose, implement, and evaluate their own machine learning experiments, but progress on frontier tasks is throttled by cost: a candidate solution can be written in minutes, whereas evaluating it can take hours to days of GPU time.