← Back to all news
arXiv AI

UTP-Bench: Uncertainty-aware Travel Planning Benchmark

UTP-Bench is a new benchmark for uncertainty-aware travel planning that evaluates large language models on their ability to generate robust itineraries under real-world stochastic conditions. The dataset covers 504 Indian cities, incorporating attractions, restaurants, accommodations, and multi-modal transportation networks, and includes empirical delay distributions and crowd-density patterns to simulate realistic disruptions. Three new metrics—Buffer Adequacy Score, Crowd-Aware Timing Score, and Transport Delay Absorption Score—measure how well generated plans maintain robustness against transit delays and crowd variability, revealing significant gaps between state-of-the-art LLMs and human-authored itineraries.

Related stories

Hugging Face Trending Papers
Sep 2

UTP-Bench: Uncertainty-aware Travel Planning Benchmark

UTP‑Bench is a new benchmark that evaluates large language models on uncertainty‑aware travel planning, incorporating real‑world data from 504 Indian cities and empirical delay and crowd patterns. It introduces three metrics—Buffer Adequacy Score, Crowd‑Aware Timing Score, and Transport Delay Absorption Score—to measure how well generated itineraries remain robust under stochastic conditions. Experiments show that current LLMs lag behind human planners in buffering, delay‑aware scheduling, and crowd sensitivity.

llmsmultimodalbenchmarks
More like this →
arXiv AI
Jun 2

TravelEval: A Comprehensive Benchmarking Framework for Evaluating LLM-Powered Travel Planning Agents

arXiv:2606. 01046v1 Announce Type: new Abstract: The development of Large Language Models (LLMs) has significantly improved travel planning applications, yet evaluating such models is limited by existing benchmarks' limitations: 1) overemphasis on constraint compliance, neglecting multi-dimensional qualities like spatio-temporal cost; 2) datasets lacking real-world authenticity and coverage in key areas (e.

By Weiyi Chen, Shuaixiong Wang, Ziyun Gao, Kaichun Hu, Wangze Ni, Shimin Di, Chen Jason Zhang, Lei Chen
llmsagentsbenchmarks
More like this →
arXiv Machine Learning
Sep 22

CALM: A Calibrated LLM Choice Network Framework for Activity-Based Traveler Simulation

CALM is a reproducible hybrid framework that combines an optional large language model (LLM) activity planner with calibrated stochastic choice, shared network feedback, memory and habit, typed feasibility checks, and deterministic offline replay. It executes a closed traveler‑day loop and evaluates each generative module against an empirical, reproducible baseline, using the 2024 New York City Citywide Mobility Survey data. The framework demonstrates significant improvements in mode‑choice accuracy, quantifies trade‑offs through live‑LLM ablation, and supports controlled stress testing and deterministic replay of downstream simulations.

By Yezhou Cheng
llmsbenchmarks
More like this →
arXiv Machine Learning
Aug 24

Fine-tuning LLMs for Tourist Trajectory Prediction using Field Experiment Data

The paper demonstrates that fine‑tuning large language models (LLMs) on local tourist trajectory data can predict visitor movements under varying conditions. Using 566 trajectories from Wakayama Castle Park, Japan, the authors fine‑tuned Llama‑3.1‑8B, achieving 49.1% accuracy for next point‑of‑interest predictions and maintaining strong performance even on undersampled scenarios such as rainy days. This shows that LLMs can serve as high‑fidelity, context‑aware behavior models for tourist prediction and enable counterfactual analysis of mobility interventions.

By Tatsuya Amano, Hirozumi Yamaguchi
llmsfine-tuning
More like this →