Evaluating Local Language Model Agents for Reproducible Data Engineering: An Empirical Software Engineering Study of Mobility Workflows

The study evaluates local open‑weight large language model agents for data‑engineering tasks, using a benchmark of fifteen mobility‑workflow tasks that cover discovery, processing, enrichment, and reporting. Ten local model configurations were tested in one‑shot and closed‑loop conditions, producing 1,500 scored attempts on a consumer‑grade GPU. Results show that models larger than two billion parameters achieve up to 85.3% artifact‑level success, with closed‑loop workspaces boosting pass rates by 26.7–52.0 percentage points, and a quantized 9‑billion‑parameter model reaching 69.3% success with a 6.5 GB memory footprint.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• agents
• efficiency
• benchmarks
• data-engineering