QUORUM is a budget‑aware routing framework that dynamically assigns each annotation instance to either human or large language model (LLM) annotators, using feature‑based signals to estimate difficulty and multiple annotations per instance. It combines annotations through agreement‑based rewards to improve reliability, without relying on model confidence or uncertainty estimates. Experiments on diverse closed‑ and open‑ended tasks in English and multilingual settings show that QUORUM can boost annotation quality by up to 34.4% while cutting costs by 8.8% compared to existing methods.
arXiv:2511. 09373v2 Announce Type: replace-cross Abstract: LLMs now tackle a wide range of software-related tasks, yet we show that their performance varies markedly both across and within these tasks.
The paper introduces VDAR-Router, a routing framework for large language models that uses verbalized query difficulty analysis to guide model selection. It first generates an explicit difficulty profile for each query, retrieves historical examples with similar profiles, and then estimates model suitability to choose a model based on a reward function balancing performance and cost. Experiments on three datasets show that VDAR-Router consistently outperforms existing baselines in cost‑performance trade‑offs, and case studies confirm that explicit difficulty analysis improves example relevance and routing reliability.
arXiv:2606. 24734v1 Announce Type: cross Abstract: High-quality annotations of structured representations are expensive to collect over large corpora.
arXiv:2608. 10939v1 Announce Type: cross Abstract: Multilingual short-text classification supports operational systems such as content moderation, customer support routing, and intent recognition, yet aggregate evaluation often hides large differences between high-resource and low-resource languages.
arXiv:2608. 08700v1 Announce Type: new Abstract: Reliable evaluation of tool routing is critical as Large Language Models increasingly operate as autonomous agents.
arXiv:2603.04445v3 Announce Type: replace-cross Abstract: The rapid growth of large language models (LLMs) with diverse capabilities, costs, and domains has created a critical need for intelligent mo...
MiCRo is a two‑stage framework that improves personalized preference learning for large language models. It first uses a context‑aware mixture model to capture diverse human preferences from large binary preference datasets, then applies an online routing strategy to dynamically adjust mixture weights based on context, reducing ambiguity. Experiments on multiple datasets show that MiCRo captures diverse preferences and enhances downstream personalization.
arXiv:2604. 17289v2 Announce Type: replace Abstract: Supervised fine-tuning of large language models relies on human-annotated data, yet annotation pipelines routinely involve multiple crowdworkers of heterogeneous expertise.
arXiv:2607. 11399v1 Announce Type: cross Abstract: Large language model agents are increasingly executed not by a single model call, but by an execution harness that manages observation, context, control, action, state, and verification.
The paper "Towards Optimizing SQL Generation via LLM Routing" proposes a routing approach for Text-to-SQL tasks that dynamically selects the most cost‑effective large language model (LLM) for each query. Two routing strategies—score‑based and classification‑based—are introduced, achieving accuracy comparable to the best LLM while reducing latency and monetary cost. The authors design the routers for easy training and efficient inference, and demonstrate a practical accuracy‑cost trade‑off on the BIRD dataset.
arXiv:2609.38660v1 Announce Type: cross Abstract: Long-form subtitle translation requires reasoning over discourse and cultural context spanning episodes or entire series, while maintaining consisten...
arXiv:2608.30224v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly used to annotate structured product data in e-commerce, but early deployment often begins as a cold-sta...