arXiv:2605.30018v3 Announce Type: replace-cross Abstract: Large language models (LLMs) frequently achieve impressive scores on standardized benchmarks, yet accuracy alone offers a limited view of the...
PROOF is a benchmark that profiles the reliability of object-level facts in instruction-tuned language models by converting a frozen Wikidata snapshot into 18,486 English multiple-choice questions grounded in 11,779 semantic facts across 101 classes, 392 properties, and 14 domains. Each question includes an explicit "I don't know" option, a "No correct option" control, and nine controlled formulations, with 1,849 questions designed as no-correct-option traps. The study evaluates 18 open-weight model deployments on 166,374 prompts, revealing wide variability in factual accuracy, sensitivity to wording changes, and the impact of decoder perturbations.
ULME introduces a post‑training framework that unifies text embedding and generation within a single causal decoder‑only LLM by inserting learnable slots for semantic and reasoning embeddings. It trains three objectives—contrastive, language‑modeling, and stop‑gradient predictive—so that generation actively shapes the embedding space. Across Qwen2.5 and Qwen3 models (0.5B–8B), ULME improves MTEB scores by 3.3–4.1 points, tightens BRIGHT and AdvBench‑IR, and matches generation‑only fine‑tunes on standard accuracy benchmarks.
ULME is a post‑training framework that unifies text embedding and language modeling within a single causal decoder‑only LLM by inserting learnable slots for semantic and reasoning embeddings. It trains three objectives—contrastive loss for semantic alignment, language‑modeling loss for answer generation, and a stop‑gradient predictive loss linking reasoning to the answer—so that generation actively shapes the embedding. Across Qwen2.5 and Qwen3 models (0.5B–8B), ULME improves MTEB scores by 3.3–4.1 points, tightens BRIGHT and AdvBench‑IR, and matches generation‑only fine‑tunes on standard accuracy benchmarks.
arXiv:2605. 11374v5 Announce Type: replace Abstract: Test-time compute is widely believed to benefit only large reasoning models, leaving small models with nothing to gain.
arXiv:2510. 20535v2 Announce Type: replace-cross Abstract: Recent techniques such as retrieval-augmented generation or chain-of-thought reasoning have led to longer contexts and increased inference costs.
The paper introduces SCX Router, a lightweight GLiClass-based model selector that assigns suitability scores to inference-time language models without autoregressive generation. It uses a 0.6B-parameter Qwen3 decoder with a shallow bidirectional scorer, preserving a text-only key–value cache across sessions and predicting task attributes such as type, difficulty, and expected output length. The authors build a comprehensive task ontology with 23 families, 115 types, and 1,173 synthetic examples, generating 150,000 verifier-scored tasks to train the router, which outperforms baseline models on LiveBench subsets with a top‑1 score of 0.707 versus 0.696 for the strongest fixed model.
arXiv:2410.02343v2 Announce Type: replace Abstract: Large language models (LLMs) routinely fail to output the correct option in multiple-choice question answering (MCQA) while encoding the answer int...
arXiv:2605.16893v2 Announce Type: replace Abstract: Recent studies have introduced memory-augmented architectures that decouple knowledge storage from neural computation, enabling more direct access...
arXiv:2608. 09351v1 Announce Type: cross Abstract: Test-time scaling improves LLM accuracy but multiplies inference cost, making the accuracy gained per unit of compute the metric that matters in deployment.
arXiv:2608. 06933v1 Announce Type: cross Abstract: Today, we improve models by training and evaluating them on problems at the frontier of their abilities.
Should you replace your text-embedding pipeline with a large language model? We answer this with a controlled, cost-aware comparison of ten LLMs across six families and 26 embedding models (118M to 14B parameters) on 37 tasks spanning classification, semantic textual similarity (STS), clustering, pair classification, and retrieval.