arXiv AI
Jun 11
Soft-Prompt Tuning for Fair and Efficient LLM Benchmark Evaluation
arXiv:2606. 12117v1 Announce Type: cross Abstract: Benchmark scores often misrepresent a large language model's (LLM's) knowledge, because they rely, e.
Beyond Prompt and Parser Engineering: Soft-Prompt Tuning for Automated LLM Evaluation
The paper introduces soft‑prompt tuning as a lightweight, automated method for evaluating large language models (LLMs). By optimizing only ten soft‑prompt vectors—about 0.0006% of a 7B model’s parameters—within 150 steps, the approach adapts models to specific benchmark formats, improving format compliance while preserving knowledge accuracy. Experiments across seven models and eleven datasets show that this method matches strong evaluation‑harness baselines, enhances evaluation signals on challenging datasets, and yields cleaner outputs that align better with exact‑match scoring and independent LLM judges.
Related stories
arXiv AI
Sep 4
RECAST: Expanding the Boundaries of LLMs' Complex Instruction Following with Multi-Constraint Data
RECAST is a new framework that generates datasets with far more constraints per example than existing benchmarks, aiming to push large language models (LLMs) to better follow complex instructions. The authors built RECAST-30K, a 30,000‑instance dataset covering 19 constraint types extracted from real prompt‑response pairs, and showed that fine‑tuning on it improves LLMs’ ability to handle complex tasks without harming general performance. RECAST also provides rule‑based and LLM‑based validators for automatic constraint verification, enabling reward‑based reinforcement learning to further enhance model performance on challenging tasks.
arXiv AI
Jun 9
Knowing How to Edit: Reliable Evaluation Signals for Diagnosing and Optimizing Prompts at Query Level
arXiv:2511. 19829v3 Announce Type: replace Abstract: Prompt optimization has become a central mechanism for eliciting strong performance from LLMs, and recent work has made substantial progress by proposing diverse prompt evaluation metrics and optimization strategies.
arXiv Machine Learning
Jul 14
FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale
arXiv:2601. 22146v2 Announce Type: replace-cross Abstract: Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data.
arXiv AI
Jun 24
Matching Tasks to Objectives: Fine-Tuning and Prompt-Tuning Strategies for Encoder-Decoder Pre-trained Language Models
arXiv:2606. 24841v1 Announce Type: new Abstract: Prompt-based learning has emerged as a dominant paradigm in natural language processing.
arXiv AI
Jun 11
APEX: Automated Prompt Engineering eXpert with Dynamic Data Selection
arXiv:2606. 11459v1 Announce Type: cross Abstract: Large Language Models are highly sensitive to prompt formulation, necessitating automatic prompt optimization to unlock their full potential.
arXiv AI
Aug 24
PromptResponse: Optimizing Prompts for LLM Coding Tasks
arXiv:2608.21074v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used in research workflows and software development pipelines, yet their output remains sensitive to in...
arXiv AI
Jun 26
Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement
arXiv:2606. 27226v1 Announce Type: new Abstract: Evaluating LLM outputs remains a major bottleneck in NLP: human evaluation is expensive and slow, lexical metrics correlate poorly with human judgments on open-ended generation, and holistic LLM judges often produce opaque scores that are hard to debug.
arXiv AI
Jun 26
NebulaExp-8B: An Empirical Post-Training Pipeline via Full-Scale Ablation Research
arXiv:2606. 26671v1 Announce Type: new Abstract: Post-training alignment determines the reasoning and human preference following capabilities of large language models, yet most existing works withhold detailed data construction, filtering rules and training recipes, which hinders community reproducibility and lightweight model optimization.
Hugging Face Trending Papers
Jun 25
Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement
Evaluating LLM outputs remains a major bottleneck in NLP: human evaluation is expensive and slow, lexical metrics correlate poorly with human judgments on open-ended generation, and holistic LLM judges often produce opaque scores that are hard to debug. We propose BINEVAL, a framework that decomposes evaluation criteria into atomic binary questions and aggregates the resulting verdicts into interpretable, multi-dimensional scores.
Hugging Face Trending Papers
Jul 30
IFHierBench: Hierarchical Instruction Following for Large Language Models
Instruction-following ability is critical for deploying large language models in real-world applications, where downstream components depend on the output satisfying specific constraints. Modern deployments increasingly handle the full task in a single LLM call, with one prompt specifying a layered output whose overall artifact, structural sections, and nested fields must each satisfy concrete constraints.
arXiv Machine Learning
Jun 5
Can LLMs Write Correct TLA+ Specifications? Evaluating Natural-Language-to-TLA+ Generation
arXiv:2606. 05792v1 Announce Type: cross Abstract: TLA+ has supported industrial verification at companies such as Amazon and Microsoft, yet writing correct TLA+ specifications from natural language still requires time and expertise, which limits adoption.