The paper introduces a six‑dimension prompt‑side structural‑complexity index to assess code‑generation reliability before a model generates output. Using 5,000 Python prompts and 21 large language models, the authors find that pass rates exhibit a non‑monotonic breakpoint around a composite score of 13.75, with task‑type and construction‑frame adjustments shifting this threshold. The study also reports high inter‑rater reliability (ICC = 0.872) and demonstrates that the index can predict failure likelihood without relying on output correctness.
Complexity measured from generated code is failure-dependent: a difficult prompt can yield a short failing program and be assigned low output complexity. We introduce a six-dimension prompt-side struc...
arXiv:2606. 00308v1 Announce Type: cross Abstract: Large-language-model code generation has shifted from single-shot prompting to multi-agent orchestrations - analyst, coder, tester, and debugger pipelines - and is evaluated almost exclusively on functional correctness.
arXiv:2608. 08822v1 Announce Type: new Abstract: Cognitive decision-making research depends on diverse scenarios with carefully controlled complexity, yet manual production is slow, inconsistent, and biased.
arXiv:2607. 01903v1 Announce Type: new Abstract: LLM-integrated applications blend natural language prompts with program code, and much of their runtime behavior originates in the prompt layer rather than in the code itself.
arXiv:2607. 11981v1 Announce Type: cross Abstract: Aggregate reliability estimates can obscure heterogeneity in measurement-design burden across response conditions, so a single G- or D-study may mischaracterize a design's adequacy for particular strata.
LLM-integrated applications blend natural language prompts with program code, and much of their runtime behavior originates in the prompt layer rather than in the code itself. Existing complexity metrics, however, operate solely at the code level and therefore overlook this behavioral logic entirely.
arXiv:2609.37405v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used for software engineering tasks that require understanding existing source code, including behavior...
The study evaluates zero‑shot essay scoring using sub‑3B open language models that run locally on a single 8 GB consumer GPU. Four instruction‑tuned models (Qwen2.5‑0.5B, 1.5B, 3B and SmolLM2‑1.7B) were tested on all eight ASAP‑AES prompts, comparing rubric‑decomposed versus holistic prompting, different aggregation methods, and trait‑mapping strategies. Results show rubric‑decomposed prompting consistently outperforms holistic prompting, trait‑mapping is sensitive to calibration, and longer essays reduce error, yet the best local configuration (macro QWK 0.388) still falls short of human agreement and a length‑only baseline, suggesting these models are best suited for formative, human‑supervised feedback.
arXiv:2607. 19257v1 Announce Type: cross Abstract: Practitioners make three prompt-design decisions with almost no controlled evidence behind them: how to format instructions and context (markdown, plain text, prose, or tabular), how many simultaneous instructions a system prompt can carry before compliance degrades, and how much context a model can hold before recall and honesty degrade.
Aggregate reliability estimates can obscure heterogeneity in measurement-design burden across response conditions, so a single G- or D-study may mischaracterize a design's adequacy for particular strata. This study introduces a conditional generalizability framework with three components.
arXiv:2609.09245v1 Announce Type: new Abstract: Repeated-sampling evaluations increasingly extrapolate pass@k far beyond the number n of samples collected per problem. We show that, in the pooled/ran...
arXiv:2608. 12426v1 Announce Type: new Abstract: Large language models are increasingly deployed in settings that require simultaneous adherence to multiple explicit constraints - reasoning structure, safety boundaries, output schemas.