The paper introduces SME-C, a human‑AI workflow that combines lightweight expert critiques with LLM‑driven refinement to create high‑quality, domain‑specific evaluation datasets and rubrics. In a financial business brief domain, SME‑C gathered 3,853 critiques from 411 professionals and used two LLM refinement strategies—one guided by written critiques and another by preference votes—to improve response quality. Blind validation with 100 additional professionals showed both strategies significantly outperformed the experts’ top‑ranked baseline, with critique‑based refinement yielding more domain‑grounded improvements and vote‑based refinement better at removing irrelevant content.
arXiv:2605. 25240v2 Announce Type: replace-cross Abstract: Two methodologies dominate current practices of benchmarking: rubric-based scoring evaluates items against predefined criteria, whereas comparative judgment elicits pairwise preferences between outputs.
arXiv:2608. 04374v1 Announce Type: cross Abstract: Large language models can produce fluent financial analysis, but fluency alone does not establish whether a report is suitable for institutional delivery.
FinAutoRubric is a system that lets experts define reusable evaluation guidance for financial research agents, which the agents then use to automatically generate, review, and validate task‑specific rubrics. The approach replaces costly fixed rubrics with query‑specific ones that incorporate expert standards and can be updated as new information becomes available. On three expert‑authored finance benchmarks, the automatically generated rubrics closely match human scoring and are preferred by in‑house analysts in blind reviews.
arXiv:2608.21374v1 Announce Type: new Abstract: Literature reviews are essential to scientific progress, but rigorously evaluating automatically generated reviews remains difficult because many aspec...
The paper examines LLM-as-a-Judge systems used to assess AI-generated text, questioning the assumption that judgments are derived from reasoning over responses and rubrics. It finds that classifiers trained solely on rubric text can predict judge outputs, indicating that rubrics contain recoverable evaluative signals independent of the responses. Counterfactual experiments show judges often fail to adjust decisions when either the response or rubric criterion is reversed, raising doubts about the reliability of rubric-based LLM evaluation.
The paper proposes using large language models (LLMs) to identify disagreements among models as a way to focus expert effort on revising codebooks for large‑scale text annotation. Three expert feedback methods are evaluated: editing LLM‑generated revisions (Codebook Verifying), answering questions about disagreements (Question Answering), and labeling disagreement cases with rationales (Rationale Labeling). Experiments on tutoring‑session transcripts show that Rationale Labeling achieves the highest LLM‑labeling accuracy (64.9%) compared to the expert‑revised codebook (57.8%), with Question Answering also outperforming the baseline (60.5%).
The paper introduces a lifecycle framework for LLM-as-a-Judge systems used to evaluate recommendation explanations at Netflix. It outlines four phases—Birth, Training, Deployment, and Monitoring—detailing how each stage addresses specific technical and operational challenges. The authors report that after five weeks of A/B testing, judge-aligned explanations increased novel content viewing and successful browse-to-play sessions without quality takedowns.
RubricAtlas is a framework that extracts type-level analytical requirements from expert reports to evaluate deep research agents. It organizes detailed binary checkpoints around recurring analytical questions, linking them to report passages and separating completeness from argument validity. Experiments on 3,079 Chinese financial reports show that individual reports cover only about half of their type’s questions, and that using different reports for requirements changes evaluation scores and agent rankings, with high agreement between judge pipelines.
FinLLM-KG is a model‑agnostic framework that enhances financial sentiment analysis by integrating causal large language models with a financial knowledge graph derived from SEC 10‑K filings. It injects temporally grounded, company‑level context at inference time without updating model weights, allowing sentiment predictions to consider broader firm‑level information. The framework also attributes sentiment scores to individual sentences and retrieved knowledge‑graph facts, improving explainability and supporting human‑in‑the‑loop review, and simulations show it consistently outperforms state‑of‑the‑art baselines in portfolio performance.
RubricAtlas is a framework that extracts type-level analytical rubrics from expert reports to evaluate deep research agents. It organizes binary checkpoints around recurring analytical questions, linking them to report passages and assessing completeness separately from argument validity. Experiments on 3,079 Chinese financial reports show that individual reports cover only about half of their type’s questions, and that using different reports for requirements changes evaluation scores and agent rankings.
The paper proposes a method to capture expert human judgment and reasoning for automated evaluation systems that can scale across high‑stakes domains such as healthcare, defense, finance, governance, and politics. By assembling expert networks and encoding their reasoning into structured editorial standards, the authors demonstrate that automated judges trained on these standards outperform uncalibrated frontier models on source‑quality and neutrality metrics in AI news reporting. The study validates the approach with strong inter‑annotator agreement and high F1 scores, suggesting a viable framework for expert‑calibrated evaluation in any domain where expertise is scarce.
The paper introduces EquiReview‑R, an AI‑assisted review system that treats omission and over‑critique as distinct risks and refines a structured concern set using evidence‑linked reasoning. It demonstrates that more criticism does not guarantee a better review, showing that many high‑recall reviews lack definitive evidence for concerns and that revision before further search is essential. On a held‑out set of papers, EquiReview‑R meets non‑inferiority for major omission, cuts major over‑critique from 15.5 % to 8.1 %, and stops on 52.4 % of papers, with gains attributed to revision rather than extra inference.