AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

Simon Willison
Oct 13

Redesigning source control at Anthropic

Anthropic redesigned its source control system, creating a custom alternative called Scion to replace Git. The change was driven by a dramatic rise in code changes—from about 2,500 per day in May 2026 to over 30,000 per day by September—requiring a system that could handle a 100 GB main‑branch history and 25 million daily checkouts. Scion was built by a seven‑person team, migrated from Git in five months, and includes a new code‑review system named Antique, while offline support was deemed unnecessary due to Claude’s constant online use.

Hugging Face Trending Papers
Oct 13

PixelMoE: Scaling Pixel-Space Diffusion with LLM-Style Mixture-of-Experts

PixelMoE introduces a pixel-space generative framework that applies LLM-style Mixture-of-Experts (MoE) to diffusion models, enabling sparse scaling of model capacity without diffusion-specific routing. The approach combines Token-Choice routing, fine-grained routed experts, a shared expert, and bias-based load balancing, while also employing Pixel-Space Hierarchical Flow Matching to adjust spatial resolution during denoising. PixelMoE outperforms dense counterparts at comparable active parameter counts, achieving strong FID scores on class-conditional ImageNet-1K.

Hugging Face Trending Papers
Oct 13

Self-Evolving Agent Harness

The paper introduces the Self‑Evolving Agent Harness, a framework that allows an agent’s software scaffolding—prompts, memory, workflows, and executable components—to evolve while keeping the underlying model weights unchanged. It formalizes a feedback‑driven process where accepted revisions persist across runs, separating update targets from proposal and admission mechanisms, and distinguishes changes to interpreted state from revisions to tools and code. The authors analyze evolutionary search, provide generalization and retention bounds under PAC‑learning assumptions, and discuss the impact of persistent harness revisions on knowledge sharing, organizational workflows, and broader implications for scientific reliability, safety, and accountability.

Hugging Face Trending Papers
Oct 13

Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judges

The paper introduces ReVeR, a pointwise rubric reward model that derives hard rules and graded principles directly from instructions, avoiding response-induced bias. It employs tool-augmented judges and label‑blind distillation to create ReVeR‑Traces, a large dataset of rubric reasoning and judging trajectories. Experiments show ReVeR surpasses pairwise rubric models in preference accuracy, especially for instruction following, and reveal that higher preference accuracy does not always lead to better downstream policies.

Hugging Face Trending Papers
Oct 13

Beyond Answer Voting: Fine-Grained Correction of LLM Reasoning through Graph Alignment across Sampled Traces

Beyond Answer Voting: Fine-Grained Correction of LLM Reasoning through Graph Alignment across Sampled Traces proposes DDRG, a framework that uses graph alignment to detect claim-level disagreements in reasoning traces and selectively corrects them with a verifier. DDRG builds source‑grounded claim graphs, sends localized conflicts to an answer‑blind verifier, and applies a focused repair only when the verifier supports the majority answer’s path. Experiments on five benchmarks show that zero‑shot DDRG improves macro‑average accuracy by 8.65–15.83 percentage points over direct answering and outperforms CoT, GoT, GoV, and SC on this metric.

Hugging Face Trending Papers
Oct 13

Do Factuality Methods Make LLMs More Factual? Reproducibility and Fair Comparison

The paper reproduces five factuality alignment methods for large language models—three on‑policy RL approaches (PPO, GRPO, Dr.GRPO) and two offline preference optimization methods (DPO, KTO)—using a common benchmark and evaluation metrics. None of the methods consistently outperforms the base model, and the rewards appear to favor format, length, or brevity rather than factuality or relevance. The study also highlights problems with current evaluation practices, noting that metrics and benchmarks often fail to measure prompt relevance and rely on fragile LLM judges and retrieval for verification.

Hugging Face Trending Papers
Oct 13

FALLNEST: Dual-Domain Dataset Condensation of Imbalanced IMU Data for Wearable Fall Detection

FALLNEST is a data‑centric framework that condenses large, imbalanced inertial measurement unit (IMU) datasets into tiny, class‑balanced synthetic training sets for wearable fall detection. It combines dual‑domain time‑series condensation, normalized‑gradient matching, MMD‑based embedding alignment, and multi‑view Fourier augmentation to generate 32–146 synthetic windows that enable a 102K‑parameter TinyCNN to achieve 93.5–95.8% mean accuracy and 0.89–0.98 PR‑AUC on SisFall, KFall, and MobiFall benchmarks.

Hugging Face Trending Papers
Oct 13

LiFUSE: Frequency-Spatial Unified Self-Supervised Pretraining for Liver Disease Analysis on Non-contrast CT

LiFUSE introduces a unified frequency‑spatial self‑supervised pretraining framework for liver disease analysis on non‑contrast CT scans. The method employs a frequency branch that uses random band‑stop masking and spectrum reconstruction, and a spatial branch that performs hybrid‑granularity masked volumetric reconstruction, with a cross‑domain consistency scheme to fuse the two branches. Experiments on three clinical tasks and a public transfer setting demonstrate that LiFUSE consistently outperforms strong baselines, indicating its potential to enhance liver disease screening using low‑contrast NCCT images.