← Back to all news
arXiv Machine Learning

Profile-aware diagnostics of discarded parameter-Hessian mass

The paper evaluates block‑diagonal curvature models by measuring the squared off‑block fraction of the parameter Hessian across different group sizes and curvature profiles. Experiments on CIFAR‑100 ResNet‑18 and a skip‑free variant show that initial discarded fractions are similar but profiles differ, and that shifting from stage to layer groups changes the estimated off‑block mass relative to the profile envelope. Further comparisons on ResNet‑50/Stanford Cars examine off‑group Hessian actions, residuals, and combined candidates, revealing that off‑group input alone often increases minibatch loss, while combined candidates can reduce loss in some cases, with preferences depending on scale and correction‑norm matching.

Related stories

arXiv Machine Learning
Sep 23

Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference

The paper demonstrates that greedy decoding from large language models is not precision‑invariant: the same model, prompt, and decoding algorithm can produce different outputs when run in BF16 versus FP16 on identical hardware. Across six models (1.1B–7B parameters, four families, and 12B) and three benchmarks, 49–100 % of prompts diverge, with a single token flip often cascading into trajectory‑level divergence. The authors develop an empirical error‑propagation analysis that identifies the top‑two logit margin at the LM head as the key factor, and they propose a low‑overhead intervention—selective FP32 LM head recomputation—that improves exact agreement by 22–36 percentage points with less than 4 % latency overhead. "whyItMatters":"The findings reveal that precision choices can fundamentally alter model outputs, challenging the assumption of deterministic greedy decoding and highlighting the need for precision‑aware inference strategies."

By Gaoyuan Du, Anam Nawaz Khan, Rex Zhou, Xiaoyang Liu, Deepayan Chakrabarti, Fnu Suya, Xueping Li
llmsbenchmarks
More like this →
arXiv Machine Learning
Jul 27

A Defense of the Quadratic Model

arXiv:2607. 21716v1 Announce Type: new Abstract: Due to the complexity of neural network loss landscapes, optimization theory is forced to rely on idealized models, and there is generally a tradeoff between how theoretically tractable the model is, and how accurately it describes the true optimization dynamics.

By Alexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan, Sham Kakade, Alex Damian
llms
More like this →
Hugging Face Trending Papers
Oct 13

A Controlled Evaluation of Curvature Inspired Scalar Feature Scaling for Domain Generalization

The paper evaluates a narrow implementation of domain‑specific curvature called HyperDG, which rescales Euclidean encoder features during training with learned positive scalars and uses a standard linear classifier at inference. Across six conditions on four benchmark datasets (PACS, VLCS, Office Home, TerraIncognita) and 384 runs, the method’s macro performance is 1.61 percentage points below native DomainBed ERM, with no statistically significant difference after adjustment. The study concludes that this scalar‑scaling approach does not provide evidence of improvement over ERM and offers a bounded negative result for this specific training package.

More like this →
arXiv AI
Oct 27

Fantastic Pretraining Optimizers and Where to Find Them

The paper evaluates ten deep learning optimizers across models ranging from 0.1B to 1.2B parameters and varying data-to-model ratios, highlighting that fair comparisons require rigorous hyperparameter tuning and end-of-training evaluations. It finds that claimed speedups over AdamW are lower than advertised, dropping to only 1.1× for the largest models, and that intermediate checkpoint comparisons can mislead due to learning‑rate decay. The study shows that matrix‑based optimizers like Muon and Soap achieve the highest speedups, but these benefits diminish as model size increases.

By Kaiyue Wen, David Hall, Tengyu Ma, Percy Liang
llms
More like this →
arXiv Machine Learning
Aug 20

Learned, Then Lost: A Measured Single-Example Counterfactual in Pre-training

The study measured the impact of a single training example on a GPT‑2 model by running 24 counterfactual experiments. 32 models were trained from scratch on OpenWebText, and at a specific training step a single batch row was replaced with a 194‑token passage under three conditions (fluent prose, fabricated subject, random characters) or left unchanged. Results showed that the passage was learned from one exposure and decayed, with measurable differences in cross‑entropy up to 50 steps after injection but no lasting effect at the final step.

By Zachary Speck, Asa Shepard
llmssafety
More like this →