← Back to all news
arXiv Machine Learning By Maxim Bolshim (ITMO University, Saint Petersburg, Russia), Alexander Kugaevskikh (ITMO University, Saint Petersburg, Russia)

Profile-aware diagnostics of discarded parameter-Hessian mass

Read the original on arXiv Machine Learning →

The paper evaluates block‑diagonal curvature models by measuring the squared off‑block fraction of the parameter Hessian across different group sizes and curvature profiles. Experiments on CIFAR‑100 ResNet‑18 and a skip‑free variant show that initial discarded fractions are similar but profiles differ, and that shifting from stage to layer groups changes the estimated off‑block mass relative to the profile envelope. Further comparisons on ResNet‑50/Stanford Cars examine off‑group Hessian actions, residuals, and combined candidates, revealing that off‑group input alone often increases minibatch loss, while combined candidates can reduce loss in some cases, with preferences depending on scale and correction‑norm matching.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Sep 23

Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference

The paper demonstrates that greedy decoding from large language models is not precision‑invariant: the same model, prompt, and decoding algorithm can produce different outputs when run in BF16 versus FP16 on identical hardware. Across six models (1.1B–7B parameters, four families, and 12B) and three benchmarks, 49–100 % of prompts diverge, with a single token flip often cascading into trajectory‑level divergence. The authors develop an empirical error‑propagation analysis that identifies the top‑two logit margin at the LM head as the key factor, and they propose a low‑overhead intervention—selective FP32 LM head recomputation—that improves exact agreement by 22–36 percentage points with less than 4 % latency overhead. "whyItMatters":"The findings reveal that precision choices can fundamentally alter model outputs, challenging the assumption of deterministic greedy decoding and highlighting the need for precision‑aware inference strategies."

By Gaoyuan Du, Anam Nawaz Khan, Rex Zhou, Xiaoyang Liu, Deepayan Chakrabarti, Fnu Suya, Xueping Li
llmsbenchmarks
More like this →
arXiv Machine Learning
Jul 27

A Defense of the Quadratic Model

arXiv:2607. 21716v1 Announce Type: new Abstract: Due to the complexity of neural network loss landscapes, optimization theory is forced to rely on idealized models, and there is generally a tradeoff between how theoretically tractable the model is, and how accurately it describes the true optimization dynamics.

By Alexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan, Sham Kakade, Alex Damian
llms
More like this →
Hugging Face Trending Papers
Oct 13

A Controlled Evaluation of Curvature Inspired Scalar Feature Scaling for Domain Generalization

The paper evaluates a narrow implementation of domain‑specific curvature called HyperDG, which rescales Euclidean encoder features during training with learned positive scalars and uses a standard linear classifier at inference. Across six conditions on four benchmark datasets (PACS, VLCS, Office Home, TerraIncognita) and 384 runs, the method’s macro performance is 1.61 percentage points below native DomainBed ERM, with no statistically significant difference after adjustment. The study concludes that this scalar‑scaling approach does not provide evidence of improvement over ERM and offers a bounded negative result for this specific training package.

More like this →