← Back to all news
arXiv Machine Learning By Xiaoyang Li, Runni Zhou, Xinghao Yan

Branch Geometry and Finite-Radius Sensitivity of Hard-ReLU Training

Read the original on arXiv Machine Learning →

The paper investigates how hard‑ReLU training behaves when perturbations have a finite radius. It shows that the usual infinitesimal sensitivities are insufficient to predict the response at a chosen radius, and it characterizes the intermediate regime where the perturbation radius scales with the gradient‑descent step. The authors derive crossing indices, a uniform endpoint expansion for separated transverse events, and provide explicit remainder terms in contractive affine regions to certify finite candidate comparisons, supported by experiments on nonlinear networks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Sep 1

Singular Curvature in ReLU Training:Differentiation and the Gradient-Flow Limit Need Not Commute

The paper investigates the relationship between discrete gradient descent (GD) and its continuous-time gradient-flow counterpart in the context of ReLU neural networks. It shows that while GD states converge over a finite horizon, the exact discrete derivatives obtained via automatic differentiation do not necessarily match the derivative of the limiting flow, due to singular curvature at activation events. The authors provide a Stieltjes representation that separates continuous regional Hessians from atomic interface curvature, revealing rank-one discrepancies at activation jumps and demonstrating that even globally strongly convex residual-ReLU losses can exhibit large sensitivity ratios on certain initialization sets.

By Xiaoyang Li, Runni Zhou
agents
More like this →
arXiv Machine Learning
Sep 4

Hard-ReLU Gradient Descent Selects an Event-Free Sensitivity Limit

The paper investigates how exact automatic differentiation behaves under hard‑ReLU gradient descent. It shows that while gradient‑descent states converge to the piecewise‑smooth gradient flow, the derivative of the training map does not, due to missing event‑time sensitivities captured by saltation matrices. The study demonstrates that for convex objectives, activation events can create large sensitivity gaps, and provides empirical evidence that event‑aware corrections are necessary for accurate flow derivatives.

By Xiaoyang Li, Runni Zhou
More like this →
arXiv Machine Learning
2d ago

Certified Inference and Training for Deep Equilibrium Networks: A Continuation Framework with Polynomial Complexity Guarantees

The paper introduces a certified continuation framework for computing and training deep equilibrium networks (DEQs). It uses compact input homotopy and a rounded Newton tracker for inference, and augments local-plus-low-rank recurrence with programmable dormant bilinear rank‑one channels for training. The approach guarantees polynomial‑time bit complexity, with certified bounds on inference and training error budgets.

By Alex Borisevich
More like this →