26  Neural Networks Don’t Escape Statistics

Connects gradient explosion, initialization sensitivity, and gradient clipping back to the loss-function discussion in Chapter 20 — clipping is framed explicitly as “Huber loss’s cousin, applied to the gradient instead of the loss.”

def clip_gradient(grad, max_norm):
    norm = sum(g**2 for g in grad) ** 0.5
    if norm > max_norm:
        return [g * (max_norm / norm) for g in grad]
    return grad