Quick fix
# 1. Safety net: gradient clipping (PyTorch standard API, max_norm=1.0 is a common start)
torch.nn.utils.clip_grad_norm_(model.parameters, max_norm=1.0)
# 2. Resume from the last healthy checkpoint + reduce lr to 1/3~1/10
# (replay the bad batch and a normal batch first to confirm the root cause)
Measured mechanism: lr too high → loss explodes from ~0.05 to inf within ~100 steps (no recovery); same lr with max_norm=1.0 clipping → stays finite and stable;
after injecting a single-step spike, resuming from a healthy checkpoint with a lower lr → normal convergence.
What you will run into
Mid-training (fine-tuning or from-scratch LLM training), the loss curve suddenly spikes: a single-step spike that recovers by itself, consecutive spikes, a spike that never recovers, or a direct NaN / inf.
The most common advice online is "lower the learning rate" — but a symptom is not a cause.
Classify first: data, hyperparameter, or training-chain problem.
Why this happens
One parameter update = learning rate × gradient. A loss spike is fundamentally an out-of-control update size, but three root-cause families share the same symptom:
- Data: bad batch (corrupted samples, over-long sequences, invalid labels) — usually a single-step spike that recovers;
- Hyperparameters / chain: learning rate too high, missing gradient clipping, AMP precision issues — usually consecutive spikes or no recovery;
- Objective / implementation: loss mask, labels, multi-GPU reduction errors — usually "smooth loss but validation does not improve" or multi-GPU-only anomalies.
Route by spike type (article diagnostic table + measured mechanism):
| Symptom | Check first | Minimal verification | Act only after confirmed |
|---|---|---|---|
| Single-step spike, recovers | batch context (tokens/seq len/source/label), grad norm | replay same batch | fix corrupted data; keep valid hard samples |
| Consecutive spikes | grad norm, clip coefficient, relative update, lr | align these metrics | adjust confirmed hyperparam/structure issue |
| Spike, never recovers | relative update, optimizer state | compare with healthy checkpoint | fix root cause, resume from healthy checkpoint |
| First-step NaN | inputs, activation, loss | single-batch FP32 replay | fix the first non-finite op or precision config |
In the training loop, a parameter update is essentially param += lr x gradient, and a loss spike is one (or a few consecutive) out-of-range updates.
The update size is a product of two factors: the learning rate (a hyperparameter, constant or schedule-driven) and the gradient (computed by backprop from the current batch's loss).
Either factor exploding makes the product explode: lr too high = every update is too large, divergence even with normal gradients;
gradient anomaly = a bad batch (corrupted samples / over-long sequences / invalid labels) produces an abnormal gradient direction or magnitude, so a single update is huge.
That's the structure behind "symptom != cause": the spike is the result; you work backwards to which factor lost control.
Two troubleshooting tools map to the two factors: replay the same batch (reproduces reliably with the same data -> data factor; doesn't reproduce -> randomness/parallelism/state factor); watch grad norm (norm persistently rising before clipping -> gradient-factor risk;
norm normal but updates still large -> lr factor).
That's also the mechanism behind clipping as the safety net: it caps the gradient norm at a threshold, adding a fuse to the gradient factor — even if data or computation goes occasionally wrong, the per-step update is clamped to a safe range.
Step by step
1. Classify the spike type — Required
Single-step recovery / consecutive / never recovers / direct NaN. The type decides the debugging route (table above).
2. Replay the same batch — Required
- Reproducible → data problem first (corrupted samples, labels, masks);
- Not reproducible → check random ops, parallel execution, previous-state differences.
3. Watch grad norm, not just loss — Required
Rising pre-clip grad norm → update-chain risk. Use the relative update size (update magnitude / parameter magnitude) to judge whether training is still learning effectively.
4. Pick the fix by root cause — Required
- Corrupted data → fix data; valid hard samples → keep;
- Out-of-control updates → gradient clipping / lower lr (change one variable at a time);
- Already NaN / no recovery → resume from a healthy checkpoint; do not patch model params and keep using suspicious optimizer state.
5. Prevent recurrence: pilot run — Required
Small-scale run before real training: full warmup, a stretch at peak lr, representative data, at least one validation, fixed random seed. Monitor at three timescales: per-step (finite loss/gradients), window (trend deviation), periodic (validation metrics).
How to confirm it works
Loss regains a downward trend with no new spikes; grad norm returns to a healthy range; training progresses after resuming from the healthy checkpoint; validation metrics improve in sync.
Other environments
| Environment | Notes |
|---|---|
| Single-GPU OK, multi-GPU broken | Check reduction and control flow first (global valid token count, loss reduction, consistent backward across ranks), not hyperparameters |
| FP16/AMP | Unscale before computing and clipping gradients (GradScaler decision flow in the article) |
| Framework differences | Every framework has a clipping API (PyTorch clip_grad_norm_ / TF clip_by_global_norm etc.); same mechanism |
| Not measured | This entry is an equivalent minimal mechanism reproduction (numpy 2-layer MLP), not a full LLM training framework run; LLaMA-Factory fine-tuning stack: combo-local-llm-finetune |
Symptom
Loss curve suddenly spikes: single-step spike with recovery, consecutive spikes, spike with no recovery, or direct NaN/inf; grad norm rises abnormally at the same time.
Cause
Update size = learning rate × gradient; a spike means the update size is out of control. Three root-cause families: data (bad batch), hyperparameters/chain (lr too high, missing clipping, AMP precision), objective/implementation (mask, label, multi-GPU reduction errors). Symptom ≠ cause; evidence first, then act.
Solution
- Locate by spike type (single/consecutive/no-recovery/NaN);
- Replay the same batch + inspect grad norm / relative update;
- Fix by root cause: repair data / gradient clipping / lower lr / resume from healthy checkpoint (one variable at a time);
- Pilot run + tiered monitoring before real training to prevent recurrence.
Verification Evidence
- Measured (2026-08-30, Python 3.14 + numpy 2.5.2, venv + Tsinghua mirror): 2-layer MLP regression y=sin(x) mechanism reproduction — lr=0.01 converges (loss 0.065→0.026); lr=1.0 diverges (loss 5.4e51 at step 25, inf at step 98, no recovery); lr=1.0+clip=1.0 stays finite (grad norm clipped to 1.0); after injecting a single-step spike (loss 0.048→0.224), resuming from the healthy checkpoint with a lower lr converges (0.028). local repro log 2026-08-30.
- Source: cnblogs author "Shitou Kaihui": "Model Training Stability: From the Training Pipeline to the Troubleshooting Manual" (Chinese original, 2026-08-28) — <https://www.cnblogs.com/GlenTt/p/22742530>.
- Not measured: live reproduction on a full LLM training framework (PyTorch/LLaMA-Factory) — heavy environment; labeled "mechanism reproduced, environment not reproduced" (SM-076).
Applicable tools (1)
This problem is linked from the tool pages below — reachable from either side.
pythonBundled solutions (combos)
This problem is part of the following verified bundles.
Source & evidence
← Local LLM domain ｜ Problem Catalog Experience ｜ Facing another issue? Go to the Helpdesk