PyTorch: RuntimeError: CUDA error: device-side assert triggered
RuntimeError: CUDA error: device-side assert triggered. Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Immediate Remediation
bash
CUDA_LAUNCH_BLOCKING=1 python train.py
Root Cause Analysis
A CUDA kernel launched asynchronously encountered an assertion failure (e.g., index out of bounds) but the error is only reported at the next synchronization point, masking the actual line. Running with CUDA_LAUNCH_BLOCKING=1 forces synchronous execution to pinpoint the exact location.
Verification & Guardrails
- May slow down training significantly
- Also set TORCH_USE_CUDA_DSA=1 for more detailed diagnostics
- Check target tensor bounds and loss function inputs
Have a custom or uncategorized crash?
Run your trace through our in-memory client privacy sandbox for instant SRE remediation.