PyTorch: RuntimeError: CUDA error: an illegal memory access was encountered / misaligned address
RuntimeError: CUDA error: an illegal memory access was encountered at /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142
Immediate Remediation
bash
CUDA_LAUNCH_BLOCKING=1 TORCH_USE_CUDA_DSA=1 python train.py 2>&1 | tee cuda_dsa.log
Root Cause Analysis
A CUDA kernel attempted to read or write outside the bounds of a tensor's allocated memory, often due to invalid indexing (e.g., out-of-range indices, negative strides, or non-contiguous tensors passed to index operations). This corrupts memory and triggers an asynchronous illegal memory access error at the next synchronization point.
Verification & Guardrails
- TORCH_USE_CUDA_DSA=1 requires PyTorch built with CUDA 10.2+ and may slow execution; use only for debugging.
- After fixing the indexing logic, verify with a small test tensor and `torch.cuda.synchronize()` to confirm no residual errors.
- If the error persists, check for mixed precision or custom CUDA extensions that may cause misaligned memory access.
Have a custom or uncategorized crash?
Run your trace through our in-memory client privacy sandbox for instant SRE remediation.