[distributed] test_distributed_checkpoint.py got atl_status: FAILURE randomly.

[daisyden]

🐛 Describe the bug

On PVC 1100 ZE_AFFINITY_MASK=0,1
GPU 0/0 GPU 1/0 GPU 2/0 GPU 3/0 CPU Affinity
GPU 0/0 S XL4 SYS SYS 0-55,112-167
GPU 1/0 XL4 S SYS SYS 0-55,112-167
GPU 2/0 SYS SYS S XL24 56-111,168-223
GPU 3/0 SYS SYS XL24 S 56-111,168-223

branch https://github.com/daisyden/pytorch/tree/distributed_2.8 + torch-xpu-ops 11320f3, build with 2025.0 test_distributed_checkpoint.py got random failures, sometimes it can pass.
pytest -v test_distributed_checkpoint.py -k test_distributed_checkpoint_state_dict_type0_xpu

11 ../../../../test/distributed/fsdp/test_distributed_checkpoint.py::TestDistributedCheckpointXPU::test_distributed_checkpoint_state_dict_type0_xpu 2025:04:02-03:30:40:(4119976) |CCL_WARN| did not find MPI-launcher specific v ariables, switch to ATL/OFI, to force enable ATL/MPI set CCL_ATL_TRANSPORT=mpi
12 2025:04:02-03:30:40:(4119976) |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL
13 2025:04:02-03:30:40:(4119977) |CCL_WARN| did not find MPI-launcher specific variables, switch to ATL/OFI, to force enable ATL/MPI set CCL_ATL_TRANSPORT=mpi
14 2025:04:02-03:30:40:(4119977) |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL
15 2025:04:02-03:30:40:(4119976) |CCL_WARN| host: a4bf01946f22.jf.intel.com, rank: 0, local_port_id: { 65559 0 1 }, port issue: { status: failed, details: training timeout }
16 2025:04:02-03:30:40:(4119977) |CCL_WARN| host: a4bf01946f22.jf.intel.com, rank: 1, local_port_id: { 65563 0 1 }, port issue: { status: failed, details: training timeout }
17 2025:04:02-03:30:40:(4119976) |CCL_WARN| host: a4bf01946f22.jf.intel.com, rank: 0, local_port_id: { 65559 0 2 }, port issue: { status: failed, details: training timeout }
18 2025:04:02-03:30:40:(4119977) |CCL_WARN| host: a4bf01946f22.jf.intel.com, rank: 1, local_port_id: { 65563 0 2 }, port issue: { status: failed, details: training timeout }
19 2025:04:02-03:30:40:(4119976) |CCL_WARN| host: a4bf01946f22.jf.intel.com, rank: 0, local_port_id: { 65559 0 3 }, port issue: { status: failed, details: training timeout }
20 2025:04:02-03:30:40:(4119977) |CCL_WARN| host: a4bf01946f22.jf.intel.com, rank: 1, local_port_id: { 65563 0 3 }, port issue: { status: failed, details: training timeout }
21 2025:04:02-03:30:40:(4119976) |CCL_WARN| host: a4bf01946f22.jf.intel.com, rank: 0, local_port_id: { 65559 0 4 }, port issue: { status: failed, details: training timeout }
22 2025:04:02-03:30:40:(4119977) |CCL_WARN| host: a4bf01946f22.jf.intel.com, rank: 1, local_port_id: { 65563 0 4 }, port issue: { status: failed, details: training timeout }
23 2025:04:02-03:30:40:(4119976) |CCL_WARN| host: a4bf01946f22.jf.intel.com, rank: 0, local_port_id: { 65559 0 6 }, port issue: { status: failed, details: training timeout }
24 2025:04:02-03:30:40:(4119977) |CCL_WARN| host: a4bf01946f22.jf.intel.com, rank: 1, local_port_id: { 65563 0 6 }, port issue: { status: failed, details: training timeout }
25 /home/sdp/mambaforge/envs/dist_2.7/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:831: UserWarning: FSDP got the argument device_id xpu on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call torch.xpu.set_device() before FSDP initialization or pass in the explicit device index as the device_id argument.
26 warnings.warn(
27 /home/sdp/mambaforge/envs/dist_2.7/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:831: UserWarning: FSDP got the argument device_id xpu on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call torch.xpu.set_device() before FSDP initialization or pass in the explicit device index as the device_id argument.
28 warnings.warn(
29 /home/sdp/mambaforge/envs/dist_2.7/lib/python3.10/site-packages/torch/testing/comparison.py:290: UserWarning: Converting a tensor with requires_grad=True to a scalar may lead to unexpected behavior.
30 Consider using tensor.detach() first. (Triggered internally at /home/sdp/penghuic/pytorch/aten/src/ATen/native/Scalar.cpp:22.)
31 abs_diff=max_abs_diff.item(),
32 /home/sdp/mambaforge/envs/dist_2.7/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:680: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please u se APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
33 warnings.warn(
34 /home/sdp/mambaforge/envs/dist_2.7/lib/python3.10/site-packages/torch/testing/comparison.py:290: UserWarning: Converting a tensor with requires_grad=True to a scalar may lead to unexpected behavior.
35 Consider using tensor.detach() first. (Triggered internally at /home/sdp/penghuic/pytorch/aten/src/ATen/native/Scalar.cpp:22.)
36 abs_diff=max_abs_diff.item(),
37 /home/sdp/mambaforge/envs/dist_2.7/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:680: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please u se APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
38 warnings.warn(
39 2025:04:02-03:30:40:(4120139) |CCL_ERROR| atl_ofi.cpp:935 prov_ep_handle_cq_err: fi_cq_readerr: err: -265, prov_err: Truncation error(265)
40 2025:04:02-03:30:40:(4120139) |CCL_ERROR| worker.cpp:338 ccl_worker_func: worker 0 caught internal exception: oneCCL: recv_entry.hpp:63 update: EXCEPTION: RECV entry failed. atl_status: FAILURE
41 terminate called after throwing an instance of 'ccl::v1::exception'
42 what(): oneCCL: recv_entry.hpp:63 update: EXCEPTION: RECV entry failed. atl_status: FAILURE
43 SIGABRT(6), PID: 4119976, Thread 4119976:
44 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f02042fd65a in /home/sdp/mambaforge/envs/dist_2.7/lib/python3.10/site-packages/torch/lib/libc10.so)
45 frame #1: + 0x42520 (0x7f023d0e8520 in /lib/x86_64-linux-gnu/libc.so.6)
46 frame #2: ccl_request::is_completed() const + 0xb (0x7f01ff84e70b in /home/sdp/intel/oneapi/2025.0/lib/libccl.so.1)
47 frame #3: ccl_executor::wait(ccl_request const*) + 0x58 (0x7f01ff86dcd8 in /home/sdp/intel/oneapi/2025.0/lib/libccl.so.1)
48 frame #4: + 0x1343b9d (0x7f01ff743b9d in /home/sdp/intel/oneapi/2025.0/lib/libccl.so.1)
49 frame #5: ccl_allgather_impl(void const*, void*, unsigned long, ccl::v1::datatype, ccl_coll_attr const&, ccl_comm*, ccl_stream const*, std::vector<ccl::v1::event, std::allocatorccl::v1::event > const&) + 0x4f (0x7f01ff74 281f in /home/sdp/intel/oneapi/2025.0/lib/libccl.so.1)
50 frame #6: ccl_comm::allgather_impl(void const*, void*, unsigned long, ccl::v1::datatype, std::shared_ptr<ccl_stream> const&, ccl::v1::allgather_attr const&, std::vector<ccl::v1::event, std::allocatorccl::v1::event > cons t&) + 0x81 (0x7f01ff7acc41 in /home/sdp/intel/oneapi/2025.0/lib/libccl.so.1)
51 frame #7: ccl_comm::allgather(void const*, void*, unsigned long, ccl::v1::datatype, std::shared_ptr<ccl_stream> const&, ccl::v1::allgather_attr const&, std::vector<ccl::v1::event, std::allocatorccl::v1::event > const&) + 0x20 (0x7f01ff7c4c30 in /home/sdp/intel/oneapi/2025.0/lib/libccl.so.1)
52 frame #8: ccl::v1::allgather(void const*, void*, unsigned long, ccl::v1::datatype, ccl::v1::communicator const&, ccl::v1::stream const&, ccl::v1::allgather_attr const&, std::vector<ccl::v1::event, std::allocator<ccl::v1::e vent> > const&) + 0x37 (0x7f01ff93d407 in /home/sdp/intel/oneapi/2025.0/lib/libccl.so.1)
53 frame #9: c10d::ProcessGroupXCCL::allgather(std::vector<std::vector<at::Tensor, std::allocatorat::Tensor >, std::allocator<std::vector<at::Tensor, std::allocatorat::Tensor > > >&, std::vector<at::Tensor, std::allocator at::Tensor >&, c10d::AllgatherOptions const&) + 0xd7d (0x7f020b2088ad in /home/sdp/mambaforge/envs/dist_2.7/lib/python3.10/site-packages/torch/lib/libtorch_xpu.so)

Versions

Collecting environment information...
PyTorch version: 2.8.0a0+git124ff16
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.5 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: 14.0.0-1ubuntu1.1
CMake version: version 4.0.0
Libc version: glibc-2.35

Python version: 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0] (64-bit runtime)
Python platform: Linux-5.15.47+prerelease6469.7-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 224
On-line CPU(s) list: 0-223
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Platinum 8480+
CPU family: 6
Model: 143
Thread(s) per core: 2
Core(s) per socket: 56
Socket(s): 2
Stepping: 8
CPU max MHz: 3800.0000
CPU min MHz: 800.0000
BogoMIPS: 4000.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr avx512_fp16 flush_l1d arch_capabilities
Virtualization: VT-x
L1d cache: 5.3 MiB (112 instances)
L1i cache: 3.5 MiB (112 instances)
L2 cache: 224 MiB (112 instances)
L3 cache: 210 MiB (2 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-55,112-167
NUMA node1 CPU(s): 56-111,168-223
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

Versions of relevant libraries:
[pip3] mypy==1.14.0
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.22.4
[pip3] onnx==1.17.0
[pip3] onnxscript==0.2.2
[pip3] optree==0.13.0
[pip3] torch==2.8.0a0+git124ff16
[conda] mkl-include 2025.1.0 pypi_0 pypi
[conda] mkl-static 2025.1.0 pypi_0 pypi
[conda] numpy 1.22.4 pypi_0 pypi
[conda] optree 0.13.0 pypi_0 pypi
[conda] torch 2.8.0a0+git124ff16 pypi_0 pypi
(dist_2.7) sdp@a4bf01946f22:~/daisyden/dist_2.8_2025.0/third_party/torch-xpu-ops/test/xpu$ vi xpu_test_utils.py

[ratnampa]

@zhangxiaoli73 I am able to run above test successfully with DL-essentials kit: 2025.1.0.539, latest xpu-ops commit and pytorch commit: https://github.com/daisyden/pytorch/tree/distributed_2.8 and oneCCL: release/ccl_2025.16-gold on Borealis.

[zhangxiaoli73]

Thanks to @ratnampa confirm. @daisyden Please confirm with @chuanqi129 and @riverliuintel to see if we need to care about issues in basekit 2025.0.

[zhangxiaoli73]

Synced with @daisyden, she will help to verify if those issues also happen in latest release (basekit 2025.1).

[daisyden]

Verified this issue also exists on 2025.1 + ccl master, reproduced 3/4 times on my machine.
Intel(R) oneAPI DPC++/C++ Compiler 2025.1.0 (2025.1.0.20250212)

commit 33e2c582599d9f0b9dc075c2c3080e9dbf6a2b95 (HEAD -> master, origin/master, origin/HEAD)
Merge: 5f7fbdd5 40219443
Author: Ilia Kosarev ilya.kosarev@intel.com
Date: Wed Apr 2 00:34:04 2025 +0200

Merge pull request #3237 from jgaffrey-intel/jgaffrey/fix-provider

scripts: fix provider override cases

xpu-smi topology -m
GPU 0/0 GPU 1/0 CPU Affinity
GPU 0/0 S XL4 0-55,112-167
GPU 1/0 XL4 S 0-55,112-167

sycl-ls
[level_zero:gpu][level_zero:0] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Data Center GPU Max 1100 12.60.7 [1.3.30049+10]
[level_zero:gpu][level_zero:1] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Data Center GPU Max 1100 12.60.7 [1.3.30049+10]
[opencl:gpu][opencl:0] Intel(R) OpenCL Graphics, Intel(R) Data Center GPU Max 1100 OpenCL 3.0 NEO [24.26.30049.10]
[opencl:gpu][opencl:1] Intel(R) OpenCL Graphics, Intel(R) Data Center GPU Max 1100 OpenCL 3.0 NEO [24.26.30049.10]

[zhangxiaoli73]

@ashokei Could you please take a look on this issue?