ROC kernel faulting upon having AMDGPU and CUDA loaded

[luraess]

Executing a ROCKernel fails if both AMDGPU and CUDA are loaded. The order matters, if CUDA is loaded first, then no error occurs.

If using first AMDGPU, then CUDA:

julia> using AMDGPU

julia> import AMDGPU: GPUCompiler

julia> methods(GPUCompiler.runtime_module)
# 2 methods for generic function "runtime_module" from GPUCompiler:
 [1] runtime_module(::GPUCompiler.CompilerJob{GPUCompiler.GCNCompilerTarget, AMDGPU.Compiler.ROCCompilerParams})
     @ AMDGPU.Compiler /scratch/project_465000139/lurass/julia_local/julia_depot/packages/AMDGPU/ia4rb/src/compiler.jl:59
 [2] runtime_module(job::GPUCompiler.CompilerJob)
     @ /scratch/project_465000139/lurass/julia_local/julia_depot/packages/GPUCompiler/07qaN/src/interface.jl:188

julia> nx=ny=11
11

julia> b = AMDGPU.ones(Float64, nx, ny);

julia> using CUDA

julia> methods(GPUCompiler.runtime_module)
# 3 methods for generic function "runtime_module" from GPUCompiler:
 [1] runtime_module(::GPUCompiler.CompilerJob{GPUCompiler.GCNCompilerTarget, AMDGPU.Compiler.ROCCompilerParams})
     @ AMDGPU.Compiler /scratch/project_465000139/lurass/julia_local/julia_depot/packages/AMDGPU/ia4rb/src/compiler.jl:59
 [2] runtime_module(job::GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams})
     @ CUDA /scratch/project_465000139/lurass/julia_local/julia_depot/packages/CUDA/DfvRa/src/compiler/gpucompiler.jl:58
 [3] runtime_module(job::GPUCompiler.CompilerJob)
     @ /scratch/project_465000139/lurass/julia_local/julia_depot/packages/GPUCompiler/07qaN/src/interface.jl:188

julia> c = AMDGPU.ones(Float64, nx, ny, 3)
ERROR: InvalidIRError: compiling kernel #5#6(AMDGPU.ROCKernelContext, ROCDeviceArray{Float64, 3, 1}, Float64) resulted in invalid LLVM IR
Reason: unsupported call to an unknown function (call to __ockl_hsa_signal_cas)
Stacktrace:

Stack-trace here Stacktrace.txt

So it looks like using CUDA changes something in GPUCompiler that now breakes ROCKernel compilation/execution 👀 .

cc @vchuravy

Versions: AMDGPU#master, CUDA v3.12, Julia 1.9.0-DEV.1584

Closing JuliaGPU/AMDGPU.jl#312 in favour of this.

[vchuravy]

Unable to reproduce on:

(gpuc) pkg> st
Status `~/builds/julia/gpuc/Project.toml`
  [21141c5a] AMDGPU v0.4.3
  [052768ef] CUDA v3.12.0
  [61eb1bfa] GPUCompiler v0.16.4

and 1.9.0-DEV.1649 (0c382c245a)

[luraess]

Oh interesting! Let me add an update here after re-testing. Thanks!

[luraess]

@vchuravy on Version 1.9.0-DEV.1510 (fe7f67300a1) 19 days old master, all works fine for me as well, however with more recent version such as today's or the one from 10 days ago, it fails with various GPUCompiler related errors as reported above. Could you maybe try with today's baked master build?

[jpsamaroo]

Reported upstream as JuliaLang/julia#47331

[luraess]

Thanks @jpsamaroo !