Sporadic cudnn/convolution test failures

[maleadt]

For example: https://buildkite.com/julialang/cuda-dot-jl/builds/968#93c70c29-8c56-466d-b31f-c15b5c4a65b6

The failed comparison:

Float32[11.339898 -0.35388422 … -7.7320366 -2.0372493; 0.5171652 6.98606 … 3.625992 0.3606677; … ; -9.337527 -0.4385929 … -0.64571404 3.9338949; 0.101695895 5.5190907 … -4.3475685 4.563487] 
Float32[-8.411543 1.8871636 … 4.951878 2.7781992; -0.23128414 -4.977843 … -2.2405014 -1.0332452; … ; 5.2111654 -5.9086237 … -8.254432 -2.6942174; -6.5529013 -0.50722265 … -9.206887 -9.598008] 
Float32[-1.6625764 8.651685 … 1.7940903 0.66811204; 3.4547272 -0.50751054 … 8.584299 -7.924706; … ; 1.3431497 -0.9462663 … 4.18754 -9.402357; 10.6235895 0.16038725 … -4.0804334 -2.2619858] 
Float32[-3.5070534 -10.729431 … -5.116676 0.07223022; -1.1512243 -3.9527044 … -0.3750069 -0.8288319; … ; -10.928001 -4.0239186 … -3.5830576 5.979162; -15.044505 6.4413824 … -3.9082098 10.073008] 
Float32[2.2557168 3.4730556 … 14.446819 3.9426515; 2.093703 10.0018835 … 0.6170838 3.8267064; … ; -0.7535105 -3.3166182 … 6.767151 -6.547782; 1.9609634 -1.9800158 … 9.211072 9.225448] 
Float32[-10.500388 -10.639041 … -8.234325 -9.571053; -6.0109186 0.68985677 … -3.269639 2.5057364; … ; -0.5645257 -5.4851894 … 3.207436 6.2571173; 3.6976657 0.78052115 … -7.2735963 -5.4585905] 
Float32[-12.282933 -5.8298993 … 6.5901723 -8.0629225; -4.5100265 7.1448364 … -9.869208 -3.397947; … ; -6.022368 1.8941798 … -1.4661267 8.301998; 1.3205339 9.675554 … 7.059723 3.29212] 
Float32[3.5465102 4.6437807 … -3.3740168 -7.524283; 3.4188838 -1.7698078 … 9.001716 4.050767; … ; 1.0820111 4.962797 … 0.8179325 -11.103408; 4.108812 -3.8109338 … -1.9185336 -5.115817] 
Float32[-1.0308669 -3.125521 … 8.702576 -3.4893954; -5.392028 -1.5024531 … -11.045694 5.366046; … ; -3.8732483 -6.105731 … 12.94564 4.690816; 2.3938022 3.942337 … -9.809627 1.3426454] 
Float32[-1.4626034 2.5595913 … -1.9670656 6.0024447; 8.564247 -1.0000148 … -4.654773 -4.868887; … ; -0.2185952 2.8668485 … -17.018162 -6.6957855; -3.7243886 -6.5969763 … -2.436799 -8.443428] 
Float32[-13.363389 -6.2885346 … 0.73612255 -5.259095; 15.318377 -3.9310389 … 7.770155 -1.9279139; … ; -2.285467 -3.0098808 … -5.6914015 5.2188597; 5.895041 -15.136935 … 5.8277507 -12.661768] 
Float32[1.5873765 9.462805 … -3.631779 0.81165874; 0.25209653 -5.1471214 … -6.5175776 -2.815481; … ; 7.669592 0.88133085 … -3.2477484 -0.97609794; -4.4345603 -5.8064327 … 0.5163051 5.663567] 
Float32[7.7663336 -0.51701593 … 9.043473 0.7726512; 4.446763 0.08444798 … 4.876623 1.6694326; … ; -7.0494223 2.9762216 … 0.81327534 9.576384; 4.7767763 -12.290512 … -3.427613 -5.423974] 
Float32[-3.4994054 -6.9185987 … 2.087229 -5.8122; -0.89573777 -2.3741922 … -5.81844 6.4371805; … ; 2.978869 1.5207782 … -2.6192136 3.8913846; -5.6063857 1.0591991 … -3.9058766 4.0126796] 
Float32[-0.35167167 4.223628 … -1.8557506 1.8335615; 1.748302 -7.696812 … -7.374843 3.1636536; … ; 2.4781594 -10.38644 … -3.313611 5.485118; 4.7394867 13.607738 … -3.6534462 0.16832009] 
Float32[1.1288794 -9.963065 … -6.94676 -5.674263; -2.3580742 2.2461686 … 0.32166493 -5.263524; … ; -2.6695936 7.412922 … -0.5336609 6.5138025; -6.144022 -1.0293213 … -3.5078764 -1.2475352]

Float32[11.339903 0.0 … 0.0 0.0; 0.51716113 6.986063 … 3.6259923 0.36066985; … ; 0.0 0.0 … 0.0 3.9338973; 0.10169387 5.5190845 … 0.0 4.563491]
Float32[0.0 1.8871601 … 4.9518867 2.7781963; 0.0 0.0 … 0.0 0.0; … ; 5.211176 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0]
Float32[0.0 8.651692 … 1.7940888 0.6681142; 3.4547334 0.0 … 8.584289 0.0; … ; 1.3431506 0.0 … 4.187529 0.0; 10.623588 0.16038755 … 0.0 0.0]
Float32[0.0 0.0 … 0.0 0.072229505; 0.0 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 5.979165; 0.0 6.4413757 … 0.0 10.073009]
Float32[2.2557414 3.4730494 … 14.446819 3.942652; 2.0936968 10.001884 … 0.6170825 3.8267062; … ; 0.0 0.0 … 6.767151 0.0; 1.9609501 0.0 … 9.211074 9.225449]
Float32[0.0 0.0 … 0.0 0.0; 0.0 0.68985546 … 0.0 2.5057378; … ; 0.0 0.0 … 3.207428 6.257118; 3.6976624 0.78052354 … 0.0 0.0]
Float32[0.0 0.0 … 6.5901685 0.0; 0.0 7.1448345 … 0.0 0.0; … ; 0.0 1.8941854 … 0.0 8.302006; 1.3205338 9.675554 … 7.0597215 3.2921138]
Float32[3.5465317 4.6437883 … 0.0 0.0; 3.418889 0.0 … 9.001716 4.050769; … ; 1.0820035 4.9627867 … 0.8179501 0.0; 4.1088295 0.0 … 0.0 0.0]
Float32[0.0 0.0 … 8.70258 0.0; 0.0 0.0 … 0.0 5.366045; … ; 0.0 0.0 … 12.945637 4.6908035; 2.3938 3.942341 … 0.0 1.3426487]
Float32[0.0 2.559598 … 0.0 6.0024376; 8.564245 0.0 … 0.0 0.0; … ; 0.0 2.8668516 … 0.0 0.0; 0.0 0.0 … 0.0 0.0]
Float32[0.0 0.0 … 0.73613024 0.0; 15.3183775 0.0 … 7.770169 0.0; … ; 0.0 0.0 … 0.0 5.2188725; 5.895043 0.0 … 5.827754 0.0]
Float32[1.587366 9.4628 … 0.0 0.8116602; 0.252097 0.0 … 0.0 0.0; … ; 7.669587 0.88133657 … 0.0 0.0; 0.0 0.0 … 0.5163039 5.6635704]
Float32[7.766321 0.0 … 9.043472 0.7726433; 4.446766 0.08445144 … 4.876629 1.6694307; … ; 0.0 2.9762156 … 0.81328607 9.576385; 4.7767687 0.0 … 0.0 0.0]
Float32[0.0 0.0 … 2.0872164 0.0; 0.0 0.0 … 0.0 6.4371824; … ; 2.97888 1.520792 … 0.0 3.8913813; 0.0 1.059197 … 0.0 4.012678]
Float32[0.0 4.223618 … 0.0 1.8335633; 1.7482939 0.0 … 0.0 3.1636562; … ; 2.478148 0.0 … 0.0 5.4851203; 4.739488 13.607737 … 0.0 0.16831973]
Float32[1.1289243 0.0 … 0.0 0.0; 0.0 2.2461672 … 0.32166946 0.0; … ; 0.0 7.412915 … 0.0 6.51381; 0.0 0.0 … 0.0 0.0]

So negative numbers have/should become zero. A clamping issue?

[denizyuret]

Your observation is correct: the first output (cpu) has negative numbers, the second call (gpu) seems to have applied relu and replaced them with zeros. As usual I cannot replicate on my machine. Which answer is correct depends on what keyword arguments were used to call cudnnConvolutionForward. Any way to find out which keyword argument combination(s) caused the failure? Contrary to our expectations the stack traces do not seem to have this info, or I don't know how to read them. I could not replicate on my machines, should I try the CI machine?

[maleadt]

I have to run it in a loop, too, so I don't think it's limited to CI machines. After half an hour or so I got the first failure:

$ while julia --project=test -L test/setup.jl test/cudnn/convolution.jl; do; done;

blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=1, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0: Test Failed at /home/tim/Julia/pkg/CUDA/test/cudnn/convolution.jl:143
  Expression: ay1 ≈ cudnnConvolutionForward(cw0, cx; bias, activation, mode, padding, stride, dilation, group, mathType, reorderType, alpha) |> Array
Stacktrace:
 [1] macro expansion
   @ ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:143 [inlined]
 [2] macro expansion
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
 [3] (::var"#convtest#6"{var"#convtest#4#7"{CuArray{Float32, 4}, CuArray{Float32, 4}, Array{Float32, 4}, Array{Float32, 4}, DataType}, CuArray{Float32, 4}})(; blendz::Bool, bias::CuArray{Float32, 4}, activation::cudnnActivationMode_t, mode::cudnnConvolutionMode_t, padding::Int64, stride::Int64, dilation::Int64, group::Int64, dataType::Type, mathType::cudnnMathType_t, reorderType::cudnnReorderType_t, alpha::Int64, beta::Int64)
   @ Main ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:98
blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=1, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0: Test Failed at /home/tim/Julia/pkg/CUDA/test/cudnn/convolution.jl:146
  Expression: ay1 ≈ cudnnConvolutionForward(cw0, cx, d; bias, activation, alpha) |> Array
Stacktrace:
 [1] macro expansion
   @ ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:146 [inlined]
 [2] macro expansion
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
 [3] (::var"#convtest#6"{var"#convtest#4#7"{CuArray{Float32, 4}, CuArray{Float32, 4}, Array{Float32, 4}, Array{Float32, 4}, DataType}, CuArray{Float32, 4}})(; blendz::Bool, bias::CuArray{Float32, 4}, activation::cudnnActivationMode_t, mode::cudnnConvolutionMode_t, padding::Int64, stride::Int64, dilation::Int64, group::Int64, dataType::Type, mathType::cudnnMathType_t, reorderType::cudnnReorderType_t, alpha::Int64, beta::Int64)
   @ Main ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:98
blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=1, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0: Test Failed at /home/tim/Julia/pkg/CUDA/test/cudnn/convolution.jl:147
  Expression: ay2 ≈ cudnnConvolutionForward!(cy0, cw0, cx; z = cz0, bias, activation, mode, padding, stride, dilation, group, mathType, reorderType, alpha, beta) |> Array
Stacktrace:
 [1] macro expansion
   @ ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:147 [inlined]
 [2] macro expansion
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
 [3] (::var"#convtest#6"{var"#convtest#4#7"{CuArray{Float32, 4}, CuArray{Float32, 4}, Array{Float32, 4}, Array{Float32, 4}, DataType}, CuArray{Float32, 4}})(; blendz::Bool, bias::CuArray{Float32, 4}, activation::cudnnActivationMode_t, mode::cudnnConvolutionMode_t, padding::Int64, stride::Int64, dilation::Int64, group::Int64, dataType::Type, mathType::cudnnMathType_t, reorderType::cudnnReorderType_t, alpha::Int64, beta::Int64)
   @ Main ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:98
blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=1, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0: Test Failed at /home/tim/Julia/pkg/CUDA/test/cudnn/convolution.jl:150
  Expression: ay2 ≈ cudnnConvolutionForward!(cy1, cw0, cx, d; z = cz1, bias, activation, alpha, beta) |> Array
Stacktrace:
 [1] macro expansion
   @ ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:150 [inlined]
 [2] macro expansion
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
 [3] (::var"#convtest#6"{var"#convtest#4#7"{CuArray{Float32, 4}, CuArray{Float32, 4}, Array{Float32, 4}, Array{Float32, 4}, DataType}, CuArray{Float32, 4}})(; blendz::Bool, bias::CuArray{Float32, 4}, activation::cudnnActivationMode_t, mode::cudnnConvolutionMode_t, padding::Int64, stride::Int64, dilation::Int64, group::Int64, dataType::Type, mathType::cudnnMathType_t, reorderType::cudnnReorderType_t, alpha::Int64, beta::Int64)
   @ Main ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:98
blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH_ALLOW_CONVERSION, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0: Error During Test at /home/tim/Julia/pkg/CUDA/test/cudnn/convolution.jl:143
  Test threw exception
  Expression: ay1 ≈ cudnnConvolutionForward(cw0, cx; bias, activation, mode, padding, stride, dilation, group, mathType, reorderType, alpha) |> Array
  CUDNNError: CUDNN_STATUS_BAD_PARAM (code 3)
  Stacktrace:
    [1] throw_api_error(res::CUDA.CUDNN.cudnnStatus_t)
      @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/cudnn/error.jl:19
    [2] macro expansion
      @ ~/Julia/pkg/CUDA/lib/cudnn/error.jl:30 [inlined]
    [3] cudnnConvolutionBiasActivationForward(handle::Ptr{Nothing}, alpha1::Base.RefValue{Float32}, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, x::CuArray{Float32, 4}, wDesc::CUDA.CUDNN.cudnnFilterDescriptor, w::CuArray{Float32, 4}, convDesc::cudnnConvolutionDescriptor, algo::cudnnConvolutionFwdAlgo_t, workSpace::CuArray{UInt8, 1}, workSpaceSizeInBytes::Int64, alpha2::Base.RefValue{Float32}, zDesc::CUDA.CUDNN.cudnnTensorDescriptor, z::CuArray{Float32, 4}, biasDesc::CUDA.CUDNN.cudnnTensorDescriptor, bias::CuArray{Float32, 4}, activationDesc::CUDA.CUDNN.cudnnActivationDescriptor, yDesc::CUDA.CUDNN.cudnnTensorDescriptor, y::CuArray{Float32, 4})
      @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/utils/call.jl:26
    [4] macro expansion
      @ ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:117 [inlined]
    [5] macro expansion
      @ ~/Julia/pkg/CUDA/lib/utils/call.jl:144 [inlined]
    [6] cudnnConvolutionForwardAD(w::CuArray{Float32, 4}, x::CuArray{Float32, 4}, bias::CuArray{Float32, 4}, z::Nothing; y::CuArray{Float32, 4}, activation::cudnnActivationMode_t, convDesc::cudnnConvolutionDescriptor, wDesc::CUDA.CUDNN.cudnnFilterDescriptor, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, yDesc::CUDA.CUDNN.cudnnTensorDescriptor, zDesc::Nothing, biasDesc::CUDA.CUDNN.cudnnTensorDescriptor, alpha::Base.RefValue{Float32}, beta::Base.RefValue{Float32}, dw::Base.RefValue{Any}, dx::Base.RefValue{Any}, dz::Base.RefValue{Any}, dbias::Base.RefValue{Any}, dready::Base.RefValue{Bool})
      @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:103
    [7] cudnnConvolutionForwardWithDefaults(w::CuArray{Float32, 4}, x::CuArray{Float32, 4}; padding::Int64, stride::Int64, dilation::Int64, mode::cudnnConvolutionMode_t, mathType::cudnnMathType_t, reorderType::cudnnReorderType_t, group::Int64, format::cudnnTensorFormat_t, convDesc::cudnnConvolutionDescriptor, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, wDesc::CUDA.CUDNN.cudnnFilterDescriptor, y::CuArray{Float32, 4}, yDesc::CUDA.CUDNN.cudnnTensorDescriptor, alpha::Int64, beta::Int64, bias::CuArray{Float32, 4}, z::Nothing, biasDesc::CUDA.CUDNN.cudnnTensorDescriptor, zDesc::Nothing, activation::cudnnActivationMode_t, dw::Base.RefValue{Any}, dx::Base.RefValue{Any}, dz::Base.RefValue{Any}, dbias::Base.RefValue{Any})
      @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:96
    [8] #cudnnConvolutionForward#104
      @ ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:50 [inlined]
    [9] macro expansion
      @ ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:143 [inlined]
   [10] macro expansion
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [11] (::var"#convtest#6"{var"#convtest#4#7"{CuArray{Float32, 4}, CuArray{Float32, 4}, Array{Float32, 4}, Array{Float32, 4}, DataType}, CuArray{Float32, 4}})(; blendz::Bool, bias::CuArray{Float32, 4}, activation::cudnnActivationMode_t, mode::cudnnConvolutionMode_t, padding::Int64, stride::Int64, dilation::Int64, group::Int64, dataType::Type, mathType::cudnnMathType_t, reorderType::cudnnReorderType_t, alpha::Int64, beta::Int64)
      @ Main ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:98
blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH_ALLOW_CONVERSION, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0: Error During Test at /home/tim/Julia/pkg/CUDA/test/cudnn/convolution.jl:146
  Test threw exception
  Expression: ay1 ≈ cudnnConvolutionForward(cw0, cx, d; bias, activation, alpha) |> Array
  CUDNNError: CUDNN_STATUS_BAD_PARAM (code 3)
  Stacktrace:
    [1] throw_api_error(res::CUDA.CUDNN.cudnnStatus_t)
      @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/cudnn/error.jl:19
    [2] macro expansion
      @ ~/Julia/pkg/CUDA/lib/cudnn/error.jl:30 [inlined]
    [3] cudnnConvolutionBiasActivationForward(handle::Ptr{Nothing}, alpha1::Base.RefValue{Float32}, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, x::CuArray{Float32, 4}, wDesc::CUDA.CUDNN.cudnnFilterDescriptor, w::CuArray{Float32, 4}, convDesc::cudnnConvolutionDescriptor, algo::cudnnConvolutionFwdAlgo_t, workSpace::CuArray{UInt8, 1}, workSpaceSizeInBytes::Int64, alpha2::Base.RefValue{Float32}, zDesc::CUDA.CUDNN.cudnnTensorDescriptor, z::CuArray{Float32, 4}, biasDesc::CUDA.CUDNN.cudnnTensorDescriptor, bias::CuArray{Float32, 4}, activationDesc::CUDA.CUDNN.cudnnActivationDescriptor, yDesc::CUDA.CUDNN.cudnnTensorDescriptor, y::CuArray{Float32, 4})
      @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/utils/call.jl:26
    [4] macro expansion
      @ ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:117 [inlined]
    [5] macro expansion
      @ ~/Julia/pkg/CUDA/lib/utils/call.jl:144 [inlined]
    [6] cudnnConvolutionForwardAD(w::CuArray{Float32, 4}, x::CuArray{Float32, 4}, bias::CuArray{Float32, 4}, z::Nothing; y::CuArray{Float32, 4}, activation::cudnnActivationMode_t, convDesc::cudnnConvolutionDescriptor, wDesc::CUDA.CUDNN.cudnnFilterDescriptor, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, yDesc::CUDA.CUDNN.cudnnTensorDescriptor, zDesc::Nothing, biasDesc::CUDA.CUDNN.cudnnTensorDescriptor, alpha::Base.RefValue{Float32}, beta::Base.RefValue{Float32}, dw::Base.RefValue{Any}, dx::Base.RefValue{Any}, dz::Base.RefValue{Any}, dbias::Base.RefValue{Any}, dready::Base.RefValue{Bool})
      @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:103
    [7] cudnnConvolutionForwardWithDefaults(w::CuArray{Float32, 4}, x::CuArray{Float32, 4}; padding::Int64, stride::Int64, dilation::Int64, mode::cudnnConvolutionMode_t, mathType::cudnnMathType_t, reorderType::cudnnReorderType_t, group::Int64, format::cudnnTensorFormat_t, convDesc::cudnnConvolutionDescriptor, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, wDesc::CUDA.CUDNN.cudnnFilterDescriptor, y::CuArray{Float32, 4}, yDesc::CUDA.CUDNN.cudnnTensorDescriptor, alpha::Int64, beta::Int64, bias::CuArray{Float32, 4}, z::Nothing, biasDesc::CUDA.CUDNN.cudnnTensorDescriptor, zDesc::Nothing, activation::cudnnActivationMode_t, dw::Base.RefValue{Any}, dx::Base.RefValue{Any}, dz::Base.RefValue{Any}, dbias::Base.RefValue{Any})
      @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:96
    [8] #cudnnConvolutionForward#106
      @ ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:52 [inlined]
    [9] macro expansion
      @ ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:146 [inlined]
   [10] macro expansion
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [11] (::var"#convtest#6"{var"#convtest#4#7"{CuArray{Float32, 4}, CuArray{Float32, 4}, Array{Float32, 4}, Array{Float32, 4}, DataType}, CuArray{Float32, 4}})(; blendz::Bool, bias::CuArray{Float32, 4}, activation::cudnnActivationMode_t, mode::cudnnConvolutionMode_t, padding::Int64, stride::Int64, dilation::Int64, group::Int64, dataType::Type, mathType::cudnnMathType_t, reorderType::cudnnReorderType_t, alpha::Int64, beta::Int64)
      @ Main ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:98
blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH_ALLOW_CONVERSION, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0: Error During Test at /home/tim/Julia/pkg/CUDA/test/cudnn/convolution.jl:147
  Test threw exception
  Expression: ay2 ≈ cudnnConvolutionForward!(cy0, cw0, cx; z = cz0, bias, activation, mode, padding, stride, dilation, group, mathType, reorderType, alpha, beta) |> Array
  CUDNNError: CUDNN_STATUS_BAD_PARAM (code 3)
  Stacktrace:
    [1] throw_api_error(res::CUDA.CUDNN.cudnnStatus_t)
      @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/cudnn/error.jl:19
    [2] macro expansion
      @ ~/Julia/pkg/CUDA/lib/cudnn/error.jl:30 [inlined]
    [3] cudnnConvolutionBiasActivationForward(handle::Ptr{Nothing}, alpha1::Base.RefValue{Float32}, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, x::CuArray{Float32, 4}, wDesc::CUDA.CUDNN.cudnnFilterDescriptor, w::CuArray{Float32, 4}, convDesc::cudnnConvolutionDescriptor, algo::cudnnConvolutionFwdAlgo_t, workSpace::CuArray{UInt8, 1}, workSpaceSizeInBytes::Int64, alpha2::Base.RefValue{Float32}, zDesc::CUDA.CUDNN.cudnnTensorDescriptor, z::CuArray{Float32, 4}, biasDesc::CUDA.CUDNN.cudnnTensorDescriptor, bias::CuArray{Float32, 4}, activationDesc::CUDA.CUDNN.cudnnActivationDescriptor, yDesc::CUDA.CUDNN.cudnnTensorDescriptor, y::CuArray{Float32, 4})
      @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/utils/call.jl:26
    [4] macro expansion
      @ ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:117 [inlined]
    [5] macro expansion
      @ ~/Julia/pkg/CUDA/lib/utils/call.jl:144 [inlined]
    [6] cudnnConvolutionForwardAD(w::CuArray{Float32, 4}, x::CuArray{Float32, 4}, bias::CuArray{Float32, 4}, z::CuArray{Float32, 4}; y::CuArray{Float32, 4}, activation::cudnnActivationMode_t, convDesc::cudnnConvolutionDescriptor, wDesc::CUDA.CUDNN.cudnnFilterDescriptor, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, yDesc::CUDA.CUDNN.cudnnTensorDescriptor, zDesc::CUDA.CUDNN.cudnnTensorDescriptor, biasDesc::CUDA.CUDNN.cudnnTensorDescriptor, alpha::Base.RefValue{Float32}, beta::Base.RefValue{Float32}, dw::Base.RefValue{Any}, dx::Base.RefValue{Any}, dz::Base.RefValue{Any}, dbias::Base.RefValue{Any}, dready::Base.RefValue{Bool})
      @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:103
    [7] #cudnnConvolutionForwardWithDefaults#108
      @ ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:96 [inlined]
    [8] #cudnnConvolutionForward!#105
      @ ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:51 [inlined]
    [9] macro expansion
      @ ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:147 [inlined]
   [10] macro expansion
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [11] (::var"#convtest#6"{var"#convtest#4#7"{CuArray{Float32, 4}, CuArray{Float32, 4}, Array{Float32, 4}, Array{Float32, 4}, DataType}, CuArray{Float32, 4}})(; blendz::Bool, bias::CuArray{Float32, 4}, activation::cudnnActivationMode_t, mode::cudnnConvolutionMode_t, padding::Int64, stride::Int64, dilation::Int64, group::Int64, dataType::Type, mathType::cudnnMathType_t, reorderType::cudnnReorderType_t, alpha::Int64, beta::Int64)
      @ Main ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:98
blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH_ALLOW_CONVERSION, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0: Error During Test at /home/tim/Julia/pkg/CUDA/test/cudnn/convolution.jl:150
  Test threw exception
  Expression: ay2 ≈ cudnnConvolutionForward!(cy1, cw0, cx, d; z = cz1, bias, activation, alpha, beta) |> Array
  CUDNNError: CUDNN_STATUS_BAD_PARAM (code 3)
  Stacktrace:
    [1] throw_api_error(res::CUDA.CUDNN.cudnnStatus_t)
      @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/cudnn/error.jl:19
    [2] macro expansion
      @ ~/Julia/pkg/CUDA/lib/cudnn/error.jl:30 [inlined]
    [3] cudnnConvolutionBiasActivationForward(handle::Ptr{Nothing}, alpha1::Base.RefValue{Float32}, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, x::CuArray{Float32, 4}, wDesc::CUDA.CUDNN.cudnnFilterDescriptor, w::CuArray{Float32, 4}, convDesc::cudnnConvolutionDescriptor, algo::cudnnConvolutionFwdAlgo_t, workSpace::CuArray{UInt8, 1}, workSpaceSizeInBytes::Int64, alpha2::Base.RefValue{Float32}, zDesc::CUDA.CUDNN.cudnnTensorDescriptor, z::CuArray{Float32, 4}, biasDesc::CUDA.CUDNN.cudnnTensorDescriptor, bias::CuArray{Float32, 4}, activationDesc::CUDA.CUDNN.cudnnActivationDescriptor, yDesc::CUDA.CUDNN.cudnnTensorDescriptor, y::CuArray{Float32, 4})
      @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/utils/call.jl:26
    [4] macro expansion
      @ ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:117 [inlined]
    [5] macro expansion
      @ ~/Julia/pkg/CUDA/lib/utils/call.jl:144 [inlined]
    [6] cudnnConvolutionForwardAD(w::CuArray{Float32, 4}, x::CuArray{Float32, 4}, bias::CuArray{Float32, 4}, z::CuArray{Float32, 4}; y::CuArray{Float32, 4}, activation::cudnnActivationMode_t, convDesc::cudnnConvolutionDescriptor, wDesc::CUDA.CUDNN.cudnnFilterDescriptor, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, yDesc::CUDA.CUDNN.cudnnTensorDescriptor, zDesc::CUDA.CUDNN.cudnnTensorDescriptor, biasDesc::CUDA.CUDNN.cudnnTensorDescriptor, alpha::Base.RefValue{Float32}, beta::Base.RefValue{Float32}, dw::Base.RefValue{Any}, dx::Base.RefValue{Any}, dz::Base.RefValue{Any}, dbias::Base.RefValue{Any}, dready::Base.RefValue{Bool})
      @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:103
    [7] #cudnnConvolutionForwardWithDefaults#108
      @ ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:96 [inlined]
    [8] #cudnnConvolutionForward!#107
      @ ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:53 [inlined]
    [9] macro expansion
      @ ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:150 [inlined]
   [10] macro expansion
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [11] (::var"#convtest#6"{var"#convtest#4#7"{CuArray{Float32, 4}, CuArray{Float32, 4}, Array{Float32, 4}, Array{Float32, 4}, DataType}, CuArray{Float32, 4}})(; blendz::Bool, bias::CuArray{Float32, 4}, activation::cudnnActivationMode_t, mode::cudnnConvolutionMode_t, padding::Int64, stride::Int64, dilation::Int64, group::Int64, dataType::Type, mathType::cudnnMathType_t, reorderType::cudnnReorderType_t, alpha::Int64, beta::Int64)
      @ Main ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:98
Test Summary:                                                                                                                                                                                                                                                          | Pass  Fail  Error  Total
cudnn/convolution                                                                                                                                                                                                                                                      |  102     4      4    110
  blendz=false, bias=Nothing, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0                              |    4                   4
  blendz=false, bias=Nothing, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=1, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0                              |    4                   4
  blendz=false, bias=Nothing, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=2, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0                              |    4                   4
  blendz=false, bias=Nothing, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=2, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0                              |    4                   4
  blendz=false, bias=Nothing, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=2, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0                              |    4                   4
  blendz=false, bias=Nothing, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_DEFAULT_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0                                |    4                   4
  blendz=false, bias=Nothing, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0                              |    4                   4
  blendz=false, bias=Nothing, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH_ALLOW_CONVERSION, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0             |    4                   4
  blendz=false, bias=Nothing, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_NO_REORDER, alpha=1, beta=0                                   |    4                   4
  blendz=false, bias=Nothing, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=2, beta=0                              |    4                   4
  blendz=false, bias=Nothing, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=2                              |    4                   4
  blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0                  |    4                   4
  blendz=true, bias=Nothing, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0                               |    4                   4
  blendz=false, bias=Nothing, activation=CUDNN_ACTIVATION_RELU, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0                                  |    4                   4
  blendz=true, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0                   |    4                   4
  blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_RELU, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0                      |    4                   4
  blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=1, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0                  |          4             4
  blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=2, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0                  |    4                   4
  blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=2, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0                  |    4                   4
  blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=2, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0                  |    4                   4
  blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_DEFAULT_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0                    |    4                   4
  blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0                  |    4                   4
  blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH_ALLOW_CONVERSION, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0 |                 4      4
  blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_NO_REORDER, alpha=1, beta=0                       |    4                   4
  blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=2, beta=0                  |    4                   4
  blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=2                  |    4                   4
  blendz=true, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=2                   |    4                   4

Notice the 4 STATUS_BAD_PARAM errors too!

I used this for logging the kwargs:

diff --git a/test/cudnn/convolution.jl b/test/cudnn/convolution.jl
index 43976fd6..2c2501e6 100644
--- a/test/cudnn/convolution.jl
+++ b/test/cudnn/convolution.jl
@@ -94,6 +94,7 @@ using CUDA.CUDNN:
                       reorderType = CUDNN_DEFAULT_REORDER,
                       alpha = 1,
                       beta = 0)
+        @testset "blendz=$blendz, bias=$(typeof(bias)), activation=$activation, mode=$mode, padding=$padding, stride=$stride, dilation=$dilation, group=$group, dataType=$dataType, mathType=$mathType, reorderType=$reorderType, alpha=$alpha, beta=$beta" begin
         if group == 1
             cdims = NNlib.DenseConvDims(ax, aw; stride, padding, dilation, flipkernel = (mode === CUDNN_CROSS_CORRELATION))
             ay = NNlib.conv(ax, aw, cdims)
@@ -148,6 +149,7 @@ using CUDA.CUDNN:
                                              reorderType, alpha, beta) |> Array
         @test ay2 ≈ cudnnConvolutionForward!(cy1, cw0, cx, d; z=cz1, bias, activation,
                                              alpha, beta) |> Array
+        end
     end
 
     # These call cudnnConvolutionForward

[denizyuret]

Hi Tim, I am running the while loop on my machine right now, 60 minutes in and nothing, will let you know if I get something. Browsing through the error messages: convtest() calls cudnnConvolutionForward with default options. At each test I change one of the keyword arguments to a non-default value to make sure things work as intended (I change one at a time because trying every combination would make the tests unreasonably long). error#1,2,3,4: come from convtest(padding=1). Looks like the target array is filled with garbage, so the Array conversion possibly happened before cudnnConvolutionForward was complete. My only hypothesis is unreliable gpu hardware ;) You may have some smarter explanations. (Did we ever stress test the old version of lib/cudnn? padding=1 is a very standard call that we have always used) error#5,6,7,8: come from convtest(mathType=CUDNN_TENSOR_OP_MATH_ALLOW_CONVERSION) and gives CUDNN_STATUS_BAD_PARAM. The only place the mathType is used in the code is with cudnnSetConvolutionMathType <https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cudnnSetConvolutionMathType>. This mathType is either legal for Float32 or not. If it is, it should always error, if not it should never. I don't understand why this can cause sporadic errors. We can just comment out this test, or remove this keyword argument and always use the default CUDNN.math_mode(). best, deniz

…

On Mon, Feb 22, 2021 at 1:41 PM Tim Besard ***@***.***> wrote: I have to run it in a loop, too, so I don't think it's limited to CI machines. After half an hour or so I got the first failure: $ while julia --project=test -L test/setup.jl test/cudnn/convolution.jl; do; done; blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=1, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0: Test Failed at /home/tim/Julia/pkg/CUDA/test/cudnn/convolution.jl:143 Expression: ay1 ≈ cudnnConvolutionForward(cw0, cx; bias, activation, mode, padding, stride, dilation, group, mathType, reorderType, alpha) |> Array Evaluated: Float32[-7.625949 -6.12281 … -5.9059563 -4.710522; 0.5002041 -0.22734165 … -11.341359 -0.029361725; … ; -5.955898 -0.8519391 … 9.367512 3.6044025; -1.369788 -4.7792273 … -1.0492474 -5.542263] Float32[0.93721855 -1.0799328 … -4.7547307 -1.3063779; 2.664013 3.7837796 … 15.958762 -1.1090627; … ; 0.11813876 0.18879864 … -1.8676813 -3.0879667; 0.8384547 4.255574 … 1.7350059 -4.358764] Float32[1.0326179 -9.130896 … -1.3046414 -1.638468; -0.403609 -3.2207842 … 6.939209 -1.9991591; … ; 1.3053217 7.436456 … -8.234082 -7.437218; 1.4383951 -0.64058656 … -6.64644 2.529673] Float32[2.1496897 -0.2559681 … -2.7595649 1.112249; 0.8892468 -0.2949109 … -2.3872485 -1.0089985; … ; 8.299812 7.852995 … 3.1898193 -1.2525343; 4.876601 4.505626 … -2.9399366 -2.8759317] Float32[-2.7328777 -7.9601955 … -7.8990173 1.8676727; -1.3844309 -7.88463 … 0.6634226 -9.988422; … ; -2.9437788 -3.2956274 … -8.971467 6.3641863; -3.4225056 -3.73323 … -0.21863937 1.5250564] Float32[-3.3950315 8.367449 … -4.26904 2.1378958; -3.221842 9.269204 … 5.919178 -8.4661875; … ; -2.908155 -4.207021 … 4.4138117 -1.76969; 0.3866739 6.6738005 … -2.8421426 -4.412014] Float32[4.9955983 8.106892 … 2.7535727 -14.277321; 2.3030026 3.396777 … -13.603948 4.5709653; … ; 2.3793406 -10.423798 … -0.92979616 7.863477; 6.555591 -1.0130867 … -2.5044403 0.09186761] Float32[1.9763883 0.7442095 … -2.7087684 2.416678; 2.5668755 -2.578361 … 4.1207623 -9.057405; … ; -3.7044673 2.5795178 … 8.583115 3.2888846; 1.4762158 -2.704668 … -0.1852051 -0.94584596] Float32[-9.431355 -5.288068 … -2.950151 2.0531697; -6.699032 4.6442776 … 1.7998717 -1.6313852; … ; -10.450587 -10.025318 … -15.630849 -3.5284545; -4.011667 -8.436462 … -7.4438815 -0.9473691] Float32[-3.8612556 8.199958 … 0.776072 0.7085123; -6.236965 0.6840241 … 0.012681454 2.3597157; … ; -2.7117143 -4.8041277 … -6.732696 6.679207; 1.4808843 4.483176 … 4.4658012 -0.3740994] Float32[-1.4137484 -1.9556055 … 2.8444076 -5.4156904; 0.91588575 -0.56438535 … -4.6698 -3.8603404; … ; -1.1119218 -0.7578048 … -5.5975842 -1.3475385; 0.93386126 -4.766769 … -3.479992 3.774753] Float32[4.8634553 0.3752916 … 2.1437469 1.2139807; 0.52895725 0.697806 … 4.3122315 -1.3506848; … ; 4.6959987 3.9705896 … 4.081906 -0.7670711; -0.97394645 0.110408664 … 0.3872782 6.2378907] Float32[-3.463529 11.649901 … 2.8375607 -0.6578778; -2.9366102 2.9042091 … -4.8822327 -1.283011; … ; -4.0112977 -0.26579285 … -7.6523533 2.3017254; -5.8749695 -13.766709 … 5.2382183 4.569278] Float32[1.980134 -0.5140625 … 2.485562 0.44694278; -6.756843 1.4986404 … 5.4092026 -0.13282302; … ; -7.779603 12.7077875 … -1.4024372 -2.7751215; 1.4063643 2.7959778 … 0.9731282 -2.4291978] Float32[0.49171367 5.022867 … -15.067985 2.414992; -2.4574265 -9.237441 … -4.8011904 13.332268; … ; -6.1316247 -8.423361 … -3.69942 4.917993; 0.96881276 2.3289487 … 1.381468 -5.249476] Float32[3.049631 7.306164 … 0.13163137 -2.8691373; -1.7166475 5.0266857 … 3.5797625 6.234812; … ; -4.7470737 -2.4605598 … 2.95617 2.1210446; 2.0376394 8.286376 … -8.101818 -2.9176269] ≈ Float32[0.0 0.0 … 0.0 0.0; 0.500196 0.0 … 0.0 0.0; … ; 0.0 0.0 … 9.367514 3.6044035; 0.0 0.0 … 0.0 0.0] Float32[0.9372158 0.0 … 0.0 0.0; 2.6640186 3.7837784 … 15.958762 0.0; … ; 0.11814305 0.18879744 … 0.0 0.0; 0.83845544 4.255575 … 1.7350042 0.0] Float32[1.0326269 0.0 … 0.0 0.0; 0.0 0.0 … 6.9392095 0.0; … ; 1.3053309 7.436457 … 0.0 0.0; 1.4383941 0.0 … 0.0 2.5296755] Float32[2.1497054 0.0 … 0.0 1.1122471; 0.8892542 0.0 … 0.0 0.0; … ; 8.299812 7.852998 … 3.1898184 0.0; 4.876604 4.505626 … 0.0 0.0] Float32[0.0 0.0 … 0.0 1.8676558; 0.0 0.0 … 0.6634207 0.0; … ; 0.0 0.0 … 0.0 6.3641863; 0.0 0.0 … 0.0 1.5250559] Float32[0.0 8.367454 … 0.0 2.1378973; 0.0 9.269204 … 5.91918 0.0; … ; 0.0 0.0 … 4.413811 0.0; 0.38668057 6.6737986 … 0.0 0.0] Float32[4.995608 8.106895 … 2.75357 0.0; 2.30298 3.3967743 … 0.0 4.5709653; … ; 2.3793416 0.0 … 0.0 7.8634763; 6.55559 0.0 … 0.0 0.091865465] Float32[1.976395 0.744218 … 0.0 2.4166713; 2.5668888 0.0 … 4.12076 0.0; … ; 0.0 2.5795188 … 8.583115 3.288885; 1.4762126 0.0 … 0.0 0.0] Float32[0.0 0.0 … 0.0 2.053174; 0.0 4.644282 … 1.7998714 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0] Float32[0.0 8.199955 … 0.7760706 0.7085104; 0.0 0.6840248 … 0.012680501 2.3597167; … ; 0.0 0.0 … 0.0 6.6792064; 1.4808874 4.4831758 … 4.4658027 0.0] Float32[0.0 0.0 … 2.8444114 0.0; 0.9158835 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.9338594 0.0 … 0.0 3.7747526] Float32[4.863457 0.37529337 … 2.1437478 1.2139851; 0.5289551 0.6978088 … 4.3122315 0.0; … ; 4.696 3.9705887 … 4.081905 0.0; 0.0 0.11040878 … 0.3872763 6.2378902] Float32[0.0 11.649898 … 2.8375564 0.0; 0.0 2.904213 … 0.0 0.0; … ; 0.0 0.0 … 0.0 2.3017268; 0.0 0.0 … 5.2382174 4.5692763] Float32[1.9801435 0.0 … 2.4855614 0.4469435; 0.0 1.4986401 … 5.4092045 0.0; … ; 0.0 12.707787 … 0.0 0.0; 1.4063678 2.7959795 … 0.9731283 0.0] Float32[0.49171594 5.0228662 … 0.0 2.414989; 0.0 0.0 … 0.0 13.332266; … ; 0.0 0.0 … 0.0 4.917992; 0.96881 2.328949 … 1.3814683 0.0] Float32[3.0496302 7.3061557 … 0.13163531 0.0; 0.0 5.0266824 … 3.5797634 6.234811; … ; 0.0 0.0 … 2.9561696 2.1210437; 2.0376396 8.286372 … 0.0 0.0] Stacktrace: [1] macro expansion @ ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:143 [inlined] [2] macro expansion @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined] [3] (::var"#convtest#6"{var"#convtest#4#7"{CuArray{Float32, 4}, CuArray{Float32, 4}, Array{Float32, 4}, Array{Float32, 4}, DataType}, CuArray{Float32, 4}})(; blendz::Bool, bias::CuArray{Float32, 4}, activation::cudnnActivationMode_t, mode::cudnnConvolutionMode_t, padding::Int64, stride::Int64, dilation::Int64, group::Int64, dataType::Type, mathType::cudnnMathType_t, reorderType::cudnnReorderType_t, alpha::Int64, beta::Int64) @ Main ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:98 blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=1, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0: Test Failed at /home/tim/Julia/pkg/CUDA/test/cudnn/convolution.jl:146 Expression: ay1 ≈ cudnnConvolutionForward(cw0, cx, d; bias, activation, alpha) |> Array Evaluated: Float32[-7.625949 -6.12281 … -5.9059563 -4.710522; 0.5002041 -0.22734165 … -11.341359 -0.029361725; … ; -5.955898 -0.8519391 … 9.367512 3.6044025; -1.369788 -4.7792273 … -1.0492474 -5.542263] Float32[0.93721855 -1.0799328 … -4.7547307 -1.3063779; 2.664013 3.7837796 … 15.958762 -1.1090627; … ; 0.11813876 0.18879864 … -1.8676813 -3.0879667; 0.8384547 4.255574 … 1.7350059 -4.358764] Float32[1.0326179 -9.130896 … -1.3046414 -1.638468; -0.403609 -3.2207842 … 6.939209 -1.9991591; … ; 1.3053217 7.436456 … -8.234082 -7.437218; 1.4383951 -0.64058656 … -6.64644 2.529673] Float32[2.1496897 -0.2559681 … -2.7595649 1.112249; 0.8892468 -0.2949109 … -2.3872485 -1.0089985; … ; 8.299812 7.852995 … 3.1898193 -1.2525343; 4.876601 4.505626 … -2.9399366 -2.8759317] Float32[-2.7328777 -7.9601955 … -7.8990173 1.8676727; -1.3844309 -7.88463 … 0.6634226 -9.988422; … ; -2.9437788 -3.2956274 … -8.971467 6.3641863; -3.4225056 -3.73323 … -0.21863937 1.5250564] Float32[-3.3950315 8.367449 … -4.26904 2.1378958; -3.221842 9.269204 … 5.919178 -8.4661875; … ; -2.908155 -4.207021 … 4.4138117 -1.76969; 0.3866739 6.6738005 … -2.8421426 -4.412014] Float32[4.9955983 8.106892 … 2.7535727 -14.277321; 2.3030026 3.396777 … -13.603948 4.5709653; … ; 2.3793406 -10.423798 … -0.92979616 7.863477; 6.555591 -1.0130867 … -2.5044403 0.09186761] Float32[1.9763883 0.7442095 … -2.7087684 2.416678; 2.5668755 -2.578361 … 4.1207623 -9.057405; … ; -3.7044673 2.5795178 … 8.583115 3.2888846; 1.4762158 -2.704668 … -0.1852051 -0.94584596] Float32[-9.431355 -5.288068 … -2.950151 2.0531697; -6.699032 4.6442776 … 1.7998717 -1.6313852; … ; -10.450587 -10.025318 … -15.630849 -3.5284545; -4.011667 -8.436462 … -7.4438815 -0.9473691] Float32[-3.8612556 8.199958 … 0.776072 0.7085123; -6.236965 0.6840241 … 0.012681454 2.3597157; … ; -2.7117143 -4.8041277 … -6.732696 6.679207; 1.4808843 4.483176 … 4.4658012 -0.3740994] Float32[-1.4137484 -1.9556055 … 2.8444076 -5.4156904; 0.91588575 -0.56438535 … -4.6698 -3.8603404; … ; -1.1119218 -0.7578048 … -5.5975842 -1.3475385; 0.93386126 -4.766769 … -3.479992 3.774753] Float32[4.8634553 0.3752916 … 2.1437469 1.2139807; 0.52895725 0.697806 … 4.3122315 -1.3506848; … ; 4.6959987 3.9705896 … 4.081906 -0.7670711; -0.97394645 0.110408664 … 0.3872782 6.2378907] Float32[-3.463529 11.649901 … 2.8375607 -0.6578778; -2.9366102 2.9042091 … -4.8822327 -1.283011; … ; -4.0112977 -0.26579285 … -7.6523533 2.3017254; -5.8749695 -13.766709 … 5.2382183 4.569278] Float32[1.980134 -0.5140625 … 2.485562 0.44694278; -6.756843 1.4986404 … 5.4092026 -0.13282302; … ; -7.779603 12.7077875 … -1.4024372 -2.7751215; 1.4063643 2.7959778 … 0.9731282 -2.4291978] Float32[0.49171367 5.022867 … -15.067985 2.414992; -2.4574265 -9.237441 … -4.8011904 13.332268; … ; -6.1316247 -8.423361 … -3.69942 4.917993; 0.96881276 2.3289487 … 1.381468 -5.249476] Float32[3.049631 7.306164 … 0.13163137 -2.8691373; -1.7166475 5.0266857 … 3.5797625 6.234812; … ; -4.7470737 -2.4605598 … 2.95617 2.1210446; 2.0376394 8.286376 … -8.101818 -2.9176269] ≈ Float32[0.0 0.0 … 0.0 0.0; 0.500196 0.0 … 0.0 0.0; … ; 0.0 0.0 … 9.367514 3.6044035; 0.0 0.0 … 0.0 0.0] Float32[0.9372158 0.0 … 0.0 0.0; 2.6640186 3.7837784 … 15.958762 0.0; … ; 0.11814305 0.18879744 … 0.0 0.0; 0.83845544 4.255575 … 1.7350042 0.0] Float32[1.0326269 0.0 … 0.0 0.0; 0.0 0.0 … 6.9392095 0.0; … ; 1.3053309 7.436457 … 0.0 0.0; 1.4383941 0.0 … 0.0 2.5296755] Float32[2.1497054 0.0 … 0.0 1.1122471; 0.8892542 0.0 … 0.0 0.0; … ; 8.299812 7.852998 … 3.1898184 0.0; 4.876604 4.505626 … 0.0 0.0] Float32[0.0 0.0 … 0.0 1.8676558; 0.0 0.0 … 0.6634207 0.0; … ; 0.0 0.0 … 0.0 6.3641863; 0.0 0.0 … 0.0 1.5250559] Float32[0.0 8.367454 … 0.0 2.1378973; 0.0 9.269204 … 5.91918 0.0; … ; 0.0 0.0 … 4.413811 0.0; 0.38668057 6.6737986 … 0.0 0.0] Float32[4.995608 8.106895 … 2.75357 0.0; 2.30298 3.3967743 … 0.0 4.5709653; … ; 2.3793416 0.0 … 0.0 7.8634763; 6.55559 0.0 … 0.0 0.091865465] Float32[1.976395 0.744218 … 0.0 2.4166713; 2.5668888 0.0 … 4.12076 0.0; … ; 0.0 2.5795188 … 8.583115 3.288885; 1.4762126 0.0 … 0.0 0.0] Float32[0.0 0.0 … 0.0 2.053174; 0.0 4.644282 … 1.7998714 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0] Float32[0.0 8.199955 … 0.7760706 0.7085104; 0.0 0.6840248 … 0.012680501 2.3597167; … ; 0.0 0.0 … 0.0 6.6792064; 1.4808874 4.4831758 … 4.4658027 0.0] Float32[0.0 0.0 … 2.8444114 0.0; 0.9158835 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.9338594 0.0 … 0.0 3.7747526] Float32[4.863457 0.37529337 … 2.1437478 1.2139851; 0.5289551 0.6978088 … 4.3122315 0.0; … ; 4.696 3.9705887 … 4.081905 0.0; 0.0 0.11040878 … 0.3872763 6.2378902] Float32[0.0 11.649898 … 2.8375564 0.0; 0.0 2.904213 … 0.0 0.0; … ; 0.0 0.0 … 0.0 2.3017268; 0.0 0.0 … 5.2382174 4.5692763] Float32[1.9801435 0.0 … 2.4855614 0.4469435; 0.0 1.4986401 … 5.4092045 0.0; … ; 0.0 12.707787 … 0.0 0.0; 1.4063678 2.7959795 … 0.9731283 0.0] Float32[0.49171594 5.0228662 … 0.0 2.414989; 0.0 0.0 … 0.0 13.332266; … ; 0.0 0.0 … 0.0 4.917992; 0.96881 2.328949 … 1.3814683 0.0] Float32[3.0496302 7.3061557 … 0.13163531 0.0; 0.0 5.0266824 … 3.5797634 6.234811; … ; 0.0 0.0 … 2.9561696 2.1210437; 2.0376396 8.286372 … 0.0 0.0] Stacktrace: [1] macro expansion @ ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:146 [inlined] [2] macro expansion @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined] [3] (::var"#convtest#6"{var"#convtest#4#7"{CuArray{Float32, 4}, CuArray{Float32, 4}, Array{Float32, 4}, Array{Float32, 4}, DataType}, CuArray{Float32, 4}})(; blendz::Bool, bias::CuArray{Float32, 4}, activation::cudnnActivationMode_t, mode::cudnnConvolutionMode_t, padding::Int64, stride::Int64, dilation::Int64, group::Int64, dataType::Type, mathType::cudnnMathType_t, reorderType::cudnnReorderType_t, alpha::Int64, beta::Int64) @ Main ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:98 blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=1, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0: Test Failed at /home/tim/Julia/pkg/CUDA/test/cudnn/convolution.jl:147 Expression: ay2 ≈ cudnnConvolutionForward!(cy0, cw0, cx; z = cz0, bias, activation, mode, padding, stride, dilation, group, mathType, reorderType, alpha, beta) |> Array Evaluated: Float32[-7.625949 -6.12281 … -5.9059563 -4.710522; 0.5002041 -0.22734165 … -11.341359 -0.029361725; … ; -5.955898 -0.8519391 … 9.367512 3.6044025; -1.369788 -4.7792273 … -1.0492474 -5.542263] Float32[0.93721855 -1.0799328 … -4.7547307 -1.3063779; 2.664013 3.7837796 … 15.958762 -1.1090627; … ; 0.11813876 0.18879864 … -1.8676813 -3.0879667; 0.8384547 4.255574 … 1.7350059 -4.358764] Float32[1.0326179 -9.130896 … -1.3046414 -1.638468; -0.403609 -3.2207842 … 6.939209 -1.9991591; … ; 1.3053217 7.436456 … -8.234082 -7.437218; 1.4383951 -0.64058656 … -6.64644 2.529673] Float32[2.1496897 -0.2559681 … -2.7595649 1.112249; 0.8892468 -0.2949109 … -2.3872485 -1.0089985; … ; 8.299812 7.852995 … 3.1898193 -1.2525343; 4.876601 4.505626 … -2.9399366 -2.8759317] Float32[-2.7328777 -7.9601955 … -7.8990173 1.8676727; -1.3844309 -7.88463 … 0.6634226 -9.988422; … ; -2.9437788 -3.2956274 … -8.971467 6.3641863; -3.4225056 -3.73323 … -0.21863937 1.5250564] Float32[-3.3950315 8.367449 … -4.26904 2.1378958; -3.221842 9.269204 … 5.919178 -8.4661875; … ; -2.908155 -4.207021 … 4.4138117 -1.76969; 0.3866739 6.6738005 … -2.8421426 -4.412014] Float32[4.9955983 8.106892 … 2.7535727 -14.277321; 2.3030026 3.396777 … -13.603948 4.5709653; … ; 2.3793406 -10.423798 … -0.92979616 7.863477; 6.555591 -1.0130867 … -2.5044403 0.09186761] Float32[1.9763883 0.7442095 … -2.7087684 2.416678; 2.5668755 -2.578361 … 4.1207623 -9.057405; … ; -3.7044673 2.5795178 … 8.583115 3.2888846; 1.4762158 -2.704668 … -0.1852051 -0.94584596] Float32[-9.431355 -5.288068 … -2.950151 2.0531697; -6.699032 4.6442776 … 1.7998717 -1.6313852; … ; -10.450587 -10.025318 … -15.630849 -3.5284545; -4.011667 -8.436462 … -7.4438815 -0.9473691] Float32[-3.8612556 8.199958 … 0.776072 0.7085123; -6.236965 0.6840241 … 0.012681454 2.3597157; … ; -2.7117143 -4.8041277 … -6.732696 6.679207; 1.4808843 4.483176 … 4.4658012 -0.3740994] Float32[-1.4137484 -1.9556055 … 2.8444076 -5.4156904; 0.91588575 -0.56438535 … -4.6698 -3.8603404; … ; -1.1119218 -0.7578048 … -5.5975842 -1.3475385; 0.93386126 -4.766769 … -3.479992 3.774753] Float32[4.8634553 0.3752916 … 2.1437469 1.2139807; 0.52895725 0.697806 … 4.3122315 -1.3506848; … ; 4.6959987 3.9705896 … 4.081906 -0.7670711; -0.97394645 0.110408664 … 0.3872782 6.2378907] Float32[-3.463529 11.649901 … 2.8375607 -0.6578778; -2.9366102 2.9042091 … -4.8822327 -1.283011; … ; -4.0112977 -0.26579285 … -7.6523533 2.3017254; -5.8749695 -13.766709 … 5.2382183 4.569278] Float32[1.980134 -0.5140625 … 2.485562 0.44694278; -6.756843 1.4986404 … 5.4092026 -0.13282302; … ; -7.779603 12.7077875 … -1.4024372 -2.7751215; 1.4063643 2.7959778 … 0.9731282 -2.4291978] Float32[0.49171367 5.022867 … -15.067985 2.414992; -2.4574265 -9.237441 … -4.8011904 13.332268; … ; -6.1316247 -8.423361 … -3.69942 4.917993; 0.96881276 2.3289487 … 1.381468 -5.249476] Float32[3.049631 7.306164 … 0.13163137 -2.8691373; -1.7166475 5.0266857 … 3.5797625 6.234812; … ; -4.7470737 -2.4605598 … 2.95617 2.1210446; 2.0376394 8.286376 … -8.101818 -2.9176269] ≈ Float32[0.0 0.0 … 0.0 0.0; 0.500196 0.0 … 0.0 0.0; … ; 0.0 0.0 … 9.367514 3.6044035; 0.0 0.0 … 0.0 0.0] Float32[0.9372158 0.0 … 0.0 0.0; 2.6640186 3.7837784 … 15.958762 0.0; … ; 0.11814305 0.18879744 … 0.0 0.0; 0.83845544 4.255575 … 1.7350042 0.0] Float32[1.0326269 0.0 … 0.0 0.0; 0.0 0.0 … 6.9392095 0.0; … ; 1.3053309 7.436457 … 0.0 0.0; 1.4383941 0.0 … 0.0 2.5296755] Float32[2.1497054 0.0 … 0.0 1.1122471; 0.8892542 0.0 … 0.0 0.0; … ; 8.299812 7.852998 … 3.1898184 0.0; 4.876604 4.505626 … 0.0 0.0] Float32[0.0 0.0 … 0.0 1.8676558; 0.0 0.0 … 0.6634207 0.0; … ; 0.0 0.0 … 0.0 6.3641863; 0.0 0.0 … 0.0 1.5250559] Float32[0.0 8.367454 … 0.0 2.1378973; 0.0 9.269204 … 5.91918 0.0; … ; 0.0 0.0 … 4.413811 0.0; 0.38668057 6.6737986 … 0.0 0.0] Float32[4.995608 8.106895 … 2.75357 0.0; 2.30298 3.3967743 … 0.0 4.5709653; … ; 2.3793416 0.0 … 0.0 7.8634763; 6.55559 0.0 … 0.0 0.091865465] Float32[1.976395 0.744218 … 0.0 2.4166713; 2.5668888 0.0 … 4.12076 0.0; … ; 0.0 2.5795188 … 8.583115 3.288885; 1.4762126 0.0 … 0.0 0.0] Float32[0.0 0.0 … 0.0 2.053174; 0.0 4.644282 … 1.7998714 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0] Float32[0.0 8.199955 … 0.7760706 0.7085104; 0.0 0.6840248 … 0.012680501 2.3597167; … ; 0.0 0.0 … 0.0 6.6792064; 1.4808874 4.4831758 … 4.4658027 0.0] Float32[0.0 0.0 … 2.8444114 0.0; 0.9158835 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.9338594 0.0 … 0.0 3.7747526] Float32[4.863457 0.37529337 … 2.1437478 1.2139851; 0.5289551 0.6978088 … 4.3122315 0.0; … ; 4.696 3.9705887 … 4.081905 0.0; 0.0 0.11040878 … 0.3872763 6.2378902] Float32[0.0 11.649898 … 2.8375564 0.0; 0.0 2.904213 … 0.0 0.0; … ; 0.0 0.0 … 0.0 2.3017268; 0.0 0.0 … 5.2382174 4.5692763] Float32[1.9801435 0.0 … 2.4855614 0.4469435; 0.0 1.4986401 … 5.4092045 0.0; … ; 0.0 12.707787 … 0.0 0.0; 1.4063678 2.7959795 … 0.9731283 0.0] Float32[0.49171594 5.0228662 … 0.0 2.414989; 0.0 0.0 … 0.0 13.332266; … ; 0.0 0.0 … 0.0 4.917992; 0.96881 2.328949 … 1.3814683 0.0] Float32[3.0496302 7.3061557 … 0.13163531 0.0; 0.0 5.0266824 … 3.5797634 6.234811; … ; 0.0 0.0 … 2.9561696 2.1210437; 2.0376396 8.286372 … 0.0 0.0] Stacktrace: [1] macro expansion @ ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:147 [inlined] [2] macro expansion @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined] [3] (::var"#convtest#6"{var"#convtest#4#7"{CuArray{Float32, 4}, CuArray{Float32, 4}, Array{Float32, 4}, Array{Float32, 4}, DataType}, CuArray{Float32, 4}})(; blendz::Bool, bias::CuArray{Float32, 4}, activation::cudnnActivationMode_t, mode::cudnnConvolutionMode_t, padding::Int64, stride::Int64, dilation::Int64, group::Int64, dataType::Type, mathType::cudnnMathType_t, reorderType::cudnnReorderType_t, alpha::Int64, beta::Int64) @ Main ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:98 blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=1, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0: Test Failed at /home/tim/Julia/pkg/CUDA/test/cudnn/convolution.jl:150 Expression: ay2 ≈ cudnnConvolutionForward!(cy1, cw0, cx, d; z = cz1, bias, activation, alpha, beta) |> Array Evaluated: Float32[-7.625949 -6.12281 … -5.9059563 -4.710522; 0.5002041 -0.22734165 … -11.341359 -0.029361725; … ; -5.955898 -0.8519391 … 9.367512 3.6044025; -1.369788 -4.7792273 … -1.0492474 -5.542263] Float32[0.93721855 -1.0799328 … -4.7547307 -1.3063779; 2.664013 3.7837796 … 15.958762 -1.1090627; … ; 0.11813876 0.18879864 … -1.8676813 -3.0879667; 0.8384547 4.255574 … 1.7350059 -4.358764] Float32[1.0326179 -9.130896 … -1.3046414 -1.638468; -0.403609 -3.2207842 … 6.939209 -1.9991591; … ; 1.3053217 7.436456 … -8.234082 -7.437218; 1.4383951 -0.64058656 … -6.64644 2.529673] Float32[2.1496897 -0.2559681 … -2.7595649 1.112249; 0.8892468 -0.2949109 … -2.3872485 -1.0089985; … ; 8.299812 7.852995 … 3.1898193 -1.2525343; 4.876601 4.505626 … -2.9399366 -2.8759317] Float32[-2.7328777 -7.9601955 … -7.8990173 1.8676727; -1.3844309 -7.88463 … 0.6634226 -9.988422; … ; -2.9437788 -3.2956274 … -8.971467 6.3641863; -3.4225056 -3.73323 … -0.21863937 1.5250564] Float32[-3.3950315 8.367449 … -4.26904 2.1378958; -3.221842 9.269204 … 5.919178 -8.4661875; … ; -2.908155 -4.207021 … 4.4138117 -1.76969; 0.3866739 6.6738005 … -2.8421426 -4.412014] Float32[4.9955983 8.106892 … 2.7535727 -14.277321; 2.3030026 3.396777 … -13.603948 4.5709653; … ; 2.3793406 -10.423798 … -0.92979616 7.863477; 6.555591 -1.0130867 … -2.5044403 0.09186761] Float32[1.9763883 0.7442095 … -2.7087684 2.416678; 2.5668755 -2.578361 … 4.1207623 -9.057405; … ; -3.7044673 2.5795178 … 8.583115 3.2888846; 1.4762158 -2.704668 … -0.1852051 -0.94584596] Float32[-9.431355 -5.288068 … -2.950151 2.0531697; -6.699032 4.6442776 … 1.7998717 -1.6313852; … ; -10.450587 -10.025318 … -15.630849 -3.5284545; -4.011667 -8.436462 … -7.4438815 -0.9473691] Float32[-3.8612556 8.199958 … 0.776072 0.7085123; -6.236965 0.6840241 … 0.012681454 2.3597157; … ; -2.7117143 -4.8041277 … -6.732696 6.679207; 1.4808843 4.483176 … 4.4658012 -0.3740994] Float32[-1.4137484 -1.9556055 … 2.8444076 -5.4156904; 0.91588575 -0.56438535 … -4.6698 -3.8603404; … ; -1.1119218 -0.7578048 … -5.5975842 -1.3475385; 0.93386126 -4.766769 … -3.479992 3.774753] Float32[4.8634553 0.3752916 … 2.1437469 1.2139807; 0.52895725 0.697806 … 4.3122315 -1.3506848; … ; 4.6959987 3.9705896 … 4.081906 -0.7670711; -0.97394645 0.110408664 … 0.3872782 6.2378907] Float32[-3.463529 11.649901 … 2.8375607 -0.6578778; -2.9366102 2.9042091 … -4.8822327 -1.283011; … ; -4.0112977 -0.26579285 … -7.6523533 2.3017254; -5.8749695 -13.766709 … 5.2382183 4.569278] Float32[1.980134 -0.5140625 … 2.485562 0.44694278; -6.756843 1.4986404 … 5.4092026 -0.13282302; … ; -7.779603 12.7077875 … -1.4024372 -2.7751215; 1.4063643 2.7959778 … 0.9731282 -2.4291978] Float32[0.49171367 5.022867 … -15.067985 2.414992; -2.4574265 -9.237441 … -4.8011904 13.332268; … ; -6.1316247 -8.423361 … -3.69942 4.917993; 0.96881276 2.3289487 … 1.381468 -5.249476] Float32[3.049631 7.306164 … 0.13163137 -2.8691373; -1.7166475 5.0266857 … 3.5797625 6.234812; … ; -4.7470737 -2.4605598 … 2.95617 2.1210446; 2.0376394 8.286376 … -8.101818 -2.9176269] ≈ Float32[0.0 0.0 … 0.0 0.0; 0.500196 0.0 … 0.0 0.0; … ; 0.0 0.0 … 9.367514 3.6044035; 0.0 0.0 … 0.0 0.0] Float32[0.9372158 0.0 … 0.0 0.0; 2.6640186 3.7837784 … 15.958762 0.0; … ; 0.11814305 0.18879744 … 0.0 0.0; 0.83845544 4.255575 … 1.7350042 0.0] Float32[1.0326269 0.0 … 0.0 0.0; 0.0 0.0 … 6.9392095 0.0; … ; 1.3053309 7.436457 … 0.0 0.0; 1.4383941 0.0 … 0.0 2.5296755] Float32[2.1497054 0.0 … 0.0 1.1122471; 0.8892542 0.0 … 0.0 0.0; … ; 8.299812 7.852998 … 3.1898184 0.0; 4.876604 4.505626 … 0.0 0.0] Float32[0.0 0.0 … 0.0 1.8676558; 0.0 0.0 … 0.6634207 0.0; … ; 0.0 0.0 … 0.0 6.3641863; 0.0 0.0 … 0.0 1.5250559] Float32[0.0 8.367454 … 0.0 2.1378973; 0.0 9.269204 … 5.91918 0.0; … ; 0.0 0.0 … 4.413811 0.0; 0.38668057 6.6737986 … 0.0 0.0] Float32[4.995608 8.106895 … 2.75357 0.0; 2.30298 3.3967743 … 0.0 4.5709653; … ; 2.3793416 0.0 … 0.0 7.8634763; 6.55559 0.0 … 0.0 0.091865465] Float32[1.976395 0.744218 … 0.0 2.4166713; 2.5668888 0.0 … 4.12076 0.0; … ; 0.0 2.5795188 … 8.583115 3.288885; 1.4762126 0.0 … 0.0 0.0] Float32[0.0 0.0 … 0.0 2.053174; 0.0 4.644282 … 1.7998714 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0] Float32[0.0 8.199955 … 0.7760706 0.7085104; 0.0 0.6840248 … 0.012680501 2.3597167; … ; 0.0 0.0 … 0.0 6.6792064; 1.4808874 4.4831758 … 4.4658027 0.0] Float32[0.0 0.0 … 2.8444114 0.0; 0.9158835 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.9338594 0.0 … 0.0 3.7747526] Float32[4.863457 0.37529337 … 2.1437478 1.2139851; 0.5289551 0.6978088 … 4.3122315 0.0; … ; 4.696 3.9705887 … 4.081905 0.0; 0.0 0.11040878 … 0.3872763 6.2378902] Float32[0.0 11.649898 … 2.8375564 0.0; 0.0 2.904213 … 0.0 0.0; … ; 0.0 0.0 … 0.0 2.3017268; 0.0 0.0 … 5.2382174 4.5692763] Float32[1.9801435 0.0 … 2.4855614 0.4469435; 0.0 1.4986401 … 5.4092045 0.0; … ; 0.0 12.707787 … 0.0 0.0; 1.4063678 2.7959795 … 0.9731283 0.0] Float32[0.49171594 5.0228662 … 0.0 2.414989; 0.0 0.0 … 0.0 13.332266; … ; 0.0 0.0 … 0.0 4.917992; 0.96881 2.328949 … 1.3814683 0.0] Float32[3.0496302 7.3061557 … 0.13163531 0.0; 0.0 5.0266824 … 3.5797634 6.234811; … ; 0.0 0.0 … 2.9561696 2.1210437; 2.0376396 8.286372 … 0.0 0.0] Stacktrace: [1] macro expansion @ ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:150 [inlined] [2] macro expansion @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined] [3] (::var"#convtest#6"{var"#convtest#4#7"{CuArray{Float32, 4}, CuArray{Float32, 4}, Array{Float32, 4}, Array{Float32, 4}, DataType}, CuArray{Float32, 4}})(; blendz::Bool, bias::CuArray{Float32, 4}, activation::cudnnActivationMode_t, mode::cudnnConvolutionMode_t, padding::Int64, stride::Int64, dilation::Int64, group::Int64, dataType::Type, mathType::cudnnMathType_t, reorderType::cudnnReorderType_t, alpha::Int64, beta::Int64) @ Main ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:98 blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH_ALLOW_CONVERSION, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0: Error During Test at /home/tim/Julia/pkg/CUDA/test/cudnn/convolution.jl:143 Test threw exception Expression: ay1 ≈ cudnnConvolutionForward(cw0, cx; bias, activation, mode, padding, stride, dilation, group, mathType, reorderType, alpha) |> Array CUDNNError: CUDNN_STATUS_BAD_PARAM (code 3) Stacktrace: [1] throw_api_error(res::CUDA.CUDNN.cudnnStatus_t) @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/cudnn/error.jl:19 [2] macro expansion @ ~/Julia/pkg/CUDA/lib/cudnn/error.jl:30 [inlined] [3] cudnnConvolutionBiasActivationForward(handle::Ptr{Nothing}, alpha1::Base.RefValue{Float32}, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, x::CuArray{Float32, 4}, wDesc::CUDA.CUDNN.cudnnFilterDescriptor, w::CuArray{Float32, 4}, convDesc::cudnnConvolutionDescriptor, algo::cudnnConvolutionFwdAlgo_t, workSpace::CuArray{UInt8, 1}, workSpaceSizeInBytes::Int64, alpha2::Base.RefValue{Float32}, zDesc::CUDA.CUDNN.cudnnTensorDescriptor, z::CuArray{Float32, 4}, biasDesc::CUDA.CUDNN.cudnnTensorDescriptor, bias::CuArray{Float32, 4}, activationDesc::CUDA.CUDNN.cudnnActivationDescriptor, yDesc::CUDA.CUDNN.cudnnTensorDescriptor, y::CuArray{Float32, 4}) @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/utils/call.jl:26 [4] macro expansion @ ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:117 [inlined] [5] macro expansion @ ~/Julia/pkg/CUDA/lib/utils/call.jl:144 [inlined] [6] cudnnConvolutionForwardAD(w::CuArray{Float32, 4}, x::CuArray{Float32, 4}, bias::CuArray{Float32, 4}, z::Nothing; y::CuArray{Float32, 4}, activation::cudnnActivationMode_t, convDesc::cudnnConvolutionDescriptor, wDesc::CUDA.CUDNN.cudnnFilterDescriptor, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, yDesc::CUDA.CUDNN.cudnnTensorDescriptor, zDesc::Nothing, biasDesc::CUDA.CUDNN.cudnnTensorDescriptor, alpha::Base.RefValue{Float32}, beta::Base.RefValue{Float32}, dw::Base.RefValue{Any}, dx::Base.RefValue{Any}, dz::Base.RefValue{Any}, dbias::Base.RefValue{Any}, dready::Base.RefValue{Bool}) @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:103 [7] cudnnConvolutionForwardWithDefaults(w::CuArray{Float32, 4}, x::CuArray{Float32, 4}; padding::Int64, stride::Int64, dilation::Int64, mode::cudnnConvolutionMode_t, mathType::cudnnMathType_t, reorderType::cudnnReorderType_t, group::Int64, format::cudnnTensorFormat_t, convDesc::cudnnConvolutionDescriptor, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, wDesc::CUDA.CUDNN.cudnnFilterDescriptor, y::CuArray{Float32, 4}, yDesc::CUDA.CUDNN.cudnnTensorDescriptor, alpha::Int64, beta::Int64, bias::CuArray{Float32, 4}, z::Nothing, biasDesc::CUDA.CUDNN.cudnnTensorDescriptor, zDesc::Nothing, activation::cudnnActivationMode_t, dw::Base.RefValue{Any}, dx::Base.RefValue{Any}, dz::Base.RefValue{Any}, dbias::Base.RefValue{Any}) @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:96 [8] #cudnnConvolutionForward#104 @ ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:50 [inlined] [9] macro expansion @ ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:143 [inlined] [10] macro expansion @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined] [11] (::var"#convtest#6"{var"#convtest#4#7"{CuArray{Float32, 4}, CuArray{Float32, 4}, Array{Float32, 4}, Array{Float32, 4}, DataType}, CuArray{Float32, 4}})(; blendz::Bool, bias::CuArray{Float32, 4}, activation::cudnnActivationMode_t, mode::cudnnConvolutionMode_t, padding::Int64, stride::Int64, dilation::Int64, group::Int64, dataType::Type, mathType::cudnnMathType_t, reorderType::cudnnReorderType_t, alpha::Int64, beta::Int64) @ Main ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:98 blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH_ALLOW_CONVERSION, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0: Error During Test at /home/tim/Julia/pkg/CUDA/test/cudnn/convolution.jl:146 Test threw exception Expression: ay1 ≈ cudnnConvolutionForward(cw0, cx, d; bias, activation, alpha) |> Array CUDNNError: CUDNN_STATUS_BAD_PARAM (code 3) Stacktrace: [1] throw_api_error(res::CUDA.CUDNN.cudnnStatus_t) @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/cudnn/error.jl:19 [2] macro expansion @ ~/Julia/pkg/CUDA/lib/cudnn/error.jl:30 [inlined] [3] cudnnConvolutionBiasActivationForward(handle::Ptr{Nothing}, alpha1::Base.RefValue{Float32}, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, x::CuArray{Float32, 4}, wDesc::CUDA.CUDNN.cudnnFilterDescriptor, w::CuArray{Float32, 4}, convDesc::cudnnConvolutionDescriptor, algo::cudnnConvolutionFwdAlgo_t, workSpace::CuArray{UInt8, 1}, workSpaceSizeInBytes::Int64, alpha2::Base.RefValue{Float32}, zDesc::CUDA.CUDNN.cudnnTensorDescriptor, z::CuArray{Float32, 4}, biasDesc::CUDA.CUDNN.cudnnTensorDescriptor, bias::CuArray{Float32, 4}, activationDesc::CUDA.CUDNN.cudnnActivationDescriptor, yDesc::CUDA.CUDNN.cudnnTensorDescriptor, y::CuArray{Float32, 4}) @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/utils/call.jl:26 [4] macro expansion @ ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:117 [inlined] [5] macro expansion @ ~/Julia/pkg/CUDA/lib/utils/call.jl:144 [inlined] [6] cudnnConvolutionForwardAD(w::CuArray{Float32, 4}, x::CuArray{Float32, 4}, bias::CuArray{Float32, 4}, z::Nothing; y::CuArray{Float32, 4}, activation::cudnnActivationMode_t, convDesc::cudnnConvolutionDescriptor, wDesc::CUDA.CUDNN.cudnnFilterDescriptor, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, yDesc::CUDA.CUDNN.cudnnTensorDescriptor, zDesc::Nothing, biasDesc::CUDA.CUDNN.cudnnTensorDescriptor, alpha::Base.RefValue{Float32}, beta::Base.RefValue{Float32}, dw::Base.RefValue{Any}, dx::Base.RefValue{Any}, dz::Base.RefValue{Any}, dbias::Base.RefValue{Any}, dready::Base.RefValue{Bool}) @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:103 [7] cudnnConvolutionForwardWithDefaults(w::CuArray{Float32, 4}, x::CuArray{Float32, 4}; padding::Int64, stride::Int64, dilation::Int64, mode::cudnnConvolutionMode_t, mathType::cudnnMathType_t, reorderType::cudnnReorderType_t, group::Int64, format::cudnnTensorFormat_t, convDesc::cudnnConvolutionDescriptor, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, wDesc::CUDA.CUDNN.cudnnFilterDescriptor, y::CuArray{Float32, 4}, yDesc::CUDA.CUDNN.cudnnTensorDescriptor, alpha::Int64, beta::Int64, bias::CuArray{Float32, 4}, z::Nothing, biasDesc::CUDA.CUDNN.cudnnTensorDescriptor, zDesc::Nothing, activation::cudnnActivationMode_t, dw::Base.RefValue{Any}, dx::Base.RefValue{Any}, dz::Base.RefValue{Any}, dbias::Base.RefValue{Any}) @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:96 [8] #cudnnConvolutionForward#106 @ ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:52 [inlined] [9] macro expansion @ ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:146 [inlined] [10] macro expansion @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined] [11] (::var"#convtest#6"{var"#convtest#4#7"{CuArray{Float32, 4}, CuArray{Float32, 4}, Array{Float32, 4}, Array{Float32, 4}, DataType}, CuArray{Float32, 4}})(; blendz::Bool, bias::CuArray{Float32, 4}, activation::cudnnActivationMode_t, mode::cudnnConvolutionMode_t, padding::Int64, stride::Int64, dilation::Int64, group::Int64, dataType::Type, mathType::cudnnMathType_t, reorderType::cudnnReorderType_t, alpha::Int64, beta::Int64) @ Main ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:98 blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH_ALLOW_CONVERSION, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0: Error During Test at /home/tim/Julia/pkg/CUDA/test/cudnn/convolution.jl:147 Test threw exception Expression: ay2 ≈ cudnnConvolutionForward!(cy0, cw0, cx; z = cz0, bias, activation, mode, padding, stride, dilation, group, mathType, reorderType, alpha, beta) |> Array CUDNNError: CUDNN_STATUS_BAD_PARAM (code 3) Stacktrace: [1] throw_api_error(res::CUDA.CUDNN.cudnnStatus_t) @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/cudnn/error.jl:19 [2] macro expansion @ ~/Julia/pkg/CUDA/lib/cudnn/error.jl:30 [inlined] [3] cudnnConvolutionBiasActivationForward(handle::Ptr{Nothing}, alpha1::Base.RefValue{Float32}, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, x::CuArray{Float32, 4}, wDesc::CUDA.CUDNN.cudnnFilterDescriptor, w::CuArray{Float32, 4}, convDesc::cudnnConvolutionDescriptor, algo::cudnnConvolutionFwdAlgo_t, workSpace::CuArray{UInt8, 1}, workSpaceSizeInBytes::Int64, alpha2::Base.RefValue{Float32}, zDesc::CUDA.CUDNN.cudnnTensorDescriptor, z::CuArray{Float32, 4}, biasDesc::CUDA.CUDNN.cudnnTensorDescriptor, bias::CuArray{Float32, 4}, activationDesc::CUDA.CUDNN.cudnnActivationDescriptor, yDesc::CUDA.CUDNN.cudnnTensorDescriptor, y::CuArray{Float32, 4}) @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/utils/call.jl:26 [4] macro expansion @ ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:117 [inlined] [5] macro expansion @ ~/Julia/pkg/CUDA/lib/utils/call.jl:144 [inlined] [6] cudnnConvolutionForwardAD(w::CuArray{Float32, 4}, x::CuArray{Float32, 4}, bias::CuArray{Float32, 4}, z::CuArray{Float32, 4}; y::CuArray{Float32, 4}, activation::cudnnActivationMode_t, convDesc::cudnnConvolutionDescriptor, wDesc::CUDA.CUDNN.cudnnFilterDescriptor, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, yDesc::CUDA.CUDNN.cudnnTensorDescriptor, zDesc::CUDA.CUDNN.cudnnTensorDescriptor, biasDesc::CUDA.CUDNN.cudnnTensorDescriptor, alpha::Base.RefValue{Float32}, beta::Base.RefValue{Float32}, dw::Base.RefValue{Any}, dx::Base.RefValue{Any}, dz::Base.RefValue{Any}, dbias::Base.RefValue{Any}, dready::Base.RefValue{Bool}) @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:103 [7] #cudnnConvolutionForwardWithDefaults#108 @ ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:96 [inlined] [8] #cudnnConvolutionForward!#105 <#105> @ ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:51 [inlined] [9] macro expansion @ ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:147 [inlined] [10] macro expansion @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined] [11] (::var"#convtest#6"{var"#convtest#4#7"{CuArray{Float32, 4}, CuArray{Float32, 4}, Array{Float32, 4}, Array{Float32, 4}, DataType}, CuArray{Float32, 4}})(; blendz::Bool, bias::CuArray{Float32, 4}, activation::cudnnActivationMode_t, mode::cudnnConvolutionMode_t, padding::Int64, stride::Int64, dilation::Int64, group::Int64, dataType::Type, mathType::cudnnMathType_t, reorderType::cudnnReorderType_t, alpha::Int64, beta::Int64) @ Main ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:98 blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH_ALLOW_CONVERSION, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0: Error During Test at /home/tim/Julia/pkg/CUDA/test/cudnn/convolution.jl:150 Test threw exception Expression: ay2 ≈ cudnnConvolutionForward!(cy1, cw0, cx, d; z = cz1, bias, activation, alpha, beta) |> Array CUDNNError: CUDNN_STATUS_BAD_PARAM (code 3) Stacktrace: [1] throw_api_error(res::CUDA.CUDNN.cudnnStatus_t) @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/cudnn/error.jl:19 [2] macro expansion @ ~/Julia/pkg/CUDA/lib/cudnn/error.jl:30 [inlined] [3] cudnnConvolutionBiasActivationForward(handle::Ptr{Nothing}, alpha1::Base.RefValue{Float32}, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, x::CuArray{Float32, 4}, wDesc::CUDA.CUDNN.cudnnFilterDescriptor, w::CuArray{Float32, 4}, convDesc::cudnnConvolutionDescriptor, algo::cudnnConvolutionFwdAlgo_t, workSpace::CuArray{UInt8, 1}, workSpaceSizeInBytes::Int64, alpha2::Base.RefValue{Float32}, zDesc::CUDA.CUDNN.cudnnTensorDescriptor, z::CuArray{Float32, 4}, biasDesc::CUDA.CUDNN.cudnnTensorDescriptor, bias::CuArray{Float32, 4}, activationDesc::CUDA.CUDNN.cudnnActivationDescriptor, yDesc::CUDA.CUDNN.cudnnTensorDescriptor, y::CuArray{Float32, 4}) @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/utils/call.jl:26 [4] macro expansion @ ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:117 [inlined] [5] macro expansion @ ~/Julia/pkg/CUDA/lib/utils/call.jl:144 [inlined] [6] cudnnConvolutionForwardAD(w::CuArray{Float32, 4}, x::CuArray{Float32, 4}, bias::CuArray{Float32, 4}, z::CuArray{Float32, 4}; y::CuArray{Float32, 4}, activation::cudnnActivationMode_t, convDesc::cudnnConvolutionDescriptor, wDesc::CUDA.CUDNN.cudnnFilterDescriptor, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, yDesc::CUDA.CUDNN.cudnnTensorDescriptor, zDesc::CUDA.CUDNN.cudnnTensorDescriptor, biasDesc::CUDA.CUDNN.cudnnTensorDescriptor, alpha::Base.RefValue{Float32}, beta::Base.RefValue{Float32}, dw::Base.RefValue{Any}, dx::Base.RefValue{Any}, dz::Base.RefValue{Any}, dbias::Base.RefValue{Any}, dready::Base.RefValue{Bool}) @ CUDA.CUDNN ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:103 [7] #cudnnConvolutionForwardWithDefaults#108 @ ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:96 [inlined] [8] #cudnnConvolutionForward!#107 <#107> @ ~/Julia/pkg/CUDA/lib/cudnn/convolution.jl:53 [inlined] [9] macro expansion @ ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:150 [inlined] [10] macro expansion @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined] [11] (::var"#convtest#6"{var"#convtest#4#7"{CuArray{Float32, 4}, CuArray{Float32, 4}, Array{Float32, 4}, Array{Float32, 4}, DataType}, CuArray{Float32, 4}})(; blendz::Bool, bias::CuArray{Float32, 4}, activation::cudnnActivationMode_t, mode::cudnnConvolutionMode_t, padding::Int64, stride::Int64, dilation::Int64, group::Int64, dataType::Type, mathType::cudnnMathType_t, reorderType::cudnnReorderType_t, alpha::Int64, beta::Int64) @ Main ~/Julia/pkg/CUDA/test/cudnn/convolution.jl:98 Test Summary: | Pass Fail Error Total cudnn/convolution | 102 4 4 110 blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=1, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0 | 4 4 blendz=false, bias=CuArray{Float32, 4}, activation=CUDNN_ACTIVATION_IDENTITY, mode=CUDNN_CONVOLUTION, padding=0, stride=1, dilation=1, group=1, dataType=Float32, mathType=CUDNN_TENSOR_OP_MATH_ALLOW_CONVERSION, reorderType=CUDNN_DEFAULT_REORDER, alpha=1, beta=0 | 4 4 Notice the 4 STATUS_BAD_PARAM errors too! — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#725 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAN43JSMH565WFSLC6475L3TAIYHPANCNFSM4X4WOIJQ> .

[maleadt]

This is just too breaking for CI, so I'm disabling the CUDNN/convolution tests for now: 20abc44.
It isn't a case of faulty hardware, as it happens locally and on different CI servers.

[denizyuret]

Sounds good. It broke on my machine sometime after the 60 minute mark as well. We'll keep looking into this. If it is caused by calls with particular keyword arguments we can turn on the other convolution tests. If it occurs generally to any call then it is best to disable all until we can figure out what is going on.

[maleadt]

Sometimes there's also memory errors during testing, e.g. https://buildkite.com/julialang/cuda-dot-jl/builds/1055, so I wonder if there's some buffers being mishandled...

[kshyatt]

I'm going to try to rerun these tests on a loop - perhaps the situation has improved since 2021? If so it would be nice to be able to cover this logic again

[kshyatt]

In particular, CUDA 11.4 (which is the oldest we support) came out in November 2021 so perhaps there's hope this was fixed?