`PHI nodes not grouped at top of basic block!` when testing some packages

[KristofferC]

When running the tests for

"DistributionsAD"
"AdvancedHMC"
"Pathfinder"

with assertions enabled they error with

PHI nodes not grouped at top of basic block!
  %64 = phi <2 x double> [ %36, %idxend15.us ], [ %106, %idxend44.us.postloop ], [ %54, %idxend44.us ]
label %L97.us
julia: /source/src/llvm-lower-handlers.cpp:243: llvm::PreservedAnalyses LowerExcHandlersPass::run(llvm::Function&, llvm::FunctionAnalysisManager&): Assertion `!verifyFunction(F, &errs())' failed.

Example PkgEval log: https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_hash/c9a32f4_vs_e4ee485/DistributionsAD.primary.log

[maleadt]

Bisected to #49798 (cc @pchintalapudi). I'm looking into reducing to an MWE, but there's quite some code involved with these packages.

[maleadt]

Reduced to:

using LinearAlgebra
Base.:*(x, partials)  = scale_tuple(partials, x)
tupexpr(f, N) = Expr(:tuple, [f(i) for i=1:N]...)
@generated function scale_tuple(tup::NTuple{N}, x) where N
    tupexpr(i -> :(tup[$i] * x), N)
end
struct Dual{V,N}
    value::V
    partials::NTuple{N,V}
end
Dual(value, partials::NTuple{N,V}) where {N,V} = Dual{V,N}(value, partials)
Base.:*(x, y::Dual) = Dual(x * y.value, x * y.partials)
Base.:\(x, y::Dual) = Dual(x \ y.value, one(y.value) / x * y.partials)
Base.:-(:, y) = y
struct PDMat{T} <: AbstractMatrix{T} end
Base.size(PDMat) = (adim, adim)
function logdensity_and_gradient(ℓ, x)
    duals = Dual{Float64,10}[]
    cholesky(ℓ).factors' \ duals
end
phasepoint(h, θ::T) where T<:AbstractVector = h(θ)
find_good_stepsize(h, θ) = phasepoint(h, θ)

sig = Tuple{typeof(find_good_stepsize), Base.Fix1{typeof(logdensity_and_gradient), PDMat{Float64}}, Array{Float64}}
CC = Core.Compiler
world = Base.get_world_counter()
match, = CC._findsup(sig, nothing, world)
mi = CC.specialize_method(match)
ccall(:jl_compile_method_internal, Any, (Any, UInt), mi, world)

Run with LLVM assertions enabled, and --check-bounds=yes:

PHI nodes not grouped at top of basic block!
  %28 = phi <4 x double> [ %114, %guard_pass90 ], [ %72, %main.exit.selector ], [ %107, %guard_pass85.loopexit ]
label %L86
julia: /source/src/llvm-lower-handlers.cpp:243: llvm::PreservedAnalyses LowerExcHandlersPass::run(llvm::Function&, llvm::FunctionAnalysisManager&): Assertion `!verifyFunction(F, &errs())' failed.

[pchintalapudi]

This is not actually our fault, this is llvm/llvm-project#60649, fixed by llvm/llvm-project@af39acd, which we should backport. cc @vchuravy