Regression in time for `using Revise`

[timholy]

On 1.1,

$ julia --startup-file=no -q
julia> using Revise

takes ~1.5s to produce the next julia> prompt. On master, it's around 4s. I tried timing this and didn't have time to figure out what's up before needing to switch to other things. It's not the raw package import, @time using Revise is almost the same on both 1.1 and master. But there's clearly some aspect of this that's not being included in that @time; perhaps has more to do with tasks and/or stealing the REPL backend?

I have the sense that this is quite a recent change, perhaps the last week or two.

[JeffBezanson]

It's probably re-compilation that happens when the repl regains control.

[timholy]

I found a sec and did this:

tinfo = @snoopi tmin=0.01 begin
           using Revise
           Revise.steal_repl_backend()
           REPL.run_repl(Base.active_repl)
       end

The result is weird because it gets delayed one cycle in the REPL, but what came out was

12-element Array{Tuple{Float64,Core.MethodInstance},1}:
 (0.015212059020996094, MethodInstance for setindex!(::Dict{String,Revise.WatchList}, ::Revise.WatchList, ::String))                                                                                                                                                                
 (0.015861034393310547, MethodInstance for setindex!(::Dict{Base.PkgId,CodeTracking.PkgFiles}, ::CodeTracking.PkgFiles, ::Base.PkgId))
 (0.01951885223388672, MethodInstance for setindex!(::Dict{Base.PkgId,Revise.PkgData}, ::Revise.PkgData, ::Base.PkgId))
 (0.021203041076660156, MethodInstance for (::getfield(Base, Symbol("#kw##showerror")))(::NamedTuple{(:backtrace,),Tuple{Bool}}, ::typeof(showerror), ::IOContext{REPL.Terminals.TTYTerminal}, ::UndefVarError, ::Array{Union{Ptr{Nothing}, Base.InterpreterIP},1}))
 (0.025014162063598633, MethodInstance for (::getfield(REPL.LineEdit, Symbol("##112#164")))(::Any, ::Any, ::Vararg{Any,N} where N))
 (0.02947688102722168, MethodInstance for #with_output_color#673(::Bool, ::typeof(Base.with_output_color), ::Function, ::Symbol, ::IOContext{REPL.Terminals.TTYTerminal}, ::String))
 (0.029732942581176758, MethodInstance for setindex!(::OrderedCollections.OrderedDict{Module,OrderedCollections.OrderedDict{Revise.RelocatableExpr,Union{Nothing, Array{Any,1}}}}, ::OrderedCollections.OrderedDict{Revise.RelocatableExpr,Union{Nothing, Array{Any,1}}}, ::Module))
 (0.04018902778625488, MethodInstance for watch_files_via_dir(::String))
 (0.06658196449279785, MethodInstance for run_repl(::AbstractREPL))
 (0.08636188507080078, MethodInstance for (::getfield(Revise, Symbol("##20#21")){Base.PkgId})())
 (0.10401010513305664, MethodInstance for __init__())
 (1.5831129550933838, MethodInstance for (::Revise.Rescheduler{typeof(Revise.watch_manifest),Tuple{String}})())

Those setindex! methods are typical and the motivation for #31466. The last entry (which takes 1.58s to infer, by far the slowest of them) is new, and it's normally precompiled: https://github.com/timholy/Revise.jl/blob/09e26525bc0d4b2c8e78bf703d7def07041b8aca/src/precompile.jl#L6. But that seems not to be working on master.

In the absence of type-piracy, should there be a big difference between the amount of recompilation in 1.1 and 1.2? I'm wondering if some methods are being flagged for recompilation that don't need to be.

[JeffBezanson]

In the absence of type-piracy, should there be a big difference between the amount of recompilation in 1.1 and 1.2? I'm wondering if some methods are being flagged for recompilation that don't need to be.

Detecting what needs to be recompiled can only be approximated; we hope to improve the accuracy over time but it will never be perfect. All sorts of code changes can change the dependency graph and sometimes change the amount of re-compilation by a surprising amount. It's independent of type piracy as well; if Base defines f(::Any) and a package adds f(::OtherType) we might need to recompile callers of f even though there is no piracy.

[timholy]

OK. It seems odd, though, that the watch_manifest method would need to be recompiled given that it's precompiled with exactly the same "environment" (meaning, exactly the same loaded packages).

[JeffBezanson]

Yes, all that being said, we can dig into this and see if we are doing something egregiously wrong in this case.