-
Notifications
You must be signed in to change notification settings - Fork 45
add basic benchmarks for Julia-level compilation pipeline #288
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
a8f0a4a to
d386165
Compare
|
The failure in Julia nightly is because this added benchmark suite isn't tuned yet, and thus it's tuned like I confirmed this benchmark suite works just correctly on my machine. |
|
I think you need to specify |
|
Even though I set it manually here?
|
|
That will work, assuming no other code later calls tune |
|
Ah, BaseBenchmarks.jl/test/runtests.jl Lines 10 to 14 in 0254882
|
This commit setups a basic infrastructure for benchmarking Julia-level compilation pipeline. `InferenceBenchmarks` is based on `InferenceBenchmarker <: AbstractInterpreter`, which maintains its own global inference cache, and so it allows us to run the compilation pipeline multiple times while avoiding caches generated by previous compilation to be reused. I set up a top-level benchmark group named `"inference": InferenceBenchmarks`, which is composed of the following subgroups: - `"inference"`: just benchmarks overall Julia-level compilation pipeline - `"abstract interpretation"`: benchmarks only abstract interpretation, i.e. without optimization - `"optimization"`: benchmarks only optimization Here is an example of benchmark result obtained by comparing these two commits of `JuliaLang/julia` [`5c357e9`](JuliaLang/julia@5c357e9) and [`d515f05`](JuliaLang/julia@d515f05): ```julia \# built on 5c357e9 using BenchmarkTools, BaseBenchmarks BaseBenchmarks.load!("inference") results = run(BaseBenchmarks.SUITE; verbose = true) BenchmarkTools.save("5c357e9.json", results) \# built on d515f05 using BenchmarkTools, BaseBenchmarks BaseBenchmarks.load!("inference") results = run(BaseBenchmarks.SUITE; verbose = true) BenchmarkTools.save("d515f05.json", results) \# compare using BenchmarkTools, BaseBenchmarks base = BenchmarkTools.load("5c357e9.json")[1] target = BenchmarkTools.load("d515f05.json")[1] ``` ``` julia> leaves(regressions(judge(minimum(target), minimum(base)))) Any[] julia> leaves(improvements(judge(minimum(target), minimum(base)))) 6-element Vector{Any}: (Any["inference", "inference", "rand(Float64)"], TrialJudgement(-2.85% => invariant)) (Any["inference", "inference", "sin(42)"], TrialJudgement(-2.44% => invariant)) (Any["inference", "inference", "abstract_call_gf_by_type"], TrialJudgement(-1.97% => invariant)) (Any["inference", "inference", "println(::QuoteNode)"], TrialJudgement(-0.96% => invariant)) (Any["inference", "optimization", "sin(42)"], TrialJudgement(+1.26% => invariant)) (Any["inference", "optimization", "println(::QuoteNode)"], TrialJudgement(-6.97% => improvement)) ``` This result is very satisfying because the refactor added in `d515f05` certainly improved Julia-level compilation performance by avoiding domtree construction in the SROA pass in many cases.
This commit setups a basic infrastructure for benchmarking
Julia-level compilation pipeline.
InferenceBenchmarksis based onInferenceBenchmarker <: AbstractInterpreter,which maintains its own global inference cache, and so it allows us to
run the compilation pipeline multiple times while avoiding caches generated
by previous compilation to be reused.
I set up a top-level benchmark group named
"inference": InferenceBenchmarks,which is composed of the following subgroups:
"inference": just benchmarks overall Julia-level compilation pipeline"abstract interpretation": benchmarks only abstract interpretation,i.e. without optimization
"optimization": benchmarks only optimizationHere is an example of benchmark result obtained by comparing these two
commits of
JuliaLang/julia5c357e9andd515f05:This result is very satisfying because the refactor added in
d515f05certainly improved Julia-level compilation performance by avoiding
domtree construction in the SROA pass in many cases.