Skip to content

Performance degradation when reducing SparseMatrixCSC #27836

@bicycle1885

Description

@bicycle1885

I've found an extreme performance degradation (~30x slower) on Julia 0.7.0-beta vs. on Julia 0.6.2 when reducing a large SparseMatrixCSC matrix using maximum. I couldn't find differences in the data, so I think this is a problem in SparseArrays.

Julia 0.6.2

julia> size(counts)
(27998, 1306127)

julia> typeof(counts)
SparseMatrixCSC{Int32,Int64}

julia> @time maximum(counts, 2);
  9.171888 seconds (103.28 k allocations: 5.739 MiB, 0.06% gc time)

julia> @time sum(counts .> 0)
 10.630171 seconds (42.21 k allocations: 22.013 GiB, 0.08% gc time)
2624828308

julia> versioninfo()
Julia Version 0.6.2
Commit d386e40c17 (2017-12-13 18:08 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, broadwell)

Julia 0.7.0-beta:

julia> size(counts)
(27998, 1306127)

julia> typeof(counts)
SparseMatrixCSC{Int32,Int64}

julia> @time maximum(counts, dims=2);
271.014620 seconds (5.28 M allocations: 34.628 GiB, 0.09% gc time)

julia> @time sum(counts .> 0)
 10.284461 seconds (348.48 k allocations: 22.028 GiB, 0.69% gc time)
2624828308

julia> versioninfo()
Julia Version 0.7.0-beta.0
Commit f41b1ecaec (2018-06-24 01:32 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, broadwell)

Metadata

Metadata

Assignees

No one assigned

    Labels

    performanceMust go fasterpotential benchmarkCould make a good benchmark in BaseBenchmarksregressionRegression in behavior compared to a previous versionsparseSparse arrays

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions