Issue: Parallel executor performance is mostly linear #326
Description
We use capistrano to deploy on several hundred servers at once, and we noticed capistrano's performance was heavily tied to the number of servers.
To prove it I ran the following benchmark:
require 'benchmark'
require 'csv'
task :bench do
servers = roles(:borg).to_a
timings = {}
if ENV['PREWARM']
on servers do
execute :echo, 'pong > /dev/null'
end
end
[1, 10, 50, 100].each do |count|
timings[count] = Benchmark.measure do
on servers.pop(count) do
execute :echo, 'pong > /dev/null'
end
end
end
res = [%w(count user system real)]
timings.each do |count, timing|
res << [count, timing.utime, timing.stime, timing.real]
end
puts res.map(&:to_csv).join
endHere are the results: Spreadsheet
The first graph is with "cold" connections, meaning it's connection establishment plus the command execution. In the second graph, all the connections were pre established before the benchmark.
I'm still investigating to figure out where exactly the bottleneck (or bottlenecks) exactly is. I know the GIL is not for nothing, but capistrano / SSHKit being IO heavy, I think there is other reasons.
cc @kirs as well as @csfrancis