Persistence

Format

A set of YAML files, saved by absolute path of the benchmark file, which contains multiple documents, each of them being a "Result":

{
  metadata: {
    # mandatory
    file: "/full/path/to/benchmark/file.rb",
    session: "Array sorting", # session name
    job: "#sort!", # job title
    ruby: RUBY_DESCRIPTION,
    run_time: Time.now, # time at which the benchmark tool started
                        # Benchmarks ran in the same context/session should have identical :run_time
                        # So :run_time should be initialized once in the application

    # optional
    # configuration
    minimal_time: 1.0,
    measurements: 10,
    verbose: true,

    # git, git data of the current project, if available
    git_branch: "master",
    git_commit: "3433deb...",

    # benchmark type dependent
    iterations: 1234567, # numbers of iterations
    n: 1024, # input size

    # others
    command_line: "perfer run ...", # full command line
    bench_file_checksum: "78e6530...", # benchmark file SHA-1 checksum
  }
  data: [
    {
      # mandatory
      real: 3.14, # real time
      # optional
      utime: 0.01, # user time
      stime: 0.01, # system time
    },
    ...
  ]
}

Given the benchmark file /path/to/benchmark/file.rb, results are saved in ~/.perfer/results/path/to/benchmark/file.rb. (On Windows, the drive letter is added: ~/.perfer/results/C/path/...)

Notions

• necessary for history comparisons
• allows to reuse data a nice way (by keeping at least last run results):

$ perfer run bench_sort
$ perfer report bench_sort
$ perfer graph bench_sort

Dimensions:

• benchmark:
  • project (detected by folder containing .git?)
  • file (should be "relocatable" if absolute path)
  • job
  • version (md5, git hash?)
• time: which code/implementation it ran on (commit/blob hash or "working directory version")
• ruby implementation (RUBY_DESCRIPTION)
• hardware

on_change:

• benchmark code: either merge or remove old data OR manage bench versions
• implementation code: should manage with impl code commit hash change (overwrite last if working dir version? (=> ctime?) )
• hardware change: should still allow comparison, but not mix results by default

Where to save?

• ~/.perfer: avoid generated files in project dir (persist if repo deleted, better for multi-repos)
• in project bench{,mark}/results (has to be git-ignored, but easy to find and keep around, always relative paths to bench{,mark} dir?)

Format:

• plain ruby Hashes, one per line (but lot of space wasted, no explicit results grouping)
• group them, and add a header before each set of results
• or inline results in Arrays (but grouping by result less clear, and never-ending-line)

Typical use cases:

• new user: a few benches, mostly last results, but then maybe compare a few versions (does not want too much setup like git and config files)
• CLI guru: do everything in "-e" (does not want any setup at all)
• ruby implementation dev: perf history across versions, commits
• library developer: keep benchmarks in bench/, run regularly, perf history