Basic Instrumentation

[terhorstd]

Is your feature request related to a problem? Please describe.

This requested feature solves two problems. First, in the research on simulation technology carried out with NEST and the continuous improvement of NEST it is necessary to measure execution times of different phases of the simulation. Across projects it is difficult to communicate how exactly these times are determined and every project has to re-implement the instrumentation. This is inefficient and limits reproducibility. Second, the NEST team regularly receives support requests where NEST is reported to be slow. The timing data provided by the client will speed determination of the origin of the problem.

Interface present status

At present GetKernelStatus gives the following timing information:

• time
• time_collocate
• time_communicate

These variables are no longer filled consistently in the present master.

Interface desired situation

Proposal of different status dictionary entry entries providing timing information:

• biological_time : biological time simulated since the last ResetKernel (This is not based on stopwatches but needed to calculate the real-time factor)

These basic timers do not overlap and should usually not impact performance:

• time_construction : time for constructing the network in the nest kernel, cumulative build time for neurons and connections
• time_communicate_prepare : setup of presynaptic connection infrastructure (before actual simulation)
• time_simulate : total time of simulation without construction

These detailed timers contribute to time_simulate:

• time_communicate : time for communicating spikes between compute nodes
• time_update : time for neuron update
• time_deliver : time to deliver events from the MPI recv buffers to their
  local synaptic targets (including synaptic update, e.g. STDP synapses) and to
  the spike ring buffers of the corresponding postsynaptic neurons
• time_collocate : time to collocate MPI send buffer from spike register

These timers, except the first cumulating the biological time, should be reset at the start of the program/ResetKernel (for network construction) and Simulate (for timers concerned with dynamics). The timers that may negatively impact performance (e.g. time_communicate and time_deliver) should only be active if the compiler flag is set.

The proposed name of the compiler option is --with-basic-timers (may become default) and --with-detailed-timers.

Implementation

Use the POSIX timing commands already built into NEST which are the basis of SLI tic and toc commands and the existing StopWatch class. Document the compiler options and the interface in the user-level documentation.

Reasons

0. increase reproducibility of timing measurements, ease of use
1. In 5g part of network construction is happening at the first call of simulate(). The time of this needs to be considered as network construction time not state propagation time. The time can easily be measured by timing a call of Simulate(0.0) from the Python level, but one needs to think about it and it requires explanation
2. Timing a Simulate(T) call from the Python level is trivially possible but this always includes the time of reloading Python into the caches. This is in particular relevant and visible if all cores on a compute node are participating in a simulation. The perturbation can be overcome by using large T, but a perturbation will remain and the problem can easily be overcome as described above.

Alternatives

1. no action, leave as is
2. implement but use the timers of the boost library which in theory give better portability as a language feature of C++ and better resolution (but in the last years POSIX was not a problem). Risk is that boost may initially not be available on supercomputers (but risk seems to shrink and the dependence of NEST on boost is increasing anyway.

Limitations

1. default instrumentation is not suitable for all timers. For example, communication consists of 10,000 of communication calls which require high-resolution measurements and the measurement influences performance. Therefore some timers require a non-default compiler option.
2. instrumentation only covers timing not memory consumption

[diesmann]

I sent comments to @terhorstd for updating the text of the issue

[terhorstd]

Updated the description according to suggested changes by @diesmann.