[CPP-62][CPP-69]Frontend Benchmark CPU Usage - Windows self hosted runner.

[john-michaelburke]

Details

• Added a benchmark for validating the frontend using hyperfine on a self hosted runner (Local VM will eventually set up permanent).
• Switched naming convention for produced binaries from "Swift Navigation Console" to "swift_navigation_console", as some issues with running via subprocess causing issues for hyperfine catching all additional arguments.
• Added a few extra glob patterns to the glob used for finding python files, also added a list for blacklisting patterns.
• The CI for the frontend benchmark runs after the "Building Binaries" job. It relies on a zip created in the ci-build.sh script for each OS type which basically throws everything needed to run the binary into the zip + Makefile.toml + benchmark folder.
• Fixed the git lfs patterns.
• Added a command line flag to automatically connect once the console has opened up. If a file has been passed in it will be read when console opens otherwise connect to hardcoded remote piksi.
• Another feature which may need to be rethought, when a file is read once the file has been completely read the console will close. This is necessary for the action of the frontend benchmark.
• The frontend benchmark is fairly similar to the new backend benchmark but follows the pattern used in libsbp running hyperfine on the console executable from a python script, then reads the local file generated for expected means within a tolerance.

Self Hosted Runner Setup

Currently tested only on Windows.
Assumes the host has these dependencies preinstalled:

• hyperfine
• bash
• cargo make
• 7z
• python3 (only need std libs)

Before attempting to run the benchmark an "rm -rf" is performed on the working folder to remove any trace from previous runs.

[john-michaelburke]

Fixes https://swift-nav.atlassian.net/browse/CPP-69

[silverjam]

LGTM - silly question though, if this is running in the VM on your laptop... should we wait to setup the self-hosted runner in the lab before merging? Or merge but make passing optional?

[john-michaelburke]

LGTM - silly question though, if this is running in the VM on your laptop... should we wait to setup the self-hosted runner in the lab before merging? Or merge but make passing optional?

Very good point. I added a continue-on-error to the job as well as a timeout. I also turned off the runner in my VM to see the behavior in CI. Interestingly enough it seems that when the runner was not available the CI just waits until one becomes available; however, I think it would fail after the default timeout of 360min. This is actually why I added the timeout as I think it will just wait if no runner is online.

UPDATE It appears GA just indefinitely waits for a self hosted runner to become available. Seeing as how this does not incur a charge AFAIK and this is at the end of all stages this should be okay until we set up a more permanent solution? @silverjam

[silverjam]

LGTM - silly question though, if this is running in the VM on your laptop... should we wait to setup the self-hosted runner in the lab before merging? Or merge but make passing optional?

Very good point. I added a continue-on-error to the job as well as a timeout. I also turned off the runner in my VM to see the behavior in CI. Interestingly enough it seems that when the runner was not available the CI just waits until one becomes available; however, I think it would fail after the default timeout of 360min. This is actually why I added the timeout as I think it will just wait if no runner is online.

UPDATE It appears GA just indefinitely waits for a self hosted runner to become available. Seeing as how this does not incur a charge AFAIK and this is at the end of all stages this should be okay until we set up a more permanent solution? @silverjam

Should be fine for now, let's try to get this resolved this week though.

[silverjam]

Is 2 minutes a realistic timeout?

[john-michaelburke]

The benchmark runs in under a minute now. We can increase it as we add more benchmarks.