Skip to content

Commit e485ba9

Browse files
diegorussodiegorusso
authored
Add example scripts to run pyperformance on a generic host (#436)
1 parent cdbf33b commit e485ba9

File tree

5 files changed

+317
-0
lines changed

5 files changed

+317
-0
lines changed

‎examples/benchmarking-scripts/README.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Benchmarking Scripts Toolkit
2+
3+
Companion assets for running `pyperformance` benchmarks on hosts that provide isolated CPUs and for backfilling historical CPython revisions to [speed.python.org](https://speed.python.org/).
4+
5+
## Contents
6+
- `run-pyperformance.sh` – shell wrapper that reserves an isolated CPU (175–191) via lockfiles, renders `benchmark.conf` from `benchmark.conf.in` with `m4`, sets up a virtual environment, and runs `pyperformance` with upload enabled.
7+
- `benchmark.conf.in` – template consumed by the wrapper; placeholders `TMPDIR` and `CPUID` are filled in so each run has its own working tree, build directory, and CPU affinity.
8+
- `backfill.py` – Python helper that reads revisions from `backfill_shas.txt` and launches multiple `run-pyperformance.sh` jobs in parallel, capturing stdout/stderr per revision under `output/`.
9+
- `backfill_shas.txt` – example list of `sha=branch` pairs targeted by the backfill script.
10+
11+
## Typical Workflow
12+
1. Ensure kernel CPU isolation (`isolcpus=175-191`) and the `lockfile` utility are available so the wrapper can pin workloads without contention.
13+
2. Invoke `./run-pyperformance.sh -- compile benchmark.conf <sha> <branch>` for an ad-hoc run; the script installs `pyperformance==1.13.0`, clones CPython, and uploads results using the environment label configured in `benchmark.conf.in`.
14+
3. Populate `backfill_shas.txt` with the revisions you want to replay and run `python backfill.py` to batch process them; individual logs land in `output/<branch>-<sha>.out|.err`.
15+
16+
Adjust `benchmark.conf.in` if you need to change build parameters (PGO/LTO, job count, upload target, etc.).
17+
18+
## Scheduled Runs
19+
If you want a daily unattended run, drop an entry like this into `crontab -e` on the host:
20+
21+
```
22+
0 0 * * * cd /home/user/pyperformance/examples/benchmarking-scripts && ./run-pyperformance.sh -- compile_all benchmark.conf > /home/pyperf/pyperformance/cron.log 2>&1
23+
```

‎examples/benchmarking-scripts/backfill.py‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
import signal
2+
import subprocess
3+
from multiprocessing import Pool
4+
from pathlib import Path
5+
6+
"""
7+
Parallel backfilling helper for pyperformance runs on isolated CPUs.
8+
9+
Reads `sha=branch` pairs from backfill_shas.txt, invokes run-pyperformance.sh
10+
for each revision, and lets that wrapper pin the workload to an isolated CPU,
11+
materialize benchmark.conf, build CPython, and upload results to
12+
speed.python.org. Stdout/stderr for each revision are captured under
13+
output/<branch>-<sha>.(out|err).
14+
"""
15+
16+
17+
def get_revisions() -> tuple[str, str]:
18+
revisions = []
19+
with open("backfill_shas.txt", "r") as f:
20+
for line in f:
21+
sha, branch = line.split("=")
22+
revisions.append((sha, branch.rstrip()))
23+
return revisions
24+
25+
26+
def run_pyperformance(revision):
27+
sha, branch = revision
28+
print(f"Running run-pyperformance.sh with sha: {sha}, branch: {branch}")
29+
output_dir = Path("output")
30+
output_dir.mkdir(parents=True, exist_ok=True)
31+
out_file = output_dir / f"{branch}-{sha}.out"
32+
err_file = output_dir / f"{branch}-{sha}.err"
33+
with open(out_file, "w") as output, open(err_file, "w") as error:
34+
subprocess.run(
35+
[
36+
"./run-pyperformance.sh",
37+
"-x",
38+
"--",
39+
"compile",
40+
"benchmark.conf",
41+
sha,
42+
branch,
43+
],
44+
stdout=output,
45+
stderr=error,
46+
)
47+
48+
49+
if __name__ == "__main__":
50+
original_sigint_handler = signal.signal(signal.SIGINT, signal.SIG_IGN)
51+
signal.signal(signal.SIGINT, original_sigint_handler)
52+
with Pool(8) as pool:
53+
res = pool.map_async(run_pyperformance, get_revisions())
54+
# Without the timeout this blocking call ignores all signals.
55+
res.get(86400)

‎examples/benchmarking-scripts/backfill_shas.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
5d2edf72d25c2616f0e13d10646460a8e69344fa=main
2+
bd2c7e8c8b10f4d31eab971781de13844bcd07fe=main
3+
29b38b7aae884c14085a918282ea7f0798ed7a2a=main

‎examples/benchmarking-scripts/benchmark.conf.in‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
[config]
2+
# Directory where JSON files are written.
3+
# - uploaded files are moved to json_dir/uploaded/
4+
# - results of patched Python are written into json_dir/patch/
5+
json_dir = TMPDIR/json
6+
7+
# If True, compile CPython is debug mode (LTO and PGO disabled),
8+
# run benchmarks with --debug-single-sample, and disable upload.
9+
#
10+
# Use this option used to quickly test a configuration.
11+
debug = False
12+
13+
14+
[scm]
15+
# Directory of CPython source code (Git repository)
16+
repo_dir = TMPDIR/cpython
17+
18+
# Update the Git repository (git fetch)?
19+
update = True
20+
21+
# Name of the Git remote, used to create revision of
22+
# the Git branch. For example, use revision 'remotes/origin/3.6'
23+
# for the branch '3.6'.
24+
git_remote = remotes/origin
25+
26+
27+
[compile]
28+
# Create files into bench_dir:
29+
# - bench_dir/bench-xxx.log
30+
# - bench_dir/prefix/: where Python is installed
31+
# - bench_dir/venv/: Virtual environment used by pyperformance
32+
bench_dir = TMPDIR/bench_tmpdir
33+
34+
# Link Time Optimization (LTO)?
35+
lto = True
36+
37+
# Profiled Guided Optimization (PGO)?
38+
pgo = True
39+
40+
# The space-separated list of libraries that are package-only,
41+
# i.e., locally installed but not on header and library paths.
42+
# For each such library, determine the install path and add an
43+
# appropriate subpath to CFLAGS and LDFLAGS declarations passed
44+
# to configure. As an exception, the prefix for openssl, if that
45+
# library is present here, is passed via the --with-openssl
46+
# option. Currently, this only works with Homebrew on macOS.
47+
# If running on macOS with Homebrew, you probably want to use:
48+
# pkg_only = openssl readline sqlite3 xz zlib
49+
# The version of zlib shipping with macOS probably works as well,
50+
# as long as Apple's SDK headers are installed.
51+
pkg_only =
52+
53+
# Install Python? If false, run Python from the build directory
54+
#
55+
# WARNING: Running Python from the build directory introduces subtle changes
56+
# compared to running an installed Python. Moreover, creating a virtual
57+
# environment using a Python run from the build directory fails in many cases,
58+
# especially on Python older than 3.4. Only disable installation if you
59+
# really understand what you are doing!
60+
install = True
61+
62+
# Specify '-j' parameter in 'make' command
63+
jobs = 24
64+
65+
[run_benchmark]
66+
# Run "sudo python3 -m pyperf system tune" before running benchmarks?
67+
system_tune = False
68+
69+
# --manifest option for 'pyperformance run'
70+
manifest =
71+
72+
# --benchmarks option for 'pyperformance run'
73+
benchmarks =
74+
75+
# --affinity option for 'pyperf system tune' and 'pyperformance run'
76+
affinity = CPUID
77+
78+
# Upload generated JSON file?
79+
#
80+
# Upload is disabled on patched Python, in debug mode or if install is
81+
# disabled.
82+
upload = True
83+
84+
# Configuration to upload results to a Codespeed website
85+
[upload]
86+
url = https://speed.python.org/
87+
# environment-name should be created on speed.python.org
88+
environment = environment-name
89+
executable = lto-pgo
90+
project = CPython
91+
92+
[compile_all]
93+
# List of CPython Git branches
94+
branches = main
95+
96+
97+
# List of revisions to benchmark by compile_all
98+
[compile_all_revisions]
99+
# list of 'sha1=' (default branch: 'main') or 'sha1=branch'
100+
# used by the "pyperformance compile_all" command
101+
# e.g.:
102+
# 11159d2c9d6616497ef4cc62953a5c3cc8454afb =

‎examples/benchmarking-scripts/run-pyperformance.sh‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
#!/bin/bash
2+
3+
4+
# Wrapper around pyperformance for hosts with isolated CPUs. Reserves a CPU
5+
# (175-191) via lockfiles, renders benchmark.conf with m4, bootstraps a venv,
6+
# and runs pyperformance pinned to that CPU. Requires kernel isolcpus=175-191
7+
# and the lockfile utility so concurrent runs do not collide, which is
8+
# especially helpful when backfilling multiple revisions.
9+
10+
11+
set -e
12+
set -u
13+
set -o pipefail
14+
15+
lock_file=
16+
tmpdir=
17+
cleanup()
18+
{
19+
if [[ -n "${lock_file:-}" ]]; then
20+
echo "Removing $lock_file"
21+
rm -f "$lock_file"
22+
fi
23+
if [[ -n "${tmpdir:-}" ]]; then
24+
echo "Removing $tmpdir"
25+
rm -fr "$tmpdir"
26+
fi
27+
exit
28+
}
29+
30+
trap cleanup EXIT
31+
32+
usage()
33+
{
34+
cat <<EOF
35+
36+
usage: run-pyperformance.sh [OPTION]...
37+
38+
-h, --help
39+
print some basic usage information and exit
40+
-x
41+
enable tracing in this shells script
42+
43+
Note: if you want to pass arguments to pyperformance append "--" followed by the arguments.
44+
EOF
45+
}
46+
47+
args=$(getopt -o+hx -l help -n $(basename "$0") -- "$@")
48+
eval set -- "$args"
49+
while [ $# -gt 0 ]; do
50+
if [ -n "${opt_prev:-}" ]; then
51+
eval "$opt_prev=\$1"
52+
opt_prev=
53+
shift 1
54+
continue
55+
elif [ -n "${opt_append:-}" ]; then
56+
if [ -n "$1" ]; then
57+
eval "$opt_append=\"\${$opt_append:-} \$1\""
58+
fi
59+
opt_append=
60+
shift 1
61+
continue
62+
fi
63+
case $1 in
64+
-h | --help)
65+
usage
66+
exit 0
67+
;;
68+
69+
-x)
70+
set -x
71+
;;
72+
73+
--)
74+
shift
75+
break 2
76+
;;
77+
esac
78+
shift 1
79+
done
80+
81+
82+
# We have the latest 16 CPUs (ID 175-191) for running pyperformance and we want
83+
# to make sure that pyperformance runs with affinity on one of these CPUs.
84+
# In order to do that a locking mechanism is implemented in order to "reserve"
85+
# a CPU being used.
86+
# Locking files are in /var/lock/pyperformance-CPUID (where CPUID is 175, 176... 191)
87+
# Linux is booted with
88+
#
89+
# GRUB_CMDLINE_LINUX="isolcpus=175-191 mitigations=off"
90+
#
91+
# in the /etc/default/grub file
92+
lock_prefix_path="/var/lock/pyperformance-"
93+
94+
for i in $(seq 175 191); do
95+
lock_file="$lock_prefix_path$i"
96+
# lockfile is provided byt the `procmail` package
97+
if lockfile -r0 "$lock_file"; then
98+
# Let's save the CPUID to set the affinity later
99+
cpuid=$i
100+
break
101+
fi
102+
done
103+
104+
if [ -z ${cpuid+x} ]; then
105+
echo "Cannot find an available CPU to run pyperformance on. Exiting...";
106+
# Disable the trap as we don't need to clean up anything
107+
trap - EXIT
108+
exit 1
109+
fi
110+
111+
# Create a temporary directory
112+
tmpdir=$(mktemp -d -t pyperformance.XXXXXXXXX)
113+
114+
echo "Pyperformance will be run on CPU $cpuid"
115+
echo "Working directory is $tmpdir"
116+
117+
# Snapshot the benchmark.conf file
118+
m4 \
119+
-DTMPDIR="$tmpdir" \
120+
-DCPUID="$cpuid" \
121+
benchmark.conf.in > "$tmpdir/benchmark.conf"
122+
123+
# This is our working directory from now on
124+
cd "$tmpdir"
125+
126+
# Install pyperformance in a virtual env.
127+
python3 -m venv venv
128+
venv/bin/pip install pyperformance==1.13.0
129+
130+
# Clone cpython
131+
git clone https://github.com/python/cpython.git
132+
133+
# Run pyperformance
134+
venv/bin/pyperformance "$@"

0 commit comments

Comments
 (0)