Skip to content

Commit 848cab1

Browse files
richamishra006Richa MishraNishant BarolaHadesArchitectmichaelsembwever
committed
K8s immutable provisioning of ci-cassandra.apache.org jenkins instances
.build/run-ci is a a python script used to create and interact with k8s provisioned ci-cassandra.apache.org clones See .build/run-ci.d/README.md for docs on usage. .jenkins/k8s/ contains the k8s jenkins helm chart, JSaC configuration, and docker image for provisioning. .build/run-ci.d/ contains python requirements and tests for .build/run-ci patch by Richa Mishra, Nishant Barola, Aleksandr Volochnev, Mick Semb Wever; reviewed by Richa Mishra, Nishant Barola, Aleksandr Volochnev, Brandon Hsieh, Mick Semb Wever, Brandon Williams for CASSANDRA-18145 Co-authored-by: Richa Mishra <[email protected]> Co-authored-by: Nishant Barola <[email protected]> Co-authored-by: Aleksandr Volochnev <[email protected]> Co-authored-by: Mick Semb Wever <[email protected]>
1 parent 4226a77 commit 848cab1

16 files changed

+1568
-12
lines changed

.build/ci/generate-ci-summary.sh

Lines changed: 19 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -44,11 +44,25 @@ cat >${DIST_DIR}/ci_summary.html <<EOL
4444
<html>
4545
<head></head>
4646
<body>
47-
<h1>CI Summary</h1>
48-
<h2>sha: $(git ls-files -s ${CASSANDRA_DIR} | git hash-object --stdin)</h2>
49-
<h2>branch: $(git -C ${CASSANDRA_DIR} branch --remote --verbose --no-abbrev --contains | sed -rne 's/^[^\/]*\/([^\ ]+).*$/\1/p')</h2>
50-
<h2>repo: $(git -C ${CASSANDRA_DIR} remote get-url origin)</h2>
51-
<h2>Date: $(date)</h2>
47+
<h1>CI Summary ${BUILD_TAG}</h1>
48+
<h2>Build State</h2>
49+
<ul>
50+
<li>sha: $(git ls-files -s ${CASSANDRA_DIR} | git hash-object --stdin)</li>
51+
<li>repo: $(git -C ${CASSANDRA_DIR} remote get-url origin)</li>
52+
<li>branch: $(git -C ${CASSANDRA_DIR} branch --remote --verbose --no-abbrev --contains | sed -rne 's/^[^\/]*\/([^\ ]+).*$/\1/p')</li>
53+
<li>date: $(date)</li>
54+
</ul>
55+
<h2>Build Parameters</h2>
56+
<ul>
57+
<li>repository: ${REPOSITORY}</li>
58+
<li>branch: ${BRANCH}</li>
59+
<li>profile: ${PROFILE}</li>
60+
<li>profile_custom_regexp: ${PROFILE_CUSTOM_REGEXP}</li>
61+
<li>architecture: ${ARCHITECTURE}</li>
62+
<li>jdk: ${JDK}</li>
63+
<li>dtest_repository: {DTEST_REPOSITORY}</li>
64+
<li>dtest_branch: ${DTEST_BRANCH}</li>
65+
</ul>
5266
</body>
5367
</html>
5468
...

.build/ci/requirements.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,4 +18,4 @@
1818
# Changes to this file must also be put into
1919

2020
beautifulsoup4==4.12.3
21-
jinja2==3.1.3
21+
jinja2==3.1.5

.build/docker/_set_java.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,4 +62,4 @@ else
6262
sudo alternatives --set javac $(alternatives --display javac | grep "family java-${java_version}-openjdk" | cut -d' ' -f1)
6363
fi
6464
export JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:/bin/javac::")
65-
echo "Using Java ${java_version}"
65+
echo "Using Java ${java_version}"

.build/docker/run-tests.sh

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -204,7 +204,7 @@ case ${test_target/-repeat/} in
204204
[[ ${mem} -gt $((5 * 1024 * 1024 * 1024 * ${jenkins_executors})) ]] || { error 1 "${target} require minimum docker memory 6g (per jenkins executor (${jenkins_executors})), found ${mem}"; }
205205
;;
206206
*)
207-
error 1 "unrecognized test type \"${target}\""
207+
error 1 "unrecognized test type \"${target}\""
208208
;;
209209
esac
210210

@@ -305,8 +305,8 @@ docker exec --user cassandra ${container_name} bash -c "${docker_command}" | tee
305305
status=$?
306306
set +o pipefail
307307

308-
if [ "$status" -ne 0 ] ; then
309-
echo "${docker_id} failed (${status}), debug…"
308+
if [ "$status" -ne 0 ] && [ -z $SKIP_DOCKER_DEBUG_ON_FAIL ] ; then
309+
echo "${docker_id} failed (${status}), debug… (set SKIP_DOCKER_DEBUG_ON_FAIL to quiet)"
310310
docker inspect ${docker_id}
311311
echo "–––"
312312
docker logs ${docker_id}

.build/run-ci

Lines changed: 837 additions & 0 deletions
Large diffs are not rendered by default.

.build/run-ci.d/README.md

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# Help for `.build/run-ci`
2+
3+
```
4+
➤ .build/run-ci --help
5+
usage: run-ci [-h] [-c KUBECONFIG] [-x KUBECONTEXT] [-i URL] [-u USER] [-r REPOSITORY] [-b BRANCH] [-p {packaging,skinny,pre-commit,pre-commit w/ upgrades,post-commit,custom}] [-e PROFILE_CUSTOM_REGEXP] [-j JDK] [-d DTEST_REPOSITORY] [-k DTEST_BRANCH]
6+
[-s] [--only-setup] [--tear-down] [--only-tear-down] [--only-node-cleaner] [-o DOWNLOAD_RESULTS]
7+
8+
Run CI pipeline for Cassandra on K8s using Jenkins.
9+
10+
options:
11+
-h, --help show this help message and exit
12+
-c KUBECONFIG, --kubeconfig KUBECONFIG
13+
Path to a different kubeconfig.
14+
-x KUBECONTEXT, --kubecontext KUBECONTEXT
15+
Use a different Kubernetes context.
16+
-i URL, --url URL Jenkins url. Suitable when kubectl access in not available. Can also be specified via the JENKINS_URL environment variable (and in .build/.run-ci.env)
17+
-u USER, --user USER Jenkins user. Can also be specified via the JENKINS_USER environment variable (and in .build/.run-ci.env)
18+
-r REPOSITORY, --repository REPOSITORY
19+
Repository URL. Defaults to current tracking remote.
20+
-b BRANCH, --branch BRANCH
21+
Repository branch. Defaults to current branch.
22+
-p {packaging,skinny,pre-commit,pre-commit w/ upgrades,post-commit,custom}, --profile {packaging,skinny,pre-commit,pre-commit w/ upgrades,post-commit,custom}
23+
CI pipeline profile. Defaults to skinny.
24+
-e PROFILE_CUSTOM_REGEXP, --profile-custom-regexp PROFILE_CUSTOM_REGEXP
25+
Regexp for stages when using custom profile. See `testSteps` in Jenkinsfile for list of stages. Example: 'stress.*|jvm-dtest.'
26+
-j JDK, --jdk JDK Specify JDK version. Defaults to all JDKs the current branch supports.
27+
-d DTEST_REPOSITORY, --dtest-repository DTEST_REPOSITORY
28+
DTest repository URL.
29+
-k DTEST_BRANCH, --dtest-branch DTEST_BRANCH
30+
DTest repository branch.
31+
-s, --setup Set up Jenkins before the build.
32+
--only-setup Only install Jenkins into the k8s cluster.
33+
--tear-down Tear down Jenkins after the build.
34+
--only-tear-down Only tear down Jenkins.
35+
--only-node-cleaner Only run the node cleaner. The node cleaner scans the k8s nodes, eagerly terminating those unused.
36+
-o DOWNLOAD_RESULTS, --download-results DOWNLOAD_RESULTS
37+
Just download the results for the specificed build number. Naming of local artefacts assumes current tracking remote and branch, use -r and -b otherwise.
38+
```
39+
40+
## Examples
41+
Run the current directory's fork and branch through the default "pre-commit" pipeline, connecting via your default kubeconfig
42+
```
43+
.build/run-ci
44+
```
45+
46+
Do the same but connecting via a jenkins url
47+
```
48+
.build/run-ci --url pre-ci.cassandra.apache.org --user myuser
49+
```
50+
51+
Run the the specified fork and branch through the "skinny" pipeline restricted to tests on jdk11
52+
```
53+
.build/run-ci -r "https://github.com/jrwest/cassandra.git" -b "jwest/15452-5.0" -p "skinny" -j 11
54+
```
55+
56+
Run the the specified fork and branch through just the "fqltool-test" tests
57+
```
58+
.build/run-ci -r "https://github.com/jrwest/cassandra.git" -b "jwest/15452-5.0" -p "custom" -e "fqltool-test"
59+
```
60+
61+
Setup/Update Jenkins Helm into your current kubeconfig
62+
```
63+
.build/run-ci --only-setup
64+
```
65+
66+
Uninstall Jenkins from your current kubeconfig
67+
```
68+
.build/run-ci --only-tear-down
69+
```

.build/run-ci.d/requirements.txt

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
bs4
2+
dotenv
3+
kubernetes
4+
python-jenkins
5+
requests
6+
7+
# optional for different clouds
8+
boto3
9+
google-cloud-compute

.build/run-ci.d/run-ci-test.py

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
#!/usr/bin/env python
2+
# -*- coding: utf-8 -*-
3+
#
4+
# Licensed to the Apache Software Foundation (ASF) under one
5+
# or more contributor license agreements. See the NOTICE file
6+
# distributed with this work for additional information
7+
# regarding copyright ownership. The ASF licenses this file
8+
# to you under the Apache License, Version 2.0 (the
9+
# "License"); you may not use this file except in compliance
10+
# with the License. You may obtain a copy of the License at
11+
#
12+
# http://www.apache.org/licenses/LICENSE-2.0
13+
#
14+
# Unless required by applicable law or agreed to in writing, software
15+
# distributed under the License is distributed on an "AS IS" BASIS,
16+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
17+
# See the License for the specific language governing permissions and
18+
# limitations under the License.
19+
#
20+
# Used to test `.build/run-ci`
21+
# Run with `python .build/run-ci.d/run-ci-test.py`
22+
#
23+
#
24+
# lint with:
25+
# `pylint --disable=C0301,W0511,C0114,C0103,W0702,C0415,C0116,C0115,R0914,W0603,R0915,R0913,R0911 run-ci-test.py`
26+
27+
28+
import argparse
29+
from pathlib import Path
30+
import unittest
31+
from unittest.mock import patch, MagicMock
32+
33+
34+
# Import the functions from the script
35+
from run_ci import (
36+
debug,
37+
install_jenkins,
38+
get_jenkins,
39+
trigger_jenkins_build,
40+
spin_while,
41+
delete_remote_junit_files,
42+
cleanup_and_maybe_teardown,
43+
helm_installation_lock,
44+
)
45+
46+
class TestCIPipeline(unittest.TestCase):
47+
48+
def setUp(self):
49+
print("\ntesting ", self._testMethodName)
50+
51+
@patch('run_ci.os.environ.get')
52+
@patch('run_ci.print')
53+
def test_debug(self, mock_print, mock_get):
54+
mock_get.return_value = "1"
55+
debug("Test message")
56+
mock_print.assert_called_with("Test message")
57+
58+
@patch('run_ci.subprocess.run')
59+
def test_install_jenkins(self, mock_run):
60+
mock_run.return_value = MagicMock(returncode=0)
61+
install_jenkins("test-namespace", Path("/fake/cassandra/dir"), "default")
62+
mock_run.assert_any_call(["helm", "repo", "add", "jenkins", "https://charts.jenkins.io"], check=True)
63+
mock_run.assert_any_call(["helm", "repo", "update"], check=True)
64+
65+
@patch('run_ci.subprocess.run')
66+
@patch('run_ci.jenkins.Jenkins')
67+
def test_get_jenkins(self, mock_jenkins, mock_run):
68+
mock_k8s_client = MagicMock()
69+
mock_run.return_value = MagicMock(stdout="fake-password")
70+
mock_jenkins_instance = MagicMock()
71+
mock_jenkins.return_value = mock_jenkins_instance
72+
# hack – use False values instead of None
73+
args = argparse.Namespace(kubeconfig="/fake/kubeconfig", kubecontext="test-context", user=False, url=False)
74+
_, server = get_jenkins(mock_k8s_client, args, "default")
75+
self.assertEqual(server, mock_jenkins_instance)
76+
77+
@patch('run_ci.jenkins.Jenkins.build_job')
78+
@patch('run_ci.wait_for_build_number')
79+
def test_trigger_jenkins_build(self, mock_wait_for_build_number, mock_build_job):
80+
mock_server = MagicMock()
81+
mock_build_job.return_value = mock_server.build_job.return_value = 123
82+
mock_wait_for_build_number.return_value = 456
83+
with patch('run_ci.spin_while', side_effect=lambda msg, condition: 0):
84+
queue_item = trigger_jenkins_build(mock_server, "test-job", param1="value1")
85+
self.assertEqual(queue_item, 123)
86+
87+
def test_spin_while(self):
88+
result = spin_while("Testing", lambda: True)
89+
self.assertEqual(result, 0)
90+
91+
@patch('run_ci.stream.stream')
92+
def test_delete_remote_junit_files(self, mock_stream):
93+
mock_k8s_client = MagicMock()
94+
delete_remote_junit_files(mock_k8s_client, "test-pod", "test-namespace", 456)
95+
mock_stream.assert_called()
96+
97+
@patch('run_ci.subprocess.run')
98+
def test_cleanup_and_maybe_teardown(self, mock_run):
99+
cleanup_and_maybe_teardown(None, None, "test-namespace", True)
100+
mock_run.assert_called_with(["helm", "--namespace", "test-namespace", "uninstall", "cassius"], check=True)
101+
102+
@patch('run_ci.fcntl.flock')
103+
def test_helm_installation_lock(self, mock_flock):
104+
with helm_installation_lock(Path("/tmp/.fake.lock")):
105+
mock_flock.assert_called()
106+
107+
if __name__ == '__main__':
108+
unittest.main()

.build/run-ci.d/run_ci.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../run-ci

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
.gitignore
22

33
# C*
4+
.build/.run-ci.env
45
build
56
build/
67
src/gen-java/

0 commit comments

Comments
 (0)