Skip to content

Commit 5cfd1d5

Browse files
committed
HBASE-23779 Up the default fork count to make builds complete faster; make count relative to CPU count
Halve the build speed by doubling the -C count from 0.25C to 0.5C. Pass mvn a -T of 0.5C too.
1 parent 611c62f commit 5cfd1d5

File tree

2 files changed

+16
-6
lines changed

2 files changed

+16
-6
lines changed

dev-support/hbase-personality.sh

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -140,7 +140,15 @@ function personality_modules
140140

141141
clear_personality_queue
142142

143-
extra="-DHBasePatchProcess"
143+
# Set a fork count based off the host cpu count. Pass maven a -T argument.
144+
# Default -T is one thread. 0.5C on an apache box of 16 cores and 2 jenkins
145+
# 'executors' per host should make for 8 threads. Setting this here for yetus
146+
# to pick up. See
147+
# https://yetus.apache.org/documentation/0.11.1/precommit-advanced/#global-definitions
148+
# See below for more on -T:
149+
# https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
150+
forkcount="0.5C"
151+
extra="-T${forkcount} -Dsurefire.firstPartForkCount=${forkcount} -Dsurefire.secondPartForkCount=${forkcount} -DHBasePatchProcess"
144152
if [[ "${PATCH_BRANCH}" = branch-1* ]]; then
145153
extra="${extra} -Dhttps.protocols=TLSv1.2"
146154
fi

pom.xml

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1413,13 +1413,15 @@
14131413
by what apache jenkins nightly builds will tolerate (See HBASE-24072). Up this
14141414
value is you want to burn through tests faster (could make for more failures
14151415
if more contention around resources). There is a matching MAVEN_ARG
1416-
in our yetus personality where we set the maven -T command to 0.25C too.
1417-
For example, to run at a rate that is more furious than our 0.25C, do
1416+
in our yetus personality where we set the maven -T command to 0.5C too.
1417+
For example, to run at a rate that is more furious than our 0.5C, do
14181418
something like this:
1419-
f="0.5C" ; mvn -T$f -Dsurefire.firstPartForkCount=$f -Dsurefire.secondPartForkCount=$f test -PrunAllTests
1419+
f="0.75C" ; mvn -T$f -Dsurefire.firstPartForkCount=$f -Dsurefire.secondPartForkCount=$f test -PrunAllTests
1420+
Note, tests seem to crash or fail nonsensically if the fork count is set too
1421+
high. On high-CPU boxes, a fork count of 1.0 doesn't always 'work'.
14201422
-->
1421-
<surefire.firstPartForkCount>0.25C</surefire.firstPartForkCount>
1422-
<surefire.secondPartForkCount>0.25C</surefire.secondPartForkCount>
1423+
<surefire.firstPartForkCount>0.5C</surefire.firstPartForkCount>
1424+
<surefire.secondPartForkCount>0.5C</surefire.secondPartForkCount>
14231425
<surefire.firstPartGroups>org.apache.hadoop.hbase.testclassification.SmallTests</surefire.firstPartGroups>
14241426
<surefire.secondPartGroups>org.apache.hadoop.hbase.testclassification.MediumTests</surefire.secondPartGroups>
14251427
<surefire.testFailureIgnore>false</surefire.testFailureIgnore>

0 commit comments

Comments
 (0)