Skip to content

Commit 17ce605

Browse files
committed
[SPARK-32556][INFRA] Fix release script to have urlencoded passwords where required
### What changes were proposed in this pull request? 1. URL encode the `ASF_PASSWORD` of the release manager. 2. Update the image to install `qpdf` and `jq` dep 3. Increase the JVM HEAM memory for maven build. ### Why are the changes needed? Release script takes hours to run, and if a single failure happens about somewhere midway, then either one has to get down to manually doing stuff or re run the entire script. (This is my understanding) So, I have made the fixes of a few failures, discovered so far. 1. If the release manager password contains a char, that is not allowed in URL, then it fails the build at the clone spark step. `git clone "https://$ASF_USERNAME:$ASF_PASSWORD$ASF_SPARK_REPO" -b $GIT_BRANCH` ^^^ Fails with bad URL `ASF_USERNAME` may not be URL encoded, but we need to encode `ASF_PASSWORD`. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? By running the release for branch-2.4, using both type of passwords, i.e. passwords with special chars and without it. Closes #29373 from ScrapCodes/release-script-fix2. Lead-authored-by: Prashant Sharma <[email protected]> Co-authored-by: Prashant Sharma <[email protected]> Signed-off-by: Prashant Sharma <[email protected]> (cherry picked from commit 6c3d0a4) Signed-off-by: Prashant Sharma <[email protected]>
1 parent 4d8642f commit 17ce605

File tree

3 files changed

+8
-2
lines changed

3 files changed

+8
-2
lines changed

dev/create-release/release-tag.sh

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,8 +64,12 @@ init_maven_sbt
6464

6565
ASF_SPARK_REPO="gitbox.apache.org/repos/asf/spark.git"
6666

67+
function uriencode { jq -nSRr --arg v "$1" '$v|@uri'; }
68+
69+
declare -r ENCODED_ASF_PASSWORD=$(uriencode "$ASF_PASSWORD")
70+
6771
rm -rf spark
68-
git clone "https://$ASF_USERNAME:$ASF_PASSWORD@$ASF_SPARK_REPO" -b $GIT_BRANCH
72+
git clone "https://$ASF_USERNAME:$ENCODED_ASF_PASSWORD@$ASF_SPARK_REPO" -b $GIT_BRANCH
6973
cd spark
7074

7175
git config user.name "$GIT_NAME"

dev/create-release/release-util.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -229,7 +229,7 @@ function init_maven_sbt {
229229
if [[ $JAVA_VERSION < "1.8." ]]; then
230230
# Needed for maven central when using Java 7.
231231
SBT_OPTS="-Dhttps.protocols=TLSv1.1,TLSv1.2"
232-
MVN_EXTRA_OPTS="-Dhttps.protocols=TLSv1.1,TLSv1.2"
232+
MVN_EXTRA_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dhttps.protocols=TLSv1.1,TLSv1.2"
233233
MVN="$MVN $MVN_EXTRA_OPTS"
234234
fi
235235
export MVN MVN_EXTRA_OPTS SBT_OPTS

dev/create-release/spark-rm/Dockerfile

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,8 @@ RUN apt-get clean && apt-get update && $APT_INSTALL gnupg ca-certificates && \
6363
$APT_INSTALL nodejs && \
6464
# Install needed python packages. Use pip for installing packages (for consistency).
6565
$APT_INSTALL libpython3-dev python3-pip python3-setuptools && \
66+
# qpdf is required for CRAN checks to pass.
67+
$APT_INSTALL qpdf jq && \
6668
# Change default python version to python3.
6769
update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1 && \
6870
update-alternatives --install /usr/bin/python python /usr/bin/python3.6 2 && \

0 commit comments

Comments
 (0)