Skip to content

Commit c34c422

Browse files
gengliangwangsrowen
authored andcommitted
[SPARK-26076][BUILD][MINOR] Revise ambiguous error message from load-spark-env.sh
## What changes were proposed in this pull request? When I try to run scripts (e.g. `start-master.sh`/`start-history-server.sh ` in latest master, I got such error: ``` Presence of build for multiple Scala versions detected. Either clean one of them or, export SPARK_SCALA_VERSION in spark-env.sh. ``` The error message is quite confusing. Without reading `load-spark-env.sh`, I didn't know which directory to remove, or where to find and edit the `spark-evn.sh`. This PR is to make the error message more clear. Also change the script for less maintenance when we add or drop Scala versions in the future. As now with #22967, we can revise the error message as following(in my local setup): ``` Presence of build for multiple Scala versions detected (/Users/gengliangwang/IdeaProjects/spark/assembly/target/scala-2.12 and /Users/gengliangwang/IdeaProjects/spark/assembly/target/scala-2.11). Remove one of them or, export SPARK_SCALA_VERSION=2.12 in /Users/gengliangwang/IdeaProjects/spark/conf/spark-env.sh. Visit https://spark.apache.org/docs/latest/configuration.html#environment-variables for more details about setting environment variables in spark-env.sh. ``` ## How was this patch tested? Manual test Closes #23049 from gengliangwang/reviseEnvScript. Authored-by: Gengliang Wang <[email protected]> Signed-off-by: Sean Owen <[email protected]>
1 parent a00aaf6 commit c34c422

File tree

1 file changed

+16
-11
lines changed

1 file changed

+16
-11
lines changed

bin/load-spark-env.sh

Lines changed: 16 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -26,35 +26,40 @@ if [ -z "${SPARK_HOME}" ]; then
2626
source "$(dirname "$0")"/find-spark-home
2727
fi
2828

29+
SPARK_ENV_SH="spark-env.sh"
2930
if [ -z "$SPARK_ENV_LOADED" ]; then
3031
export SPARK_ENV_LOADED=1
3132

3233
export SPARK_CONF_DIR="${SPARK_CONF_DIR:-"${SPARK_HOME}"/conf}"
3334

34-
if [ -f "${SPARK_CONF_DIR}/spark-env.sh" ]; then
35+
SPARK_ENV_SH="${SPARK_CONF_DIR}/${SPARK_ENV_SH}"
36+
if [[ -f "${SPARK_ENV_SH}" ]]; then
3537
# Promote all variable declarations to environment (exported) variables
3638
set -a
37-
. "${SPARK_CONF_DIR}/spark-env.sh"
39+
. ${SPARK_ENV_SH}
3840
set +a
3941
fi
4042
fi
4143

4244
# Setting SPARK_SCALA_VERSION if not already set.
4345

4446
if [ -z "$SPARK_SCALA_VERSION" ]; then
47+
SCALA_VERSION_1=2.12
48+
SCALA_VERSION_2=2.11
4549

46-
ASSEMBLY_DIR2="${SPARK_HOME}/assembly/target/scala-2.11"
47-
ASSEMBLY_DIR1="${SPARK_HOME}/assembly/target/scala-2.12"
48-
49-
if [[ -d "$ASSEMBLY_DIR2" && -d "$ASSEMBLY_DIR1" ]]; then
50-
echo -e "Presence of build for multiple Scala versions detected." 1>&2
51-
echo -e 'Either clean one of them or, export SPARK_SCALA_VERSION in spark-env.sh.' 1>&2
50+
ASSEMBLY_DIR_1="${SPARK_HOME}/assembly/target/scala-${SCALA_VERSION_1}"
51+
ASSEMBLY_DIR_2="${SPARK_HOME}/assembly/target/scala-${SCALA_VERSION_2}"
52+
ENV_VARIABLE_DOC="https://spark.apache.org/docs/latest/configuration.html#environment-variables"
53+
if [[ -d "$ASSEMBLY_DIR_1" && -d "$ASSEMBLY_DIR_2" ]]; then
54+
echo "Presence of build for multiple Scala versions detected ($ASSEMBLY_DIR_1 and $ASSEMBLY_DIR_2)." 1>&2
55+
echo "Remove one of them or, export SPARK_SCALA_VERSION=$SCALA_VERSION_1 in ${SPARK_ENV_SH}." 1>&2
56+
echo "Visit ${ENV_VARIABLE_DOC} for more details about setting environment variables in spark-env.sh." 1>&2
5257
exit 1
5358
fi
5459

55-
if [ -d "$ASSEMBLY_DIR2" ]; then
56-
export SPARK_SCALA_VERSION="2.11"
60+
if [[ -d "$ASSEMBLY_DIR_1" ]]; then
61+
export SPARK_SCALA_VERSION=${SCALA_VERSION_1}
5762
else
58-
export SPARK_SCALA_VERSION="2.12"
63+
export SPARK_SCALA_VERSION=${SCALA_VERSION_2}
5964
fi
6065
fi

0 commit comments

Comments
 (0)