[SPARK-40833][K8S] Cleanup apt lists cache

[Yikun]

What changes were proposed in this pull request?

Remove unused apt lists cache

Why are the changes needed?

Clean cache to reduce docker image size.

This is also recommanded by docker community:

$ docker run --user 0:0 -ti apache/spark bash
root@5d1ca347279e:/opt/spark/work-dir# ls /var/lib/apt/lists/
auxfiles								     lock
deb.debian.org_debian_dists_bullseye-updates_InRelease			     partial
deb.debian.org_debian_dists_bullseye-updates_main_binary-arm64_Packages.lz4  security.debian.org_debian-security_dists_bullseye-security_InRelease
deb.debian.org_debian_dists_bullseye_InRelease				     security.debian.org_debian-security_dists_bullseye-security_main_binary-arm64_Packages.lz4
deb.debian.org_debian_dists_bullseye_main_binary-arm64_Packages.lz4
root@5d1ca347279e:/opt/spark/work-dir# du --max-depth=1 -h /var/lib/apt/lists/
4.0K	/var/lib/apt/lists/partial
4.0K	/var/lib/apt/lists/auxfiles
17M	/var/lib/apt/lists/

Does this PR introduce any user-facing change?

Yes in some level, image size is reduced.

How was this patch tested?

K8s CI passed

[Yikun]

cc @dongjoon-hyun

[dongjoon-hyun]

+1, LGTM. Thank you, @Yikun .
Merged to master for Apache Spark 3.4.0.

[dongjoon-hyun]

Ur, @Yikun . Why do you use a different style per repo?

• https://github.com/apache/spark-docker/pull/14/files

- rm -rf /var/cache/apt/*
+ rm -rf /var/cache/apt/* && \
+ rm -rf /var/lib/apt/lists/*

• This PR

- rm -rf /var/cache/apt/*
+ rm -rf /var/cache/apt/* && rm -rf /var/lib/apt/lists/*

[Yikun]

@dongjoon-hyun After spark-docker PR merged, I found the Python docker file are using one line style, so keep the same style in spark repo.

I'd also like to sync this style to spark-docker. : ), from functions view it's same, put similar function cmd (rm in here) in one line is better.

[Yikun]

BTW, what do you think about the dockerfile being in sync between spark-docker and spark?

spark-docker contains some changes for non-k8s to meet the requirements of DOI. such as, remove print line in entrypoint, add users, expose port. Do we want to also sync them in spark repo?

[dongjoon-hyun]

We don't need to sync~ If we really want to sync, we had better make spark-docker git repo as a submodule of spark repo.

[Yikun]

@dongjoon-hyun OK, thanks, let's just keep different for now! If a submodule is really needed, we can do it in future.

[dongjoon-hyun]

Ya, feel free to propose any changes if it helps you and don't be blocked by me. You are now a committer with more power, @Yikun . :)