[SPARK-12972][CORE] update httpclient to 4.5.1

[blbradley]

looking to fix some dependency issues

[srowen]

My major question is simply, does this break anything? the problem is that not all transitive dependencies use this version of the httpclient. Although I suspect it would be OK, assuming 4.5 is backwards-compatible with 4.3, and tests would help reveal problems (and you're going to have to update the dependencies file in the repo anyway to reflect the change), then this is substantially OK.

Is there a particular fix we need in 4.5?

One good thing to do is examine the current set of dependencies and evaluate all the different versions of this we use, and from where, and skim release notes to see if there are likely any breaking changes.

Secondly, we probably need to clean up handling of this dependency more thoroughly. Version needs to be declared once in the parent, and we probably need to remove most if not all exclusions of this at this stage. They should be redundant, but that also bears thinking through.

It's not a trivial exercise, but it would certainly help to study the state of this fairly important dependency and clean it up. We can tolerate a little breakage risk with Spark 2.x.

[blbradley]

I don't know if it breaks anything yet. I ran the tests locally, but I'm not sure how to tell if/how many tests failed.

[blbradley]

@srowen SPARK-11796 brought the Docker integration tests to 4.5. They still break depending on how they are run (in IntelliJ, for example). Until docker-client uses shaded jars (see this comment), updating this dependency may be the best approach.

[blbradley]

Tests on core sub-project passed. Can we give Jenkins a roll?

[SparkQA]

Test build #2449 has finished for PR 10887 at commit 4eea1c8.

• This patch fails build dependency tests.
• This patch merges cleanly.
• This patch adds no public classes.

[blbradley]

I didn't know there was a dependency test. That helps!

[JoshRosen]

I'm on the verge of just publishing my own shaded docker-client since nobody seems to want to review / shepherd my shading fixes for the Spotify version (and also because it's going to be a larger change to publish with dependency-reduced POMs).

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

@blbradley are you interested in following up on this? this could be a good update for 2.0. This would need a rebase, and needs to also edit the "deps/*" files so we can see what dependencies changed. The thing I would be worried about is that the transitive dependency graph would still contain older versions of Commons Http-related artifacts, which may require a little more massaging. Conceptually, this is good though.