AWS EC2 IMDS requests dropping headers

[jackye1995]

Describe the bug

After apache/arrow-rs#7183, the request to IMDS no longer uses the reqwest client directly to make request, but instead now goes through http.connect(&self.client_options.metadata_options()) which gives a HttpClient.

Looks like when making a request to IMDS to get credentials, the header is overridden, so original headers like User-Agent set by the user will be dropped. This causes access issue when the IMDS server is behind a corporate firewall.

To Reproduce

#[test]
fn test_issue() {
    use object_store::client::{ClientOptions, HttpClient};
    use http::{Method, HeaderValue};
    
    let options = ClientOptions::new()
        .with_user_agent(HeaderValue::from_static("my-test-app/1.0"));
    
    let reqwest_client = options.client().unwrap();

    let http_client = HttpClient::new(reqwest_client);
    
    let request = http_client
        .request(Method::GET, "http://169.254.169.254/latest/meta-data/")
        .build()
        .unwrap();

    let has_user_agent = request.headers().contains_key("user-agent");
    
    // fails here
    assert!(has_user_agent)
}

Expected behavior

Should preserve the original headers

Additional context

I will put up a PR later with more details

[tustvold]

Looking more closely at your reproducer I wouldn't expect it to pass, you're setting a header at the client level which only gets applied after the request has been passed to it, it's transparent to the rest of the application.

Have you confirmed the request lack the header, e.g. using wireshark or an test http server?