Follow redirects and eliminate them for sub-resources

[GoogleCodeExporter]

Consider:

<img src="http://example.com/foo.png">

where:

requesting http://example.com/foo.png returns a 301 redirect to 
http://www.example.com/foo.png which is the actual image.

I believe that right now we will not rewrite or cache-extend this image because 
Serf will fail when not getting the 200 OK response for 
http://example.com/foo.png .

This is a big opportunity because not only can we rewrite and cache-extend the 
image, but we could remove unnecessary redirects.

Original issue reported on code.google.com by sligocki@google.com on 8 Apr 2011 at 6:58

[jeffkaufman]

This is a bit tricky because we would need to handle caching properly. Imagine you have:

<img src="http://example.com/foo.png">

http://example.com/foo.png -> www.example.com/foo.png
http://www.example.com/foo.png -> https://www.example.com/foo.png
https://www.example.com/foo.png -> https://www.example.com/foo.webp if chrome

There's a lot of potential for us to optimize the site by resolving those redirects on the server, but we need to make sure the foo.webp redirect doesn't apply to firefox. Plus any Cache-Control headers need to be respected. One option would be to only handle 301 (permanent) redirects but not 302/303/307, but that does miss out on a bunch of potential optimizations.

The next step here is for someone to propose how to handle these cases, and then we can move on to code changes.

[jeffkaufman]

Added to Accepted Feature Requests.

[jmaessen]

I'm going to point Jeff at this one, as he already has some idea how to attack the problem.

[jmaessen]

One other issue I realized could present problems here: http-to-https redirects.

[jeffkaufman]

I think we should probably turn on https fetching by default in the next release, so http-to-https should be fine.

We should talk offline about prioritizing following redirects vs other work.

[crowell]

https fetching is already on by default in master as of b43781d