[Cookie] HttpClient does not properly parse all Set-Cookie headers

[Nothing4You]

Hello,

I'm currently trying to build a .Net Core application that automates some HTTP requests.
In the first request I receive several cookies which should be sent back to the server on the second request.
For demonstration purposes I have created a small PHP script that sends some Set-Cookie headers similar to the style I receive them from the actual application.
The source code of this script is https://gist.github.com/Nothing4You/6623cda16eb2c2c5b4d3d9106b95a6ce. You can find a live version at https://hosting.rep.pm/cookietest.php.
I have tested the script using a web browser (Firefox), curl (in a linux shell) aswell as using python requests, which all handle cookies and show me the expected output in the second request:

Cookie[test1]: test1value
Cookie[test2]: test=2&value=2
Cookie[test3]: $test-3
Cookie[test4]: test4

However, if I try to do the same thing using System.Net.Http.HttpClient it only sets the 2 out of 4 cookies.

According to Visual Studio I'm using SDK version 1.1.1.
HttpClient is the following version:

Assembly: System.Net.Http, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
Assembly location: .nuget\packages\system.net.http\4.3.0\ref\netstandard1.3\System.Net.Http.dll

Example code: https://gist.github.com/Nothing4You/c04af6781f8520efb4921ef144733731

[davidsh]

Does this repro on just Windows or Linux or both?

[Nothing4You]

I have only tested on windows so far.

[Nothing4You]

Same results on linux.

[davidsh]

However, if I try to do the same thing using System.Net.Http.HttpClient it only sets the 2 out of 4 cookies.

Please post the specific of which cookies are being "skipped"? I assume being "skipped" implies that they are not being placed into the HttpClientHandler.CookieContainer?

It would be useful if you could post a detailed Fiddler trace of the request and response as seen by Fiddler and then post which cookies on the responses 'Set-Cookie' headers are being "skipped" by HttpClient.

[davidsh]

cc: @stephentoub

[Nothing4You]

In the gist with the code you can see which headers are working and which are not.

[Nothing4You]

https://hosting.rep.pm/httpclient-cookies.saz I believe this is the fiddler trace you're looking for?

[stephentoub]

This repros on desktop as well. This might be a red herring, but the difference between the cookies that work and those that don't appears to be the domain. The ones that don't work set a domain of domain=hosting.rep.pm. The ones that do work either don't have a domain or set one as domain=.hosting.rep.pm.

[davidsh]

Usually the rules for 'Set-Cookie' require the leading prefix of ".". I'll need to lookup the applicable RFCs and it also differs depnding on which cookie RFC is being interpreted.

[davidsh]

Not sure if this is related to https://github.com/dotnet/corefx/issues/11795 but as you can see cookie parsing changes are risky to make due to app-compat.

[Nothing4You]

I have seen that bug, however, in this case it seems like the exact opposite is happening as those without a dot prefix don't work while the one with dot prefix aswell as the unspecified one work.

[davidsh]

If you look at the current cookie parsing code:
https://github.com/dotnet/corefx/blob/master/src/System.Net.Primitives/src/System/Net/CookieContainer.cs#L41

These comments here are relevant:

A cookie with Domain=ajax.com will be rejected because the value for Domain does not begin with a dot.

// According to RFC2109/RFC2965, the cookie will be rejected for matching if:
// * The value for the Domain attribute contains no embedded dots or does not start with a dot.
// * The value for the request-host does not domain-match the Domain attribute.
// " The request-host is a FQDN (not IP address) and has the form HD, where D is the value of the Domain 
//  attribute, and H is a string that contains one or more dots.
//
// Examples:
// * A cookie from request-host y.x.foo.com for Domain=.foo.com would be rejected, because H is y.x 
//  and contains a dot.
// 
// * A cookie from request-host x.foo.com for Domain=.foo.com would be accepted.
//
// * A cookie with Domain=.com or Domain=.com., will always be rejected, because there is no embedded dot.
//
// * A cookie with Domain=ajax.com will be rejected because the value for Domain does not begin with a dot.

[Nothing4You]

RFC2965 doesn't seeem to be relevant for Set-Cookie.
In RFC2109 I found the following:

Domain=domain

 Optional.  The Domain attribute specifies the domain for which the
 cookie is valid.  An explicitly specified domain must always start
 with a dot.

However, as already written in my original post, firefox, curl and python-requests all support this method just fine, so I believe there should be a way to use the appearently commonly accepted way with .NET aswell.

According to RFC2109 is obsoleted by RFC2965, which in turn is obsoleted by RFC6265. RFC6265 is in the state PROPOSED STANDARD, which according to wikipedia usually means it should be fine to be used in production code (correct me if I'm wrong).

This RFC has the relevant part:

If the first character of the attribute-value string is %x2E ("."):

 Let cookie-domain be the attribute-value without the leading %x2E
 (".") character.

Otherwise:

 Let cookie-domain be the entire attribute-value.

Convert the cookie-domain to lower case.

Append an attribute to the cookie-attribute-list with an attribute-
name of Domain and an attribute-value of cookie-domain.

This would just strip a leading dot and parse domains the same way it does now with a leading dot.

[karelz]

Triage: Same behavior as on Desktop, so not critical for 2.0. Moving to Future.