Decode with utf8 by default for non-text (or all?) content types

[cdvv7788]

I am requesting some information from the a server, which returns the following (using postman):

Headers:
Allow →GET, HEAD, OPTIONS
CF-RAY →439e4801cf3db955-MIA
Connection →keep-alive
Content-Encoding →gzip
Content-Type →application/json

Body:
{
"name": "SARA LUCIA OSSA PEÑA",
}

But, using http, I am getting the following (using a simple request get('url')):
{
"name": "SARA LUCIA OSSA PE�A",
}

To fix this, I had to do something like:
UTF8.decode(response.bodyBytes)

This works as expected, and the information is retrieved fine. This, however, is a pain to setup (and inconsistent with post, where utf8 is used as default encoding).

Is there a better way to handle this? An argument to the get parameter to force encoding? shouldn't application/json assume utf8 by default?

I came up with the solution after reading https://pub.dartlang.org/documentation/http/latest/http/Response-class.html and the body property. Probably it is encoding the body with a wrong format.

Anyway, thanks for the hard work. Awesome library.

[zoechi]

Good comments about this in
https://stackoverflow.com/questions/9254891/what-does-content-type-application-json-charset-utf-8-really-mean

I don't think it's the HTTP clients job to do the decoding. The header is no guarantee that the content will be of that type.
You can always build your own wrapper that does the UTF8-decoding for you so you don't have to repeat yourself.

[cdvv7788]

@zoechi The client has to try to decode using a best effort approach. The http client is already decoding, but using the wrong encoding. There are 2 attributes in the response, body and bodyBytes.
In the same link you send, it is mentioned:

Designating the encoding is somewhat redundant for JSON, since the default (only?) encoding for JSON is UTF-8

Knowing that the content type is json should be enough for the client to interpret it, or at least to avoid intepreting it with the wrong encoding. This is not an edge case, json is probably the most popular format for api communication at the moment. You are right on the wrapper part, but if the headers are coherent, the client should be able to handle the body properly too.

So, the situation is a bit murky, so bear with me..

The first layer in any of this is HTTP.
HTTP, AFAICT, defines the default character set to be ISO-8859-1, which is why the Content-Type charset parameter exists as an explicit override.
In Content-Type all parameters (charset included) are optional, but specific types may define their own required parameters.

JSON, in turn, explicitly does not define the charset parameter.
JSON originally declared that it "shall be encoded in Unicode" with the default byte encoding being UTF-8.
But that's a rather vague declaration, which is perhaps why they later amended it to be a requirement that JSON be encoded in UTF-8.

Next, the Dart http API defines (as mentioned above) two ways of interacting with the response data:

body → String
The body of the response as a string. This is converted from bodyBytes using the charset parameter of the Content-Type header field, if available. If it's unavailable or if the encoding name is unknown, latin1 is used by default, as per RFC 2616.

bodyBytes → Uint8List
The bytes comprising the body of this response.

So as discussed above, HTTP in absence of a defined charset is assumed to be encoded in ISO-8859-1 (Latin-1). And body from its description is consistent with this behaviour.
If the server response sets the Content-Type header to application/json; charset=utf-8 the body should work as expected.

The problem of course is that there are servers out there that do not set charset for JSON (which is valid), but which is also a bit of a grey area in between the two specs:

• JSON is always supposed to be UTF-8, and for that reason says you don't need to set charset, but ..
• HTTP is always by default ISO-8859-1, unless the charset is explicitly set.

A "smart" HTTP client could choose to follow the JSON definition closer than the HTTP definition and simply say any application/json is by default UTF-8 - technically violating the HTTP standard.
However, the most robust solution is ultimately for the server to explicitly state the charset which is valid according to both standards.

As for this bug I'm inclined to say that http is working as intended, though the standards are a bit at odds with each other on this one.
@cdvv7788, if you are able to you could add charset to your Content-Type on the server which should fix your issue.
Alternatively if you're stuck with your server as-is, I recommend you try something like the httpserver example:

  HttpClientRequest request = await HttpClient().post(_host, 4049, path) /*1*/
    ..headers.contentType = ContentType.json /*2*/
    ..write(jsonEncode(jsonData)); /*3*/
  HttpClientResponse response = await request.close(); /*4*/
  await response.transform(utf8.decoder /*5*/).forEach(print);

Hope it helps.

I'll close this issue assuming all open questions are resolved.

[cdvv7788]

Got it. Thanks for this.

[lovelydinosaur]

So as discussed above, HTTP in absence of a defined charset is assumed to be encoded in ISO-8859-1

Note that applies to "text" media types. JSON is "application/json".

Correct clients should treat JSON (or any other non-text media type) responses as bytestrings, rather than text. (ie. use response.bodyBytes.)

@tomchristie, right. I also elaborated a bit on this in #186, but basically saying the same but with more words. :)

[fabiocarneiro]

I still believe this is a bad behavior and added comments to #186