Properly deal with console codepages on windows

[be5invis]

Under CP936, CJK characters in REPL are misencoded. For example, print("测试") is encoded into print("\262\342\312\324"), where 0xB2E2 is the CP936 encoding of "测" and 0xCAD4 is encoded "试".

libuv's TTY adapter (for windows) might help.

---

• cooked mode libuv uses A version of console read/write functions instead of W (with a note in the code that it should be fixed)
• provide information on how to switch away from the (default) raster font
• force CP=437 for the console (and elsewhere), so we at least know we support ASCII

[jiahao]

I don't think we support any encodings other than UTF8.

[be5invis]

However UTF-8 is NOT a native-supported encoding for Windows console. You should add a conversion which converts CP-sensitive strings, or WCHARs if you use Unicode version API to read from console, into UTF-8.

[Keno]

Libuv should do that for us.

[vtjnash]

@loladiro I can confirm that pasting the above characters into the repl, they do not survive the round-trip.

[vtjnash]

this feels embarrassing: I can no longer reproduce the problem. the only known change was installing the german language.

however, it appears that the windows command prompt only supports the current codepage (which means we can't display these characters anyways)

switching the codepage seemed to sometimes makes this work, even showing the right characters (chcp 65001 for utf8), but usually just makes libuv crash in a Console. perhaps we can get this to work? http://msdn.microsoft.com/en-us/library/windows/desktop/ms686013(v=vs.85).aspx

edit: apparently switch the codepage to 65001 once permanently fixes the issue for mintty and only worked the first time for cmd (all later attempts resulted in crashes), even though this setting is transient for the window

[tkelman]

Not sure whether this is the same issue, but I'm seeing only a subset of the new Latex-to-Unicode substitutions show up properly in cmd.

 _/ |\__'_|_|_|\__'_|  |  Commit 631b022* (1 day old master)
|__/                   |  x86_64-w64-mingw32
julia> \alpha α \beta ß \gamma ? \delta δ \epsilon ? \zeta ? \eta ? \theta ? \io
ta ? \kappa ? \lambda ? \mu µ \nu ? \xi ? \pi π \rho ? \sigma σ \tau τ \upsilon
? \phi ? \chi ? \psi ? \omega ?
julia> \Gamma Γ \Delta δ \Theta Θ \Lambda ? \Xi ? \Pi π \Sigma Σ \Upsilon ? \Phi
 Φ \Psi ? \Omega Ω

[vtjnash]

if you enter ;chcp 65001, those will be converted to crashes in all future (supposedly independent) julia execution shells, but will work for that one -- if my testing analysis is was correct

[tkelman]

crashes the first time in my case

[stevengj]

Can't we just ship Julia with some free/open-source alternative to the Windows console that doesn't suck so badly? e.g. are this console or this console any better? How about Mintty, which is the Cygwin default terminal, supports UTF-8, and seems to have lots of other nice features?

[be5invis]

@stevengj I am already using Conemu. And Node.js can handle codepages properly EVEN IN DEFAULT CONSOLE.

[vtjnash]

We use the same backend as node, so we have the same support for the console code pages. However, the issue is that the console only works in the current code page -- it doesn't fully support Unicode. Switching the code page to UTF8 nearly works, but for some reason it also causes writing Unicode characters to return an error.

Conemu is better, but that doesn't say anything

+1 for shipping mintty (which node doesn't support :), as soon as someone figures out why the new repl waits for another character press, after getting a newline, before processing input.

[stevengj]

Conemu also claims to be Unicode-aware and support UTF-8. Although there are reports that Conemu doesn't properly handle combining characters. Any reasonable free-software console that we can ship would be fine with me.

Frankly, even if we get Unicode output working in the default console, we should still ship a better console with Julia. When users double-click on the Julia program in windows, by default it should pop up a window that doesn't suck. Defaulting to the Windows console is shooting ourselves in the foot.