Fix color support on Windows via VT sequences

[davidanthoff]

This is a follow up on #7267 (comment). The issue is that right now that @KristofferC's https://github.com/KristofferC/Crayons.jl package doesn't print colors properly on Windows, because the julia process on Windows doesn't configure proper VT support on newer versions of Windows 10.

I'm sketching what I found so far, but don't have the time to fix this. I believe this needs to be fixed somewhere either in libuv or julia, not sure. But maybe what I found so far is helpful for doing that.

I started out with starting julia on Windows 10 via the start menu link. I then ran:

julia> hOutput = ccall(:GetStdHandle, Int32, (Int32,), -11)
88

julia> dwMode = Ref{UInt32}(0)
Base.RefValue{UInt32}(0x00000000)

julia> ccall(:GetConsoleMode, Int32, (Int32, Ptr{Nothing}), hOutput, dwMode)
1

julia> dwMode[]
0x00000003

The first thing to note is that by default the ENABLE_VIRTUAL_TERMINAL_PROCESSING flag is not set for this process, which means that none of the new VT support in Windows 10 is active.

Alright, lets set that flag (the value of it is 0x004):

julia> newMode = dwMode[] | 0x0004
0x00000007

julia> ccall(:SetConsoleMode, Int32, (Int32, Int32), hOutput, newMode)
1

If I now call

julia> using Crayons

julia> Crayons.test_256_colors(false)

things still don't look good. BUT, if I do the following, I get the correct output:

julia> buf = IOBuffer()

julia> Crayons.test_256_colors(buf, false)

julia> tr_buf = transcode(UInt16, String(take!(buf)))

julia> ccall(:WriteConsoleW, Int32, (Int32, Ptr{UInt8}, Int32, Ptr{Nothing}, Ptr{Nothing}), hOutput, tr_buf , length(tr_buf ), C_NULL, C_NULL)

I'm not sure what is going on, but it looks like somehow that print and friends don't send that string in such a way to the console that the VT sequences make it there. Maybe libuv is intercepting these sequences in some form? I have no idea, but clearly whatever print is doing is preventing things from working as they should in this case.

I believe ENABLE_VIRTUAL_TERMINAL_INPUT is also not set by default. I don't think that matters for the color example, but it probably means that some other VT feature that is in principle supported on Windows is not active when julia runs.

[davidanthoff]

Maybe libuv is intercepting these sequences in some form?

I poked around a bit in the libuv code, and that seems to be the explanation. If I understand it properly, then libuv is actually intercepting VT sequences that one writes to the console and tries to translate them into Windows Console API calls. So these VT sequences never make it out of julia/libuv! I think for newer Windows versions the proper strategy is probably to just skip this custom tty implementation in libuv entirely, and instead set ENABLE_VIRTUAL_TERMINAL_PROCESSING for the julia process and let the windows console handle these VT sequences.

[mgkuhn]

Sounds like something that should be fixed upstream in libuv: if the Windows version is sufficiently new (e.g., 1809), then libuv should make use of the new ConPTY architecture, that is stop using the old Windows Console API, and instead pass on UTF-8 + VT100 codes directly to stdout, just as on Unix.

[davidanthoff]

Yes, agreed. I think on recent Windows versions one could also get rid of the whole UTF-8 to UCS-2 conversion that is happening in libuv on Windows, one can just pass UTF-8 strings directly to WriteConsoleA on modern windows system, as long as one sets the right code page for the console, see here.

CCing @vtjnash because he seems the resident expert on these issues.

[vtjnash]

Yeah, currently we're blocked on several libuv bugs with VTP handling (libuv/libuv#2129 and libuv/libuv#1965). I've lost track of what their status is. But if someone wants this to happen and can help push those over the line, we can maybe get this turned back on.

[musm]

So to be clear this is all upstream problems in UV and not julia related?

[davidanthoff]

That seems a fair assessment, as far as I can tell.

I think libuv just is not hooking into any of the new console improvements that have come to Windows lately. On a modern Windows, it really shouldn't need to handle TTY stuff itself, and it probably also shouldn't do any of the string encoding conversions. I suspect that the code in libuv could look a lot more like the one for the other unix systems on modern Windows systems.

The only thing in julia that I found that one probably doesn't want on modern Windows is this, assuming one just uses the UTF-8 versions of the API directly.

I don't understand this. It might be a way to disable the whole TTY handling that libuv provides on Windows? But I'm not sure... And in any case, it seems better if this would be fixed in libuv directly.

[musm]

What's the advantage or why are we using libuv for tty ? In order to resolve this issue, should we just wait until libuv fixes the problem or try to fix the issue by circumventing libuv and addressing the problem by handling tty stuff ourselves?

[musm]

julia/src/support/libsupportinit.c

Line 16 in c446444

SetConsoleCP(1252); // ANSI Latin1; Western European (Windows)

I wonder why this is set, it doesn't seem necessary to have a functioning repl

[mgkuhn]

So far, Windows console input/output has been very different from Unix (using a C Console API rather than VT100 sequences, no pty, forcing terminal emulators and sshd to screen-scrape a character matrix, no UTF-8 support), and libuv had to work hard to emulate Unix terminal semantics under Windows in a (limited) portable way. However, in 2018, Microsoft finally saw the light (a result of their efforts to create WSL and port OpenSSH to Windows), and they now aim to make the Windows 10 console work much more like xterm on Unix, with a lot of Microsoft-specific backwards-compatibility mechanics (ConPTY). Once that is completed, and Julia drops support for older (i.e., pre-2019) Windows and terminal versions, libuv should no longer be needed for tty output, but for a while the Windows console may remain a moving target. Initial ConPTY support came with Windows 10 1809, with more improvements and bug fixes expected in the 2019 releases.

[davidanthoff]

Probably best to just hope this gets addressed in libuv? It could simply detect at runtime what Windows version is running, and then, depending on that activate all its emulation (for old Windows versions) or not. I assume Julia will want to support older versions of Windows for a while, and it seems easiest if handling that complexity is happening in libuv.

[davidanthoff]

Fun fact: the julia fork of libuv just blanket disables support for this here :)

What I don't understand is why the julia fork took that route, instead of just applying libuv/libuv#1965 on the julia fork, while at the same time leaving the existing libuv support for VT sequences on modern Windows system enabled?

@vtjnash As far as I can tell, all the pieces that are needed to make this work actually exist, the only thing we would need to do at this point is make two changes to the julia fork of libuv: a) undo JuliaLang/libuv@587ebd3 and b) apply libuv/libuv#1965 to the julia fork of libuv.

As far as I can tell libuv/libuv#2129 really doesn't matter for any of this. It would provide a simple API to control some of this behavior, but as far as I can tell we wouldn't actually need that for julia.

[davidanthoff]

I also spent some time investigating what we would gain if our libuv just passed these VT sequences through.

1. it would fix the color issues in the default Windows terminal.
2. I tried out to alternative terminals FluentTerminal and Terminus. Both seem to use conpty, and for both the color issues is solved when the VT sequences are just passed through.
3. When I use VS Code Insider on Windows Insider, the color problems are solved in the integrated REPL. I think that is explained by the fact that VS Code is only using conpty on Windows 18309+ right now.
4. It wouldn't fix ConEmu. I think that is because conpty support hasn't landed there yet (but apparently they are working on it).

So my sense is that the various terminals on Windows are all in the process of moving over to conpty, and as they do, fixing this on the julia side would smooth things considerably for a large number of terminals.

[musm]

@davidanthoff thank you for spearheading this and the great examples / discussion

With the following patch to libsupportinit.c

// This file is a part of Julia. License is MIT: https://julialang.org/license

#include <locale.h>
#include "libsupport.h"
#ifdef _OS_WINDOWS_
#include "windows.h"
#ifndef ENABLE_VIRTUAL_TERMINAL_PROCESSING
#define ENABLE_VIRTUAL_TERMINAL_PROCESSING 0x0004
#endif
#endif

#ifdef __cplusplus
extern "C" {
#endif

static int isInitialized = 0;

void libsupport_init(void)
{
    if (!isInitialized) {
#ifdef _OS_WINDOWS_
        HANDLE hConsole = { GetStdHandle(STD_OUTPUT_HANDLE) };
        DWORD consoleMode = 0;
        GetConsoleMode(hConsole, &consoleMode);
        SetConsoleMode(hConsole, consoleMode | ENABLE_VIRTUAL_TERMINAL_PROCESSING);
#endif
        setlocale(LC_ALL, ""); // set to user locale
        setlocale(LC_NUMERIC, "C"); // use locale-independent numeric formats

        ios_init_stdstreams();

        isInitialized=1;
    }
}

#ifdef __cplusplus
}
#endif

We then fix the issue of having the wrong Console Mode set. In other words, we can directly run the following example without manually changing the console mode.

using Crayons
buf = IOBuffer()
Crayons.test_256_colors(buf, false)
hOutput = ccall(:GetStdHandle, Int32, (Int32,), -11)
tr_buf = transcode(UInt16, String(take!(buf)))
ccall(:WriteConsoleW, Int32, (Int32, Ptr{UInt8}, Int32, Ptr{Nothing}, Ptr{Nothing}), hOutput, tr_buf , length(tr_buf ), C_NULL, C_NULL)