add syntax highlighting to the REPL input

[KristofferC]

Python now ships their default REPL with syntax highlighting (https://realpython.com/python-repl-autocompletion-highlighting/#syntax-highlighting), and I find it unacceptable that the default Python REPL should be (in some cases) better than ours ( 😉 ).

It has long been possible to get syntax highlighting from the external OhMyREPL package, but now, since we have StyledStrings + JuliaSyntaxHighlighting as stdlibs, it seems kind of a waste not to use those for this. OhMyREPL also has to touch a lot of internals and there are some awkward corner cases with it so proper first-class support would avoid this.

In addition to the highlighting it also adds a little marker showing what parenthesis you are inside (analogous to https://kristofferc.github.io/OhMyREPL.jl/latest/features/bracket_highlighting/). The existing region marking is also implemented in the "pass framework" here.

By default, the JuliaSyntaxHighlighting is quite conservative:

But with a custom faces.toml file (as shown in the docs) you can fancy it up to your liking:

WIP Because of tests.

Written in a beautiful symbiosis between man and machine.

[tecosaur]

Python now ships their default REPL with syntax highlighting and I find it unacceptable that the default Python REPL should be (in some cases) better than ours (😉).

Heh, I was thinking similar thoughts myself reading the 3.14 announcement post. Thanks for doing this Kristoffer.

I've just read through the diff, and at first read it all seems pretty reasonable to me 👍

[tecosaur]

One thing I do wonder is if we're going to put a sample "fancier theme" in the config whether it would be better to use an example that is based on terminal colours rather than hex codes, so it blends in better with people's terminal themes as a drop-in config.

For instance, this is what I've got in my faces.toml

[julia]
macro.fg = "magenta"
symbol.fg = "magenta"
singleton_identifier.inherit = "julia_symbol"
type.fg = "yellow"
typedec.fg = "bright_blue"
comment.fg = "grey"
string.fg = "green"
string_delim.fg = "bright_green"
regex.inherit = "julia_string"
backslash_literal.fg = "magenta"
cmd.inherit = "julia_string"
cmd_delim.inherit = "julia_macro"
char.inherit = "julia_string"
char_delim.inherit = "julia_string_delim"
number.fg = "bright_magenta"
bool.fg = "light_yellow"
funcall.fg = "cyan"
broadcast = { fg = "bright_blue", weight = "bold" }
builtin.fg = "yellow"
operator = "blue"
comparator.fg = "bright_blue"
assignment.fg = "bright_red"
keyword.fg = "red"

[julia.rainbow_paren]
1.fg = "bright_green"
2.fg = "bright_blue"
3.fg = "bright_red"
[julia.rainbow_bracket]
1.fg = "blue"
2.fg = "bright_magenta"
[julia.rainbow_curly]
1.fg = "bright_yellow"
2.fg = "yellow"

That said, Monokai is rather inoffensive and most people use a dark theme, so I'm probably overthinking this.

[KristofferC]

I don't really have an opinion, I just wanted to make it clear that it can be customized and how it is done. Monokai was the first the came to my head.

[IanButterworth]

Is there anything we can learn from the python change about conforming with accessibility, especially in the default color scheme?

[KristofferC]

especially in the default color scheme?

See JuliaLang/JuliaSyntaxHighlighting.jl#3 for the 🖌️ 🚲🏠 . BYOB (Bring Your Own Brush).

[tecosaur]

Is there anything we can learn from the python change about conforming with accessibility, especially in the default color scheme?

Regarding defaults and accessibility, besides the range of opinions about how much syntax highlighting is good (some people like monochrome, or occasional hints of colour, others like comment emphasis, and yes some people like rainbow vomit), default coloring will use ANSI 4-bit colours. Since these tend to be configurable by the terminal emulator, in my mind that's where the responsibility for making them accessible lies.

[KristofferC]

On Windows, pasting a large text becomes significantly slower and also holding a key gives the spinner from #39538. I thought the functionality in #39538 would save us from this type of quadratic behavior but I guess not.

Some breadcrumbs:

• Pasting in new REPL is slow python/cpython#130328
• GH-130328: pasting in new REPL is slow on Windows python/cpython#132884

On nightly, pasing a 1000 line functions takes ~40s. On this PR it is much slower. But even the current 40s is quite bad...

[tecosaur]

I just checked out this PR to see if any of the blame for 40s might lie with JuliaSyntaxHighlighting. While src/JuliaSyntaxHighlighting.jl is just ~450 loc, benchmarking:

apply_styling_passes($jsh_src_code, $([StylingPasses.SyntaxHighlightPass(), StylingPasses.EnclosingParenHighlightPass()]), StylingContext(0))

I get a time of ~1.5ms. I also checked the timings for:

collect(eachsplit(JuliaSyntaxHighlighting.highlight($jsh_src_code, '\n'))

and that gave ~40ms. I suspect there's room for improvement/optimisation here, but this leads me to suspect the poor performance is due to how the REPL handles updates.

---

On another note, it feels great seeing syntax highlighting as part of the OOTB Julia experience. I actually stoped using OhMyREPL a while back to try to motivate me to actually reimplement syntax highlighting + nice history completion in REPL. I've got the latter mostly implemented now, but here you are swooping in with the former 😁

[KristofferC]

The issue is that it rehighlights from scratch character by character when you paste in windows. But the 40s was also without highlighting.

[KristofferC]

We can continue the discussion on what the exact default colors should be, but I think it is good to get some testing done in the meantime.