The Fanciest REPL History in the Land

[tecosaur]

The Fanciest REPL History in the Land ✨

Do you dread typing out code for the second time? Are you a particular enjoyed of REPL history?

Well, I know I am, and for years I have yearned for something better than the current readline-style completion, better than OhMyREPL.jl's fzf-driven completion, better than any REPL history I've seen before!

It's not quite finished baking, but we're onto the final stretch 😀

repl_history_demo.webm

Thanks to @kdheepak, @jakobnissen, and @digital-carver for helping me design the UI and UX over on Zulip (#repl > Revamped REPL history).

Features

• Zippy searching
  • Event-driven asynchronous filtering UI
  • Incremental, resumable searching with dynamic batch sizes
  • Log-structured search checkpoints
• Multi-selection
• Faster histfile parsing (~2x)
• Multiple search modes
• A friendly help page
• Syntax highlighting
• Save multiple items to a file or your clipboard

TODO

• Introduce annotation-preserving replace method
• Thoroughly test the new replace method
• Ask somebody more compiler-y about the performance pitfalls of the replace method (see: the REVIEW: ... code comments in annotated_io.jl)
• Implement flashy REPL history
• Restore up/down arrow history rotation in the REPL (collateral damage of over-zealous deleting)
• Create a new test set for the new history
• Make sure that enough is precompiled to be relatively snappy

---

This PR is on top of #59778, because I think I can safely assume that will be merged first.

[DilumAluthge]

Anyone else getting "video playback aborted due to a network error"?

[KristofferC]

If possible, it would be nice to cut this up into some orthogonal pieces. For example, the AnnotatedString perf improvements could be a separate PR that (with benchmarks) could be merged and would make the diff here smaller and easier to review.

[kdheepak]

Maybe it should default write to a location? The prompt could default to ~/.julia/history.jl (or a more appropriate location).

---

Ideally, the prompt for the filename would also allow scrolling up and down with the up arrow and down arrow, typing some characters of the file and hitting up arrow to narrow the search, have a ghost text of the last file name or the default file name so right arrow completes it etc. But I can imagine this being awkward the way things are set up in the REPL?

[tecosaur]

Ideally, the prompt for the filename would also allow scrolling up and down with the up arrow and down arrow, typing some characters of the file and hitting up arrow to narrow the search, have a ghost text of the last file name or the default file name so right arrow completes it etc. But I can imagine this being awkward the way things are set up in the REPL?

I've had exactly the same thought. I just can't be bothered to make it happen 😅

[tecosaur]

If possible, it would be nice to cut this up into some orthogonal pieces.

I'm happy to split this up, but there are really just three parts to it:

1. A refactor to _replace_final (needed to reuse some code for annotation-preserving replacement)
2. Introducing support for annotation-preserving replacement (needed for changing multi-line annotated strings into single-line with linebreak symbols)
3. Replacing the old history system with the fancy new one

I could split 1-2 off into a separate PR to review, if that sounds like a nice idea?

the AnnotatedString perf improvements

There's only suspicious performance here, not perf improvements, unfortunately 😞. I say "suspicious" because if I remove the keyword argument from replace skip reconstructing a named tuple with a range shifed the total time of a small replace op drops from ~4400ns to ~200ns. Given that an equivalent annotation-less replace takes ~65ns a time of ~200ns would be decent I think, but ~4400ns is strange — I can't figure it out though. If I can interest anyone else in investigating, that would be great.

For now, I've left these REVIEW code comments:

julia/base/strings/annotated_io.jl

Lines 202 to 226 in cc7ed22

	# REVIEW: For some reason, construction of `newannot`
	# can be a significant contributor to the overall runtime
	# of this function. For instance, executing:
	#
	# replace(AnnotatedIOBuffer(), S"apple",
	# 'e' => S"{red:x}", 'p' => S"{green:y}")
	#
	# results in 3 calls to `_insert_annotations!`. It takes
	# ~570ns in total, compared to ~200ns if we push `annot`
	# instead of `newannot`. Commenting out the `_insert_annotations!`
	# line reduces the runtime to ~170ns, from which we can infer
	# that constructing `newannot` is somehow responsible for
	# a ~30ns -> ~400ns (~13x) increase in runtime!!
	# This also comes with a marginal increase in allocations
	# (compared to the commented out version) of 2 -> 14 (250b -> 720b).
	#
	# This seems quite strange, but I haven't dug into the generated
	# LLVM or ASM code. If anybody reading this is interested in checking
	# this out, that would be brilliant 🙏.
	#
	# What I have done is found that "direct tuple reconstruction"
	# (as below) is several times faster than using `setindex`.
	newannot = (region = start+offset:stop+offset,
	label = annot.label,
	value = annot.value)

julia/base/strings/annotated_io.jl

Lines 236 to 240 in cc7ed22

	# REVIEW: For some reason the `Core.kwcall` indirection seems to cause a
	# substantial slowdown here. If we remove `; count` from the signature
	# and run the sample code above in `_insert_annotations!`, the runtime
	# drops from ~4400ns to ~580ns (~7x faster). I cannot guess why this is.
	function replace(out::AnnotatedIOBuffer, str::AnnotatedString, pat_f::Pair...; count = typemax(Int))

[tecosaur]

I've just added back ~400 loc of the original code for using up/down arrow keys to go through recent history without ^R.

If you start a new REPL and immediately press up arrow, it does nothing the first time. Not quite sure why... (any ideas?)

I would like to add session-awareness. Maybe I'll try to drop that in this PR too?

[tecosaur]

Rebased to the latest REPL Syntax Highlighting HEAD.

[tecosaur]

Fixed some typos.

[tecosaur]

The precompilation situation is looking pretty decent, this is all I'm seeing with --trace-compile:

#=   19.4 ms =# precompile(Tuple{Type{Base.IOContext{IO_t} where IO_t<:IO}, Base.GenericIOBuffer{Memory{UInt8}}, Base.TTY})
#=   12.9 ms =# precompile(Tuple{typeof(Base.get), Base.Dict{Tuple{Symbol, Any}, Int64}, Tuple{Symbol, Symbol}, Int64})
#=   19.2 ms =# precompile(Tuple{typeof(Base.setindex!), Base.Dict{Tuple{Symbol, Any}, Int64}, Int64, Tuple{Symbol, Symbol}})
#=    2.5 ms =# precompile(Tuple{typeof(Base.print), Base.IOContext{Base.GenericIOBuffer{Memory{UInt8}}}, Base.AnnotatedString{String}})
#=    1.6 ms =# precompile(Tuple{typeof(Base._str_sizehint), UInt64})
#=    1.9 ms =# precompile(Tuple{typeof(Base.print), Base.GenericIOBuffer{Memory{UInt8}}, UInt64})
#=    2.5 ms =# precompile(Tuple{typeof(Base.AnnotatedDisplay.ansi_write), typeof(Base.write), Base.GenericIOBuffer{Memory{UInt8}}, Base.AnnotatedString{String}})
#=   14.6 ms =# precompile(Tuple{typeof(Base.getindex), Base.JuliaSyntax.GreenNode{Base.JuliaSyntax.SyntaxHead}, Int64})
#=    7.8 ms =# precompile(Tuple{typeof(Base.Terminals.cmove_up), Base.Terminals.TerminalBuffer})

[tecosaur]

Rebased now that #59778 has been merged.

[tecosaur]

If you start a new REPL and immediately press up arrow, it does nothing the first time. Not quite sure why... (any ideas?)

Fixed.

[tecosaur]

Thanks to Miguel, Camillo, and Sundar for taking this for a test-drive and providing feedback 🙂

Changes:

• Clarified that ; uses and logic in quick help
• Strip meaningless whitespace around filter segments
• Changed "negative" → "negated" search term
• Arrow keys (and others) no longer accidentally abort save
• Prevent runaway redisplays
• More minimal display updates
• Make mmap work on Windows

[tecosaur]

Rebase (get the REPL precompilation improvements)

[tecosaur]

• Made synchronous updates dependent on terminfo
• Made face names match module capitalisation
• Use informatively named consts for less common ANSI codes
• Changed the default symbols used on Mac, based on testing on a device without extra fonts installed
• Added warnings for different kinds of invalid REPL histories
• Rename repl_history_hook! to history_search in LineEdit
• Set hp.last_mode + hp.last_buffer in history_search, just to be safe (since enter_search did). It may not be necessary, but it seems potentially prudent
• More intuitive handling of whitespace with search filters
• Fix filter separator escaping
• Count _ as word separator in initialism searches
• Add support for modifying history during a REPL session
• Add history parsing + filtering tests

[tecosaur]

Since almost* all tests were passing before, and all the new REPL History tests (~100) are passing locally, I expect that we'll see the same result after CI finishes.

*there was one issue, a hang on FreeBSD, with the process terminated during jl_swap_fiber, while the cmdlineargs tests were running by the looks of things. This seems unlikely to be related to these changes to me.

[tecosaur]

The history display/interface is not currently covered by tests, not because I don't think it's worth testing, but because it's not clear how best to do so when the display is done asynchronously (two @spawn'd tasks each running a while loop, with a communications channel), and heavily depends on the particular terminal setup (e.g. window size, how can I test an expanding/shrinking terminal window?).

Furthermore, a lot of what we want to test is visual and behaviour varies by terminal emulator (e.g. see the sync / Terminal.app issue noted earlier).

While not a substitute for unit tests, I am rather comforted by the fact that throughout the development of this feature, none of the testers have run into any major failures (the kind that would make someone want to reach for the current basic history finder: there's been no issing history, failure to parse, failure to filter, hang, etc.). The most substantial issues found by testers was (1) a precompilation issue on Mac, (2) Windows not supporting mmaping an IO. The remaining feedback has been entirely on minor visual bugs and behaviour suggestions.

So, while I expect there to be a somewhat long tail of corner cases that come up, I am confident that in its current state this feature is a strict improvement over the current history experience, with no regressions in basic functionality.

[tecosaur]

• Resolve merge conflict with Pkg.version
• A tweak to another default symbol on Mac
• Make clipboard selection just use a SubString{String} all the time
• Since end-to-end/display testing is hard, add a bunch of display state tests

[tecosaur]

In my estimation, this PR should now be good to merge (not squashed, preferably, because the commits actually are meaningful).

[tecosaur]

Tests are soley failing because of a Pkg/SHA issue:

Error During Test at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZQ07RJYVY.0/build/default-grannysmith-C07ZQ07RJYVY-0/julialang/julia-master/julia-f70652db66/share/julia/stdlib/v1.13/Pkg/test/artifacts.jl:355
  Got exception outside of a @test
  Unsatisfiable requirements detected for package SHA [ea8e919c]:
   SHA [ea8e919c] log:
   ├─possible versions are: 1.0.0 or uninstalled (package in sysimage!)
   └─restricted to versions 0.7 by Pkg [44cfe95a] — no versions left
     └─Pkg [44cfe95a] log:
       ├─possible versions are: 1.13.0 or uninstalled
       └─Pkg [44cfe95a] is fixed to version 1.13.0

I'll bump the Pkg bump in an effort to get a green tick, it looks like JuliaLang/Pkg.jl@8d74d35 is needed.

[KristofferC]

Been test driving this and it is very nice. Some comments:

I see -1h for some new changes:

---

Trying to search for "Pkg" mode showed nothing. I figured out later I needed "pkg":

---

Every time you go into the help mode of the reverse search a new " Interactive history search" header is shown. Or rather, I think a spurious new line is inserted at the top.

---

Trying to save to the clipboard gives:

history> save to  Clipboard   File    Tab to toggle ⋅ ⏎ to select┌ Error: Error in the keymap
│   exception =
│    TaskFailedException
│    Stacktrace:
│      [1] #wait#587
...
  nested task error: UndefVarError: `out` not defined in `REPL.History`
│        Suggestion: check for spelling errors or missing imports.
│        Stacktrace:
│         [1] saveclipboard(content::SubString{String})
│           @ REPL.History ~/julia/usr/share/julia/stdlib/v1.13/REPL/src/History/search.jl:389
│         [2] saveprompt(term::Base.Terminals.TTYTerminal, content::SubString{String})
│           @ REPL.History ~/julia/usr/share/julia/stdlib/v1.13/REPL/src/History/search.jl:357

---

I am not sure I like the

# Julia REPL history excerpt

that gets added when saving to a file.

---

You cannot seem to tab-complete Unicode,

---

If I type something longer than the terminal width and then exit out, it doesn't clear the lines above properly:

---

In the previous reverse search, if you select a line, you can press down to get the next line

julia> 'a'
'a': ASCII/Unicode U+0061 (category Ll: Letter, lowercase)

julia> 'b'
'b': ASCII/Unicode U+0062 (category Ll: Letter, lowercase)

julia> 'c'
'c': ASCII/Unicode U+0063 (category Ll: Letter, lowercase)

# reverse search select 'a'
julia> 'a'
'a': ASCII/Unicode U+0061 (category Ll: Letter, lowercase)

# press down
julia> 'b'

With this PR pressing down after selecting an entry does nothing.

---

looks for an initialism

I don't know what this means.

[giordano]

I accidentally pressed twice C-s and ended up in the history saving menu, then instinctively pressed C-g in the hope of aborting, but that didn't work (first issue), it only toggled the option, which is supposed to happen only with TAB. It looks like pressing any key toggles the options (second issue). Then I pressed C-c to try and exit from the history saving menu, that worked, but, also, I lost the cursor (third issue).

[kdheepak]

ctrl + s seems to be launching the interactive history for me

julia-ctrl-s.mov

[kdheepak]

If I type something longer than the terminal width and then exit out, it doesn't clear the lines above properly:

Similar issue when searching for function foo()<kbd>ESC</kbd><kbd>Enter</kbd> return. Maybe making the line prompt have horizontal scrolling would be better than trying to deal with multiple lines.

[tecosaur]

@KristofferC

I see -1h for some new changes

So, currently, times are written as YYYY-MM-DD hh:mm:ss <timezone>. However, we can't interpret <timezone> with the stdlib. We can either:
(a) Ignore the timezone, or (b) switch to using UTC.

I've gone with (b) here, changing the format to YYYY-MM-DD hh:mm:ssZ. That does mean that recent entries from an earlier Julia version appear off by <timezone>.

Trying to search for "Pkg" mode showed nothing. I figured out later I needed "pkg"

This does seem confusing. I've just adjusted the mode to be consistently stored/read in lowercase, and searched case-insensitive.

Every time you go into the help mode of the reverse search a new " Interactive history search" header is shown. Or rather, I think a spurious new line is inserted at the top.

Curious, I have yet to see this behaviour. What's your platform/terminal?

Trying to save to the clipboard gives

That's what I get for an overly quick/casual refactor. Fixed.

I am not sure I like the # Julia REPL history excerpt that gets added when saving to a file.

Removed.

If I type something longer than the terminal width and then exit out, it doesn't clear the lines above properly:

Urgh. I'm not sure how to address this well. I'd like to make it so that long prompts cause horizontal scrolling, but then I can't just reuse LineEdit.Prompt.

With this PR pressing down after selecting an entry does nothing.

Is there something extra you mean by "selecting"? Just going up arrow x3 down arrow x2 gets me back to 'c'.

"looks for an initialism" I don't know what this means

Initials, for instance, Iterators.filter is `if, DataFrame is `df, etc.

@giordano

then instinctively pressed C-g in the hope of aborting, but that didn't work (first issue)

Hmm, I thought I put C-g in the abort keymap. I missed it. Added now.

which is supposed to happen only with TAB

Currently, it happens with any non-abort non-select key.

Then I pressed C-c to try and exit from the history saving menu, that worked, but, also, I lost the cursor (third issue).

🤔 That's a surprise. There's an initial print(out, get(Base.current_terminfo(), :cursor_invisible, "")), but that's within a try ... finally and the finally arm contains print(out, get(Base.current_terminfo(), :cursor_visible, "")).

I could make it so that aborting now takes you back to the history selection UI, that seems potentially more intuitive to me?

[KristofferC]

Another one I hit (enter >Pkg then press ctrl-C)

▪: ┌ Error: Error in the keymap
│   exception =arkTools                                                                                    1h 
...
│        nested task error: MethodError: no method matching Base.AnnotatedString{String}(::SubString{String})
│        The type `Base.AnnotatedString{String}` exists, but no method is defined for this combination of argument types when trying to construct it.
│        
│        Closest candidates are:
│          (::Type{Base.AnnotatedString{S}} where S<:AbstractString)(::Any, ::Any)
│           @ Base strings/annotated.jl:65
│        
│        Stacktrace:
│         [1] convert(::Type{Base.AnnotatedString{String}}, s::SubString{String})
│           @ Base ./strings/basic.jl:228
│         [2] convert(::Type{SubString{Base.AnnotatedString{String}}}, s::SubString{String})
│           @ Base ./strings/substring.jl:66
│         [3] push!
│           @ ./array.jl:1344 [inlined]
│         [4] (::REPL.History.var"#addcond!#ConditionSet##0")(condset::REPL.History.ConditionSet{Base.AnnotatedString{String}}, cond::SubString{Base.AnnotatedString{String}})
│           @ REPL.History ~/julia/usr/share/julia/stdlib/v1.13/REPL/src/History/resumablefiltering.jl:98
│         [5] REPL.History.ConditionSet(spec::Base.AnnotatedString{String})
│           @ REPL.History ~/julia/usr/share/julia/stdlib/v1.13/REPL/src/History/resumablefiltering.jl:139
│         [6] redisplay_prompt(io::IOContext{IOBuffer}, oldstate::REPL.History.SelectorState, newstate::REPL.History.SelectorState, pstate::REPL.LineEdit.PromptState)
│           @ REPL.History ~/julia/usr/share/julia/stdlib/v1.13/REPL/src/History/display.jl:164
│         [7] redisplay_all(io::Base.TTY, oldstate::REPL.History.SelectorState, newstate::REPL.History.SelectorState, pstate::REPL.LineEdit.PromptState; buf::IOContext{IOBuffer})
│           @ REPL.History ~/julia/usr/share/julia/stdlib/v1.13/REPL/src/History/display.jl:80
│         [8] run_display!(::@NamedTuple{term::Base.Terminals.TTYTerminal, prompt::REPL.LineEdit.Prompt, istate::REPL.LineEdit.MIState, pstate::REPL.LineEdit.PromptState}, events::Channel{Symbol}, hist::Vector{REPL.History.HistEntry})
│           @ REPL.History ~/julia/usr/share/julia/stdlib/v1.13/REPL/src/History/search.jl:160
│         [9] (::REPL.History.var"#runsearch##2#runsearch##3"{REPL.History.HistoryFile, @NamedTuple{term::Base.Terminals.TTYTerminal, prompt::REPL.LineEdit.Prompt, istate::REPL.LineEdit.MIState, pstate::REPL.LineEdit.PromptState}, Channel{Symbol}})()
│           @ REPL.History ~/julia/usr/share/julia/stdlib/v1.13/REPL/src/History/search.jl:16
└ @ REPL.LineEdit ~/julia/usr/share/julia/stdlib/v1.13/REPL/src/LineEdit.jl:2985
julia> 

[tecosaur]

Thanks, fixed in cb0894a. I've also had a chat with Mosè and figured out the disappearing cursor issue:

# NOTE: While it may look like `:cursor_visible` would be the
# appropriate choice to reverse `:cursor_invisible`, unfortunately
# tmux-256color declares a sequence that doesn't actually make
# the cursor become visible again 😑.
print(out, get(Base.current_terminfo(), :cursor_normal, ""))

[tecosaur]

• Now when you cancel copying to the clipboard/save to file, you return to the history search

[tecosaur]

• Rebase on master/HEAD