Correctly map UTF-8 positions to UTF-16

[aochagavia]

The LSP specification says that positions are UTF-16 based, but the positions we use are UTF-8 based. We should correctly map from UTF-8 positions to UTF-16 positions.

See the code in https://github.com/rust-analyzer/rust-analyzer/blob/d1b242262a6617b22140bddd0bed23115c260e74/crates/ra_lsp_server/src/conv.rs#L52

@matklad proposed the following approach on Discord:

We need to add a thing, like LineIndex, but which track non-ascii characters.
I think such index could be an HashSet of lines which contain non-ascii chars. Then, for such lines, we'll do a linear scan to switc between UTF-16/UTF-8 offsets

[matklad]

LineIndex is constructed here: https://github.com/rust-analyzer/rust-analyzer/blob/d1b242262a6617b22140bddd0bed23115c260e74/crates/ra_analysis/src/db.rs#L110-L113

[aochagavia]

For anyone interested in fixing this issue, you can test whether it is fixed using the following byte string literal: "€ \u ". Here € is three bytes wide, but VSCode treats it as a single character (because UTF-16). The effect is that the diagnostic shown for \u (invalid unicode escape) is placed two characters to the right, instead of under the \u, as shown below:

Expected:

"€ \u  "
   ~~

Current:

"€ \u  "
     ~~