make S"..." and "..." throw errors identically

[StefanKarpinski]

We're close to this, but the error messaging of these two forms is still quite different — the cases in which errors occur is (nearly) the same, but the way errors are shown is quite different. For maximum seamlessness between the two forms, all errors should appear the same.

[vtjnash]

@StefanKarpinski when do these throw an error? and what is an example of the output in each case?

[JeffBezanson]

I guess this is an example:

julia> "\xz"
ERROR: syntax: invalid escape sequence

julia> E"\xz"
ERROR: \x used with no following hex digits

I almost wonder if escape sequences and interpolation should just be features of all strings, period, and custom string literals are just a way to invoke a macro on that parser output. For interpolation, you'd get multiple arguments, some of which are strings. This has the advantage of being really predictable --- the escaping rules never change --- plus custom string literals can support interpolation, and you don't have to know about calling unescape or interp_parse. \ and " are unavoidably metacharacters (for example L"\" is never going to work), so why not just add $ to that list.
Or if there were no interpolation, it would be even better, but let's not gild the lilly :)

cc @nolta

[StefanKarpinski]

Regular expressions would not fare well under such a plan.

[JeffBezanson]

All you have to do is escape $ and . I'm not even sure it's unequivocally good that regexes don't support escape sequences right now.

[pao]

Not sure what you're getting at Jeff. PCRE supports escapes just fine:

julia> m = match(r"\012", "\012")
RegexMatch("\n")

[JeffBezanson]

Oh, I saw this:

julia> r"\u2200"
ERROR: compile: PCRE does not support \L, \l, \N{name}, \U, or \u at position 2 in "\\u2200"

So we are dealing with 2 escape syntaxes, which is also not great.

[pao]

Ahh, fair enough.

[StefanKarpinski]

Ah, yeah, that's a good point. Slashes, however still present a problem as far as I can tell. How can you possibly make r"\d" work?

[StefanKarpinski]

Backslashes, rather.

[JeffBezanson]

Under this design r"<string>" would always be equivalent to Regex("<string>") with the exact same text in both cases. So you would write r"\\d".

[StefanKarpinski]

Honestly, I feel that's unacceptable.

[JeffBezanson]

In that case it might help just to get rid of the str macros that produce strings, and only use custom string literals for different types of objects. The idea then is that custom string literals are different enough not to want the same features that normal string literals have.

Meanwhile, b_str and I_str are now special cases in the parser since they support interpolation, so they are no longer legit string macros. But maybe b_str isn`t as useful as we thought; we haven't used it anywhere (though maybe somebody has).

So you see I find this messy and unsatisfying. True, r"\\d" is very annoying, but there are no deep weird fiddly bugs involved. However we could improve the situation without changing the language by removing all str macros except r_str and v_str.

One telling flaw is that a str macro can't end with a single backslash. Another one, AFAICT, is that there is no way to put a " in a regex literal without also inserting a backslash. Granted those cases are rarer than \d, but they are why I prefer to have one universal escaping scheme and stick with it. Nobody knows what r"\"" means.

Come to think of it, maybe this is a regression?

julia> r"\""
r"\\""

If these things don't support escapes, then we should say you just can't have embedded quote characters, and you get whatever text is before the next " no matter what.

[pao]

Another one, AFAICT, is that there is no way to put a " in a regex literal without also inserting a backslash.

That could be fixed with allowing triplequoting for regexes, which would be useful with the m suffix anyways.

[StefanKarpinski]

There are a few separate issues at play here for non-standard string literals:

1. The parser handling quote escaping (good).
2. The parser handling general escaping (bad).
3. The parser handling interpolation (mixed).

The reason the L"\"" thing is weird is the same reason you can't put a single " into a regex literal without a backslash: the parser knows to ignore escaped quotes for parsing but doesn't remove them. This is not really right and there's no fully correct way around it; the solution is that the parser should handle a layer of escaping. This is slightly tricky to make so it interacts well with additional layers of escaping, but it can be done.

The issue of how regular expressions handle things like r"\u2200" is just a bug. I passed things through to PCRE as-is because it seemed to work and I didn't really know what else to do at the time. The correct behavior is to provide a full, correct translation from surface escapes to escapes and/or raw unicode data that PCRE understands. This is actually an argument against the parser handling escaping, rather than for it, since otherwise we'll never have the flexibility and power to get this right.

We've come around to the idea that the parser should handle interpolation in standard string literals, which I think is good, and has nothing to do with general escaping in standard or non-standard string literals. When x"\a$b\c" is encountered, this could be parsed as a call to @x_str L"\a" b L"\c" so that the parser handles everything dealing with quoting but the varargs @x_str macro still gets to handle escaping (typically using the escape_string utility function). This would even allow interpolation into regular expressions, although, of course, that would entail recompilation on every execution. Non-standard string literals that don't handle interpolation at all could simply emit a clear error message if you try to do interpolation.

The main downside of doing interpolation for all string literals is that you can't have non-standard literals where $ as just a character. That's especially problematic for regular expressions where $ already has a meaning and is extremely common. Perl handles this in classic fashion by cleverly disambiguating the two cases for you in the parser. We could try to do something like that, but it would get pretty insane.

I want to point out that my original proposal was that the parser handle string interpolation for standard string literals while no interpolation would be done for non-standard string literals. I'm not clear on why that's not an acceptable solution. The only counterargument I've seen anywhere is b"" strings, which do binary interpolation in their current form, but as @JeffBezanson pointed out, are not used anywhere.