Lack of precedence in OR clause causes lexical ambiguity for comments

[kayabaNerve]

/**/* */*/, in its entirety, is a valid BLOCK_COMMENT_OR_DOC if and only if the outer comment is interpreted as an OUTER_BLOCK_DOC AFAICT. The beginning of the stream also matches as a BLOCK_COMMENT however, which will match only /**/, leaving the rest as remaining tokens (which appears to be the compiler's current behavior).

The notation gives precedence to sequences over the OR operator but does not define precedence when multiple branches are candidates.

I believe the latter may be too notable of a change over updating the comment syntax here, but I'm also unsure the likelihood this was an assumption never actually written which should be included. I also know the comment syntax is also incredibly verbose. I'm happy to leave that question to those with experience.

Also, apologies if I did make a mistake or miss something. I'd be happy to hear why this is invalid :)

[mattheww]

The Reference doesn't say what formalism it's using for its Lexer blocks.

I think it's intended to be a form of context-free grammar, in which case there wouldn't be a precedence between parallel branches: the grammar would be interpreted as saying that BLOCK_COMMENT_OR_DOC accepts a string if there's any choice of branches and sub-branches which amounts to a match of that string.

That leaves a problem if the grammar is ambiguous, as the Reference's lexical grammar is.

The Reference doesn't say how to resolve ambiguities. Most of it seems to be written under the assumption that the lexer repeatedly chooses the longest matching token.

Your /**/* */*/ case is an example where rustc doesn't agree with the longest match given by the Reference's lexical grammar. As you say, rustc takes the initial /**/ as a complete comment.

But the Reference's grammar for block comments is wrong in many ways, and I'm not sure whether this particular ambiguity problem would remain if it was fixed. I wrote up some examples recently at rust-lang/fls#611 (the FLS is using basically the same grammar here).

[mattheww]

I also know the comment syntax is also incredibly verbose.

I think the Reference would be much more readable if it didn't try to make the distinction between doc and non-doc comments as part of the lexical grammar.

I think it would also be easier to avoid making mistakes in the grammar that way (rust-lang/fls#610 probably applies to the Reference, for example).

[ehuss]

The longer-term intention is that alternation gives precedence to alternates in order (first one wins), but we have not been able to go through and update the rules to ensure that holds.

[mattheww]

There is a fully worked out, and tested, lexical grammar using that form of precedence at https://mjw.woodcraft.me.uk/2025-lexeywan/ .