macro hygine escape stripped late for macros calling macros

[c42f]

In julia 0.7, escaping with esc seems to be stripped out later than in julia 0.6.
This means that macros which are called by other macros need to have code for
dealing with input ASTs including Expr(:escape, ...). For example:

macro a(ex)
    # ex will be Expr(:escape, :(x=10)) on julia 0.7, but :(x=10) on 0.6.
    @show ex
    esc(ex)
end

macro b(ex)
    quote
        @a $(esc(ex)) # esc() required for julia 0.6.0
        #@a $ex       # Can be used in 0.7.0-DEV.1300
    end
end

@b x=10
x

Is this intentional? From a user perspective the 0.6 behavior made more sense
to me, as it allowed the inner macro to work with surface syntax not including
Expr(:escape, ...) nodes. It also meant that the outer macro would be required
to properly escape $ex in the example above.

[vtjnash]

Yes, incorrect placement of esc was required as a workaround for #22307

Most of the time, the simplest replacement is simply to move the esc outside:

esc(:( @a $ex ))

However, note that this also affects the scope resolution of @a too.

[yuyichao]

What is the correct way to combine expressions that needs to be resolved in caller scope and callee scope then?

[yuyichao]

i.e. the equivalent of

macro m(ex)
    quote
        a = 1
        @another_macro_in_callee_module a $(esc(ex)) global_in_callee_module
    end
end

[vtjnash]

If you know it must be a global, my current recommendation is to splice it in explicitly: global_in_callee_module = GlobalRef(@__MODULE__, :myglobal)

We also may need to update some macro implementations to be hygiene-annotation aware.

[yuyichao]

Seems to be a step backward since there's almost no way one can get things working without wrapping everything in a big esc if there's nested macros.....

And there's also the a. Is it possible to do that correctly without gensym? Doing gensym will mess up the error messages (UndefVarError and debug info) so it does not provide an equivalence.

[vtjnash]

manual hygiene (gensym(:a)) is definitely always a feasible option

[yuyichao]

That's why I mentioned why gensym isn't really a solution.

It also means that whenever one include a nested macro, they'll likely need to replace all local variables that is used in the macro with gensyms. That can be a pretty tricky work.

This doesn't seem like the behavior we want. We should (and previous did) make sure that at macro templates can be written as normal code apart from the part that needs to deal with user input. That's the lowest level of hygiene-by-default we can/should provide.

[vtjnash]

We should (and previous did) make sure that at macro templates

Let's not romanticize past bugs. In the past, we got macro hygiene wrong, but did so randomly and unpredictably. Hygiene doesn't (and can't) know whether to replace a symbol with a gensym or a globalref until after it has finished expanding all macros. Using esc is tricky work, but it was made exponentially worse in the past by not behaving predictably (ref your PRs on the topic as well).

[yuyichao]

Let's not romanticize past bugs

Well, in both cases there are correct way to write code that will give the correct result and the previous behavior was much easier to use. None of them should be the final solution but I don't think we should make writing correct macros much harder.

[yuyichao]

And so the predictable behavior now is that

1. Basically writing macro template like normal code never work
2. Due to 1, you basically always want to escape the whole result
3. Due to 2, you will get hygiene wrong by default
4. If you managed to overcome 3, you will not get good debug info

[ararslan]

This may be expected behavior, but the change introduced in #22985 has proven to be hugely breaking, so something about this should be addressed.