@evaluate macro

[impact-basin]

Carrying on the work from #22 - this is rebased on top of the @grammar pull, though, as most of the nasty bits are the same!

Likewise - I still need to write the docs, so probably not ready for merging yet, but good to track it here :-).

[exaexa]

Just fyi -- the tests actually seem to work, the CI failure is related to something different (will fix).

It's going to need some changes (e.g. the huge module-finding hack is unnecessary because you already hold the module :) ) but should be doable.

Same case for the other PR I guess.

[impact-basin]

Sounds good! Very open to feedback, and glad that the test suite runs!

Regarding testing - I've written an ODL parser via these macros over the last two days (the grammar + evaluator is ~120 LOC); it's part of a larger codebase which isn't ready for release, but I would be happy to share it with you, as it's a reasonable stress-test for these macros. Could you shoot me an email at henry.eshbaugh@physics.ox.ac.uk so I can link you the repo? (It's on a secret self-hosted server).

So - I'll admit that this code is somewhat hacky, but I'm not sure what the best option is, and the decision is up to you. I'll try to summarise it here and what the options are for it.

TL;DR about the hack

What we do depends on whether within the macro we'd like to write clauses as e.g. first(seq(...)) or P.first(P.seq(...)) - I prefer the former, as for complicated grammars this feels less "cluttered" - but this comes with some challenges that the hack addresses.

Those problems relate to the gap between the evaluation context of the code in the macro, and the code produced by the macro; we need to know about the caller's context to carry out this functionality. It's precisely the second context that we need to get information about to make this functionality work.

As I see it, we have five options:

• Keep the macro-expansion-time reflection in;
• Do a @gensym pikaparser in the macro head and import PikaParser as $(pikaparser) in the macro body to ensure we have PikaParser in scope under a known name;
• Have the user supply the module name to the macro;
• Have a separate macro to do the rewriting, leaving it up to the user to decide what to do;
• Drop the functionality completely.

I went with the first option as it's the first one that occurred to me and got the tests to pass; however, import has load-once semantics given:

• Precompilation/caching is enabled (as usual);
• There's no __precompile__(false) directive in the imported module (and there isn't in PikaParser),

so the second option is probably better - fewer lines of code, less hacky, and also works if you import via using PikaParser.

Specifics around the problem

The issue is that the @grammar macro runs in the PikaParser module, but the code it generates runs in Main - so, any PikaParser symbols we want to use have to be qualified according to the way PikaParser was imported. For instance, without this hack and hardcoded PikaParser (as in the gists), if you write import PikaParser, you can invoke the macro, but the test suite will break (as PikaParser is imported as P).

Check out this code at the REPL:

eshbaugh_pu @ atmlxdaq03 ~/src/MoonTools meshing
❯ julia -q
  Activating project at `~/src/MoonTools`
julia> import PikaParser as P

julia> names(@__MODULE__; imported=true)
7-element Vector{Symbol}:
 :Base
 :Core
 :Main
 :P
 :Pkg
 :Revise
 :StyledStrings

julia> PikaParser
ERROR: UndefVarError: `PikaParser` not defined in `Main`
Suggestion: check for spelling errors or missing imports.
Hint: PikaParser is loaded but not imported in the active module Main.
Stacktrace:
 [1] top-level scope
   @ REPL:1

julia> nameof(P)
:PikaParser

Because we have import PikaParser as P, we have the symbol P in the namespace, but not PikaParser.

If we hardcode PikaParser into the grammar (as I was doing in the gist), we can use the macros just fine:

julia> import PikaParser

julia> test1 = PikaParser.@grammar :top begin
       :ws    => first(satisfy(isspace), epsilon),
       :ident => r"[a-z_]+[a-z0-9_]*"i,
       :decl  => seq(:ident, :ws),
       :top   => some(:decl)
       end
#3 (generic function with 1 method)

julia> test2 = PikaParser.@evaluate :top m v begin
       :ident => Symbol(m.view)
       :decl  => v[1]
       end
#9 (generic function with 1 method)

julia> test2(test1("foo bar baz"))
3-element Vector{Any}:
 :foo
 :bar
 :baz

But if we change what PikaParser is called, we have issues:

julia> import PikaParser as P

julia> test1 = PikaParser.@grammar :top begin
       :ws    => first(satisfy(isspace), epsilon),
       :ident => r"[a-z_]+[a-z0-9_]*"i,
       :decl  => seq(:ident, :ws),
       :top   => some(:decl)
       end
ERROR: LoadError: UndefVarError: `PikaParser` not defined in `Main`
Suggestion: check for spelling errors or missing imports.
Hint: PikaParser is loaded but not imported in the active module Main.
Stacktrace:
 [1] top-level scope
   @ REPL:1
in expression starting at REPL[2]:1

julia> nameof(P)
:PikaParser

So we need to know what PikaParser is called in Main (or __module__ more generally - Julia helpfully provides us information about the module our macro's code will be executed in at compile-time).

The trick used in the code at the moment is that the @eval macro can be used to evaluate code in other modules. This can be used for evil (e.g. mutating global variables in other modules), but here we're only doing it to snoop around. This snooping via @eval essentially does three things in the calling context:

• It lists all the available symbols in the calling module;
• It finds the symbols which correspond its in-scope modules;
• It finds symbol corresponding to the module with the name :PikaParser -- which will always be us.

Of course, there are other ways to do this: we can just re-import (which should be instant) and bind PikaParser to a gensym, so that we know we have a reference to it (and this also won't break if you import via using PikaParser, as the current approach does) - or we separate out the functionality.

I'm happy to do whatever you think is best! This turned out to be an interesting problem yesterday when I was baking the PRs and trying to ensure the tests ran. There also might be more elegant ways of solving this problem, but I haven't managed to think up any yet.

[impact-basin]

Update here