Macros Can't find any functions on the REPL or try-wisp

[vshesh]

A lot of macros will fail to be accepted on the repl command line.
I give this ns first (so you know everything has been imported)

(ns dev
  (:require
     ; normal runtime comparison and creation functions...
     [wisp.runtime :refer [identity odd? even?
                           dictionary keys vals key-values merge map-dictionary
                           satisfies? contains-vector?
                           nil? true? false? string? number?
                           dictionary? date? boolean? error? re-pattern? object?
                           re-find re-matches re-pattern
                           inc dec str char int subs
                           == > >= < <= + - / *
                           and or print]]

     ; seq abstractions common to most functional languages
     [wisp.sequence :refer [lazy-seq  seq  list  vec  cons  conj
                            lazy-seq? seq? list? sequential? empty?
                            reverse map filter reduce count
                            first second third nth rest last butlast
                            take take-while drop assoc concat
                            repeat every? some
                            partition interleave sort]]

     ;; string functions:
     ;; split split-lines join
     ;; upper-case lower-case capitalize pattern-escape
     ;; replace-first replace blank? reverse
     [wisp.string :as s]))

And then I try something like:

=> (defn tack-on [tack coll] (map (fn [x] [x tack]) coll))
; everything is good here - the function works as expected
=> (defmacro tackmacro [vecs] (tack-on "=" vecs))
=> (tackmacro [1 2 3])
ReferenceError: tackOn is not defined
    at tackmacro (eval at <anonymous> (/Users/Vishesh/repos/dng3/node_modules/wisp/backend/escodegen/generator.js:126:32), <anonymous>:3:9)
    at /Users/Vishesh/repos/dng3/node_modules/wisp/expander.js:72:36
    at expand (/Users/Vishesh/repos/dng3/node_modules/wisp/expander.js:74:7)
    at /Users/Vishesh/repos/dng3/node_modules/wisp/expander.js:176:33
    at macroexpand1 (/Users/Vishesh/repos/dng3/node_modules/wisp/expander.js:177:11)
    at loop (/Users/Vishesh/repos/dng3/node_modules/wisp/expander.js:183:30)
    at macroexpand (/Users/Vishesh/repos/dng3/node_modules/wisp/expander.js:188:11)
    at /Users/Vishesh/repos/dng3/node_modules/wisp/analyzer.js:597:31
    at analyzeList (/Users/Vishesh/repos/dng3/node_modules/wisp/analyzer.js:601:11)
    at analyze (/Users/Vishesh/repos/dng3/node_modules/wisp/analyzer.js:663:160)

Which looks like there's something else going on here with the way regular functions and macros interact - it seems like they're not finding each other?

I can replicate this with a much simpler example:

=> (defn test [x] (+ 1 x))
=> (defmacro testm [x] (test x))
=> (testm 1)
ReferenceError: test is not defined
    at testm (eval at <anonymous> (/Users/Vishesh/repos/dng3/node_modules/wisp/backend/escodegen/generator.js:126:32), <anonymous>:3:9)
    at /Users/Vishesh/repos/dng3/node_modules/wisp/expander.js:72:36
    at expand (/Users/Vishesh/repos/dng3/node_modules/wisp/expander.js:74:7)
    at /Users/Vishesh/repos/dng3/node_modules/wisp/expander.js:176:33
    at macroexpand1 (/Users/Vishesh/repos/dng3/node_modules/wisp/expander.js:177:11)
    at loop (/Users/Vishesh/repos/dng3/node_modules/wisp/expander.js:183:30)
    at macroexpand (/Users/Vishesh/repos/dng3/node_modules/wisp/expander.js:188:11)
    at /Users/Vishesh/repos/dng3/node_modules/wisp/analyzer.js:597:31
    at analyzeList (/Users/Vishesh/repos/dng3/node_modules/wisp/analyzer.js:601:11)
    at analyze (/Users/Vishesh/repos/dng3/node_modules/wisp/analyzer.js:663:160)
=> 

It's even replicable with the base functions

=> (defmacro component [m] (assoc m :restrict :EA))
=> (component {})
ReferenceError: assoc is not defined
    at component (eval at <anonymous> (/Users/Vishesh/repos/dng3/node_modules/wisp/backend/escodegen/generator.js:126:32), <anonymous>:3:9)
    at /Users/Vishesh/repos/dng3/node_modules/wisp/expander.js:72:36
    at expand (/Users/Vishesh/repos/dng3/node_modules/wisp/expander.js:74:7)
    at /Users/Vishesh/repos/dng3/node_modules/wisp/expander.js:176:33
    at macroexpand1 (/Users/Vishesh/repos/dng3/node_modules/wisp/expander.js:177:11)
    at loop (/Users/Vishesh/repos/dng3/node_modules/wisp/expander.js:183:30)
    at macroexpand (/Users/Vishesh/repos/dng3/node_modules/wisp/expander.js:188:11)
    at /Users/Vishesh/repos/dng3/node_modules/wisp/analyzer.js:597:31
    at analyzeList (/Users/Vishesh/repos/dng3/node_modules/wisp/analyzer.js:601:11)
    at analyze (/Users/Vishesh/repos/dng3/node_modules/wisp/analyzer.js:663:160)
=> 

[Gozala]

Yes this has being one of the most painful blockers in getting a macro imports working and in general making macros really useful. Right now defmacro is a macro itself defined here:
https://github.com/Gozala/wisp/blob/0bcaeb6376086314a13b51a3c7b5ac507f5209f9/src/backend/escodegen/generator.wisp#L53-L65

The issue is that defmacro evaluates macro function in the scope of that module rather than in the scope it is defined in. There has being some effort to fix this by compiling (defmacro ...) to regular functions (see #102) instead although there is still work to be done.

I'm afraid I don't have much time to dedicate to this, but I'm more then happy to help to any volunteer ;)

[Gozala]

To be clear that problem is quite tricky and it requires proper implementation of compile and runtime phases. Meaning that during complete time defmacro form can't be just converted to function, instead it needs to be analysed for all the references & those references need to be also compiled and evaluated so that macros could make use of them. In practice this is even more tricky as references may be imported from other modules there for compiler would need to compile imported modules first.

I suspect simple hacky implementation could be made by changing defmacro so that it just compiles to a definition (as illustrated in #102) and does not actually eval but rather eval the full module incrementally form after form afterwards (so that forms after macro could make use of that macro)

[vshesh]

Ok, I don't know enough about macro systems to work on this, but I'm curious how the chain macro example works then, since it calls reduce/cons/first/rest and those all have to be imported.

@vshesh Well, through the use of reader macro's for the most of it I guess. Syntax-quote just does that. But also since there is a precompilation process which makes it a lot easier. I added the chain operator (or thread-first macro as its called in Clojure) to src/expander.wisp and this is how it looks after compilation. Since I have npm link, I can npm link wisp in other projects and use it locally everywhere because this is what it compiled down to:

var expandThreadFirst = exports.expandThreadFirst = function expandThreadFirst() {
    var operations = Array.prototype.slice.call(arguments, 0);
    return reduce(function (form, operation) {
        return cons(first(operation), cons(form, rest(operation)));
    }, first(operations), rest(operations));
};
installMacro('->', expandThreadFirst);

And they would work for a few reasons, one being JS is lenient with non-existing objects and functions and given the correct context, suddenly (when executed there) the variables may be around in scope or, even more importantly I guess, because they are referenced explicitly in the expander.wisp source. I've had more than enough cases where I needed to reference the namespace by fully qualified name (e.g. wisp.string/replace) in order for it to work.