Compiling Lisp Code with Read Macros

Question

I am having trouble understanding what happens with read macros when compiling a lisp code file into bytecode or a raw assembly (or fasl file, for that matter). Or maybe I understand that, but I don’t know. I'm just confused.

When you use a read macro, you do not need to have an accessible source?

If you do this, you need to execute the source code, which is the function of the read macro. If you don't, how can they work when you can do things like read-char ?

To do this, if you want a read macro to use predefined variables, you must execute all the code in front of it, so that it becomes a runtime that will ruin everything.

If you do not run the code in front of it, then the information above will not be available.

What about compiler functions or macros that define read macros? I would suggest that they will not work at all unless you require or load file or something that is not compiled. But if they were compiled, then they will not be able to use them?

If some of my guesses are correct, this means that there is a big difference in “what data will be available for macros” and “which macros will be available for functions” depending on whether you compile the whole file to run later or interpret the file one line at a time (i.e. read, compile and evaluate one expression after another).

In short, it seems that to compile one line into a form where it can be executed without further processing of macros or anything else, you should read, compile and run the previous lines.

Remember once again that these questions relate to compiling lisp without interpreting it, where you can run each line when it comes in.

Sorry for my rambling, but I'm new to lisp and want to learn more about how this works.

Answer 1

This is actually an interesting question, and many beginner Lisp programmers are faced with this. One of the main reasons for this is that everything basically works “as expected,” and you just start thinking about it when you start using the more complex Lisp functions.

The short answer to your question is that yes, for the code to be compiled correctly, some of the previous code had to be executed. Note the word some, which is the key. Let's make a small example. Consider a file with the following contents:

 (print 'a) (defmacro bar (x) `(print ,x)) (bar 'b) 

As you already found out, if you run COMPILE-FILE in this file, the resulting .fasl file will simply contain a compiled version of the following code:

 (print 'a) (print 'b) 

"But," you may ask: "Why was the DEFMACRO form executed at compile time, but the PRINT form was not?". The answer is explained in Hyperspec 3.2.3 . It contains the following sentence:

Typically, top-level forms appear in a compiled file using a compile file and are evaluated only when the resulting compiled file is loaded, and not when the file is compiled. However, as a rule, the case is when some forms in the file must be evaluated at compilation so that the rest of the file can be correctly read and compiled.

There is a form that can be used to control exactly when evaluating the form. For this, EVAL-WHEN . In fact, this is how the Lisp compiler implements DEFMACRO itself. You can see how your Lisp implements it by entering the following from REPL:

 (macroexpand '(defmacro bar (x) `(print ,x))) 

Obviously, various Lisp implementations will implement this differently, but the main thing is that it wraps the definition in the form: (eval-when (:compile-toplevel :load-toplevel :execute) ...) . This tells the compiler that the form should be evaluated both when compiling the file and when loading the file. If he did not, you will not be able to use the macro in the same file that was defined. If the form was evaluated only when the file was compiled, you cannot use the macro in another file after downloading it.

Answer 2

File compilation is defined in Common Lisp: CLHS Section 3.2.3 File Compilation

At compilation: To use a form using a read macro, you must make this implementation of the read macros available to the compiler.

Typically, these dependencies are processed using the defsystem , which describes the dependencies between the various files of the system (something like a project). To compile a specific file, another file (preferably a compiled version) must be loaded into Lisp compilation.

Now, if you want to define a read macro and have forms using its notation in the same file, you need to make sure again that the compiler knows about the read macro and its implementation. The file compiler has a compilation environment. It does not load compiled functions of the same file into this environment by default.

To make the compiler aware of specific code in the file, it compiles Common Lisp provides EVAL-WHEN .

See an example of a read macro:

 (set-syntax-from-char #\] #\)) (defun reader-example (stream char) (declare (ignore char)) (let ((class (read stream t nil t)) (args (read-delimited-list #\] stream t))) (apply #'make-instance class args))) (set-macro-character #\[ 'reader-example) (defclass example () ((name :initarg :name))) (defvar *examples* (list [example :name e1] [example :name e2] [example :name e3])) 

If you download the above source, everything is in order. But if we use a file compiler, it does not compile without loading it in the first place. For example, a file compiler is called by calling the COMPILE-FILE function with the path name.

Now compile the file:

 (set-syntax-from-char #\] #\)) 

The above will not be executed at compile time. A new syntax change will not be available at compile time.

 (defun reader-example (stream char) (declare (ignore char)) (let ((class (read stream t nil t)) (args (read-delimited-list #\] stream t))) (apply #'make-instance class args))) 

The above function compiles but does not load. This implementation is not available to the compiler in subsequent steps.

 (set-macro-character #\[ 'reader-example) 

Again, the above form is not executed - only the code for it is generated.

 (defclass example () ((name :initarg :name))) 

The compiler notices the class, but cannot do it later.

 (defvar *examples* (list [example :name e1] [example :name e2] [example :name e3])) 

The above code throws an error because the read macro is not available at compile time - unless it has been loaded before.

Now there are two simple solutions:

• put the implementation of the read macro in a separate file and make sure that it is compiled and loaded before any file that uses the read macro.

• put EVAL-WHEN around the code, which should have an effect at compile time:

Example:

 (EVAL-WHEN (:compile-toplevel :load-toplevel :execute) (do-something-also-at-compile-time)) 

The compiler will be visible above, and then it will be executed. Now you need to make sure that the code has everything that it calls (all necessary definitions) at compile time.

Needless to say: this is a good style to reduce such compilation dependencies as much as possible. Usually add the necessary functions to a separate file and make sure that this file is compiled and loaded into the Lisp compilation before compiling the file that uses it.

Answer 3

Macros (including read macros) are nothing more than functions, and they are processed just like all other functions. You do not need to save the source code after the function or macro has been compiled.

Many Lisp implementations will not interpret at all. For example, SBCL will compile by default without switching to interpretation mode even for eval . An important nuance is that Common Lisp compilation is incremental (unlike the separate compilation common in many implementations of the Scheme and languages ​​such as C and Java), which allows you to compile a function or macro and use it immediately, in the same "block compilation. "