CL-WHO demo: The Coad Pit

⁰⁹⁵ July 5, 2019 -- (tech tmsr)

In the midst of an ongoing battle with the monster that is Hunchentoot, I have received the suggestion that I can easily communicate intermediate results by simply posting said code on the web in an annotatable form. I judge this to be an excellent idea, even though the items in question represent merely coad¹. Moreover, since I've previously promised to provide examples for other CL WWWisms, it occurs to me that this is just the right occasion to do so.

Thus this post answers the following couple of questions: a. how can one use CL-WHO to generate a simple web site? and b. given a code base consisting of source code, how can we turn that into a HTMListic representation with per-line references, using our tool²?

As previously described, CL-WHO gives us a template language, wrapped around the with-html-output macro. For example, to print a simple web page to standard output, one would write:

(cl-who:with-html-output (*standard-output*)
  (:html (:body
          (:b (cl-who:str "Hello, world!")))))

which yields the output:

<html><body><b>Hello, world!</b></body></html>

First, let's make some sense of what we just wrote. The full description of the template language can be found in the docs, but in short, we: 1. define a context for HTML output; in which we 2. generate a "<html>" whose body contains; 3. a "bold" context, containing 4. the string (str) "Hello, world!".

Now let's say we want to do something fancier, such as mechanically generating a list of items. Then:

(cl-who:with-html-output (*standard-output* nil)
   (:html (:body
           (:ul
            (loop for x in '(a b c d e f)
             for i = 1 then (1+ i) do
             (cl-who:htm
              (:li (cl-who:esc
                (format nil "[~d] ~s" i x))))))))))

gives us:

<html><body><ul><li>[1] A</li><li>[2] B</li><li>[3] C</li><li>[4] D</li><li>[5] E</li><li>[6] F</li></ul></body></html>

In addition to the loop construct, this example has two new elements, namely the htm and the esc symbols. Firstly, esc is very similar to the previously-presented str, only that it does a HTML escape (of e.g. ">" into ">").

Secondly, htm is a sort of a shorthand for with-html-output that is useful for the following: the HTML generation mechanism of WHO looks at cons cells whose first element is a keyword (interpreted as a HTML tag) and it collects them, until it reaches another form -- e.g. our loop construct above. When it gets there, it treats the input as a standard S-expression, to be evaluated by the Lisp evaluator as-is. However, this S-expression now also contains the definition of htm in its macro-scope, which causes further occurences of htm to be expanded, which allows nested HTML-in-Lisp-in-HTML-... expressions.

To put this convoluted³ explanation in simpler⁴ words: with-html-output throws us into a HTML template context; loop (or some other CL control structure) gets us out of there, but htm brings us back in, which is precisely the "mix HTML and CL" machinery at work. Use it wisely, and it shall serve you well.

Now that we have these basics settled, we can get to what really interests us, that is, a HTML representation of source code with line-by-line references. For the sake of simplicity and modularity, we are going to break this into a set of macros, each doing one thing. For example, the CL-WHO template for a line number:

(defmacro tcp-line-number (n &optional (padding 0))
  "Template for the line number."
  `(cl-who:str (format nil "~vd " ,padding ,n)))

Note: the ~vd format string allows us to do variable padding with spaces, which is quite useful here to keep our lines aligned. Next, the line content and the actual line:

(defmacro tcp-line-content (line)
  "Template for the line content."
  `(cl-who:esc ,line))

(defmacro tcp-line (number padding content)
  "Template for the line."
  `(cl-who:htm
    (:span :class "line"
       :id (format nil "L~d" ,number)
       (:a :href  (format nil "#L~d" ,number)
           :class "lineno"
           (tcp-line-number ,number ,padding))
       (:span :class "linecont"
          (tcp-line-content ,content)))
    (:br)))

Next we need to put all the lines together in a code block, which for our convenience will be delimited by a "<pre>" tag:

(defmacro tcp-block (lines)
  "Template for generating a code block out of a list of lines."
  (with-gensyms (i padding line)
   `(cl-who:htm
     (:pre :class "coadblock"
      (loop for ,padding = (length (format nil "~d" (length ,lines)))
     for ,i = 1 then (1+ ,i)
     for ,line in ,lines do
       (tcp-line ,i ,padding ,line))))))

... and, for even more convenience, we'll output a HTML header with some CSS to bellify the coad loox:

(defmacro tcp-html-header (title)
  `(cl-who:htm
    (:title (cl-who:str ,title))
    (:style :type "text/css"
        (cl-who:str "
.coadblock {background-color:#e8e8e8;
            border-style:solid;border-width:1px;padding-left:5px;}
.line {float:left; width:100%;}
.lineno {font-weight:bold;text-decoration:none;color:black;}
.lineno:visited {color:black;}
:target {background-color:lightyellow;}"))))

Finally, we put everything together in a function that receives a page title, a list of strings, each string a line in the source file, and outputs them into a file:

(defun write-code-page (title lines out-path)
  "Write LINES to OUT-PATH as a HTML file."
  (with-open-file (out out-path
               :direction :output
               :if-exists :supersede)
    (cl-who:with-html-output (out nil
                  :prologue t
                  :indent nil)
      (cl-who:htm
       (:html (:head (tcp-html-header title))
          (:body (:b (cl-who:str title))
             (tcp-block lines)))))))

That's it, really, our coad-pit-generating coad. Go ahead and test it. This isn't quite the whole thing, but I'm deliberately omitting the rest since there's a lot of functionality going beyond our CL-WHO demo: we need to a. take a code base given as input, and; b. based on its input path and output path and; c. based on its path inside the website we're deploying to; d. we must process the code tree; and e. for each source file, output the generated page, and f. for each directory, create the output directory and output an index page. I initially thought this would fit in about two hundred lines of Lisp, but it runs a bit over three hundred, so I'm illustrating the demo by having it generate itself.

The full demo coad can be examined on this page. For the future I am preparing a new site, The Coad Pit, available over at coad.thetarpit.org, containing e.g. CL-WHO, Hunchentoot and other coads and codes that I've gathered over time. It's still very rough around the edges, but don't hesitate to play with it.

"Coad" is a term of art, representing, well... let's see:

mircea_popescu: is "coad" code ?
asciilifeform: yea, but think 'with lower case c'
asciilifeform: i.e snippets

In other words, coad is code that comes with no guarantees that it does what the author says it does, nor, in some cases, that it does anything useful at all, nor that it's usable, readable, fittable in head and so on -- making its value as a published item somewhat questionable. However, (some) coad may have the potential to be turned into actual code, i.e. into something that I for example could sign without thinking twice, which is why putting coad on the 'net in some form or the other is not at all a useless endeavour.↩
There's nothing fundamentally new in "code sites". Shithub does it, owing to ye olde LXR; meanwhile, Phf's btcbase has a very neat V patch explorer that's been in Republican use for years now.

So the only thing I'm adding to it is the didactic aspect, I guess.↩
The reader might get a reasonably good insight on what happens behind the scenes by running macroexpand-1 on the example presented. Other than that, it's just CL macros all the way down, which exercise unfortunately goes way beyond the scope of this humble article.↩
Update, July 6: The esteemed readers inform me that I'm all over the place with my attempt at helping them make sense of this, and I doubt that my second attempt is helping much either. So let's take a step back and do a third attempt.

Take any arbitrary (Common) Lisp function f, that we've just finished writing. We naturally want to execute this new function. In order to do that, the function gets compiled and then the resulting code gets evaluated. In fact there's much more happening there, but I'm trying to get the general readership familiar with this, so bear with me, mkay?

So in our scenario, before the compilation phase, one of the steps involves taking the expressions used within f that were previously defined as "CL macros" and macroexpanding them. That is, there is, potentially, code in f that gets executed at compile-time. Most of the time we're doing this because a. we want to preserve code modularity; while b. not fucking up run-time performance; but there's a deeper consideration to be had in mind, namely that CL macros allow the user to define "sub-languages" that, when employed correctly, will result in code that fits in head.

Let us, for example, say that we wanted to output a HTML page that contained "Hello, $name!", where "$name" is to be replaced with a user-provided variable. With nothing but our bare-bones Lisp, we would write something along the lines of:
```
(defun f (name stream)
  (write-string "<html><body>Hello, " stream)
  (write-string name stream)
  (write-string "!</body></html>"))
```
and then we'd say, e.g.:
```
> (f "spyked" *standard-output*)
<html><body>Hello, spyked!</body></html>
```
which does precisely what we want, only this quickly gets uglier with the amount of content we put in our HTML page. So one thing we could do is to wrap some of our previous code into macros:
```
(defmacro html-body (stream)
  `(write-string "<html><body>" ,stream))
;;
(defmacro hello-to (name stream)
  `(progn
     (write-string "Hello, " ,stream)
     (write-string ,name ,stream)
     (write-string "!" ,stream)))
;;
(defmacro /body-/html (stream)
  `(write-string "</body></html>" ,stream))
```
Notice how the macro definitions wrap code that we're not executing at compile time into backquotes and commas, which form a syntactic mechanism for controlling evaluation. This footnote is becoming a blogpost of its own, so I won't get into further details on the particular topic.

Getting back to our f, its definition becomes:
```
(defun f (name stream)
  (html-body stream)
  (hello-to name stream)
  (/body-/html stream))
```
which looks a lot better than the previous, but does the same thing. Say we wanted to look at the macro expansion process by ourselves. Then we would do:
```
> (macroexpand-1 '(hello-to "spyked" *standard-output*))
(PROGN
 (WRITE-STRING "Hello, " *STANDARD-OUTPUT*)
 (WRITE-STRING "spyked" *STANDARD-OUTPUT*)
 (WRITE-STRING "!" *STANDARD-OUTPUT*))
```
What CL-WHO does is to generalize this HTML-outputting mechanism into its own templating language. For the sake of illustrating how growing complexity is handled, we can rewrite a variation on the example above in CL-WHO as:
```
(defun f (name stream)
  (cl-who:with-html-output (stream)
    (:html (:body
            (cl-who:str "Hello, ")
            (if (string= name "spyked")
                (cl-who:htm (:b (cl-who:str "my man")))
                (cl-who:str name))
            (cl-who:str "!")))))
```
which works like:
```
> (f "gigi" *standard-output*)
<html><body>Hello, gigi!</body></html>
> (f "spyked" *standard-output*)
<html><body>Hello, <b>my man</b>!</body></html>
```
and expands to:
```
> (macroexpand-1
   '(cl-who:with-html-output (stream)
     (:html (:body
             (cl-who:str "Hello, ")
             (if (string= name "spyked")
                 (cl-who:htm (:b (cl-who:str "my man")))
                 (cl-who:str name))
             (cl-who:str "!")))))
;; The result:
(LET ((STREAM STREAM))
  (CHECK-TYPE STREAM STREAM)
  (MACROLET ((CL-WHO:HTM (&BODY CL-WHO::BODY)
           `(CL-WHO:WITH-HTML-OUTPUT (,'STREAM NIL :PROLOGUE NIL :INDENT
                                      ,NIL)
              ,@CL-WHO::BODY))
         (CL-WHO:FMT (&REST CL-WHO::ARGS)
           `(FORMAT ,'STREAM ,@CL-WHO::ARGS))
         (CL-WHO:ESC (CL-WHO::THING)
           (CL-WHO::WITH-UNIQUE-NAMES (CL-WHO::RESULT)
             `(LET ((,CL-WHO::RESULT ,CL-WHO::THING))
                (WHEN ,CL-WHO::RESULT
                  (WRITE-STRING (CL-WHO:ESCAPE-STRING ,CL-WHO::RESULT)
                                ,'STREAM)))))
         (CL-WHO:STR (CL-WHO::THING)
           (CL-WHO::WITH-UNIQUE-NAMES (CL-WHO::RESULT)
             `(LET ((,CL-WHO::RESULT ,CL-WHO::THING))
                (WHEN ,CL-WHO::RESULT (PRINC ,CL-WHO::RESULT ,'STREAM))))))
(WRITE-STRING "<html><body>" STREAM)
(LET ((CL-WHO::*INDENT* NIL))
  NIL
  (CL-WHO:STR "Hello, "))
(LET ((CL-WHO::*INDENT* NIL))
  NIL
  (IF (STRING= NAME "spyked")
      (CL-WHO:HTM (:B (CL-WHO:STR "my man")))
      (CL-WHO:STR NAME)))
(LET ((CL-WHO::*INDENT* NIL))
  NIL
  (CL-WHO:STR "!"))
(WRITE-STRING "</body></html>" STREAM)))
```
Maybe not the most beautiful piece of code, but it all becomes clear once you become familiar with the CL-WHO implementation.

So, to summarize: CL-WHO is a compiler that parses a templating language that represents HTML nodes as Lisp lists starting with keywords, e.g. ":em" is the representation of "<em>"; but the same language allows us to express mechanical tasks, e.g. looping over a list of items, which requires a special symbol (cl-who:htm) to move us back in the "HTML page" context. The result of a "with-html-output" code is a program that outputs HTML to wherever we want.

So then, I guess that makes CL-WHO a HTML generator generator?↩