The Tar Pit development log [ii]

In our previous log entry we set the goal of hosting The Tar Pit on its own HTTP server, as opposed to a general-purpose setup, and then using it to serve dynamic content, e.g. comments. We skimmed through a few Common Lisp HTTP server implementations -- why reinvent the wheel when we can steal the blueprints? -- and found at least two such implementations that are minimal enough to be useful for us: Sven Van Caekenberghe's s-http-server and Tomo Matsumoto's cl-http-server. To continue, we'll give a brief overview of one of the former and afterwards get a better look at the latter.

Let's begin with s-http-server. Its interface is fairly simple to use, as stated in the README: creating, starting and stopping servers as well as setting handlers for given URIs are all a piece of cake, except... the whole thing doesn't really work. On a first glance, we notice that the start-server method calls s-sysdeps:start-standard-server, while stop-server calls s-sysdeps:kill-process, which denotes a confusion between the two abstractions¹. Then we notice s-sysdeps contains its own general-purpose stop-server routine, so we can make that public and then use it (instead of kill-process) to stop the HTTP server.

All this turns our server into a working machine until trying to serve a static file, any file at all. Leaving aside the fact that for some reason SLIME doesn't integrate too well with SBCL's add-fd-handler (sb-impl::serve-event needs to be called manually to trigger the handling routine), the HTTP connection handler² ends up trying to write a sequence of raw bytes to a stream that expects characters. I'm not willing to put up with that; what other bugs await in the rabbit hole of s-http-server? If you're interested, find out yourself and let us know.

cl-http-server is very similar to s-http-server, only it's somewhat heavier on the dependency side: it depends among others on Bordeaux Threads, CL-FAD, flexi-streams and trivial-shell, which themselves have some dependencies, which make our in-depth exploration somewhat difficult. There's also no tutorial on the cl-http-server page, so we're going to go through the first steps ourselves. Let us proceed.

Assuming that all the packages are in place and ASDF knows how to load them³, let's begin by importing the library:

CL-USER> (asdf:load-system :cl-http-server)
...
CL-USER> (in-package :cl-http-server) ; ... for convenience

Now that we have cl-http-server loaded, how do we use it? The examples directory has a file called test.lisp which gives us a basic idea. Also, looking at the definition of start-server (server.lisp), we see that the default value of the server argument is the result of a call to make-server, the constructor of server. Let's take a look at how a server object looks:

(defstruct server
  (host                   "127.0.0.1"             :type string)
  (port                   8080                    :type integer)
  ...
  (public-dir             "/tmp/cl-srv/public/"   :type string)
  ...
  (session-save-dir       "/tmp/cl-srv/session/"  :type string)
  ...
  (thread))

There are quite a few fields whose meaning we don't understand exactly quite yet, but we notice there is a public-dir which can be used to serve static files⁴, some fields related to session management⁵ and finally some configuration related to logging, cookies and garbage collection. We don't care about most of them right now, but we'd like to be able to do the bare minimum of serving The Tar Pit in its current form, which means that we can set public-dir to wherever the site is. Let's give it a try:

CL-HTTP-SERVER> (defparameter *my-server*
                  (start-server (make-server :public-dir *lbs-site*
                                             :port 8000)))
*MY-SERVER*
CL-HTTP-SERVER> (describe *my-server*)
#S(SERVER..
  [structure-object]

Slots with :INSTANCE allocation:
...

Where *lbs-site* is the path to The Tar Pit Lisp Blog Scaffolding site defined in config.lisp.

Now, pointing the browser at localhost:8000 displays us a "Default index page" message, which is actually the same page as that returned by the index-page function, which we notice is called from default-page. Either way, by trying out localhost:8000/index.html we actually get the index of our site. This is pretty inconvenient, since we'd like / (the root) to point to this page. Before looking at how to do that, let's get an overview of a few of the abstractions that cl-http-server offers us:

• (get-page path): retrieves the page associated with path
• (set-page path fn): associates the page given by the handler fn with path
• (page-lambda (args ..) body): macro that creates a handler to be used in conjunction with set-page
• (defpage name (args ..) body): alias for set-page and page-lambda; read the code
• (page path args): sort of an alias for get-page; read the code
• (serve-file path): pretty self-explanatory

We notice that default-page tries to get the page named default (which, intuitively enough, maps to the URI /default). If it finds a definition for it, then it serves it; otherwise, it returns the "Default index page" message. So all we need to do is to define it using defpage:

CL-HTTP-SERVER> (defpage default ()
                  (serve-file
                   (merge-pathnames #p"index.html"
                                    (namestring (public-dir)))))

Pointing the browser to / or /default should throw up the index page now.

Now we want to test dynamic stuff, so let's say that we wanted to create a dynamic page, /my-page, that increments a variable on each access and displays it. First, let's define the variable:

CL-HTTP-SERVER> (defparameter *my-lispy-var* 0)
*MY-LISPY-VAR*

Now let's define the page. For convenience, we'll use cl-http-server's html utility function that returns a very simple HTML page with a title and a body. We won't need this for the actual blog, since we already use CL-WHO for templating, but for this prototype it should suffice. We will wrap the page definition in a progn that does two things: it increments *my-lispy-var* and it calls a templated html which formats the value of *my-lispy-var* to a string:

CL-HTTP-SERVER> (defpage my-page ()
                  (progn
                    (incf *my-lispy-var*)
                    (html :body
                          (format nil "Lispy var: ~s"
                                  *my-lispy-var*))))

Now let's suppose we want to do more sophisticate things. Let's say that we want to do some processing of the URL of /my-page, e.g. if we want to search for a specific page when /my-page/a is entered⁶, etc. For this, cl-http-server provides us with the uri-path function, which tokenizes the URL by slashes and it allows us the access the nth token in the URL. Let's put this into an example:

CL-HTTP-SERVER> (defpage my-page ()
                  (progn
                    (incf *my-lispy-var*)
                    (html :body
                          (format nil
                                  "URL suffix: ~s, lispy var: ~s"
                                  (uri-path 2)
                                  *my-lispy-var*))))

Now pointing the browser to /my-page will display "URL suffix: NIL, lispy var: 7"; then if we point it to /my-page/a, it will display "URL suffix: "a", lispy var: 8" and so on.

Now we can stop the server (not that we would ever need to):

CL-HTTP-SERVER> (stop-server *my-server*)
T

To sum things up, I am quite satisfied. At some point I will need to do some basic benchmarking and stress testing to make sure that the server performs well under basic workloads, but even as a test setup this is quite satisfactory. The achievement here is that we've managed to throw together a basic server that adheres to the fits-in-head principle, i.e. even if the length of this post is above the tarpitian average, it still fits within the bounds of decency.

In the following instances of our series we will adapt The Tar Pit LBS to output posts and their metadata as S-expressions and we will integrate the HTTP server component to serve said S-expressions as web pages from what will be a database of blog items.