Placing the cornerstone for The new Tar Pit

Back when I started this journey I had to make a few decisions of a purely technical nature. These decisions shaped the way this blog looks¹, the way it is structured², the software it is based on and the manner in which thoughts are conveyed to the reader. The first and the second I don't plan to change in the foreseeable future; the fourth is hard to figure as far as my own mind's structure is concerned and may change in unknown ways; the third was a purely arbitrary choice based on things I liked at the time, and will be discussed here.

In other words, while I am largely satisfied with my blog, the time has come for me to question the way its so-called "scaffolding" works and set a new stake in the ground regarding it. This is mostly of interest to myself, although I hope bloggers reading this will find here a few bits of experience to integrate into their own.

The Tar Pit has so far used Hakyll as its backing blog software. Hakyll is sexy, being made in Haskell, and it's also customizable, in that it allows me to automatically generate this particular web-thing you're reading. And while it works without giving me any outstanding headaches of the time, it has a few problems that makes it an undesirable tool.

Problem number one (in an arbitrary order) is also its sexiest point, which is that it works within the Haskell environment. This wouldn't have been a problem had GHC not gone the systemd way. Alas, it has, and I do not like it, because it generates among others problem number two. Problem number two is that the modern Haskell ecosystem has now for long been riddled with issues such as a packaging hell, which requires³ me to basically reinstall Hakyll whenever I receive a major update to GHC. More unfortunate is the fact that Hakyll blog (re)installations are very uneconomical, as good practice requires setting up a Cabal sandbox that weighs about 500MB of storage space.

Problem number three consists of some bad bits in Hakyll itself, bits that throw control out of my hands and into the maintainer's⁴. For example a few months ago I for some reasons had to go through the process described in problem number two, which resulted in a new version of Hakyll popping up in my lap. This was all okay until I figured out that post identifier strings were no longer parsed properly, because of some YAML functionality added to the metadata parser. Thus I now had in my hands a feature that I did not ask for but that I got anyway, that I couldn't disable and that I had to hack my way around by manually modifying a large number of files.

Problem number four is the sum of the first three. It's clear to me that such behaviour cannot go unpunished, as ad-hoc changes in the specification⁵ are the bane of engineering. The only question is how to go about this in a sane way.

There are actually two⁶ potential solutions to this problem. The first solution is to fork Hakyll and maintain it myself; the second solution is to start from a clean slate and build the blog exactly the way I want it. Guess which one I chose.

The trade-off goes as follows. The first solution is easier to apply in the short term, but may still end up hurting me in the long term: Hakyll's entire code base, including libraries, is huge in size and as the Haskell ecosystem will continue going systemd I will have to go through pains to keep it up to date while making sure it does what I want it to. The second solution is quite the opposite: it requires a lot of work in the short term⁷ but once it's done it's done, just like real engineering is supposed to work.

So it's about time we applied our principles to this whole thing, isn't it now?

⁂

From a purely Tarpitian point of view⁸, the software that backs a blog up, which I will from now on call a (blog) scaffolding, is a program that takes as input a bunch of files, typically (but not necessarily) Markdown text files, and spits to the output another bunch of files that make up the blog you are now reading. This is the abstract description, which we will refine in the paragraphs to come.

More precisely, files taken as input by The Tar Pit blog scaffolding may be:

• Template files, which are used to generate pages whose content changes over time; this may mean either specific pages (e.g. the blog's index.html) or types of pages (e.g. a list of posts for each blog tag).
• Post files, which are usually written in a lightweight markup language such as Markdown, that will be converted to HTML, and on which a standard page template will be applied⁹.
• Other files, e.g. CSS files or images, that will be inserted into the site as they are.

Template files need to be written in a specific templating language which can be understood by a processing machine. Post files are a tad more complicated: their metadata (e.g. title, post identifier) needs to be parsed, they need to be converted to HTML and then the metadata needs to be applied as per each template's substitution rules. Other things happen there, such as the post metadata being collected into a list in order to be used to e.g. build the post archive, generate one web page per tag, etc.

We additionally require a few mechanisms to make our lives easier. For example we need to take all the post files in a directory tree¹⁰ and automatically generate the output files in a similar hierarchy; idem for files residing in a dedicated uploads directory. Optionally we could have a simple HTTP server to allow us to test the newly generated site¹¹. As an optimization we could maintain a cache of generated posts and check for modifications and update only the modified posts as necessary. There are other features (e.g. RSS feeds), but overall the above should be enough to have a basic blog running.

After pondering for a while¹² I have reached the conclusion that the best trade-off between stability, features and minimalism is the Common Lisp ecosystem. I had actually considered using Scheme¹³ at first, but this one lacks all the cool portable libraries that Common Lisp has¹⁴, so I decided to stick to the latter.

As far as HTML templating goes, CL-WHO is both small enough and usable enough for my purposes, so I'm sticking to that. Directory walking and file copying can be done using built-in Lisp support. The thing that I need to do is build a set of operations around these basic primitives and a small management component to keep track of all the posts in the blog. This is all scriptable, so no troubles here. I'm also including CL-PPCRE to parse tags and post metadata.

The greatest challenge is Markdown-to-HTML conversion. Hakyll uses Pandoc for that and as far as I've used it, it works and it works well, even for extensions such as highlighted code blocks and MathJax¹⁵. As far as I know there are a few Markdown processors written in Lisp¹⁶, but I haven't looked at what libraries they depend on and how compatible they are with Pandoc's Markdown, and figuring this part out would take quite a bit of time. So for the time being I'm conceding on using Pandoc, even though it comes from the same Haskell territory I've decided to abandon.

Things are similar regarding importing a HTTP server. I'm not yet sure this is worth it, but assuming it were, I would take something such as Hunchentoot and severely amputate some of its features¹⁷. Until I figure this out I'm using Busybox's httpd to serve site files locally.

⁂

The new Tar Pit blog scaffolding is still very much a work in progress, and there is no such thing such as an "estimated date of release" for it. However most pieces are already working, the main aspect I need to decide on being the interface the scaffolding will expose to the blogger. There is also the matter of making the code look better, but I don't expect to spend too much time on this¹⁸.

So, dear reader, with a bit of luck you will be able to read this after the nuclear apocalypse will have come to pass and luckily both you and I will have survived it. All you will need is to find¹⁹ a Lisp machine and make only a few modifications to the software in order to be able to enjoy The Tar Pit in all its horrible splendor.