Building your Blog the Hard Way

This is my blog. And it’s basically just a collection of static HTML pages that are hosted on gitlab pages through the complex CI/CD step of doing nothing on each commit and therefore just publishing the existing /public directory of my git repository.

Until recently, that is.

I’ve chosen to not use some static site generator like jekyll but do stuff myself the hard way, because that’s generally more fun and I learn more like that. To give myself a little bit of comfort, I still write my posts with markdown and use the markdown tool from David Parsons, which is in the default arch package repositories, to compile it to HTML. A little Makefile then finds all *.md files and uses a glue.sh script to combine the compiled markdown with some header and footer stubs to create valid HTML pages by using cat how it was originally intended, for concatenating files.

So my basic “build” process was running watch make in one shell to build the project every two seconds while writing the Post in another. To see the compiled result I still have to open the page in my browser and reload manually, but that’s acceptable for me. I would then commit the finished post and the compiled HTML page to my repository and let the “do-nothing” CI do it’s thing.

The only problem with this approach was that I didn’t have a landing page (yet). I didn’t want to write one manually, because I’d have to edit it every time I created a new post. But I couldn’t generate it either, because I wanted it to list all posts by date and the git repository messes up timestamps so that a plain ls -St wouldn’t do.

I was kinda stuck on this issue for a while. I didn’t know a good solution to this problem, and I wasn’t motivated enough to look one up with just a few posts sitting on the page. I didn’t write new ones either, because one couldn’t find one without a landing page anyways. A vicious cycle emerged.

Then I stumbled upon yaml frontmatter. Of course, other projects and static site generators have the same problem as well, and solve it by just stuffing the metadata into the markdown file itself. The format is pretty simple and there exists a python library to parse it. My markdown to HTML compiler is a bit confused by it, but that’s not a problem when you can just cut the frontmatter out with a simple awk script.

So now I had markdown with metadata, but I still wasn’t using it anywhere, and I wasn’t particularly fond of getting frustrated with python, so I snatched a friend who’s using python more regularly than me to do my index.py script with a little pair programming. ¹

It’s a pretty simple process, really. The python script gets all input markdown files from my Makefile as a list of arguments, parses the frontmatter using python-frontmatter, which returns a weird dictionary with content hybrid. I don’t really need the post content, so I convert it into a honest dictionary, and then sort everything by the published yaml matadata key. The python parser is nice enough to convert that into a datetime class automatically.

The only thing that I have to duplicate is the markdown to html path translation to add correct links into my landing page. ² A simple ^markdown/(.*).md$ captures the file name in a, uh… capture group… to append the correct .html ending afterwards. It’d be better if I could somehow use Makefile magic to create (.*).md and (.*).html dependency pairs, but this solution just works, for now.

Then all relevant metadata keys get printed to standard output in a html formatted way using python format strings. The Makefile rule for my index.html then just calls the script with appropriate arguments and redirects the output to $@. ³ In the end, building the whole script was faster and more fun then expected, and I’m really glad that I have people who are nice enough to do pair programming with.

The other thing bugging me with my current implementation was that I’m not really using the CI/CD in the intended way, because I don’t really build my blog on the CI server. Some stack overflow post was hinting that the default runner has some ubuntu image, so I tried modifying the .gitlabci-yml file to install the tools I’m using and then building the blog by calling make clean; make, but committing a single line change in the CI file and then waiting a minute for the build to fail wasn’t as fun.

The first roadblock was installing the correct python version, because the default on Ubuntu still seems to be python2. And not only do you somehow have to install the python3 packages, but to install library with pip you also have to call pip3. After noticing that my markdown to html compiler is not available on Ubuntu I gave up and just used a custom Docker image. Fortunatly, it was fairly simple to use the archlinux:latest image and then to install the tools that I’m using on my own machine.

So now I should be just able to write some markdown in the gitlab Web-IDE and have everything build and deployed automatically, yay!