AM010 – On Tolerating Complexity

It was late at night when I realized it was about to happen again.

The existing publishing tools I had at hand weren’t particularly good to showcase what I had in mind, but they were convenient enough that I could just write and leave the problem of actually reading them to whoever happened to land on that blog.

I mean, it was just a blog. Starting out with 8 patrons, the traffic it got was mostly friends. They could bear with it.

But the unfortunate reality is that I can’t quite stand watching something being done poorly. It just broils in my gut the feeling that we can do better. Editing could be simpler. Publishing could be faster. Thinking could be easier.

I will try to best it, even if the end result ends up being worse, and while I’ll be the first to admit that perhaps it isn’t the most attractive of qualities, it always teaches me something new. Every now and then I do manage to do it better. The world around me is a little better for it.

This time I had to sit down and write something. I didn’t quite know what yet, but I knew that how I had been writing was getting in the way of what I wanted to write. So I took a step back and asked around how are the cool kids doing it this days.

Many seem to like tools like Next.js, Gatsby, or Hugo. They all provide some sort of structure or template you fit your content into, a couple of ways to do theming and navigation, and the possibility to run the whole shebang as a static or dynamically generated service.

I just needed some static files, and I needed them in the structure they were already in, so from the choices I was offered it looked like the winner was in fact Next.js. It came with a few interesting features that I dismissed as whistles & bells, and frowned at the requirement that my pages would have to be translated into React components.

Surely a tool with such an emphasis on the “Developer Experience” would consider the many formats my data could be in, and accommodate for it. But I nonetheless decided to translate an essay into some Javascript code, and I finally felt like I could start writing.

Until I had to embed some code.

Suddenly I was somehow recommended to go down the rabbit of hole of finding good Javascript component libraries that would do the highlighting for me, that worked with Next, from withing my own writing, which was now no longer just text but in fact a computer program posing as an essay.

A quick look under the hood showed me there were over 100,000 lines of Javascript code to turn my untranscendental words into a god damned website.

I sighed.

There is a tolerable amount of complexity involved in doing our every day work. I use vim and, on my computer, it runs on the Linux kernel. I tolerate the many millions of lines of code from there down to the hardware it runs. Tolerance here is the key word. They afford me things. Not physical things like food, but the affordance that your thinking gains when you discover a new idea.

When I first learned about git I suddenly had a new tool to think. I wasn’t familiar with branching models of Subversion or other versioning tools, but git’s branches, merges, and code history extended my thinking. I have a vague understanding of the internals of git, but I tolerate this complexity because of what it affords me.

What was this tool really affording me with this complexity? I already had to pay the cost of translating my content into the format it wanted of me, what else is there? Hot reloading of assets. Great. My assets are stylesheet files, the occasional image, and one or two embedded scripts. I can already refresh a browser with a keypress. What else is there?

From the looks of it, this tools just doesn’t support my use-case very well. How can something so enormously popular and complicated as Next.js not support my so seemingly simple use-case? I had to be wrong. Where is all this complexity heading that I can’t leverage it without bringing in even more of it?

Some complexity exists because the underlying problem is in fact complex. It needs to be dealt with and it cannot be reduced any further. We tend to call this Essential Complexity. Did all the complexity in Next exist only to justify making me feel productive?

I started wondering what really was so essentially complex about what I needed to do, but it didn’t take long to put a list of the things that seemed the most independent from each other:

• I’d like my Markdown files to be compiled into their corresponding HTML files, respecting their existing structure
• I’d like them to be optionally templated, to share some framing information
• I’d like to only do work that needs to be done
• I’d like my files to be served in a browser fast, and
• I’d like my files to be updated automatically in the browser

There were them. 5 requirements that this problem could be broken into. It shouldn’t take more than a couple of days to explore each one of them in enough depth to understand whether I was looking at massive Essential Complexity, or not.

I decided to build this tool.

1. Compiling Markdown to HTML

Markdown was introduced as a more humane way to write HTML. It has evolved from a rather moving target of inconsistent syntaxes into a series of standards, some describing a fairly complex format with plenty of features.

To build a Markdown to HTML compiler I’d have to be clear about which Markdown format I’d be supporting. Since my content was currently written mostly following Github Flavored Markdown, that seemed like the Markdown to target.

Every compiler has a series of stages that take the initial source code, or a similar specification of a program, and turns into another language. Some compilers turn this source code into machine language, some others just turn it into another high-level language.

Whichever your target is, chances are the compiler will read some binary strings (sometimes this is just UTF-8 text, sometimes its actual binary encoded data), and transform them into something that it can operate on. Then it proceeds to transform these data structures into something that more closely resembles the desired output, maybe making some checks along the way.

In my case, I designed it to have 3 stages:

• Parsing of Markdown text — a parsing phase would require a Markdown parser that would deal with the quirks of GFM, and the CommonMark spec it builds on.
• Transformation between Markdown structures and an HTML tree — this would take data structures like Paragraph { content: String } or List { elements: Vec<ListElement> } and turn them into the appropriate HTML tree.
• Writing out the HTML tree — which would take an DomNode { tag: DomTag, attributes: Vec<DomAttribute>, children: Vec<DomNode> } and turn it into a String that can be written into a file.

You can imagine some scaffolded code for this to look like:

enum MarkdownNode {
  Heading1(Vec<MarkdownNode>), // corresponding to a #
  Heading2(Vec<MarkdownNode>), // corresponding to a ##
  Blockquote(Vec<MarkdownNode>), // corresponding to series of >
  // ...
}

struct MarkdownDoc { nodes: Vec<MarkdownNode> }

enum HtmlTag { P, H1, /* ... */ }
struct HtmlAttribute { key: String, value: String }
enum HtmlNode {
  Tagged { 
    tag: HtmlTag,
    attributes: Option<Vec<HtmlAttribute>>,
    children: Option<Vec<HtmlNode>>
  },
  Literal {
    child: String
  }
}

fn string_to_markdown(input: String) -> Result<MarkdownNode, Error> {}

fn markdown_to_html(md: MarkdownDoc) -> Result<HtmlNode, Error> {}

fn html_to_string(html: HtmlNode) -> String {}

It took me about an hour of reading the specification to realize that implementing a parser for the entire syntax would easily take me over a week, and I don’t have that kind of time. It would likely be a very error prone process as well.

A very fun thing to work on, for sure, but after understanding this specific problem better, I can tolerate the complexity of bringing in a 3rd party Markdown compiler into the table.

2. Templating

Templating can take many shapes and forms. From full blown programming language support in the style of ERB (Embedded RuBy), to string matching and replacing in more mundane forms.

Considering I do not need to perform any specific logic, my templating needs are closer to a string-matching followed by some splitting and joining.

I have a template.html file that somewhere in the middle has a keyword that I want replaced with the actual content of the essay I’m writing.

In pseudocode, this should be enough to achieve my goal:

do_template(template, content) do
  [before, after] = template.split_at_word("$$document")
  return [before, content, after]
end

How exactly we are finding the word to be replaced by the content is less important, but from the vast bibliography out there I keep a copy of Flexible Pattern Matching in Strings that is a good resource to implement some of these algorithms.

Once you know where to split the template to inject your content, the rest is just string concatenation.

Thankfully, modern programming languages excel at providing us with string manipulation tools, so putting it together took almost as much code as the pseudocode:

// assume content and template are strings already
let compiled = template.replace("$$document", content);

For my use-case, the complexity of having an entire incrementally-rendering component framework like React, to reuse components across pages that would achieve the same effect, is simply not tolerable.

3. Only doing work that needs to be done

A lot of the tools we work with do the same thing over and over again. Sometimes that is okay. Sometimes that is mandatory.

For my use-case, because I’d like to keep the output of the compilation versioned, the compilation process should only redo the work that needs to be done.

This has the side-benefit that recompiling these documents should be relatively fast, since I tend to work on a single document at a time. Occasionally a change in a template file would trigger only the recompilation of the documents using it, but that work is necessary.

It is not “fast enough that I do not mind having it done over and over again”. It would yield a broken artifact if it wasn’t done.

So a hidden requirement here is that the build process should ensure that work that depends on each other is properly linked, and that if upstream work has to be redone, then dependent downstream work will be done again too.

When we think of work and its dependencies, a very useful mental tool to put to use is Graph Theory. Now I’m no expert mathematician, but building a tree-like data structure is something that we do in functional languages on a daily basis. Even if we don’t define new ones ourselves, we have worked with inductive definitions of Lists, which can be thought of as Directed Acyclic Graphs where every node can have a single children.

enum BuildPlan {
  WithDependants { todo: CompilationUnit, then: Vec<BuildPlan> },
  NoDependants { todo: CompilationUnit }
}

With this mental tool we can model our entire build plan, the steps we have to carry out to achieve our goal, without necessarily doing the actual compilation. However, we will have to describe each step along the way with all the information needed to carry it out in the future.

So far we have to:

1. Compile markdown files to html
2. Maybe create some folders along the way
3. Copy some files verbatim (like images or stylesheets)
4. Template some files

We can imagine a type WorkUnit that specifies the required information for each step:

enum CompilationUnit {
    // To create a directory, we need to know its path
    CreateDir { path: String },

    // To copy a file we need to know its source and destination
    Copy { src: String, dst: String },

    // To compile a file, we need to know what to compile and where to put it
    Compile { input: String, output: String },

    // To template a file we need to also know what template to use
    Template { input: String, output: String, template: String },
}

It seems that because we are only working with files here, we can make use of the expected outputs to verify whether they are already where they should be and use this to avoid repeating that work.

There has been plenty of work done by tools like Google’s Bazel in this space, which are phenomenal at handling distributed and parallel execution of massive compilation graphs across a variety of languages.

Nonetheless, the essential complexity of the problem can be boiled down, for my use case, to a tree-traversal and some file-system operations. Bringing in an external graph library capable of more than that I would consider not tolerable in this case.

4. Serving files to a browser

Having built the bulk of the document compiler, I looked ahead at making this outputs available in a web browser.

At the bottom of this stack, we would need to listen on a TCP socket that a browser could connect to, “speak” HTTP so we could understand what the browser is requesting, and have access to the file system to read, encode, and reply back to the browser with the appropriate HTTP responses.

I already include the IP/TCP stack in the tolerable collection of things, but let’s examine quickly why. Reimplementing the TCP stack would require me to sit down and read through IETF’s RFC 793. I’m not sure I can make sense of everything that’s described in that 80 page document, so I’d immediately say that this could possibly take me months of work. I don’t have that kind of time.

I chose then, sensibly, to reuse existing lower level primitives to open a TCP socket.

The next layer here is HTTP. This falls into the group of tolerable things for very similar reasons. Reading and reimplementing either the original RFC 2616 for HTTP/1.1 or any of the subsequent updates that obsolete the first specification would take me weeks if not months of work.

Lastly, there is the work of turning HTTP requests into specific actions against the file system, and putting together the appropriate responses. This shouldn’t be that hard.

// pseudocode illustrating the idea
fn handle_request(req: Request) -> Response {
  let requested_path = req.path();
  let actual_path = project_folder.join(requested_path);
  let file_contents = read_file(actual_path);
  Response::ok(file_contents)
}

If I can get my hands on some structure or object that represents the browser’s request, I should be able to extract a few things, such as the path, and use that to construct the path to the actual underlying file.

Reading a file is something most programming languages are already equipped to do, so that’s done.

Next is putting together the appropriate response.

The essential complexity of routing a request to a file in disk that I already know exists is mostly absorbed by the TCP and HTTP protocols, which I will not implement.

The rest is manageable, and thus bringing in a 3rd party static file serving solution seems not tolerable.

5. Updating files in a browser automatically

And there I was. I could build my documents, and I could serve them. But they were as static as things can be.

I’d fire up vim, save a change. Recompile. Refresh. The iterative loop.

I’m going to be frank here. That hot reloading is the closest the browser can natively get to that feeling Smalltalk gives you when you modify the actual running system.

Where the source and the artifact meet, fuse, commune.

Once you get your hands on a workflow like that it is pretty hard to go back to anything else that asks anything of you other than expressing your intent.

Just like I find it crippling to leave my keyboard to select a word with my mouse, manually refreshing the browser to see my changes suddenly was something I just could not do without frowning.

So I sat down and I wondered what was it that made Hot Reloading work in this context. How can I tell the browser to redownload and reapply styles on the existing DOM nodes? What about images? Can I update them on the fly too?

Poked around in the console, grabbed a handle to the <link /> field, and tried to update the hred attribute.

let link = document.querySelector('link')
let old = link.href
link.href = "wat"

A request flew by in the Network tab. This change was picked up automatically by the browser and it tried to download the new, obviously missing, stylesheet.

link.href = old

I returned the old value back in and I saw the old stylesheet being downloaded once again. It was clear that assigning a new value to the attribute would retrigger the download of the asset.

After changing the CSS in disk, I tried again, and the styles got updated too. This worked as well for images.

Happy with the discovery, I realized that if I could put together a list of assets that have changed then I could have a single request/response cycle that could tell the browser what needed to be updated.

I put hands to work, made a new handler in the HTTP server, and made that handler wait until there was in fact newly created artifacts from the compilation. This was easy because planning the build already returned a value with the amount of work that has to be done.

fn wait_for_changes(req: Request) -> Response {
  let changed_files = [];
  loop {
    let build_plan = plan_build();
    if build.has_work_to_do() {
      build.execute();
      changed_files = build.new_files();
      break;
    }
    sleep("100ms");
  }
  Response::ok(changed_files)
}

Slightly naive, I’ll grant you that, but considering the recompilation is already very efficient and it only happens as long as there is someone listening for changes I’d say its not that bad.

On the browser side, however, I’d have to make this request to wait for changes happen automatically, without any user intervention whatsoever. It would not make any sense for me to have to write some quirky live-reloading code side-by-side with my documents, so I needed to find a way of injecting a small amount of Javascript into the responses to automatically make this request, and act on the response it returned.

This process on its own would require parsing the HTML that was about to be served, and stitching in a new DOM element that would run the needed Javascript. In turn, I would have to implement an HTML parser that followed the W3C HTML Specification, and I do not have time for this, so the complexity of bringing one in is justified.

After this it all tied together fairly nicely, but the complexity involved in a 3rd party hot reloader for my use case was just not tolerable.

Rinse and Repeat

And so a week went and I had done what I feared: I spent an entire week writing a document builder instead of writing the actual documents.

hotstuff, yet another document compiler, was fathomed into existence.

I am still unsure if the 1,000 lines of fairly amateur Rust code, reading RFCs, language specifications, and bashing my head at the implementation of the incredibly talented work of the authors of Bazel, dune, Next.js, TCP, HTTP, and many others that I’ll miss to mention, has actually made the world even slightly better.

I am unsure if anyone I know would’ve encouraged me to do it myself anyway. If you are reading this and you would’ve, reach out.

What I am sure about is that I’ve learned more about what are the things that I value in the tools that I use, and how they help me do work.

I learned that I like better tools that afford me a lot with a reasonable investment, like how git gives me time-traveling powers for a few metaphors.

I learned that I like tools that respect my data above everything else, like how dune and bazel respect my folder structures, however crazy they may be.

I learned that there is a gap between my intent and the effects it has, and that hot-reloading is a good way of bridging it in a browser.

Most importantly, I learned that I value tools that help me think clearer, and that I think clearest when I understand why something is complex and can make voluntary decisions to tolerate certain complexity or to stand against it.

And that’s what software seems to need the most nowadays. Less flash and more understanding. Less bait and more questions.

Tools to think clearly.

That seems to be the real hot stuff.

References

• Bazel [web]
• CC2 Entry on Whistles and Bells [wiki]
• Dune, a composable build system for OCaml [web]
• Flexible Pattern Matching in Strings [web]
• Gatsby [web]
• Github Flavored Markdown Spec [spec]
• Hugo, The World’s Fastest Framework for Building Websites [web]
• Hypertext Transfer Protocol – HTTP/1.1 [rfc]
• Introducing Markdown [blog]
• Next.js [web]
• No Silver Bullet — Essence and Accident in Software engineering [pdf]
• Rust Language [web]
• Smalltalk [web]
• Stuart Ellis’ Introduction to ERB Templating [blog]
• Tranmission Control Protocol [rfc]
• WHATWG HTML Living Standard [spec]
• WikiWikiWeb [wiki]
• hotstuff, a composable turnkey document compiler [github]