+++
date = "2023-01-30T05:23:28+00:00"
publishdate = "2023-12-29T07:08:55+00:00"
title = "Incremental Writes in Hugo"
slug = "incremental-writes-in-hugo"
author = "Thedro"
tags = ["hugo"]
type = "posts"
summary =  "Lately I’ve been twiddling with a few sufficiently complex Hugo themes and thought I’d share an interesting approach for rendering small changes quickly."
draft =  ""
syntax =  "1"
toc =  ""
updated =  "2023-01-31"
+++

![Hugo's landing page](/images/incremental-writes-in-hugo.png "
  Hugo's [homepage](https://gohugo.io/)."
)

Lately I've been twiddling with a few sufficiently complex
[Hugo](https://gohugo.io/) themes and thought I'd share an interesting approach
for rendering small changes quickly. In absence of a better term ---
"incremental writes" will do and this method works best when there's a lot
happening inside a theme that renders in the `1,000` ~ `10,000+` and beyond
page range.

So what's Hugo and why is it great? Well it's the ultimate static site generator
(`SSG`) that just gets things done. The reason it's great is due to its
{{< sidenote mark="good" set="left" >}}
Low complexity, easy maintenance, and little to no shenanigans.
{{< /sidenote >}}
build and install story along with its 
{{< sidenote mark="hands--off" set="right" >}}
In terms of web development.
{{< /sidenote >}}
abstraction of things.

The code discussed here isn't necessarily intended to be used with
[`hugo server`](https://gohugo.io/commands/hugo_server/) because it has its own
fast rendering mode that works well enough if a theme is adequately simple and
well written. This is more of an approach for rendering and previewing a large,
complex, and messy theme quickly with just the plain `hugo` command. My `hugo`
version is `v0.108.0`.

```shell
$ hugo version
hugo v0.108.0+extended linux/amd64 BuildDate=unknown
```

## Overview of Site Rendering

Before going into the details of making `hugo` write files incrementally it's
useful to have a simple idea of the minimum requirements of a theme and the way
a site is rendered.

```shell
|-- config.json
|-- config.toml
|-- config.yaml
|-- content
|   `-- markdown
|       |-- first.md
|       |-- second.md
|       `-- third.md
`-- themes
    `-- base
        `-- layouts
            `-- _default
                `-- baseof.html
                `-- index.html
```

The directory structure above along with one of the basic
[configuration formats](https://gohugo.io/getting-started/configuration/#configuration-file)
are all that's required to render a theme. Inside the content folder there's the
[Markdown](https://commonmark.org/help/) source and inside the themes folder are
the layout files. Below is a minimal `config.yaml` that disables specific
outputs because for this explanation the `home` and `page` outputs are the
only concern.

```yaml {options="hl_lines=13-17 20-21 28-29",caption="config.yaml"}
---
baseURL:
theme: base
title: Base
languageCode: en-us

paginate: 10
summaryLength: 1

taxonomies:
  tag: tags

disableKinds:
- sitemap
- term
- section
- taxonomy

outputs:
  home:
  - html
  section:
  - html
  taxonomy:
  - html
  term:
  - html
  page:
  - html
```

Default formats are output
["kinds"](https://gohugo.io/templates/lookup-order/#hugo-layouts-lookup-rules)
that are split across home, section, taxonomy, term, and page. This separation
avoids thinking about redundant logic and common repetitive patterns. Home pages
are rendered by `index.html`, sections by `section.html`, taxonomies by
`taxonomy.html`, terms by `term.html` and pages by `single.html`.

| Kinds           | Layouts                   |
| :-------------: | :------------------------: |
| `home`          | `_default/index.html`      |
| `section`       | `_default/section.html`    |
| `taxonomy`      | `_default/taxonomy.html`   |
| `term`          | `_default/term.html`       |
| `page`          | `_default/single.html`     |

[Base layouts](https://gohugo.io/templates/base/#define-the-base-template) are
the authoritative boilerplate that markup and
{{< sidenote mark="data" set="left" >}}
Generally it's best to avoid templating data directly --- but if you're
careful enough you can get away with it handily.
{{< /sidenote >}}
templates of different kinds (home, section, taxonomy, term, page) extend and
render. This is a common approach for most frameworks when smashing together
`HTML` (Hypertext Markup Language). The ideal scenario is to only extend base
layouts and keep visual interpolation and noise to a minimum.

```shell {caption="Base Templates"}
`-- themes
    `-- base
        `-- layouts
            `-- _default
                `-- baseof.html
                `-- baseof.json
                `-- baseof.xml
                `-- baseof.txt
```

For `HTML` and other boilerplate if there's confusion over what type or kind is
in use, then appending that information to the markup or data in a
non--obstructive way works wonders. A `block` opens up a space inside a base
template and a `define` elsewhere fills in or extends that space with the
desired content.

```html {options="hl_lines=4-5 9-11 18-23",caption="themes/base/layouts/_default/baseof.html"}
<!DOCTYPE html>
<html
  lang="en-us"
  data-kind="{{ .Page.Kind }}"
  data-type="{{ .Page.Type }}"
>
  <head>
    <title>
      {{- block "title" . -}}
       {{ .Site.Title }}
      {{- end -}}
    </title>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
  </head>

  <body>
    {{- block "main" . -}}
      <p>
        This is what you see if you don't
        extend the base template's main block.
      </p>
    {{- end -}}
  </body>
</html>
```

## Incremental Writes

Writing output incrementally exploits the way `hugo` evaluates base templates.
The whole site still has to be read but like a novel reading is faster than
writing --- and the computer is not much different. So the speed up here makes
`hugo` mostly read things and not write them. It just so happens that if a base
template for a particular kind evaluates to nothing well then --- `hugo` does
nothing.

Multiple conditions exist for choosing which files to process and write but the
most obvious one compares the modification dates of the source `markdown` with
the resulting `index.html` in the
{{< sidenote mark="public" set="left" >}}
There's a bit of string cutting going on inside the `$page` variable and that's
due to attempts at supporting both domain and sub--directory detection from the
base `URL` (Uniform Resource Locator).
{{< /sidenote >}}
folder. A
[function template](https://gohugo.io/templates/partials/#returning-a-value-from-a-partial)
below checks the modification time using the page's context as input and returns
its modified state as a `true` or `false` output.

```go-html-template {options="hl_lines=1 4 6-10 24-27 31",caption="themes/base/layouts/partials/function-page-modified.html"}
{{- $input := . -}}
{{- $pageContext := $input -}}

{{- $markdown := print "content/" $pageContext.File -}}
{{- $markdownModTime := "" -}}
{{- $page := print "public/" (strings.TrimPrefix
      $pageContext.Page.Site.BaseURL
      $pageContext.Page.Permalink
    ) "index.html"
-}}
{{- $pageModTime := "" -}}

{{- if fileExists $markdown -}}
  {{- $markdownModTime = (os.Stat $markdown).ModTime -}}
{{- end -}}

{{- if fileExists $page -}}
  {{- $pageModTime = (os.Stat $page).ModTime -}}
{{- end -}}

{{- $modified := gt $markdownModTime $pageModTime -}}

{{- $output := or $modified (in (slice
    "home"
    "section"
    "taxonomy"
    "term"
  ) $pageContext.Page.Kind)
-}}

{{- return $output -}}
```

One of the upsides to this particular function is that it also allows turning
off writes for particular parts of the site with an `in slice` check against
the current page's kind.

```html {options="hl_lines=1 3 34",caption="themes/base/layouts/_default/baseof.html"}
{{- $modified := partial "function-page-modified.html" . -}}

{{- if $modified -}}
<!DOCTYPE html>
<html
  lang="en-us"
  data-kind="{{ .Page.Kind }}"
  data-type="{{ .Page.Type }}"
>
  <head>
    <title>
      {{- block "title" . -}}
       {{ .Site.Title }}
      {{- end -}}
    </title>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />

    {{- $default := resources.Get "css/default.css" -}}
    <link title ="Default" rel="stylesheet" href="{{ $default.Permalink }}" />
  </head>

  <body>
    <main class="container">
      {{- block "main" . -}}
        <p>
          This is what you see if you don't
          extend the base template's main block.
        </p>
      {{- end -}}
    </main>
  </body>
</html>
{{- end -}}
```

Absolutely no stray characters or
{{< sidenote mark="white--space" set="left" >}}
Isn't it sort of neat how much white--space matters in programming?
{{< /sidenote >}}
must render in the logic surrounding the base template --- when a page is
unmodified nothing should evaluate. The dash (`-`) in the template logic removes
leading `{{-` and trailing `-}}` white--space. Repeat for every base template
type. The clip below shows a demonstration of this function with `20,000`
pages.

{{< video 
  poster="/images/incremental-writes-in-hugo-poster.png" 
  source="/videos/incremental-writes-in-hugo.mp4" 
  options="loop muted" 
  width="854" 
>}}
In this clip the home page is always rebuilt plus any other recently edited and
non--existent pages. All other previous `index.html` outputs are left untouched.
{{< /video >}}

See
[this repository](https://www.thedroneely.com/git/thedroneely/hugo-theme-base/)
for trying out different numbers and adjusting the complexity.

## Applications and Utility

So how in the world is this useful? Mostly in the case of a theme that executes
lots of work. An upper bound of `20,000` pages for the device in the above clip
{{< sidenote mark="should" set="right" >}}
[The page](https://en.wikipedia.org/wiki/Page_cache) and disk cache are also on
your side.
{{< /sidenote >}}
take around `5` seconds hot plus write time for a single modified page change
even in the context of heavy image processing, bundling, and data
unmarshalling.

In other words no matter the complexity on previously rendered pages editing,
previewing, and building a new page will take at most `~5` seconds because
that's how long it took my device to read all the source `markdown`.

It's also important to know that the code paths for `hugo` and
[`hugo server`](https://gohugo.io/commands/hugo_server/) are different and the
server will likely assume that the previously generated pages don't exist even
if they are in the public folder. If this post makes any sense --- you're
probably using another web server anyway.

Finally be aware that care must be taken with
[shortcodes](https://gohugo.io/content-management/shortcodes/#what-a-shortcode-is).
Even shortcodes that are not in use are evaluated immediately at run time ---
this is an easy way to make builds slow. Keep them as simple as possible.

## Conclusion

More could be said in regards to applying incremental switches especially to
pagination logic but my time was too limited to think that one through. The goal
here however is to avoid diving too far into incremental shenanigans and instead
find a good cut off point for archive rotation while still having fast previews
on pages when running the `hugo` command on a beefy theme. The
{{< sidenote mark="rotation" set="right" >}}
Either truncation or disjointed archives that are joined back together for a
full rebuild.
{{< /sidenote >}}
cut off could be `10,000` pages or even `100,000+` pages --- more or less
depending on a theme's template efficiency, disk `I/O` (Input/Output), and
device memory.