Posts from Thedro Neely

Guix in a Linux Container

Sat, 29 Apr 2023 02:10:18 +0000

GNU Guix (as in geeks) is one of the new type of Linux distributions that focuses on reproducibility. It’s in the class of systems that can allow for immutability, in addition to atomic updates and root less user package management. NixOS is another similar new type in this class of reproducible operating systems.

Placing a Linux distribution into a container is a good way to learn about the underlying structure before installing it onto a real device. This let’s play around with Guix.

Commands

The commands in this post are for running on Debian but it’s possible on any Linux distribution via the Guix shell installer script. Debian makes this easy because it has a guix package in its repository. It may be necessary to prefix sudo to the following commands for root privileges.

apt install guix

There are other packages that might be required for guix to work correctly.

apt install netbase gpg xz-utils ca-certificates

Guix needs a daemon to build and answer different requests. Running it without root is possible (from a container point of view). Start the daemon if it’s not running.

guix-daemon &

From there guix is now installed. Running guix pull brings in sources and upgrades guix to the latest version.

guix pull

The version string should reflect the current git repository commit at the time of pulling. In this post the guix version is the short commit identifier 3ab983d.

$ guix --version
guix (GNU Guix) 3ab983d630a95a29b9418b1ba8a26e5ca2836ec0

If the version string of guix isn’t up to date then check that the shell has properly sourced the guix specific environment variables like PATH from the home folder.

. /etc/profile.d/guix.sh

In other instances, it might be necessary to hash or update the new location of guix using the built–in shell command hash.

hash guix

To view the exact revision the system is running use the describe subcommand.

$ guix describe
Generation 1 Apr 11 2023 22:06:05 (current)
  guix 3ab983d
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 3ab983d630a95a29b9418b1ba8a26e5ca2836ec0

Searching for programs to install locally on Guix is powerful. There’s also a portal for previewing the available packages online.

Overview

The Guix configuration system works from a declared file that represents the desired state of the machine. That file, let’s say config.scm needs to describe a configuration in Guile Scheme that returns an operating-system.

(use-modules (gnu))
(operating-system)

Guix already comes with a system subcommand that allows creating varied formats from a defined configuration for containers, images, and virtual machines (vm).

guix system container config.scm
guix system image --image-type=tarball config.scm
guix system vm config.scm

The list below shows the various presets for different image output format types.

$ guix system image --list-image-types
The available image types are:
   - rock64-raw
   - pinebook-pro-raw
   - pine64-raw
   - novena-raw
   - hurd-raw
   - hurd-qcow2
   - efi32-raw
   - qcow2
   - iso9660
   - wsl2
   - docker
   - uncompressed-iso9660
   - tarball
   - efi-raw
   - raw-with-offset

All of these work great and are extremely useful. In my case however, I’ll explore a bit further for a file system that can be plugged into a few more places like a LXC (Linux Containers). A brief look around the net seems to suggest that there are multiple approaches.

But nothing beats source code right? The guix system container commands appear to run from the gnu/system/linux-container.scm module definition. the source further reveals a gnu/system/examples folder that lists different configuration templates including a docker-image.tmpl and a vm-image.tmpl.

The linux-container.scm looks enough to massage into a config.scm with a bit of guess work. Guix also has a %base-services list in gnu/services/base.scm that can work as a reference to build up a minimal service configuration. GNU Shepherd is the program that starts up the guix system.

Guix system init sets up a target directory with the needed files to start the OS (Operating System). A --dry-run tests the configuration file for correctness. The --no-bootloader argument is passed because well it’s not a real device — not yet at least.

mkdir filesystem
guix system --dry-run init --no-bootloader config.scm filesystem/

Configuration

A setup more suited for a container is added to the operating-system based on the previously mentioned linux-container.scm source file.

(use-modules (gnu) (gnu system locale))
(use-service-modules networking ssh)
(use-package-modules ssh bash package-management)

(operating-system
 (host-name "guix")
 (timezone "America/Nassau")
 (locale "en_US.utf8")
 (firmware `())
 (initrd-modules `())
 (kernel hello)
 (packages (list guix coreutils))

 (essential-services (modify-services
                      (operating-system-default-essential-services this-operating-system)
                      (delete firmware-service-type)
                      (delete (service-kind %linux-bare-metal-service))))

 (locale-definitions (list (locale-definition
                            (name "en_US.utf8")
                            (source "en_US")
                            (charset "UTF-8"))))

 (bootloader (bootloader-configuration
              (bootloader grub-bootloader)
              (targets '("/dev/null"))))

 (file-systems (list (file-system
                      (device "/dev/null")
                      (mount-point "/")
                      (type "dummy"))))

 (services (list (service dhcp-client-service-type)
                 (service syslog-service-type)
                 (service guix-service-type)
                 (service static-networking-service-type
                          (list %loopback-static-networking))
                 (service special-files-service-type
                          `(("/bin/sh" ,(file-append bash "/bin/sh"))
                            ("/usr/bin/env" ,(file-append coreutils "/bin/env"))))
                 (service udev-service-type
                          (udev-configuration
                           (rules '())))
                 (service openssh-service-type
                          (openssh-configuration
                           (openssh openssh-sans-x))))))

Generate the files needed to boot the system by running the init subcommand without a --dry-run.

guix system init --no-bootloader config.scm filesystem/

One of the best ways to pass around file systems like guix with hard links and such is to use rsync.

rsync -aAXHv filesystem/ rootfs
rsync --archive --acls --xattrs --hard-links --verbose filesystem/ rootfs

Container

The configuration below contains guix specific options that allow the container to start up.

# Distribution configuration
lxc.arch = linux64
lxc.include = /usr/share/lxc/config/common.conf

# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:48:0d:f8

# Container configuration
lxc.uts.name = guix
lxc.environment = HOME=/root
lxc.environment = GUIX_NEW_SYSTEM=/gnu/store/n3yd660amqnhcmyv3qj2ghc4xhnfcj5z-system
lxc.rootfs.path = dir:/home/thedro/.local/share/lxc/guix/rootfs
lxc.init.cmd = /gnu/store/vbcdwklck7cd28j4jv9wchfw60abivfd-guile-3.0.9/bin/guile /gnu/store/hgi3i9ydjhi5bxnpz7zmzaqn60iw3ak1-boot

Guix will try to activate-modprobe and activate-firmware. If those activation paths (proc or sys) are exposed to the container and are read only, then the container may fail to start. The work around is to remove the activation scripts by modifying the list of essential-services in the operating-system as seen in the configuration from before.

In addition, the lxc.init.cmd has to be populated with the system environment variable (GUIX_NEW_SYSTEM), the boot script path, and the guile executable path needed to start the system. Setting up a special-files-service-type for an equivalent to /sbin/init is probably a better approach after the first boot, but for now manually locating the paths with the find command from the root directory works well.

find . -maxdepth 3 -regex '.*system' -print -quit
find . -maxdepth 3 -regex '.*boot' -print -quit
find . -maxdepth 3 -regex '.*guile-3.*' -print -quit

Entering the container with lxc-attach may require sourcing /etc/profile to populate all of the guix specific environment variables.

. /etc/profile

Comparisons

The GNU Guix system is inspired by NixOS and there are some initial comparisons that can be made between the two within the context of containers. In terms of file sizes for the containers, consider the following shaved down NixOS container configuration.

{
  imports = [
    <nixpkgs/nixos/modules/profiles/headless.nix>
    <nixpkgs/nixos/modules/profiles/minimal.nix>
  ];

  disabledModules = [
    <nixpkgs/nixos/modules/profiles/all-hardware.nix>
    <nixpkgs/nixos/modules/profiles/base.nix>
  ];

  system.stateVersion = "22.11";
  nixpkgs.system = "x86_64-linux";
  networking.hostName = "nixos-container";
  environment.defaultPackages = [ ];

  boot.isContainer = true;
  xdg.mime.enable = false;
  xdg.icons.enable = false;
  xdg.sounds.enable = false;
  users.mutableUsers = false;
  documentation.enable = false;
  security.polkit.enable = false;
  fonts.fontconfig.enable = false;
  services.udisks2.enable = false;
  nixpkgs.config.allowUnfree = true;
}

At a quick glance, operations on Guix are slightly slower than NixOS but as seen from both configurations Guix is much easier to hack on and with naive attempts at minimization clocks in at a size.

$ du -h guix
834M guix/

$ du -h nixos
1.1G nixos/

Guix also runs in an unprivileged container easily as shown in the video below but for NixOS it’s rather difficult to do.

Conclusion

That’s a wrap. setups allow for experimenting with lots of configurations and software before going all in on real devices! On another note — here’s a quick overview and rundown of the Guix system.

Sorry, your browser doesn't support embedded videos. Here is a link to the video instead.

Poking around the container to run the guix command and as a bonus neofetch

Incremental Writes in Hugo

Mon, 30 Jan 2023 05:23:28 +0000

Lately I’ve been twiddling with a few sufficiently complex Hugo themes and thought I’d share an interesting approach for rendering small changes quickly. In absence of a better term — “incremental writes” will do and this method works best when there’s a lot happening inside a theme that renders in the 1,000 ~ 10,000+ and beyond page range.

So what’s Hugo and why is it great? Well it’s the ultimate static site generator (SSG) that just gets things done. The reason it’s great is due to its build and install story along with its abstraction of things.

The code discussed here isn’t necessarily intended to be used with hugo server because it has its own fast rendering mode that works well enough if a theme is adequately simple and well written. This is more of an approach for rendering and previewing a large, complex, and messy theme quickly with just the plain hugo command. My hugo version is v0.108.0.

$ hugo version
hugo v0.108.0+extended linux/amd64 BuildDate=unknown

Overview of Site Rendering

Before going into the details of making hugo write files incrementally it’s useful to have a simple idea of the minimum requirements of a theme and the way a site is rendered.

|-- config.json
|-- config.toml
|-- config.yaml
|-- content
|   `-- markdown
|       |-- first.md
|       |-- second.md
|       `-- third.md
`-- themes
    `-- base
        `-- layouts
            `-- _default
                `-- baseof.html
                `-- index.html

The directory structure above along with one of the basic configuration formats are all that’s required to render a theme. Inside the content folder there’s the Markdown source and inside the themes folder are the layout files. Below is a minimal config.yaml that disables specific outputs because for this explanation the home and page outputs are the only concern.

---
baseURL:
theme: base
title: Base
languageCode: en-us

paginate: 10
summaryLength: 1

taxonomies:
  tag: tags

disableKinds:
- sitemap
- term
- section
- taxonomy

outputs:
  home:
  - html
  section:
  - html
  taxonomy:
  - html
  term:
  - html
  page:
  - html

Default formats are output “kinds” that are split across home, section, taxonomy, term, and page. This separation avoids thinking about redundant logic and common repetitive patterns. Home pages are rendered by index.html, sections by section.html, taxonomies by taxonomy.html, terms by term.html and pages by single.html.

Kinds	Layouts
`home`	`_default/index.html`
`section`	`_default/section.html`
`taxonomy`	`_default/taxonomy.html`
`term`	`_default/term.html`
`page`	`_default/single.html`

Base layouts are the authoritative boilerplate that markup and templates of different kinds (home, section, taxonomy, term, page) extend and render. This is a common approach for most frameworks when smashing together HTML (Hypertext Markup Language). The ideal scenario is to only extend base layouts and keep visual interpolation and noise to a minimum.

`-- themes
    `-- base
        `-- layouts
            `-- _default
                `-- baseof.html
                `-- baseof.json
                `-- baseof.xml
                `-- baseof.txt

For HTML and other boilerplate if there’s confusion over what type or kind is in use, then appending that information to the markup or data in a non–obstructive way works wonders. A block opens up a space inside a base template and a define elsewhere fills in or extends that space with the desired content.

<!DOCTYPE html>
<html
  lang="en-us"
  data-kind="{{ .Page.Kind }}"
  data-type="{{ .Page.Type }}"
>
  <head>
    <title>
      {{- block "title" . -}}
       {{ .Site.Title }}
      {{- end -}}
    </title>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
  </head>

  <body>
    {{- block "main" . -}}
      <p>
        This is what you see if you don't
        extend the base template's main block.
      </p>
    {{- end -}}
  </body>
</html>

Incremental Writes

Writing output incrementally exploits the way hugo evaluates base templates. The whole site still has to be read but like a novel reading is faster than writing — and the computer is not much different. So the speed up here makes hugo mostly read things and not write them. It just so happens that if a base template for a particular kind evaluates to nothing well then — hugo does nothing.

Multiple conditions exist for choosing which files to process and write but the most obvious one compares the modification dates of the source markdown with the resulting index.html in the folder. A function template below checks the modification time using the page’s context as input and returns its modified state as a true or false output.

{{- $input := . -}}
{{- $pageContext := $input -}}

{{- $markdown := print "content/" $pageContext.File -}}
{{- $markdownModTime := "" -}}
{{- $page := print "public/" (strings.TrimPrefix
      $pageContext.Page.Site.BaseURL
      $pageContext.Page.Permalink
    ) "index.html"
-}}
{{- $pageModTime := "" -}}

{{- if fileExists $markdown -}}
  {{- $markdownModTime = (os.Stat $markdown).ModTime -}}
{{- end -}}

{{- if fileExists $page -}}
  {{- $pageModTime = (os.Stat $page).ModTime -}}
{{- end -}}

{{- $modified := gt $markdownModTime $pageModTime -}}

{{- $output := or $modified (in (slice
    "home"
    "section"
    "taxonomy"
    "term"
  ) $pageContext.Page.Kind)
-}}

{{- return $output -}}

One of the upsides to this particular function is that it also allows turning off writes for particular parts of the site with an in slice check against the current page’s kind.

{{- $modified := partial "function-page-modified.html" . -}}

{{- if $modified -}}
<!DOCTYPE html>
<html
  lang="en-us"
  data-kind="{{ .Page.Kind }}"
  data-type="{{ .Page.Type }}"
>
  <head>
    <title>
      {{- block "title" . -}}
       {{ .Site.Title }}
      {{- end -}}
    </title>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />

    {{- $default := resources.Get "css/default.css" -}}
    <link title ="Default" rel="stylesheet" href="{{ $default.Permalink }}" />
  </head>

  <body>
    <main class="container">
      {{- block "main" . -}}
        <p>
          This is what you see if you don't
          extend the base template's main block.
        </p>
      {{- end -}}
    </main>
  </body>
</html>
{{- end -}}

Absolutely no stray characters or must render in the logic surrounding the base template — when a page is unmodified nothing should evaluate. The dash (-) in the template logic removes leading {{- and trailing -}} white–space. Repeat for every base template type. The clip below shows a demonstration of this function with 20,000 pages.

Sorry, your browser doesn't support embedded videos. Here is a link to the video instead.

In this clip the home page is always rebuilt plus any other recently edited and non–existent pages. All other previous index.html outputs are left untouched.

See this repository for trying out different numbers and adjusting the complexity.

Applications and Utility

So how in the world is this useful? Mostly in the case of a theme that executes lots of work. An upper bound of 20,000 pages for the device in the above clip take around 5 seconds hot plus write time for a single modified page change even in the context of heavy image processing, bundling, and data unmarshalling.

In other words no matter the complexity on previously rendered pages editing, previewing, and building a new page will take at most ~5 seconds because that’s how long it took my device to read all the source markdown.

It’s also important to know that the code paths for hugo and hugo server are different and the server will likely assume that the previously generated pages don’t exist even if they are in the public folder. If this post makes any sense — you’re probably using another web server anyway.

Finally be aware that care must be taken with shortcodes. Even shortcodes that are not in use are evaluated immediately at run time — this is an easy way to make builds slow. Keep them as simple as possible.

Conclusion

More could be said in regards to applying incremental switches especially to pagination logic but my time was too limited to think that one through. The goal here however is to avoid diving too far into incremental shenanigans and instead find a good cut off point for archive rotation while still having fast previews on pages when running the hugo command on a beefy theme. The cut off could be 10,000 pages or even 100,000+ pages — more or less depending on a theme’s template efficiency, disk I/O (Input/Output), and device memory.

Extensible Stylesheets

Wed, 30 Nov 2022 19:57:00 +0000

The browser has a peculiar dimensionality that natively supports moving around complexity between different data and language domains. One of those domains is XML (Extensible Markup Language) and it comes practically in the form of RSS (Really Simple Syndication) Feeds, Atom Feeds, Site Maps, OPML (Outline Processor Markup Language) Outlines and other various XML representations.

XSLT (Extensible Stylesheet Language Transformations) is a that transforms XML into different output formats. You can expose and transform big or tiny blobs of atom.xml, rss.xml, sitemap.xml, and opml.xml files into presentation formats.

In some respects, XSLT is considered “dead” technology, but take the word dead with a of salt. It’s not uncommon to right click a site, and lo and behold see a DTD (Document Type Definition) for XHTML (EXtensible HyperText Markup Language) in its source generated from XML — you’d be surprised.

XSLT operates as an XML templating language and a rather verbose one at that. If you go deep enough, the verbosity gets unwieldy and like all programming shenanigans it’s a perpetual rabbit hole. Here’s my practical notes for working with XML and XSLT in a web context for adding style and presentation while maintaining a bit of sanity.

Formats, File Extensions, and MIME types

The XSLT transformations discussed here will be limited to XHTML output. Raw XML in the browser has no styles associated with it (example) so styles are added with XSLT (example).

The MIME (Multipurpose Internet Mail Extensions) type definition for XSLT is application/xslt+xml. A file extension ending in .xsl or .xslt is the commonly accepted and used form.

The mimetype definition for XHTML is application/xhtml+xml but it’s usually served using the text/html content type for browsers to assume HTML instead of XML parsing. XHTML has from HTML which you can take a look at in this XHTML in a nutshell article.

XML Validation and Formatting

You can validate and check an XML document for well formedness using xmllint from the libxml2 W3C (The World Wide Web Consortium) offers an online feed validation service, but an offline validator sets up a better feedback loop and is a lot more robust and

XML has multiple validation grammars in the form of schemas. RELAX NG (REgular LAnguage for XML Next Generation) is one of those schema language formats. Schema examples can be found in RFCs (Request for Comments) or in niche places around the web — for example here’s a RSS rng file, an ATOM rnc file, and an ATOM rng file. The catch is that these validation schema files may have differing use cases or may be out of spec due to time, but they’re still worth looking at.

RELAX NG has both a standard xml.rng syntax and a compact xml.rnc syntax. Offline validation with xmllint does not rnc compact schema syntax — but rng works. Schema between rnc and rng can be achieved with the Java program trang (usually goes by the name jing-trang in package repositories).

Trang: Trang converts between different schema languages for XML. RELAX NG (XML syntax), RELAX NG compact syntax, XML 1.0 DTDs and W3C XML Schema (WXS).

In my case, and maybe yours, it’s easier to run trang on an already well specified and well formed XML document. This produces a basic rng schema file for validation and adding more rules.

trang rss.xml rss.rng
trang atom.xml atom.rng
trang opml.xml opml.rng

Validate XML using the rng file with xmllint and the --relaxng flag. The --noout flag disables printing the output to the command line.

$ xmllint --noout --relaxng rss.rng rss.xml
rss.xml validates

If it fails to validate it will return the error message defined by the schema’s grammar.

$ xmllint --noout --relaxng rss.rng rss.xml
rss.xml:25: element description: Relax-NG validity error : Did not expect element description there
rss.xml fails to validate

Pretty print XML with --pretty 1 for basic formatting or --pretty 2 for “one attribute per line” white space formatting.

xmllint --pretty 1 rss.xml
xmllint --pretty 2 rss.xml

Stylesheet Processing and Validation

The command line XSLT processor xsltproc can be used to process stylesheets offline and works only on stylesheets up to version 1.1. If using xsltproc as a validation tool for xsl files, you’ll have to downgrade the version declaration from version 3.0 to version 1.1 and a few features.

xsltproc rss.xsl

xsltproc rss.xsl rss.xml

Other processors like Xalan–Java supports XSLT up to version 1.0 and Saxon up to version 3.0.

Stylesheet Boilerplate and Transformations

Below is one variation of a stylesheet that transforms XML to XHTML. A typical XHTML document skeleton is embedded within along with XSLT elements for processing and transformation.

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:atom="http://www.w3.org/2005/Atom"
  xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd"
  xmlns:dc="http://purl.org/dc/elements/1.1/"
  version="1.1"
>
  <xsl:output method="html" version="1.0" encoding="UTF-8" indent="yes"/>
  <xsl:template match="/">
    <html xmlns="http://www.w3.org/1999/xhtml">
      <head>
        <title>XHTML Document</title>
        <meta
          http-equiv="Content-Type"
          content="text/html; charset=utf-8"
        />
        <meta
          name="viewport"
          content="width=device-width, initial-scale=1, maximum-scale=1"
        />
      </head>
      <body>
      </body>
    </html>
  </xsl:template>
</xsl:stylesheet>

In the above, namespace attributes in the form xmlns:itunes extend the document. You could think of them as imports for extending features and avoiding naming conflicts. The URL points to the “allowed” specified by the namespace.

For example, the Atom Activity Streams namespace could be added under xmlns:activity and extend the stylesheet with an understanding of Activity Streams related vocabulary. Namespaces can also be used to extend processing instructions like xmlns:xsl for XSLT processors that support them.

<xsl:stylesheet
  xmlns:activity="http://activitystrea.ms/specs/atom/1.0/"
  version="1.1"
>

<activity:verb>post</activity:verb>

Drop the xsl stylesheet inside a XML document with the xml-stylesheet declaration and the browser handles the rest.

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<?xml-stylesheet href="/rss.xsl" type="text/xsl"?>

XSLT works in conjunction with XPath (the XML Path Language) and is somewhat similar to CSS (Cascading Style Sheets) selectors. Command line programs like xmlstarlet make use of XPath expressions for selecting data from parts of an XML document.

<xsl:value-of select="/rss/channel/atom:link[@rel='previous']/@href"/>

The XPath expression from the select attribute above gets the href value from the <link> tag in the atom namespace which is equal to https://example.com/page/2/rss.xml.

<atom:link rel="next" href="https://example.com/page/2/rss.xml" />

If you’re familiar with CSS, then you’re in luck. Cheat sheets for XPath are everywhere across the Internet in a “CSS to XPath” format. Test expressions locally with xsltproc, xmlstarlet or an online Xpath expression test bed.

rss > channel > link[rel="previous"][href] {
  display: inline;
}

Value selections, for loops, and, switch statements are the more commonly used XSLT elements.

Attributes and Value Selection

Create attributes with the xsl:attribute element. The attributes are added to the parent tag. Select values with the xsl:value-of element.

<a>
  <xsl:attribute name="href">
    <xsl:value-of select="/rss/channel/atom:link[@rel='next']/@href"/>
  </xsl:attribute>
</a>

<!--  Output: <a href="https://example.com/page/2/rss.xml"></a> -->

<img>
  <xsl:attribute name="alt"><xsl:value-of select="/rss/channel/category"/></xsl:attribute>
  <xsl:attribute name="title"><xsl:value-of select="/rss/channel/category"/></xsl:attribute>
  <xsl:attribute name="src"><xsl:value-of select="/rss/channel/image/url"/></xsl:attribute>
</img>

<!--  Output: <img alt="image" title="image" src="/image"></img> -->

For Each

A typical for each construction executes over a range of XML tags with the xsl:for-each element.

<xsl:for-each select="/rss/channel/item">
  <h2>
    <xsl:value-of select="title" />
  </h2>
</xsl:for-each>

Switch Statements

A switch statement construction is executed with a combination of the xsl:choose, xsl:otherwise, and xsl:when elements. The test attribute on xsl:when contains the condition.

<xsl:choose>

  <xsl:when test="/rss/channel/atom:link[@rel='previous']/@href">
    <xsl:attribute name="href">
      <xsl:value-of select="/rss/channel/atom:link[@rel='previous']/@href"/>
    </xsl:attribute>
  </xsl:when>

  <xsl:otherwise>
    <xsl:attribute name="href">/</xsl:attribute>
  </xsl:otherwise>

</xsl:choose>

View the XSLT elements and function reference for the complete list of instructions.

Conclusion

There you have it — a basic overview and approach to working with XML, XSLT, and XHTML.

Making Web Pages

Thu, 29 Sep 2022 08:26:21 +0000

Introduction

A recent look at my web server configuration revealed that my error pages were set up in an odd way. From the base route of a domain, it’s possible to iterate over each error page starting with /400, /401, /402, /403, /404 and so on. I guess that my self wanted to verify an error page’s content and existence.

It did seem like more was going on but upon further investigation, these error pages were just statically generated HTML (HyperText Markup Language). They were made with HTML and XML (Extensible Markup Language) utilities many moons ago.

Just as there are too many books to read in a lifetime, there are too many programs to try out in a lifetime. If you look online it’s easy to get the impression that making a website is rocket science. The reason for this is obvious — almost all discussions online are oriented towards engineers working in the industry. The reality is that a website or even an “application” can be as simple as a single HTML text file.

The blessed ways of crafting and delivering a website have become beyond the pale complicated, but composing HTML is still as simple as it always was, and perhaps even easier thanks to some improvements in the HTML specification.

HTML and XML Utilities

HTML and XML utilities or html-xml-utils are a simple set of programs for manipulating HTML and XML files. Here’s the master list that describes the purpose of each utility taken right from the readme.

asc2xml      -  convert from UTF-8 to &#nnn; entities
hxaddid      -  add IDs to selected elements
hxcite       -  replace bibliographic references by hyperlinks
hxcite-mkbib -  expand references and create bibliography
hxclean      -  apply heuristics to correct an HTML file
hxcopy       -  copy an HTML file while preserving relative links
hxcount      -  count elements and attributes in HTML or XML files
hxextract    -  extract selected elements
hxincl       -  expand included HTML or XML files
hxindex      -  create an alphabetically sorted index
hxmkbib      -  create bibliography from a template
hxmultitoc   -  create a table of contents for a set of HTML files
hxname2id    -  move some ID= or NAME= from A elements to their parents
hxnormalize  -  pretty-print an HTML file
hxnsxml      -  convert output of hxxmlns back to normal XML
hxnum        -  number section headings in an HTML file
hxpipe       -  convert XML to a format easier to parse with Perl or AWK
hxprintlinks -  number links & add table of URLs at end of an HTML file
hxprune      -  remove marked elements from an HTML file
hxref        -  generate cross-references
hxselect     -  extract elements that match a (CSS) selector
hxtoc        -  insert a table of contents in an HTML file
hxuncdata    -  replace CDATA sections by character entities
hxunent      -  replace HTML predefined character entities to UTF-8
hxunpipe     -  convert output of pipe back to XML format
hxunxmlns    -  replace "global names" by XML Namespace prefixes
hxwls        -  list links in an HTML file
xml2asc      -  convert from &#nnn; entities to UTF-8

These 28 programs (primitives) allow you to do a lot of magic with HTML (and XML). If you’re on a Linux distribution this package exists on Debian, Arch, Alpine and others as html-xml-utils.

Basic Templating

The web was initially designed with the purpose of passing documents around. Below is a modern skeleton of a basic document.

<!DOCTYPE html>
<html lang="en">
  <head>
    <title>The Document</title>
    <meta charset="utf-8" />
    <meta
      name="viewport"
      content="width=device-width, initial-scale=1"
    />
  </head>
  <body>
  </body>
</html>

The document type is HTML with a language attribute set to English, the <head> includes the title, character set and viewport while the <body> begins the rest of the document and ends it with a closing </body>.

The package html-xml-utils has a program hxincl that allows augmenting that base index.html document. Here’s what a rough template of those error pages look like using the inclusion directives of hxincl.

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta
      name="viewport"
      content="width=device-width, initial-scale=1"
    />
    <style>
      <!-- include "index.css" -->
    </style>
  </head>
  <body>
    <article>
      <section>
        <h1>
          <code>
            <b>
              <!-- include "error.code.html" -->
            </b>
          </code>
          <!-- include "error.title.html" -->
        </h1>
      </section>
      <p>You can try:</p>
      <ul id="action">
        <li>
          Going back to the
          <a onclick="window.history.go(-1); return false;" href="/">
            previous page.
          </a>
        </li>
        <li>Returning <a href="/">to the home page.</a></li>
      </ul>
    </article>
    <script>
      <!-- include "index.js" -->
    </script>
  </body>
</html>

Running hxincl on this template expands the contents of the included file directives which are conveniently just HTML comments. It expands into a final artifact: the error code value and title, along with a traditional separation of concerns for CSS (Cascading Style Sheets) as index.css and JavaScript as index.js. The source code of an error page shows the full expansion.

hxincl -f -x template.html

The -f flag removes all comments after the expansion and the -x flag enforces XML conventions when generating void elements. The program hxnormalize can be used to format and pretty print the final HTML document.

Conclusion

This was a very simple demonstration and hxincl can be used much more cleverly with substitutions. There’s no props, slots, or template inheritance but I thought it’d be rather nice to demonstrate that you don’t need complex tools to start generating HTML.

A JavaScript Executable

Tue, 30 Aug 2022 01:11:38 +0000

There’s an interesting command line browser called Lynx. It’s an extremely useful and versatile text based browser. In lynx you can dump all links on a web page as references. That’s neat.

$ lynx -dump -listonly https://example.com/

References

   1. https://www.iana.org/domains/example

I’ve been meaning to replicate this feature, generate a list of references and stick them at the end of each post. This would allow counting the number of links on a page, and visually or automatically checking for dead links (link rot) in the future.

Lynx: A fully–featured World Wide Web (WWW) client for users running cursor–addressable character–cell display devices — Lynx Users Guide

Let’s make a quick and dirty command line program that replicates the reference list feature of lynx, but to HTML instead of text.

JavaScript with Deno Runtime

JavaScript is the master at traversing the DOM (Document Object Model). Pulling down a list of links is easy with native browser JavaScript APIs (Application Programming Interfaces). Deno is a versatile runtime for JavaScript and TypeScript that sits at just the right primitives and abstractions that allow for a variety of use cases. Using Deno, we’ll compile a simple command line program written in JavaScript that generates references. The output of my deno --version is 1.23.0.

$ deno --version
deno 1.23.0 (release, x86_64-unknown-linux-gnu)
v8 10.4.132.5
typescript 4.7.2

File and Directory Setup

The program will be exoference. The source directory will contain the entry–point main.ts, the TypeScript compiler option manifest tsconfig.json, and eventually a compiled binary that assumes the name of the parent

exoference/
  |__ exoference
  |__ main.ts
  |__ tsconfig.json

Running the script inside main.ts is usually achieved with deno run.

deno run --allow-net --config tsconfig.json main.ts

Compiling a program into a runnable or binary uses deno compile.

deno compile --allow-net --config tsconfig.json main.ts

The tsconfig.json manifest configures compiler options for TypeScript which provides type checking. The following is a slightly relaxed tsconfig.json that imports browser DOM API libraries.

{
  "compilerOptions": {
    "noFallthroughCasesInSwitch": true,
    "noImplicitAny": false,
    "noImplicitReturns": true,
    "noImplicitThis": true,
    "noUncheckedIndexedAccess": true,
    "noUnusedLocals": false,
    "noUnusedParameters": false,
    "strict": true,
    "strictBindCallApply": true,
    "strictFunctionTypes": true,
    "strictNullChecks": false,
    "strictPropertyInitialization": false,
    "lib": [
      "dom",
      "dom.iterable",
      "dom.asynciterable",
      "deno.ns"
    ]
  }
}

Specifics & Details

Since this is supposed to be a command line program, the first thing is setting up a boilerplate. The program name, version, help flags, and help function are declared.

const program = "exoference";
const version = "0.0.1";
const helpFlags = ["-h", "-help", "--help"];
const help = () => {
  return `
Usage: ${program} [FLAGS]... [ARGUMENTS]...

The program ${program} shall generate a list of anchor reference links
as partial HTML output.

  Command List:

    ${program} https://example.com           Dump anchor reference links from
                                             specified URL to HTML output.

    ${program} --pretty https://example.com  Dump anchor reference links from
                                             specified URL to text output.

    ${program} --help                        Show this help menu.

Version: ${version}
`.trim();
};

Import DOMparser for DOM traversal. The references function accepts a URL (Uniform Resource Locator) and finds all elements that are anchors on the page. The URL API adds implicit formatting checks to the input URL.

The second argument resolves relative URLs and completes them with the base. Duplicates are removed by briefly turning the references array [] into a set {}. The spread operator expands the items (iterables) inside the set back into an array.

import { DOMParser } from "https://deno.land/x/deno_dom@v0.1.31-alpha/deno-dom-wasm.ts";

const references = async (address) => {
  const url = new URL(address);
  const page = await fetch(url.href);
  const dom = new DOMParser().parseFromString(await page.text(), "text/html");
  const anchors = dom.getElementsByTagName("a");
  const references = [];

  for (const anchor of anchors) {
    references.push(new URL(anchor.attributes.href, url).href);
  }

  return [...new Set(references)];
};

The html function returns partial HTML (HyperText Markup Language) to the console as an ordered list of references.

const html = (references) => {
  console.log("<ol>");
  for (const reference in references) {
    console.log(`<li><a href="${references[reference]}">${references[reference]}</a></li>`,);
  }
  console.log("</ol>");
};

The pretty function replicates the -dump and -listonly arguments of lynx and returns a list of references as plain text. Command line flags allow switching to this function.

const prettyFlags = ["-p", "-pretty", "--pretty"];
const pretty = (references) => {
  console.log("References", "\n");
  for (const reference in references) {
    console.log((parseInt(reference) + 1) + ".", references[reference]);
  }
};

The main function handles switching between each case by reading the first, and second arguments along with the argument length. The function switches based on the helpFlags and prettyFlags array in addition to the argument position.

const firstArgument: string = Deno.args[0];
const secondArgument: string = Deno.args[1];
const numberOfArguments: number = Deno.args.length;

const main = async (numberOfArguments) => {
  if (numberOfArguments === 0) return console.log(help());
  if (helpFlags.includes(firstArgument)) return console.log(help());
  if (prettyFlags.includes(firstArgument)) {
    if (new URL(secondArgument)) {
      return pretty(await references(secondArgument));
    }
  }
  if (new URL(firstArgument)) return html(await references(firstArgument));
  return console.log(`Unkown argument ${Deno.args[0]}`);
};

main(numberOfArguments);

This is good enough to compile the and output a list of references for this article.

Sorry, your browser doesn't support embedded videos. Here is a link to the video instead.

A quick run

Conclusion

This might be a very simple starting point, but the opportunities for more features are endless. Filtering, automatic link rot detection, and caching the reference sources are just a small subset of possible features.

Literate Programming

Wed, 29 Jun 2022 22:29:50 +0000

Literate Programming is a documentation first programming style pioneered by Donald Knuth. It’s writing a blog post as a light specification but with the added bonus of producing a runnable program at the end.

But this isn’t a post about my hacked together attempts at literate programming — that won’t be very interesting. This is more of a small research gathering exercise on the different literate programming tools and workflows.

Ideal

In my opinion, literate programming is still an incomplete paradigm. The biggest drawback comes in the form of programming resistance. This resistance comes mostly from the fact that a problem space is unknown ahead of time and so, as the story goes — the implementation becomes the specification.

That’s the norm, but ideally anyone should be able to jump into a source tree, make changes, and reconcile those changes later on correctly with some sort of generated readable documentation or specification. The goal is not perfection but accurate agility, and the tooling should be super simple. What would that look like? Who knows, so here’s what I think a more complete or “agile” literate programming workflow might entail.

The workflow should facilitate calculating and visualizing sources of documentation drift. Difference in the specification (documentation) and implementation (source code) should be easy to reconcile through varied means. This could be by the seat of your pants, automated diffing and fuzzing algorithms between the source and doc, or from signals that inform specificity and constraints. Test driven development and type systems are useful signals but are exposed as low level source code rather than high level human readable constraints.
The initial documentation entry point should not matter. Edit source files directly or enter top down through an overarching literate system. If documentation drift is easy to identify then there is flexibility to experiment “freely” with the implementation. Primitives could be at a singular source file level and ideally any source file can act as an entry point for jump starting a code repository’s documentation in part, or in whole.
Literate workflows should facilitate producing documents in whole or in part as discrete outputs based on section, topic, idea, or source file. This can be in any output format such as PDF (Portable Document Format), HTML (HyperText Markup Language), or Markdown. This is important because sometimes only select portions of a program are worth documenting or reading.
Ideally multiple formats should be available as inputs and outputs. Markdown is popular but there are a range of other markup document languages that offer different advantages. reStructuredText, AsciiDoc, and Troff/Groff are a few. The shortcut approach is to leverage Pandoc, a universal document converter.

There are nigh infinite literate programming tools and workflows. I’ve tried some of them, but not in any meaningful way to write about at great length.

Literate Programming Tools and Programs

The literate programming workflow is based on a simple tangle and weave process. The literate source file contains splices of code chunks with their accompanying explanations. The code chunks are tangled to construct a complete source file or executable. The explanations around the chunks are weaved to create a document that fully explains the program.

Web [pdf]: Web is Donald Knuth’s system for literate programming. Tangling and weaving macros process different parts of a web file to produce multiple outputs. There’s cweb, nuweb, and noweb.

Documented LaTeX Files [pdf]: LaTeX Packages from CTAN (Comprehensive TeX Archive Network) achieve a literate style by weaving and tangling documentation and code with the doc and docstrip packages. Package code is mixed in with commented typesetting inside a documented LaTeX file (dtx).

Babel: Babel converts Emacs’ Org Mode into a powerful workflow more suitable for literate programming. Babel is popular in some circles and has numerous use cases including developer operations and general system crafting and architecture.

Tsodings’ lit: A simple literate implementation based on the Literate Haskell approach. The literate program reads a document written in LaTeX and comments out every line not contained in a \code{} section block. The output is an executable source file and the literate document takes advantage of LaTeX’s output formats.

Zyedidia’s Literate: A literate system based on Markdown. The literate document is written inside .lit files and outputs to HTML and CSS. Since it outputs directly to HTML, there’s a lot of leverage in refining and adapting the output.

Jupyter Notebook: A more popular kind of simple literate programming in the form of literate “computation”. These are interactive GUI (Graphical User Interface) documents for explaining and running code from a single view.

lit.sh: A literate programming preprocessor in pure shell because why not?

Conclusion

Literate programming is difficult and hasn’t caught on because in the majority of cases, programs are built with the specification made up along the way. Put a customer in the mix and your specification changes randomly. This is the status quo because often the range of inputs are unknown along with a full scope of the problem space.

Quantifying documentation drift and adding flexibility for limited but accurate documentation could assist in making literate programming much easier. Testing and type systems are becoming much more robust — perhaps Donald Knuth was just a tad bit too early for literate programming to fully catch on? My hunch is that if someone or a company figures this out, it will be super simple to use and would look like git but for documentation.

In terms of popularity, the most used literate programming tools seem to be Babel and Noweb.

Writing Strategy

Thu, 05 May 2022 09:23:57 +0000

My micro blog homepage.

If you’ve visited my repositories recently you might have noticed there’s new code on the block. My writing strategy is going to change a bit with the start of my brand new micro blog. If you’d like to read more about the technical details, check out my personal projects article on the stack.

The micro blog concept has become very popular. Long form content is great but many find it easier to ruminate on short posts that are much more palatable. In my case, I’ve been timid on writing long form web development related articles. Web development is cumbersome, fast moving, nuanced, and rather complicated. I’d like to instead have an escape chute for writing shorter posts that are easier to maintain, while still being able to write long form charades here and there.

Hugo is Good

Wed, 30 Mar 2022 21:21:29 +0000

Hugo is a static site generator program and a mighty good piece of free software. This blog has been mostly powered by hugo since about 4 years ago beginning with version 0.44. Usually, my summary description of hugo is that it’s a blog templating system, but hugo has enough generality and versatility to be called a data templating system.

Lately, I’ve been catching up to some of its newer features, and after revisiting conundrums like the smart quotes edge cases — it’s great to see a lot of interesting details resolved. The Goldmark Markdown is clean.

If you’re early adopting software, sometimes you’ve got to take a primitive approach to using its interfaces since future changes may catch up to you in degrees. I’m now on hugo version 0.94.2 and have extricated myself from a myriad of hacks with effort. Writing posts in markdown was already easy, but now it’s even easier?

Let’s take a look at features this random blog uses or could use from Hugo’s feature set. In order not to make this too long, my micro blog hugo theme which uses semantic HTML (HyperText Markup Language) explores more of the specific features discussed below.

Markdown

The text file is king. Form separated from content, and content edited from a plurality of user interfaces — graphical or otherwise is hard to beat in my opinion. This article’s single source of truth is a Markdown text file that is transformed. You can the raw text file that my system transforms to see the underlying markdown format for this article.

Diagrams

If desired, diagrams can be drawn on the fly as an ASCII (American Standard Code for Information Interchange) representation using the GoAT — the Go ASCII tool built into hugo. The ASCII representation is rendered to SVG (Scalable Vector Graphics). You can also shim in the more popular Mermaid diagrams using render code block hooks and JavaScript includes.

Drawing this for the first time was an interesting fiddle and dance with the vim editor. There are tools like ASCIIFlow (repository) that makes this sort of thing quicker, but Emacs in artist mode is probably the way.

      +-------------------.                       .->  thedroneely.com/posts/hugo-is-good/          
     |        hugo         |     .-->  public ---+                                                  
      '---------+---------'     |                 '->  thedroneely.com/images/placeholder.png/      
                |               |                                                                   
    .-----------.------------.  |                                                                   
   |            |             |  '---.                                                              
   |            |             |       |              .---- baseof.html --->  "Base Template"        
config.toml  config.yaml  config.json |             |                                               
   |            |             |       |             +----- single.html --->  "Article Content"      
   |            |             |       |             |                                               
    '-----------+------------'         '-.          +----- section.html -->  "Article Card"         
                |                         |     .---+                                               
  .------------' '--------------------+---+.   |    +----- list.html ----->  "List of Article Cards"
 |                                       .| |  |    |                                               
 +--->  content/posts/hugo-is-good.md --->| |  |    +----- rss.xml ------->  "RSS Feed"            
 |                                        | |  |    |                                               
 ++-->  resources/_gen --------------.    | |  |     '---- sitemap.xml --->  "Site Map"             
 |                                    |   | |  |                                                    
 +'-->  assets/images/placeholder.png  '->| |  |                                                    
 |                                        ^ |  |                                                    
 +--->  themes/tdro/layouts/_default  <---+-+-'                                                     
 |                                          |                                                       
  '----------------------------------------'

Configuration Formats

Hugo makes use of TOML (Tom’s Obvious Minimal Language), YAML (YAML Ain’t Markup Language), and JSON (JavaScript Object Notation) as configuration formats for itself and its front matter data. YAML is probably the most popular format but hugo favours .

In the front matter, TOML uses +++ delimiters, YAML, --- delimiters, and JSON {} delimiters. The typical hugo configuration can configure many options, and you can add pretty much anything to the config.toml, config.yaml, or config.json file.

---
baseURL:
theme: base
title: Base
languageCode: en-us

author:
  name: Opulene

menu:
  main:
  - name: Home
    url: /
    weight: 1
  - name: Data
    url: /data/
    weight: 2
  - name: Tags
    url: /tags/
    weight: 3
  - name: Archives
    url: /archives/
    weight: 4
  - name: JSON Feed
    url: /index.json
    weight: 5
  - name: RSS
    url: /rss.xml
    weight: 6

paginate: 1
summaryLength: 20

markup:
  highlight:
    anchorLineNos: true
    codeFences: true
    guessSyntax: true
    lineNos: false
    lineNumbersInTable: false
    noClasses: false
    noHl: false

taxonomies:
  tag: tags

outputFormats:
  html:
    baseName: index
    mediaType: text/html
  json:
    baseName: index
    mediaType: application/json
  rss:
    baseName: rss
    mediaType: application/xml

outputs:
  home:
  - html
  - rss
  - json
  section:
  - html
  - rss
  taxonomy:
  - html
  - rss
  term:
  - html
  - rss
  page:
  - html

By default, hugo sets up logic for tag and category taxonomies but tags are usually good enough for most people. Categories can be hidden with the taxonomies key as seen in the above code block by only showing tags.

Code Fences and Syntax Highlighting

Speaking of code blocks, hugo allows for a lot of flexibility when rendering code syntax highlighting. Hugo uses Chroma for syntax highlighting which is based on the Pygments Python syntax highlighter. Chroma supports a wide variety of languages and you can highlight specific lines.

---
markup:
  highlight:
    anchorLineNos: true
    codeFences: true
    guessSyntax: true
    lineNos: false
    lineNumbersInTable: false
    noClasses: false
    noHl: false

You can even configure hugo to generate class based tokens on the HTML output and use a separate CSS style sheet to colorize the tokens. No need to pull out the JavaScript hammer for code syntax highlighting.

Short Codes

Shortcodes allow for creating micro templates that are throughout a markdown text file. Hugo provides a set of built–in shortcodes for popular sites around the web. If you want, you could embed a GitHub Gist with a short code.

Render Hooks

Render hooks let hugo change specific markdown rendering behavior. For example, HTML output for section headings can be changed by altering the markup in layouts/_default/_markup/render-heading.html. This is a handy feature for controlling specific HTML output across the entire layout.

<h{{ .Level }} id="{{ .Anchor | safeURL }}">
  <a
    title="{{ .Text | safeHTML }}"
    href="#{{ .Anchor | safeURL }}">
    {{ .Text | safeHTML }}
  </a>
</h{{ .Level }}>

Output Formats

The custom output format is the most important feature in my opinion. If you take a look at the configuration file above, you’ll see that hugo can be parked onto multiple data types. HTML is not the only format you can work with — it can be anything. For example, it’s not too hard to get a bit barbaric and forge a JSON using Hugo’s extension of the go template at layouts/_default/index.json.

{
  "version": "https://jsonfeed.org/version/1.1",
  "title": "{{ .Site.Title }}",
  "home_page_url": "{{ .Site.BaseURL }}",
  "feed_url": "{{ .Site.BaseURL }}/index.json",
  "items": [
    {{- range $index, $data := where .Site.RegularPages ".Params.hidden" "!=" "true" -}}
    {{- if ne $data.Type "json" -}}
    {{- if and $index (gt $index 0) -}},{{- end }}
    {
      "id": "{{ md5 $data.Permalink }}",
      "url": "{{ $data.Permalink }}",
      "title": {{ $data.Summary | htmlUnescape | jsonify }},
      "summary": {{ $data.Summary | htmlUnescape | jsonify }},
      "date_modified": "{{ $data.Date | time.Format "2006-01-02T15:04:05Z" }}",
      "date_published": "{{ $data.PublishDate | time.Format "2006-01-02T15:04:05Z" }}",
      "_metadata": {
        "slug": "{{ $data.Slug }}",
        "type": "{{ $data.Type }}"
      },
      "author": {
        "name": "{{ .Site.Author.name }}"
      },
      "tags": [
        {{- range $tags, $tag := $data.Params.tags -}}
          {{- if $tags -}}
          ,
          {{- end -}}
          "
          {{- $tag | htmlEscape -}}
          "
        {{- end -}}
      ],
      "content_text": {{ $data.Plain | jsonify }},
      "content_html": {{ $data.Content | jsonify }}
    }
    {{- end -}}
    {{ end }}
  ]
}

Now my site now has a JSON Feed outlet that could be useful on the client side.

Headless Content Management

Static site generators in general offer the flexibility of using any administrative or presentation interface as the front end. This decoupling of the front from the back is sometimes called a headless content management system and in the wider marketing sphere, “JAMStack”.

For example, this article can be comfortably edited using the editor or through a based content management system like Cockpit CMS. If you’re clever enough, you can contort anything to act as your “headless” CMS (even utterly horrific content management systems (CMS) like WordPress) since a headless CMS at its source exposes an extremely flexible API (Application Programming Interface). A lot of headless CMS options are out there.

Conclusion

Hugo is a good static site generator, and has a lot of interesting use cases — hopefully, at some point I’ll take a closer look into hugo modules. Hugo modules tap into the general go module ecosystem, allowing you to mount the contents of modules as file directories or import them for use inside your project. Check out this article on the hugo module system.

Web Browsers

Tue, 08 Feb 2022 06:56:52 +0000

Web browsers, search engines, and the open source technologies underpinning them have more or less bootstrapped what could be called a pseudo home shopping network (HSN). In many respects, the dominant language, behaviour, and environment on The Internet mirrors that of a tightly controlled, but free-to-view corporate marketplace where almost everything looks, feels, and perhaps is an advertisement (ad).

Laplace’s Demon is an interesting idea that if there is enough information about the position and momentum of the properties in a system, then perhaps you could predict its future state. In my first article for this year — let’s don the hat of that demon and a simplified of what the Web Browser and more generally the World Wide Web (www) could look like in the next two decades (2042).

To come up with plausible predictions, my technological enthusiasm and bias for free and open source software (FOSS) has to be suspended to build up my best arguments for the combined incentives of the stakeholders who affect the direction of the web. These stakeholders are governments, companies, technologists, and users. This will effectively be an argument against myself, but it should be delightfully fun.

I’m not a market expert nor business analyst but just a random developer that every so often works on websites for web browsers. Let’s take a look at my predictions. It is currently the year 2022.

The URL Bar is Replaced by the Everything Search Bar

The Uniform Resource Locator (URL) will be replaced by search and its visibility left as a browser implementation detail. The blur between URL and search term will lead to users dropping the distinction between a URL bar and a search bar. The primary mode of web navigation and discovery becomes search and the first web browser to get away with it wins big.

The frictionless transition and user behavior will compel other browsers to reluctantly do the same under the pressure for shared The significance of the Uniform Resource Identifier (URI) scheme such as http:// or ftp:// will follow suit and disappear completely on the user facing side. Reduced parsing complexity will open up the path towards a full focus on natural language processing (NLP) as more protocol supports are due to security concerns and no signal consensus.

Google’s omnibox, or “everything box” becomes the standard and search first “omnibars” become a ubiquitous feature in all mainstream browsers as a user friendly prompt using search engine NLP to navigate the web.

# install firefox and chromium now
$ pacman -S firefox chromium

# find and open the 11th edition of ecmascript
$ mupdf "$(locate --all ecmascript 11 | head --lines=1)"

"take me to amazon" -> https://www.amazon.com/

Bookmarks and History Replaced by the Integrated Search Engine

The web browser’s bookmarks and history interface will be revamped into a thin client integrated search engine, and an offline client replica with lightly cached bookmarks, history and search data. Current iterations of the “new tab page” trend will motivate this new design. The history and bookmarks page will look exactly like a search engine results page (SERP).

Startups will offer fully centralized bookmark and history storage replication to servers direct from the browser. This will be a feature that users want as history management and bookmarking become frictionless and obsolete.

Users in aggregate rarely care about the way their personal data is used and bar legal restraints — bookmarks and history will be a new data collection source to improve search results if it isn’t already. Monetizing bookmarks and history data will be normalized, and companies will use this data to further perfect key ranking algorithms for advertisers and develop new products.

Web browser based startups will need to have both a search engine and a browser to compete in the new space of the integrated search engine. The search engine by itself becomes obsolete, since the web browser or rather the operating system (OS) will do the heavy lifting of discovery.

The Web Browser as an Internet Operating System and Specification

The Internet operating system (Internet OS) will make its grand resurgence with rivaling feature parity to entrenched operating systems. Google’s Chromebook and Chrome OS will become the most important predictive for the Internet OS. This is a device that runs Google’s Chrome Browser as the operating system. Competitors in this space will use the open source Chromium OS to bootstrap their own operating system derivatives.

Web Assembly (WASM) will become the wild card that fills the eventual application vacuum as the Internet OS paradigm goes mainstream. WASM will pave the way for complete by exposing more programming languages to the browser.

Google’s Chrome and Chromium based derivatives will knock all other browsers off the market except Apple’s Safari, leaving Safari as their only contender. Chrome and Chromium application programming interfaces (APIs) will become the additional specification that web applications passively adopt.

Deprioritization of Open Source Software and Ascendancy of Open Source Hardware

Since most open source software (OSS) in the web space is “obsolete” and has bootstrapped many big players to a “good enough” position, the next destination is to model the Apple ecosystem and that requires bootstrapping a hardware stack with the help of open source hardware (OSH).

The elusive technological walled garden requires stack supremacy — which involves tight control of the technological stack down from the hardware and up to the software. The race towards stack supremacy will cause open source hardware to ascend from obscurity into the light and mature OSH startups will be bought with capital.

Specialized Hardware, Eternal Defaults, and Challenging Legacy Carrier Features

Companies that reach stack supremacy will iterate on new devices and software quickly without obstruction to create substantially better user communication experiences. This sets the stage for “eternal defaults” and pseudo carrier like status. Traditional carrier networks will begin to see a significant usage decline in legacy communication schemes like short message service (SMS) as the Internet backbone subsumes its function more reliably.

Big players in the market will be the equivalent of wireless carriers or Internet Service Providers (ISP). In the environment of specialized hardware and eternal defaults, users would have to buy another device to have a completely different Internet experience. Installing, removing, or bolting on software will be nigh impossible and hard to change defaults will be everything in these markets.

Deprecation of the Developer Tools and Transition to Developer Programs

Companies that reach garden status will phase out their specific browser developer tools to further tighten iteration speed with web application developers by introducing developer programs. Security concerns will also place into question the viability of exposing developer tools to all users and will be disabled by default. As more web applications sensitive data in plain sight, pressure will mount from stakeholders to resolve security issues.

Initially developer programs will be free to get many developers on board. Implementation fragmentation will be removed by laying down strict guidelines on web application development within the Internet OS ecosystem. The average user’s knowledge on basic aspects of the legacy web will be effectively non–existent at this point.

Property Stewardship above Ownership and Co-ownership

Software as a service (SaaS) and Infrastructure as a service (IaaS) are lucrative business models in the software and hardware space and will have even greater significance. The complexity of specialized hardware combined with increasing software results in users preferring that companies be stewards of their data and property. The process of self installing and setting up software becomes obsolete.

The subscription model of SaaS and IaaS will extend to almost all software and into the hardware space with phones, laptops, and tablets. Schools, governments, and other major institutions will switch over to renting software and hardware indefinitely. The maintenance and repair of rented hardware and software will be relegated to approved and authorized companies. The software industry will begin to formalize and standardize key operations into professional licences.

Defragmentation Markets: Buying Interoperability and Integration

Open standards don’t make for easy business models — pockets of incompatible fragmentation do. The lucrative unbundling effect will give rise to defragmentation markets and product bundling. The standard’s interoperable by design approach fades away as a complete user experience on the web is achieved by buying interoperability and integration. The majority of content on the web will be generated in silos that do not communicate with each other unless through partnerships or “product bundling”.

If incompatible fragmentation rests in few players, bigger players will buy interoperability directly in partnerships or integration deals with an Internet OS. If fragmentation has numerous players, the biggest Internet OS players will facilitate users being able to buy and bundle services directly in deals with smaller players. Installing multiple applications that have similar functionality and siloed network effects will be tough on users causing product interoperability bundling to become the competitive feature for the Internet OS.

For example, product interoperability bundling for chat applications would mean that users of one chat application could communicate with users of another application.

Chat: Facebook Messenger, Discord, Slack, Microsoft Teams, Skype, WhatsApp, Snapchat, Telegram, Signal, Viber, WeChat, Line, FaceTime, Google Chat

Vertical search engines will be reduced to APIs that the omnibar consumes. Search bars on websites and web applications will disappear to avoid user confusion and developer programs will ensure tight integration. Capital will determine interoperability in this market, and Internet operating systems will have integration deals for prominent vertical search engines on major web applications.

Site specific search is available to the omnibar powered by contextual suggestions.

Modern protocols are also up for defragmentation. The Internet OS gains competitive advantage by amalgamating disparate protocols by either them seamlessly or indirectly with web applications. The more alternative protocols the more potential for user capturing power.

Protocols: IPFS (Inter Planetary File System), Tor (The Onion Router), I2P (Invisible Internet Project), ZeroNet, BitTorrent, Matrix, IRC (Internet Relay Chat)

Unblockable Advertisements, Cryptocurrency and Interaction Fees

Advertisements (ads) will be baked into the experience of the Internet in a way it cannot be easily or freely disabled. Native ad–blocking will become an integral part of this achievement with death by a thousand white–lists. The increasing popularity of ad–blocking will incentivize a shift to unblockable advertisements. Stakeholders, mainly users, supported by downstream advertisement revenue will Internet operating systems with unblockable ads.

Direct cryptocurrency payments will become a short term alternative to the advertisement model, but its fragmented and controversial influence will carry it into a merged union with established financial institutions and ad markets as it meets the requirements of too big to fail (TBTF) doctrine.

The centralized union of crypto markets with established financial markets will allow for seamless integration of Internet interaction fees. Micropayments will become the norm for many interactions you can currently still do for free, such as clicking links, copying text, and downloading images.

More businesses will converge around advertising, and there will be few alternative revenue models. Since all users operate within the frame of an ad market, those who wish to avoid advertisements will pay subscription fees to partially or completely unblockable advertisements. The ad–free subscription model will become extremely popular and lucrative as the complexity and awareness of free ad–blocking increases.

Deprecating Registration Forms: Universal Password and Data Management

Internet operating systems will have feature complete password, data, and role management functionality. Users will have to enter their personal data on their systems only once to enable one click registration to any service. This is one step up from the single sign-on (sign in with) buttons that are ubiquitous across the web — to a new one click registration and permission based API model.

One click registration will obsolete completing registration forms in web applications, and pave the way for product bundling integrations. Personal data will be stored on remote servers while the Internet OS locally stores transitory and key metadata. Security tokens will become ubiquitous and modern devices will be authenticated with physical digital keys.

Market dominance will hinge particularly on the tablet and phone mobile markets. Companies in this space will have their own Internet operating system for mobile devices. Chrome and Chromium’s specification will be the base, but dominant companies will eventually diverge and create their own proprietary APIs with their specific devices.

Internet OS companies with dominant market share influence and multi–national cloud infrastructure will become “global carriers” and manage communications services in more jurisdictions on the Internet backbone, eventually competing more directly with legacy carrier networks.

# Hypothetical Market Share

"Google/ChromeOS"           : 55
"Microsoft/EdgeOS"          : 14
"Apple/Safari"              : 12
"* Facebook/MetaOS"         : 9
"Brave Browser/BraveOS"     : 5
"* Amazon Silk/FireOS"      : 1
"* SpaceX/StarlinkOS"       : 1
"DuckDuckGo Browser/DuckOS" : 1
"Firefox Browser/FirefoxOS" : 1
"Opera Browser/OperaOS"     : 1

Microsoft and Brave Software will have high upward swing potential. Facebook (now known as Meta Platforms), Amazon, and SpaceX will enter the mobile phone Internet OS market by onboarding a new user base.

Conclusion and Alternative Disruptions

That was a wild ride, and while my hope is that none of the above happens — anything is possible because money can align the most distant stars. Let’s conclude with a brief look at possible disruptions that can easily nullify the plausibility of everything written before.

Out of a of disruptions, one has caught my eye, and it’s the advertisement model itself that supplants most businesses on the Internet. The encroaching influence and ubiquity of ads increasingly suggest that it’s a grand house of cards in the sand.

Let he that is not a shill among you cast the first stone.

Search engines are increasingly looking more like ad brochures and streaming services, like home shopping networks. One might as well the job. The already massive and increasing ad volume push could mean that there is an opening for a new type of fully classified advertising product over the current hybrid personalized and search advertising offerings.

Personally, it’s difficult to find convincing information on the efficacy of ads from the advertiser’s point of view in aggregate. Do advertisers have clear beneficial signals from personalized advertisements? It’s not easy to shake the feeling that it only takes a single company to create an online product that that advertisers can achieve the same value for less money and this party ends quickly. Here’s a few questions that would be interesting to see an answer to from a company with a product.

Is there a way to guarantee a product was bought because of an ad?

What would cause people to pay to watch ads or read classifieds for product leads?

Can we search for, read, and watch by category every advertisement in one place?

What would a strictly advertisement only global streaming service or proper classifieds search engine look like?

Why do people watch Super Bowl ads?

The technology at our disposal, particularly solutions, should hypothetically allow advertisers to reach more people transparently with better metrics, for a vastly lower price point than it is to advertise today — or at least that’s what I’d like to believe. This appears easy, but perhaps there is a combined incentive by stakeholders to avoid lowering the price of advertising.

NixOS Pins and Needles

Mon, 22 Nov 2021 09:55:32 +0000

NixOS is a Linux distribution built around the nix package manager. Since writing my first article on NixOS, it has quickly become my personal OS (Operating System) of choice for desktops, servers, and The transition process to become almost as as my imperative environments has been marred by a lot of pins and needles on the usability front, but it has been worth it.

For context, you can use a single configuration.nix to build machines which is super convenient, but my inventory is diverse and has at least five machines. Configuring these machines to play nice together ultimately means having to control complexity — and that’s not always fun.

This is a fairly opinionated and under the hood take on using NixOS, you shouldn’t take any of this as advice — I’m not a functional or LISP programming expert. Lasciate ogni speranza, voi ch’entrate.

Testing Single Files Quickly

My inventory includes some low end hardware so rebuilding a small subset of machines is in NixOS. This hacky script of mine dry builds single files in isolation and foregoes the need of rebuilding entire configuration sets (with some trade–offs), saving precious time. Before, I’d usually have to wait for a complete nixos-rebuild or nixops evaluation, only to discover missing imports or other syntax errors. Single file evaluation allows working on small sections of a larger set of configurations quickly without rebuilding or activating machines.

Sorry, your browser doesn't support embedded videos. Here is a link to the video instead.

Using my hacky script nixos-test to quickly evaluate small files and even a full configuration.

GNU Guix (as in geeks) is currently superior to NixOS in this regard with commands like guix system that provide this type of fast inspection. If Guix is interesting to you, out this cool video introduction.

Searching For Packages and Files

In most Linux distributions you can search to see which package owns a file. For example, on Arch Linux you would run pacman -Fy <file> to search the database for a package that contains the <file> you’re seeking. This is great when compiling or writing a program. On NixOS, my preference is to use nix-locate to find packages and files. NixOS package search and naming conventions can be unintuitive. To ease my pain, the ideas of this shell script are adapted for my specific purposes. My hacky version sets up an alternate home environment, drops the desired program into a temporary nix-shell with fish and prints some useful paths afterwards.

Sorry, your browser doesn't support embedded videos. Here is a link to the video instead.

Poking around VLC’s package files.

Running Configurations

NixOS comes with powerful virtual machine integration testing modules. I’m a bit more lazy and like to jump directly into a virtual machine from my shell with QEMU (sometimes called Quick Emulator) to try out a configuration. The nixos-generate command generates different system output formats for a specified configuration. The ISO is my favorite. Often times I’ll spin up a system in the cloud as a read only ISO.

$ nixos-generate --format iso --configuration server.nix --option builders ''
$ qemu-system-x86_64 -nographic -enable-kvm -m 1024 -cdrom nixos.iso

Sorry, your browser doesn't support embedded videos. Here is a link to the video instead.

Trying out a configuration with a virtual machine.

DRY is Evil Incarnate

The functional ecosystem has taught me that programming isn’t worth the pain. Don’t repeat yourself (DRY) is the devil incarnate in my book, especially in a functional programming context. After throwing away my first directory structure, I’ve since implemented what may be a kind of naive stratified design approach to prevent myself from wasting pulling useful logic out of deep abstractions or forcing my poorly crafted NixOS modules to work for a purpose that it was not designed for.

Nixpkgs, the NixOS implementation, lets you spin up system abstractions in your sleep and while this is good for writing packages if your hastily made abstractions are poor — you’ll be banished to decomposing the world. My self made rule now is that anything that does not require imports and does something useful is a basic knowledge piece. Every singular piece of knowledge is saved, and whether used or not can be combined to create higher functionality independent of itself. These knowledge pieces should be easy to change, check, and reuse. For that simple reason, duplication is good, really really good, and as a bonus — sharing working snippets with others is easier.

Sorry, your browser doesn't support embedded videos. Here is a link to the video instead.

WET: Write everything twice.

Modules Should Do One Thing

Or rather — it seems useful to distinguish between lower level modules (modules containing systemd modules only) and higher level modules (modules containing multiple modules). The high level interfacing inside a module is often too hard to pick apart. Situations can arise where you try to do something with a module that is out of its scope. The reality is that a lot of NixOS modules have two or more smaller modules struggling to get out. Sometimes you’ll have to reduce upstream modules down to its basics to get what you want done — this can be painful.

My use cases typically require running multiple copies of the same module. The easiest way to achieve this is to keep custom made modules as small as possible (just a systemd service and basic configuration). It’s impossible to satisfy all use cases so everything else ends up as a test case of combinations that can be used as higher level modules. “Lower” level modules are easier to understand and manipulate too, since they are just an indirect version of plain systemd files.

Sorry, your browser doesn't support embedded videos. Here is a link to the video instead.

The lib.attrsets.mapAttrs' function is great for duplicating systemd services.

Separate System and User Environments

As a user, if I can avoid a nixos-rebuild, nixops deploy, sudo, or doas — sign me up. My user packages are split from the system side with config.nix abuse and nix-env. The nix community’s home manager is nice, but avoided because the less dependencies the better when connecting my home folder to another Linux distribution. My programming style incorporates the Unix shell mostly as an integrated development environment (IDE), so my dotfiles and default presets for linting and such are all that’s necessary. The new nix flake feature puts in the guard rails for proper reproducibility, but I don’t think I’ll be using that for some time.

Imperative to Declarative User Experience

Once in a while I’ll take a look at the discussions on social media around the NixOS, Guix, and functional operating system community. Recent discussions seem to center around the lack of popularity of NixOS and GNU Guix and their complexity.

The unpopularity of NixOS and GNU Guix sounds about right. Functional Linux distributions will not become generally popular anytime due to misaligned developer and user experience expectations. Popularity may be limited to self–hosting, developer operations, and system administration circles.

I’m fairly certain that nixos-generate-config is the command with the best user experience (UX) on the whole system — everything else is pins and needles. NixOS needs more imperative to declarative The NixOS install procedure achieves this workflow with two separate commands. The first command generates a declarative configuration and the second implements it. If distribution installation speed runs were a thing, then NixOS would have a leg up especially if a configuration.nix is baked into a custom installer with automatic formatting options like fileSystems.<name>.autoFormat.

$ nixos-generate-config --root /mnt
$ nixos-install

In my experience, a casual NixOS user doesn’t even want to open the configuration.nix to manually write stuff and connect modules together unless absolutely necessary. In their world — configuration blocks should simply be “generated” so that they can just copy and paste things.

“Why can’t it just generate the configs like it did to install my system?” A casual user

Docker, Kubernetes, Tailwind, and React all have the same incentives that ensure popularity. It’s not necessarily the underlying technology that makes a software ecosystem popular or “good”, but the ease of the higher level tooling to create, share, and copy from a near endless library of pre–defined documents, templates and components. In short, it’s very little effort for great reward — that’s the nature of the beast. Most users abhor starting from scratch and because of that it’s sort of easy to anticipate what will become popular — NixOS and GNU Guix are not there yet. Searching the NixOS options index to build up configuration is almost like starting from scratch in a docker compose.

Conclusion

Future iterations of NixOS, Guix and its derivatives can only get more robust. may be rejected in most real world applications, but will definitely become the name of the game. styles will undoubtedly always be around.

If easily composed and shared declarative tooling (infrastructure as code) becomes the dominant paradigm in user space, then issues in the Linux desktop space can be resolved elegantly by lassoing the majority of desktop configuration types and quirks. The NixOS hardware quirks repository is brilliant.

Interestingly enough, the nixpkgs repository is a very way of getting a general overview of the current state of the upstream software development world. Every dirty hack, every non–standard project setup, every network call during installation — the sandbox sees it all.

Finally, The nix language is just great in my opinion, without functions, it reads like JSON (JavaScript Object Notation) with comments minus the dangling comma shenanigans. It can be used as metadata or as a language. If I could ever manage to get my NixOS configuration under control (specifically secrets and metadata), I’d open source it, but for the time being, I just don’t have the time.

A Few Links

Thu, 30 Sep 2021 02:39:44 +0000

Image showing a few links from my list of bookmarks.

Over the course of cross browser testing websites, my bookmarks have become scattered and out of sync. I’ve grown too accustomed to using multiple web browsers and their various styles of bookmark management.

I’ve begin consolidating my bookmarks into one location in an attempt to reduce my Internet usage. The constant researching of websites and articles that are already bookmarked or known from memory isn’t particularly fun.

To make matters worse, website rediscovery through a search engine is abysmal nowadays with sites becoming increasingly ranked by comment recency on social media aggregators, spam, and bot generated content. Rediscovering an original website or source is work than necessary. If the origin website isn’t mentioned recently by a top ten social media website or aggregator, then as far as search engines go it doesn’t exist.

Nothing yet in my opinion beats a list of links in a file. The awk code below dumps a list of records from a bookmarks.txt file as partial html. I’m hoping to eventually integrate my public bookmarks into this website using partial html output. These bookmarks can be serialized into any format and it doesn’t have to be html.

#!/usr/bin/env -S awk -f

# Record separator (RS) will split on blank lines.
# Field separator (FS) will split on new lines.
BEGIN {
  RS = ""
  FS = "\n"
}

{
  printf "<a rel=\"nofollow noopener\" id=\"%s\" title=\"%s\" href=\"%s\">\n", NR, $1, $2
  printf "<span>%s</span>\n", $2
  printf "<h1>%s</h1>\n", $1
  printf "</a>\n\n"
}

I’m a lazy programmer. My requirements are simple; the website title is my tag, and the link is the additional context needed to find the exact site or source. The bookmarks.txt file that this awk code parses is a list of records containing a link and a title. A description snippet can also be appended to the record for extra fuzzy search context, but that’s not a priority.

CSS Icons
https://cssicon.space/

NixOS: the good, the bad, and the ugly
http://www.willghatch.net/blog/2020/06/27/nixos-the-good-the-bad-and-the-ugly/

Learn X in Y minutes
https://learnxinyminutes.com/

An Introduction to the World of ThinkPad
https://mckinley.cc/blog/20200411.html

All Abstractions Lie
https://wiki.c2.com/?AllAbstractionsLie

Interestingly enough, many browsers incentivize the use of tabs as bookmarks, others, history as bookmarks, and some, tabs as history — and vice versa. Bookmarks, tabs, and browser history eventually start to look the same and perhaps may end up unified in some way, or at least that’s my impression after using many different web browsers for so long.

Once my bookmarks are unified again, I’ll return to my old Internet browsing style; no history or tab dependency, and no extension or plugin dependency — just bookmarks. That provides the best case scenario for little dependency on my choice of browser.

Conclusion

It goes without saying that a list of links is not a particularly useful way for most people to find or store content, but with the trend of increasingly poor and siloed re–discoverability of original content, even search engines will eventually loose significant utility. That’s not a bad trade–off though, perhaps it’s not to long before someone implements a way for web browsers, given a specific prompt, to surf the Internet as an agent on the user’s behalf, display the resulting websites, and skip the annoying parts of Internet browsing entirely.

Typesetting in LaTeX

Fri, 30 Jul 2021 17:12:40 +0000

A LaTeX document.

LaTeX pronounced (/la:tech/), is a free typesetting system that allows you to efficiently create articles, books, white papers and other technical documents. Most scientific documents and white papers are built using this typesetting system. LaTeX is the opposite of a (What You See Is What You Get) editor. The key difference is that format is separated from content. Documents are pre–formatted to a specification that the content follows. LaTeX is a markup language similar to HTML (Hyper Text Markup Language).

The separation of format and content allows for the automation of many parts of a typical document. The table of contents, bibliography, page numbers, footnotes, and references are all auto generated. The document, for lack of better term, is “alive” — a living document that is programmable.

Typesetting isn’t a very useful skill in general, at least in my experience, because most people prefer WYSIWYG editors, but it’s useful in large groups if only to eliminate arguments over formatting, positioning, and other details. Let’s take a look at my preferred typesetting programs and workflow.

Typesetting Workflow

My preferred workflow for typesetting is a combination of the programs TexWorks and ViM. The LaTeX editor texworks acts as a heads up display and gives a general overview of what’s going on. The magnifying glass, jump to source, and jump to PDF (Portable Document Format) are pretty much the killer features of texworks. Honestly it’s the only reason texworks is in my toolkit.

Sorry, your browser doesn't support embedded videos. Here is a link to the video instead.

LaTeX Workflow

The editing action happens in vim while texworks polls and updates its graphical display for both the tex source file and pdf, even when external programs are mutating them. Editing can happen really fast as it’s only necessary to keep the current line number inside my head when inter-operating between each program.

If the sense of position is lost, then the line number becomes a guide, and a jump from tex to pdf or from pdf to tex puts me back into the flow. The desktop windows are tiled to maximize use of the viewport and the recompilation commands below are executed on every manual save in vim.

#!/bin/sh -eu

if ! {
      pdflatex -halt-on-error "$1" \
      || xelatex -halt-on-error "$1";
     }
then cat
fi

Conclusion

That’s all there is to my typesetting work flow. Programs in general tend to be either frustrating or slow, but texworks and vim happen to be one of those nice and rare experiences. Gummi is also another interesting LaTeX editor that is used mainly for quick edits.

Cooking and Baking Linux Distributions in Nix

Mon, 28 Jun 2021 21:47:37 +0000

Note: The code in this article fails on newer versions of nix (2.4+) due to an implementation change. Take a look at this shell.nix that uses a relaxed sandbox along with the __noChroot option.

The Linux distribution space is vast and fragmented. It’s the wild wild west, and that’s all right in my book. Having one distribution installed, to a lesser degree means having installed. Packages in one distribution may be unavailable in another but it’s always nice to be able to run unavailable programs without much ado. No one can stop you from spinning up a “container” or virtual machine with Arch Linux to try out an idea, while mixing in packages from NixOS to compare or hack around. The process of binaries and creating ad–hoc virtual machines is sometimes called “cooking” or “baking”.

The terms “cook” and “bake” is a bit of jargon. In this context and for my purposes, “cooking” refers to the cherry picking of various binaries and files from multiple Linux distributions into a final system, and “baking” refers to the process of injecting a Linux kernel into a “cooked” distribution image for boot ability. The recipe for a “cake” — a cook and bake, can be composed in any preferred programming language or process.

Countless programs offer a way to cook or bake systems in some form. LinuxKit, Darch, BitBake, and BuildRoot are just a few. Manual chroots (change root), Toolbox, and Distrobox also allow for fully mutable distribution environments. Let’s use Nix, nix-shell, and PRoot (user space chroot) to produce an experimental environment that lets us cook and bake various distributions.

Nix Shell

The nix-shell command sets up an interactive shell environment according to the specified nix code. A shell.nix file below creates an interactive shell environment with a custom PS1 (Prompt String #1).

let

  name = "nix-shell";
  pkgs = import <nixpkgs> { };

in pkgs.mkShell {

  inherit name;

  buildInputs = [ ];

  shellHook = ''
    export PS1='\h (${name}) \W \$ '
  '';
}

This shell.nix sets a name that describes the environment, buildInputs that adds a list of programs to the new shell path, and a shellHook that runs arbitrary commands after nix-shell invocation.

Executing nix-shell without a file path acts on the default.nix or shell.nix in the current directory.

$ nix-shell

Adding --pure as an argument to nix-shell clears the majority of the environment variables before entering the nix shell environment.

$ nix-shell --pure

The --command argument can be used to further cleanup the environment variables by specifying a shell of choice and the sourcing procedure.

$ nix-shell --pure --command 'bash --login --norc --noprofile'

Cooking

Now that the nix-shell summary is out of the way — let’s set up a basic shell.nix that implements the cooking functionality. Using the previous minimal shell.nix setup, the interactive environment is named, the pkgs attribute locked to a specific version of nixpkgs, and a file system for Alpine Linux 3.12 using pkgs.dockerTools.pullImage.

let

  name = "nix-shell.cake";

  pkgs = import (builtins.fetchTarball {
    url = "https://releases.nixos.org/nixos/21.05/nixos-21.05.650.eaba7870ffc/nixexprs.tar.xz";
    sha256 = "08fpds1bkv9106c6s5w3p5r4v3dc24bhk9asm9vqbxxypjglqg9l";
  }) { };

  alpine-3-12-amd64 = pkgs.dockerTools.pullImage rec {
    imageName = "alpine";
    imageDigest = "sha256:2a8831c57b2e2cb2cda0f3a7c260d3b6c51ad04daea0b3bfc5b55f489ebafd71";
    sha256 = "1px8xhk0a3b129cc98d3wm4s0g1z2mahnrxd648gkdbfsdj9dlxp";
    finalImageName = imageName;
    finalImageTag = "3.12";
  };

in pkgs.mkShell {

  inherit name;

  buildInputs = [ ];

  shellHook = ''
    export PS1='\h (${name}) \W \$ '
  '';
}

The cook function creates a derivation that extracts a rootfs, passes it to proot for bootstrapping, and cherry picks contents into the file system. The nix sandbox defaults to a strict policy that prohibits network access so the derivation’s hash mode is made recursive to allow network access inside the proot. If sandboxing is to relaxed, then network access is allowed by setting the __noChroot option to true inside the derivation without the need for a recursive hash. Most attributes defined in a derivation are exposed as bash in a phase.

{
  cook = { name, src, contents ? [ ], path ? [ ], script ? "", prepare ? "", cleanup ? "", sha256 ? pkgs.lib.fakeSha256 }: pkgs.stdenvNoCC.mkDerivation {
    inherit name src contents;
    phases = [ "unpackPhase" "installPhase" ];
    buildInputs = [ pkgs.proot pkgs.rsync pkgs.tree pkgs.kmod ];
    bootstrap = pkgs.writeScript "bootstrap-${name}" ''
      ${script}
      rm "$0"
    '';
    installPhase = ''
      set -euo pipefail
      mkdir --parents rootfs $out/rootfs
      tar --extract --file=layer.tar -C rootfs

      ${prepare}

      cp $bootstrap rootfs/bootstrap
      proot --cwd=/ --root-id --rootfs=rootfs /usr/bin/env - /bin/sh -euc '. /etc/profile && /bootstrap'
      printf 'PATH=${pkgs.lib.strings.makeBinPath path}:$PATH' >> rootfs/etc/profile

      [ -n "$contents" ] && {
        printf "\n"
        for paths in $contents; do
          printf "Cooking... Adding %s \n" "$paths"
          rsync --copy-dirlinks --relative --archive --chown=0:0 "$paths/" "rootfs" || exit 1
        done
        printf "\n"
      } || printf '\n%s\n' 'No contents to cook.';

      ${cleanup}

      printf '\n%s\n\n' "$(du --all --max-depth 1 --human-readable rootfs | sort --human-numeric-sort)"
      cp -rT rootfs $out/rootfs
    '';
    outputHashAlgo = "sha256";
    outputHashMode = "recursive";
    outputHash = sha256;
  };
}

This soft abstraction allows us to minimally cook a distribution. Binaries are added in from the nix store or from other distributions. The NixOS packages glibc and awk are thrown into the mix just for fun. GNU’s awk is added to the system path and serial console ttyS0 activated for low level virtual machine access.

{
  alpine = cook {
    name = "alpine";
    src = alpine-3-12-amd64;
    contents = [ pkgs.glibc pkgs.gawk ];
    path = [ pkgs.gawk ];
    script = ''
      apk update
      apk upgrade
      apk add openrc
      sed -i 's/#ttyS0/ttyS0/' /etc/inittab
      printf 'migh7Lib\nmigh7Lib\n' | adduser alpine
    '';
  };
}

The cooking sha256 is unknown until the shell completes the derivation because running apk update breaks the immutability requirement. Once the system is cooked to my liking, the generated hash is added to save the derivation as a copy.

This cooked alpine file system is good enough for useful work. Inside pkgs.mkShell serve up the system by running proot from the shellHook. Pass through selected host directories to the guest using the --bind argument with proot. These host directories will inside the guest.

pkgs.mkShell {

  inherit name;

  buildInputs = [ pkgs.proot pkgs.qemu ];

  shellHook = ''
    export PS1='\h (${name}) \W \$ '
    proot --cwd=/ --rootfs=${alpine}/rootfs --bind=/proc --bind=/dev /usr/bin/env - /bin/sh -c '. /etc/profile && sh'
    exit
  '';
}

Take a look at my cake.nix shell that creates this environment. You can this shell using the nix-shell argument --expr on a raw format URL. Never ever run code from a URL in your shell without first downloading and auditing the code.

nix-shell -E 'import (builtins.fetchurl "https://raw.githubusercontent.com/tdro/dotfiles/b9a9051a06d7bba2911c2ed0da0f2e73b4b5de81/.config/nixpkgs/shells/cake.nix")'

Baking

The bake function begins by cooking the initrd (initial ramdisk) image. The initial RAM (Random Access Memory) disk contains the first root file system the kernel sees after initialization. The simplest initrd file system (initramfs) my mind can come up with is worked out in a cooking script. Kernel modules are prepared and cooked before baking.

{
  bake = { name, image, size ? "1G", debug ? false, kernel ? pkgs.linux, options ? [ ], modules ? [ ], uuid ? "99999999-9999-9999-9999-999999999999", sha256 ? pkgs.lib.fakeSha256 }: let
    initrd = cook {
      inherit sha256;
      name = "initrd-${name}";
      src = alpine-3-12-amd64;
      script = ''
        rm -rf home opt media root run srv tmp var
        printf '#!/bin/sh -eu
        mount -t devtmpfs none /dev
        mount -t proc none /proc
        mount -t sysfs none /sys
        sh /lib/modules/initrd/init
        ${pkgs.lib.optionalString (debug) "sh +m"}
        mount -r "$(findfs UUID=${uuid})" /mnt
        mount -o move /dev /mnt/dev
        umount /proc /sys
        exec switch_root /mnt /sbin/init
        ' > init
        chmod +x init
        find . ! -name bootstrap ! -name initramfs.cpio | cpio -H newc -ov > initramfs.cpio
        gzip -9 initramfs.cpio
      '';
      prepare = ''
      modules='${pkgs.lib.strings.concatMapStringsSep " " (module: module) modules}'
      initrd_directory=rootfs/lib/modules/initrd
      [ -n "$modules" ] && {
      mkdir --parents "$initrd_directory"
      printf "\n"
      for module in $modules; do
        module_file=$(find ${kernel} -name "$module.ko*" -type f)
        module_basename=$(basename "$module_file")
        printf "Cooking initrd... Adding module %s \n" "$module"
        cp "$module_file" "$initrd_directory" || exit 1
        printf 'insmod /lib/modules/initrd/%s\n' "$module_basename" >> "$initrd_directory/init"
      done
      } || printf '\n%s\n' 'No modules to cook.'
      '';
    }; in pkgs.writeScript name ''
      # Baking Script
    '';
}

The script for the bake function creates a disk image, formats its partitions, installs the syslinux bootloader, and injects the kernel. Some commands are partially parameterized for more control.

{
  pkgs.writeScript name ''
    set -euo pipefail
    PATH=${pkgs.lib.strings.makeBinPath [
        pkgs.coreutils
        pkgs.e2fsprogs
        pkgs.gawk
        pkgs.rsync
        pkgs.syslinux
        pkgs.tree
        pkgs.utillinux
      ]
    }
    IMAGE=${name}.img
    LOOP=/dev/loop0
    ROOTFS=rootfs
    rm "$IMAGE" || true
    fallocate --length ${size} $IMAGE && chmod o+rw "$IMAGE"
    printf 'o\nn\np\n1\n2048\n\na\nw\n' | fdisk "$IMAGE"
    dd bs=440 count=1 conv=notrunc if=${pkgs.syslinux}/share/syslinux/mbr.bin of="$IMAGE"
    mkdir --parents "$ROOTFS"
    umount --verbose "$ROOTFS" || true
    losetup --detach "$LOOP" || true
    losetup --offset "$((2048 * 512))" "$LOOP" "$IMAGE"
    mkfs.ext4 -U ${uuid} "$LOOP"
    mount --verbose "$LOOP" "$ROOTFS"
    rsync --archive --chown=0:0 "${image}/rootfs/" "$ROOTFS";
    mkdir --parents "$ROOTFS/boot"
    cp ${kernel}/bzImage "$ROOTFS/boot/vmlinux"
    cp ${initrd}/rootfs/initramfs.cpio.gz "$ROOTFS/boot/initrd"
    printf '
    DEFAULT linux
    LABEL linux
      LINUX  /boot/vmlinux
      INITRD /boot/initrd
      APPEND ${pkgs.lib.strings.concatMapStringsSep " " (option: option) options}
    ' > "$ROOTFS/boot/syslinux.cfg"
    extlinux --heads 64 --sectors 32 --install $ROOTFS/boot
    printf '\n%s\n\n' "$(du --max-depth 1 --human-readable $ROOTFS | sort --human-numeric-sort)"
    umount --verbose "$ROOTFS"
    rm -r "$ROOTFS"
    losetup --detach "$LOOP"
  '';
}

The desired kernel version is injected, and kernel options for connecting the virtual console tty1 and serial console ttyS0 are added. The modules are baked in dependency order using a minimal Module dependency order is determined using modprobe with the --show-depends argument on a live machine. These modules are just to sustain a virtio based block device and a root switch to an ext4 file system.

{
  alpine-machine = bake {
    name = "alpine-machine";
    image = alpine;
    kernel = pkgs.linuxPackages_5_10.kernel;
    options = [ "console=tty1" "console=ttyS0" ];
    size = "128M";
    modules = [
      "virtio"
      "virtio_ring"
      "virtio_blk"
      "virtio_pci"
      "jbd2"
      "mbcache"
      "crc16"
      "crc32c_generic"
      "ext4"
    ];
  };
}

Baking delegates to the shellHook because keeping the implementation inside the nix derivation sandbox requires specialized wizardry. It’s easier to use privileged programs such as doas or sudo within the shell environment to bake the image. Once baked, initialize the image inside a virtual machine using QEMU (Quick Emulator) in -vga std, -curses or -nographic serial mode. You can also -enable-kvm for better performance and set memory size with -m 1024.

pkgs.mkShell {

  inherit name;

  buildInputs = [ pkgs.proot pkgs.qemu ];

  shellHook = ''
    export PS1='\h (${name}) \W \$ '
    doas ${alpine-machine}
    sudo ${alpine-machine}
    qemu-system-x86_64 -nographic -drive if=virtio,file=./${alpine-machine.name}.img,format=raw
    exit
  '';
}

Conclusion

The nix-shell paradigm is extremely powerful. The functional compositions give way to replacing a lot of tools with minimal effort through abstraction. A lot more could be done with more advance usage of this type of nix-shell but it’s good enough to bind a few distributions into my user environment and cook up small virtual machines quickly.

My main systems run somewhat lean NixOS setups, and while spinning out nix code is easier for me now, the language seems more like a glue, and getting too much of it everywhere may slow you down. Escape pods like virtual machines and easily bound user space chroots are nice to have.

A Few Abstracts

Mon, 31 May 2021 20:26:07 +0000

Abstracts Page

The intention this month was to release a post on “cooking and baking” Linux distributions from a NixOS perspective, but instead I’ll release the beginnings of my abstracts page.

The abstracts page will contain short on books, media, games, and other material not directly related to programming and tech in general. We’ll kick it off with abstracts of The Art of Being Right, Aria the Animation, and Golden Sun.

Site Updates: Navigation

Fri, 30 Apr 2021 08:28:01 +0000

Updated navigation bar

Time has granted a small reprieve to work on the greatest pain point of my personal website — the navigation bar. Due to sheer laziness, the navigation bar was initially implemented as three separate bars for each of the main views (tablet, phone, and desktop devices). This created unnecessary friction when adding new items. In the long term, the goal of integrating more programs into the website will require a certain consistency and so the navigation bar must be unified across all views.

Improvements

My website is glued together with on the back–end. You’ll often hear that PHP has native abilities and that is true. To solve my problem of multiple navigation bars, a templating system is used to create scoped components using null coalescing operators to set the defaults. This bespoke component navigator.link describes a navigation link. Three parts receive or pass data: the attributes, the content, and optionally, a faux return clause.

<a href="<?php echo $route ?? null; ?>" class="
  <?php
    $hiddenLabels = [ 'Profile' ];
    if (in_array($label, $hiddenLabels)) { echo ' data-hidden'; }
    if ($navigation->isActive($route ?? null)) { echo ' data-active'; }
  ?>
">
  <?php echo icon($icon ?? null); ?>
  <span title="<?php echo $label ?? null; ?>">
    <?php echo $label ?? null; ?>
  </span>
</a>

<?php $route = $label = $icon = null; ?>

Since this component is described in a file, it is cascaded anywhere using require.

<column-middle>
  <?php
    $route = '/';          $label = 'Home';     $icon = 'home';      require views('components', 'navigator.link');
    $route = '/posts/';    $label = 'Posts';    $icon = 'feather';   require views('components', 'navigator.link');
    $route = '/projects/'; $label = 'Projects'; $icon = 'package';   require views('components', 'navigator.link');
    $route = '/about/';    $label = 'Profile';  $icon = 'file-text'; require views('components', 'navigator.link');
    $route = '/contact/';  $label = 'Contact';  $icon = 'mail';      require views('components', 'navigator.link');
  ?>
</column-middle>

In frameworks like Laravel, this templating method can take on a more form where components may contain scoped CSS or JavaScript that can be “extracted” into one final payload. Laravel uses Blade syntax for templating.

@component ('components.navigator')

  @slot ('left')
    @include ('navigator.partials.menu')
    @include ('navigator.partials.logo')
    @include ('navigator.partials.header')
  @endslot

  @slot ('middle')
    @include ('navigator.partials.title')
  @endslot

  @slot ('right')
    @include ('navigator.partials.search')
    @include ('navigator.partials.list')
    @include ('navigator.partials.user')
    @include ('navigator.partials.options')
  @endslot

@endcomponent

@component ('components.context-menu', ['extract' => 'style']) @endcomponent
@component ('components.context-overlay', ['extract' => 'style']) @endcomponent

Templates can also be nested indefinitely. Here’s the unified context menu component that contains nested HTML elements inside the label that is stacked or cascaded in different places around the website.

<?php
  $links = true;
  $id = 'navigation';
  $icon = icon('arrow-down-circle');
  $label = <<<insert
      $icon
      <span>More</span>
    insert;
  require views('components', 'context.menu');
?>

You’ll notice that namespacing could become an issue, particularly for code below the fold, and ideally data would be passed as to the views function, but this somewhat CSS like behavior is preferred for duplicating components of a similar type quickly.

Basic context menu component

Goals

More interesting tasks are on my to–do list and may be completed if the time allows.

Comprehensive Dark and Light Mode

The dark mode on the site isn’t implemented all that well. One of my goals is to implement comprehensive dark mode features. In reality CSS is the only necessity, but adding some of the minor features may prove useful. One such feature involves situations where light and dark mode changes apply to all open tabs of the site. The most comprehensive guide on implementing dark mode that I’ve come across is written in this blog post.

Search Engine

The idea of a vertical search engine where the search scope is limited to a specific set of curated sites looks promising. Here’s to hoping somebody makes a nice and vertical search engine that is easy to integrate into a blog, or maybe I’ll make my own. Lately, I’ve been playing around with building a search engine from scratch for something else that I’m interested in personally.

Sorry, your browser doesn't support embedded videos. Here is a link to the video instead.

Searching through a small index in a personal work in progress application

Deployment

The deployment situation for the website is looking up. The CI/CD (Continuous Integration and Deployment) situation, previously Drone, was replaced with NixOS and now the website deploys instantly. The deployment logic lives on the server itself. The development, staging, and production environments are setup and activated with a simple NixOS module.

{
  services.thedroneely.enable = true;
  services.thedroneely.development.enable = true;
  services.thedroneely.staging.enable = true;
  services.thedroneely.production.enable = true;
}

Writing Articles with Vale

Mon, 29 Mar 2021 22:55:17 +0000

Using Vale in the Vim text editor

If you’ve ever done any sort of continuous long form writing, you’ll eventually discover that grammar and spell checking break down and become more of a distraction than a benefit. The “one size fits all” rules for grammar and spell checking cannot hope to account for the indefinite amount of writing styles and constantly evolving word constructions.

Writing is time consuming. How long does it take to write a long form document? The answer not only depends on the writer, but also on the draft and review process which often requires another pair of eyes. Many documents are checked by hired professional editors and others yet freelancers to write for them.

For the soloist, and or casual writer, building a personal editor or another set of eyes is useful, provided that you’re willing to take on that time investment. The idea is to target a specific writing style, construct the rules, and apply it consistently to your documents.

Vale is an open source program that facilitates this approach — it’s a linter for writing documents. Vale offers no opinion on how to write. You decide that for yourself by a set of grammatical rules and word primitives. Such flexibility allows you to emulate the behaviour of applications like the Hemingway Editor. Vale is not the only program for prose — there are a plethora of similar programs such as textlint, proselint, redpen, or just basic regular expressions.

Installation

Vale is available on major platforms. Install vale for your specific operating system by following the install documentation. Predefined style templates are available for download if you need a starting point.

In my case, a wrapper in ~/.local/bin/vale is setup around the canonical vale to allow downloading a few writing style templates and setting up my configuration directory.

#!/bin/sh -eu

directory="$XDG_CONFIG_HOME/vale"
styles="$directory/styles"
mkdir -p "$directory" "$styles";

if [ "${1:-}" = "--update" ]; then
  svn export --force https://github.com/errata-ai/Google/trunk/Google         "$styles/google";
  svn export --force https://github.com/errata-ai/joblint/trunk/Joblint       "$styles/joblint";
  svn export --force https://github.com/errata-ai/Microsoft/trunk/Microsoft   "$styles/microsoft";
  svn export --force https://github.com/errata-ai/proselint/trunk/proselint   "$styles/proselint";
  svn export --force https://github.com/errata-ai/write-good/trunk/write-good "$styles/write-good";
  exit;
fi

$(which vale --all | grep --invert-match "local/bin" | head -n 1) --config "$directory/vale.ini" "$@";

The vale.ini configuration file is simple. Set a directory containing the styles, adjust the minimum alerting level, and activate the desired writing styles. The typical usage pattern is vale document.txt at the command line.

StylesPath = /home/thedro/.config/vale/styles
MinAlertLevel = suggestion

[*.{md,txt}]
#BasedOnStyles = google
#BasedOnStyles = joblint
#BasedOnStyles = microsoft
#BasedOnStyles = proselint
#BasedOnStyles = write-good
BasedOnStyles = hemingway
BasedOnStyles = dickens
BasedOnStyles = technical
BasedOnStyles = thedro

Emulating a style requires reading into the patterns of an author’s writing. Here is a snippet of a rule set taken from the write-good style that checks for weasel words.

extends: existence
message: "'%s' is a weasel word!"
ignorecase: true
level: warning
tokens:
  - absolutely
  - accidentally
  - additionally
  - allegedly
  - alternatively
  - angrily
  - anxiously
  - approximately

Styles are the named directories that contain a set of YAML rule sets. Construct as many rules as you want to complement the desired style.

$ tree -L 3 ~/.config/vale
___ styles
|   |__ google
|   |   |__ Contractions.yml
|   |__ joblint
|   |   |__ Acronyms.yml
|   |__ microsoft
|   |   |__ Wordiness.yml
|   |__ proselint
|   |   |__ Spelling.yml
|   |__ write-good
|       |__ Weasel.yml
|__ vale.ini

Conclusion

This approach obviously requires time as you work to build up a robust rule set for a particular style. The general idea is to correct mistakes, tokenize them as rules, and iterate constantly. You’ll have a tight feedback loop that at a minimum guards the consistency of your writing, and improves your writing speed over time. Perhaps we should take my work in progress clickbait writing style for sometime.

BasedOnStyles = clickbait

Writing NixOS Modules and Switching to Cgit

Sat, 27 Feb 2021 02:19:52 +0000

Note: Since NixOS version 23.05 a service module for cgit/nginx is available. Read only if you want a summary on making modules.

NixOS, and by consequence nix are for freelance programmers like myself. After trying out Gitea for a while, it’s about time to switch back to cgit for hosting my public repositories. Gitea and cgit are both self hosted git front-ends — the former being more resource heavy and featureful than the latter.

This website runs on a NixOS server so switching to cgit means setting up a basic module. Modules are a set of lower level implementations that combine to create a higher level system configuration. A cgit module is already available in NixOS using lighttpd but it does not fit my use case. Let’s wire up a module that implements cgit.

Module Interface

Writing a module yourself provides the luxury of defining the interface that invokes the implementation. I’ll create an imaginary interface that matches my case and worry about the implementation details later. The upstream service module is preemptively by adding it to disabledModules to possibly prevent future conflicts.

{
  imports = [
    ../modules/cgit/module.nix
  ];

  disabledModules = [ "services/misc/cgit.nix" ];

  services.cgit = {
    enable = true;
    package = pkgs.callPackage ../servers/cgit/default.nix {};
    domain = "thedroneely.com";
    subDirectory = "/git";
    authorizedKeys = [
      "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8aD3uJ937DKFXN1BYDAezG2umwj4k6 key"
    ];
    mirrors = {
      dotfiles = { owner = "thedroneely"; url = "https://github.com/tdro/dotfiles.git"; };
      "thedroneely.com" = { owner = "thedroneely"; url = "https://github.com/tdro/thedroneely.com"; };
    clones = {
      cgit = { owner = "thedroneely"; url = "https://git.zx2c4.com/cgit"; };
    };
    extraConfig = ''
      robots=noindex
    '';
  };
}

My imaginary interface is quite ambitious. Given an imported service module, it wants to be able to swap in and out the source code, change the domain name, support website subdomains and subdirectories, mirror and clone repositories, expose the git shell, and append configuration on the fly. Let’s try to make that happen and turn it into a real interface.

Module Framework

The simplest module has a structure where if enabled — nothing happens. My cgit module in modules/cgit/module.nix starts off bare.

{ pkgs, lib, config, ... }:

let

  service = "cgit";
  cfg = config.services.${service};

in {

  options.services.${service} = {
    enable = lib.mkOption {
      type = lib.types.bool;
      default = false;
    };
  };

  config = lib.mkIf cfg.enable { };
}

To sum it up — if services.cgit.enable = true then everything inside config evaluates.

Module Options

The config options of the services.cgit attribute set is derived from the options.services.cgit attribute set. Each option has a name which is the attribute set itself, a type, a default value, and optionally a description and example field that generates higher level The module will contain all options needed for the implementation.

{
  options.services.${service} = {

    enable = lib.mkOption {
      type = lib.types.bool;
      default = false;
    };

    package = lib.mkOption {
      type = lib.types.package;
      default = pkgs.cgit;
    };

    domain = lib.mkOption {
      type = lib.types.str;
      default = "${cfg.user}.example";
    };

    user = lib.mkOption {
      type = lib.types.str; default = service;
    };

    authorizedKeys = lib.mkOption {
      type = lib.types.listOf lib.types.str; default = [ ];
    };

    title = lib.mkOption {
      type = lib.types.str; default = "Repositories";
    };

    description = lib.mkOption {
      type = lib.types.str; default = "Browse repositories";
    };

    directory = lib.mkOption {
      type = lib.types.str; default = "/srv/${cfg.user}";
    };

    repository = lib.mkOption {
      type = lib.types.str; default = "${cfg.directory}/repos";
    };

    subDirectory = lib.mkOption {
      type = lib.types.str; default = "";
    };

    extraConfig = lib.mkOption {
      type = lib.types.str; default = "";
    };

    customStatic.enable = lib.mkOption {
      type = lib.types.bool; default = false;
    };
  };
}

Module Implementation

The difficulty of the implementation will depend on your familiarity with a piece of software and its operational dependencies. Lets create a cgit user, set its shell to git, bind the ssh authorized keys from the interface to the cgit user, and make the home directory the location that stores the repositories.

{
  config = lib.mkIf cfg.enable {
    users = {
      groups.${cfg.user} = { };
      users.${cfg.user} = {
        createHome = true;
        home = cfg.repository;
        isSystemUser = true;
        shell = "${pkgs.git}/bin/git-shell";
        openssh.authorizedKeys.keys = cfg.authorizedKeys;
        group = cfg.user;
      };
    };
  };
}

The home directory as the repository directory avoids configuring the git shell’s fetch and push remote urls. The ssh remote urls would be of the form owner/repository relative to that folder.

origin	cgit@thedroneely.com:thedroneely/cgit (fetch)
origin	cgit@thedroneely.com:thedroneely/cgit (push)

Using systemd as a crutch we ensure that the top level directory and the repository directory have the right permissions and exist. If services.cgit.customStatic.enable is true, then the module creates a static directory for custom assets.

{
  config = lib.mkIf cfg.enable {
    systemd.tmpfiles.rules = [
      "z ${cfg.directory} 755 ${cfg.user} ${cfg.user} - -"
      "d ${cfg.repository} 700 ${cfg.user} ${cfg.user} - -"
      (lib.optionalString (cfg.customStatic.enable) "d ${cfg.directory}/static 755 ${cfg.user} ${cfg.user} - -")
    ];
  };
}

At this point, enable old school FastCGI and let nginx know about the socket address. The regular for nginx are such that if services.cgit.subDirectory is an empty string it defaults to the root of the domain, otherwise it would be a subdirectory.

{
  config = lib.mkIf cfg.enable {
    services.fcgiwrap = { enable = true; user = cfg.user; group = cfg.user; };

    services.nginx.virtualHosts.${cfg.domain} = {

      locations."~* ^${cfg.subDirectory}/static/(.+.(ico|css|png))$" = {
        extraConfig = ''
          ${lib.optionalString (!cfg.customStatic.enable) "alias ${cfg.package}/cgit/$1;"}
          ${lib.optionalString (cfg.customStatic.enable) "alias ${cfg.directory}/static/$1;"}
        '';
      };

      locations."${cfg.subDirectory}/" = {
        extraConfig = ''
          include ${pkgs.nginx}/conf/fastcgi_params;
          fastcgi_param CGIT_CONFIG ${cgitrc};
          fastcgi_param SCRIPT_FILENAME ${cfg.package}/cgit/cgit.cgi;
          fastcgi_split_path_info ^(${cfg.subDirectory}/?)(.+)$;
          fastcgi_param PATH_INFO $fastcgi_path_info;
          fastcgi_param QUERY_STRING $args;
          fastcgi_param HTTP_HOST $server_name;
          fastcgi_pass unix:${config.services.fcgiwrap.socketAddress};
        '';
      };
    };
  };
}

Next we would need the most basic cgitrc configuration passed to nginx and fastcgi for cgit to operate, keeping in mind the possible options defined at the onset.

{ pkgs, lib, config, ... }:

let

  cgitrc = pkgs.writeText "cgitrc" ''
    css=${cfg.subDirectory}/static/cgit.css
    logo=${cfg.subDirectory}/static/cgit.png
    favicon=${cfg.subDirectory}/static/favicon.ico
    root-title=${cfg.title}
    root-desc=${cfg.description}
    snapshots=tar.gz tar.bz2 zip

    readme=:README
    readme=:readme
    readme=:readme.txt
    readme=:README.txt
    readme=:readme.md
    readme=:README.md

    ${cfg.extraConfig}

    about-filter=${cfg.package}/lib/cgit/filters/about-formatting.sh
    source-filter=${cfg.package}/lib/cgit/filters/syntax-highlighting.py
    remove-suffix=1
    section-from-path=1
    scan-path=${cfg.repository}
  '';

in { ... }

The final part is to set up mirroring and cloning from the interface. This is where it gets hairy. We can lean on systemd services to set up git mirroring and cloning services. The cloning services will probably be a one time action, but mirroring will happen periodically using timers. One could jam everything into a single systemd service but it would be nice to sort of “orchestrate” git cloning and mirroring over a set of systemd services and timers. To try this out — let’s add two options that expect a named attribute set of sub-modules where a repository url and a local owner are the options for cloning and mirroring.

{
  options.services.${service} = {

    mirrors = lib.mkOption {
      type = lib.types.attrsOf (lib.types.submodule {
        options = {
          owner = lib.mkOption { type = lib.types.str; };
          url = lib.mkOption { type = lib.types.str; };
          description = lib.mkOption {
            type = lib.types.str;
            default = "Unnamed mirrored repository; edit the description file to name the repository.";
          };
        };
      });
      default = { };
    };

    clones = lib.mkOption {
      type = lib.types.attrsOf (lib.types.submodule {
        options = {
          owner = lib.mkOption { type = lib.types.str; };
          url = lib.mkOption { type = lib.types.str; };
          description = lib.mkOption {
            type = lib.types.str;
            default = "Unnamed cloned repository; edit the description file to name the repository.";
          };
        };
      });
      default = { };
    };
  };
}

Writing the mirror and clone urls directly as attribute sets containing those sub-options is now acceptable.

{
  imports = [
    ../modules/cgit/module.nix
  ];

  services.cgit = {
    mirrors = {
      dotfiles = { owner = "thedroneely"; url = "https://github.com/tdro/dotfiles.git"; };
      "thedroneely.com" = { owner = "thedroneely"; url = "https://github.com/tdro/thedroneely.com"; };
    };
    clones = {
      cgit = { owner = "thedroneely"; url = "https://git.zx2c4.com/cgit"; };
    };
  };
}

Here’s a disclaimer; I’m not too with functional programming, but to achieve my configuration, it looks like somehow we would need to map over every attribute set and instantiate the option name–value pairs across every systemd service and timer.

In the Nix Package Manual there is an interesting function called lib.attrsets.mapAttrs' that allows changing the attributes of a given set by the name–value pairs of another attribute set. The mapping function has to return a name–value pair. this function inside nix repl confirms that behavior.

{
  nix-repl> :l <nixpkgs>
  nix-repl> :p lib.attrsets.mapAttrs' (name: value: lib.attrsets.nameValuePair "systemd-service-name-${name}" { option = value.one; } ) { service1 = { one = 1; two = 2; }; service2 = { one = 1; two = 2; }; }
  { systemd-service-name-service1 = { option = 1; }; systemd-service-name-service2 = { option = 1; }; }
}

Great now let’s set up the framework of systemd services that will mirror cgit repositories automatically using git clone --mirror.

{
  config = lib.mkIf cfg.enable {
    systemd.services = lib.attrsets.mapAttrs' (repo: mirror:
      lib.attrsets.nameValuePair "cgit-mirror-${mirror.owner}-${repo}" {
        description = "cgit repository mirror for ${mirror.url}";
        after = [ "network.target" ];
        path = [ pkgs.git pkgs.shellcheck ];
        script = ''
          set -euxo pipefail
          shellcheck "$0" || exit 1
          repo () { printf '${mirror.owner}/${repo}'; }
          mkdir -p "$(repo)"
          git clone --mirror '${mirror.url}' "$(repo)" || \
            git --git-dir="$(repo)" fetch --force --all
        '';
        serviceConfig = {
          User = cfg.user;
          Group = cfg.user;
          WorkingDirectory = cfg.repository;
        };
      }) cfg.mirrors;
  };
}

Using the operator (//), merge the systemd service attributes into a complete set that will also clone cgit repositories automatically using git clone --bare.

{
  config = lib.mkIf cfg.enable {
    systemd.services = lib.attrsets.mapAttrs' (repo: mirror:
      lib.attrsets.nameValuePair "cgit-mirror-${mirror.owner}-${repo}" {
        description = "cgit repository mirror for ${mirror.url}";
        after = [ "network.target" ];
        path = [ pkgs.git pkgs.shellcheck ];
        script = ''
          set -euxo pipefail
          shellcheck "$0" || exit 1
          repo () { printf '${mirror.owner}/${repo}'; }
          mkdir -p "$(repo)"
          git clone --mirror '${mirror.url}' "$(repo)" || \
            git --git-dir="$(repo)" fetch --force --all
        '';
        serviceConfig = {
          User = cfg.user;
          Group = cfg.user;
          WorkingDirectory = cfg.repository;
        };
      }) cfg.mirrors // lib.attrsets.mapAttrs' (repo: clone:
        lib.attrsets.nameValuePair "cgit-clone-${clone.owner}-${repo}" {
          description = "cgit repository clone for ${clone.url}";
          wantedBy = [ "multi-user.target" ];
          after = [ "network.target" ];
          path = [ pkgs.git pkgs.shellcheck ];
          script = ''
            set -euxo pipefail
            shellcheck "$0" || exit 1
            repo () { printf '${clone.owner}/${repo}'; }
            mkdir -p "$(repo)"
            git clone --bare '${clone.url}' "$(repo)" || true
          '';
          serviceConfig = {
            User = cfg.user;
            Group = cfg.user;
            WorkingDirectory = cfg.repository;
            RemainAfterExit = "yes";
          };
        }) cfg.clones;
  };
}

To wrap up, activate the mirroring services using systemd timers instantiated with the same mapping method.

{
  config = lib.mkIf cfg.enable {
    systemd.timers = lib.attrsets.mapAttrs' (repo: mirror:
      lib.attrsets.nameValuePair "cgit-mirror-${mirror.owner}-${repo}" {
        description = "cgit repository mirror for ${mirror.url}";
        wantedBy = [ "timers.target" ];
        timerConfig = {
          OnBootSec = "1";
          OnUnitActiveSec = "3600";
          RandomizedDelaySec = "1800";
        };
      }) cfg.mirrors;
  };
}

Maintenance

To keep the repositories well oiled, run git maintenance after mirroring and cloning. Running the maintenance command will reset cgit's idle time measurements. Fix the idle times by getting the latest commit time stamp and touching each repository’s packed-refs with that time stamp.

git --git-dir="$(repo)" maintenance run
touch -m --date "@$(git --git-dir="$(repo)" log -1 --format=%ct)" "$(repo)"/packed-refs || true

Conclusion

The power of nix in NixOS with its functional approach is truly wild. The language is viewed as and obscure by a but in about an hour of having to think recursively one could set up logic that connects the various parts of a software stack seamlessly with a high degree of correctness. Automation systems like Ansible begin to look like mere toys. Imagine if it ever reaches a point where one could swap out entire init systems seamlessly. Nix however comes with trade–offs, operations are slow, can take up lots of disk space and memory, and if it breaks — well, good luck.

Good, Evil, and The Law

Sat, 30 Jan 2021 04:30:47 +0000

Image by Prawny from Pixabay

intersects heavily with morality, ethics, and philosophy. The great moral quandaries of its intellectuals are boundless, but outside of their towers are those in the moral trenches compelled to answer the hard questions right now, consequences and all.

Individuals who probably did not expect to hold such deciding power are now answering the weighty philosophical questions one way or the other, and with the advent of encryption, federation, machine learning and digital currencies — one of those questions is the question of evil.

We could opine at great length about the present affairs but let’s instead take up an view to explore the question of evil as answered in the past. The King James will act as the entry point to discuss a subset of the historical answers to this philosophical question. I’m not a philosopher and surely don’t claim to be one, consider me a layman. There be multiple dragons below.

Why does evil exist?

Evil exists because the law exists. That was the easy answer in past ecclesiologies; but today it is and supposes an unconventional conclusion. If one accepts that evil exists because the law exists, then it must follow that without the law there is neither good nor evil, as there is no This idea is ancient. Paul, a reformed persecutor and Apostle of the first–century period ecclesia, draws a similar argument when reasoning with the Greeks in his letter to the Romans.


`Romans 4:15`	`Because the law worketh wrath: for where no law is, there is no transgression.`

Qualifying what it means to have no law is but first, let’s explore an implicit conclusion that follows if there is such a relationship between the law and evil. That conclusion is simple: To establish the law is to explicitly create good and evil. That surely sounds asinine, but the ecclesia embraced this summation completely. Take for example the 8th century BC prophet Isaiah as he speaks of Cyrus the Great, the prophet Micah speaking of Maroth, or Paul as he draws a strong comparison of a potter molding clay when speaking of supreme sovereignty.


`Isaiah 45:7`	`I form the light, and create darkness: I make peace, and create evil: I the LORD do all these things.`


`Micah 1:12`	`For the inhabitant of Maroth waited carefully for good: but evil came down from the LORD unto the gate of Jerusalem.`


`Romans 9:21`	`Hath not the potter power over the clay, of the same lump to make one vessel unto honour, and another unto dishonour?`

The implicit side effect of establishing the law is sometimes referred to as the on The paradoxical absurdity between, good, evil, and the law (God) under specific premises is sometimes called the logical problem of evil. For the sake of entertaining the question of evil further, this ecclesiological view of the law sidesteps the problem of evil as asking the wrong question.

Instead, if an establishes the law, and as a side effect creates good and evil, and thereby wields and enforces a monopoly on violence; then what is, in essence, good and evil? That question is easier to entertain with another question.

Who determines good and evil?

Why the government of course. If we can accept that a creator of the law possesses implicit knowledge of good and evil, then who better an expert on good and evil than the government itself. This isn’t a novel concept either, the book of Genesis says explicitly that those who know good and evil are gods.


`Genesis 3:5`	`For God doth know that in the day ye eat thereof, then your eyes shall be opened, and ye shall be as gods, knowing good and evil.`


`Matthew 22:21`	`They say unto him, Caesar's. Then saith he unto them, Render therefore unto Caesar the things which are Caesar's; and unto God the things that are God's.`

Let’s further say that relative to the government the average citizen possesses no knowledge of good or evil, that is — until that citizen hires a lawyer or becomes a lawyer. The lawyer adorns the government’s likeness and appears to wield its powers. In the ecclesia, lack of knowledge in the law is no saving grace. Violation in any point confers punishment just the same.


`Romans 2:11-12`	`For there is no respect of persons with God. For as many as have sinned without law shall also perish without law: and as many as have sinned in the law shall be judged by the law;`

Such conjecture reveals a more puzzling problem: Governments and citizens contemplate the nature of good and evil. This presents a set of dualisms and controversies that unravel between contemplators of good and evil and further expands into two opposing and genres of good and evil with its encapsulated. The knowledge of good and evil appears to differ between observers.

Good	Evil	Good	Evil
Rich	Poor	White	Black
Right	Wrong	Light	Dark


`Malachi 1:3`	`And I hated Esau, and laid his mountains and his heritage waste for the dragons of the wilderness.`


`Romans 9:13`	`As it is written, Jacob have I loved, but Esau have I hated.`

The existence of evil as we have seen is paralleled to the identification of good and evil. However, if the creator and the observer of the law both know good and evil — then who holds authority in knowing? Is it the creator of the law, or the observer of the law? This higher level dualism is unavoidable and is sometimes expressed today as apparent and absolute authority.


`Isaiah 5:20`	`Woe unto them that call evil good, and good evil; that put darkness for light, and light for darkness; that put bitter for sweet, and sweet for bitter!`

One could pontificate at length about the influence of knowledge on but one thing is apparent: If good and evil are opposing and abstract, then even observers create for others to follow. These observers paradoxically, become creators. This introduces the peculiar perplexity of multiple creators — a controversy of gods among gods. The spheres of influence from each god are in conflict continually, and the winner sets the rules

The principle of authority spheres is foundational, with examples all throughout the ecclesia. In short, any entity imputing good or evil, whether alive or dead, is not just like a god — but becomes one.


`Exodus 7:1`	`And the LORD said unto Moses, See, I have made thee a god to Pharaoh: and Aaron thy brother shall be thy prophet.`


`Isaiah 44:17`	`And the residue thereof he maketh a god, even his graven image: he falleth down unto it, and worshippeth it, and prayeth unto it, and saith, Deliver me; for thou art my god.`


`John 10:34`	`Jesus answered them, Is it not written in your law, I said, Ye are gods?`

Ideological battles are fought constantly and indefinitely within a larger divination war with varying levels of Many observe the law but only few create due to its paradoxical requirement of monopolized violence. Perpetual battles between our gods and would be overlords on Earth are fought to decide the decider of good and evil, which are opposing and abstract mights. In some ancient civilizations, child sacrifice was upheld as a moral good — Canaan’s ruler Molech is one such example.


`Leviticus 18:21`	`And thou shalt not let any of thy seed pass through the fire to Molech, neither shalt thou profane the name of thy God: I am the LORD.`

If we continue with this ancient ecclesiastical philosophy it starts to become easier to answer the question of why evil exists. The controversy of many gods fighting amongst each other is too problematic. A better approach requires peeking into the very essence of evil itself. One could sneak a glimpse at the essence of evil by removing all gods and consequently removing all laws. Paul a similar point with the Greeks.


`Romans 2:14-15`	`For when the Gentiles, which have not the law, do by nature the things contained in the law, these, having not the law, are a law unto themselves: Which shew the work of the law written in their hearts, their conscience also bearing witness, and their thoughts the mean while accusing or else excusing one another;)`

Those who have no laws are a law unto themselves, meaning observers in the face of no external laws, create internal laws. The laws that they make are scoped to a more natural and daresay noble This line of reasoning appears to zero in on a more satisfying answer, but asking the question of no law is to ask a final and more sinister question.

What is evil?

Take away all laws, by removing all gods, and we stare into the face of evil itself. is the source of fear — the primary essence of terror. A rather anticlimactic conclusion but a circumspectual answer in the ancient ecclesiologies.

Evil as the source of fear is a word play, but should not be taken in the sense of fear as the absolute root of all evil. The sense is to the idea that if pain is a source of fear, and suffering a source of fear, then pain, suffering, and evil would be apparent equivalents. To put it another way; fear becomes the source of a new evil.

The ultimate end of fear is the terror of death, preempting the final natural law that no man has escaped; all eventually die. This axiom pervades the minds and perturbs mortals continually. In the ecclesia, this internal law and its derivatives guards the fear of the final evil, the king of evils, and the king of terrors — death itself.


`Job 18:14`	`His confidence shall be rooted out of his tabernacle, and it shall bring him to the king of terrors.`


`Ecclesiastes 8:8`	`There is no man that hath power over the spirit to retain the spirit; neither hath he power in the day of death: and there is no discharge in that war;`

In the mortal’s mind, anything that so much as points towards death is evil. Within the scope of this idea, laws act as the fundamental instruments that regulate fear; the essence of terror, and subsequently the propensity of death. Any external entity perpetuating terror challenges the law directly in the most blasphemous way. It would be trivial to usurp or gather the powers of a god quickly with alone.


`Romans 13:3`	`For rulers are not a terror to good works, but to the evil. Wilt thou then not be afraid of the power? do that which is good, and thou shalt have praise of the same:`

Evil is also mutual; one who is afraid evil to remove the source of one’s fear. This is not particularly eye opening, but the fearful within the ecclesia are considered to be some of the most dangerous and unpredictable.


`Genesis 3:10`	`And he said, I heard thy voice in the garden, and I was afraid, because I was naked; and I hid myself.`


`Numbers 13:32`	`And they brought up an evil report of the land which they had searched unto the children of Israel, saying, The land, through which we have gone to search it, is a land that eateth up the inhabitants thereof; and all the people that we saw in it are men of a great stature.`


`Numbers 14:37`	`Even those men that did bring up the evil report upon the land, died by the plague before the LORD.`

Death, the king of terrors, additionally invites the infinite unknown. Foreign and unknown entities become implicitly evil. The unknown imputes fear, fear brings terror, and ultimate terror motivates death. Many civilizations have destroyed the unknown without second thought.

Paul’s take on the conscience in a calculative sense, turns the problem of mutual evil into multiple bounded or scoped problems of evil. The conscience acts as a delimit, such that if the evil then there is some route to The greater of the convergent evils isn’t always the target.

This is primarily the idea that the enemy of my enemy is my friend. If an external entity attacked the Earth, peace among Earthlings would be realized The attacking forces would become an convergence of evil, but in practicality, evil as it is perceived is mostly divergent — it cannot be pinpointed quickly or identified easily.

Lastly, the conscience and the knowledge of good and evil appear as two entirely different domains. Paul discusses this distinction when summarizing the ecclesia’s laws of liberty in the context of two men’s opposing consciences.


`1 Corinthians 10:29`	`Conscience, I say, not thine own, but of the other: for why is my liberty judged of another man's conscience?`

Conclusion

The shapeless conflict between good, evil, and the law is continual, its winners and losers painted throughout the halls of history. The technological landscape is transforming fast and in the real world, fear appears to be increasing steadily. What new evils are on the radar? Who will be the kingmakers? Who will be the Those who answer such questions are possibly the ones you least expect.

A Few Linux Distributions

Tue, 22 Dec 2020 03:41:52 +0000

Tux

My devices have seen numerous Linux distributions over the course of my computing journey. A Linux distribution is an operating system built around the Linux kernel. Many new comers to the Linux community are evangelised in, but others come in from a lack of computing freedom.

I’m of the latter camp. If you are a new comer to the Linux community, then choose the first distribution that suits your fancy. Don’t waste time determining which is best. To put it bluntly — arguments of the “best” are over mind share and collective use cases.

Programmers and computing enthusiasts enjoy dictating how software should be used. Terms such as “best practices”, and “anti-patterns” pop up frequently — ignore them for the most part. In time, you’ll be able to discern if such advice warrants a change in operating systems, use cases, or specifications. That being said, here’s my personal review of a few Linux distributions used over the years.

Ubuntu

Ubuntu is probably the most well known distribution as it is backed by the company Canonical. This was my first ever Linux distribution, and most new users often end up here. If casual computing is the goal then it’s an okay distribution. Occasionally the business interests of Canonical clash with the user base, motivating some to other distributions. The updates are and the package managers are apt and snap.

In the server world Ubuntu is also popular. You can be assured that many companies run Ubuntu as production servers. My reason for keeping Ubuntu around is due to its popularity in production.

Debian

Debian is a distant cousin to Ubuntu. This is the only distribution that one can install on a user’s computer and receive no complaints. Debian is built to be stable, conservative, and “boring”. The updates are fixed and the package manager is apt.

You can’t go wrong using Debian as a server in production. The conservative package repository boasts exceptional reliability at the cost of lacking the latest features.

Arch

Arch is a popular distribution that is best known by its Arch Wiki and Arch User Repository (AUR). The package repository is expansive and bleeding edge. Arch adheres to the philosophy, making it a simple but flexible distribution. Due to its massive mind share, a user can trivially find and install anything they want. The updates are and the package manager is pacman.

Using Arch as a production server is frowned upon, but can be done. A production Arch server software faster than any other distribution, but at the cost of extreme volatility from moving targets.

Manjaro

Manjaro is an Arch based distribution with the aim of being more user friendly. Many casual users enjoy this Linux distribution. It’s a feeder distribution into the Arch philosophy. The updates are rolling release and the package managers are pacman and pamac.

Void

Void is sometimes referred to as a Linux distribution. The base system is compact and has sane defaults. Void Linux is popular among technical users as it cuts through a lot of the fat in other modern distributions. The updates are rolling release and the package manager is xbps.

Alpine

Alpine is a lightweight security focused distribution. Personally, Alpine is superior to all other distributions in terms of size, simplicity, and productivity. Alpine is not user desktop oriented, but for servers it is the dream. If you have a choice, use Alpine as a server. The updates are both rolling and fixed and the package manager is apk — probably the fastest and simplest package manager to date.

CentOS (Rocky Linux)

CentOS, soon to be Rocky Linux is an enterprise oriented distribution backed by Redhat. CentOS is inherently stable and conservative due to its focus on the enterprise use case. Business types prefer safe options, so you’ll only tend to see this system running at companies and corporations as CentOS or Red Hat Enterprise Linux (RHEL). The updates are fixed and the package managers are yum and dnf.

NixOS

NixOS is a new of Linux distribution that focuses on reproducibility and immutability. The operating system is built from a single file that resembles a specification. NixOS allows you to remove the complexity of other domains entirely and stand on the shoulders of — but at the cost of more complexity in the form of an incongruent abstraction layer.

This is my least recommended Linux distribution because some use cases require extensive knowledge of how a typical Linux operating system works. The abstraction is not intuitively consistent and you must know how to fix an underlying problem when it breaks. When you don’t have to fight the abstraction, it’s truly a computing dream come true. The updates are both rolling and fixed and the package manager is nix.

Keeping Up with Open Source

Sat, 31 Oct 2020 02:48:35 +0000

RSS feeds of interest

A good way to improve your programming skills is to read the code that other people write. Reading code for the sake of reading code isn’t optimal — it’s more beneficial to read the code of programs that you like or use often in bite-sized chunks.

My favorite ways to track open source programs are RSS (Really Simple Syndication) feeds, mailing lists, and forum discussions. If one is able to sufficiently reason about a code base, then keeping on top of changes is easy, and takes minutes a day.

The benefits are practical and time saving. You’ll know the expected changes in upcoming releases, issues that other people have, patches that contributors want to integrate, and other useful information.

RSS Feeds

RSS (Really Simple Syndication) is a great way to keep track of various open source projects. GitHub hosts most of the world’s open source projects and provides RSS feeds (atom) for commits and releases.

If a project is slow moving subscribe to the commits. https://github.com/Ralim/ts100/commits.atom
If it has good release notes subscribe to the releases. https://github.com/Ralim/ts100/releases.atom

Good change logs are a rarity these days, but an RSS feed will allow you to know what has changed as soon as it happens. Countless RSS feed readers are available. Use one that you

Forum Discussions

Forum discussions are excellent for news on open source programs that you use often. Subscribe to any forums either by registering or by using RSS. Here’s a subset of forums that I follow through RSS.

Arch Linux: Arch Linux is a bleeding edge Linux distribution. Subscribe to their large offering of feeds for updates, and information on numerous open source programs.
Reddit: Reddit has a large offering of forums (subreddits). If you see that a subreddit exists for a program — add .rss to the URL to get the RSS feed link. If the RSS for a subreddit is broken use inline reddit. The discussions are usually niche enough to be useful.

Mailing Lists

Mailing lists are considered old school, but contain lots of good information. Subscribe to any mailing list available for software that you use. Subscribing is as easy as having an email address and following the instructions for that mailing list. Here’s a subset of mailing lists that I follow from time to time.

NixOS in The Wild

Mon, 14 Sep 2020 18:17:49 +0000

NixOS on a T420

Nix and NixOS are two technologies that my eyes have been on for a few years. Nix is the independent package manager that allows for and immutable package managment and NixOS is a Linux distribution built from the ground up using Nix.

of my personal infrastructure now resides on NixOS including the server that hosts this website. CentOS, the previous Linux distribution running this server has done

14:39:28 up 502 days, 15:30,  1 user,  load average: 0.00, 0.01, 0.05

Let’s discuss the points, use cases, and general application of NixOS in the real world for both desktops and servers but before that — here’s my experience with the package repository and documentation search discovery.

The Package Repository

Nix Packages Collection

The nix Package Repository or nixpkgs contains every package derivation. My one pain point is that GitHub chokes on this repository hard. Searching the right term over 4,000+ issues can intermittently the page. This is not common but happens often enough to notice. In my experience, GitHub issue discovery is better achieved using a search engine that supports site queries.

site:https://github.com/NixOS/nixpkgs/issues <keyword>

Online Documentation and Search Discovery

In my experience, the online documentation has poor search discoverability. Nix’s search discoverability is the result of generating the majority of the documentation as a single HTML page. These pages are impeccable under the hood with anchor id links on each section and subsection, but search are not smart enough to scope out this context and link directly to the anchor id fragments on the search engine results page (SERP). Most search queries send you to the top of the nix package manual.

Searching for phases on Google

Local Documentation and Troubleshooting

NixOS provides a lot of documentation out of the box that is specific to every installation. Users familiar with man will run man configuration.nix to see config available for a system. This means that the local configuration.nix documentation is more accurate than searching the online NixOS options index.

NixOS configuration specification

In fact, the entire package repository (nixpkgs) resides on every installation as a channel. In the above manpage nixpkgs/nixos/modules/config/appstream.nix declares the logic behind appstream.enable. View this file by listing the channel name as root and navigating to the per user channel source tree.

$ sudo nix-channel --list
nixos https://releases.nixos.org/nixos/20.03/nixos-20.03.2351.f8248ab6d9e

$ tree -L 1 /nix/var/nix/profiles/per-user/root/channels/nixos
|__ COPYING
|__ default.nix
|__ doc
|__ flake.nix
|__ lib
|__ maintainers
|__ nixos
|__ nixpkgs -> .
|__ pkgs
|__ programs.sqlite
|__ README.md
|__ svn-revision

$ nix-instantiate --eval -E '(import <nixpkgs> {}).lib.version'
"20.03.2351.f8248ab6d9e"

This is good to know even if you don’t care about the source purely as a reminder that not every system has the same state or exposes the same configuration options.

User Autologin

Most of my desktop machines do not use login managers. The auto login settings of NixOS are login manager dependent. Apparently it’s trivial to override the autovt@tty1 service and force auto login for any user without having to install a login manager. Set restartIfChanged to false to avoid restarting your desktop.

{ pkgs, ... }:

{
  systemd.services."autovt@tty1" = {
    after = [ "systemd-logind.service" ];
    restartIfChanged = false;
    serviceConfig = {
      Type = "simple";
      ExecStart = "${pkgs.utillinux}/sbin/agetty --autologin ${username} --noclear %I $TERM";
      Restart = "always";
    };
  };
}

The above will not work in future versions, as it is simplified to the option services.getty.autologinUser. Be careful though, enabling this option auto logs into every getty and the first activation restarts the currently running session.

{
  services.getty.autologinUser = "username";
}

Updating NixOS

Updating NixOS is tricky business. The short of it is that generally the system wide channel is the repository used to source updates. Most systems have one channel called nixos or nixpkgs under the user root. Every user inherits the root user’s channel unless they have set their own per user channel.

Using root list the channel with nix-channel --list
Override or add a channel with nix-channel --add
Update the channel with nix-channel --update
Upgrade every per user environment with nix-env --upgrade
the system with nixos-rebuild switch --upgrade

# nix-channel --list
nixos https://nixos.org/channels/nixos-20.03
# nix-channel --add 'https://nixos.org/channels/nixos-20.03' nixos
# nix-channel --update
# nix-env --upgrade
# nixos-rebuild switch --upgrade

If you trust the stability, you can use system.autoUpgrade.enable and system.autoUpgrade.channel, but those are scary options for my use case.

Shellcheck and systemd

Every language, system, or framework leads you down a certain path. NixOS coaxes you down the path of writing lots of shell scripts with systemd. Save yourself the pain — set the shell to a stricter mode and run shellcheck upon service execution.

{ pkgs, ... }:

{
  systemd.services.my-service = {
    description = "My service";
    wantedBy = [ "multi-user.target" ];
    path = [ pkgs.shellcheck ];
    script = ''
      set -euxo pipefail
      shellcheck "$0" || exit 1
      # Code goes here...
    '';
  };
}

LibreOffice and Spell Checking

Spell checking is a commonly used feature of LibreOffice. On my NixOS 20.03 machines, LibreOffice spell checking doesn’t seem to work. Add the missing dictionary package links and expose them using DICPATH. Log out and back in and LibreOffice should find the spell checking modules.

{ pkgs, ... }:

{
  environment.systemPackages = with pkgs; [
    hunspell
    hunspellDicts.en_US-large
    hyphen
  ];

  environment.pathsToLink = [ "/share/hunspell" "/share/myspell" "/share/hyphen" ];

  environment.variables.DICPATH = "/run/current-system/sw/share/hunspell:/run/current-system/sw/share/hyphen";
}

Hashless Git Fetching

Use the function builtins.fetchGit to fetch the HEAD of a repository branch directly without stipulating a hash. The builtin function fetchGit will fetch any git repository at evaluation time — allowing for automation, ssh-agent integration, and other niceties.

{ stdenv }:

let url = "https://github.com/koalaman/shellcheck"; in

stdenv.mkDerivation rec {

  pname = "shellcheck";
  version = "master";

  src = builtins.fetchGit { inherit url; ref = "refs/heads/master"; };

  dontBuild = true;

  installPhase = ''
    runHook preInstall
    mkdir $out
    # Code goes here...
    runHook postInstall
  '';
}

Package Tracking and systemd

In a continuous integration and deployment (CI/CD) environment you can set a systemd service to restart on package changes. Use restartTriggers to track upstream package definitions.

{ pkgs, ... }:

let package = pkgs.callPackage ./default.nix {}; in

{
  systemd.services.my-service = {
    description = "My service";
    wantedBy = [ "multi-user.target" ];
    restartTriggers = [ package ];
    path = [ pkgs.shellcheck ];
    script = ''
      set -euxo pipefail
      shellcheck "$0" || exit 1
      # Code goes here...
    '';
    serviceConfig = { RemainAfterExit = "yes"; };
  };
}

Internet Connectivity in the Nix Sandbox

Nix build environments prohibit internet connections outside the scope of defined functions like fetchgit or fetchurl. You can remove this restriction by setting the correct expected hash recursively before evaluation. This allows us to do “illegal” things like the following.

{ stdenv, pkgs }:

let url = "https://thedroneely.com/git/thedroneely/thedroneely.com.git"; in

stdenv.mkDerivation rec {

  pname = "composer";
  version = "master";

  src = builtins.fetchGit { inherit url; ref = "refs/heads/master"; };

  buildInputs = [ pkgs.cacert pkgs.php74Packages.composer ];

  dontBuild = true;

  installPhase = ''
    runHook preInstall
    composer --no-cache install
    mkdir $out
    cp -r vendor $out/vendor
    runHook postInstall
  '';

  outputHashAlgo = "sha256";
  outputHashMode = "recursive";
  outputHash = "0zkqkbwz5vg4k95s83pl0kxvphav1wzmivs5b1kmwf101wnj1m4q";
}

Check out this neat blog for more interesting approaches and tips of this nature.

Kernel Patching for Kids

Applying kernel patches using fetchurl in NixOS is a trivial endeavour. You should manually patch, configure, and compile a kernel at least once to understand what’s happening.

{
  boot.kernelPatches = [

    { name = "ck-5.6"; patch = (builtins.fetchurl {
      url = "http://ck.kolivas.org/patches/5.0/5.6/5.6-ck2/patch-5.6-ck2.xz";
      sha256 = "18rk9023b14x62n0ckbnms6ahq5yjramz7qfjagkaga95i8ha6b2"; });
    }

    { name = "uksm-5.6"; patch = (builtins.fetchurl {
      url= "https://raw.githubusercontent.com/dolohow/uksm/master/v5.x/uksm-5.6.patch";
      sha256 = "021sylwacamh8q26agcp0nmmw3ral2wl7bgibmi379irnvy0c37y"; });
    }

    { name = "userns-overlayfs"; patch = (builtins.fetchurl {
      url= "https://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/patch/?id=0c29f9eb00d76a0a99804d97b9e6aba5d0bf19b3";
      sha256 = "1j4ind31hgkjazbgfd64lpaiqps8hcsqkar4v6nvxrpysmkg9nfd"; });
    }

  ];
 }

Nginx and its Temporary Folders

The permissions and ownership on nginx’s temporary folders can change in peculiar circumstances. There is a chance that the user nginx becomes dissociated from its folders when disabling and re-enabling nginx using the option services.nginx.enable. The temp folders for nginx are in /var/spool/nginx on system version 20.03. Use systemd’s tmpfiles.d to ensure that these permissions always stay consistent. This is solved in version 20.09.

{
  systemd.tmpfiles.rules = [
    "z /var/spool/nginx                  0700 nginx nginx -"
    "z /var/spool/nginx/client_body_temp 0700 nginx nginx -"
    "z /var/spool/nginx/fastcgi_temp     0700 nginx nginx -"
    "z /var/spool/nginx/logs             0700 nginx nginx -"
    "z /var/spool/nginx/proxy_temp       0700 nginx nginx -"
    "z /var/spool/nginx/scgi_temp        0700 nginx nginx -"
    "z /var/spool/nginx/uwsgi_temp       0700 nginx nginx -"
  ];
}

The Hash as Truth

A common mode of failure when working with Nix and NixOS is that happens when you try to apply a change. Nix looks at the changes of a package’s derivation hash to decide if something should be rebuilt and applied. This also means that nix knows the hashes of already built resources.

If you want to set a new sha256 and force a rebuild: don’t place a random dummy string or change one of the characters in the original sha256. Always use lib.fakeSha256 or the command nix-prefetch-url to fetch the new sha256 hash.

The reason is simple — if you happen to use an already known hash, then nix may download the correlated binaries from a cached location and leave you with an interesting debugging session. The sha256 attributes always take precedence — changing only the pname or version attributes in most cases will have no effect.

{ lib, stdenv, fetchurl }:

stdenv.mkDerivation rec {

  pname = "puppeteer-docs";
  version = "latest";

  src = fetchurl {
    url = "https://raw.githubusercontent.com/puppeteer/puppeteer/main/docs/api.md";
    sha256 = lib.fakeSha256;
  };

  phases = [ "installPhase" ];

  installPhase = ''
    runHook preInstall
    mkdir -p $out
    cp ${src} $out/api.md
    runHook postInstall
  '';

  meta = with lib; {
    description = "Puppeteer Documentation";
    homepage = "https://github.com/puppeteer/puppeteer/blob/main/docs/api.md";
  };
}

Nix Commands and Local Caching

Nix is slow and caches to improve speed. Local caching can sometimes interfere with automation tasks when using commands like nix-build or nixops. Use the --option argument and set the tarball-ttl to 0 to ensure you are always pulling fresh sources and repositories.

$ nixops deploy -d deployment --option tarball-ttl 0
$ nix-build --option tarball-ttl 0

If you are running a lean system, make the time to live (TTL) settings permanent with nix.extraOptions in the system configuration.

{
  nix.extraOptions = ''
    tarball-ttl = 0
    narinfo-cache-negative-ttl = 0
    narinfo-cache-positive-ttl = 0
  '';
}

Logs and Logrotate

On NixOS, it’s systemd the way down. The important logs for most services route to systemd’s journald. The defaults are okay, but logrotate is a crucial program in my stack.

The abstraction for logrotate is fine, and it’s trivial to mimic my preferred setup as it would be on any traditional Linux distribution. Enable logrotate with a default preset.

{
  services.logrotate = {
    enable = true;
    config = ''
      compress
      create
      daily
      dateext
      delaycompress
      missingok
      notifempty
      rotate 31
    '';
  };
}

Create a per service setup using the config anywhere in your configuration, but preferably in the same file as the service in question (nginx).

{
  services.logrotate.config = ''
    /var/spool/nginx/logs/*.log {
      create 644 nginx nginx
      postrotate
        systemctl reload nginx
      endscript
    }
  '';
}

Using the Force

Breaking the assumptions of abstractions and frameworks is a hobby of mine. Sometimes there are special situations where the abstractions get in the way of the desired configuration. Use lib.mkForce as a forcing action to any configuration option and prevent other modules from interfering. The name service switch configuration is written to /etc/nsswitch.conf using environment.etc and lib.mkForce to prevent casualties.

{ lib, ... }:

{
  environment.etc."nsswitch.conf".text = lib.mkForce ''
    rpc: files
    shadow: files
    ethers: files
    networks: files
    services: files
    netgroup: files
    publickey: files
    protocols: files
    group: files systemd
    passwd: files systemd
    hosts: files mymachines myhostname libvirt libvirt_guest mdns4_minimal resolve [!UNAVAIL=return] dns
  '';
}

Programming Sans Internet

Thu, 30 Jul 2020 16:11:25 +0000

The Internet is down

Software development is tightly coupled to the Internet. Some programs refuse to run or fail catastrophically without an Internet connection. For most people a computer becomes a useless pile of metal without the Internet. My circumstances have required the dreaded skill of programming without the Internet. The and infrastructure in my country royally sucks.

Keeping in a state of “flow” when the Internet goes down is difficult. Let’s take a look at my basic approach to remedying the difficulty of programming without an Internet connection.

Offline The Documentation

Many kind people exist in this world. Some of them write manuals that no one reads. Linux based distributions have first class support for POSIX style programming. One could run man to bring up any function or program instruction in the POSIX programmer’s manual. If we wanted to know how to use mmap we could run man mmap.

Manpage for mmap

Manpages are so old school now that it’s somewhat rare to see anyone using man let alone localized manuals.

“Knowledge is of two kinds. We know a subject ourselves, or we know where we can find information upon it. When we enquire into any subject, the first thing we have to do is to know what books have treated of it. This leads us to look at catalogues, and at the backs of books in libraries.” Samuel Johnson, The Life of Samuel Johnson

Manpages are faster than a search engine lookup if you know what you are looking for. Search engine lookups can be expensive switching actions, but you often get what you’re looking for quickly. Good manuals contain a synopsis of a program or function, an example of its usage pattern, return value references, and most important of them all — error handling instructions.

Ensure that you have manpages for all tools and target languages that you work with if available. Check your local distribution repositories to see if there are any extra manuals available. Run man -k . to see manuals on the system. Display manuals containing a keyword with man -k <keyword>. manuals to PDF format by using man -Tpdf <keyword>. Pipe the output into the PDF reader of choice.

man -Tpdf mmap | zathura -

Zeal offers an offline graphical documentation browser for many popular programming APIs (Application Programming Interfaces). Install zeal and download the documentation sets that are relevant to your programming environment.

Zeal browser showing JavaScript’s Math.floor() function documentation.

Offline The Wiki

The Arch Wiki is a well known and respected documentation source. If you find a wiki that you visit more than once — offline it. Relegate any downloaded wiki to a directory where you can grep (global regular expression print) or search through the files.

This applies doubly so for any GitHub wiki. Extract markdown copies of a repository’s wiki by using the git clone command.

git clone https://github.com/koalaman/shellcheck.wiki.git

GitHub’s pull requests and issues contain critical information. Grab all pull requests and issues from any working using the Github API (Application Programming Interface).

https://api.github.com/repos/<repo-owner>/<repo-name>/issues?state=all
https://api.github.com/repos/<repo-owner>/<repo-name>/pulls?state=all

Offline The REPLs

REPL stands for read, evaluate, print, and loop. It’s a command line shell for your language. Get comfortable using a REPL just like any other familiar shell to reduce Internet lookups and context switching. Most of us use our heads as rudimentary REPLs — basically, we write a piece of code with a general imagination of its action and return state.

Eventually there’s a point where you’re doing that weird JavaScript style crash and burn programming. The target code is now large enough that it can’t fit inside your head, so you’ll edit what you approximate is wrong, and run it again to see if it doesn’t crash and burn. Repeat ad nauseam.

Unpacking a tricky error state is often much quicker in a REPL than with an Internet search or using a crude crash and burn method. This saves time and having to context switch to a search lookup for some common or obscure error message. Make sure to have all REPLs offline for any language or tool you use if available.

Messing around with Elixir’s REPL iex in Vim

Offline The Static Code Analysis

Static code analysis offers refactoring advice, sniffing out high level coding errors, and correcting naive approaches to simple problems. Static code analysis tools can hold your hand and carry you a good distance. In fact, there are some static analysis tools that are so that if you have good cursory knowledge of a language, you can learn the ins and outs quicker than reading a book.

The authors of static code analysis tools know more than you — it’s as simple as that. Static analysis coalesces a collection of common questions, answers, and pitfalls that experienced people have encountered time and time again. Even based programs like Nginx have their own static analysis tools like gixy. Ensure you have all static analysis tools for any language or program that you work with offline.

Offline The Editor

Most programmers use integrated development environments (IDE) nowadays which handles most of the above (with or without Internet probably), but in a situation without an Internet connection — having a flexible editor is paramount.

This means an editor like vim or emacs. In the wastelands, you don’t have the luxury of downloading a plugin or extra feature to assist you. Flexible editors have configuration that is dynamic enough to code your own plugins trivially on the fly.

Emacs uses Emacs Lisp, and Vim uses Vimscript. Vim itself has one of the most thorough manuals in existence, so implementing an ad hoc plugin for an odd workflow is not too difficult as long as you know how to read.

Offline NixOS Documentation

NixOS offline documentation is impeccable. You can run the command line tool man on configuration.nix, nix.conf, nixos-rebuild and more. The is the nix REPL. The nix REPL doesn’t have the ability to lookup function documentation strings. A function’s documentation requires a search of the source code, and that’s not fun. The plugin nix-doc adds this feature to the nix REPL. Let’s wire it up while we wait on upstream. Create a basic default.nix for nix-doc using rustPlatform.

{ lib, rustPlatform, fetchgit, pkgs }:

rustPlatform.buildRustPackage rec {
  pname = "nix-doc";
  version = "v0.3.1";

  src = fetchgit {
    rev = version;
    url = "https://github.com/lf-/nix-doc.git";
    sha256 = "1hiz0fsbsl74m585llg2n359gs8i2m6jh8zdikkwxd8xq7qmw032";
  };

  buildInputs = with pkgs; [ boost nix ];

  nativeBuildInputs = with pkgs; [ pkg-config ];

  cargoSha256 = "1d1gvx14yai4dyz44pp2ffx2swfxnm0fwvldy96dw9gqmar69cpv";

  meta = with lib; {
    homepage = url;
    license = licenses.lgpl3;
    description = "A Nix documentation lookup tool that quickly dumps the docs of the library function";
  };
}

Install the program using environment.systemPackages and its plugin using nix.extraOptions where ./nix-doc/default.nix is the path to the derivation.

{ pkgs, ... }:

{
  nix.extraOptions = ''
    plugin-files = ${pkgs.callPackage ./nix-doc/default.nix {}}/lib/libnix_doc_plugin.so
  '';

  environment.systemPackages = with pkgs; [
    (callPackage ./nix-doc/default.nix {})
  ];
}

The nix REPL now has access to the documentation lookup function builtins.doc and the command line utility nix-doc.

NixOps: Towards The Final Frontier

Mon, 29 Jun 2020 07:27:53 +0000

NixOps Source Page

NixOps (Nix Operations) is a program for deploying and provisioning multiple NixOS machines in a network or cloud environment. This write up is by no means an guide but rather a commentary on my nixops approach. You can also check out krops, morph, and deploy-rs as community alternatives to nixops. You may also want to use the simpler nixos-rebuild command.

nixos-rebuild switch \
  --target-host "nix@remote.host" \
  --build-host "localhost" \
  --no-build-nix

Bootstrapping Remote Machines

NixOps ssh access to the root user, therefore you should bootstrap a remote machine using its configuration.nix. Power users will use NixOS generators to up this process.

{
  services.openssh = {
    enable = true;
    permitRootLogin = "prohibit-password";
    passwordAuthentication = false;
    challengeResponseAuthentication = false;
    extraConfig = "Compression no";
  };

  users.users.root = {
    openssh.authorizedKeys.keyFiles = [ ./ssh-key.pub ];
  };
}

The user’s authorized_keys is set using the option users.users.<name?>.openssh.authorizedKeys.keyFiles. Do not use users.users.<name?>.openssh.authorizedKeys.keys to configure public keys as it replaces the authorized_keys that nixops sets on the remote machine.

Once nixops interfaces successfully with a remote machine it sets its public key, as well as a private called id_charon_vpn in the user’s .ssh folder. The remote machines are now ready to accept connections from nixops.

Directory Structure

Let’s take a look at my directory before creating a deployment using nixops.

|__ configuration.nix
|__ deployments.nixops
|__ configuration
|   |__ heron.nix
|   |__ pigeon.nix
|__ hardware
|   |__ heron.nix
|   |__ pigeon.nix
|__ helpers
|   |__ extra-builtins.nix
|   |__ wrappers
|   	|__ vault
|__ keys
|   |__ thedroneely.com.nix
|__ mailers
|__ packages
|__ programs
|   |__ nix.nix
|__ servers
|__ users
|   |__ pigeon.nix
|   |__ root.nix
|   |__ thedro.nix
|__ virtualizers
|__ websites
    |__ thedroneely.com.nix

The deployment entry point is at configuration.nix. The machine specific configuration.nix and hardware.nix are in the configuration and hardware folders respectively. The deployment is tracked in deployments.nixops.

This directory contains helpers, program specific configurations, as well as files responsible for secrets management. For example — this website is realized from an entry point that imports and implements every dependency it needs to function.

{ config, lib, pkgs, ... }:

{
  imports = [
    ../../keys/deployments/thedroneely.com.nix
    ../../servers/cgit/service.nix
    ../../servers/goaccess/service.nix
    ../../servers/isso/service.nix
    ../../servers/nginx/service.nix
    ../../servers/phpfpm/service.nix
    ../../servers/postgresql/service.nix
    ../../servers/rainloop/service.nix
  ];
}

NixOps Deployment and Commands

Populate the configuration entry point configuration.nix with a list of machines to provision in the deployment.

{
  network.description = "nixos";
  network.enableRollback = true;

  heron = {
    imports = [ ./hardware/heron.nix ./configuration/heron.nix ];
    deployment.targetHost = "heron.local";
  };

  talon = {
    imports = [ ./hardware/talon.nix ./configuration/talon.nix ];
    deployment.targetHost = "talon.local";
  };

  tiger = {
    imports = [ ./hardware/tiger.nix ./configuration/tiger.nix ];
    deployment.targetHost = "tiger.local";
  };

  hound = {
    imports = [ ./hardware/hound.nix ./configuration/hound.nix ];
    deployment.targetHost = "hound.local";
  };

  pigeon = {
    imports = [ ./hardware/pigeon.nix ./configuration/pigeon.nix ];
    deployment.targetHost = "pigeon.local";
  };
}

This entry point declares five machines to be operated on — each with their own hardware and configuration specific assets under the network nixos. Rollbacks are enabled for the nixops operator to switch generations in the case of a mishap.

Make a deployment using create by referencing the entry point file path and setting the location of the deployment state file using --state.

nixops create --deployment nixos --state deployments.nixops configuration.nix

You can also adjust the entry point file paths using modify. Below two files configuration.nix and extra.nix are used as entry points.

nixops modify --deployment nixos --state deployments.nixops configuration.nix extra.nix

List all available configurations using the list argument.

$ nixops list --state deployments.nixops
+--------------------------------------+-------+-------------+------------+------+
| UUID                                 | Name  | Description | # Machines | Type |
+--------------------------------------+-------+-------------+------------+------+
| f56d276b-af54-11ea-b171-02422b1a33b6 | nixos | nixos       |          5 | none |
+--------------------------------------+-------+-------------+------------+------+

the deployment nixos on all machines listed in the entry point configuration using the deploy argument.

nixops deploy --deployment nixos --state deployments.nixops

Run the deployment nixos but only on the machine with the cryptonym heron using --include.

nixops deploy --deployment nixos --state deployments.nixops --include heron

Show the state for deployment nixos using the info argument. This command lists the status and current state of each machine.

$ nixops info --deployment nixos --state deployments.nixops

Network name: nixos
Network UUID: f56d276b-af54-11ea-b171-02422b1a33b6
Network description: nixos
Nix expressions: /nixos/configuration.nix
Nix profile: /nix/var/nix/profiles/per-user/thedro/nixops/f56d276b-af54-11ea-b171-02422b1a33b6

+--------+-----------------+------+----------------------------------------------------+------------+
| Name   |      Status     | Type | Resource Id                                        | IP address |
+--------+-----------------+------+----------------------------------------------------+------------+
| heron  |  Up / Outdated  | none | nixops-f56d276b-af54-11ea-b171-02422b1a33b6-heron  |            |
| hound  |  Up / Outdated  | none | nixops-f56d276b-af54-11ea-b171-02422b1a33b6-hound  |            |
| pigeon | Up / Up-to-date | none | nixops-f56d276b-af54-11ea-b171-02422b1a33b6-pigeon |            |
| talon  |  Up / Outdated  | none | nixops-f56d276b-af54-11ea-b171-02422b1a33b6-talon  |            |
| tiger  | Up / Up-to-date | none | nixops-f56d276b-af54-11ea-b171-02422b1a33b6-tiger  |            |
+--------+-----------------+------+----------------------------------------------------+------------+

Examine the machines more closely with the check argument. This command provides extra information such as load averages and failed units.

$ nixops check --deployment nixos --state deployments.nixops --include tiger pigeon
Machines state:
+--------+--------+-----+-----------+----------+----------------+----------------------------------------+-------+
| Name   | Exists | Up  | Reachable | Disks OK | Load avg.      | Units                                  | Notes |
+--------+--------+-----+-----------+----------+----------------+----------------------------------------+-------+
| pigeon | Yes    | Yes | Yes       | N/A      | 0.16 0.07 0.04 | proc-sys-fs-binfmt_misc.mount [failed] |       |
| tiger  | Yes    | Yes | Yes       | N/A      | 0.11 0.09 0.03 | proc-sys-fs-binfmt_misc.mount [failed] |       |
+--------+--------+-----+-----------+----------+----------------+----------------------------------------+-------+

List the generations for deployment nixos. The deployment fleet is currently on generation 203.

$ nixops list-generations --deployment nixos --state deployments.nixops
 200   2020-06-26 00:28:10
 201   2020-06-26 04:14:40
 202   2020-06-26 04:28:12
 203   2020-06-26 21:58:55   (current)

Rollback the deployment to a previous generation with the rollback argument. Use --include to target specific machines.

nixops rollback 201 --deployment nixos --state deployments.nixops --include heron ferret

Delete a deployment with the delete argument but you must first remove the machine resources with destroy.

nixops destroy --deployment nixos --state deployments.nixops --include heron
nixops delete --deployment nixos --state deployments.nixops

NixOps Secrets Management

The nix store at /nix/store is world readable. Secrets entered directly into any nix configuration will be available to all users on the system. Avoid placing secrets into a nix configuration directly — as it will leak to unprivileged users. NixOps provides a powerful secrets management system in the form of password files.

Using a password file provides indirection and guards against writing secrets directly into your nix files. NixOps writes deployment keys to /run/keys. Users must be a part of the key group to access deployment keys on the system.

NixOps can set the permissions, owners, and groups of each key. In my case, the come from a Vault server wrapper defined in builtins.extraBuiltins. This blog post gives an excellent rundown on this technique, and this wiki compares the different secret management approaches. A sample of the nix secrets configuration for this website is shown

{ config, ... }:

let user = "thedroneely"; in

{
  deployment.keys = {

    thedroneely_cockpit_api_token = {
      user = user;
      text = "${builtins.extraBuiltins.vault "thedroneely/thedroneely.com" "cockpit_api_token"}";
    };

    thedroneely_pgsql_database_password = {
      user = user; group = "postgres"; permissions = "0640";
      text = "${builtins.extraBuiltins.vault "thedroneely/thedroneely.com" "pgsql_database_password"}";
    };

    thedroneely_ssh_known_hosts = {
      inherit user;
      text = "${builtins.extraBuiltins.vault "thedroneely/thedroneely.com" "ssh_known_hosts"}";
    };
  };
}

Enable extraBuiltins by linking directly to the nix-plugins package on the nix operator host. Allow users in @wheel to become trusted users and wield this ultimate power.

{ pkgs, config, ... }:

{
  nix.extraOptions = ''
    trusted-users = root @wheel
    plugin-files = ${pkgs.nix-plugins}/lib/nix/plugins/libnix-extra-builtins.so
  '';
}

Create helpers/extra-builtins.nix and extend the builtins with custom functions. The vault function below accepts a field and a path as arguments to an external wrapper.

{ exec, ... }:

{
  vault = path: field: exec [ ./wrappers/vault field path ];
}

The wrapper returns the secret result from vault as a string. NixOps is now aware of keys available from a vault server. This is done at evaluation time. Environment variables are available to wrappers called from extraBuiltins.

#!/bin/sh -eu
printf '"' && vault kv get -field "$1" "$2" && printf '"'

Load the extraBuiltins with any nixops command by the --option flag with the file path. Vault keys are called using builtins.extraBuiltins.vault.

nixops reboot --deployment nixos --state deployments.nixops --include pigeon --option extra-builtins-file $PWD/nixos/helpers/extra-builtins.nix

The keys in /run/keys are ephemeral and never touch storage. Do reboot the system through any other means except nixops. Your keys will be lost and any dependent service will Use nixops to remotely reboot a machine and its keys will be seeded back into position. Wonderful.

NixOps reboot seeding secrets

Upload deployment keys immediately without waiting on a deploy by using the send-keys argument.

nixops send-keys --deployment nixos --state deployments.nixops --include pigeon

NixOps Pinning

NixOps depends on the nix channel set on the host. This is a very interesting target. Shoot it and lock the host’s channel by prefixing or exporting a defined NIX_PATH to your nixops commands.

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/2b417708c282d84316366f4125b00b29c49df10f.tar.gz

Check the system’s channel version by running nixos-version or nix-instantiate if you prefer to use nix-channel directly.

nix-instantiate --eval --expr '(import <nixpkgs> {}).lib.version'

NixOps and Nix Channel

Syncing the channel on the host and remote with nix-channel works out for my use case. Note that you can use nix flake (manual) to completely replace the channel mechanism. The channel list is at releases.nixos.org and status at status.nixos.org. In my case, a “lock” file called versions.nix is created containing all sources of truth.

{
  # Channel Status:  https://status.nixos.org
  # Channel List:    https://releases.nixos.org/?prefix=nixos

  pkgs = import <nixpkgs> { };

  "20.03" = { channel = "https://releases.nixos.org/nixos/20.03/nixos-20.03.3061.360e2af4f87";   };
  "20.09" = { channel = "https://releases.nixos.org/nixos/20.09/nixos-20.09.2468.c6b23ba64ae";   };
  "21.05" = { channel = "https://releases.nixos.org/nixos/21.05/nixos-21.05.1486.2a96414d7e3";   };
  "21.11" = { channel = "https://releases.nixos.org/nixos/21.11/nixos-21.11.336020.2128d0aa28e"; };
  "22.05" = { channel = "https://releases.nixos.org/nixos/22.05/nixos-22.05.998.d17a56d90ec";    };

  unstable = import (builtins.fetchTarball {
    url = "https://releases.nixos.org/nixos/unstable/nixos-22.11pre386147.e0a42267f73/nixexprs.tar.xz";
    sha256 = "0y6q1j17lmhxh1pqi2jj6xr21pnmachra48336nnbcpnxizswjgg"; }) { };

  linux_5_6_10 = import (builtins.fetchTarball {
    url= "https://github.com/NixOS/nixpkgs/archive/b0e3df2f8437767667bd041bb336e9d62a97ee81.tar.gz";
    sha256 = "0d34k96l0gzsdpv14vnxdfslgk66gb0nsjz7qcqz1ykb0i7n3n07"; }) { };

  linux_5_7_7 = import (builtins.fetchTarball {
    url= "https://github.com/NixOS/nixpkgs/archive/f761c14fd2f198f64cc5483ebf9f83222f9214aa.tar.gz";
    sha256 = "09w0n8qvldanab1m6ik507nl48aszam2a5ii3z2fvk72s66zmry7"; }) { };

  linux_5_10_13 = import (builtins.fetchTarball {
    url= "https://github.com/NixOS/nixpkgs/archive/75d4d5fe851a.tar.gz";
    sha256 = "0ks99va26jsq1mdr1mk9p9r75zvj6ghlmqkdcf4yak0dwr7j48a6"; }) { };
}

A systemd service is created using restartTriggers to lazily watch versions.nix for attribute changes. The system will sync up as

{ config, pkgs, ... }:

let channel = (import ../versions.nix)."${config.system.stateVersion}".channel; in

{
  systemd.services.nix-channel-update = {
    description = "Update Nix Channels";
    wantedBy = [ "multi-user.target" ];
    after = [ "network.target" ];
    restartTriggers = [ channel ];
    path = [ pkgs.shellcheck pkgs.nix ];
    script = ''
      set -euxo pipefail
      shellcheck "$0" || exit 1

      nix-channel --add "${channel}" nixos
      nix-channel --update
    '';
    serviceConfig = { RemainAfterExit = "yes"; };
  };
}

The option config.system.stateVersion is the system’s canonical state version — query the current and default value with the command nixos-option system.stateVersion.

Declarative User Package Management in NixOS

Wed, 27 May 2020 05:26:17 +0000

NixOS

NixOS is a Linux distribution built around the nix package manager. The configuration of the entire system is allowing for immutability and reproducibility.

Let’s take a look at managing a user’s package environment declaratively in NixOS version 20.03. The of NixOS dictates that there are numerous declarative methods. We can use attribute sets, build environments, or even community like home manager.

Declarative Package Management

We will combine the command nix-env with the user’s config.nix to achieve a declarative package setup. The user specific configuration is located at ~/.config/nixpkgs/config.nix. Create this file if it does not exist.

The user’s nixpkgs configuration config.nix places us in a context to nixpkgs.config. Populate config.nix with multiple build environments and install packages using nix-env. This can be achieved more optimally with nix flake or you could use nix-shell for projects and temporary environments. The following is a basic config.nix that makes of package overriding. Check out my config.nix here.

{
  allowUnfree = true;

  packageOverrides = pkgs: with pkgs; {

    Awesome = pkgs.buildEnv {
      name = "awesome";
      paths = [
        awesome
        lxappearance
        paper-gtk-theme
      ];
    };

    Golang = pkgs.buildEnv {
      name = "golang";
      paths = [
        go
      ];
    };

    PHP = pkgs.buildEnv {
      name = "php";
      paths = [
        php74
      ];
    };

    C = pkgs.buildEnv {
      name = "c";
      paths = [
        gnumake
        meson
        ninja
        gcc
      ];
    };
  };
}

Install the above build sets in config.nix using nix-env -Air. The -A flag is for the attribute path, -i to install the packages and -r for removing all packages.

nix-env -Air nixos.Awesome nixos.Golang nixos.PHP nixos.C

Distribution independent installations of nix default to the nixpkgs as the default channel name.

nix-env -Air nixpkgs.Awesome nixpkgs.Golang nixpkgs.PHP nixpkgs.C

Querying the installed packages with nix-env -q shows that there are four packages installed each containing the programs declared in config.nix.

$ nix-env -q
awesome
c
golang
php

One could exploit the recursive aspect and group packages further by any metric. Below we add another set that groups the above declarations by machine.

{
  allowUnfree = true;

  packageOverrides = pkgs: with pkgs; rec {

    Machine1 = pkgs.buildEnv {
      name = "machine1";
      paths = [ Awesome Golang PHP C ];
    };

    # Package declarations for Awesome, Golang, PHP, & C ...
  };
}

and querying this package with nix-env shows that there is one package installed containing a program set of other program sets declared within config.nix.

$ nix-env -Air nixos.Machine1
$ nix-env -q
machine1

This is immensely beneficial for use cases that prioritize keeping root system build times Users can install packages independent of the system’s configuration.nix declaratively. The nix store also allows exposing program paths to the user’s environment.

{
  allowUnfree = true;

  packageOverrides = pkgs: with pkgs; {

    Awesome = pkgs.buildEnv {
      name = "awesome";
      paths = [
        awesome
        lxappearance
        paper-gtk-theme
      ];
      pathsToLink = [ "/etc" "/share" "/bin" ];
    };
  };
}

Instantly all packages with nix-env -Air by switching to an empty environment.

nix-env -Air

List previous environment switches with --list-generations. You can further manipulate the environment lineage by using the generation arguments in the nix-env manual.

$ nix-env --list-generations
368 2021-08-13 23:16:36
369 2021-08-22 00:01:38
370 2021-08-22 00:05:47
371 2021-08-22 00:23:03

Unstable Declarative Package Management

The above is nice and all, but what if a package is not in the nixos stable channel? We can add the channel from the system side, but what happens if its too unstable?

Since this is NixOS — we can mix and match. In config.nix bring in the unstable channel directly. We ungoogled-chromium from the channel as an example.

let
  unstable = import (builtins.fetchTarball "https://releases.nixos.org/nixos/unstable/nixos-21.03pre265961.891f607d530/nixexprs.tar.xz") {};

  # Or lock at a specific commit to stop moving targets.
  # unstable = import (builtins.fetchTarball {
  #  url = "https://releases.nixos.org/nixos/unstable/nixos-21.03pre265961.891f607d530/nixexprs.tar.xz";
  #  sha256 = "1hwwb4n15bbqxnbqffq4kfb369vz65sq74p537fqdp6i4ywpqsyh"; }) {};
in

{
  allowUnfree = true;

  packageOverrides = pkgs: with pkgs; {

    Awesome = pkgs.buildEnv {
      name = "awesome";
      paths = [
        awesome
        lxappearance
        paper-gtk-theme
        unstable.ungoogled-chromium
      ];
      pathsToLink = [ "/etc" "/share" "/bin" ];
    };
  };
}

Advanced Declarative Package Management

Here’s a scenario: You’re reading some Lua code and you need a lua formatting tool. It looks like LuaFormatter will do the job, but that package is neither in the stable or unstable channel. You install luarocks — the lua package manager, but something happens when it tries to install luaformatter. What now?

Create a for luaformatter that config.nix can call and build to expose the package to the user’s environment. Create a packages directory and a folder containing default.nix at ~/.config/nixpkgs/packages/luaformatter.

{ lib, stdenv, fetchFromGitHub, cmake }:

stdenv.mkDerivation rec {
  pname = "LuaFormatter";
  version = "1.3.3";

  src = fetchFromGitHub {
    sha256 = "1dfqsh6v8brnwzg3lgi7228lw08qqfy4ghbjyvwn7mr82fy1xcnd";
    rev = version;
    repo = pname;
    owner = "Koihik";
    fetchSubmodules = true;
  };

  buildInputs = [ cmake ];

  meta = with lib; {
    inherit (src.meta) homepage;
    description = "Code formatter for Lua";
    license = licenses.asl20;
    platforms = platforms.linux;
  };
}

Test the derivation using nix-build. Run nix with the --expr argument inside the folder containing default.nix.

nix-build -E 'with import <nixpkgs> {}; callPackage ./default.nix {}'

Use the --keep-failed (-K) argument to save the temporary build folder on failure. This allows testing fixups in the failed build environment.

nix-build -K -E 'with import <nixpkgs> {}; callPackage ./default.nix {}'

Call the derivation if the build succeeds with pkgs.callPackage from your config.nix

let
  unstable = import (builtins.fetchTarball "https://releases.nixos.org/nixos/unstable/nixos-21.03pre265961.891f607d530/nixexprs.tar.xz") {};
in {
  allowUnfree = true;

  packageOverrides = pkgs: with pkgs; {

    Awesome = pkgs.buildEnv {
      name = "awesome";
      paths = [
        awesome
        lxappearance
        paper-gtk-theme
        unstable.ungoogled-chromium
        (callPackage ./packages/luaformatter/default.nix {})
      ];
      pathsToLink = [ "/etc" "/share" "/bin" ];
    };
  };
}

Think of a derivation as an abstraction of the typical application build process. In the above derivation we imported stdenv — the standard environment that contains common tools and dependencies most programs need. The fetchFromGitHub contains methods that extend the standard environment allowing us to pull in repositories the way. The application luaformatter requires cmake, so it is sourced as one of the buildInputs — another attribute within the derivation context.

A derivation can be extended in multiple ways, allowing us to build any application. The nixpkgs repository has an overview of building artifacts in popular environments like Python. In tricky application builds, step down a level of abstraction to fix it up using the derivation’s phase attributes.

Finally a derivation can be paired with a module allowing us to hide away the above work into a simple of attributes that a user can easily understand. This is what most users care about. The following is a made up example from the simple derivation above.

{
  programs.awesome.enable = true;
  programs.awesome.luaFormat = true;

  # Or

  programs.awesome = {
    enable = true;
    luaFormat = true;
  };
}

Tweaking the Bash Shell

Wed, 29 Apr 2020 05:04:26 +0000

Bourne-Again Shell

The Bourne Again is ubiquitous among many unix-like systems. Here’s a few tweaks that are applied to my bash configuration for interactivity.

Conditional Prompt Colors

The color in the prompt gives a visual cue for the current environment. Green is indicative of a secure shell connection (SSH), red of root privilege, blue of the standard user, and white (an unmodified prompt) of a foreign system.

Prompt color changes based on the current context

This is achieved by short circuiting the $PS1 (the primary variable) in ~/.bash_profile.

PS1_USER='\[\e[0;34m\]\W\[\e[0m\] \[\e[0;34m\]\$\[\e[0m\] '
PS1_ROOT='\[\e[0;31m\]\W\[\e[0m\] \[\e[0;31m\]\$\[\e[0m\] '
PS1_SSHD='\[\e[0;32m\]\W\[\e[0m\] \[\e[0;32m\]\$\[\e[0m\] '

[ "$EUID" != 0 ] && export PS1="$PS1_USER";
[ "$EUID" = 0 ] && export PS1="$PS1_ROOT";
[ -n "$SSH_CLIENT" ] && [ "$EUID" != 0 ] && export PS1="$PS1_SSHD";

Exit Codes

Often there is a need to know the return or exit status of the last command. This is extremely useful in varied situations. The variable $? contains the exit status of the last command.

Prompt shows the exit status of the last command

In ~/.bash_profile, the logic $(E=$? && [ "$E" != 0 ] && echo "$E ") is prefixed to the $PS1. The exit status will be printed on a carriage return.

PS1_USER='$(E=$? && [ "$E" != 0 ] && echo "$E ")\[\e[0;34m\]\W\[\e[0m\] \[\e[0;34m\]\$\[\e[0m\] '
PS1_ROOT='$(E=$? && [ "$E" != 0 ] && echo "$E ")\[\e[0;31m\]\W\[\e[0m\] \[\e[0;31m\]\$\[\e[0m\] '
PS1_SSHD='$(E=$? && [ "$E" != 0 ] && echo "$E ")\[\e[0;32m\]\W\[\e[0m\] \[\e[0;32m\]\$\[\e[0m\] '

[ "$EUID" != 0 ] && export PS1="$PS1_USER";
[ "$EUID" = 0 ] && export PS1="$PS1_ROOT";
[ -n "$SSH_CLIENT" ] && [ "$EUID" != 0 ] && export PS1="$PS1_SSHD";

Fuzzy Finder

The command line fuzzy finder fzf has keybindings for common operations in bash and zsh. This gives the a more opinionated reverse lookup (CTRL+R) workflow.

Command Line Fuzzy Finder

Source the fuzzy finder’s keybindings in the ~/.bashrc configuration. This only works if the command fzf has been installed on the

[ -f '/usr/share/fzf/completion.bash' ] && . /usr/share/fzf/completion.bash
[ -f '/usr/share/fzf/key-bindings.bash' ] && . /usr/share/fzf/key-bindings.bash
[ -f '/usr/share/doc/fzf/examples/completion.bash' ] && . /usr/share/doc/fzf/examples/completion.bash
[ -f '/usr/share/doc/fzf/examples/key-bindings.bash' ] && . /usr/share/doc/fzf/examples/key-bindings.bash

Vi Mode

Bash has a built in readline vi mode. This removes the tempting dependency on the mouse for precise text manipulation. The prefixed + and : indicate insert and normal modes respectively. Pressing v in normal mode invokes vim on the current command.

Vi Mode

Use the set command in the ~/.inputrc to enable showing the vi mode prompt.

set show-mode-in-prompt on
set vi-ins-mode-string "+ "
set vi-cmd-mode-string ": "

Use the set command to enable vi by adding the directive in ~/.bashrc.

set -o vi;

Readline Configuration

The shell’s readline can be configured in the ~/.inputrc file. Here’s a few other readline tweaks for a sane setup.

# Set colors on completion results.
set colored-stats on

# Ignore case when using tab completion.
set completion-ignore-case on

# Show completion results on the first tab press.
set show-all-if-ambiguous on

# Avoid inserting tab completions in the middle of a word.
set skip-completed-text on

# Up and down reverse search will consider the currently typed command.
"\e[A": history-search-backward
"\e[B": history-search-forward

My TS100 Settings and Configuration

Tue, 31 Mar 2020 02:51:00 +0000

TS100 Soldering Iron

The TS100 is a nice complement to the many soldering irons in my toolkit. Integrating technology into traditional tools can be a frustrating experience (gimmicks) but this iron’s not too bad. The TS100 uses firmware to drive its features and core functionality. The original firmware is replaced with Ralim’s TS100 firmware You can view the complete documentation at the repository. Let’s take a look at my preferred settings for this soldering iron.

Soldering settings

Boost Mode (Enabled): Activates temperature boost mode when the front button is pressed.

Boost Temperature (450 °C): The max temperature the iron targets in boost mode. My preference is to set a baseline temperature of 200 °C — 350 °C and boost to 450 °C when needed.

Auto Start (Disabled): This setting automatically heats up the iron when it receives power. My preference is to start the iron manually.

Temperature Change Short (1 °C): This setting determines the temperature change increments on a short button press.

Temperature Change Long (10 °C): This setting determines the temperature change increments on a long button press.

Sleep Modes

Sleep Temperature (10 °C): The temperature the iron maintains while in sleep mode. My preference is the lowest setting. If you put the iron down a lot, set this temperature higher.

Sleep Timeout (10S): The time taken for the iron to enter sleep mode while idling. My preferred setting is 10 seconds. In my case, the soldering iron is rarely put down mid use, so once idle — it should go to sleep fast.

Shutdown timeout (5M): The time taken for the iron to shutdown when idle. My preferred setting is 5 minutes.

Motion Sensitivity (9): This is the sensitivity setting for the motion wake detector. User motion will determine if the iron is idle. The max sensitivity works fine for my use case.

Temperature Interface

Temperature Units (C): The units used to display the temperature. You can choose between Fahrenheit and Celsius.

Display Orientation (R): This setting controls the position of the display. You can choose either left handed, right handed, or automatic. This is set explicitly to right handed in my case.

Cool Down Blink (Disabled): When enabled, the temperature display on the cooling down screen blinks while the tip is hot. This is annoying so off she goes.

Description Scroll Speed (F): The speed at which text scrolls past the screen. My preference is fast scrolling.

Reverse Buttons (F): The button assignments are reversed when enabled.

Advanced Options

Detailed Idle Screen (Enabled): This setting determines if the detailed idle screen should be shown. The detailed idle screen has more information at a glance so my preferred setting is enabled.

Detailed Solder Screen (Disabled): This setting determines if the detailed soldering screen should be shown. This is disabled in my case, since I rarely need anything but the current temperature when soldering.

Power Limiter (Disabled): This setting will enable power output limits. This is useful when operating on battery power.

Power Limit (30W): When the power limiter is enabled this setting will control the maximum power output.

Pulse Power (Disabled): Enables pulse power which will keep battery power banks alive.

Kubernetes in a Linux Container

Sat, 29 Feb 2020 04:25:44 +0000

Kubernetes Home Page

Kubernetes is an open source container orchestration tool promising a catch all solution to issues that arise when deploying applications. Kubernetes creates a framework that models how infrastructure and services should interact.

The industry is trending in a direction where a of people will be running k8s (Kubernetes) setups in one form or the other. Let’s exploit Linux Containers (LXC) as a mock infrastructure and create a local k8s cluster to experiment.

The Base System

We will use CentOS Linux 8.1.1911 as our base system for a cluster running k8s version 1.17. You can download the CentOS version of choice using lxc-create.

lxc-create -n k8s-centos8 -t download

In my case the rootfs for lxc is with docker using a Dockerfile. We docker export to obtain the container’s filesystem. This build bootstraps openssh-server and python for ansible to quickly setup the clusters.

FROM centos:centos8.1.1911

RUN dnf -y update

RUN  dnf install -y --setopt=install_weak_deps=False \
  openssh-server network-scripts python3

RUN  systemctl enable network
RUN  systemctl disable kdump

RUN  usermod -p "$(uuidgen)" root

COPY centos8/files/ifcfg-ens3 /etc/sysconfig/network-scripts/ifcfg-ens3
COPY common/sshd_config /etc/ssh/sshd_config

RUN mkdir -p /root/.ssh
RUN chmod 700 /root/.ssh
COPY common/authorized_keys /root/.ssh/authorized_keys
RUN chmod 400 /root/.ssh/authorized_keys

The Container Configuration

For Kubernetes to work inside the container, we’ll need to tweak its lxc configuration. This is a for the base container k8s-centos8 in /var/lib/lxc/k8s-centos8/config.

# Uncomment the following line to support nesting containers
lxc.include = /usr/share/lxc/config/nesting.conf

# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.arch = x86_64

# Allow access to all cgroups
lxc.cgroup.devices.allow = a

# Set proc mount to read and write
lxc.mount.auto = cgroup:mixed proc:rw sys:mixed

# No capability drops
lxc.cap.drop =

# Bind host kmsg for oom killer
lxc.mount.entry = /dev/kmsg dev/kmsg none defaults,bind,create=file

# Apparmor unconfined and allow nesting
lxc.apparmor.profile = unconfined
lxc.apparmor.allow_nesting = 1

# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:e0:c3:1d
lxc.rootfs.path = dir:/var/lib/lxc/k8s-centos8/rootfs
lxc.uts.name = k8s-centos8

Kubernetes Cluster Installation

The base system is now ready. a master node and worker nodes by cloning the base system.

lxc-copy -n k8s-centos8 -N k8s-master
lxc-copy -n k8s-centos8 -N k8s-node1
lxc-copy -n k8s-centos8 -N k8s-node2
lxc-copy -n k8s-centos8 -N k8s-node3

Manually setting up each system is troublesome. We automate the rest using this ansible

Running Kubernetes Playbook

Running NixOS in a Linux Container

Thu, 16 Jan 2020 00:16:15 +0000

Note: Rerunning the code in this article seems to suggest an ambiguity between the container init and the init path lxc expects across multiple NixOS versions. See this thread for a possible fix.

NixOS is seen as the holy grail of management and system administration. It’s what many call a purely functional distribution [pdf]. The state is declarative — entire systems, deployment artifacts, or services can be built from a single file.

Let’s try out NixOS in a Container (LXC). The NixOS Hydra builds (aarch64) provide container images but they don’t seem to for this use case.

Bootstrapping

The best way to get a NixOS root file system is to bootstrap using nixos-generate. This will require a live NixOS environment or installing nix onto your Linux distribution. Download the latest NixOS minimal image and boot into the live environment using preferred means.

the live environment or after installing nix, bootstrap a Linux container’s rootfs using nixos-generate.

nixos-generate --format lxc

nixos-generate --format lxc --configuration configuration.nix

If the command nixos-generate is not found — install nixos-generators using nix-env or nix-shell.

nix-env -iA nixos.nixos-generators

nix-shell --packages nixos-generators

Once completed, this command will print a hashed path like /nix/store/3ipfpzhk4dllwhcnldsbfldi1favyxsm-tarball/nix-support/hydra-build-products. This file contains the location of the rootfs archive.

$ cat /nix/store/3ipfpzhk4dllwhcnldsbfldi1favyxsm-tarball/nix-support/hydra-build-products
file system-tarball /nix/store/3ipfpzhk4dllwhcnldsbfldi1favyxsm-tarball/tarball/nixos-system-x86_64-linux.tar.xz

Save the generated archive nixos-system-x86_64-linux.tar.xz and exit the live environment.

Setup the Linux Container

Now that we have a clean rootfs archive. Create an empty linux container and rootfs directory. Extract nixos-system-x86_64-linux.tar.xz to rootfs.

lxc-create -n nixos -t none
cd /var/lib/lxc/nixos
mkdir rootfs
tar -xvf nixos-system-x86_64-linux.tar.xz -C rootfs/

Let’s massage our for NixOS at /var/lib/lxc/nixos/config. The entry point is /sbin/init.

# Distribution configuration
lxc.arch = linux64
lxc.include = /usr/share/lxc/config/common.conf

# Container specific configuration
lxc.uts.name = nixos
lxc.rootfs.path = dir:/var/lib/lxc/nixos/rootfs

# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:01:77:76

# NixOS configuration
lxc.init.cmd = /sbin/init

Setup and Run NixOS

Now the container and enter the shell using the command lxc-attach.

lxc-start -n nixos
lxc-attach -n nixos

The default /etc/nixos/configuration.nix is empty. Tell NixOS that we are a container by adding boot.isContainer = true.

{ config, pkgs, ... }:

{
  imports = [ ];
  boot.isContainer = true;
}

Now run a of the system. The upgrade flag may be necessary on the first rebuild to download the channels.

 nixos-rebuild switch --upgrade

We now have a base NixOS system in a Linux container. You can build reproducible desktops, servers, deployment artifacts — anything, using your own configuration.nix and consulting the NixOS options index. This approach is great because when you make something it just works — all the time.

Trying Out This Website

Mon, 02 Dec 2019 06:48:51 +0000

Demo version of this website

Lately, there have been a few inquiries about deploying the source code of my website. Unfortunately for my own very few of my resources are deployed directly or entirely with Docker. You can however get an instance of this website running very quickly with linux

Installing and running lxc on your system should theoretically be the hardest part. To get this up and running, create a container and change to its working directory.

lxc-create --name app --template download -- --dist alpine --release 3.14 --arch amd64 --keyserver hkp://keyserver.ubuntu.com
cd /var/lib/lxc/app

the contents of the container file system. The rootfs will be replaced with the website’s file system.

rm -rf rootfs/*

Download the website here, verify its checksum, and extract the contents into rootfs.

wget https://www.thedroneely.com/download/rootfs.tar.gz
tar --numeric-owner -xzvf rootfs.tar.gz -C rootfs/

Start the website and get its IP address.

lxc-start -n app
lxc-ls -f

The output should look something like the following.

NAME     STATE   AUTOSTART GROUPS IPV4       IPV6 UNPRIVILEGED
app      RUNNING 0         -      x.x.x.x    -    false

Cockpit CMS can be accessed at http://x.x.x.x/cockpit. Isso’s administration page can be accessed at http://x.x.x.x/isso/admin. The default are listed below.

# Cockpit CMS default credentials
Username: admin
Password: admin

# Isso default credentials
Password: changethis

If this was too long to read, then see the condensed version below.

lxc-create -n app -t download -- --dist alpine --release 3.10 --arch amd64
cd /var/lib/lxc/app
rm -rf rootfs/*
wget https://www.thedroneely.com/download/rootfs.tar.gz
tar --numeric-owner -xzvf rootfs.tar.gz -C rootfs/
lxc-start -n app
lxc-ls -f

Installing Isso From Source

Sat, 30 Nov 2019 20:46:14 +0000

Isso is a lightweight open source commenting server. Sometimes changes to an open source program can take a while to make it into a packaged version. Let’s install isso from source and get the latest fixes and features. The documentation can be tricky. Here’s my approach at installing. I’m using Linux, but the distribution doesn’t matter.

In essence the installation process can be to the following crucial commands. In later versions, you won’t need to contort so much when building isso since the necessary JavaScript files are built during the python build.

mkdir /opt/isso
cd /opt/isso
git clone https://github.com/posativ/isso .
python3 -m venv .
source bin/activate
python setup.py develop
npm install bower requirejs jade
PATH=$PATH:/opt/isso/node_modules/.bin
make init
make js
touch isso.conf
isso --version

Then create an example isso.conf, run isso, and some endpoints to make sure it is working.

# Isso configuration file
# vim: set filetype=ini

[general]
dbpath = /opt/isso/isso-comments.db
name = example.com
host = https://example.com/
max-age = 15m
notify = stdout

[admin]
enabled = true
password = changethis

[moderation]
enabled = false
purge-after = 30d

[server]
reload = off
profile = off
listen = http://127.0.0.1:8100
#public-endpoint = https://example.com/isso

[smtp]
username =
password =
host = localhost
port = 587
security = starttls
to =
from =
timeout = 10

[guard]
enabled = true
ratelimit = 2
direct-reply = 3
reply-to-self = false
require-email = false
require-author = false

[markup]
options = strikethrough, autolink, fenced_code, no_intra_emphasis
allowed-elements =
allowed-attributes =

[hash]
salt = GkSOkjoIPI2GvAlnxwUsMiwn
algorithm = pbkdf2

If you’re trying this on Alpine Linux, then the following are needed.

apk add git python3 gcc python3-dev musl-dev libffi-dev npm make

Improving Paperless' Interface

Sun, 27 Oct 2019 07:12:59 +0000

Improved Paperless Interface

Paperless, now known as paperless-ng, and paperless-ngx is an open source web application for searching documents that have been indexed using OCR (Optical Character Recognition). The backend can be gutted in any way you like to run customized operations pre and post document Paperless provides a robust, quick, and hands solution for those who may want to go “paperless”. The interface is lacking though — it’s the Django admin dashboard after all. On initial deployment it looks bare.

Default Paperless Interface

The project has provided a way to override CSS and JavaScript at run time. To do this, create the files overrides.css and overrides.js in the media folder from the project’s root directory. I’m going for a clean look. Here’s a stylesheet override implementing some improvements to the interface.

body { background-color: #fcfcfc; max-width: 1800px; margin: 0 auto; }
label[for="searchbar"] { display: none !important; }
.result .header { background-color: #fff !important }
.result .header a, .result a.tag { color: #000 !important; }
.result .header .info { display: none !important; }
.result .date { color: #555 !important; }
thead th a:link, thead th a:visited { color: #000 !important; }
thead th, tfoot td { color: #000 !important; }
a.section:link, a.section:visited { color: #000 !important; }
#changelist-filter { background: transparent }
.results, #changelist-filter, #login-form, fieldset { text-align: left }
.selector .help-icon { background: url(/static/admin/img/icon-unknown.svg) 0 0 no-repeat !important; }
.xfull { margin-top: -1rem; }
.object-tools { display: inline-block; }
.module, #changelist .actions { background-color: transparent; }

#changelist #toolbar { 
  border-top: none !important; 
  border-bottom: none !important; 
  background-color: transparent !important;
}

.djangoql-toggle { display: none !important; }
.paginator { border-top: none !important; }
#changelist .paginator { border-bottom: none !important; }
#changelist .toplinks { border-bottom: none !important; }
.grid { margin-right: 0 !important; }
.quiet { margin: 0 1rem !important; color: #555; }

@media (max-width: 560px) { table#result_list { display: block; overflow:scroll } }

.paginator .this-page {font-size: 24px !important; color: #17541f; }
.paginator a:link, .paginator a:visited { background-color: transparent !important; color: #000 !important; }
.paginator a:focus, .paginator a:hover { background-color: #ccc !important; border-radius: 6px; }
#header { color: #000 !important; background-color: transparent !important; }
#header a:link, #header a:visited { color: #000 !important; }
.breadcrumbs { display: none !important; }
#content h1 { display: none !important; }
#content-related { float: none !important; width: 100% !important; position: inherit !important; margin-right: 0 !important; text-align: left; background-color: transparent !important; }
.colMS { margin: 0 auto !important; }
#branding, #user-tools, #content { text-align: center; }
.colMS div:first-child h3:first-child { color: #17541f; font-size: 2rem !important; }
.colMS div:first-child { margin-bottom: 4rem !important; }
.actions { margin-bottom: 2rem; justify-content: center; }
textarea { vertical-align: middle !important }
#changelist-search > div, #changelist-search input[type="submit"] { font-weight:bold; margin: 0 auto; }
#changelist #toolbar form input[type="submit"] { margin: 1rem auto; padding: 8px 24px; }

.button, input[type=submit], input[type=button], .submit-row input, a.button {
  color: #fff !important;
}

.button, .button:active, .button:focus, .button:hover, a.button, .submit-row input, input[type="submit"], input[type="submit"]:active, input[type="submit"]:focus, input[type="submit"]:hover, input[type="button"], input[type="button"]:active, input[type="button"]:focus, input[type="button"]:hover {
    background-color: #17541f !important;
}

.button, #changelist #toolbar form input[type="submit"] { border: none !important; box-shadow: none !important; }

#changelist-search input#searchbar { width: 100%; }

#searchbar {
  margin: 2rem 0 !important;
  height: 1rem !important; 
  padding: 0.75rem !important;
  box-shadow: 0 4px 6px -1px rgba(0, 0, 0, 0.1), 0 2px 4px -1px rgba(0, 0, 0, 0.06);
}

@media (min-width: 768px) { #searchbar { width: 100% !important; } }

.module h2, .module caption, .inline-group h2 {
    background-color: #eee;
    color: #000 !important;
}

.object-tools a:link, .object-tools a:visited { border-radius: 3px; background-color: #17541f; }

We should also replace the default logo with its darker variant, since the background colors are now significantly lighter.

diff --git a/src/documents/templates/admin/base_site.html b/src/documents/templates/admin/base_site.html
index 0f76ec58..d84574ca 100644
--- a/src/documents/templates/admin/base_site.html
+++ b/src/documents/templates/admin/base_site.html
@@ -55,7 +55,7 @@

 {% block branding %}
 <h1 id="site-name">
-	<a href="{% url 'admin:index' %}"><img src="{% static 'paperless/img/logo-light.png' %}" alt="Paperless" /></a>
+	<a href="{% url 'admin:index' %}"><img src="{% static 'paperless/img/logo-dark.png' %}" alt="Paperless" /></a>
 </h1>
 {% endblock %}

Every page of the application has been improved. Notably, the table row headers have been adjusted for responsive The overall result is decent enough.

Improved Paperless Search Interface

More functional improvements can be implemented using but I’ll leave it there. The changes listed in this article can be viewed at this repository.

Generating Archive Pages with Hugo

Mon, 30 Sep 2019 01:00:34 +0000

Archive Page

In the absence of a search engine — an page is a simple way to find older articles quickly. Here is a quick and dirty way to create multiple archive pages with Hugo.

There are a few requirements. The archive pages should be a bit modular. We should be able to just drop an archive.md file into any folder within the content directory and generate a single archive page for that topic.

hugo
|__ content
    |__ posts
        |__ archive.md
        projects
        |__ archive.md

We should be able to generate clean slugs. Hugo makes that The slugs for each archive page topic would be of the form /posts/archive and /projects/archive.

We could also take a more pragmatic approach by using the following file directory structure which results in slugs like /archives/posts or /archives/projects.

hugo
|__ content
    |__ archives
        |__ posts.md
        |__ projects.md

The content of each archive page will be determined by the parameters within the front matter, particularly the layout parameter. The front content in posts/archive.md will contain the needed parameters to generate that page.

---
title: "Posts Archive"
layout: archive
hidden: true
type: posts
summary: This page contains an archive of all posts.
---

The layout parameter will reference the template layouts/_default/archive.html to render a single archive page. That template file is shown below.

{{ define "main" }}
<section>
  {{ $type := .Type }}
  {{ $.Scratch.Set "count" 1 }}

  {{ range (.Site.RegularPages.GroupByDate "2006") }}
    {{ if and (gt .Key 1) (gt (where .Pages "Type" $type) 0) }}

      {{ range (where .Pages "Type" $type) }}
        {{ if (eq ($.Scratch.Get "count") 1) }}
          {{ $.Scratch.Set "count" 0 }}
          <h1>{{ .Date.Format "2006" }}</h1>
        {{ end }}
      {{ end }}

      {{ $.Scratch.Set "count" 1 }}

      <ul>
      {{ range (where .Pages "Type" $type) }}
        {{ if (ne .Params.hidden true) }}
          <li>
            <a href="{{ .RelPermalink }}">
              <span>{{ .Date.Format "02 Jan" }}</span> — {{ .Title }}
            </a>
          </li>
        {{ end }}
      {{ end }}
      </ul>
    {{ end }}
  {{ end }}
</section>
{{ end }}

This template groups all posts for a particular topic by year and orders them by the month, taking into account the type parameter from the front matter of archive.md. Note the use of the .Scratch There is most likely a much cleaner but this is good enough.

There are quirks when implementing an archive page as a single post. We may want to adjust the RSS and pagination templates to hide additional data produced by an archive page with the hidden: true attribute.

Webrings with Openring

Sat, 17 Aug 2019 05:11:17 +0000

Webring Generated by Openring

Lately my fascination has moved towards the IndieWeb. Maintaining your online identity and data independently of big corporations is My website has a simple webring at the bottom of every

“Webrings allow ringbearers to create a network of blogs around a shared goal where readers can choose the next, previous, and random blog in the ring.” IndieWeb: What is a Webring?

Openring is a neat that allows you to generate a webring. Pass a series of RSS links as arguments to openring in addition to a template and it generates an html file that can be included anywhere on a web page. This is particularly nice for static site generators.

openring \
  -s https://example.com/feed1.xml \
  -s https://example.com/feed2.xml \
  -s https://example.com/feed3.xml \
< template.html \
> output.html

It’s a matter of making my own template and including it within my custom

<section class="webring">

  <h3>On the Web</h3>
  <section class="articles">
    <div class="columns is-tablet is-centered">
      {{ range .Articles }}
      <div class="column is-4">
        <div class="tile is-ancestor">
          <div class="tile is-parent">
            <article class="tile is-child box is-shadowless">
              <h2><a href="{{ .Link }}" target="_blank" rel="noopener" class="serif is-size-4">{{ .Title }}</a></h2>
              <p class="summary">{{ .Summary }}</p>
              <small class="source is-block">by <a href="{{ .SourceLink }}">{{ .SourceTitle }}</a></small>
              <small class="date has-text-grey-dark">{{ .Date | date }}</small>
            </article>
          </div>
        </div>
      </div>
      {{end}}
    </div>
  </section>

  <p class="attribution has-text-right">Generated by <a href="https://git.sr.ht/~sircmpwn/openring">Open Ring</a></p>

</section>

This website uses Hugo to generate static content. It’s as simple as including the generated .html in the Go HTML template. Done.

{{- partial "openring.html" . -}}

Drone is a decent tool. Add a configuration step to fetch openring and generate the webring on every build.

---
- name: openring
  image: debian:stretch-slim
  commands:

  - chmod +x openring

  - >
    ./openring
      -s https://drewdevault.com/feed.xml
      -s https://mxb.dev/feed.xml
      -s https://www.taniarascia.com/rss.xml
    < generators/openring/template.html
    > generators/hugo/themes/tdro/layouts/partials/openring.html

Openring is integrated into the website’s deployment pipeline. The build is roughly 40 seconds. Not too

Drone Pipeline

This was easy to integrate and as more knowledge is gained about the IndieWeb, new features will be added to the site.

Gitea in a Subdirectory with Nginx

Wed, 31 Jul 2019 22:37:06 +0000

Gitea’s Home Page

Running web applications from the subdirectory of a URL has its benefits. There doesn’t seem to be much talk about this It’s useful when you need to expose a web application quickly without provisioning a new subdomain and SSL certificate.

For example let’s run Gitea from the subdirectory of a domain using nginx We will make Gitea accessible from a URL of the form https://www.example.com/gitea or https://www.example.com/git.

To accomplish this, reverse proxy using a separate location block in the nginx

location /gitea/ {
    proxy_pass http://unix:/opt/gitea/var/lib/gitea/gitea.socket:/;
}

Then in Gitea’s app.ini configuration, adjust the server block to match the location block specified

[server]
PROTOCOL         = unix
SSH_DOMAIN       = example.com
DOMAIN           = http://www.example.com/gitea/
ROOT_URL         = http://www.example.com/gitea/
HTTP_ADDR        = /opt/gitea/var/lib/gitea/gitea.socket

In future versions of Gitea, the unix protocol value is deprecated in favour of http+unix.

[server]
PROTOCOL         = http+unix
SSH_DOMAIN       = example.com
DOMAIN           = http://www.example.com/gitea/
ROOT_URL         = http://www.example.com/gitea/
HTTP_ADDR        = /opt/gitea/var/lib/gitea/gitea.socket

A user can access the full Gitea instance from the gitea/ subdirectory of the URL.

All applications are not created equal and some blow up badly in a subdirectory. Gitea supports this natively from the configuration.

Some applications require elaborate tricks, workarounds, or source code changes that are probably not worth the effort.

Tweaking GoAccess for Analytics

Tue, 11 Jun 2019 04:01:04 +0000

GoAccess Dashboard for this server

GoAccess is an open source server–side web log analyzer. Server side means that it will process web server logs and compile the results into a real–time graphical view. It works with many web log

By default goaccess will provide a picture of the traffic to your server. This includes crawlers, bots, and various HTTP requests. We can make some changes to goaccess’ configuration to make it behave a bit more like a web analytic

“I believe tracking visitors at the client level deflates the actual number of visitors. On the other hand, server–side tracking gives you a more accurate number at the cost of not knowing for sure if the client is a human behind a browser.” GoAccess Author Explains Tracking using Client vs. Server

GoAccess allows command line flags and shell piping, but we’ll do most of the work from a central goaccess.conf Let’s start off by enabling real–time HTML for live updates through the socket connection.

# Enable real-time HTML output.
real-time-html true

# Set output HTML path.
output /srv/http/goaccess/index.html

The backend web server is nginx so enable the combined log format and set the access log path.

# Set log format.
log-format COMBINED

# Specify the path to the input log file.
log-file /var/log/nginx/access.log

Exclude localhost so that goaccess ignores counting internal requests as unique visitors. We can exclude multiple public IPv4 and IPv6 addresses here as well.

# Exclude an IPv4 or IPv6 address from being counted.
exclude-ip 127.0.0.1
exclude-ip xx.xx.xx.xx

Ignore counting crawlers. This should make the unique visitors count more accurate.

# Ignore crawlers from being counted.
ignore-crawlers true

You can further refine the output by adding more crawlers to ignore. This can be done by setting a browsers-file

# Include an additional delimited list of browsers/crawlers/feeds etc.
browsers-file /opt/goaccess/config/browsers.list

Let’s enable IP address anonymization. In future versions of goaccess you’ll be able set the level of IP address anonymization with the command line flag --anonymize-level and the configuration option anonymize-level.

# IP address anonymization
anonymize-ip true

# Pedantic IP address anonymization
anonymize-level 3

By default goaccess does not add client errors to the unique visitors count.

# Do not add 4xx client errors to the unique visitors count.
4xx-to-unique-count false

We can also remove specific HTTP response codes from the visitor’s count too.

# Ignore parsing and displaying one or multiple status code(s)
ignore-status 429

Referrer spam inflates and skews the log data. visitors from a list of by using ignore-referer. My personal preference is to use Matomo’s list.

# Ignore referrer from being counted.
ignore-referer www.example.com

Sort the most important panels by visitor count, data, and bandwidth in descending order.

# Sort panels on initial load by visitors, data, and bandwidth.
sort-panel BROWSERS,BY_VISITORS,DESC
sort-panel CACHE_STATUS,BY_VISITORS,DESC
sort-panel GEO_LOCATION,BY_VISITORS,DESC
sort-panel HOSTS,BY_VISITORS,DESC
sort-panel KEYPHRASES,BY_VISITORS,DESC
sort-panel MIME_TYPE,BY_VISITORS,DESC
sort-panel NOT_FOUND,BY_BW,DESC
sort-panel OS,BY_VISITORS,DESC
sort-panel REFERRERS,BY_VISITORS,DESC
sort-panel REFERRING_SITES,BY_VISITORS,DESC
sort-panel REMOTE_USER,BY_VISITORS,DESC
sort-panel REQUESTS,BY_VISITORS,DESC
sort-panel REQUESTS_STATIC,BY_BW,DESC
sort-panel STATUS_CODES,BY_VISITORS,DESC
sort-panel TLS_TYPE,BY_VISITORS,DESC
sort-panel VIRTUAL_HOSTS,BY_VISITORS,DESC
sort-panel VISITORS,BY_DATA,DESC
sort-panel VISIT_TIMES,BY_DATA,DESC

Change the theme and table specifications on the page by using a string of json

# Set default HTML preferences.
html-prefs {"theme":"darkBlue","perPage":20,"visitors":{"plot":{"chartType":"bar"}},"visit_time":{"plot":{"chartType":"bar"}}}

Make sure that all static files — including files with a query string are categorized under the static files table.

# Include static files that contain a query string in the static files
all-static-files true

Show statistics based on country by loading in a GeoIP database. You can install a database from your Linux distribution of choice.

# Set GeoIP database path.
geoip-database /usr/share/GeoIP/GeoLiteCity.dat

Everything runs in memory. Tokyo Cabinet and store the results as a database on the file system.

### GoAccess version <= 1.3

# Persist parsed data into disk.
keep-db-files true
# Load previously stored data from disk.
load-from-disk true
# Path where the on-disk database files are stored.
db-path /tmp/

Newer versions use a different syntax and will not require setting up specific configure options.

### GoAccess version >= 1.4

# Persist parsed data into disk.
persist true
# Load previously stored data from disk.
restore true
# Path where the on-disk database files are stored.
db-path /tmp

Now stream the logs into goaccess using our souped–up config. GoAccess will process the rotated logs of nginx in addition to the current access log stipulated in goaccess.conf.

zcat --force /var/log/nginx/access.log-* | goaccess --config-file=/opt/goaccess/config/goaccess.conf -

Tailwind CSS and Beyond

Sat, 11 May 2019 07:10:35 +0000

Tailwind CSS Design

Recently, this website of mine was redesigned. The is a stylistic book and implemented using the Tailwind CSS framework.

CSS is a tight and compact CSS framework that gives one a myriad of custom design choices.

“Tailwind is different from frameworks like Bootstrap, Foundation, or Bulma in that it’s not a UI kit.” “It doesn’t have a default theme, and there are no built-in UI components.” “On the flip side, it also has no opinion about how your site should look and doesn’t impose design decisions that you have to fight to undo.” Tailwind CSS, Web Page, What is Tailwind?

Tailwind is a utility first Your HTML will contain many classes that build up into a complete component.

<div class="bg-white mx-auto max-w-sm shadow-lg rounded-lg overflow-hidden">

The above looks like bad practice but you’ll get used to its utility-like nature very quickly and may end up writing to no CSS at all. In repetitive situations we can implement abstractions by extracting components.

Automate Vultr Snapshots Using Bash

Thu, 02 May 2019 22:19:54 +0000

Note: This article is out of date. Vultr’s API v1 end of life was announced June 30, 2023

Vultr is a VPS (Virtual Private Server) hosting platform that uses KVM (Kernel–based Virtual Machine) virtualization. Signing into their website over and over again to make a snapshot can be tedious. Let’s automate this process using bash, and systemd timers.

Vultr offers an excellent API (Application Programming Interface) that lets you programmatically orchestrate a web based hypervisor dashboard, right down to account payments. We will use this API to automatically create snapshots.

Update: Vultr’s used to be free, but the pricing changed to $0.05/GB per month. Keeping this in mind the script below sets an limit of 11 snapshots.

Vultr Snapshot Disclaimer

First let’s set up the variables that our script will use to create and rotate snapshots.

#!/bin/sh -eu

Key='YourAPIKEY'
SnapshotLimit=11
SnapshotCount=$(curl -s -H "API-Key: $Key" https://api.vultr.com/v1/snapshot/list | grep -o SNAPSHOTID | wc -l)
Fields=$(curl -s -H "API-Key: $Key" https://api.vultr.com/v1/snapshot/list | grep -o '"' | wc -l)
LastSnapshotField=$((Fields - 24))
LastSnapshotID=$(curl -s -H "API-Key: $Key" https://api.vultr.com/v1/snapshot/list | grep '"' | cut -d '"' -f $LastSnapshotField)

We are using command substitutions with curl. Let’s add our API key and set the snapshot limit to 11. The API will respond in json. We can install jq but I’ll use the tools that are available to process the output. First, we’ll get the current number of snapshots using grep and wc.

SnapshotCount=$(curl -s -H "API-Key: $Key" https://api.vultr.com/v1/snapshot/list | grep -o SNAPSHOTID | wc -l)

Then use the same technique to get the number of fields in the json output delimited by ".

Fields=$(curl -s -H "API-Key: $Key" https://api.vultr.com/v1/snapshot/list | grep -o '"' | wc -l)

Since we have an upper snapshot limit of 11, we also need to know the ID of the oldest snapshot. This will allow us to destroy that snapshot when we reach the limit automatically. I already know that the oldest snapshot ID is always 28 fields away from the end of the delimited json output.

LastSnapshotField=$((Fields - 28)

Finish up by using grep and cut to extract the oldest snapshot ID.

LastSnapshotID=$(curl -s -H "API-Key: $Key" https://api.vultr.com/v1/snapshot/list | grep '"' | cut -d '"' -f $LastSnapshotField)

Execute a simple if statement to destroy that snapshot when necessary. Nothing fancy.

if [ "$SnapshotCount" -eq "$SnapshotLimit" ]
  then
    curl -H "API-Key: $Key" https://api.vultr.com/v1/snapshot/destroy --data "SNAPSHOTID=$LastSnapshotID" || exit 1;
fi

curl -H "API-Key: $Key" https://api.vultr.com/v1/snapshot/create --data 'SUBID=YourSUBID';

Our full script now looks like this and is complete.

#!/bin/sh -eu

Key='YourAPIKEY'
SnapshotLimit=11
SnapshotCount=$(curl -s -H "API-Key: $Key" https://api.vultr.com/v1/snapshot/list | grep -o SNAPSHOTID | wc -l)
Fields=$(curl -s -H "API-Key: $Key" https://api.vultr.com/v1/snapshot/list | grep -o '"' | wc -l)
LastSnapshotField=$((Fields - 24))
LastSnapshotID=$(curl -s -H "API-Key: $Key" https://api.vultr.com/v1/snapshot/list | grep '"' | cut -d '"' -f $LastSnapshotField)

if [ "$SnapshotCount" -eq "$SnapshotLimit" ]
  then
    curl -H "API-Key: $Key" https://api.vultr.com/v1/snapshot/destroy --data "SNAPSHOTID=$LastSnapshotID" || exit 1;
fi

curl -H "API-Key: $Key" https://api.vultr.com/v1/snapshot/create --data 'SUBID=YourSUBID';

Let’s drop this into a systemd service at /etc/systemd/system/vultr-snapshot.service.

[Unit]
Description=Create Vultr Snapshot

[Service]
Type=oneshot
ExecStart=/usr/local/bin/vultr-snapshot

Then run it using a timer at /etc/systemd/system/vultr-snapshot.timer. This timer runs the script at 5:30 AM every morning.

[Unit]
Description=Create Vultr Snapshot

[Timer]
OnCalendar=*-*-* 05:30:00
RandomizedDelaySec=60

[Install]
WantedBy=timers.target

That’s it. This is a quick way of supplementing your normal backups with automatic Vultr snapshots.

Variable Dumping Post Requests in PHP

Tue, 30 Apr 2019 23:16:13 +0000

PHP manual on var_dump

This is a brilliant way to dump variables to a webserver’s error log in This is especially helpful when experimenting with POST requests in barebone environments.

<?php
ob_start();
var_dump($this);
error_log(ob_get_clean());

Finding That 1% On The Internet

Sun, 21 Apr 2019 20:07:55 +0000

Image by akbaranifsolo from Pixabay

The internet is an enormous mess of information that can be difficult to search, validate, and utilize. The common refrain today is fake news. These problems have always

Nothing much has changed from then to now, except more algorithms to fight against, and information to sift through. Here are some principles learned the hard way to extract useful information (that 1%).

Prefer Utility Before Popularity

Companies like Google, YouTube, Reddit, and Facebook optimize for popularity over utility. In fact, almost everyone eventually optimizes for popularity. Popularity is a general metric for determining some utility but it is not always optimal. Utility is affected by multiple factors.

The more popular something is, the greater the chance we overestimate its utility. This is a simple trade off. External forces (i.e market pressures) may prevent individuals from making rational decisions or disseminating useful information.

YouTube is a good example of this; smaller channels often provide more valuable knowledge than larger channels. Bigger channels may have external pressures that can affect their ability to provide useful information.

You will gain much headway on the Internet if you have the insight to distinguish between what is just popular and what is actually useful. Take for instance this Amazon’s Choice being casually picked.

You Don’t Know What You Don’t Know

If you do not know what you’re looking for, you almost certainly will not find it. It is important to first understand that perhaps we know nothing.

This is a humble approach that makes the initial search easier. Many subjects involve varied levels of scope and knowledge that depend on experience.

Always Sort by Controversial First

In any knowledgeable discussion, always focus on those who are controversial first. There is something great about individuals who are critical. These are the types who often care the most about a subject.

They can also bring out more knowledgeable individuals who make great effort to rebut them. These two groups can take you from what you don’t know to what you should know.

Pretend That You Know What You Don’t Know

We do this all the time. If you go to a library and do not know what book you are looking for, you will instinctively give the librarian a set of ideas to consider.

On the Internet, this was much easier to do in the past. Today it’s a bit more tricky, but the concept still holds true. Try to envision as closely as possible what you expect to find even if you do not know what it looks like. This is a skill that is hard to put into words.

The Best Information is Usually at the Bottom

Ever been to Stack Overflow and notice that in a lot of threads the answers seem to get progressively better as you scroll down?

If it’s first, it must work — that is the motto of most question and answer forums. In the search for that elusive 1%, it’s always practical to consider new information.

Valuable Knowledge is Lost Every Day

The Internet never forgets is somewhat of a tired phrase, because the Internet does forget.

I’ve learned more useful information from archive.org than most of the stuff being pushed out today. Sometimes finding that 1% means you must travel to the past.

Adding Headroom with JavaScript

Mon, 08 Apr 2019 16:45:37 +0000

Image by janjf93 from Pixabay

Fixed navigation bars have an annoying drawback — they take up precious screen real estate. We can add headroom to any web page by automatically retracting and expanding the navigation bar on

We will create a super simple implementation with vanilla JavaScript and discuss some ways to improve its behavior. First let’s add some hacker style boilerplate to our app.js file.

/**
 * Boilerplate Functions
 */
function posf (f, a) { for (var i=0; i < a.length; i++) { if (f(a[i])) return i; } return -1; }
function apos (x, a) { return (typeof x == 'function') ? posf(x,a) : Array.prototype.indexOf.call(a,x) }
function arem (a, x) { var i = apos(x, a); if (i >= 0) { a.splice(i, 1); } return a; }
function afind (x, a) { var i = apos(x, a); return (i >= 0) ? a[i] : null; }
function addClass (el, cl) { if (el) { var a = el.className.split(' '); if (!afind(cl, a)) { a.unshift(cl); el.className = a.join(' ')}} }
function remClass (el, cl) { if (el) { var a = el.className.split(' '); arem(a, cl); el.className = a.join(' ') } }

Then setup our variables. The variable previousPosition stores the initial page position. Bind to the navigation bar with the navbar selector and grab its height using the offsetHeight.

/**
 * Setup Variables
 */
var previousPosition = window.pageYOffset;
var navbar = document.getElementById("navbar");
var navbarHeight = navbar.offsetHeight;

Add an onscroll event with a simple check that hides the navigation bar on scroll down and reveals it on scroll up.

/**
 * Headroom Action
 */
window.onscroll = function() {
  var currentPosition = window.pageYOffset;

  if (previousPosition > currentPosition) {
    remClass(navbar, 'headroom');
  } else if (currentPosition > navbarHeight) {
    addClass(navbar, 'headroom');
  }

  previousPosition = currentPosition;
};

We add headroom to the page only when the current position is greater than the navigation bar’s height.

else if (currentPosition > navbarHeight) {
  addClass(navbar, 'headroom');
}

For our CSS style sheet, force the navigation bar upwards from its initial fixed position and give it a transition time to make the movement less jarring.

.headroom { top: -5em !important; }
.navbar {
  -webkit-transition: top 0.75s;
  -o-transition: top 0.75s;
  transition: top 0.75s;
}

Demonstration and Source

Sorry, your browser doesn't support embedded videos. Here is a link to the video instead.

Basic auto hiding navigation bar.

Improvements

How can we improve this simple script? Let’s introduce velocity. We will show the navigation bar only if the user scrolls up faster than normal.

The velocity will be the difference between the previous and current position. The greater the difference, the faster the user scrolls. A scroll downwards produces a negative value and a scroll upwards produces a positive value.

var velocity = previousPosition - currentPosition;

Now that we know the velocity, we can add resistance by setting a velocity threshold of 55. The user must now overcome this velocity to reveal the navigation bar. The effect of this resistance threshold will vary based on device and browser.

The else if condition is capped to velocity < 0. This handles a situation where the user performs a flick scroll gesture which involves both acceleration and deceleration. In this scenario accidental triggers can happen if we do not consider the direction of the scroll.

if (velocity > 55 || currentPosition < navbarHeight) {
  remClass(navbar, 'headroom');
} else if (velocity < 0) {
  addClass(navbar, 'headroom');
}

Due to this implementation, add an initial check at run time to handle a refresh anywhere in the middle of the

var previousPosition = window.pageYOffset;
var navbar = document.getElementById("navbar");
var navbarHeight = navbar.offsetHeight;

if (previousPosition > navbarHeight) {
  addClass(navbar, 'headroom');
}

window.onscroll = function() {
  var currentPosition = window.pageYOffset;
  var velocity = previousPosition - currentPosition;

  if (velocity > 55 || currentPosition < navbarHeight) {
    remClass(navbar, 'headroom');
  } else if (velocity < 0) {
    addClass(navbar, 'headroom');
  }

  previousPosition = currentPosition;
};

Demonstration and Source

Sorry, your browser doesn't support embedded videos. Here is a link to the video instead.

Improved auto hiding navigation bar.

Conclusion

That’s it. We can make even improvements but this looks good enough.

Let's Customize Gitea

Fri, 29 Mar 2019 21:37:32 +0000

Gitea is an awesome and Git service that can be hosted anywhere. It is open source and written in Golang. Many of my own projects are hosted across multiple Gitea instances. A standard Gitea installation will look a bit bare.

We can many aspects of Gitea by overriding its default templates. This allows us to change the header, footer, and navigation sections with ease. You can also override public resources like images.

To start overriding the templates, create the folder custom in Gitea’s home folder or working directory.

drwxr-xr-x 4 gitea gitea 4096 Mar 29 17:03 custom
drwxr-xr-x 6 gitea gitea 4096 Jan 17 03:40 data
-rw-r--r-- 1 gitea gitea 73   Mar 29 17:29 .gitconfig
srw-rw-rw- 1 gitea gitea 0    Mar 29 17:29 gitea.socket
drwxr-xr-x 3 gitea gitea 4096 Mar 29 15:52 log
drwxr----- 3 gitea gitea 4096 Jan 17 06:44 .pki
drwxr-xr-x 3 gitea gitea 4096 Jan 17 06:44 repos
drwx------ 2 gitea gitea 4096 Jan 17 05:49 .ssh

We will change some elements of the navigation bar and override the favicons. To do this, the custom folder will have the following file structure.

custom
|__ public
|   |__ img
|       |__ favicon.ico
|       |__ favicon.png
|__ templates
    |__ base
        |__ head_navbar.tmpl

The navigation elements are templated by the source file base/head_navbar.tmpl and can be overridden by the path custom/templates/base/head_navbar.tmpl.

Match the template files with your Gitea version using the repository or you may experience runtime errors.

You can view the override structure of the templates by looking directly at the source files. You must restart the Gitea instance for changes to take effect.

The Customization

Let’s copy base/head_navbar.tmpl to our local path at custom/templates/base/head_navbar.tmpl and replace the brand icon and link.

@@ -1,7 +1,8 @@
<div class="ui container" id="navbar">
  <div class="item brand" style="justify-content: space-between;">
-   <a href="{{AppSubUrl}}/">
-     <img class="ui mini image" src="{{AppSubUrl}}/img/gitea-sm.png">
+   <a title="Head back to https://www.thedroneely.com" href="/">
+     <img style="display: inline; border-radius: 6px;" class="ui mini image" src="/android-chrome-192x192.png">
+     <span style="margin-left: 0.65em;">www.thedroneely.com</span>
    </a>
  <div class="ui basic icon button mobile-only" id="navbar-expand-toggle">
    <i class="sidebar icon"></i>

Finally let’s disable the help button in the navigation bar by adjusting the template.

@@ -118,7 +119,7 @@
{{else}}
- <a class="item" target="_blank" rel="noopener noreferrer" href="https://docs.gitea.io">{{.i18n.Tr "help"}}</a>
+ <!-- <a class="item" target="_blank" rel="noopener noreferrer" href="https://docs.gitea.io">{{.i18n.Tr "help"}}</a> -->
  <div class="right stackable menu">
    {{if .ShowRegistrationButton}}
      <a class="item{{if .PageIsSignUp}} active{{end}}" href="{{AppSubUrl}}/user/sign_up">

The can be overridden by placing the files favicon.ico and favicon.png inside custom/public/img.

Caching and Changes

Gitea uses a service worker that caches files This means that some changes may not show up immediately. Clear the browser cache (history) or “hard refresh” the to view the changes.

The Results

This guide did not contain drastic changes to Gitea, however the sky is the limit. You can the results of my changes by visiting this website’s self hosted Git server.

Extreme SSH Hardening

Sun, 10 Feb 2019 21:37:32 +0000

Results from a Mozilla SSH Observatory Scan

SSH (Secure Shell) is a great tool, but if you ever expose SSH to the internet, you’ll soon discover a multitude of internet bots and bad actors pummeling your server.

There is a clever way of mitigating this attack vector passively. Some of you might know this already, but the idea is to configure SSH to maximum

Warning: This will break legacy systems. Here’s an example of a /etc/ssh/sshd_config file in a nutshell.

HostKey /etc/ssh/ssh_host_ed25519_key
KexAlgorithms curve25519-sha256@libssh.org
Ciphers chacha20-poly1305@openssh.com
MACs hmac-sha2-512-etm@openssh.com

LogLevel VERBOSE
SyslogFacility AUTHPRIV
Subsystem sftp internal-sftp -f AUTHPRIV -l INFO

UsePAM yes
PrintMotd no
Compression no
PermitRootLogin no
GSSAPIAuthentication no
PasswordAuthentication no
AuthenticationMethods publickey
ChallengeResponseAuthentication no
AuthorizedKeysFile .ssh/authorized_keys

A configuration like this prevents almost 99% of attackers from reaching or getting pass the negotiation stage. The most you’ll see in your system log is the following.

sshd: Connection from x.x.x.x port x on x.x.x.x port 22
sshd: Connection closed by x.x.x.x port x [preauth]
sshd: Connection from x.x.x.x port x on x.x.x.x port 22
sshd: Connection closed by x.x.x.x port x [preauth]

What did we do? First we let SSH support the most modern cipher, key exhange algorithm, and MAC (message authentication code). This is what kills legacy systems and bots.

HostKey /etc/ssh/ssh_host_ed25519_key
KexAlgorithms curve25519-sha256@libssh.org
Ciphers chacha20-poly1305@openssh.com
MACs hmac-sha2-512-etm@openssh.com

Let it also log everything, including every file modified when you use SSH to mount a remote file system.

LogLevel VERBOSE
SyslogFacility AUTHPRIV
Subsystem sftp internal-sftp -f AUTHPRIV -l INFO

Force public key authentication only and deny root login.

PermitRootLogin no
PasswordAuthentication no
AuthenticationMethods publickey
ChallengeResponseAuthentication no

If you really need root login you can set prohibit-password.

PermitRootLogin prohibit-password

Or activate this setting more strictly using match patterns.

Match Address 192.168.1.*,10.0.3.1
  PermitRootLogin prohibit-password

A Better Firefox Experience

Wed, 23 Jan 2019 21:37:32 +0000

Screenshot of my Firefox Setup

Firefox is my Internet browser. I’ve stuck with it for a long time. Firefox’s market share has dropped considerably, but that doesn’t Here’s my take on a better Firefox experience. This list includes my favorite addons and interface

Step 1: Hide the tab bar

Wait what? Yes. The tab bar interface is somewhat of an anti-pattern. It’s extremely distracting, unproductive, and usually results in 200+ open tabs. In your Firefox profile directory create a folder named chrome, then inside that folder create the file userChrome.css and paste the following code

@-moz-document url("chrome://browser/content/browser.xul") {
  #TabsToolbar {
    visibility: collapse !important;
    margin-bottom: 21px !important;
  }
}

On Firefox 69 and later the following changes are needed.

@@ -1,4 +1,4 @@
-@-moz-document url("chrome://browser/content/browser.xul") {
+@-moz-document url("chrome://browser/content/browser.xhtml") {
   #TabsToolbar {
     visibility: collapse !important;
     margin-bottom: 21px !important;

Clueless about the location of your Firefox profile? Read this helpful guide. Good, onto the next step.

Step 2: Install Sidebery

Sidebery are what tabs should be. Your tabs will now be displayed in a sidebar and assembled in a like format. Every link opened in a new tab will become a branch from the root. Take a look at the image below.

Tree Style Tabs

See how the tabs naturally form many branches related to the root topic Elm? This is a visually succinct and useful way of displaying tabs. Let’s make that new tab linking behavior the default.

Step 3: Open Links in a New Tab

Install the Open Link With New Tab addon and set it to open a new tab only when the target link is different from the website you are currently browsing. This should be the second option in its settings. The tree style tabs can now build up with very little friction.

Step 4: Make DuckDuckGo the Default

Congratulations, you now have direct access to almost every engine on the planet. Bang syntax allows you to search most if not all search engines directly from DuckDuckGo.

Step 5: Bring back RSS

RSS is too important to deprecate. Install RSS Preview to bring back one of Firefox’s most important features — knowing whether a website has a RSS Feed.

Step 6: Unload Inactive Tabs

Install Tab Unloader to unload tabs that are This will save lots of memory. It’s a clever addon that is smart enough to know that tabs with active content should not be considered inactive.

Step 7: Destroy Resource Abuse with uBlock Origin

Resource abuse means ads, banners, popups, and all manner of relentless abuse websites shove down the tube. uBlock Origin is a necessary addon for internet browsing sanity.

Step 8: Disable JavaScript on Demand

Disable JavaScript is a wonderful addon that allows you to disable JavaScript on a per site This makes a lot of sites with lethal and browser crashing JavaScript behave.

Step 9: Tab Sessions

Tab Session Manager is an addon that extends the tab saving functionality of Firefox. It can serve as a simple way to backup browsing sessions.

Step 10: Tab Search

Tab Search is a good addition that allows you to search all open tabs. For persons with an insane amount of tabs, this will save you much pain and frustration.

Step 11: Disable URL Bar Suggestions

Suggestions are nice until they aren’t. To disable this annoyance, we’ll visit the about:config in Firefox and set the following preferences.

browser.urlbar.maxRichResults 0
browser.urlbar.suggest.history false
browser.urlbar.suggest.bookmark false
browser.urlbar.suggest.openpage false
browser.urlbar.suggest.searches false
browser.urlbar.searchSuggestionsChoice false
browser.urlbar.timesBeforeHidingSuggestionsHint 0
browser.search.suggest.enabled false

Then hide what’s left of the URL bar suggestions in chrome/userChrome.css.

/* Hide autocomplete results dropdown */
.urlbarView {
  display: none !important;
}

Firefox 75 introduces the expanding bar. You can disable it temporarily by setting the following preferences.

browser.urlbar.update1 false
browser.urlbar.update1.interventions false
browser.urlbar.update1.searchTips false
browser.urlbar.update1.view.stripHttps false
browser.urlbar.update2.expandTextOnFocus false

You can also opt to the expanding effect permanently in chrome/userChrome.css.

/*
 * Reduce expanding urlbar.
 * Source: https://github.com/WesleyBranton/Remove-Firefox-Megabar#readme
 */
@-moz-document url(chrome://browser/content/browser.xhtml) {
  #urlbar[breakout][breakout-extend] > #urlbar-input-container,
  #urlbar-input-container {
    height: 26px !important;
    padding-block: 0px !important;
    padding-inline: 0px !important;
    transition: none !important;
    border-radius: var(--toolbarbutton-border-radius) !important;
  }

  #urlbar[breakout][breakout-extend] {
    top: calc((var(--urlbar-toolbar-height) - var(--urlbar-height)) / 2) !important;
    left: 0 !important;
    width: 100% !important;
  }

  #urlbar[breakout][breakout-extend] > #urlbar-input-container {
    border: 1px solid var(--toolbar-field-focus-border-color) !important;
  }

  #urlbar[breakout][breakout-extend][open] > #urlbar-input-container {
    border-bottom: 1px solid transparent !important;
    border-radius: var(--toolbarbutton-border-radius) var(--toolbarbutton-border-radius) 0 0 !important;
  }
}

Conclusion

There’s much more that can be added, but this is a good start to streamlining your Firefox browsing work flow.

Website Auditing Tools

Sat, 15 Dec 2018 21:37:32 +0000

Recently Google introduced a new auditing tool that scans your website and gives it a ranking score out of 100.

Upon closer inspection, it looks like another front end to their Lighthouse tool or PageSpeed Insights application. While very informative, this audit gives you but a general glimpse into the possible issues a website might have.

Advanced website auditing can be extensive or may even incur costs but generally you can get a good idea of whether you’re up to date on certain standards by using a myriad of free online tools.

All websites have real world constraints that may inhibit their ability to meet certain standards but it doesn’t hurt to see if anything can be improved immediately.

Comprehensive Tests

A comprehensive test will try to audit as much of a website as possible, providing both succinct and focused analysis.

https://www.dareboost.com/en — My preferred starting point. This tool will give you a gloriously detailed analysis, even going so far as to pull suspect snippets of code from your website and recommend possible solutions. Highly recommended.

https://observatory.mozilla.org — A very comprehensive and detailed tool. Mozilla’s Observatory will analyze SSL, SSH, content security policy, raw headers, and anything important you can think of to test.

https://webhint.io/ — This is another in depth web scanner. The analysis is detailed and just like Dareboost, it presents suspect code snippets from your website and gives recommendations.

Speed Tests

While a web browser might be the best speed testing tool since it’s trivial to emulate different connection speeds — real world tests are great too.

https://gtmetrix.com — My preferred speed testing tool. It gives Pagespeed and YSlow analysis.

https://tools.pingdom.com — If you need an alternative analysis, Pingdom also provides a nice tool for conducting speed tests.

https://web.dev or https://developers.google.com/web/tools/lighthouse/run or https://developers.google.com/speed/pagespeed/insights — Google has so many iteration of products that all seem to do the same thing, but if you’re looking for a quick and dirty speed analysis, these tools are also great.

Content Security Policy

https://securityheaders.com — General content security policy checker.

https://csp-evaluator.withgoogle.com — Excellent tool for debugging your content security policy rules. Highly recommended.

SSL Checks

SSL (Secure Sockets Layer) is now the de facto standard, especially since search engines may alter a site’s ranking based on its security. SSL isn’t needed in all cases, but when in doubt it’s just better to have this security feature enabled. Activating SSL is just the first step.

https://sslping.com — Very quick SSL check. Covers the basics.

https://www.jitbit.com/sslcheck — An SSL crawler. This tool will check to see if any content throughout all searchable parts of the website are over plain HTTP.

https://www.ssllabs.com/ssltest — SSL Labs is pretty well known. This tool provides a very comprehensive and in depth SSL Analysis.

https://tls.imirhil.fr — SSL cipher suite analysis.

Search Engine Optimization

I’m not much of an SEO believer, but there are tools that can give you a glimpse into the world of SEO (Search Engine Optimization).

http://seositecheckup.com — General SEO checks.

http://www.seowebpageanalyzer.com — Keyword and page content analysis.

Miscellaneous

https://realfavicongenerator.net/favicon_checker — The website icon displayed in your browser tab is important too. This tool will not only check to see if your favicons are set, but will also check all other favicons for operating system platforms including Windows, Android, and Mac OS.

http://www.favicomatic.com/favicon-test — If you’re looking for an even more detailed favicon analysis. This tool will bring you full circle by checking against most if not all of the favicon formats available.

https://gdprchecklist.io — For GDPR Compliance (General Data Protection Regulation).

Bitcoin Hacked Email Scam

Mon, 19 Nov 2018 21:37:32 +0000

Example of a scam email.

This showed up in my inbox today. Scammers are apparently upping their game, but not by a lot. This one was rather clever though. The sender tries to convince me that my account was hacked and attempts to extort me into paying Bitcoin to the above highlighted address.

“Hello! I have bad news for you. 13/07/2018 - on this day I hacked your operating system and got full access to your account [email] On that day your account [email] password was: [password].” Quote from the extortionist

What makes the scam slightly convincing is that the password stated in the email is a legitimate password. In my case, this is a password from a very long time ago. Not only that, but this message appears to be sent from my own email address. Even the email headers look legit, but the message is sent from a spoofed address.

Data breach check at “have i been pwned?” website.

How’d they get the password? That’s an easy answer. My email has been exposed in at least five different security breaches — chief among them: Dropbox, Bit.ly, and Last.fm. Anyone that has access to those pawned files can troll for real passwords, emails, and other info to construct an elaborate scheme such as this one.

The Bitcoin address in question has already been reported multiple times and it’s evident that many variations of this scam have been sent to numerous This is however, a good reminder to not use the same password on multiple sites and to change passwords often.

My Sublime Text Setup

Sun, 16 Sep 2018 21:37:32 +0000

Sublime Text is a great general purpose performant text editor. It provides a lightweight interface and an excellent plugin packaging system.

While I absolutely love Vim and Geany, Sublime Text serves as a good compromise between flexibility and efficiency, especially when doing web development oriented tasks. Everyone has their preferred plugin arrangement and I’ll show you what my set up looks like.

Preferences

Sublime stores its settings in json format. You can find my preferred settings right here in my git repository. Since this write up is mainly focused on the plugins that are installed, no further details will be provided about my Sublime preferences.

Plugins

Advanced New File — An improved implementation of the new file function in Sublime. Eliminates some of the quirkiness of the native new file shortcut, providing more control and speed when creating new files.

Auto Set Syntax — This plugin automatically sets the syntax of a file based on the first typed line. This is useful when creating a new file where the language is not known because the file extension has not yet been set.

Bracket Highlighter — A tag bracket highlighter. No more hunting around to see where a tag ends.

CSS Format — Allows you to reformat CSS on the fly into different forms, i.e compact, expanded, minified, etc. This is my go-to plugin when it is necessary to reformat portions of CSS quickly for readability.

Dayle Rees Color Schemes — A variety of colour schemes by Dayle Rees. This provides a nice assortment of syntax coloring schemes.

Elm Language Support — A plugin that provides Elm syntax highlighting.

Emmet — Good plugin for improving HTML workflow. Provides the use of abbreviations to quickly define multiple tags.

Hooks — Allows a user to run commands based on event hooks like “open new file” or “save file”. Sometimes this is used to trigger a lazy live browser reload everytime a file is saved. You can view my poor man’s browser reload script in my dotfiles git repository.

Origami — Split a given window into multiple sections. This plugin eliminates the annoyance of having to juggle between two or more files.

PHP Getters and Setters — This plugin provides a quick way to generate getters and setters for a PHP class.

PHP Unit Kit — PHP unit testing kit for Sublime Text.

Predawn — This is one of my favorite themes for Sublime Text.

Sass — SASS and SCSS syntax highlighting.

Shell Command — Beautiful plugin that allows arbitrary shell commands to be ran from within Sublime Text.

Sublime Linter — General purpose code linting framework. Good for picking up errors in the syntax of multiple languages based on their documentation.

Sublime Linter-csslint — CSS code linter.

Sublime Linter-phpcs — PHP code linter.

Vue Syntax Highlight — Vue syntax highlighting.

Conclusion

That’s it. These are the plugins that I currently use in my web development work flow.

Mixing PHP into Hugo

Wed, 01 Aug 2018 21:37:32 +0000

Hugo’s Homepage

Hugo is nice because of its blog template system, static site generation abilities and generic framework structure, but sometimes you may want to add extra functionality that requires dynamic

If you are also combining static elements with dynamic functions you may want to DRY your views to make everything consistent. It could be that you only want the blog of your website to be static and everything else to operate dynamically.

We can generate files ending in .php using the media types option in Hugo, however because the template is in the layouts/_default directory, we can simply inject something like this directly within the .html

{{ safeHTML "<?php require $_SERVER['DOCUMENT_ROOT'] . '/..' . '/views/partials/navigator.php'; ?>" }}

This tells Hugo when compiling to insert the above PHP code as safe HTML into the statically generated If your web server is set up to parse .html files as PHP then you are good to go.

# nginx let fastcgi parse .php, .html, and .htm
location ~ \.(php|html|htm)$ {
    include default.d/php_fastcgi.conf;
}

More tricks can be had by carefully tiptoeing around the Go You get the best (or worst) of both

{{ safeHTML "<?php" }}
$width = getimagesize($_SERVER['DOCUMENT_ROOT'] . '{{.Get `source`}}')[0];
$height = getimagesize($_SERVER['DOCUMENT_ROOT'] . '{{.Get `source`}}')[1];
$ratio = ((($height / $width) * 100) > 100) ? $height . 'px' : $height / $width * 100 . '%';
{{ safeHTML "?>" }}

<img data-image-zoom src="{{.Get `source`}}" alt="{{.Get `title`}}" title="{{.Get `title`}}"
{{ safeHTMLAttr `<?php echo 'width=' . '"' . $width . '"'; ?>` }}
{{ safeHTMLAttr `<?php echo 'height=' . '"' . $height . '"'; ?>` }} />

If you are interested in producing .php pages with Hugo, take a look at using custom output formats. Note that since Hugo version 0.44 the MediaType.Suffix is deprecated and replaced with a plural version, MediaType.Suffixes

Now-DNS and pfSense

Thu, 26 Jul 2018 21:37:32 +0000

The alternative to a static IP address is a server that keeps track of your dynamic IP address. When the IP address changes, the client will immediately notify the server of the new IP address. This is known as Dynamic DNS. One of the free DDNS services that I use is Now-DNS.

I’ve noticed on pfSense version 2.3.5-RELEASE-p2 that custom client update entries will not work when the update url for curl looks something like this;

shell

curl -u <email>:<password> "https://now-dns.com/update?hostname=<hostname>"

It appears that the authentication fails when executing the client update. To fix this issue, remove the curl HTTP authentication option by commenting out the line below in the file /etc/inc/dyndns.class

php

<?php
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);

php

<?php
if ($this->_curlSslVerifypeer) {
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, TRUE);
  } else {
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
  }
    #curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
    curl_setopt($ch, CURLOPT_USERPWD, "{$this->_dnsUser}:{$this->_dnsPass}");
}

This does not have an adverse affect on custom client entries that use a token style update url which do not require a username or password. These urls tend to look like this;

text

https://www.duckdns.org/update?domains=exampledomain&token=a7c4d0ad-114e-40ef-ba1d-d217904a50f2&ip=%IP%

GPU Acceleration on Remote Desktop

Sun, 11 Feb 2018 21:37:32 +0000

A screenshot of my remote server’s desktop

Sometimes the solution to a problem is strange. Imagine a headless, remote, Linux based desktop setup where users can remote into a graphical environment. The remote desktop works great with the integrated graphics card but once it switches to discrete graphics it comes to a screeching halt. No graphical acceleration.

Many discrete graphic cards now enter a low power state if you do not connect a monitor. This is good for users who care about power efficiency but a pain for those who implement headless configurations. The common solution to this problem is to buy a “dummy plug” for your HDMI or Display Port to fake a head onto your headless configuration. It does work.

There is a way of doing this without a dummy plug. It turns out that we can pretend that we have a display connected by using xrandr. First tell the system to add new resolution

xrandr --newmode "1366x768p" 72.00 1366 1414 1446 1494 768 771 777 803;
xrandr --newmode "1280x800p" 24.15 1280 1312 1400 1432 800 819 822 841;
xrandr --newmode "1024x768p" 18.71 1024 1056 1120 1152 768 786 789 807;

Then add and assign those resolution modes to a port

xrandr --addmode DisplayPort-0 "1366x768p";
xrandr --addmode DisplayPort-0 "1280x800p";
xrandr --addmode DisplayPort-0 "1024x768p";

Finally pick a mode and then force output on the interface.

xrandr --output DisplayPort-0 --mode "1366x768p";

The discrete graphics card comes alive and we get GPU acceleration on the remote. This may not work for all cards, but it has good coverage. For everything else there is always the “dummy plug”.

SSH Port Fowarding Tunnels

Mon, 08 Jan 2018 21:37:32 +0000

The other day I needed to set up a Syncthing instance on a remotely networked server that had no graphical display. I didn’t want to set it up using the command line tools and I definitely did not want to install a graphical environment on the server to access the web GUI.

There is another computer with a graphical display on the local network. The only problem is that a standard syncthing installation does not have its port (8384) exposed to computers on the local network. We would be unable to access the web server.

This means that one would need to expose that port either by configuring syncthing directly or by opening up the respective port. That direction is more involved. There is a great solution to this problem and it’s called SSH port forwarding. The solution comes down to this single command.

shell

ssh -v -NL 12345:127.0.0.1:8384 machine.remote

The above command states that it will forward or bind port 8384 of 127.0.0.1 (syncthing) from machine.remote to localhost's port 12345 (computer with the graphical display) such that the user on localhost can access machine.remote's port 8384 as if it was on port 12345. The -N flag prevents the SSH command from executing on the remote so that it is forced to run as a foreground process at the command line.

Posts from Thedro Neely

Guix in a Linux Container

Commands

Overview

Configuration

Container

Comparisons

Conclusion

Incremental Writes in Hugo

Overview of Site Rendering

Incremental Writes

Applications and Utility

Conclusion

Extensible Stylesheets

Formats, File Extensions, and MIME types

XML Validation and Formatting

Stylesheet Processing and Validation

Stylesheet Boilerplate and Transformations

Attributes and Value Selection

For Each

Switch Statements

Conclusion

Making Web Pages

Introduction

HTML and XML Utilities

Basic Templating

Conclusion

A JavaScript Executable

JavaScript with Deno Runtime

File and Directory Setup

Specifics & Details

Conclusion

Literate Programming

Ideal

Literate Programming Tools and Programs

Conclusion

Writing Strategy

Hugo is Good

Markdown

Diagrams

Configuration Formats

Code Fences and Syntax Highlighting

Short Codes

Render Hooks

Output Formats

Headless Content Management

Conclusion

Web Browsers

The URL Bar is Replaced by the Everything Search Bar

Bookmarks and History Replaced by the Integrated Search Engine

The Web Browser as an Internet Operating System and Specification

Deprioritization of Open Source Software and Ascendancy of Open Source Hardware

Specialized Hardware, Eternal Defaults, and Challenging Legacy Carrier Features

Deprecation of the Developer Tools and Transition to Developer Programs

Property Stewardship above Ownership and Co-ownership

Defragmentation Markets: Buying Interoperability and Integration

Unblockable Advertisements, Cryptocurrency and Interaction Fees

Deprecating Registration Forms: Universal Password and Data Management

Market Share Dominance, New Mobile Operating Systems, and Global Carrier Wars

Conclusion and Alternative Disruptions

NixOS Pins and Needles

Testing Single Files Quickly

Searching For Packages and Files

Running Configurations

DRY is Evil Incarnate

Modules Should Do One Thing

Separate System and User Environments

Imperative to Declarative User Experience

Conclusion

A Few Links

Conclusion

Typesetting in LaTeX

Typesetting Workflow

Conclusion

Cooking and Baking Linux Distributions in Nix

Nix Shell

Cooking

Baking

Conclusion

A Few Abstracts