NixOS in The Wild

Nix and NixOS are two technologies that my eyes have been on for a while. Nix is the independent package manager that allows for reproducible Close to 99% reproducible. and immutable package managment and NixOS is a Linux distribution built from the ground up using Nix.

Most I'd like to keep some of my Alpine, Arch, Debian, and CentOS installations around. of my personal infrastructure now resides on NixOS including the server that hosts this website. CentOS, the previous Linux distribution running this server has done well. Shadowing this NixOS server just in case things blow up.

14:39:28 up 502 days, 15:30,  1 user,  load average: 0.00, 0.01, 0.05

Let’s discuss the pain Disclaimer: Some pain points are fixed in later versions. This post (which is more like a note to myself) tracks nixpkgs 20.03 commits. points, use cases, and general application of NixOS in the real world for both desktops and servers — but before that here’s my opinion on the package repository and documentation search discovery.

The Package Repository

The Nix Package Repository or nixpkgs contains every package derivation. My one gripe is that GitHub chokes on this repository hard. Searching the right term over 4,000+ issues can intermittently timeout For this reason issues and pull requests are not viewed from within GitHub. The interface is just too slow. the page. This is not common but happens often enough to notice. In my experience, GitHub issue discovery is better achieved using a search engine that supports site queries.

site:https://github.com/NixOS/nixpkgs/issues <keyword>

Online Documentation and Search Discovery

In my experience, it’s not the online documentation that is bad per se — it’s the search discoverability that’s poor. Nix’s poor This post probably makes it worse, but perhaps the title is tricky enough to fool search engines. search discoverability is the result of generating most documentation as a single HTML page.

These pages are impeccable under the hood with anchor id links to each section and subsection, but search engines Searching for phases on Bing are not smart enough to scope out this context and link directly to the anchor id from the search engine results page (SERP). Most search queries send you to the top of the Nix package manual.

Local Documentation and Troubleshooting

NixOS provides a lot of documentation out of the box that is specific to every installation. Users familiar with man will run man configuration.nix to see every Simple and fast. config option available for a system. This means that the local configuration.nix documentation is more accurate than searching the online NixOS options index.

In fact, the entire package repository (nixpkgs) resides on every installation as a channel. In the above manpage nixpkgs/nixos/modules/config/appstream.nix declares the logic behind appstream.enable.

View this declaration path by getting the channel name as root and navigating to the per user channel source tree.

$ sudo nix-channel --list
nixos https://releases.nixos.org/nixos/20.03/nixos-20.03.2351.f8248ab6d9e

$ tree -L 1 /nix/var/nix/profiles/per-user/root/channels/nixos
├── COPYING
├── default.nix
├── doc
├── flake.nix
├── lib
├── maintainers
├── nixos
├── nixpkgs -> .
├── pkgs
├── programs.sqlite
├── README.md
└── svn-revision

$ nix-instantiate --eval -E '(import <nixpkgs> {}).lib.version'
"20.03.2351.f8248ab6d9e"

This is good to know even if you don’t care about the source purely as a reminder that not every system has the same state or exposes the same configuration options.

User Autologin

Most of my desktop machines do not use login managers. The autologin settings in NixOS are login manager dependent. Apparently it’s quite trivial to override the autovt@tty1 service and force autologin for any user without having to install a login manager. Set restartIfChanged to false to avoid restarting your desktop.

{ pkgs, ... }:

{
  systemd.services."autovt@tty1" = {
    after = [ "systemd-logind.service" ];
    restartIfChanged = false;
    serviceConfig = {
      Type = "simple";
      ExecStart = "${pkgs.utillinux}/sbin/agetty --autologin ${username} --noclear %I $TERM";
      Restart = "always";
    };
  };
}

Updating NixOS

Updating NixOS is tricky business. The short of it is that generally the system wide channel is the repository used to source updates. Most systems have one channel called nixos or nixpkgs under the user root. Every user inherits the root user’s channel unless they have set their own per user channel.

To update, as root list the channel with nix-channel --list, override or add a channel with nix-channel --add, update the channel with nix-channel --update, upgrade every per user environment with nix-env --upgrade and finally That's rather cumbersome manually. My preference is to sync the channel declaratively with nixops and use a per user declarative configuration with nix-env. rebuild the system with nixos-rebuild switch --upgrade.

# nix-channel --list
nixos https://nixos.org/channels/nixos-20.03
# nix-channel --add 'https://nixos.org/channels/nixos-20.03' nixos
# nix-channel --update
# nix-env --upgrade
# nixos-rebuild switch --upgrade

If you trust the stability, you can use system.autoUpgrade.enable and system.autoUpgrade.channel, but those are scary options for my use case.

Shellcheck and systemd

Every language, system, or framework leads you down a certain path. NixOS tends to lead you down the path of writing lots of shell scripts with systemd. Save yourself the pain — set the shell to a stricter mode and run shellcheck upon service execution.

{ pkgs, ... }:

{
  systemd.services.my-service = {
    description = "My service";
    wantedBy = [ "multi-user.target" ];
    path = [ pkgs.shellcheck ];
    script = ''
      set -euxo pipefail
      shellcheck "$0" || exit 1
      # Code goes here...
    '';
  };
}

LibreOffice and Spell Checking

Spell checking is a commonly used feature of LibreOffice. On my NixOS 20.03 machines, LibreOffice spell checking doesn’t seem to work. Add the missing dictionary package links and expose them using DICPATH. Log out and back in and LibreOffice should find the spell checking modules.

{ pkgs, ... }:

{
  environment.systemPackages = with pkgs; [
    hunspell
    hunspellDicts.en_US-large
    hyphen
  ];

  environment.pathsToLink = [ "/share/hunspell" "/share/myspell" "/share/hyphen" ];

  environment.variables.DICPATH = "/run/current-system/sw/share/hunspell:/run/current-system/sw/share/hyphen";
}

Hashless Git Fetching

Use the function builtins.fetchGit to fetch the HEAD directly on a repository branch without stipulating a hash. The builtin function fetchGit will fetch any git repository at evaluation time — allowing for automation, ssh-agent integration, and other niceties.

{ stdenv }:

let url = "https://github.com/koalaman/shellcheck"; in

stdenv.mkDerivation rec {

  pname = "shellcheck";
  version = "master";

  src = fetchGit { inherit url; ref = "refs/heads/master"; };

  dontBuild = true;

  installPhase = ''
    mkdir $out
    # Code goes here...
  '';
}

Package Tracking and systemd

In a continuous integration and deployment (CI/CD) environment you can set a systemd service to restart on package changes. Use restartTriggers to track upstream package definitions.

{ pkgs, ... }:

let package = pkgs.callPackage ./default.nix {}; in

{
  systemd.services.my-service = {
    description = "My service";
    wantedBy = [ "multi-user.target" ];
    restartTriggers = [ package ];
    path = [ pkgs.shellcheck ];
    script = ''
      set -euxo pipefail
      shellcheck "$0" || exit 1
      # Code goes here...
    '';
    serviceConfig = { RemainAfterExit = "yes"; };
  };
}

Internet Connectivity in the Nix Sandbox

Nix build environments prohibit internet connections outside the scope of defined builtin functions like fetchgit or fetchurl. You can remove this restriction by setting the correct expected hash recursively before evaluation. This allows us to do “illegal” things like the following.

{ stdenv, pkgs }:

let url = "https://thedroneely.com/git/thedroneely/thedroneely.com.git"; in

stdenv.mkDerivation rec {

  pname = "composer";
  version = "master";

  src = fetchGit { inherit url; ref = "refs/heads/master"; };

  buildInputs = [ pkgs.cacert pkgs.php74Packages.composer ];

  dontBuild = true;

  installPhase = ''
    composer --no-cache install
    mkdir $out
    cp -r vendor $out/vendor
  '';

  outputHashAlgo = "sha256";
  outputHashMode = "recursive";
  outputHash = "0zkqkbwz5vg4k95s83pl0kxvphav1wzmivs5b1kmwf101wnj1m4q";
}

Kernel Patching for Kids

Applying kernel patches using fetchurl in NixOS is a rather trivial endeavour. You should manually patch, configure, and compile a kernel at least once to understand what’s happening.

{
  boot.kernelPatches = [

    { name = "ck-5.6"; patch = (builtins.fetchurl {
      url = "http://ck.kolivas.org/patches/5.0/5.6/5.6-ck2/patch-5.6-ck2.xz";
      sha256 = "18rk9023b14x62n0ckbnms6ahq5yjramz7qfjagkaga95i8ha6b2"; });
    }

    { name = "uksm-5.6"; patch = (builtins.fetchurl {
      url= "https://raw.githubusercontent.com/dolohow/uksm/master/v5.x/uksm-5.6.patch";
      sha256 = "021sylwacamh8q26agcp0nmmw3ral2wl7bgibmi379irnvy0c37y"; });
    }

    { name = "userns-overlayfs"; patch = (builtins.fetchurl {
      url= "https://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/patch/?id=0c29f9eb00d76a0a99804d97b9e6aba5d0bf19b3";
      sha256 = "1j4ind31hgkjazbgfd64lpaiqps8hcsqkar4v6nvxrpysmkg9nfd"; });
    }

  ];
 }

Nginx and its Temporary Folders

The permissions and ownership on nginx’s temporary folders can change in peculiar circumstances. On NixOS, there is a chance that the user nginx becomes dissociated from its folders when disabling and re-enabling nginx using the option services.nginx.enable. The temp folders for nginx are in /var/spool/nginx. Use systemd’s tmpfiles.d to ensure that these permissions always stay consistent.

{
  systemd.tmpfiles.rules = [
    "d /var/spool/nginx                  0700 nginx nginx -"
    "d /var/spool/nginx/client_body_temp 0700 nginx nginx -"
    "d /var/spool/nginx/fastcgi_temp     0700 nginx nginx -"
    "d /var/spool/nginx/logs             0700 nginx nginx -"
    "d /var/spool/nginx/proxy_temp       0700 nginx nginx -"
    "d /var/spool/nginx/scgi_temp        0700 nginx nginx -"
    "d /var/spool/nginx/uwsgi_temp       0700 nginx nginx -"
  ];
}

The Hash as Truth

One mode of failure when working with Nix and NixOS is that nothing happens when you try to apply a change. Nix looks at the changes of a package’s derivation hash to decide if something should be rebuilt and applied. This also means that nix knows the hashes of already built resources.

When you want to set a new sha256 and force a rebuild: don’t place a random dummy string or change one of the characters in the original sha256. Always use stdenv.lib.fakeSha256 or the command nix-prefetch-url to fetch the new sha256 hash. The reason is simple — if you happen to use a hash that is already known, there is a chance that nix will download the related binaries from the cache and leave you with an interesting debugging session.

{ stdenv, fetchurl }:

stdenv.mkDerivation rec {

  pname = "puppeteer-docs";
  version = "latest";

  src = fetchurl {
    url = "https://raw.githubusercontent.com/puppeteer/puppeteer/main/docs/api.md";
    sha256 = stdenv.lib.fakeSha256;
  };

  phases = [ "installPhase" ];

  installPhase = ''
    mkdir -p $out
    cp ${src} $out/api.md
  '';

  meta = with stdenv.lib; {
    description = "Puppeteer Documentation";
    homepage = "https://github.com/puppeteer/puppeteer/blob/main/docs/api.md";
  };
}

Nix Commands and Local Caching

Nix is slow and caches to improve speed. Local caching can sometimes interfere with automation tasks when using commands like nix-build or nixops. Use the --option argument and set the tarball-ttl to 0 to ensure you are always pulling fresh sources and repositories.

$ nixops deploy -d deployment --option tarball-ttl 0
$ nix-build --option tarball-ttl 0

Logs and Logrotate

On NixOS, it’s systemd all The seething rage. the way down. The important logs for most services route to systemd’s journald. The defaults are okay, but logrotate is a crucial program in my stack.

The abstraction for logrotate is fine, and it’s trivial to mimic my preferred setup as it would be on any traditional Linux distribution. Enable logrotate with a default preset.

{
  services.logrotate = {
    enable = true;
    config = ''
      compress
      create
      daily
      dateext
      delaycompress
      missingok
      notifempty
      rotate 31
    '';
  };
}

Create a per service setup using the config attribute This config attribute is of type types.lines which basically means it will mix down or append all declarations into a single logrotate.conf. anywhere in your configuration, but preferably in the same file as the service in question (nginx).

{
  services.logrotate.config = ''
    /var/spool/nginx/logs/*.log {
      create 644 nginx nginx
      postrotate
        systemctl reload nginx
      endscript
    }
  '';
}

Using the Force

Breaking the assumptions of abstractions and frameworks is a hobby of mine. On NixOS there are special situations where the abstractions get in the way of the desired configuration. Use lib.mkForce as a forcing action to override It looks like mkForce is an alias of mkOverride 50 which sets the override priority of a configuration option. any configuration option and prevent other modules from interfering. The name service switch configuration is written to /etc/nsswitch.conf using environment.etc and lib.mkForce to prevent casualties.

{ lib, ... }:

{
 environment.etc."nsswitch.conf".text = lib.mkForce ''
   ethers: files
   group: files systemd
   netgroup: files
   networks: files
   passwd: files systemd
   protocols: files
   publickey: files
   rpc: files
   services: files
   shadow: files

   hosts: files mymachines myhostname mdns4_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] dns
 '';
}