NixOS in The Wild

Nix and NixOS are two technologies that my eyes have been on for a few years. Nix is the independent package manager that allows for Close to 99% reproducible. and immutable package managment and NixOS is a Linux distribution built from the ground up using Nix.
Keeping my Alpine, Arch, Debian, and CentOS installations around is a must. of my personal infrastructure now resides on NixOS including the server that hosts this website. CentOS, the previous Linux distribution running this server has done Shadowing this NixOS server just in case things blow up.
14:39:28 up 502 days, 15:30, 1 user, load average: 0.00, 0.01, 0.05
Let’s discuss the Disclaimer: Some pain points are fixed in later versions. This post (which is more like a note to myself) tracks nixpkgs 20.03 commits. points, use cases, and general application of NixOS in the real world for both desktops and servers but before that — here’s my experience with the package repository and documentation search discovery.
The Package Repository

The
Nix Package Repository
or nixpkgs
contains every package derivation. My one pain point is that
GitHub
chokes on this repository hard. Searching the right term over 4,000+
issues can intermittently
For this reason issues and pull requests are not viewed from within GitHub. The interface is just too slow.
the page. This is not common but happens often enough to notice.
In my experience, GitHub issue discovery is better achieved using a search engine that supports site
queries.
site:https://github.com/NixOS/nixpkgs/issues <keyword>
Online Documentation and Search Discovery
In my experience, it’s not the online documentation that is bad per se — it’s the search discoverability that’s poor. Nix’s
This post probably makes it worse, but perhaps the title is tricky enough to fool search engines.
search discoverability is the result of generating most documentation as a single HTML
page.
These pages are impeccable under the hood with anchor id
links to each section and subsection, but search
are not smart enough to scope out this context and link directly to the anchor
id
from the search engine results page (SERP
). Most search queries send you to the top of the
Nix package manual.

Local Documentation and Troubleshooting
NixOS provides a lot of documentation out of the box that is specific to every installation. Users familiar with man
will run man configuration.nix
to see
Simple and fast.
config
Use nixos-option
if you know what you are looking for.
available for a system. This means that the local configuration.nix
documentation is more accurate than searching the online
NixOS options index.

In fact, the entire
package repository
(nixpkgs
) resides on every installation as a channel
. In the above manpage nixpkgs/nixos/modules/config/appstream.nix
declares the logic behind appstream.enable
.
View this declaration path by listing the channel
name as root
and navigating to the per user channel
source tree.
$ sudo nix-channel --list
nixos https://releases.nixos.org/nixos/20.03/nixos-20.03.2351.f8248ab6d9e
$ tree -L 1 /nix/var/nix/profiles/per-user/root/channels/nixos
├── COPYING
├── default.nix
├── doc
├── flake.nix
├── lib
├── maintainers
├── nixos
├── nixpkgs -> .
├── pkgs
├── programs.sqlite
├── README.md
└── svn-revision
$ nix-instantiate --eval -E '(import <nixpkgs> {}).lib.version'
"20.03.2351.f8248ab6d9e"
This is good to know even if you don’t care about the source purely as a reminder that not every system has the same state or exposes the same configuration options.
User Autologin
Most of my desktop machines do not use login managers. The autologin settings in NixOS are login manager dependent. Apparently it’s quite trivial to override the autovt@tty1
service and force autologin for any user without having to install a login manager. Set restartIfChanged
to false
to avoid restarting your desktop.
{ pkgs, ... }:
{
systemd.services."autovt@tty1" = {
after = [ "systemd-logind.service" ];
restartIfChanged = false;
serviceConfig = {
Type = "simple";
ExecStart = "${pkgs.utillinux}/sbin/agetty --autologin ${username} --noclear %I $TERM";
Restart = "always";
};
};
}
Updating NixOS
Updating NixOS is tricky business. The short of it is that generally the system wide channel
is the repository
used to source updates. Most systems have one channel
called nixos
or nixpkgs
under the user root
. Every user inherits the root
user’s channel
unless they have set their own per user channel
.
- Using
root
list the channel withnix-channel --list
- Override or add a channel with
nix-channel --add
- Update the channel with
nix-channel --update
- Upgrade every per user environment with
nix-env --upgrade
-
That's rather cumbersome manually (but for good reason). My preference is to sync the channel declaratively with
nixops
and use a per user declarative configuration withnix-env
. the system withnixos-rebuild switch --upgrade
# nix-channel --list
nixos https://nixos.org/channels/nixos-20.03
# nix-channel --add 'https://nixos.org/channels/nixos-20.03' nixos
# nix-channel --update
# nix-env --upgrade
# nixos-rebuild switch --upgrade
If you trust the stability, you can use system.autoUpgrade.enable
and system.autoUpgrade.channel
, but those are scary options for my use case.
Shellcheck and systemd
Every language, system, or framework leads you down a certain path. NixOS coaxes you down the path of writing lots of shell
scripts with systemd
. Save yourself the pain — set the shell
to a stricter mode and run
shellcheck
upon service execution.
{ pkgs, ... }:
{
systemd.services.my-service = {
description = "My service";
wantedBy = [ "multi-user.target" ];
path = [ pkgs.shellcheck ];
script = ''
set -euxo pipefail
shellcheck "$0" || exit 1
# Code goes here...
'';
};
}
LibreOffice and Spell Checking
Spell checking is a commonly used feature of
LibreOffice.
On my NixOS 20.03
machines, LibreOffice spell checking doesn’t seem to work. Add the missing dictionary package links and expose them using DICPATH
. Log out and back in and LibreOffice should find the spell checking modules.
{ pkgs, ... }:
{
environment.systemPackages = with pkgs; [
hunspell
hunspellDicts.en_US-large
hyphen
];
environment.pathsToLink = [ "/share/hunspell" "/share/myspell" "/share/hyphen" ];
environment.variables.DICPATH = "/run/current-system/sw/share/hunspell:/run/current-system/sw/share/hyphen";
}
Hashless Git Fetching
Use the function builtins.fetchGit
to fetch the HEAD
directly on a repository branch without stipulating a hash. The builtin function fetchGit
will fetch any git
repository at evaluation time — allowing for automation, ssh-agent
integration, and other niceties.
{ stdenv }:
let url = "https://github.com/koalaman/shellcheck"; in
stdenv.mkDerivation rec {
pname = "shellcheck";
version = "master";
src = fetchGit { inherit url; ref = "refs/heads/master"; };
dontBuild = true;
installPhase = ''
runHook preInstall
mkdir $out
# Code goes here...
runHook postInstall
'';
}
Package Tracking and systemd
In a continuous integration and deployment (CI/CD
) environment you can set a systemd
service to restart on package changes. Use restartTriggers
to track upstream package
definitions.
{ pkgs, ... }:
let package = pkgs.callPackage ./default.nix {}; in
{
systemd.services.my-service = {
description = "My service";
wantedBy = [ "multi-user.target" ];
restartTriggers = [ package ];
path = [ pkgs.shellcheck ];
script = ''
set -euxo pipefail
shellcheck "$0" || exit 1
# Code goes here...
'';
serviceConfig = { RemainAfterExit = "yes"; };
};
}
Internet Connectivity in the Nix Sandbox
Nix build environments prohibit internet connections outside the scope of defined builtin functions like fetchgit
or fetchurl
. You can remove this restriction by setting the correct expected hash recursively before evaluation. This allows us to do “illegal” things like the following.
{ stdenv, pkgs }:
let url = "https://thedroneely.com/git/thedroneely/thedroneely.com.git"; in
stdenv.mkDerivation rec {
pname = "composer";
version = "master";
src = fetchGit { inherit url; ref = "refs/heads/master"; };
buildInputs = [ pkgs.cacert pkgs.php74Packages.composer ];
dontBuild = true;
installPhase = ''
composer --no-cache install
mkdir $out
cp -r vendor $out/vendor
'';
outputHashAlgo = "sha256";
outputHashMode = "recursive";
outputHash = "0zkqkbwz5vg4k95s83pl0kxvphav1wzmivs5b1kmwf101wnj1m4q";
}
Kernel Patching for Kids
Applying kernel patches using fetchurl
in NixOS is a trivial endeavour. You should manually patch, configure, and compile a kernel at least once to understand what’s happening.
{
boot.kernelPatches = [
{ name = "ck-5.6"; patch = (builtins.fetchurl {
url = "http://ck.kolivas.org/patches/5.0/5.6/5.6-ck2/patch-5.6-ck2.xz";
sha256 = "18rk9023b14x62n0ckbnms6ahq5yjramz7qfjagkaga95i8ha6b2"; });
}
{ name = "uksm-5.6"; patch = (builtins.fetchurl {
url= "https://raw.githubusercontent.com/dolohow/uksm/master/v5.x/uksm-5.6.patch";
sha256 = "021sylwacamh8q26agcp0nmmw3ral2wl7bgibmi379irnvy0c37y"; });
}
{ name = "userns-overlayfs"; patch = (builtins.fetchurl {
url= "https://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/patch/?id=0c29f9eb00d76a0a99804d97b9e6aba5d0bf19b3";
sha256 = "1j4ind31hgkjazbgfd64lpaiqps8hcsqkar4v6nvxrpysmkg9nfd"; });
}
];
}
Nginx and its Temporary Folders
The permissions and ownership on
nginx’s
temporary folders can change in peculiar circumstances. On NixOS, there is a chance that the user nginx
becomes dissociated from its folders when disabling and re-enabling nginx
using the option services.nginx.enable
. The temp
folders for nginx
are in /var/spool/nginx
. Use systemd’s tmpfiles.d
to ensure that these permissions always stay consistent.
{
systemd.tmpfiles.rules = [
"z /var/spool/nginx 0700 nginx nginx -"
"z /var/spool/nginx/client_body_temp 0700 nginx nginx -"
"z /var/spool/nginx/fastcgi_temp 0700 nginx nginx -"
"z /var/spool/nginx/logs 0700 nginx nginx -"
"z /var/spool/nginx/proxy_temp 0700 nginx nginx -"
"z /var/spool/nginx/scgi_temp 0700 nginx nginx -"
"z /var/spool/nginx/uwsgi_temp 0700 nginx nginx -"
];
}
The Hash as Truth
A common mode of failure when working with Nix and NixOS is that
That's a good thing.
happens when you try to apply a change. Nix looks at the changes of a package’s derivation hash to decide if something should be rebuilt and applied. This also means that nix
knows the hashes of already built resources.
When you want to set a new sha256
and force a rebuild: don’t place a random dummy string or change one of the characters in the original sha256
. Always use lib.fakeSha256
or the command nix-prefetch-url
to fetch the new sha256
hash.
The reason is simple — if you happen to use a hash that is already known, there is a chance that nix
will download the related binaries from the cache and leave you with an interesting debugging session.
{ stdenv, fetchurl }:
stdenv.mkDerivation rec {
pname = "puppeteer-docs";
version = "latest";
src = fetchurl {
url = "https://raw.githubusercontent.com/puppeteer/puppeteer/main/docs/api.md";
sha256 = stdenv.lib.fakeSha256;
};
phases = [ "installPhase" ];
installPhase = ''
runHook preInstall
mkdir -p $out
cp ${src} $out/api.md
runHook postInstall
'';
meta = with stdenv.lib; {
description = "Puppeteer Documentation";
homepage = "https://github.com/puppeteer/puppeteer/blob/main/docs/api.md";
};
}
Nix Commands and Local Caching
Nix is slow and caches to improve speed. Local caching can sometimes interfere with automation tasks when using commands like nix-build
or nixops
. Use the --option
argument and set the tarball-ttl
to 0
to ensure you are always pulling fresh sources and repositories.
$ nixops deploy -d deployment --option tarball-ttl 0
$ nix-build --option tarball-ttl 0
Logs and Logrotate
On NixOS, it’s
systemd
The seething rage.
the way down. The important logs for most services route to systemd’s journald
. The defaults are okay, but logrotate
is a crucial program in my stack.
The abstraction for logrotate
is fine, and it’s trivial to mimic my preferred setup as it would be on any traditional Linux distribution. Enable logrotate
with a default preset.
{
services.logrotate = {
enable = true;
config = ''
compress
create
daily
dateext
delaycompress
missingok
notifempty
rotate 31
'';
};
}
Create a per service setup using the config
This config
attribute is of type types.lines
which basically means it will mix down or append all declarations into a single logrotate.conf
.
anywhere in your configuration, but preferably in the same file as the service in question (nginx
).
{
services.logrotate.config = ''
/var/spool/nginx/logs/*.log {
create 644 nginx nginx
postrotate
systemctl reload nginx
endscript
}
'';
}
Using the Force
Breaking the assumptions of abstractions and frameworks is a hobby of mine. On NixOS there are special situations where the abstractions get in the way of the desired configuration. Use lib.mkForce
as a forcing action to
It looks like mkForce
is an alias of mkOverride 50
which sets the override priority of a configuration option.
any configuration option and prevent other modules from interfering. The
name service switch
configuration is written to /etc/nsswitch.conf
using environment.etc
and lib.mkForce
to prevent casualties.
{ lib, ... }:
{
environment.etc."nsswitch.conf".text = lib.mkForce ''
ethers: files
group: files systemd
netgroup: files
networks: files
passwd: files systemd
protocols: files
publickey: files
rpc: files
services: files
shadow: files
hosts: files mymachines myhostname mdns4_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] dns
'';
}
Updated 23 September 2020