NixOps: Towards The Final Frontier
NixOps
(Nix Operations) is a program for deploying and provisioning multiple
NixOS
machines in a network or cloud environment. Let’s bid farewell to nix-rebuild (and many other cool tools) and say hello to nixops. This write up is by no means an
Consult your friendly NixOps documentation.
guide but rather a commentary on my
The current procedure of moving my entire external and personal infrastructure to NixOS.
nixops approach.
Bootstrapping Remote Machines
NixOps
NixOps version 2.0 supports non-root deployments.
ssh access to the root user, therefore you should bootstrap a remote machine using its configuration.nix. Power users will use NixOS
generators
to
Provision a specific NixOS configuration and convert between the many (too many) virtualization and cloud vendor formats.
up this process.
{
services.openssh = {
enable = true;
permitRootLogin = "prohibit-password";
passwordAuthentication = false;
challengeResponseAuthentication = false;
extraConfig = "Compression no";
};
users.users.root = {
openssh.authorizedKeys.keyFiles = [ ./ssh-key.pub ];
};
}
The user’s authorized_keys is
users.users.<name?> where <name?> is the actual user.
set using the option users.users.<name?>.openssh.authorizedKeys.keyFiles.
Do not use users.users.<name?>.openssh.authorizedKeys.keys to configure public keys as it replaces the authorized_keys that nixops sets on the remote machine.
Once nixops interfaces successfully with a remote machine it sets its public key, as well as a private
NixOps was first introduced as Charon [pdf].
called id_charon_vpn in the user’s .ssh folder. The remote machines are now ready to accept connections from nixops.
Directory Structure
Let’s take a look at my directory
Use a directory structure that fits your use case.
before creating a deployment using nixops.
├── configuration.nix
├── deployments.nixops
├── configuration
│ ├── heron.nix
│ └── pigeon.nix
├── hardware
│ ├── heron.nix
│ └── pigeon.nix
├── helpers
│ ├── extra-builtins.nix
│ └── wrappers
│ └── vault
├── keys
│ └── thedroneely.com.nix
├── mailers
├── packages
├── programs
│ └── nix.nix
├── servers
├── users
│ ├── pigeon.nix
│ ├── root.nix
│ └── thedro.nix
├── virtualizers
└── websites
└── thedroneely.com.nix
The deployment entry point is at configuration.nix. The machine specific configuration.nix and hardware.nix are in the configuration and hardware folders respectively. The deployment
The state file is rigid, however options for multiple state backends are in the works.
is tracked in deployments.nixops.
This directory contains helpers, program specific configurations, as well as files responsible for secrets management. This website for instance is realized from an entry point that imports and implements every dependency it needs to function.
{ config, lib, pkgs, ... }:
{
imports = [
../../keys/deployments/thedroneely.com.nix
../../servers/gitea/service.nix
../../servers/goaccess/service.nix
../../servers/isso/service.nix
../../servers/nginx/service.nix
../../servers/phpfpm/service.nix
../../servers/postgresql/service.nix
../../servers/rainloop/service.nix
];
}
NixOps Deployment and Commands
Populate the configuration entry point configuration.nix with a list of machines to provision in the deployment.
{
network.description = "nixos";
network.enableRollback = true;
heron = {
imports = [ ./hardware/heron.nix ./configuration/heron.nix ];
deployment.targetHost = "heron.local";
};
talon = {
imports = [ ./hardware/talon.nix ./configuration/talon.nix ];
deployment.targetHost = "talon.local";
};
tiger = {
imports = [ ./hardware/tiger.nix ./configuration/tiger.nix ];
deployment.targetHost = "tiger.local";
};
hound = {
imports = [ ./hardware/hound.nix ./configuration/hound.nix ];
deployment.targetHost = "hound.local";
};
pigeon = {
imports = [ ./hardware/pigeon.nix ./configuration/pigeon.nix ];
deployment.targetHost = "pigeon.local";
};
}
This entry point declares five machines to be operated on — each with their own hardware and configuration specific assets under the network nixos. Rollbacks are enabled for the nixops operator to switch generations in the case of a mishap.
Make a deployment using create by referencing the entry point file path and
The state file defaults to ~/.nixops/deployments.nixops
setting the location of the deployment state file using the -s flag.
nixops create -d nixos configuration.nix -s deployments.nixops
List all available
The output of nixops list depends on the last nixops deploy run.
configurations using the list argument.
$ nixops list -s deployments.nixops
+--------------------------------------+-------+-------------+------------+------+
| UUID | Name | Description | # Machines | Type |
+--------------------------------------+-------+-------------+------------+------+
| f56d276b-af54-11ea-b171-02422b1a33b6 | nixos | nixos | 5 | none |
+--------------------------------------+-------+-------------+------------+------+
the deployment nixos on all machines listed in the entry point configuration using the deploy argument.
nixops deploy -d nixos -s deployments.nixops
Run the deployment nixos but only on the machine with the cryptonym heron.
nixops deploy -d nixos -s deployments.nixops --include heron
Show the state for deployment nixos using the info argument. This command lists the status and current state of each machine.
$ nixops info -d nixos -s deployments.nixops
Network name: nixos
Network UUID: f56d276b-af54-11ea-b171-02422b1a33b6
Network description: nixos
Nix expressions: /nixos/configuration.nix
Nix profile: /nix/var/nix/profiles/per-user/thedro/nixops/f56d276b-af54-11ea-b171-02422b1a33b6
+--------+-----------------+------+----------------------------------------------------+------------+
| Name | Status | Type | Resource Id | IP address |
+--------+-----------------+------+----------------------------------------------------+------------+
| heron | Up / Outdated | none | nixops-f56d276b-af54-11ea-b171-02422b1a33b6-heron | |
| hound | Up / Outdated | none | nixops-f56d276b-af54-11ea-b171-02422b1a33b6-hound | |
| pigeon | Up / Up-to-date | none | nixops-f56d276b-af54-11ea-b171-02422b1a33b6-pigeon | |
| talon | Up / Outdated | none | nixops-f56d276b-af54-11ea-b171-02422b1a33b6-talon | |
| tiger | Up / Up-to-date | none | nixops-f56d276b-af54-11ea-b171-02422b1a33b6-tiger | |
+--------+-----------------+------+----------------------------------------------------+------------+
Examine the machines more closely with the check argument. This command provides extra information such as load averages and failed units.
$ nixops check -d nixos -s deployments.nixops --include tiger pigeon
Machines state:
+--------+--------+-----+-----------+----------+----------------+----------------------------------------+-------+
| Name | Exists | Up | Reachable | Disks OK | Load avg. | Units | Notes |
+--------+--------+-----+-----------+----------+----------------+----------------------------------------+-------+
| pigeon | Yes | Yes | Yes | N/A | 0.16 0.07 0.04 | proc-sys-fs-binfmt_misc.mount [failed] | |
| tiger | Yes | Yes | Yes | N/A | 0.11 0.09 0.03 | proc-sys-fs-binfmt_misc.mount [failed] | |
+--------+--------+-----+-----------+----------+----------------+----------------------------------------+-------+
List the generations for deployment nixos. The deployment fleet is currently on generation 203.
$ nixops list-generations -d nixos -s deployments.nixops
200 2020-06-26 00:28:10
201 2020-06-26 04:14:40
202 2020-06-26 04:28:12
203 2020-06-26 21:58:55 (current)
Delete a deployment with the delete argument but you must first remove the machine resources with destroy.
nixops destroy -d nixos -s deployments.nixops --include heron
nixops delete -d nixos -s deployments.nixops
NixOps Secrets Management
The nix store at /nix/store is world readable. Secrets entered directly into any nix configuration will be available to all users on the system. Avoid placing secrets into a nix configuration directly — as it will leak to unprivileged users. NixOps provides a powerful secrets management system in the form of password files.
Using a password file provides indirection and guards against writing secrets directly into your nix files. NixOps writes deployment keys to /run/keys. Users must be a part of the key
group
to access deployment keys on the system.
NixOps can set the permissions, owners, and groups of each key. The
Use any preferred program as a secrets storage.
secrets are sourced from a
Vault
server wrapper defined in builtins.extraBuiltins. This
blog post
gives an excellent rundown on this technique. A sample of the nix secrets configuration for this website is shown below.
{ config, ... }:
let user = "thedroneely"; in
{
deployment.keys = {
thedroneely_cockpit_api_token = {
user = user;
text = "${builtins.extraBuiltins.vault "thedroneely/thedroneely.com" "cockpit_api_token"}";
};
thedroneely_pgsql_database_password = {
user = user; group = "postgres"; permissions = "0640";
text = "${builtins.extraBuiltins.vault "thedroneely/thedroneely.com" "pgsql_database_password"}";
};
thedroneely_ssh_known_hosts = {
inherit user;
text = "${builtins.extraBuiltins.vault "thedroneely/thedroneely.com" "ssh_known_hosts"}";
};
};
}
Enable extraBuiltins by linking directly from the nix-plugins package on the nix operator host. Allow users in @wheel to become trusted users and wield this ultimate power.
{ pkgs, config, ... }:
{
nix.extraOptions = ''
trusted-users = root @wheel
plugin-files = ${pkgs.nix-plugins}/lib/nix/plugins/libnix-extra-builtins.so
'';
}
Create helpers/extra-builtins.nix and extend the builtins with custom functions. The vault function accepts a field and a path as arguments to an external wrapper.
{ exec, ... }:
{
vault = path: field: exec [ ./wrappers/vault field path ];
}
The wrapper returns the secret result from vault as a string. NixOps is now aware of keys available from a vault server. This is done at evaluation time. Environment variables are available to wrappers called from extraBuiltins.
#!/bin/sh -eu
printf '"' && vault kv get -field "$1" "$2" && printf '"'
Load the extraBuiltins with any nixops command by appending the --option flag with the file path. Vault keys are called using builtins.extraBuiltins.vault.
nixops reboot -d nixos -s deployments.nixops --include pigeon --option extra-builtins-file $PWD/nixos/helpers/extra-builtins.nix
The keys in /run/keys are ephemeral and never touch storage. Do
Executing a reboot with nixops on the same nixops host won't end well if secrets are set.
reboot the system through any other means except nixops. Your keys will be lost and any dependent service will
Keyless service failures can be avoided with some systemd voodoo.
Use nixops to remotely reboot a machine and its keys will be seeded back into position. Wonderful.
Upload deployment keys immediately without waiting on a deploy by using the send-keys argument.
nixops send-keys -d nixos -s deployments.nixops --include pigeon
NixOps Pinning
NixOps depends on the nix channel set on the host. This is a very interesting
Executing nixops on a machine with an uncertain channel is an assured gotcha.
target. Shoot it and lock the host’s channel by prefixing or
Currently using a nix shell until more of the code is read.
exporting a defined NIX_PATH to your nixops commands.
NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/2b417708c282d84316366f4125b00b29c49df10f.tar.gz
Check the system’s channel version with nix-instantiate if you prefer to use nix-channel directly.
nix-instantiate --eval -E '(import <nixpkgs> {}).lib.version'
NixOps and Nix Channel
Syncing the channel on the host and remote with nix-channel works out
To avoid silent drift when working with throw away nix-env or nix-shell environments.
for my use case.
The official channel list that nix-channel sources is at
release.nixos.org.
Create a “lock” file called versions.nix that contain all sources of truth.
{
# Channel list at https://releases.nixos.org/
"20.03" = { version = "20.03"; channel = "https://releases.nixos.org/nixos/20.03/nixos-20.03.3061.360e2af4f87"; };
"20.09" = { version = "20.09"; channel = "https://releases.nixos.org/nixos/20.09/nixos-20.09.1500.edb26126d98"; };
unstable = import (builtins.fetchTarball {
url= "https://github.com/NixOS/nixpkgs/archive/c308da1c4b14b0ffdfe9c2e2a948320124938220.tar.gz";
sha256 = "0ab9jhnlscxl1fl9rc7l978n26al1n77w32sr8gdbb11pfnhsizp"; }) {};
kernelLock = import (builtins.fetchTarball {
url= "https://github.com/NixOS/nixpkgs/archive/b0e3df2f8437767667bd041bb336e9d62a97ee81.tar.gz";
sha256 = "0d34k96l0gzsdpv14vnxdfslgk66gb0nsjz7qcqz1ykb0i7n3n07"; }) {};
intelKernelLock = import (builtins.fetchTarball {
url= "https://github.com/NixOS/nixpkgs/archive/f761c14fd2f198f64cc5483ebf9f83222f9214aa.tar.gz";
sha256 = "09w0n8qvldanab1m6ik507nl48aszam2a5ii3z2fvk72s66zmry7"; }) {};
}
Create a systemd service that uses restartTriggers to lazily watch versions.nix for attribute changes. The system will sync up as
This allows easier operation of nixos-rebuild if nixops is not available.
{ config, pkgs, ... }:
let channel = (import ../versions.nix)."${config.system.stateVersion}".channel; in
{
systemd.services.nix-channel-update = {
description = "Update Nix Channels";
wantedBy = [ "multi-user.target" ];
restartTriggers = [ channel ];
path = [ pkgs.shellcheck pkgs.nix ];
script = ''
set -euxo pipefail
shellcheck "$0" || exit 1
nix-channel --add "${channel}" nixos
nix-channel --update
'';
serviceConfig = { RemainAfterExit = "yes"; };
};
}
The option config.system.stateVersion is the system’s canonical version — query the current and default value with the command nixos-option system.stateVersion.
Updated 1 November 2020