+++
date = "2020-01-16T00:16:15+00:00"
publishdate = "2023-12-29T07:08:55+00:00"
title = "Running NixOS in a Linux Container"
slug = "running-nixos-linux-containers"
author = "Thedro"
tags = ["linux"]
type = "posts"
summary =  "NixOS is sort of like the holy grail of configuration management, devops, and system administration."
draft =  ""
syntax =  "1"
toc =  ""
updated =  "2023-04-28"
+++

{{< mark >}}
**Note**: Rerunning the code in this article seems to suggest an ambiguity
between the container `init` and the `init` path `lxc` expects across multiple
NixOS versions. See [this
thread](https://github.com/nix-community/nixos-generators/issues/79#issuecomment-822073364)
for a possible fix.
{{< /mark >}}

- - -

![NixOS Home Page](/images/running-nixos-linux-containers.png "
  `NixOS` [Home Page](https://nixos.org/)"
)

[`NixOS`](https://nixos.org/) is seen as the holy grail of
{{< sidenote mark="configuration" set="left" >}}My favorite configuration stack
is [Ansible](https://en.wikipedia.org/wiki/Ansible_(software)) +
[Shell](https://en.wikipedia.org/wiki/Unix_shell) +
[Alpine](https://alpinelinux.org/about/){{< /sidenote>}} management and system
administration. It's what many call a
[purely functional distribution](https://nixos.org/~eelco/pubs/nixos-icfp2008-final.pdf)
`[pdf]`. The state is declarative --- entire systems, deployment artifacts, or
services can be built from a single file.

Let's try out `NixOS` in a 
{{< sidenote mark="Linux" set="left" >}}
This is assuming you have a working [`LXC` or `LXD`](https://linuxcontainers.org/)
setup with networking.
{{< /sidenote>}} 
Container (`LXC`). The `NixOS` [Hydra
builds](https://hydra.nixos.org/job/nixos/release-22.05/nixos.containerTarball.x86_64-linux)
([`aarch64`](https://hydra.nixos.org/job/nixos/release-22.05/nixos.containerTarball.aarch64-linux))
provide container images but they don't seem to
{{< sidenote mark="work" set="right" >}}
Symlinking issues when extracting the `rootfs` (file system).
{{< /sidenote>}} 
for this use case.

## Bootstrapping

The best way to get a `NixOS` root file system is to bootstrap using
[`nixos-generate`](https://github.com/nix-community/nixos-generators#readme).
This will require a live `NixOS` environment or
[installing](https://nixos.org/download.html#download-nix) `nix` onto your Linux
distribution. Download the latest
[`NixOS` minimal image](https://nixos.org/download.html#download-nixos) and boot
into the live environment using preferred means.

{{< sidenote mark="Inside" set="left" >}}
Even at this granularity it's possible for this process to break in the future
--- chaos demands it. For better reproducibility consider
[pinning or locking](https://nixos.wiki/wiki/FAQ/Pinning_Nixpkgs) the channel
version in a `nix-shell` or
[`flake` configuration](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake.html).
{{< /sidenote>}} 
the live environment or after installing `nix`, bootstrap a Linux
container's `rootfs` using `nixos-generate`.

```shell { caption="Using the default configuration.nix" }
nixos-generate --format lxc
```

```shell { caption="Using the specified configuration.nix" }
nixos-generate --format lxc --configuration configuration.nix
```

If the command
[`nixos-generate`](https://search.nixos.org/packages?channel=unstable&query=nixos-generators)
is not found --- install `nixos-generators` using
[`nix-env`](https://nixos.org/manual/nix/unstable/command-ref/nix-env.html) or
[`nix-shell`](https://nixos.org/manual/nix/unstable/command-ref/nix-shell.html).

```shell { caption="Environment installation" }
nix-env -iA nixos.nixos-generators
```

```shell { caption="Shell installation" }
nix-shell --packages nixos-generators
```

Once completed, this command will print a hashed path like
`/nix/store/3ipfpzhk4dllwhcnldsbfldi1favyxsm-tarball/nix-support/hydra-build-products`.
This file contains the location of the `rootfs` archive.

```shell
$ cat /nix/store/3ipfpzhk4dllwhcnldsbfldi1favyxsm-tarball/nix-support/hydra-build-products
file system-tarball /nix/store/3ipfpzhk4dllwhcnldsbfldi1favyxsm-tarball/tarball/nixos-system-x86_64-linux.tar.xz
```

Save the generated archive `nixos-system-x86_64-linux.tar.xz` and exit the live
environment.

## Setup the Linux Container

Now that we have a clean `rootfs` archive. Create an empty `linux` container and
`rootfs` directory. Extract `nixos-system-x86_64-linux.tar.xz` to `rootfs`.

```shell
lxc-create -n nixos -t none
cd /var/lib/lxc/nixos
mkdir rootfs
tar -xvf nixos-system-x86_64-linux.tar.xz -C rootfs/
```

Let's massage our 
{{< sidenote mark="configuration" set="right" >}}
This has been
tested on `Debian` and `Arch Linux.`
{{< /sidenote>}} 
for `NixOS` at `/var/lib/lxc/nixos/config`. The entry point is `/sbin/init`.

```ini { options="hl_lines=16" }
# Distribution configuration
lxc.arch = linux64
lxc.include = /usr/share/lxc/config/common.conf

# Container specific configuration
lxc.uts.name = nixos
lxc.rootfs.path = dir:/var/lib/lxc/nixos/rootfs

# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:01:77:76

# NixOS configuration
lxc.init.cmd = /sbin/init
```

## Setup and Run NixOS

Now
{{< sideimage mark="start" set="right" source="/images/running-nixos-linux-containers.gif" >}}
`NixOS` container setup
{{< /sideimage >}} the container and enter the shell using the command
`lxc-attach`.

```shell
lxc-start -n nixos
lxc-attach -n nixos
```

The default `/etc/nixos/configuration.nix` is empty. Tell `NixOS` that we are a
container by adding `boot.isContainer = true`.

```nix
{ config, pkgs, ... }:

{
  imports = [ ];
  boot.isContainer = true;
}
```

Now run a 
{{< sidenote mark="rebuild" set="left" >}}
Firewall errors about `iptables` filters for `IPv6` mean you either run an old
kernel or you need to `sudo`
[modprobe](https://man.archlinux.org/man/modprobe.8) `ip6table_filter`
{{< /sidenote>}} 
of the system. The upgrade flag may be necessary on the first rebuild to
download the channels.

```shell
 nixos-rebuild switch --upgrade
```

We now have a base `NixOS` system in a Linux container. You can build
reproducible desktops, servers, deployment artifacts --- anything, using your
own `configuration.nix` and consulting the
[`NixOS` options index.](https://nixos.org/nixos/options.html) This approach is
great because when you make something it just works --- all the time.