+++
date = "2020-02-29T04:25:44+00:00"
publishdate = "2023-12-29T07:08:55+00:00"
title = "Kubernetes in a Linux Container"
slug = "kubernetes-in-a-linux-container"
author = "Thedro"
tags = ["lxc","k8s"]
type = "posts"
summary =  "Kubernetes is an open source container orchestration tool promising a catch all solution to issues that arise when deploying applications."
draft =  ""
syntax =  "1"
toc =  ""
updated =  ""
+++

{{< image source="/images/kubernetes-in-a-linux-container.png" title="Kubernetes Home Page" >}}
Kubernetes Home Page {{< /image >}}

[Kubernetes](https://kubernetes.io/) is an open source container orchestration
tool promising a catch all solution to issues that arise when deploying
applications. Kubernetes creates a framework that models how infrastructure and
services should interact.

The industry is trending in a direction where a
{{< sidenote mark="lot" set="right" >}}Tech marketing works its
wonders.{{< /sidenote>}} of people will be running `k8s` (Kubernetes) setups in
one form or the other. Let's exploit Linux Containers (`LXC`) as a mock
infrastructure and create a local `k8s` cluster to experiment.

## The Base System

We will use CentOS Linux {{< sidenote mark="version" set="left" >}}Pinning down
moving targets (versions) prevents you from losing your mind if something breaks
and things **do** break.{{< /sidenote>}} `8.1.1911` as our base system for a
cluster running `k8s` version `1.17`. You can download the CentOS version of
choice using `lxc-create`.

```shell
lxc-create -n k8s-centos8 -t download
```

In my case the `rootfs` for `lxc` is {{< sidenote mark="built" set="right" >}}In
ideal scenarios, transitioning between a `chroot`, container, virtual machine,
and bare metal should be easy.{{< /sidenote>}} with `docker` using a
[`Dockerfile`.](https://docs.docker.com/engine/reference/commandline/build/) We
[`docker export`](https://docs.docker.com/engine/reference/commandline/export/)
to obtain the container's filesystem. This build bootstraps `openssh-server` and
`python` for `ansible` to quickly setup the clusters.

```dockerfile
FROM centos:centos8.1.1911

RUN dnf -y update

RUN  dnf install -y --setopt=install_weak_deps=False \
  openssh-server network-scripts python3

RUN  systemctl enable network
RUN  systemctl disable kdump

RUN  usermod -p "$(uuidgen)" root

COPY centos8/files/ifcfg-ens3 /etc/sysconfig/network-scripts/ifcfg-ens3
COPY common/sshd_config /etc/ssh/sshd_config

RUN mkdir -p /root/.ssh
RUN chmod 700 /root/.ssh
COPY common/authorized_keys /root/.ssh/authorized_keys
RUN chmod 400 /root/.ssh/authorized_keys
```

## The Container Configuration

For Kubernetes to work inside the container, we'll need to tweak its `lxc`
configuration. This is a {{< sidenote mark="setup" set="right" >}}This should
work on Debian and Arch Linux based systems.{{< /sidenote>}} for the base
container `k8s-centos8` in `/var/lib/lxc/k8s-centos8/config`.

```ini {options="hl_lines=1-2 8-9 11-12 14-15 17-18 20-22"}
# Uncomment the following line to support nesting containers
lxc.include = /usr/share/lxc/config/nesting.conf

# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.arch = x86_64

# Allow access to all cgroups
lxc.cgroup.devices.allow = a

# Set proc mount to read and write
lxc.mount.auto = cgroup:mixed proc:rw sys:mixed

# No capability drops
lxc.cap.drop =

# Bind host kmsg for oom killer
lxc.mount.entry = /dev/kmsg dev/kmsg none defaults,bind,create=file

# Apparmor unconfined and allow nesting
lxc.apparmor.profile = unconfined
lxc.apparmor.allow_nesting = 1

# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:e0:c3:1d
lxc.rootfs.path = dir:/var/lib/lxc/k8s-centos8/rootfs
lxc.uts.name = k8s-centos8
```

## Kubernetes Cluster Installation

The base system is now ready. {{< sidenote mark="Create" set="left" >}}You can
add `-B overlayfs -s` onto the end of the `lxc-copy` command for super fast
cloning.{{< /sidenote>}} a master node and
{{< sidenote mark="three" set="right" >}}If you have a
[domain](https://angristan.xyz/2018/02/setup-network-bridge-lxc-net/) defined in
`/etc/default/lxc-net` then you can `ping` a running container by
name.{{< /sidenote>}} worker nodes by cloning the base system.

```shell
lxc-copy -n k8s-centos8 -N k8s-master
lxc-copy -n k8s-centos8 -N k8s-node1
lxc-copy -n k8s-centos8 -N k8s-node2
lxc-copy -n k8s-centos8 -N k8s-node3
```

Manually setting up each system is troublesome. We automate the rest using
[this](https://www.thedroneely.com/git/thedroneely/playbooks/tree/roles/k8s/main.yml)
`ansible` {{< sidenote mark="playbook." set="left" >}}This playbook is a
distillation of the standard installation documentation, with the addition of
the
[Kubernetes Dashboard.](https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/){{< /sidenote>}}

{{< image source="/images/ansible-kubernetes-lxc.gif" title="Running Kubernetes Playbook" >}}
Running Kubernetes Playbook {{< /image >}}