Running K3s in LXC on Proxmox with cgroup v2

Posted on 4 mins read

It has been a while since I’ve actively used Kubernetes and wanted to explore the evolution of tools like Helm and Tekton. I decided to deploy K3s, since I’ve had success with deploying it on resource-contrained Raspberry Pis in the past. I thought that this time it’d be convenient to have K3s running in a LXC container on Proxmox. This would allow for backing up the LXC container with the entire Kubernetes deployment. LXC containers also enable an efficient way to squeeze as much out of the machine resources as possible.

What is K3s?

K3s is a Kubernetes distro that advertises itself as a lightweight binary with a much smaller memory-footprint than traditional k8s. K3s is not a fork and seeks to remain as close to upstream as it can while maintaining a small set of patchest .

Configure Proxmox

This gist contains snippets and discussion on how to deploy K3s in LXC on proxmox. It mentions that bridge-nf-call-iptables should be loaded, but I did not understand the benefit of doing this.

Disable swap

There is an issue on Kubernetes regarding swap here. There claims to be support for swap in 1.22, but for now let’s disable it:

sysctl vm.swappiness=0
swapoff -a

It might be worth experimenting with swap enabled in the future to see how that might affect performance.

Enable IP Forwarding

To avoid IP Forwarding issues with Traefik, run the following on the host:

sudo sysctl net.ipv4.ip_forward=1
sudo sysctl net.ipv6.conf.all.forwarding=1
sudo sed -i 's/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/g' /etc/sysctl.conf
sudo sed -i 's/#net.ipv6.conf.all.forwarding=1/net.ipv6.conf.all.forwarding=1/g' /etc/sysctl.conf

Create LXC container

  • Uncheck unprivileged container
  • Use a LXC template (I chose a debian 11 template downloaded with pveam)
  • Set swap to 0 under memory
  • Create the container

Modify container config

Now back on the host run pct list to determine what VMID it was given.

Open /etc/pve/lxc/$VMID.conf and append:

lxc.apparmor.profile: unconfined
lxc.cap.drop:
lxc.mount.auto: "proc:rw sys:rw"
lxc.cgroup2.devices.allow: c 10:200 rwm

All of the above configurations are described in the manpages. Notice that cgroup2 is used since Proxmox VE 7.0 has switched to a pure cgroupv2 environment.

Thankfully cgroup v2 support has been enabled thanks to these contributions:

Enable shared host mounts

From within the container, run:

echo '#!/bin/sh -e
ln -s /dev/console /dev/kmsg
mount --make-rshared /' > /etc/rc.local
chmod +x /etc/rc.local
reboot

Install K3s

One of the simplest ways to install K3s on a remote host is to use k3sup. Ensure that you supply a valid CONTAINER_IP and choose the k3s-version you prefer. As of 2021/11, it is still defaulting to the 1.19 channel, so I overrode it to 1.22 for cgroup v2 support. See the published releases here.

ssh-copy-id root@$CONTAINER_IP
k3sup install --ip $CONTAINER_IP --user root --k3s-version v1.22.3+k3s1

If all goes well, you should see a path to the kubeconfig generated. I moved this into ~/.kube/config so that kubectl would read this by default.

Wrapping up

Installing K3s in LXC on proxmox works with a few tweaks to the default configuration. I later followed the Tekton’s Getting Started guide and was able to deploy in a few commands.

$ kubectl get all --namespace tekton-pipelines
NAME                                               READY   STATUS    RESTARTS      AGE
pod/tekton-pipelines-webhook-8566ff9b6b-6rnh8      1/1     Running   1 (50m ago)   12h
pod/tekton-dashboard-6bf858f977-qt4hr              1/1     Running   1 (50m ago)   11h
pod/tekton-pipelines-controller-69fd7498d8-f57m4   1/1     Running   1 (50m ago)   12h

NAME                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                              AGE
service/tekton-pipelines-controller   ClusterIP   10.43.44.245    <none>        9090/TCP,8080/TCP                    12h
service/tekton-pipelines-webhook      ClusterIP   10.43.183.242   <none>        9090/TCP,8008/TCP,443/TCP,8080/TCP   12h
service/tekton-dashboard              ClusterIP   10.43.87.97     <none>        9097/TCP                             11h

NAME                                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/tekton-pipelines-webhook      1/1     1            1           12h
deployment.apps/tekton-dashboard              1/1     1            1           11h
deployment.apps/tekton-pipelines-controller   1/1     1            1           12h

NAME                                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/tekton-pipelines-webhook-8566ff9b6b      1         1         1       12h
replicaset.apps/tekton-dashboard-6bf858f977              1         1         1       11h
replicaset.apps/tekton-pipelines-controller-69fd7498d8   1         1         1       12h

NAME                                                           REFERENCE                             TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/tekton-pipelines-webhook   Deployment/tekton-pipelines-webhook   9%/100%   1         5         1          12h

I think the next step would be to install Tailscale in the container so that I can easily access K3s from anywhere.

If I’m feeling adventurous, I might experiment with K3s rootless.