mirror of
https://github.com/davegallant/davegallant.github.io.git
synced 2025-08-14 12:20:19 +00:00
271 lines
18 KiB
HTML
271 lines
18 KiB
HTML
<!doctype html><html lang=en dir=auto><head>
|
|
<meta charset=utf-8>
|
|
<meta name=viewport content="width=device-width,initial-scale=1">
|
|
<meta name=description content>
|
|
<meta name=theme-color content="#8979b3">
|
|
<meta property="og:title" content="Running K3s in LXC on Proxmox • davegallant">
|
|
<meta property="og:description" content>
|
|
<meta property="og:url" content="/blog/2021/11/14/running-k3s-in-lxc-on-proxmox/">
|
|
<meta property="og:site_name" content="davegallant">
|
|
<meta property="og:type" content="article"><meta property="article:section" content="post"><meta property="article:tag" content="k3s"><meta property="article:tag" content="proxmox"><meta property="article:tag" content="lxc"><meta property="article:published_time" content="2021-11-14T10:07:03-05:00"><meta property="article:modified_time" content="2021-11-14T10:07:03-05:00"><meta name=twitter:card content="summary">
|
|
<meta name=generator content="Hugo 0.88.0">
|
|
<title>Running K3s in LXC on Proxmox • davegallant</title>
|
|
<link rel=canonical href=/blog/2021/11/14/running-k3s-in-lxc-on-proxmox/>
|
|
<link rel=icon href=/favicon.ico>
|
|
<link rel=stylesheet href=/assets/css/main.ab98e12b.css><link rel=stylesheet href=/css/custom.css><style>:root{--color-accent:#8979b3}</style>
|
|
<script type=application/javascript>var doNotTrack=!1;doNotTrack||(window.ga=window.ga||function(){(ga.q=ga.q||[]).push(arguments)},ga.l=+new Date,ga('create','UA-98710982-2','auto'),ga('send','pageview'))</script>
|
|
<script async src=https://www.google-analytics.com/analytics.js></script>
|
|
</head>
|
|
<body class="page type-post has-sidebar">
|
|
<div class=site><div id=sidebar class=sidebar>
|
|
<a class=screen-reader-text href=#main-menu>Skip to Main Menu</a>
|
|
<div class=container><section class="widget widget-about sep-after">
|
|
<header>
|
|
<div class=logo>
|
|
<a href=/>
|
|
<img src=/images/logo.png>
|
|
</a>
|
|
</div>
|
|
<h2 class="title site-title">
|
|
<a href=/>
|
|
davegallant
|
|
</a>
|
|
</h2>
|
|
<div class=desc>
|
|
personal blog
|
|
</div>
|
|
</header>
|
|
</section>
|
|
<section class="widget widget-taxonomy_cloud sep-after">
|
|
<header>
|
|
<h4 class="title widget-title">Tags</h4>
|
|
</header>
|
|
<div class="container list-container">
|
|
<ul class="list taxonomy-cloud"><li>
|
|
<a href=/tags/adguard/ style=font-size:1em>adguard</a>
|
|
</li><li>
|
|
<a href=/tags/aws/ style=font-size:1em>aws</a>
|
|
</li><li>
|
|
<a href=/tags/aws-vault/ style=font-size:1em>aws-vault</a>
|
|
</li><li>
|
|
<a href=/tags/containers/ style=font-size:1em>containers</a>
|
|
</li><li>
|
|
<a href=/tags/docker/ style=font-size:1em>docker</a>
|
|
</li><li>
|
|
<a href=/tags/dotfiles/ style=font-size:1em>dotfiles</a>
|
|
</li><li>
|
|
<a href=/tags/grafana/ style=font-size:1em>grafana</a>
|
|
</li><li>
|
|
<a href=/tags/home-manager/ style=font-size:1em>home-manager</a>
|
|
</li><li>
|
|
<a href=/tags/homelab/ style=font-size:1em>homelab</a>
|
|
</li><li>
|
|
<a href=/tags/jellyfin/ style=font-size:1em>jellyfin</a>
|
|
</li><li>
|
|
<a href=/tags/k3s/ style=font-size:1em>k3s</a>
|
|
</li><li>
|
|
<a href=/tags/linux/ style=font-size:1em>linux</a>
|
|
</li><li>
|
|
<a href=/tags/lxc/ style=font-size:1em>lxc</a>
|
|
</li><li>
|
|
<a href=/tags/netdata/ style=font-size:1em>netdata</a>
|
|
</li><li>
|
|
<a href=/tags/nix/ style=font-size:1em>nix</a>
|
|
</li><li>
|
|
<a href=/tags/pihole/ style=font-size:1em>pihole</a>
|
|
</li><li>
|
|
<a href=/tags/plex/ style=font-size:1em>plex</a>
|
|
</li><li>
|
|
<a href=/tags/podman/ style=font-size:1em>podman</a>
|
|
</li><li>
|
|
<a href=/tags/proxmox/ style=font-size:1em>proxmox</a>
|
|
</li><li>
|
|
<a href=/tags/python/ style=font-size:2em>python</a>
|
|
</li><li>
|
|
<a href=/tags/security/ style=font-size:1em>security</a>
|
|
</li><li>
|
|
<a href=/tags/tailscale/ style=font-size:1em>tailscale</a>
|
|
</li><li>
|
|
<a href=/tags/virtualization/ style=font-size:1em>virtualization</a>
|
|
</li><li>
|
|
<a href=/tags/vpn/ style=font-size:1em>vpn</a>
|
|
</li></ul>
|
|
</div>
|
|
</section>
|
|
<section class="widget widget-social_menu sep-after"><nav aria-label="Social Menu">
|
|
<ul><li>
|
|
<a href=https://github.com/davegallant target=_blank rel="noopener me">
|
|
<span class=screen-reader-text>Open Github account in new tab</span><svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><path d="M9 19c-5 1.5-5-2.5-7-3m14 6v-3.87a3.37 3.37.0 00-.94-2.61c3.14-.35 6.44-1.54 6.44-7A5.44 5.44.0 0020 4.77 5.07 5.07.0 0019.91 1S18.73.65 16 2.48a13.38 13.38.0 00-7 0C6.27.65 5.09 1 5.09 1A5.07 5.07.0 005 4.77 5.44 5.44.0 003.5 8.55c0 5.42 3.3 6.61 6.44 7A3.37 3.37.0 009 18.13V22"/></svg>
|
|
</a>
|
|
</li><li>
|
|
<a href=https://twitter.com/dave_gallant_ target=_blank rel="noopener me">
|
|
<span class=screen-reader-text>Open Twitter account in new tab</span><svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><path d="M23 3a10.9 10.9.0 01-3.14 1.53 4.48 4.48.0 00-7.86 3v1A10.66 10.66.0 013 4s-4 9 5 13a11.64 11.64.0 01-7 2c9 5 20 0 20-11.5a4.5 4.5.0 00-.08-.83A7.72 7.72.0 0023 3z"/></svg>
|
|
</a>
|
|
</li><li>
|
|
<a href=mailto:davegallant@gmail.com target=_blank rel="noopener me">
|
|
<span class=screen-reader-text>Contact via Email</span><svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><path d="M4 4h16c1.1.0 2 .9 2 2v12c0 1.1-.9 2-2 2H4c-1.1.0-2-.9-2-2V6c0-1.1.9-2 2-2z"/><polyline points="22,6 12,13 2,6"/></svg>
|
|
</a>
|
|
</li><li>
|
|
<a href=https://linkedin.com/in/dave-gallant target=_blank rel="noopener me">
|
|
<span class=screen-reader-text>Open Linkedin account in new tab</span><svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><path d="M16 8a6 6 0 016 6v7h-4v-7a2 2 0 00-2-2 2 2 0 00-2 2v7h-4v-7a6 6 0 016-6z"/><rect x="2" y="9" width="4" height="12"/><circle cx="4" cy="4" r="2"/></svg>
|
|
</a>
|
|
</li></ul>
|
|
</nav>
|
|
</section></div>
|
|
<div class=sidebar-overlay></div>
|
|
</div><div class=main><nav id=main-menu class="menu main-menu" aria-label="Main Menu">
|
|
<div class=container>
|
|
<a class=screen-reader-text href=#content>Skip to Content</a>
|
|
<button id=sidebar-toggler class=sidebar-toggler aria-controls=sidebar>
|
|
<span class=screen-reader-text>Toggle Sidebar</span>
|
|
<span class=open><svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><line x1="3" y1="12" x2="21" y2="12"/><line x1="3" y1="6" x2="21" y2="6"/><line x1="3" y1="18" x2="21" y2="18"/></svg>
|
|
</span>
|
|
<span class=close><svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
|
|
</span>
|
|
</button>
|
|
<ul><li class=item>
|
|
<a href=/>Home</a>
|
|
</li><li class=item>
|
|
<a href=/about/>About</a>
|
|
</li><li class=item>
|
|
<a href=/index.xml>RSS</a>
|
|
</li></ul>
|
|
</div>
|
|
</nav><div class=header-widgets>
|
|
<div class=container></div>
|
|
</div>
|
|
<header id=header class="header site-header">
|
|
<div class="container sep-after">
|
|
</div>
|
|
</header>
|
|
<main id=content>
|
|
<article lang=en class=entry>
|
|
<header class="header entry-header">
|
|
<div class="container sep-after">
|
|
<div class=header-info>
|
|
<h1 class=title>Running K3s in LXC on Proxmox</h1>
|
|
</div>
|
|
<div class=entry-meta>
|
|
<span class=posted-on><svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><rect x="3" y="4" width="18" height="18" rx="2" ry="2"/><line x1="16" y1="2" x2="16" y2="6"/><line x1="8" y1="2" x2="8" y2="6"/><line x1="3" y1="10" x2="21" y2="10"/></svg>
|
|
<span class=screen-reader-text>Posted on </span>
|
|
<time class=entry-date datetime=2021-11-14T10:07:03-05:00>2021, Nov 14</time>
|
|
</span>
|
|
<span class=reading-time><svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><circle cx="12" cy="12" r="10"/><polyline points="12 6 12 12 15 15"/></svg>
|
|
4 mins read
|
|
</span>
|
|
</div>
|
|
</div>
|
|
</header>
|
|
<div class="container entry-content">
|
|
<p>It has been a while since I’ve actively used Kubernetes and wanted to explore the evolution of tools like <a href=https://helm.sh>Helm</a> and <a href=https://tekton.dev>Tekton</a>. I decided to deploy <a href=https://k3s.io>K3s</a>, since I’ve had success with deploying it on resource-contrained Raspberry Pis in the past. I thought that this time it’d be convenient to have K3s running in a LXC container on Proxmox. This would allow for backing up the LXC container with the entire Kubernetes deployment. LXC containers provide an efficient way to squeeze as much out of the machine resources as possible.</p>
|
|
<h2 id=what-is-k3s>What is K3s?</h2>
|
|
<p>K3s is a Kubernetes distro that advertises itself as a lightweight binary with a much smaller memory-footprint than traditional k8s. K3s is not a fork of k8s as it seeks to remain as close to upstream as it possibly can.</p>
|
|
<h2 id=configure-proxmox>Configure Proxmox</h2>
|
|
<p>This <a href=https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185>gist</a> contains snippets and discussion on how to deploy K3s in LXC on proxmox. It mentions that <code>bridge-nf-call-iptables</code> should be loaded, but I did not understand the benefit of doing this.</p>
|
|
<h2 id=disable-swap>Disable swap</h2>
|
|
<p>There is an issue on Kubernetes regarding swap <a href=https://github.com/kubernetes/kubernetes/issues/53533>here</a>. There claims to be support for swap in 1.22, but for now let’s disable it:</p>
|
|
<pre tabindex=0><code>sysctl vm.swappiness=0
|
|
swapoff -a
|
|
</code></pre><p>It might be worth experimenting with swap enabled in the future to see how that might affect performance.</p>
|
|
<h3 id=enable-ip-forwarding>Enable IP Forwarding</h3>
|
|
<p>To avoid IP Forwarding issues with Traefik, run the following on the host:</p>
|
|
<div class=highlight><pre tabindex=0 style=background-color:#f0f3f3;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sh data-lang=sh>sudo sysctl net.ipv4.ip_forward<span style=color:#555>=</span><span style=color:#f60>1</span>
|
|
sudo sysctl net.ipv6.conf.all.forwarding<span style=color:#555>=</span><span style=color:#f60>1</span>
|
|
sudo sed -i <span style=color:#c30>'s/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/g'</span> /etc/sysctl.conf
|
|
sudo sed -i <span style=color:#c30>'s/#net.ipv6.conf.all.forwarding=1/net.ipv6.conf.all.forwarding=1/g'</span> /etc/sysctl.conf
|
|
</code></pre></div><h2 id=create-lxc-container>Create LXC container</h2>
|
|
<p>Create an LXC container in the Proxmox interface as you normally would. Remember to:</p>
|
|
<ul>
|
|
<li>Uncheck <code>unprivileged container</code></li>
|
|
<li>Use a LXC template (I chose a debian 11 template downloaded with <a href=https://pve.proxmox.com/wiki/Linux_Container#Create_container>pveam</a>)</li>
|
|
<li>In memory, set swap to 0</li>
|
|
<li>Create and start the container</li>
|
|
</ul>
|
|
<h3 id=modify-container-config>Modify container config</h3>
|
|
<p>Now back on the host run <code>pct list</code> to determine what VMID it was given.</p>
|
|
<p>Open <code>/etc/pve/lxc/$VMID.conf</code> and append:</p>
|
|
<div class=highlight><pre tabindex=0 style=background-color:#f0f3f3;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sh data-lang=sh>lxc.apparmor.profile: unconfined
|
|
lxc.cap.drop:
|
|
lxc.mount.auto: <span style=color:#c30>"proc:rw sys:rw"</span>
|
|
lxc.cgroup2.devices.allow: c 10:200 rwm
|
|
</code></pre></div><p>All of the above configurations are described in the <a href=https://linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html>manpages</a>.
|
|
Notice that <code>cgroup2</code> is used since Proxmox VE 7.0 has switched to a <a href=https://pve.proxmox.com/pve-docs/chapter-pct.html#pct_cgroup>pure cgroupv2 environment</a>.</p>
|
|
<p>Thankfully cgroup v2 support has been enabled with these contributions:</p>
|
|
<ul>
|
|
<li><a href=https://github.com/k3s-io/k3s/pull/2584>https://github.com/k3s-io/k3s/pull/2584</a></li>
|
|
<li><a href=https://github.com/k3s-io/k3s/pull/2844>https://github.com/k3s-io/k3s/pull/2844</a></li>
|
|
</ul>
|
|
<h2 id=enable-shared-host-mounts>Enable shared host mounts</h2>
|
|
<p>From within the container, run:</p>
|
|
<div class=highlight><pre tabindex=0 style=background-color:#f0f3f3;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sh data-lang=sh><span style=color:#366>echo</span> <span style=color:#c30>'#!/bin/sh -e
|
|
</span><span style=color:#c30>ln -s /dev/console /dev/kmsg
|
|
</span><span style=color:#c30>mount --make-rshared /'</span> > /etc/rc.local
|
|
chmod +x /etc/rc.local
|
|
reboot
|
|
</code></pre></div><h2 id=install-k3s>Install K3s</h2>
|
|
<p>One of the simplest ways to install K3s on a remote host is to use <a href=https://github.com/alexellis/k3sup>k3sup</a>.
|
|
Ensure that you supply a valid <code>CONTAINER_IP</code> and choose the <code>k3s-version</code> you prefer.
|
|
As of 2021/11, it is still defaulting to the 1.19 channel, so I overrode it to 1.22 for cgroup v2 support. See the published releases <a href=https://github.com/k3s-io/k3s/releases>here</a>.</p>
|
|
<div class=highlight><pre tabindex=0 style=background-color:#f0f3f3;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sh data-lang=sh>ssh-copy-id root@<span style=color:#033>$CONTAINER_IP</span>
|
|
k3sup install --ip <span style=color:#033>$CONTAINER_IP</span> --user root --k3s-version v1.22.3+k3s1
|
|
</code></pre></div><p>If all goes well, you should see a path to the <code>kubeconfig</code> generated. I moved this into <code>~/.kube/config</code> so that kubectl would read this by default.</p>
|
|
<h2 id=wrapping-up>Wrapping up</h2>
|
|
<p>Installing K3s in LXC on proxmox works with a few tweaks to the default configuration. I later followed the Tekton’s <a href=https://tekton.dev/docs/getting-started/>Getting Started</a> guide and was able to deploy it in a few commands.</p>
|
|
<pre tabindex=0><code class=language-console data-lang=console>$ kubectl get all --namespace tekton-pipelines
|
|
NAME READY STATUS RESTARTS AGE
|
|
pod/tekton-pipelines-webhook-8566ff9b6b-6rnh8 1/1 Running 1 (50m ago) 12h
|
|
pod/tekton-dashboard-6bf858f977-qt4hr 1/1 Running 1 (50m ago) 11h
|
|
pod/tekton-pipelines-controller-69fd7498d8-f57m4 1/1 Running 1 (50m ago) 12h
|
|
|
|
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
|
service/tekton-pipelines-controller ClusterIP 10.43.44.245 <none> 9090/TCP,8080/TCP 12h
|
|
service/tekton-pipelines-webhook ClusterIP 10.43.183.242 <none> 9090/TCP,8008/TCP,443/TCP,8080/TCP 12h
|
|
service/tekton-dashboard ClusterIP 10.43.87.97 <none> 9097/TCP 11h
|
|
|
|
NAME READY UP-TO-DATE AVAILABLE AGE
|
|
deployment.apps/tekton-pipelines-webhook 1/1 1 1 12h
|
|
deployment.apps/tekton-dashboard 1/1 1 1 11h
|
|
deployment.apps/tekton-pipelines-controller 1/1 1 1 12h
|
|
|
|
NAME DESIRED CURRENT READY AGE
|
|
replicaset.apps/tekton-pipelines-webhook-8566ff9b6b 1 1 1 12h
|
|
replicaset.apps/tekton-dashboard-6bf858f977 1 1 1 11h
|
|
replicaset.apps/tekton-pipelines-controller-69fd7498d8 1 1 1 12h
|
|
|
|
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
|
|
horizontalpodautoscaler.autoscaling/tekton-pipelines-webhook Deployment/tekton-pipelines-webhook 9%/100% 1 5 1 12h
|
|
</code></pre><p>I think the next step would be to install Tailscale in the container so that I can easily access K3s from anywhere.</p>
|
|
<p>If I’m feeling adventurous, I might experiment with <a href=https://rancher.com/docs/k3s/latest/en/advanced/#running-k3s-with-rootless-mode-experimental>K3s rootless</a>.</p>
|
|
</div>
|
|
<footer class=entry-footer>
|
|
<div class="container sep-before"><div class=tags><svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><path d="M20.59 13.41l-7.17 7.17a2 2 0 01-2.83.0L2 12V2H12l8.59 8.59a2 2 0 010 2.82z"/><line x1="7" y1="7" x2="7" y2="7"/></svg>
|
|
<span class=screen-reader-text>Tags: </span><a class=tag href=/tags/k3s/>k3s</a>, <a class=tag href=/tags/proxmox/>proxmox</a>, <a class=tag href=/tags/lxc/>lxc</a></div>
|
|
</div>
|
|
</footer>
|
|
</article>
|
|
<nav class=entry-nav>
|
|
<div class=container><div class="prev-entry sep-before">
|
|
<a href=/blog/2021/10/11/replacing-docker-with-podman-on-macos-and-linux/>
|
|
<span aria-hidden=true><svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><line x1="20" y1="12" x2="4" y2="12"/><polyline points="10 18 4 12 10 6"/></svg>
|
|
Previous</span>
|
|
<span class=screen-reader-text>Previous post: </span>Replacing docker with podman on macOS (and Linux)</a>
|
|
</div></div>
|
|
</nav>
|
|
<section id=comments class=comments>
|
|
<div class="container sep-before">
|
|
<div class=comments-area><script src=https://utteranc.es/client.js repo=davegallant/davegallant.github.io issue-term=pathname theme=github-light crossorigin=anonymous async></script>
|
|
</div>
|
|
</div>
|
|
</section>
|
|
</main>
|
|
<footer id=footer class=footer>
|
|
<div class="container sep-before"><div class=copyright>
|
|
<p> © 2020-2021 Dave Gallant </p>
|
|
</div>
|
|
</div>
|
|
</footer>
|
|
</div>
|
|
</div><script>window.__assets_js_src="/assets/js/"</script>
|
|
<script src=/assets/js/main.c3bcf2df.js></script>
|
|
</body>
|
|
</html> |