dave/site
1
0
Fork 0
mirror of https://github.com/davegallant/davegallant.github.io.git synced 2025-08-14 12:20:19 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
dd6fce9f8c519218425d724a5e7d17f6300ddbd2
site/blog/2021/11/14/running-k3s-in-lxc-on-proxmox/index.html
davegallant dd6fce9f8c deploy: 986518eeffe71f7b33f7b508f53195a1ef21a3ce
2022-03-14 02:20:30 +00:00

286 lines
19 KiB
HTML
Raw Blame History

<!doctype html><html lang=en dir=auto><head>
<meta charset=utf-8>
<meta name=viewport content="width=device-width,initial-scale=1">
<meta name=description content>
<meta name=theme-color content="#8979b3">
<meta property="og:title" content="Running K3s in LXC on Proxmox • davegallant">
<meta property="og:description" content>
<meta property="og:url" content="/blog/2021/11/14/running-k3s-in-lxc-on-proxmox/">
<meta property="og:site_name" content="davegallant">
<meta property="og:type" content="article"><meta property="article:section" content="post"><meta property="article:tag" content="k3s"><meta property="article:tag" content="proxmox"><meta property="article:tag" content="lxc"><meta property="article:published_time" content="2021-11-14T10:07:03-05:00"><meta property="article:modified_time" content="2021-11-14T10:07:03-05:00"><meta name=twitter:card content="summary">
<meta name=generator content="Hugo 0.89.4">
<title>Running K3s in LXC on Proxmox • davegallant</title>
<link rel=canonical href=/blog/2021/11/14/running-k3s-in-lxc-on-proxmox/>
<link rel=icon href=/favicon.ico>
<link rel=stylesheet href=/assets/css/main.ab98e12b.css><link rel=stylesheet href=/css/custom.css><style>:root{--color-accent:#8979b3}</style>
<script type=application/javascript>var doNotTrack=!1;doNotTrack||(window.ga=window.ga||function(){(ga.q=ga.q||[]).push(arguments)},ga.l=+new Date,ga('create','UA-98710982-2','auto'),ga('send','pageview'))</script>
<script async src=https://www.google-analytics.com/analytics.js></script>
</head>
<body class="page type-post has-sidebar">
<div class=site><div id=sidebar class=sidebar>
<a class=screen-reader-text href=#main-menu>Skip to Main Menu</a>
<div class=container><section class="widget widget-about sep-after">
<header>
<div class=logo>
<a href=/>
<img src=/images/logo.png>
</a>
</div>
<h2 class="title site-title">
<a href=/>
davegallant
</a>
</h2>
<div class=desc>
personal blog
</div>
</header>
</section>
<section class="widget widget-taxonomy_cloud sep-after">
<header>
<h4 class="title widget-title">Tags</h4>
</header>
<div class="container list-container">
<ul class="list taxonomy-cloud"><li>
<a href=/tags/adguard/ style=font-size:1em>adguard</a>
</li><li>
<a href=/tags/aws/ style=font-size:1em>aws</a>
</li><li>
<a href=/tags/aws-vault/ style=font-size:1em>aws-vault</a>
</li><li>
<a href=/tags/backup/ style=font-size:1em>backup</a>
</li><li>
<a href=/tags/containers/ style=font-size:1em>containers</a>
</li><li>
<a href=/tags/degoogle/ style=font-size:1em>degoogle</a>
</li><li>
<a href=/tags/docker/ style=font-size:1em>docker</a>
</li><li>
<a href=/tags/dotfiles/ style=font-size:1em>dotfiles</a>
</li><li>
<a href=/tags/gmail/ style=font-size:1em>gmail</a>
</li><li>
<a href=/tags/grafana/ style=font-size:1em>grafana</a>
</li><li>
<a href=/tags/home-manager/ style=font-size:1em>home-manager</a>
</li><li>
<a href=/tags/homelab/ style=font-size:1em>homelab</a>
</li><li>
<a href=/tags/jellyfin/ style=font-size:1em>jellyfin</a>
</li><li>
<a href=/tags/k3s/ style=font-size:1em>k3s</a>
</li><li>
<a href=/tags/linux/ style=font-size:1em>linux</a>
</li><li>
<a href=/tags/lxc/ style=font-size:1em>lxc</a>
</li><li>
<a href=/tags/netdata/ style=font-size:1em>netdata</a>
</li><li>
<a href=/tags/nix/ style=font-size:1em>nix</a>
</li><li>
<a href=/tags/pihole/ style=font-size:1em>pihole</a>
</li><li>
<a href=/tags/plex/ style=font-size:1em>plex</a>
</li><li>
<a href=/tags/podman/ style=font-size:1em>podman</a>
</li><li>
<a href=/tags/proxmox/ style=font-size:1em>proxmox</a>
</li><li>
<a href=/tags/python/ style=font-size:2em>python</a>
</li><li>
<a href=/tags/ransomware/ style=font-size:1em>ransomware</a>
</li><li>
<a href=/tags/security/ style=font-size:1em>security</a>
</li><li>
<a href=/tags/synology/ style=font-size:1em>synology</a>
</li><li>
<a href=/tags/tailscale/ style=font-size:1em>tailscale</a>
</li><li>
<a href=/tags/virtualization/ style=font-size:1em>virtualization</a>
</li><li>
<a href=/tags/vpn/ style=font-size:1em>vpn</a>
</li></ul>
</div>
</section>
<section class="widget widget-social_menu sep-after"><nav aria-label="Social Menu">
<ul><li>
<a href=https://github.com/davegallant target=_blank rel="noopener me">
<span class=screen-reader-text>Open Github account in new tab</span><svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><path d="M9 19c-5 1.5-5-2.5-7-3m14 6v-3.87a3.37 3.37.0 00-.94-2.61c3.14-.35 6.44-1.54 6.44-7A5.44 5.44.0 0020 4.77 5.07 5.07.0 0019.91 1S18.73.65 16 2.48a13.38 13.38.0 00-7 0C6.27.65 5.09 1 5.09 1A5.07 5.07.0 005 4.77 5.44 5.44.0 003.5 8.55c0 5.42 3.3 6.61 6.44 7A3.37 3.37.0 009 18.13V22"/></svg>
</a>
</li><li>
<a href=https://twitter.com/dave_gallant_ target=_blank rel="noopener me">
<span class=screen-reader-text>Open Twitter account in new tab</span><svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><path d="M23 3a10.9 10.9.0 01-3.14 1.53 4.48 4.48.0 00-7.86 3v1A10.66 10.66.0 013 4s-4 9 5 13a11.64 11.64.0 01-7 2c9 5 20 0 20-11.5a4.5 4.5.0 00-.08-.83A7.72 7.72.0 0023 3z"/></svg>
</a>
</li><li>
<a href=mailto:davegallant@gmail.com target=_blank rel="noopener me">
<span class=screen-reader-text>Contact via Email</span><svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><path d="M4 4h16c1.1.0 2 .9 2 2v12c0 1.1-.9 2-2 2H4c-1.1.0-2-.9-2-2V6c0-1.1.9-2 2-2z"/><polyline points="22,6 12,13 2,6"/></svg>
</a>
</li><li>
<a href=https://linkedin.com/in/dave-gallant target=_blank rel="noopener me">
<span class=screen-reader-text>Open Linkedin account in new tab</span><svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><path d="M16 8a6 6 0 016 6v7h-4v-7a2 2 0 00-2-2 2 2 0 00-2 2v7h-4v-7a6 6 0 016-6z"/><rect x="2" y="9" width="4" height="12"/><circle cx="4" cy="4" r="2"/></svg>
</a>
</li></ul>
</nav>
</section></div>
<div class=sidebar-overlay></div>
</div><div class=main><nav id=main-menu class="menu main-menu" aria-label="Main Menu">
<div class=container>
<a class=screen-reader-text href=#content>Skip to Content</a>
<button id=sidebar-toggler class=sidebar-toggler aria-controls=sidebar>
<span class=screen-reader-text>Toggle Sidebar</span>
<span class=open><svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><line x1="3" y1="12" x2="21" y2="12"/><line x1="3" y1="6" x2="21" y2="6"/><line x1="3" y1="18" x2="21" y2="18"/></svg>
</span>
<span class=close><svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
</span>
</button>
<ul><li class=item>
<a href=/>Home</a>
</li><li class=item>
<a href=/about/>About</a>
</li><li class=item>
<a href=/index.xml>RSS</a>
</li></ul>
</div>
</nav><div class=header-widgets>
<div class=container></div>
</div>
<header id=header class="header site-header">
<div class="container sep-after">
</div>
</header>
<main id=content>
<article lang=en class=entry>
<header class="header entry-header">
<div class="container sep-after">
<div class=header-info>
<h1 class=title>Running K3s in LXC on Proxmox</h1>
</div>
<div class=entry-meta>
<span class=posted-on><svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><rect x="3" y="4" width="18" height="18" rx="2" ry="2"/><line x1="16" y1="2" x2="16" y2="6"/><line x1="8" y1="2" x2="8" y2="6"/><line x1="3" y1="10" x2="21" y2="10"/></svg>
<span class=screen-reader-text>Posted on </span>
<time class=entry-date datetime=2021-11-14T10:07:03-05:00>2021, Nov 14</time>
</span>
<span class=reading-time><svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><circle cx="12" cy="12" r="10"/><polyline points="12 6 12 12 15 15"/></svg>
4 mins read
</span>
</div>
</div>
</header>
<div class="container entry-content">
<p>It has been a while since I&rsquo;ve actively used Kubernetes and wanted to explore the evolution of tools such as <a href=https://helm.sh>Helm</a> and <a href=https://tekton.dev>Tekton</a>. I decided to deploy <a href=https://k3s.io>K3s</a>, since I&rsquo;ve had success with deploying it on resource-contrained Raspberry Pis in the past. I thought that this time it&rsquo;d be convenient to have K3s running in a LXC container on Proxmox. This would allow for easy snapshotting of the entire Kubernetes deployment. LXC containers also provide an efficient way to use a machine&rsquo;s resources.</p>
<h2 id=what-is-k3s>What is K3s?</h2>
<p>K3s is a Kubernetes distro that advertises itself as a lightweight binary with a much smaller memory-footprint than traditional k8s. K3s is not a fork of k8s as it seeks to remain as close to upstream as it possibly can.</p>
<h2 id=configure-proxmox>Configure Proxmox</h2>
<p>This <a href=https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185>gist</a> contains snippets and discussion on how to deploy K3s in LXC on Proxmox. It mentions that <code>bridge-nf-call-iptables</code> should be loaded, but I did not understand the benefit of doing this.</p>
<h2 id=disable-swap>Disable swap</h2>
<p>There is an issue on Kubernetes regarding swap <a href=https://github.com/kubernetes/kubernetes/issues/53533>here</a>. There claims to be support for swap in 1.22, but for now let&rsquo;s disable it:</p>
<pre tabindex=0><code>sysctl vm.swappiness=0
swapoff -a
</code></pre><p>It might be worth experimenting with swap enabled in the future to see how that might affect performance.</p>
<h3 id=enable-ip-forwarding>Enable IP Forwarding</h3>
<p>To avoid IP Forwarding issues with Traefik, run the following on the host:</p>
<div class=highlight><pre tabindex=0 style=background-color:#f0f3f3;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sh data-lang=sh>sudo sysctl net.ipv4.ip_forward<span style=color:#555>=</span><span style=color:#f60>1</span>
sudo sysctl net.ipv6.conf.all.forwarding<span style=color:#555>=</span><span style=color:#f60>1</span>
sudo sed -i <span style=color:#c30>&#39;s/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/g&#39;</span> /etc/sysctl.conf
sudo sed -i <span style=color:#c30>&#39;s/#net.ipv6.conf.all.forwarding=1/net.ipv6.conf.all.forwarding=1/g&#39;</span> /etc/sysctl.conf
</code></pre></div><h2 id=create-lxc-container>Create LXC container</h2>
<p>Create an LXC container in the Proxmox interface as you normally would. Remember to:</p>
<ul>
<li>Uncheck <code>unprivileged container</code></li>
<li>Use a LXC template (I chose a debian 11 template downloaded with <a href=https://pve.proxmox.com/wiki/Linux_Container#Create_container>pveam</a>)</li>
<li>In memory, set swap to 0</li>
<li>Create and start the container</li>
</ul>
<h3 id=modify-container-config>Modify container config</h3>
<p>Now back on the host run <code>pct list</code> to determine what VMID it was given.</p>
<p>Open <code>/etc/pve/lxc/$VMID.conf</code> and append:</p>
<div class=highlight><pre tabindex=0 style=background-color:#f0f3f3;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sh data-lang=sh>lxc.apparmor.profile: unconfined
lxc.cap.drop:
lxc.mount.auto: <span style=color:#c30>&#34;proc:rw sys:rw&#34;</span>
lxc.cgroup2.devices.allow: c 10:200 rwm
</code></pre></div><p>All of the above configurations are described in the <a href=https://linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html>manpages</a>.
Notice that <code>cgroup2</code> is used since Proxmox VE 7.0 has switched to a <a href=https://pve.proxmox.com/pve-docs/chapter-pct.html#pct_cgroup>pure cgroupv2 environment</a>.</p>
<p>Thankfully cgroup v2 support has been supported in k3s with these contributions:</p>
<ul>
<li><a href=https://github.com/k3s-io/k3s/pull/2584>https://github.com/k3s-io/k3s/pull/2584</a></li>
<li><a href=https://github.com/k3s-io/k3s/pull/2844>https://github.com/k3s-io/k3s/pull/2844</a></li>
</ul>
<h2 id=enable-shared-host-mounts>Enable shared host mounts</h2>
<p>From within the container, run:</p>
<div class=highlight><pre tabindex=0 style=background-color:#f0f3f3;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sh data-lang=sh><span style=color:#366>echo</span> <span style=color:#c30>&#39;#!/bin/sh -e
</span><span style=color:#c30>ln -s /dev/console /dev/kmsg
</span><span style=color:#c30>mount --make-rshared /&#39;</span> &gt; /etc/rc.local
chmod +x /etc/rc.local
reboot
</code></pre></div><h2 id=install-k3s>Install K3s</h2>
<p>One of the simplest ways to install K3s on a remote host is to use <a href=https://github.com/alexellis/k3sup>k3sup</a>.
Ensure that you supply a valid <code>CONTAINER_IP</code> and choose the <code>k3s-version</code> you prefer.
As of 2021/11, it is still defaulting to the 1.19 channel, so I overrode it to 1.22 for cgroup v2 support. See the published releases <a href=https://github.com/k3s-io/k3s/releases>here</a>.</p>
<div class=highlight><pre tabindex=0 style=background-color:#f0f3f3;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sh data-lang=sh>ssh-copy-id root@<span style=color:#033>$CONTAINER_IP</span>
k3sup install --ip <span style=color:#033>$CONTAINER_IP</span> --user root --k3s-version v1.22.3+k3s1
</code></pre></div><p>If all goes well, you should see a path to the <code>kubeconfig</code> generated. I moved this into <code>~/.kube/config</code> so that kubectl would read this by default.</p>
<h2 id=wrapping-up>Wrapping up</h2>
<p>Installing K3s in LXC on Proxmox works with a few tweaks to the default configuration. I later followed the Tekton&rsquo;s <a href=https://tekton.dev/docs/getting-started/>Getting Started</a> guide and was able to deploy it in a few commands.</p>
<div class=highlight><pre tabindex=0 style=background-color:#f0f3f3;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-console data-lang=console><span style=color:#009;font-weight:700>$ </span>kubectl get all --namespace tekton-pipelines
<span style=color:#aaa>NAME READY STATUS RESTARTS AGE
</span><span style=color:#aaa>pod/tekton-pipelines-webhook-8566ff9b6b-6rnh8 1/1 Running 1 (50m ago) 12h
</span><span style=color:#aaa>pod/tekton-dashboard-6bf858f977-qt4hr 1/1 Running 1 (50m ago) 11h
</span><span style=color:#aaa>pod/tekton-pipelines-controller-69fd7498d8-f57m4 1/1 Running 1 (50m ago) 12h
</span><span style=color:#aaa></span><span style=color:#a00;background-color:#faa>
</span><span style=color:#a00;background-color:#faa></span><span style=color:#aaa>NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
</span><span style=color:#aaa>service/tekton-pipelines-controller ClusterIP 10.43.44.245 &lt;none&gt; 9090/TCP,8080/TCP 12h
</span><span style=color:#aaa>service/tekton-pipelines-webhook ClusterIP 10.43.183.242 &lt;none&gt; 9090/TCP,8008/TCP,443/TCP,8080/TCP 12h
</span><span style=color:#aaa>service/tekton-dashboard ClusterIP 10.43.87.97 &lt;none&gt; 9097/TCP 11h
</span><span style=color:#aaa></span><span style=color:#a00;background-color:#faa>
</span><span style=color:#a00;background-color:#faa></span><span style=color:#aaa>NAME READY UP-TO-DATE AVAILABLE AGE
</span><span style=color:#aaa>deployment.apps/tekton-pipelines-webhook 1/1 1 1 12h
</span><span style=color:#aaa>deployment.apps/tekton-dashboard 1/1 1 1 11h
</span><span style=color:#aaa>deployment.apps/tekton-pipelines-controller 1/1 1 1 12h
</span><span style=color:#aaa></span><span style=color:#a00;background-color:#faa>
</span><span style=color:#a00;background-color:#faa></span><span style=color:#aaa>NAME DESIRED CURRENT READY AGE
</span><span style=color:#aaa>replicaset.apps/tekton-pipelines-webhook-8566ff9b6b 1 1 1 12h
</span><span style=color:#aaa>replicaset.apps/tekton-dashboard-6bf858f977 1 1 1 11h
</span><span style=color:#aaa>replicaset.apps/tekton-pipelines-controller-69fd7498d8 1 1 1 12h
</span><span style=color:#aaa></span><span style=color:#a00;background-color:#faa>
</span><span style=color:#a00;background-color:#faa></span><span style=color:#aaa>NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
</span><span style=color:#aaa>horizontalpodautoscaler.autoscaling/tekton-pipelines-webhook Deployment/tekton-pipelines-webhook 9%/100% 1 5 1 12h
</span></code></pre></div><p>I made sure to install Tailscale in the container so that I can easily access K3s from anywhere.</p>
<p>If I&rsquo;m feeling adventurous, I might experiment with <a href=https://rancher.com/docs/k3s/latest/en/advanced/#running-k3s-with-rootless-mode-experimental>K3s rootless</a>.</p>
</div>
<footer class=entry-footer>
<div class="container sep-before"><div class=tags><svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><path d="M20.59 13.41l-7.17 7.17a2 2 0 01-2.83.0L2 12V2H12l8.59 8.59a2 2 0 010 2.82z"/><line x1="7" y1="7" x2="7" y2="7"/></svg>
<span class=screen-reader-text>Tags: </span><a class=tag href=/tags/k3s/>k3s</a>, <a class=tag href=/tags/proxmox/>proxmox</a>, <a class=tag href=/tags/lxc/>lxc</a></div>
</div>
</footer>
</article>
<nav class=entry-nav>
<div class=container><div class="prev-entry sep-before">
<a href=/blog/2021/10/11/replacing-docker-with-podman-on-macos-and-linux/>
<span aria-hidden=true><svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><line x1="20" y1="12" x2="4" y2="12"/><polyline points="10 18 4 12 10 6"/></svg>
Previous</span>
<span class=screen-reader-text>Previous post: </span>Replacing docker with podman on macOS (and Linux)</a>
</div><div class="next-entry sep-before">
<a href=/blog/2022/03/13/backing-up-gmail-with-synology/>
<span class=screen-reader-text>Next post: </span>Backing Up Gmail With Synology<span aria-hidden=true>Next<svg class="icon" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" aria-hidden="true"><line x1="4" y1="12" x2="20" y2="12"/><polyline points="14 6 20 12 14 18"/></svg>
</span>
</a>
</div></div>
</nav>
<section id=comments class=comments>
<div class="container sep-before">
<div class=comments-area><script src=https://utteranc.es/client.js repo=davegallant/davegallant.github.io issue-term=pathname theme=github-light crossorigin=anonymous async></script>
</div>
</div>
</section>
</main>
<footer id=footer class=footer>
<div class="container sep-before"><div class=copyright>
<p> &copy; 2020-2022 Dave Gallant </p>
</div>
</div>
</footer>
</div>
</div><script>window.__assets_js_src="/assets/js/"</script>
<script src=/assets/js/main.c3bcf2df.js></script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink