dave/site
1
0
Fork 0
mirror of https://github.com/davegallant/davegallant.github.io.git synced 2025-08-05 08:13:40 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
9d730521f72f95aca7e04ef56d68ab1103c94939
site/public/blog/2021/11/14/running-k3s-in-lxc-on-proxmox/index.html
Dave Gallant 9d730521f7 Add pfsense draft
2022-04-03 21:36:37 -04:00

478 lines
22 KiB
HTML
Raw Blame History

<!DOCTYPE html>
<html lang='en' dir='auto'><head>
<meta charset='utf-8'>
<meta name='viewport' content='width=device-width, initial-scale=1'>
<meta name='description' content=''>
<meta name='theme-color' content='#8979b3'>
<meta property='og:title' content='Running K3s in LXC on Proxmox • davegallant'>
<meta property='og:description' content=''>
<meta property='og:url' content='/blog/2021/11/14/running-k3s-in-lxc-on-proxmox/'>
<meta property='og:site_name' content='davegallant'>
<meta property='og:type' content='article'><meta property='article:section' content='post'><meta property='article:tag' content='k3s'><meta property='article:tag' content='proxmox'><meta property='article:tag' content='lxc'><meta property='article:published_time' content='2021-11-14T10:07:03-05:00'/><meta property='article:modified_time' content='2021-11-14T10:07:03-05:00'/><meta name='twitter:card' content='summary'>
<meta name="generator" content="Hugo 0.92.2" />
<title>Running K3s in LXC on Proxmox • davegallant</title>
<link rel='canonical' href='/blog/2021/11/14/running-k3s-in-lxc-on-proxmox/'>
<link rel='icon' href='/favicon.ico'>
<link rel='stylesheet' href='/assets/css/main.ab98e12b.css'><link rel='stylesheet' href='/css/custom.css'><style>
:root{--color-accent:#8979b3;}
</style>
<script type="application/javascript">
var doNotTrack = false;
if (!doNotTrack) {
window.ga=window.ga||function(){(ga.q=ga.q||[]).push(arguments)};ga.l=+new Date;
ga('create', 'UA-98710982-2', 'auto');
ga('send', 'pageview');
}
</script>
<script async src='https://www.google-analytics.com/analytics.js'></script>
</head>
<body class='page type-post has-sidebar'>
<div class='site'><div id='sidebar' class='sidebar'>
<a class='screen-reader-text' href='#main-menu'>Skip to Main Menu</a>
<div class='container'><section class='widget widget-about sep-after'>
<header>
<div class='logo'>
<a href='/'>
<img src='/images/logo.png'>
</a>
</div>
<h2 class='title site-title '>
<a href='/'>
davegallant
</a>
</h2>
<div class='desc'>
personal blog
</div>
</header>
</section>
<section class='widget widget-taxonomy_cloud sep-after'>
<header>
<h4 class='title widget-title'>Tags</h4>
</header>
<div class='container list-container'>
<ul class='list taxonomy-cloud'><li>
<a href='/tags/adguard/' style='font-size:1em'>adguard</a>
</li><li>
<a href='/tags/aws/' style='font-size:1em'>aws</a>
</li><li>
<a href='/tags/aws-vault/' style='font-size:1em'>aws-vault</a>
</li><li>
<a href='/tags/backup/' style='font-size:1em'>backup</a>
</li><li>
<a href='/tags/containers/' style='font-size:1em'>containers</a>
</li><li>
<a href='/tags/degoogle/' style='font-size:1em'>degoogle</a>
</li><li>
<a href='/tags/docker/' style='font-size:1em'>docker</a>
</li><li>
<a href='/tags/dotfiles/' style='font-size:1em'>dotfiles</a>
</li><li>
<a href='/tags/gmail/' style='font-size:1em'>gmail</a>
</li><li>
<a href='/tags/grafana/' style='font-size:1em'>grafana</a>
</li><li>
<a href='/tags/home-manager/' style='font-size:1em'>home-manager</a>
</li><li>
<a href='/tags/homelab/' style='font-size:1em'>homelab</a>
</li><li>
<a href='/tags/jellyfin/' style='font-size:1em'>jellyfin</a>
</li><li>
<a href='/tags/k3s/' style='font-size:1em'>k3s</a>
</li><li>
<a href='/tags/linux/' style='font-size:1em'>linux</a>
</li><li>
<a href='/tags/lxc/' style='font-size:1em'>lxc</a>
</li><li>
<a href='/tags/netdata/' style='font-size:1em'>netdata</a>
</li><li>
<a href='/tags/nix/' style='font-size:1em'>nix</a>
</li><li>
<a href='/tags/openwrt/' style='font-size:1em'>openwrt</a>
</li><li>
<a href='/tags/pfsense/' style='font-size:1em'>pfsense</a>
</li><li>
<a href='/tags/pihole/' style='font-size:1em'>pihole</a>
</li><li>
<a href='/tags/plex/' style='font-size:1em'>plex</a>
</li><li>
<a href='/tags/podman/' style='font-size:1em'>podman</a>
</li><li>
<a href='/tags/proxmox/' style='font-size:2em'>proxmox</a>
</li><li>
<a href='/tags/python/' style='font-size:2em'>python</a>
</li><li>
<a href='/tags/ransomware/' style='font-size:1em'>ransomware</a>
</li><li>
<a href='/tags/router/' style='font-size:1em'>router</a>
</li><li>
<a href='/tags/router-on-a-stick/' style='font-size:1em'>router-on-a-stick</a>
</li><li>
<a href='/tags/security/' style='font-size:1em'>security</a>
</li><li>
<a href='/tags/synology/' style='font-size:1em'>synology</a>
</li><li>
<a href='/tags/tailscale/' style='font-size:1em'>tailscale</a>
</li><li>
<a href='/tags/virtualization/' style='font-size:1em'>virtualization</a>
</li><li>
<a href='/tags/vlan/' style='font-size:1em'>vlan</a>
</li><li>
<a href='/tags/vpn/' style='font-size:1em'>vpn</a>
</li></ul>
</div>
</section>
<section class='widget widget-social_menu sep-after'><nav aria-label='Social Menu'>
<ul><li>
<a href='https://github.com/davegallant' target='_blank' rel='noopener me'>
<span class='screen-reader-text'>Open Github account in new tab</span><svg
class="icon"
xmlns="http://www.w3.org/2000/svg"
viewbox="0 0 24 24"
stroke-linecap="round"
stroke-linejoin="round"
stroke-width="2"
aria-hidden="true"
><path d="M9 19c-5 1.5-5-2.5-7-3m14 6v-3.87a3.37 3.37 0 0 0-.94-2.61c3.14-.35 6.44-1.54 6.44-7A5.44 5.44 0 0 0 20 4.77 5.07 5.07 0 0 0 19.91 1S18.73.65 16 2.48a13.38 13.38 0 0 0-7 0C6.27.65 5.09 1 5.09 1A5.07 5.07 0 0 0 5 4.77a5.44 5.44 0 0 0-1.5 3.78c0 5.42 3.3 6.61 6.44 7A3.37 3.37 0 0 0 9 18.13V22" />
</svg>
</a>
</li><li>
<a href='https://twitter.com/dave_gallant_' target='_blank' rel='noopener me'>
<span class='screen-reader-text'>Open Twitter account in new tab</span><svg
class="icon"
xmlns="http://www.w3.org/2000/svg"
viewbox="0 0 24 24"
stroke-linecap="round"
stroke-linejoin="round"
stroke-width="2"
aria-hidden="true"
><path d="M23 3a10.9 10.9 0 0 1-3.14 1.53 4.48 4.48 0 0 0-7.86 3v1A10.66 10.66 0 0 1 3 4s-4 9 5 13a11.64 11.64 0 0 1-7 2c9 5 20 0 20-11.5a4.5 4.5 0 0 0-.08-.83A7.72 7.72 0 0 0 23 3z" />
</svg>
</a>
</li><li>
<a href='mailto:davegallant@gmail.com' target='_blank' rel='noopener me'>
<span class='screen-reader-text'>Contact via Email</span><svg
class="icon"
xmlns="http://www.w3.org/2000/svg"
viewbox="0 0 24 24"
stroke-linecap="round"
stroke-linejoin="round"
stroke-width="2"
aria-hidden="true"
><path d="M4 4h16c1.1 0 2 .9 2 2v12c0 1.1-.9 2-2 2H4c-1.1 0-2-.9-2-2V6c0-1.1.9-2 2-2z" />
<polyline points="22,6 12,13 2,6" />
</svg>
</a>
</li><li>
<a href='https://linkedin.com/in/dave-gallant' target='_blank' rel='noopener me'>
<span class='screen-reader-text'>Open Linkedin account in new tab</span><svg
class="icon"
xmlns="http://www.w3.org/2000/svg"
viewbox="0 0 24 24"
stroke-linecap="round"
stroke-linejoin="round"
stroke-width="2"
aria-hidden="true"
><path d="M16 8a6 6 0 0 1 6 6v7h-4v-7a2 2 0 0 0-2-2 2 2 0 0 0-2 2v7h-4v-7a6 6 0 0 1 6-6z" />
<rect x="2" y="9" width="4" height="12" />
<circle cx="4" cy="4" r="2" />
</svg>
</a>
</li></ul>
</nav>
</section></div>
<div class='sidebar-overlay'></div>
</div><div class='main'><nav id='main-menu' class='menu main-menu' aria-label='Main Menu'>
<div class='container'>
<a class='screen-reader-text' href='#content'>Skip to Content</a>
<button id='sidebar-toggler' class='sidebar-toggler' aria-controls='sidebar'>
<span class='screen-reader-text'>Toggle Sidebar</span>
<span class='open'><svg
class="icon"
xmlns="http://www.w3.org/2000/svg"
viewbox="0 0 24 24"
stroke-linecap="round"
stroke-linejoin="round"
stroke-width="2"
aria-hidden="true"
><line x1="3" y1="12" x2="21" y2="12" />
<line x1="3" y1="6" x2="21" y2="6" />
<line x1="3" y1="18" x2="21" y2="18" />
</svg>
</span>
<span class='close'><svg
class="icon"
xmlns="http://www.w3.org/2000/svg"
viewbox="0 0 24 24"
stroke-linecap="round"
stroke-linejoin="round"
stroke-width="2"
aria-hidden="true"
><line x1="18" y1="6" x2="6" y2="18" />
<line x1="6" y1="6" x2="18" y2="18" />
</svg>
</span>
</button>
<ul><li class='item'>
<a href='/'>Home</a>
</li><li class='item'>
<a href='/about/'>About</a>
</li><li class='item'>
<a href='/index.xml'>RSS</a>
</li></ul>
</div>
</nav><div class='header-widgets'>
<div class='container'></div>
</div>
<header id='header' class='header site-header'>
<div class='container sep-after'>
</div>
</header>
<main id='content'>
<article lang='en' class='entry'>
<header class='header entry-header'>
<div class='container sep-after'>
<div class='header-info'>
<h1 class='title'>Running K3s in LXC on Proxmox</h1>
</div>
<div class='entry-meta'>
<span class='posted-on'><svg
class="icon"
xmlns="http://www.w3.org/2000/svg"
viewbox="0 0 24 24"
stroke-linecap="round"
stroke-linejoin="round"
stroke-width="2"
aria-hidden="true"
><rect x="3" y="4" width="18" height="18" rx="2" ry="2" />
<line x1="16" y1="2" x2="16" y2="6" />
<line x1="8" y1="2" x2="8" y2="6" />
<line x1="3" y1="10" x2="21" y2="10" />
</svg>
<span class='screen-reader-text'>Posted on </span>
<time class='entry-date' datetime='2021-11-14T10:07:03-05:00'>2021, Nov 14</time>
</span>
<span class='reading-time'><svg
class="icon"
xmlns="http://www.w3.org/2000/svg"
viewbox="0 0 24 24"
stroke-linecap="round"
stroke-linejoin="round"
stroke-width="2"
aria-hidden="true"
><circle cx="12" cy="12" r="10" />
<polyline points="12 6 12 12 15 15" />
</svg>
4 mins read
</span>
</div>
</div>
</header>
<div class='container entry-content'>
<p>It has been a while since I&rsquo;ve actively used Kubernetes and wanted to explore the evolution of tools such as <a href="https://helm.sh">Helm</a> and <a href="https://tekton.dev">Tekton</a>. I decided to deploy <a href="https://k3s.io">K3s</a>, since I&rsquo;ve had success with deploying it on resource-contrained Raspberry Pis in the past. I thought that this time it&rsquo;d be convenient to have K3s running in a LXC container on Proxmox. This would allow for easy snapshotting of the entire Kubernetes deployment. LXC containers also provide an efficient way to use a machine&rsquo;s resources.</p>
<h2 id="what-is-k3s">What is K3s?</h2>
<p>K3s is a Kubernetes distro that advertises itself as a lightweight binary with a much smaller memory-footprint than traditional k8s. K3s is not a fork of k8s as it seeks to remain as close to upstream as it possibly can.</p>
<h2 id="configure-proxmox">Configure Proxmox</h2>
<p>This <a href="https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185">gist</a> contains snippets and discussion on how to deploy K3s in LXC on Proxmox. It mentions that <code>bridge-nf-call-iptables</code> should be loaded, but I did not understand the benefit of doing this.</p>
<h2 id="disable-swap">Disable swap</h2>
<p>There is an issue on Kubernetes regarding swap <a href="https://github.com/kubernetes/kubernetes/issues/53533">here</a>. There claims to be support for swap in 1.22, but for now let&rsquo;s disable it:</p>
<pre tabindex="0"><code>sysctl vm.swappiness=0
swapoff -a
</code></pre><p>It might be worth experimenting with swap enabled in the future to see how that might affect performance.</p>
<h3 id="enable-ip-forwarding">Enable IP Forwarding</h3>
<p>To avoid IP Forwarding issues with Traefik, run the following on the host:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f3f3;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">sudo sysctl net.ipv4.ip_forward<span style="color:#555">=</span><span style="color:#f60">1</span>
sudo sysctl net.ipv6.conf.all.forwarding<span style="color:#555">=</span><span style="color:#f60">1</span>
sudo sed -i <span style="color:#c30">&#39;s/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/g&#39;</span> /etc/sysctl.conf
sudo sed -i <span style="color:#c30">&#39;s/#net.ipv6.conf.all.forwarding=1/net.ipv6.conf.all.forwarding=1/g&#39;</span> /etc/sysctl.conf
</code></pre></div><h2 id="create-lxc-container">Create LXC container</h2>
<p>Create an LXC container in the Proxmox interface as you normally would. Remember to:</p>
<ul>
<li>Uncheck <code>unprivileged container</code></li>
<li>Use a LXC template (I chose a debian 11 template downloaded with <a href="https://pve.proxmox.com/wiki/Linux_Container#Create_container">pveam</a>)</li>
<li>In memory, set swap to 0</li>
<li>Create and start the container</li>
</ul>
<h3 id="modify-container-config">Modify container config</h3>
<p>Now back on the host run <code>pct list</code> to determine what VMID it was given.</p>
<p>Open <code>/etc/pve/lxc/$VMID.conf</code> and append:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f3f3;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">lxc.apparmor.profile: unconfined
lxc.cap.drop:
lxc.mount.auto: <span style="color:#c30">&#34;proc:rw sys:rw&#34;</span>
lxc.cgroup2.devices.allow: c 10:200 rwm
</code></pre></div><p>All of the above configurations are described in the <a href="https://linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html">manpages</a>.
Notice that <code>cgroup2</code> is used since Proxmox VE 7.0 has switched to a <a href="https://pve.proxmox.com/pve-docs/chapter-pct.html#pct_cgroup">pure cgroupv2 environment</a>.</p>
<p>Thankfully cgroup v2 support has been supported in k3s with these contributions:</p>
<ul>
<li><a href="https://github.com/k3s-io/k3s/pull/2584">https://github.com/k3s-io/k3s/pull/2584</a></li>
<li><a href="https://github.com/k3s-io/k3s/pull/2844">https://github.com/k3s-io/k3s/pull/2844</a></li>
</ul>
<h2 id="enable-shared-host-mounts">Enable shared host mounts</h2>
<p>From within the container, run:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f3f3;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh"><span style="color:#366">echo</span> <span style="color:#c30">&#39;#!/bin/sh -e
</span><span style="color:#c30">ln -s /dev/console /dev/kmsg
</span><span style="color:#c30">mount --make-rshared /&#39;</span> &gt; /etc/rc.local
chmod +x /etc/rc.local
reboot
</code></pre></div><h2 id="install-k3s">Install K3s</h2>
<p>One of the simplest ways to install K3s on a remote host is to use <a href="https://github.com/alexellis/k3sup">k3sup</a>.
Ensure that you supply a valid <code>CONTAINER_IP</code> and choose the <code>k3s-version</code> you prefer.
As of 2021/11, it is still defaulting to the 1.19 channel, so I overrode it to 1.22 for cgroup v2 support. See the published releases <a href="https://github.com/k3s-io/k3s/releases">here</a>.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f3f3;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">ssh-copy-id root@<span style="color:#033">$CONTAINER_IP</span>
k3sup install --ip <span style="color:#033">$CONTAINER_IP</span> --user root --k3s-version v1.22.3+k3s1
</code></pre></div><p>If all goes well, you should see a path to the <code>kubeconfig</code> generated. I moved this into <code>~/.kube/config</code> so that kubectl would read this by default.</p>
<h2 id="wrapping-up">Wrapping up</h2>
<p>Installing K3s in LXC on Proxmox works with a few tweaks to the default configuration. I later followed the Tekton&rsquo;s <a href="https://tekton.dev/docs/getting-started/">Getting Started</a> guide and was able to deploy it in a few commands.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f3f3;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console"><span style="color:#009;font-weight:bold">$ </span>kubectl get all --namespace tekton-pipelines
<span style="color:#aaa">NAME READY STATUS RESTARTS AGE
</span><span style="color:#aaa">pod/tekton-pipelines-webhook-8566ff9b6b-6rnh8 1/1 Running 1 (50m ago) 12h
</span><span style="color:#aaa">pod/tekton-dashboard-6bf858f977-qt4hr 1/1 Running 1 (50m ago) 11h
</span><span style="color:#aaa">pod/tekton-pipelines-controller-69fd7498d8-f57m4 1/1 Running 1 (50m ago) 12h
</span><span style="color:#aaa"></span><span style="color:#a00;background-color:#faa">
</span><span style="color:#a00;background-color:#faa"></span><span style="color:#aaa">NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
</span><span style="color:#aaa">service/tekton-pipelines-controller ClusterIP 10.43.44.245 &lt;none&gt; 9090/TCP,8080/TCP 12h
</span><span style="color:#aaa">service/tekton-pipelines-webhook ClusterIP 10.43.183.242 &lt;none&gt; 9090/TCP,8008/TCP,443/TCP,8080/TCP 12h
</span><span style="color:#aaa">service/tekton-dashboard ClusterIP 10.43.87.97 &lt;none&gt; 9097/TCP 11h
</span><span style="color:#aaa"></span><span style="color:#a00;background-color:#faa">
</span><span style="color:#a00;background-color:#faa"></span><span style="color:#aaa">NAME READY UP-TO-DATE AVAILABLE AGE
</span><span style="color:#aaa">deployment.apps/tekton-pipelines-webhook 1/1 1 1 12h
</span><span style="color:#aaa">deployment.apps/tekton-dashboard 1/1 1 1 11h
</span><span style="color:#aaa">deployment.apps/tekton-pipelines-controller 1/1 1 1 12h
</span><span style="color:#aaa"></span><span style="color:#a00;background-color:#faa">
</span><span style="color:#a00;background-color:#faa"></span><span style="color:#aaa">NAME DESIRED CURRENT READY AGE
</span><span style="color:#aaa">replicaset.apps/tekton-pipelines-webhook-8566ff9b6b 1 1 1 12h
</span><span style="color:#aaa">replicaset.apps/tekton-dashboard-6bf858f977 1 1 1 11h
</span><span style="color:#aaa">replicaset.apps/tekton-pipelines-controller-69fd7498d8 1 1 1 12h
</span><span style="color:#aaa"></span><span style="color:#a00;background-color:#faa">
</span><span style="color:#a00;background-color:#faa"></span><span style="color:#aaa">NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
</span><span style="color:#aaa">horizontalpodautoscaler.autoscaling/tekton-pipelines-webhook Deployment/tekton-pipelines-webhook 9%/100% 1 5 1 12h
</span></code></pre></div><p>I made sure to install Tailscale in the container so that I can easily access K3s from anywhere.</p>
<p>If I&rsquo;m feeling adventurous, I might experiment with <a href="https://rancher.com/docs/k3s/latest/en/advanced/#running-k3s-with-rootless-mode-experimental">K3s rootless</a>.</p>
</div>
<footer class='entry-footer'>
<div class='container sep-before'><div class='tags'><svg
class="icon"
xmlns="http://www.w3.org/2000/svg"
viewbox="0 0 24 24"
stroke-linecap="round"
stroke-linejoin="round"
stroke-width="2"
aria-hidden="true"
><path d="M20.59,13.41l-7.17,7.17a2,2,0,0,1-2.83,0L2,12V2H12l8.59,8.59A2,2,0,0,1,20.59,13.41Z" />
<line x1="7" y1="7" x2="7" y2="7" />
</svg>
<span class='screen-reader-text'>Tags: </span><a class='tag' href='/tags/k3s/'>k3s</a>, <a class='tag' href='/tags/proxmox/'>proxmox</a>, <a class='tag' href='/tags/lxc/'>lxc</a></div>
</div>
</footer>
</article>
<nav class='entry-nav'>
<div class='container'><div class='prev-entry sep-before'>
<a href='/blog/2021/10/11/replacing-docker-with-podman-on-macos-and-linux/'>
<span aria-hidden='true'><svg
class="icon"
xmlns="http://www.w3.org/2000/svg"
viewbox="0 0 24 24"
stroke-linecap="round"
stroke-linejoin="round"
stroke-width="2"
aria-hidden="true"
><line x1="20" y1="12" x2="4" y2="12" />
<polyline points="10 18 4 12 10 6" />
</svg>
Previous</span>
<span class='screen-reader-text'>Previous post: </span>Replacing docker with podman on macOS (and Linux)</a>
</div><div class='next-entry sep-before'>
<a href='/blog/2022/03/13/backing-up-gmail-with-synology/'>
<span class='screen-reader-text'>Next post: </span>Backing Up Gmail With Synology<span aria-hidden='true'>Next <svg
class="icon"
xmlns="http://www.w3.org/2000/svg"
viewbox="0 0 24 24"
stroke-linecap="round"
stroke-linejoin="round"
stroke-width="2"
aria-hidden="true"
><line x1="4" y1="12" x2="20" y2="12" />
<polyline points="14 6 20 12 14 18" />
</svg>
</span>
</a>
</div></div>
</nav>
<section id='comments' class='comments'>
<div class='container sep-before'>
<div class='comments-area'><script src='https://utteranc.es/client.js'
repo='davegallant/davegallant.github.io'
issue-term='pathname'
theme='github-light'
crossorigin='anonymous' async>
</script>
</div>
</div>
</section>
</main>
<footer id='footer' class='footer'>
<div class='container sep-before'><div class='copyright'>
<p> &copy; 2020-2022 Dave Gallant </p>
</div>
</div>
</footer>
</div>
</div><script>window.__assets_js_src="/assets/js/"</script>
<script src='/assets/js/main.c3bcf2df.js'></script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink