dave/site
1
0
Fork 0
mirror of https://github.com/davegallant/davegallant.github.io.git synced 2025-08-06 00:33:39 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
592b3d8d071162e8a90249116b1101e0a26243e7
site/public/blog/2021/11/14/running-k3s-in-lxc-on-proxmox/index.html
Dave Gallant 592b3d8d07 Remove ### from headers
2023-12-24 22:31:36 -05:00

287 lines
19 KiB
HTML
Raw Blame History

<!DOCTYPE html>
<html><head>
<meta charset="utf-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge"><title>Running K3s in LXC on Proxmox - davegallant</title><link rel="icon" type="image/png" href=https://davegallant.ca/favicon.ico /><meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="It has been a while since I&rsquo;ve actively used Kubernetes and wanted to explore the evolution of tools such as Helm and Tekton. I decided to deploy K3s, since I&rsquo;ve had success with deploying it on resource-contrained Raspberry Pis in the past. I thought that this time it&rsquo;d be convenient to have K3s running in a LXC container on Proxmox. This would allow for easy snapshotting of the entire Kubernetes deployment." />
<meta property="og:image" content=""/>
<meta property="og:title" content="Running K3s in LXC on Proxmox" />
<meta property="og:description" content="It has been a while since I&rsquo;ve actively used Kubernetes and wanted to explore the evolution of tools such as Helm and Tekton. I decided to deploy K3s, since I&rsquo;ve had success with deploying it on resource-contrained Raspberry Pis in the past. I thought that this time it&rsquo;d be convenient to have K3s running in a LXC container on Proxmox. This would allow for easy snapshotting of the entire Kubernetes deployment." />
<meta property="og:type" content="article" />
<meta property="og:url" content="/blog/2021/11/14/running-k3s-in-lxc-on-proxmox/" /><meta property="article:section" content="post" />
<meta property="article:published_time" content="2021-11-14T10:07:03-05:00" />
<meta property="article:modified_time" content="2021-11-14T10:07:03-05:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Running K3s in LXC on Proxmox"/>
<meta name="twitter:description" content="It has been a while since I&rsquo;ve actively used Kubernetes and wanted to explore the evolution of tools such as Helm and Tekton. I decided to deploy K3s, since I&rsquo;ve had success with deploying it on resource-contrained Raspberry Pis in the past. I thought that this time it&rsquo;d be convenient to have K3s running in a LXC container on Proxmox. This would allow for easy snapshotting of the entire Kubernetes deployment."/>
<script src="https://cdn.jsdelivr.net/npm/feather-icons/dist/feather.min.js"></script>
<link href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:ital,wght@1,500&display=swap" rel="stylesheet">
<link href="https://fonts.googleapis.com/css2?family=Fira+Sans&display=swap" rel="stylesheet">
<link href="https://fonts.googleapis.com/css?family=Roboto+Mono" rel="stylesheet">
<link rel="stylesheet" type="text/css" media="screen" href="/css/main.5ff68256cc04aa4360600458bf9767c053d7b990cfb09caa0724dd24045068c7.css" />
<link id="darkModeStyle" rel="stylesheet" type="text/css" href="/css/dark.d61dc67f7e7db9e2fb71ee6e28efc53086daed613da0bbeb0ef66045420afd17.css" disabled />
<script type="text/javascript"
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
tex2jax: {
inlineMath: [['$','$'], ['\\(','\\)']],
displayMath: [['$$','$$'], ['\[','\]']],
processEscapes: true,
processEnvironments: true,
skipTags: ['script', 'noscript', 'style', 'textarea', 'pre'],
TeX: { equationNumbers: { autoNumber: "AMS" },
extensions: ["AMSmath.js", "AMSsymbols.js"] }
}
});
</script>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.15.2/dist/katex.min.css">
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.15.2/dist/katex.min.js"></script>
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.15.2/dist/contrib/auto-render.min.js" onload="renderMathInElement(document.body);"></script>
<script>
document.addEventListener("DOMContentLoaded", function() {
renderMathInElement(document.body, {
delimiters: [
{left: "$$", right: "$$", display: true},
{left: "$", right: "$", display: false}
]
});
});
</script>
<link rel="stylesheet" type="text/css" href="/css/custom.d96bfb9e3314a7699144ab6ae7331d424cbd7fb34a2e890b17e7bb7db4e30f3a.css">
</head>
<body>
<div class="content"><header>
<div class="main">
<a href="/">davegallant</a>
</div>
<nav>
<a href="/">Posts</a>
<a href="/about">About</a>
<a href="/index.xml">RSS</a>
</nav>
<script
type="text/javascript"
src="https://storage.ko-fi.com/cdn/widget/Widget_2.js"
></script>
<span id="dark-mode-toggle" onclick="toggleTheme()"></span>
<script src="/js/themetoggle.js"></script>
<link href="/css/copy-code-button.min.css" rel="stylesheet">
</header>
<main>
<article>
<div class="title">
<h1 class="title">Running K3s in LXC on Proxmox</h1>
<div class="meta">Posted on Nov 14, 2021</div>
</div>
<section class="body"><p>It has been a while since I&rsquo;ve actively used Kubernetes and wanted to explore the evolution of tools such as <a href="https://helm.sh">Helm</a> and <a href="https://tekton.dev">Tekton</a>. I decided to deploy <a href="https://k3s.io">K3s</a>, since I&rsquo;ve had success with deploying it on resource-contrained Raspberry Pis in the past. I thought that this time it&rsquo;d be convenient to have K3s running in a LXC container on Proxmox. This would allow for easy snapshotting of the entire Kubernetes deployment. LXC containers also provide an efficient way to use a machine&rsquo;s resources.</p>
<h2 id="what-is-k3s">What is K3s?<a href="#what-is-k3s" class="hanchor" ariaLabel="Anchor">#</a></h2>
<p>K3s is a Kubernetes distro that advertises itself as a lightweight binary with a much smaller memory-footprint than traditional k8s. K3s is not a fork of k8s as it seeks to remain as close to upstream as it possibly can.</p>
<h2 id="configure-proxmox">Configure Proxmox<a href="#configure-proxmox" class="hanchor" ariaLabel="Anchor">#</a></h2>
<p>This <a href="https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185">gist</a> contains snippets and discussion on how to deploy K3s in LXC on Proxmox. It mentions that <code>bridge-nf-call-iptables</code> should be loaded, but I did not understand the benefit of doing this.</p>
<h2 id="disable-swap">Disable swap<a href="#disable-swap" class="hanchor" ariaLabel="Anchor">#</a></h2>
<p>There is an issue on Kubernetes regarding swap <a href="https://github.com/kubernetes/kubernetes/issues/53533">here</a>. There claims to be support for swap in 1.22, but for now let&rsquo;s disable it:</p>
<div class="highlight"><pre tabindex="0" style="color:#d8dee9;background-color:#2e3440;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>sudo sysctl vm.swappiness<span style="color:#81a1c1">=</span><span style="color:#b48ead">0</span>
</span></span><span style="display:flex;"><span>sudo swapoff -a
</span></span></code></pre></div><p>It might be worth experimenting with swap enabled in the future to see how that might affect performance.</p>
<h3 id="enable-ip-forwarding">Enable IP Forwarding<a href="#enable-ip-forwarding" class="hanchor" ariaLabel="Anchor">#</a></h3>
<p>To avoid IP Forwarding issues with Traefik, run the following on the host:</p>
<div class="highlight"><pre tabindex="0" style="color:#d8dee9;background-color:#2e3440;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>sudo sysctl net.ipv4.ip_forward<span style="color:#81a1c1">=</span><span style="color:#b48ead">1</span>
</span></span><span style="display:flex;"><span>sudo sysctl net.ipv6.conf.all.forwarding<span style="color:#81a1c1">=</span><span style="color:#b48ead">1</span>
</span></span><span style="display:flex;"><span>sudo sed -i <span style="color:#a3be8c">&#39;s/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/g&#39;</span> /etc/sysctl.conf
</span></span><span style="display:flex;"><span>sudo sed -i <span style="color:#a3be8c">&#39;s/#net.ipv6.conf.all.forwarding=1/net.ipv6.conf.all.forwarding=1/g&#39;</span> /etc/sysctl.conf
</span></span></code></pre></div><h2 id="create-lxc-container">Create LXC container<a href="#create-lxc-container" class="hanchor" ariaLabel="Anchor">#</a></h2>
<p>Create an LXC container in the Proxmox interface as you normally would. Remember to:</p>
<ul>
<li>Uncheck <code>unprivileged container</code></li>
<li>Use a LXC template (I chose a debian 11 template downloaded with <a href="https://pve.proxmox.com/wiki/Linux_Container#Create_container">pveam</a>)</li>
<li>In memory, set swap to 0</li>
<li>Create and start the container</li>
</ul>
<h3 id="modify-container-config">Modify container config<a href="#modify-container-config" class="hanchor" ariaLabel="Anchor">#</a></h3>
<p>Now back on the host run <code>pct list</code> to determine what VMID it was given.</p>
<p>Open <code>/etc/pve/lxc/$VMID.conf</code> and append:</p>
<div class="highlight"><pre tabindex="0" style="color:#d8dee9;background-color:#2e3440;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>lxc.apparmor.profile: unconfined
</span></span><span style="display:flex;"><span>lxc.cap.drop:
</span></span><span style="display:flex;"><span>lxc.mount.auto: &#34;proc:rw sys:rw&#34;
</span></span><span style="display:flex;"><span>lxc.cgroup2.devices.allow: c 10:200 rwm
</span></span></code></pre></div><p>All of the above configurations are described in the <a href="https://linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html">manpages</a>.
Notice that <code>cgroup2</code> is used since Proxmox VE 7.0 has switched to a <a href="https://pve.proxmox.com/pve-docs/chapter-pct.html#pct_cgroup">pure cgroupv2 environment</a>.</p>
<p>Thankfully cgroup v2 support has been supported in k3s with these contributions:</p>
<ul>
<li><a href="https://github.com/k3s-io/k3s/pull/2584">https://github.com/k3s-io/k3s/pull/2584</a></li>
<li><a href="https://github.com/k3s-io/k3s/pull/2844">https://github.com/k3s-io/k3s/pull/2844</a></li>
</ul>
<h2 id="enable-shared-host-mounts">Enable shared host mounts<a href="#enable-shared-host-mounts" class="hanchor" ariaLabel="Anchor">#</a></h2>
<p>From within the container, run:</p>
<div class="highlight"><pre tabindex="0" style="color:#d8dee9;background-color:#2e3440;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span><span style="color:#81a1c1">echo</span> <span style="color:#a3be8c">&#39;#!/bin/sh -e
</span></span></span><span style="display:flex;"><span><span style="color:#a3be8c">ln -s /dev/console /dev/kmsg
</span></span></span><span style="display:flex;"><span><span style="color:#a3be8c">mount --make-rshared /&#39;</span> &gt; /etc/rc.local
</span></span><span style="display:flex;"><span>chmod +x /etc/rc.local
</span></span><span style="display:flex;"><span>reboot
</span></span></code></pre></div><h2 id="install-k3s">Install K3s<a href="#install-k3s" class="hanchor" ariaLabel="Anchor">#</a></h2>
<p>One of the simplest ways to install K3s on a remote host is to use <a href="https://github.com/alexellis/k3sup">k3sup</a>.
Ensure that you supply a valid <code>CONTAINER_IP</code> and choose the <code>k3s-version</code> you prefer.
As of 2021/11, it is still defaulting to the 1.19 channel, so I overrode it to 1.22 for cgroup v2 support. See the published releases <a href="https://github.com/k3s-io/k3s/releases">here</a>.</p>
<div class="highlight"><pre tabindex="0" style="color:#d8dee9;background-color:#2e3440;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>ssh-copy-id root@$CONTAINER_IP
</span></span><span style="display:flex;"><span>k3sup install --ip $CONTAINER_IP --user root --k3s-version v1.22.3+k3s1
</span></span></code></pre></div><p>If all goes well, you should see a path to the <code>kubeconfig</code> generated. I moved this into <code>~/.kube/config</code> so that kubectl would read this by default.</p>
<h2 id="wrapping-up">Wrapping up<a href="#wrapping-up" class="hanchor" ariaLabel="Anchor">#</a></h2>
<p>Installing K3s in LXC on Proxmox works with a few tweaks to the default configuration. I later followed the Tekton&rsquo;s <a href="https://tekton.dev/docs/getting-started/">Getting Started</a> guide and was able to deploy it in a few commands.</p>
<div class="highlight"><pre tabindex="0" style="color:#d8dee9;background-color:#2e3440;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span><span style="color:#4c566a;font-weight:bold">$</span> kubectl get all --namespace tekton-pipelines
</span></span><span style="display:flex;"><span>NAME READY STATUS RESTARTS AGE
</span></span><span style="display:flex;"><span>pod/tekton-pipelines-webhook-8566ff9b6b-6rnh8 1/1 Running 1 (50m ago) 12h
</span></span><span style="display:flex;"><span>pod/tekton-dashboard-6bf858f977-qt4hr 1/1 Running 1 (50m ago) 11h
</span></span><span style="display:flex;"><span>pod/tekton-pipelines-controller-69fd7498d8-f57m4 1/1 Running 1 (50m ago) 12h
</span></span><span style="display:flex;"><span><span style="color:#bf616a">
</span></span></span><span style="display:flex;"><span><span style="color:#bf616a"></span>NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
</span></span><span style="display:flex;"><span>service/tekton-pipelines-controller ClusterIP 10.43.44.245 &lt;none&gt; 9090/TCP,8080/TCP 12h
</span></span><span style="display:flex;"><span>service/tekton-pipelines-webhook ClusterIP 10.43.183.242 &lt;none&gt; 9090/TCP,8008/TCP,443/TCP,8080/TCP 12h
</span></span><span style="display:flex;"><span>service/tekton-dashboard ClusterIP 10.43.87.97 &lt;none&gt; 9097/TCP 11h
</span></span><span style="display:flex;"><span><span style="color:#bf616a">
</span></span></span><span style="display:flex;"><span><span style="color:#bf616a"></span>NAME READY UP-TO-DATE AVAILABLE AGE
</span></span><span style="display:flex;"><span>deployment.apps/tekton-pipelines-webhook 1/1 1 1 12h
</span></span><span style="display:flex;"><span>deployment.apps/tekton-dashboard 1/1 1 1 11h
</span></span><span style="display:flex;"><span>deployment.apps/tekton-pipelines-controller 1/1 1 1 12h
</span></span><span style="display:flex;"><span><span style="color:#bf616a">
</span></span></span><span style="display:flex;"><span><span style="color:#bf616a"></span>NAME DESIRED CURRENT READY AGE
</span></span><span style="display:flex;"><span>replicaset.apps/tekton-pipelines-webhook-8566ff9b6b 1 1 1 12h
</span></span><span style="display:flex;"><span>replicaset.apps/tekton-dashboard-6bf858f977 1 1 1 11h
</span></span><span style="display:flex;"><span>replicaset.apps/tekton-pipelines-controller-69fd7498d8 1 1 1 12h
</span></span><span style="display:flex;"><span><span style="color:#bf616a">
</span></span></span><span style="display:flex;"><span><span style="color:#bf616a"></span>NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
</span></span><span style="display:flex;"><span>horizontalpodautoscaler.autoscaling/tekton-pipelines-webhook Deployment/tekton-pipelines-webhook 9%/100% 1 5 1 12h
</span></span></code></pre></div><p>I made sure to install Tailscale in the container so that I can easily access K3s from anywhere.</p>
<p>If I&rsquo;m feeling adventurous, I might experiment with <a href="https://rancher.com/docs/k3s/latest/en/advanced/#running-k3s-with-rootless-mode-experimental">K3s rootless</a>.</p>
</section>
<script
type="text/javascript"
src="https://storage.ko-fi.com/cdn/widget/Widget_2.js"
></script>
<script type="text/javascript">
kofiwidget2.init("Buy me a coffee", "#a15bc2", "F1F2S4LWI");
kofiwidget2.draw();
</script>
<div class="post-tags">
<nav class="nav tags">
<ul class="tags">
<li><a href="/tags/k3s">k3s</a></li>
<li><a href="/tags/proxmox">proxmox</a></li>
<li><a href="/tags/lxc">lxc</a></li>
<li><a href="/tags/self-hosted">self-hosted</a></li>
</ul>
</nav>
</div>
</article>
</main>
<section id='comments' class='comments'>
<div class='container sep-before'>
<div class='comments'><script>
var getTheme = window.localStorage && window.localStorage.getItem("theme-storage");
getTheme = getTheme == null ? 'light' : getTheme;
let theme = getTheme === 'dark' ? 'github-dark' : 'github-light';
let s = document.createElement('script');
s.src = 'https://utteranc.es/client.js';
s.setAttribute('repo', 'davegallant\/davegallant.github.io');
s.setAttribute('issue-term', 'pathname');
s.setAttribute('theme', theme);
s.setAttribute('crossorigin', 'anonymous');
s.setAttribute('async', '');
document.querySelector('div.comments').innerHTML = '';
document.querySelector('div.comments').appendChild(s);
</script>
</div>
</div>
</section><footer>
<script src="https://storage.ko-fi.com/cdn/scripts/overlay-widget.js"></script>
<div style="display: flex"><a class="soc" href="mailto:me@davegallant.ca" rel="me" title="Email"
><i data-feather="at-sign"></i
></a>
<a class="border"></a><a class="soc" href="https://linktr.ee/davegallant" rel="me" title="LinkTree"
><i data-feather="compass"></i
></a>
<a class="border"></a><a class="soc" href="https://github.com/davegallant" rel="me" title="GitHub"
><i data-feather="github"></i
></a>
<a class="border"></a><a class="soc" href="https://mastodon.social/@davegallant" rel="me" title="Mastodon"
><i data-feather="speaker"></i
></a>
<a class="border"></a><a class="soc" href="https://www.linkedin.com/in/dave-gallant/" rel="me" title="LinkedIn"
><i data-feather="linkedin"></i
></a>
<a class="border"></a></div>
<div class="footer-info">
2023 Dave Gallant
</div>
<script src="/js/copy-code-button.js"></script>
</footer>
<script async src="https://www.googletagmanager.com/gtag/js?id=G-V8WJDERTX9"></script>
<script>
var doNotTrack = false;
if (!doNotTrack) {
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'G-V8WJDERTX9', { 'anonymize_ip': false });
}
</script>
<script>
feather.replace();
</script></div>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink