diff --git a/.envrc b/.envrc new file mode 100644 index 0000000..1d953f4 --- /dev/null +++ b/.envrc @@ -0,0 +1 @@ +use nix diff --git a/.gitignore b/.gitignore index 1bfde2a..99d1c92 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,6 @@ controlplane.yaml worker.yaml talosconfig patched.yaml +secrets.yaml +keys/ +.direnv/ diff --git a/README.md b/README.md index d0614a2..d68c810 100644 --- a/README.md +++ b/README.md @@ -2,54 +2,90 @@ > Following [https://www.talos.dev/v1.7](https://www.talos.dev/v1.7) +Also following the guide: +> https://blog.dalydays.com/post/kubernetes-homelab-series-part-1-talos-linux-proxmox/ + + ## Configuration layout -The main talos configuration is in the `controlplane.yaml` and `worker.yaml` files. They contain the configuration for the controlplane and worker nodes as well as the certificates and keys for the cluster. **They cannot be checked into version control**. +The bulk of the talos configuration is left as default. Only select patches are applied. -Luckily, the boilerplate configuration can be generated with `talosctl gen config ...` and we are not deviating significantly from the defaults. We will only need small patches as described below. - -- The main configuration is given by the `controlplane.yaml`, `worker.yaml` and `talosconfig` files. They need to be generated by `talosctl`. -- The patches are in the `patch` directory. - - Patches applied to **all** nodes are in `common.yaml`. - - Patches applied +The configuration is generated and output to the `controlplane.yaml` and `worker.yaml` files. They contain the configuration for the controlplane and worker nodes as well as the certificates and keys for the cluster. **They cannot be checked into version control**. -## High-availability setup -For this setup we want to use a virtual IP (VIP) for the controlplane nodes. This only works once etcd is running on at least one of the controlplane nodes. +### Patches +For patches we differentiate between: +- those that are applied to all (all controlplane) nodes +- those that are applied to particular nodes -- Prerequisites: - ``` - export TALOSCONFIG="talosconfig" +We also differentiate: +- patches that are required at install-time +- patches that can be applied after installation. + + +## Setup + +> For this setup we want to use a virtual IP (VIP) for the controlplane nodes. +> Note that this IP is only available if etcd was running to begin with. Meaning it can be used by kubectl but should not be used by talosctl itself. + +1. Generate the install media with the addons we want, by visiting https://factory.talos.dev/ + +1. Choose a virtual controle plane ip. Nodes will auto-negotiate who actually uses that ip. + +1. Generate the required files to setup the talos cluster: + ```bash + talosctl gen secrets + talosctl gen config https://:6443 --with-secrets secrets.yaml --output-dir _out --install-image + + # in this case: + talosctl gen config kluster https://192.168.5.0:6443 --with-secrets secrets.yaml --output-dir _out --install-image factory.talos.dev/metal-installer/235b109dafe508dc2e3329cacf7735c8f9154f433a04da96e1bb4b70b1437b49:v1.11.0 ``` -- Setup a cluster on one node. I chose a vm node. - ``` - talosctl gen config kluster https://192.168.1.199:6443 - ``` - ``` - talosctl machineconfig patch controlplane.yaml --patch @patch/common.yaml --patch @patch/controlplane.vm.yaml > patched.yaml - talosctl apply-config --insecure --nodes 192.168.1.199 --file patched.yaml - - talosctl config endpoint 192.168.1.199 - talosctl config node 192.168.1.199 - talosctl bootstrap +1. This also generates a `talosconfig` at `_out` which I don't want to manually reference. + ```bash + EXPORT TALOSCONFIG=_out/talosconfig ``` -- Now modify the controlplane configuration to use a VIP, by changing all occurrences of the IP address (...199) to the VIP (...222). +1. Install talos. Since my VMs are on different hypervisors their disks have different names. So I apply a per-node patch at install time: + ```bash + talosctl apply-config --insecure --file _out/controlplane.yaml --nodes -p @ -- Apply the configuration to the rest of the controlplane nodes. - ``` - talosctl machineconfig patch controlplane.yaml --patch @patch/common.yaml --patch @patch/controlplane.vm.yaml > patched.yaml - talosctl apply-config --insecure --nodes 192.168.1.201 --file patched.yaml - # For the second node as well - talosctl machineconfig patch controlplane.yaml --patch @patch/common.yaml --patch @patch/controlplane.metal.yaml > patched.yaml - talosctl apply-config --insecure --nodes 192.168.1.108 --file patched.yaml + # in this case + talosctl apply-config --insecure --file _out/controlplane.yaml --nodes 192.168.5.1 -p @patch/controlplane.proxmox.yaml + talosctl apply-config --insecure --file _out/controlplane.yaml --nodes 192.168.5.2 -p @patch/controlplane.xoa.yaml + talosctl apply-config --insecure --file _out/controlplane.yaml --nodes 192.168.5.3 -p @patch/controlplane.xoa.yaml ``` -- Also add the new nodes to the talosctl configuration, by manually editing the `endpoints` and `nodes` sections in the `talosconfig` file. - -- Finally, get the kubeconfig file. - ``` - talosctl kubeconfig . +1. We are now ready to use all these nodes and not specify `--node` every time + ```bash + talosctl config endpoint 192.168.5.1 192.168.5.2 192.168.5.3 + talosctl config node 192.168.5.1 192.168.5.2 192.168.5.3 ``` -- Proceed to cluster bootstrapping. +1. Now we can apply the late-stage patches (common and individual) + ```bash + talosctl patch mc --patch @ --node + + # in this case + talosctl patch mc --patch @patch/common.yaml + # no individual patches + ``` + +1. Bootstrap the kubernetes cluster **on a single node** + ```bash + talosctl bootstrap -n + + # in this case + talosctl bootstrap -n 192.168.5.1 + ``` + +1. Get the `kubeconfig`: + ```bash + talosctl kubeconfig --node + + # in this case + talosctl kubeconfig -n 192.168.5.1 + + ``` + +1. Proceed to apps bootstrap: https://git.kluster.moll.re/remoll/k3s-infra + diff --git a/default.nix b/default.nix new file mode 100644 index 0000000..899a6bc --- /dev/null +++ b/default.nix @@ -0,0 +1,10 @@ +{ pkgs ? import {} }: + +pkgs.mkShell { + buildInputs = [ + pkgs.talosctl + pkgs.yq + ]; + # set env + TALOSCONFIG="talosconfig"; +} diff --git a/patch/common.yaml b/patch/common.yaml index 6af3f29..65a3f98 100644 --- a/patch/common.yaml +++ b/patch/common.yaml @@ -3,10 +3,10 @@ machine: interfaces: # we select the first interface and give it a virtual IP. - deviceSelector: - busPath: "0*" + physical: true dhcp: true vip: - ip: 192.168.1.222 + ip: 192.168.5.0 nameservers: - 8.8.8.8 @@ -15,4 +15,3 @@ machine: cluster: allowSchedulingOnControlPlanes: true - diff --git a/patch/controlplane.vm.yaml b/patch/controlplane.proxmox.yaml similarity index 98% rename from patch/controlplane.vm.yaml rename to patch/controlplane.proxmox.yaml index 0413abf..11828a2 100644 --- a/patch/controlplane.vm.yaml +++ b/patch/controlplane.proxmox.yaml @@ -4,4 +4,3 @@ machine: nodeLabels: machineType: vm - diff --git a/patch/controlplane.xoa.yaml b/patch/controlplane.xoa.yaml new file mode 100644 index 0000000..636f6f0 --- /dev/null +++ b/patch/controlplane.xoa.yaml @@ -0,0 +1,7 @@ +machine: + install: + disk: /dev/xvda + + nodeLabels: + machineType: vm +