nix ready, cleaner setup

.envrc

@@ -0,0 +1 @@
+use nix

.gitignore

@@ -2,3 +2,6 @@ controlplane.yaml
 worker.yaml
 talosconfig
 patched.yaml
+secrets.yaml
+keys/
+.direnv/

README.md

@@ -2,54 +2,90 @@
 
 > Following [https://www.talos.dev/v1.7](https://www.talos.dev/v1.7)
 
+Also following the guide:
+> https://blog.dalydays.com/post/kubernetes-homelab-series-part-1-talos-linux-proxmox/
+
+
 ## Configuration layout
-The main talos configuration is in the `controlplane.yaml` and `worker.yaml` files. They contain the configuration for the controlplane and worker nodes as well as the certificates and keys for the cluster. **They cannot be checked into version control**.
+The bulk of the talos configuration is left as default. Only select patches are applied.
 
-Luckily, the boilerplate configuration can be generated with `talosctl gen config ...` and we are not deviating significantly from the defaults. We will only need small patches as described below.
+The configuration is generated and output to the `controlplane.yaml` and `worker.yaml` files. They contain the configuration for the controlplane and worker nodes as well as the certificates and keys for the cluster. **They cannot be checked into version control**.
-
-- The main configuration is given by the `controlplane.yaml`, `worker.yaml` and `talosconfig` files. They need to be generated by `talosctl`.
-- The patches are in the `patch` directory.
-    - Patches applied to **all** nodes are in `common.yaml`.
-    - Patches applied
 
 
-## High-availability setup
+### Patches
-For this setup we want to use a virtual IP (VIP) for the controlplane nodes. This only works once etcd is running on at least one of the controlplane nodes.
+For patches we differentiate between:
+- those that are applied to all (all controlplane) nodes
+- those that are applied to particular nodes
 
-- Prerequisites:
+We also differentiate:
-    ```
+- patches that are required at install-time
-    export TALOSCONFIG="talosconfig"
+- patches that can be applied after installation.
+
+
+## Setup
+
+> For this setup we want to use a virtual IP (VIP) for the controlplane nodes.
+> Note that this IP is only available if etcd was running to begin with. Meaning it can be used by kubectl but should not be used by talosctl itself.
+
+1. Generate the install media with the addons we want, by visiting https://factory.talos.dev/
+
+1. Choose a virtual controle plane ip. Nodes will auto-negotiate who actually uses that ip.
+
+1. Generate the required files to setup the talos cluster:
+    ```bash
+    talosctl gen secrets
+    talosctl gen config <CLUSTER_NAME> https://<CONTROL_PLANE_IP>:6443 --with-secrets secrets.yaml --output-dir _out --install-image <FACTORY_IMAGE>
+
+    # in this case:
+    talosctl gen config kluster https://192.168.5.0:6443 --with-secrets secrets.yaml --output-dir _out --install-image  factory.talos.dev/metal-installer/235b109dafe508dc2e3329cacf7735c8f9154f433a04da96e1bb4b70b1437b49:v1.11.0
     ```
 
-- Setup a cluster on one node. I chose a vm node.
+1. This also generates a `talosconfig` at `_out` which I don't want to manually reference.
-    ```
+    ```bash
-    talosctl gen config kluster https://192.168.1.199:6443
+    EXPORT TALOSCONFIG=_out/talosconfig
-    ```
-    ```
-    talosctl machineconfig patch controlplane.yaml --patch @patch/common.yaml --patch @patch/controlplane.vm.yaml > patched.yaml
-    talosctl apply-config --insecure --nodes 192.168.1.199 --file patched.yaml
-
-    talosctl config endpoint 192.168.1.199
-    talosctl config node 192.168.1.199
-    talosctl bootstrap
     ```
 
-- Now modify the controlplane configuration to use a VIP, by changing all occurrences of the IP address (...199) to the VIP (...222).
+1. Install talos. Since my VMs are on different hypervisors their disks have different names. So I apply a per-node patch at install time:
+    ```bash
+    talosctl apply-config --insecure --file _out/controlplane.yaml --nodes <NODE_IP> -p @<PATCH_FILE>
 
-- Apply the configuration to the rest of the controlplane nodes.
+    # in this case
-    ```
+    talosctl apply-config --insecure --file _out/controlplane.yaml --nodes 192.168.5.1 -p @patch/controlplane.proxmox.yaml
-    talosctl machineconfig patch controlplane.yaml --patch @patch/common.yaml --patch @patch/controlplane.vm.yaml > patched.yaml
+    talosctl apply-config --insecure --file _out/controlplane.yaml --nodes 192.168.5.2 -p @patch/controlplane.xoa.yaml
-    talosctl apply-config --insecure --nodes 192.168.1.201 --file patched.yaml
+    talosctl apply-config --insecure --file _out/controlplane.yaml --nodes 192.168.5.3 -p @patch/controlplane.xoa.yaml
-    # For the second node as well
-    talosctl machineconfig patch controlplane.yaml --patch @patch/common.yaml --patch @patch/controlplane.metal.yaml > patched.yaml
-    talosctl apply-config --insecure --nodes 192.168.1.108 --file patched.yaml
     ```
 
-- Also add the new nodes to the talosctl configuration, by manually editing the `endpoints` and `nodes` sections in the `talosconfig` file.
+1. We are now ready to use all these nodes and not specify `--node` every time
-
+    ```bash
-- Finally, get the kubeconfig file.
+    talosctl config endpoint 192.168.5.1 192.168.5.2 192.168.5.3
-    ```
+    talosctl config node 192.168.5.1 192.168.5.2 192.168.5.3
-    talosctl kubeconfig .
     ```
 
-- Proceed to cluster bootstrapping.
+1. Now we can apply the late-stage patches (common and individual)
+    ```bash
+    talosctl patch mc --patch @<PATCH_FILE> --node <NODE_IP (OPTIONAL)>
+
+    # in this case
+    talosctl patch mc --patch @patch/common.yaml
+    # no individual patches
+    ```
+
+1. Bootstrap the kubernetes cluster **on a single node**
+    ```bash
+    talosctl bootstrap -n <NODE_IP>
+
+    # in this case
+    talosctl bootstrap -n 192.168.5.1
+    ```
+
+1. Get the `kubeconfig`:
+    ```bash
+    talosctl kubeconfig --node <NODE_IP>
+
+    # in this case
+    talosctl kubeconfig -n 192.168.5.1
+
+    ```
+
+1. Proceed to apps bootstrap: https://git.kluster.moll.re/remoll/k3s-infra
+

default.nix

@@ -0,0 +1,10 @@
+{ pkgs ? import <nixpkgs> {} }:
+
+pkgs.mkShell {
+  buildInputs = [
+    pkgs.talosctl
+    pkgs.yq
+  ];
+  # set env
+  TALOSCONFIG="talosconfig";
+}

patch/common.yaml

@@ -3,10 +3,10 @@ machine:
     interfaces:
       # we select the first interface and give it a virtual IP.
       - deviceSelector:
-          busPath: "0*"
+          physical: true
         dhcp: true
         vip:
-          ip: 192.168.1.222
+          ip: 192.168.5.0
 
     nameservers:
       - 8.8.8.8
@@ -15,4 +15,3 @@ machine:
 
 cluster:
   allowSchedulingOnControlPlanes: true
-

patch/controlplane.proxmox.yaml

@@ -4,4 +4,3 @@ machine:
 
   nodeLabels:
     machineType: vm
-

patch/controlplane.xoa.yaml

@@ -0,0 +1,7 @@
+machine:
+  install:
+    disk: /dev/xvda
+
+  nodeLabels:
+    machineType: vm
+