diff --git a/ansible/kubernetes/README.md b/ansible/kubernetes/README.md new file mode 100644 index 0000000..0768604 --- /dev/null +++ b/ansible/kubernetes/README.md @@ -0,0 +1,97 @@ +# Ansible – Kubernetes nodes + +Prepares the Kubernetes VM nodes provisioned by the +[`kubernetes_vms` Terraform project](../../terraform/projects/kubernetes_vms/). + +## What it does + +The `k8s_node` playbook configures each node for `kubeadm`/`kubelet`: + +1. **System** – disables swap (runtime + fstab), loads `overlay` and `br_netfilter`, + persists kernel modules, applies the Kubernetes `sysctl` settings. +2. **Packages** – installs prerequisites, adds the official Kubernetes apt repo/key, + installs and holds `kubeadm` and `kubelet`. +3. **containerd** – installs it, generates the default config, enables the + `SystemdCgroup` driver, starts and enables the service. +4. **Firewall** – configures UFW per role: SSH + common Kubernetes ports from the admin + subnet, control-plane ports (`2379:2380`, `10257`, `10259`) on masters, worker ports + (`10256`, `30000:32767`) on workers, default deny incoming. + +Group membership (who is a `master` / `worker`) is resolved from the Terraform inventory, +so the role-specific firewall rules apply automatically. + +> Cluster bootstrap (`kubeadm init` / `kubeadm join`) is **not** part of this playbook +> and is done manually per the migration roadmap. + +## Inventory + +The inventory is generated from Terraform state via the `cloud.terraform` inventory +plugin: + +```yaml +# inventories/tf_k8s_vms.yml +--- +plugin: cloud.terraform.terraform_provider +project_path: + - /tmp/semaphore/project_1/repository_1_template_7/terraform/projects/kubernetes_vms +``` + +> The `project_path` is a Semaphore runtime checkout path. If you run locally, point it +> at your own `terraform/projects/kubernetes_vms` directory. The same `ansible` provider +> that writes the inventory is used by Terraform, so hosts/groups are always in sync with +> the applied VMs. + +## Requirements + +Install the collections referenced by `collections/requirements.yml`: + +```bash +ansible-galaxy collection install -r collections/requirements.yml +``` + +- `cloud.terraform` – Terraform inventory plugin +- `community.general` – `ufw` module + +## Usage + +```bash +# Connectivity check +ansible-playbook -i inventories/tf_k8s_vms.yml ping.yml + +# Prepare all Kubernetes nodes +ansible-playbook -i inventories/tf_k8s_vms.yml k8s_node.yml +``` + +Both playbooks target `all`; the inventory plugin runs from the Terraform project path. + +## Role variables + +Defined in `roles/k8s_node/defaults/main.yml` (override as needed): + +| Variable | Default | Description | +| ------------------------------- | ---------------------------- | -------------------------------------- | +| `k8s_version` | `1.37` | Kubernetes minor version | +| `k8s_repo` | `pkgs.k8s.io` core v | Kubernetes apt repository | +| `k8s_repo_key` | matching `Release.key` URL | Repository signing key | +| `k8s_packages` | `kubelet`, `kubeadm` | Packages installed and held | +| `k8s_prerequisite_packages` | `apt-transport-https`, `ca-certificates`, `curl`, `gnupg` | Build deps for the repo | +| `k8s_firewall_admin_subnet` | `192.168.66.0/24` | Subnet allowed through UFW | +| `k8s_firewall_ssh_port` | `22` | SSH port | +| `k8s_firewall_common_ports` | `6443`, `10250` | Ports open on all nodes | +| `k8s_firewall_master_ports` | `2379:2380`, `10257`, `10259` | Ports open on masters | +| `k8s_firewall_worker_ports` | `10256`, `30000:32767` | Ports open on workers | + +## File reference + +| Path | Purpose | +| ---------------------------------------- | ------------------------------------ | +| `k8s_node.yml` | Main playbook (role `k8s_node`) | +| `ping.yml` | Connectivity smoke-test playbook | +| `inventories/tf_k8s_vms.yml` | Terraform-backed inventory | +| `collections/requirements.yml` | Required Ansible collections | +| `roles/k8s_node/tasks/main.yml` | Task dispatcher | +| `roles/k8s_node/tasks/system.yml` | Swap, kernel modules, sysctl | +| `roles/k8s_node/tasks/packages.yml` | Kubernetes repo + kubeadm/kubelet | +| `roles/k8s_node/tasks/containerd.yml` | containerd install/config | +| `roles/k8s_node/tasks/firewall.yml` | Per-role UFW rules | +| `roles/k8s_node/defaults/main.yml` | Role defaults | \ No newline at end of file diff --git a/terraform/projects/kubernetes_vms/README.md b/terraform/projects/kubernetes_vms/README.md new file mode 100644 index 0000000..3e65e20 --- /dev/null +++ b/terraform/projects/kubernetes_vms/README.md @@ -0,0 +1,93 @@ +# Terraform – Kubernetes VMs + +Provisions the Kubernetes node VMs on Proxmox for the homelab Kubernetes cluster. + +## Overview + +This Terraform project clones the `ubuntu-noble-cloud` template into virtual machines +on the proxmox nodes, following a capacity-aware placement algorithm: + +1. **Control plane (master)** VMs are spread one per node, on the nodes with the most + free memory that still fit the role's `ram`. +2. **Pack roles (worker)** are water-filled over the remaining capacity, most-available + node first, without a per-node limit. + +Each VM is tagged with the environment (`vm_env`) and registered in a Terraform-backed +Ansible inventory (via the `ansible` provider), so the `k8s_node` playbook can target +them by role group. + +## Placement details + +- Online Proxmox nodes are ordered by available memory (most available first, node name + as tie-break). +- The control-plane role must fit entirely on distinct nodes with `free_mib >= ram`. +- Workers are packed into residual capacity (`free_mib - ram_per_cp_vm_on_node`), + again most-available first. +- Naming: `kub--m` for masters and `kub--w` for workers, e.g. + `kub-prod-m01`, `kub-prod-w01`. + +Deployment is blocked by `check` blocks when the fleet cannot be placed: + +- no online node is available, +- not enough nodes have the free memory required for the control-plane role, +- residual capacity cannot fit all worker VMs. + +## Prerequisites + +- Proxmox with: + - a cloneable template named `ubuntu-noble-cloud` (from the `pve_vm` module), + - a `ceph-workload` datastore, + - VLAN bridge `vlan` (default `vlan66`), + - an API token and an SSH user/password with rights to create VMs on the nodes. +- Terraform with the `bpg/proxmox` and `ansible/ansible` providers (init downloads them). + +## Input variables + +| Variable | Type | Required | Default | Description | +| ---------------------------- | -------- | -------- | ---------------------------- | -------------------------------------------------- | +| `proxmox_endpoint` | string | yes | – | Proxmox API URL | +| `proxmox_api_token` | string | yes | – | Proxmox API token (secret) | +| `proxmox_ssh_username` | string | yes | – | Proxmox SSH user for placement (secret) | +| `proxmox_ssh_password` | string | yes | – | Proxmox SSH password (secret) | +| `vm_env` | string | no | `"prod"` | Environment, part of the VM name and tags | +| `master_count` | number | no | `null` → `vm_attr` default | Number of master VMs | +| `master_cpu` | number | no | `null` → `vm_attr` default | vCPU cores for masters | +| `master_ram` | number | no | `null` → `vm_attr` default | Master RAM in **GiB** (converted to MiB internally)| +| `master_vlan` | number | no | `null` → `vm_attr` default | VLAN for masters | +| `worker_count` | number | no | `null` → `vm_attr` default | Number of worker VMs | +| `worker_cpu` | number | no | `null` → `vm_attr` default | vCPU cores for workers | +| `worker_ram` | number | no | `null` → `vm_attr` default | Worker RAM in **GiB** | +| `worker_vlan` | number | no | `null` → `vm_attr` default | VLAN for workers | +| `vm_attr` | map(… ) | no | see below | Full per-role VM defaults | + +The `vm_attr` map holds the fallback attributes used when the corresponding +`master_*`/`worker_*` override is `null`: + +```hcl +vm_attr = { + master = { ram = 2048, cpu = 2, vlan = 66, count = 3 } + worker = { ram = 4096, cpu = 2, vlan = 66, count = 3 } +} +``` + +> `ram` values inside `vm_attr` are MiB; the `master_ram` / `worker_ram` overrides are +> set in GiB and converted internally (`× 1024`). The placement math always works in MiB. + +## Usage + +```bash +terraform init +terraform plan +terraform apply +``` + +The `vm_ip` output maps VM name to assigned IP and feeds the Ansible inventory +(`inventories/`) of the `k8s_node` playbook. + +## File reference + +| File | Purpose | +| ------------ | --------------------------------------------------------- | +| `main.tf` | Placement algorithm, `pve_vm` module wiring, checks | +| `variables.tf` | Clusters, Proxmox credentials, per-role defaults/overrides | +| `provider.tf` | `proxmox` and `ansible` provider configuration | \ No newline at end of file