docs: create README.me
This commit is contained in:
@@ -0,0 +1,97 @@
|
||||
# Ansible – Kubernetes nodes
|
||||
|
||||
Prepares the Kubernetes VM nodes provisioned by the
|
||||
[`kubernetes_vms` Terraform project](../../terraform/projects/kubernetes_vms/).
|
||||
|
||||
## What it does
|
||||
|
||||
The `k8s_node` playbook configures each node for `kubeadm`/`kubelet`:
|
||||
|
||||
1. **System** – disables swap (runtime + fstab), loads `overlay` and `br_netfilter`,
|
||||
persists kernel modules, applies the Kubernetes `sysctl` settings.
|
||||
2. **Packages** – installs prerequisites, adds the official Kubernetes apt repo/key,
|
||||
installs and holds `kubeadm` and `kubelet`.
|
||||
3. **containerd** – installs it, generates the default config, enables the
|
||||
`SystemdCgroup` driver, starts and enables the service.
|
||||
4. **Firewall** – configures UFW per role: SSH + common Kubernetes ports from the admin
|
||||
subnet, control-plane ports (`2379:2380`, `10257`, `10259`) on masters, worker ports
|
||||
(`10256`, `30000:32767`) on workers, default deny incoming.
|
||||
|
||||
Group membership (who is a `master` / `worker`) is resolved from the Terraform inventory,
|
||||
so the role-specific firewall rules apply automatically.
|
||||
|
||||
> Cluster bootstrap (`kubeadm init` / `kubeadm join`) is **not** part of this playbook
|
||||
> and is done manually per the migration roadmap.
|
||||
|
||||
## Inventory
|
||||
|
||||
The inventory is generated from Terraform state via the `cloud.terraform` inventory
|
||||
plugin:
|
||||
|
||||
```yaml
|
||||
# inventories/tf_k8s_vms.yml
|
||||
---
|
||||
plugin: cloud.terraform.terraform_provider
|
||||
project_path:
|
||||
- /tmp/semaphore/project_1/repository_1_template_7/terraform/projects/kubernetes_vms
|
||||
```
|
||||
|
||||
> The `project_path` is a Semaphore runtime checkout path. If you run locally, point it
|
||||
> at your own `terraform/projects/kubernetes_vms` directory. The same `ansible` provider
|
||||
> that writes the inventory is used by Terraform, so hosts/groups are always in sync with
|
||||
> the applied VMs.
|
||||
|
||||
## Requirements
|
||||
|
||||
Install the collections referenced by `collections/requirements.yml`:
|
||||
|
||||
```bash
|
||||
ansible-galaxy collection install -r collections/requirements.yml
|
||||
```
|
||||
|
||||
- `cloud.terraform` – Terraform inventory plugin
|
||||
- `community.general` – `ufw` module
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Connectivity check
|
||||
ansible-playbook -i inventories/tf_k8s_vms.yml ping.yml
|
||||
|
||||
# Prepare all Kubernetes nodes
|
||||
ansible-playbook -i inventories/tf_k8s_vms.yml k8s_node.yml
|
||||
```
|
||||
|
||||
Both playbooks target `all`; the inventory plugin runs from the Terraform project path.
|
||||
|
||||
## Role variables
|
||||
|
||||
Defined in `roles/k8s_node/defaults/main.yml` (override as needed):
|
||||
|
||||
| Variable | Default | Description |
|
||||
| ------------------------------- | ---------------------------- | -------------------------------------- |
|
||||
| `k8s_version` | `1.37` | Kubernetes minor version |
|
||||
| `k8s_repo` | `pkgs.k8s.io` core v<ver> | Kubernetes apt repository |
|
||||
| `k8s_repo_key` | matching `Release.key` URL | Repository signing key |
|
||||
| `k8s_packages` | `kubelet`, `kubeadm` | Packages installed and held |
|
||||
| `k8s_prerequisite_packages` | `apt-transport-https`, `ca-certificates`, `curl`, `gnupg` | Build deps for the repo |
|
||||
| `k8s_firewall_admin_subnet` | `192.168.66.0/24` | Subnet allowed through UFW |
|
||||
| `k8s_firewall_ssh_port` | `22` | SSH port |
|
||||
| `k8s_firewall_common_ports` | `6443`, `10250` | Ports open on all nodes |
|
||||
| `k8s_firewall_master_ports` | `2379:2380`, `10257`, `10259` | Ports open on masters |
|
||||
| `k8s_firewall_worker_ports` | `10256`, `30000:32767` | Ports open on workers |
|
||||
|
||||
## File reference
|
||||
|
||||
| Path | Purpose |
|
||||
| ---------------------------------------- | ------------------------------------ |
|
||||
| `k8s_node.yml` | Main playbook (role `k8s_node`) |
|
||||
| `ping.yml` | Connectivity smoke-test playbook |
|
||||
| `inventories/tf_k8s_vms.yml` | Terraform-backed inventory |
|
||||
| `collections/requirements.yml` | Required Ansible collections |
|
||||
| `roles/k8s_node/tasks/main.yml` | Task dispatcher |
|
||||
| `roles/k8s_node/tasks/system.yml` | Swap, kernel modules, sysctl |
|
||||
| `roles/k8s_node/tasks/packages.yml` | Kubernetes repo + kubeadm/kubelet |
|
||||
| `roles/k8s_node/tasks/containerd.yml` | containerd install/config |
|
||||
| `roles/k8s_node/tasks/firewall.yml` | Per-role UFW rules |
|
||||
| `roles/k8s_node/defaults/main.yml` | Role defaults |
|
||||
@@ -0,0 +1,93 @@
|
||||
# Terraform – Kubernetes VMs
|
||||
|
||||
Provisions the Kubernetes node VMs on Proxmox for the homelab Kubernetes cluster.
|
||||
|
||||
## Overview
|
||||
|
||||
This Terraform project clones the `ubuntu-noble-cloud` template into virtual machines
|
||||
on the proxmox nodes, following a capacity-aware placement algorithm:
|
||||
|
||||
1. **Control plane (master)** VMs are spread one per node, on the nodes with the most
|
||||
free memory that still fit the role's `ram`.
|
||||
2. **Pack roles (worker)** are water-filled over the remaining capacity, most-available
|
||||
node first, without a per-node limit.
|
||||
|
||||
Each VM is tagged with the environment (`vm_env`) and registered in a Terraform-backed
|
||||
Ansible inventory (via the `ansible` provider), so the `k8s_node` playbook can target
|
||||
them by role group.
|
||||
|
||||
## Placement details
|
||||
|
||||
- Online Proxmox nodes are ordered by available memory (most available first, node name
|
||||
as tie-break).
|
||||
- The control-plane role must fit entirely on distinct nodes with `free_mib >= ram`.
|
||||
- Workers are packed into residual capacity (`free_mib - ram_per_cp_vm_on_node`),
|
||||
again most-available first.
|
||||
- Naming: `kub-<env>-m<NN>` for masters and `kub-<env>-w<NN>` for workers, e.g.
|
||||
`kub-prod-m01`, `kub-prod-w01`.
|
||||
|
||||
Deployment is blocked by `check` blocks when the fleet cannot be placed:
|
||||
|
||||
- no online node is available,
|
||||
- not enough nodes have the free memory required for the control-plane role,
|
||||
- residual capacity cannot fit all worker VMs.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Proxmox with:
|
||||
- a cloneable template named `ubuntu-noble-cloud` (from the `pve_vm` module),
|
||||
- a `ceph-workload` datastore,
|
||||
- VLAN bridge `vlan<id>` (default `vlan66`),
|
||||
- an API token and an SSH user/password with rights to create VMs on the nodes.
|
||||
- Terraform with the `bpg/proxmox` and `ansible/ansible` providers (init downloads them).
|
||||
|
||||
## Input variables
|
||||
|
||||
| Variable | Type | Required | Default | Description |
|
||||
| ---------------------------- | -------- | -------- | ---------------------------- | -------------------------------------------------- |
|
||||
| `proxmox_endpoint` | string | yes | – | Proxmox API URL |
|
||||
| `proxmox_api_token` | string | yes | – | Proxmox API token (secret) |
|
||||
| `proxmox_ssh_username` | string | yes | – | Proxmox SSH user for placement (secret) |
|
||||
| `proxmox_ssh_password` | string | yes | – | Proxmox SSH password (secret) |
|
||||
| `vm_env` | string | no | `"prod"` | Environment, part of the VM name and tags |
|
||||
| `master_count` | number | no | `null` → `vm_attr` default | Number of master VMs |
|
||||
| `master_cpu` | number | no | `null` → `vm_attr` default | vCPU cores for masters |
|
||||
| `master_ram` | number | no | `null` → `vm_attr` default | Master RAM in **GiB** (converted to MiB internally)|
|
||||
| `master_vlan` | number | no | `null` → `vm_attr` default | VLAN for masters |
|
||||
| `worker_count` | number | no | `null` → `vm_attr` default | Number of worker VMs |
|
||||
| `worker_cpu` | number | no | `null` → `vm_attr` default | vCPU cores for workers |
|
||||
| `worker_ram` | number | no | `null` → `vm_attr` default | Worker RAM in **GiB** |
|
||||
| `worker_vlan` | number | no | `null` → `vm_attr` default | VLAN for workers |
|
||||
| `vm_attr` | map(… ) | no | see below | Full per-role VM defaults |
|
||||
|
||||
The `vm_attr` map holds the fallback attributes used when the corresponding
|
||||
`master_*`/`worker_*` override is `null`:
|
||||
|
||||
```hcl
|
||||
vm_attr = {
|
||||
master = { ram = 2048, cpu = 2, vlan = 66, count = 3 }
|
||||
worker = { ram = 4096, cpu = 2, vlan = 66, count = 3 }
|
||||
}
|
||||
```
|
||||
|
||||
> `ram` values inside `vm_attr` are MiB; the `master_ram` / `worker_ram` overrides are
|
||||
> set in GiB and converted internally (`× 1024`). The placement math always works in MiB.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
terraform init
|
||||
terraform plan
|
||||
terraform apply
|
||||
```
|
||||
|
||||
The `vm_ip` output maps VM name to assigned IP and feeds the Ansible inventory
|
||||
(`inventories/`) of the `k8s_node` playbook.
|
||||
|
||||
## File reference
|
||||
|
||||
| File | Purpose |
|
||||
| ------------ | --------------------------------------------------------- |
|
||||
| `main.tf` | Placement algorithm, `pve_vm` module wiring, checks |
|
||||
| `variables.tf` | Clusters, Proxmox credentials, per-role defaults/overrides |
|
||||
| `provider.tf` | `proxmox` and `ansible` provider configuration |
|
||||
Reference in New Issue
Block a user