Vezpi/Homelab
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat: deploy k8s vms with Terraform and Ansible(#8)

Browse Source 
Reviewed-on: #8
This commit was merged in pull request #8.
This commit is contained in:
Vezpi
2026-09-18 15:37:00 +02:00
parent 6470af9bfd
commit dfadd55977
15 changed files with 783 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
terraform/projects/kubernetes_vms/README.md
+93
View File
@@ -0,0 +1,93 @@
# Terraform – Kubernetes VMs
Provisions the Kubernetes node VMs on Proxmox for the homelab Kubernetes cluster.
## Overview
This Terraform project clones the `ubuntu-noble-cloud` template into virtual machines
on the proxmox nodes, following a capacity-aware placement algorithm:
1. **Control plane (master)** VMs are spread one per node, on the nodes with the most
free memory that still fit the role's `ram`.
2. **Pack roles (worker)** are water-filled over the remaining capacity, most-available
node first, without a per-node limit.
Each VM is tagged with the environment (`vm_env`) and registered in a Terraform-backed
Ansible inventory (via the `ansible` provider), so the `k8s_node` playbook can target
them by role group.
## Placement details
- Online Proxmox nodes are ordered by available memory (most available first, node name
as tie-break).
- The control-plane role must fit entirely on distinct nodes with `free_mib >= ram`.
- Workers are packed into residual capacity (`free_mib - ram_per_cp_vm_on_node`),
again most-available first.
- Naming: `kub-<env>-m<NN>` for masters and `kub-<env>-w<NN>` for workers, e.g.
`kub-prod-m01`, `kub-prod-w01`.
Deployment is blocked by `check` blocks when the fleet cannot be placed:
- no online node is available,
- not enough nodes have the free memory required for the control-plane role,
- residual capacity cannot fit all worker VMs.
## Prerequisites
- Proxmox with:
- a cloneable template named `ubuntu-noble-cloud` (from the `pve_vm` module),
- a `ceph-workload` datastore,
- VLAN bridge `vlan<id>` (default `vlan66`),
- an API token and an SSH user/password with rights to create VMs on the nodes.
- Terraform with the `bpg/proxmox` and `ansible/ansible` providers (init downloads them).
## Input variables
| Variable | Type | Required | Default | Description |
| ---------------------------- | -------- | -------- | ---------------------------- | -------------------------------------------------- |
| `proxmox_endpoint` | string | yes | – | Proxmox API URL |
| `proxmox_api_token` | string | yes | – | Proxmox API token (secret) |
| `proxmox_ssh_username` | string | yes | – | Proxmox SSH user for placement (secret) |
| `proxmox_ssh_password` | string | yes | – | Proxmox SSH password (secret) |
| `vm_env` | string | no | `"prod"` | Environment, part of the VM name and tags |
| `master_count` | number | no | `null` → `vm_attr` default | Number of master VMs |
| `master_cpu` | number | no | `null` → `vm_attr` default | vCPU cores for masters |
| `master_ram` | number | no | `null` → `vm_attr` default | Master RAM in **GiB** (converted to MiB internally)|
| `master_vlan` | number | no | `null` → `vm_attr` default | VLAN for masters |
| `worker_count` | number | no | `null` → `vm_attr` default | Number of worker VMs |
| `worker_cpu` | number | no | `null` → `vm_attr` default | vCPU cores for workers |
| `worker_ram` | number | no | `null` → `vm_attr` default | Worker RAM in **GiB** |
| `worker_vlan` | number | no | `null` → `vm_attr` default | VLAN for workers |
| `vm_attr` | map(… ) | no | see below | Full per-role VM defaults |
The `vm_attr` map holds the fallback attributes used when the corresponding
`master_*`/`worker_*` override is `null`:
```hcl
vm_attr = {
master = { ram = 2048, cpu = 2, vlan = 66, count = 3 }
worker = { ram = 4096, cpu = 2, vlan = 66, count = 3 }
}
```
> `ram` values inside `vm_attr` are MiB; the `master_ram` / `worker_ram` overrides are
> set in GiB and converted internally (`× 1024`). The placement math always works in MiB.
## Usage
```bash
terraform init
terraform plan
terraform apply
```
The `vm_ip` output maps VM name to assigned IP and feeds the Ansible inventory
(`inventories/`) of the `k8s_node` playbook.
## File reference
| File | Purpose |
| ------------ | --------------------------------------------------------- |
| `main.tf` | Placement algorithm, `pve_vm` module wiring, checks |
| `variables.tf` | Clusters, Proxmox credentials, per-role defaults/overrides |
| `provider.tf` | `proxmox` and `ansible` provider configuration |
terraform/projects/kubernetes_vms/main.tf
+198
View File
@@ -0,0 +1,198 @@
module "pve_vm" {
source = "../../modules/pve_vm"
for_each = local.vm_list
node_name = each.value.node_name
vm_name = each.key
vm_cpu = each.value.vm_cpu
vm_ram = each.value.vm_ram
vm_vlan = each.value.vm_vlan
vm_tags = ["${var.vm_env}"]
}
locals {
node_stats = [
for i, name in data.proxmox_virtual_environment_nodes.pve_nodes.names : {
name = name
free_mib = floor(data.proxmox_virtual_environment_nodes.pve_nodes.memory_available[i] / 1024 / 1024)
}
if data.proxmox_virtual_environment_nodes.pve_nodes.online[i]
]
max_free_mib = length(local.node_stats) > 0 ? max([for node in local.node_stats : node.free_mib]...) : 0
# Online nodes ordered by available memory, most available first (name as tie-break)
node_order = [
for key in sort([
for node in local.node_stats : "${format("%012d", local.max_free_mib - node.free_mib)}|${node.name}"
]) : split("|", key)[1]
]
node_free = { for node in local.node_stats : node.name => node.free_mib }
# Per-role simple overrides merged over vm_attr defaults (null keeps the default)
role_overrides = {
for role, cfg in var.vm_attr : role => lookup({
"master" = { count = var.master_count, ram = var.master_ram != null ? var.master_ram * 1024 : null, cpu = var.master_cpu, vlan = var.master_vlan }
"worker" = { count = var.worker_count, ram = var.worker_ram != null ? var.worker_ram * 1024 : null, cpu = var.worker_cpu, vlan = var.worker_vlan }
}, role, {})
}
# Internal placement rule: the control-plane role (master) sits one per node, pack roles are unlimited
cp_role_name = "master"
role_max_per_node = { for role, cfg in var.vm_attr : role => role == local.cp_role_name ? 1 : 0 }
effective_vm_attr = {
for role, cfg in var.vm_attr : role => merge(cfg, {
for attr, value in local.role_overrides[role] : attr => value if value != null
}, { max_per_node = local.role_max_per_node[role] })
}
# Control plane (max_per_node = 1): one VM per node, only on nodes with enough free memory
cp_role = [for role, cfg in local.effective_vm_attr : role if cfg.max_per_node == 1][0]
cp_attr = local.effective_vm_attr[local.cp_role]
cp_feasible = [
for node in local.node_order : node
if local.node_free[node] >= local.cp_attr.ram
]
cp_nodes = length(local.cp_feasible) >= local.cp_attr.count ? [
for i in range(local.cp_attr.count) : local.cp_feasible[i]
] : []
cp_hosts = toset(local.cp_nodes)
# Residual capacity per node after control-plane VMs are reserved
residual_mib = {
for node in local.node_order :
node => local.node_free[node] - (contains(local.cp_hosts, node) ? local.cp_attr.ram : 0)
}
max_residual_mib = length(local.node_order) > 0 ? max([for node in local.node_order : local.residual_mib[node]]...) : 0
# Remaining roles (max_per_node != 1): pack nodes with residual capacity, most available first
pack_roles = [for role, cfg in local.effective_vm_attr : role if cfg.max_per_node != 1]
pack_slots = {
for role in local.pack_roles : role => {
for node in local.node_order :
node => min(
floor(local.residual_mib[node] / local.effective_vm_attr[role].ram),
local.effective_vm_attr[role].max_per_node == 0
? floor(local.residual_mib[node] / local.effective_vm_attr[role].ram)
: local.effective_vm_attr[role].max_per_node
)
}
}
pack_precedence = {
for role in local.pack_roles : role => [
for key in sort([
for node in local.node_order :
"${format("%012d", local.max_residual_mib - local.residual_mib[node])}|${node}"
if local.pack_slots[role][node] >= 1
]) : split("|", key)[1]
]
}
pack_slots_list = {
for role in local.pack_roles : role =>
[for node in local.pack_precedence[role] : local.pack_slots[role][node]]
}
# Water-fill: each node takes VMs until it runs out of capacity or demand is met
pack_placed = {
for role in local.pack_roles : role => [
for i in range(length(local.pack_precedence[role])) :
min(local.pack_slots_list[role][i], max(0, local.effective_vm_attr[role].count - try(sum([for j in range(i) : local.pack_slots_list[role][j]]), 0)))
]
}
pack_prefix = {
for role in local.pack_roles : role => [
for i in range(length(local.pack_placed[role])) :
try(sum([for j in range(i) : local.pack_placed[role][j]]), 0)
]
}
cp_placements = [
for i in range(length(local.cp_nodes)) : {
vm_name = "kub-${var.vm_env}-${substr(local.cp_role, 0, 1)}${format("%01d", i + 1)}"
vm_role = local.cp_role
vm_node = local.cp_nodes[i]
}
]
pack_placements = flatten([
for role in local.pack_roles : [
for p in range(length(local.pack_precedence[role])) : [
for k in range(local.pack_placed[role][p]) : {
vm_name = "kub-${var.vm_env}-${substr(role, 0, 1)}${format("%01d", local.pack_prefix[role][p] + k + 1)}"
vm_role = role
vm_node = local.pack_precedence[role][p]
}
]
]
])
vm_list = {
for vm in concat(local.cp_placements, local.pack_placements) :
vm.vm_name => {
node_name = vm.vm_node
vm_cpu = local.effective_vm_attr[vm.vm_role].cpu
vm_ram = local.effective_vm_attr[vm.vm_role].ram
vm_vlan = local.effective_vm_attr[vm.vm_role].vlan
vm_role = vm.vm_role
}
}
roles = toset([for vm in local.vm_list : vm.vm_role])
}
data "proxmox_virtual_environment_nodes" "pve_nodes" {}
check "proxmox_nodes" {
assert {
condition = length(local.node_order) > 0
error_message = "No online Proxmox node available for deployment."
}
}
check "control_plane_capacity" {
assert {
condition = length(local.cp_nodes) == local.cp_attr.count
error_message = "Cannot place ${local.cp_attr.count} ${local.cp_role} VM(s): only ${length(local.cp_feasible)} online node(s) have at least ${local.cp_attr.ram} MiB free memory."
}
}
check "pack_capacity" {
assert {
condition = length(local.pack_roles) == 0 ? true : alltrue([
for role in local.pack_roles :
try(sum(local.pack_placed[role]), 0) == local.effective_vm_attr[role].count
])
error_message = "Cannot place all pack VMs: ${join(", ", [
for role in local.pack_roles :
"${role} needs ${local.effective_vm_attr[role].count} VM(s) (${local.effective_vm_attr[role].ram} MiB each), residual free capacity fits only ${try(sum(local.pack_placed[role]), 0)} VM(s)."
if try(sum(local.pack_placed[role]), 0) != local.effective_vm_attr[role].count
])}"
}
}
output "vm_ip" {
value = { for k, v in module.pve_vm : k => v.vm_ip }
}
resource "ansible_group" "vm_groups" {
for_each = local.roles
name = each.key
}
resource "ansible_host" "vm_hosts" {
for_each = module.pve_vm
name = each.key
variables = {
ansible_host = each.value.vm_ip
}
groups = [local.vm_list[each.key].vm_role]
}
terraform/projects/kubernetes_vms/provider.tf
+21
View File
@@ -0,0 +1,21 @@
terraform {
required_providers {
proxmox = {
source = "bpg/proxmox"
}
ansible = {
source = "ansible/ansible"
}
}
}
provider "proxmox" {
endpoint = var.proxmox_endpoint
api_token = var.proxmox_api_token
insecure = false
ssh {
agent = false
username = var.proxmox_ssh_username
password = var.proxmox_ssh_password
}
}
terraform/projects/kubernetes_vms/variables.tf
+135
View File
@@ -0,0 +1,135 @@
variable "proxmox_endpoint" {
description = "Proxmox URL endpoint"
type = string
}
variable "proxmox_api_token" {
description = "Proxmox API token"
type = string
sensitive = true
}
variable "proxmox_ssh_username" {
description = "Proxmox SSH username"
type = string
sensitive = true
}
variable "proxmox_ssh_password" {
description = "Proxmox SSH password"
type = string
sensitive = true
}
variable "vm_attr" {
description = "VM attributes per role. The control-plane role (master) is placed one per node; other roles are packed without a per-node limit."
type = map(object({
ram = number
cpu = number
vlan = number
count = number
}))
default = {
"master" = { ram = 2048, cpu = 2, vlan = 66, count = 3 }
"worker" = { ram = 4096, cpu = 2, vlan = 66, count = 3 }
}
validation {
condition = alltrue([for v in values(var.vm_attr) : v.count >= 1])
error_message = "Each role must deploy at least 1 VM."
}
}
variable "master_count" {
description = "Master role VM count override. null keeps the vm_attr default."
type = number
default = null
validation {
condition = var.master_count == null || var.master_count >= 1
error_message = "master_count must be null or at least 1."
}
}
variable "master_cpu" {
description = "Master role CPU override. null keeps the vm_attr default."
type = number
default = null
validation {
condition = var.master_cpu == null || var.master_cpu >= 1
error_message = "master_cpu must be null or at least 1."
}
}
variable "master_ram" {
description = "Master role RAM override in GiB (converted to MiB internally). null keeps the vm_attr default."
type = number
default = null
validation {
condition = var.master_ram == null || var.master_ram >= 1
error_message = "master_ram must be null or at least 1 GiB."
}
}
variable "master_vlan" {
description = "Master role VLAN override. null keeps the vm_attr default."
type = number
default = null
validation {
condition = var.master_vlan == null || var.master_vlan >= 1
error_message = "master_vlan must be null or at least 1."
}
}
variable "worker_count" {
description = "Worker role VM count override. null keeps the vm_attr default."
type = number
default = null
validation {
condition = var.worker_count == null || var.worker_count >= 1
error_message = "worker_count must be null or at least 1."
}
}
variable "worker_cpu" {
description = "Worker role CPU override. null keeps the vm_attr default."
type = number
default = null
validation {
condition = var.worker_cpu == null || var.worker_cpu >= 1
error_message = "worker_cpu must be null or at least 1."
}
}
variable "worker_ram" {
description = "Worker role RAM override in GiB (converted to MiB internally). null keeps the vm_attr default."
type = number
default = null
validation {
condition = var.worker_ram == null || var.worker_ram >= 1
error_message = "worker_ram must be null or at least 1 GiB."
}
}
variable "worker_vlan" {
description = "Worker role VLAN override. null keeps the vm_attr default."
type = number
default = null
validation {
condition = var.worker_vlan == null || var.worker_vlan >= 1
error_message = "worker_vlan must be null or at least 1."
}
}
variable "vm_env" {
description = "VM environment"
type = string
default = "prod"
}