add: skip backuo

This reverts commit 3c8375d6f4.

ansible/kubernetes/README.md

@@ -1,97 +0,0 @@
-# Ansible – Kubernetes nodes
-
-Prepares the Kubernetes VM nodes provisioned by the
-[`kubernetes_vms` Terraform project](../../terraform/projects/kubernetes_vms/).
-
-## What it does
-
-The `k8s_node` playbook configures each node for `kubeadm`/`kubelet`:
-
-1. **System** – disables swap (runtime + fstab), loads `overlay` and `br_netfilter`,
-   persists kernel modules, applies the Kubernetes `sysctl` settings.
-2. **Packages** – installs prerequisites, adds the official Kubernetes apt repo/key,
-   installs and holds `kubeadm` and `kubelet`.
-3. **containerd** – installs it, generates the default config, enables the
-   `SystemdCgroup` driver, starts and enables the service.
-4. **Firewall** – configures UFW per role: SSH + common Kubernetes ports from the admin
-   subnet, control-plane ports (`2379:2380`, `10257`, `10259`) on masters, worker ports
-   (`10256`, `30000:32767`) on workers, default deny incoming.
-
-Group membership (who is a `master` / `worker`) is resolved from the Terraform inventory,
-so the role-specific firewall rules apply automatically.
-
-> Cluster bootstrap (`kubeadm init` / `kubeadm join`) is **not** part of this playbook
-> and is done manually per the migration roadmap.
-
-## Inventory
-
-The inventory is generated from Terraform state via the `cloud.terraform` inventory
-plugin:
-
-```yaml
-# inventories/tf_k8s_vms.yml
----
-plugin: cloud.terraform.terraform_provider
-project_path:
-  - /tmp/semaphore/project_1/repository_1_template_7/terraform/projects/kubernetes_vms
-```
-
-> The `project_path` is a Semaphore runtime checkout path. If you run locally, point it
-> at your own `terraform/projects/kubernetes_vms` directory. The same `ansible` provider
-> that writes the inventory is used by Terraform, so hosts/groups are always in sync with
-> the applied VMs.
-
-## Requirements
-
-Install the collections referenced by `collections/requirements.yml`:
-
-```bash
-ansible-galaxy collection install -r collections/requirements.yml
-```
-
-- `cloud.terraform` – Terraform inventory plugin
-- `community.general` – `ufw` module
-
-## Usage
-
-```bash
-# Connectivity check
-ansible-playbook -i inventories/tf_k8s_vms.yml ping.yml
-
-# Prepare all Kubernetes nodes
-ansible-playbook -i inventories/tf_k8s_vms.yml k8s_node.yml
-```
-
-Both playbooks target `all`; the inventory plugin runs from the Terraform project path.
-
-## Role variables
-
-Defined in `roles/k8s_node/defaults/main.yml` (override as needed):
-
-| Variable                        | Default                      | Description                            |
-| ------------------------------- | ---------------------------- | -------------------------------------- |
-| `k8s_version`                   | `1.37`                       | Kubernetes minor version               |
-| `k8s_repo`                      | `pkgs.k8s.io` core v<ver>    | Kubernetes apt repository              |
-| `k8s_repo_key`                  | matching `Release.key` URL   | Repository signing key                 |
-| `k8s_packages`                  | `kubelet`, `kubeadm`         | Packages installed and held            |
-| `k8s_prerequisite_packages`     | `apt-transport-https`, `ca-certificates`, `curl`, `gnupg` | Build deps for the repo   |
-| `k8s_firewall_admin_subnet`     | `192.168.66.0/24`            | Subnet allowed through UFW             |
-| `k8s_firewall_ssh_port`         | `22`                         | SSH port                               |
-| `k8s_firewall_common_ports`     | `6443`, `10250`              | Ports open on all nodes                |
-| `k8s_firewall_master_ports`     | `2379:2380`, `10257`, `10259` | Ports open on masters                |
-| `k8s_firewall_worker_ports`     | `10256`, `30000:32767`       | Ports open on workers                  |
-
-## File reference
-
-| Path                                     | Purpose                              |
-| ---------------------------------------- | ------------------------------------ |
-| `k8s_node.yml`                           | Main playbook (role `k8s_node`)      |
-| `ping.yml`                               | Connectivity smoke-test playbook     |
-| `inventories/tf_k8s_vms.yml`             | Terraform-backed inventory           |
-| `collections/requirements.yml`           | Required Ansible collections         |
-| `roles/k8s_node/tasks/main.yml`          | Task dispatcher                      |
-| `roles/k8s_node/tasks/system.yml`        | Swap, kernel modules, sysctl         |
-| `roles/k8s_node/tasks/packages.yml`      | Kubernetes repo + kubeadm/kubelet    |
-| `roles/k8s_node/tasks/containerd.yml`    | containerd install/config            |
-| `roles/k8s_node/tasks/firewall.yml`      | Per-role UFW rules                   |
-| `roles/k8s_node/defaults/main.yml`       | Role defaults                        |

ansible/kubernetes/collections/requirements.yml

@@ -1,4 +0,0 @@
----
-collections:
-  - name: cloud.terraform
-  - name: community.general

ansible/kubernetes/inventories/tf_k8s_vms.yml

@@ -1,4 +0,0 @@
----
-plugin: cloud.terraform.terraform_provider
-project_path:
-  - /tmp/semaphore/project_1/repository_1_template_7/terraform/projects/kubernetes_vms

ansible/kubernetes/k8s_node.yml

@@ -1,6 +0,0 @@
----
-- name: Prepare Kubernetes nodes
-  hosts: all
-
-  roles:
-    - role: k8s_node

ansible/kubernetes/ping.yml

@@ -1,5 +0,0 @@
----
-- name: Prepare Kubernetes nodes
-  hosts: all
-  tasks:
-    - ansible.builtin.ping:

ansible/kubernetes/roles/k8s_node/defaults/main.yml

@@ -1,32 +0,0 @@
----
-k8s_version: "1.37"
-
-k8s_repo: "https://pkgs.k8s.io/core:/stable:/v{{ k8s_version }}/deb"
-k8s_repo_key: "https://pkgs.k8s.io/core:/stable:/v{{ k8s_version }}/deb/Release.key"
-
-k8s_packages:
-  - kubelet
-  - kubeadm
-
-k8s_prerequisite_packages:
-  - apt-transport-https
-  - ca-certificates
-  - curl
-  - gnupg
-
-k8s_firewall_admin_subnet: "192.168.88.0/24"
-k8s_firewall_ssh_port: "22"
-k8s_firewall_common_ports:
-  - "6443"
-  - "10250"
-k8s_firewall_master_ports:
-  - "2379:2380"
-  - "10257"
-  - "10259"
-k8s_firewall_worker_ports:
-  - "10256"
-  - "30000:32767"
-k8s_firewall_cilium_udp_ports:
-  - "8472"
-k8s_firewall_cilium_tcp_ports:
-  - "4240"

ansible/kubernetes/roles/k8s_node/tasks/containerd.yml

@@ -1,34 +0,0 @@
----
-- name: Install containerd
-  ansible.builtin.apt:
-    name: containerd
-    state: present
-    update_cache: true
-  become: yes
-
-- name: Create containerd configuration directory
-  ansible.builtin.file:
-    path: /etc/containerd
-    state: directory
-    mode: "0755"
-  become: yes
-
-- name: Generate default containerd configuration
-  ansible.builtin.shell:
-    cmd: containerd config default > /etc/containerd/config.toml
-    creates: /etc/containerd/config.toml
-  become: yes
-
-- name: Enable systemd cgroup driver in containerd
-  ansible.builtin.replace:
-    path: /etc/containerd/config.toml
-    regexp: '^(\s*)SystemdCgroup(\s*=\s*)false'
-    replace: '\1SystemdCgroup\2true'
-  become: yes
-
-- name: Enable and start containerd
-  ansible.builtin.systemd:
-    name: containerd
-    state: restarted
-    enabled: true
-  become: yes

ansible/kubernetes/roles/k8s_node/tasks/firewall.yml

@@ -1,62 +0,0 @@
----
-- name: Allow SSH from admin subnet
-  community.general.ufw:
-    rule: allow
-    port: "{{ k8s_firewall_ssh_port }}"
-    proto: tcp
-    from_ip: "{{ k8s_firewall_admin_subnet }}"
-  become: yes
-
-- name: Allow common Kubernetes ports from admin subnet
-  community.general.ufw:
-    rule: allow
-    port: "{{ item }}"
-    proto: tcp
-    from_ip: "{{ k8s_firewall_admin_subnet }}"
-  become: yes
-  loop: "{{ k8s_firewall_common_ports }}"
-
-- name: Allow etcd and control plane ports on control-plane nodes
-  community.general.ufw:
-    rule: allow
-    port: "{{ item }}"
-    proto: tcp
-    from_ip: "{{ k8s_firewall_admin_subnet }}"
-  become: yes
-  loop: "{{ k8s_firewall_master_ports }}"
-  when: inventory_hostname in groups['master']
-
-- name: Allow kube-proxy and NodePort ports on worker nodes
-  community.general.ufw:
-    rule: allow
-    port: "{{ item }}"
-    proto: tcp
-    from_ip: "{{ k8s_firewall_admin_subnet }}"
-  become: yes
-  loop: "{{ k8s_firewall_worker_ports }}"
-  when: inventory_hostname in groups['worker']
-
-- name: Allow Cilium VXLAN and health check UDP ports from admin subnet
-  community.general.ufw:
-    rule: allow
-    port: "{{ item }}"
-    proto: udp
-    from_ip: "{{ k8s_firewall_admin_subnet }}"
-  become: yes
-  loop: "{{ k8s_firewall_cilium_udp_ports }}"
-
-- name: Allow Cilium health check TCP ports from admin subnet
-  community.general.ufw:
-    rule: allow
-    port: "{{ item }}"
-    proto: tcp
-    from_ip: "{{ k8s_firewall_admin_subnet }}"
-  become: yes
-  loop: "{{ k8s_firewall_cilium_tcp_ports }}"
-
-- name: Enable ufw and deny incoming by default
-  community.general.ufw:
-    state: enabled
-    policy: deny
-    direction: incoming
-  become: yes

ansible/kubernetes/roles/k8s_node/tasks/main.yml

@@ -1,5 +0,0 @@
----
-- ansible.builtin.import_tasks: system.yml
-- ansible.builtin.import_tasks: packages.yml
-- ansible.builtin.import_tasks: containerd.yml
-- ansible.builtin.import_tasks: firewall.yml

ansible/kubernetes/roles/k8s_node/tasks/packages.yml

@@ -1,36 +0,0 @@
----
-- name: Install prerequisite packages
-  ansible.builtin.apt:
-    name: "{{ k8s_prerequisite_packages }}"
-    state: present
-    update_cache: true
-  become: yes
-
-- name: Add Kubernetes apt key
-  ansible.builtin.apt_key:
-    url: "{{ k8s_repo_key }}"
-    keyring: /etc/apt/keyrings/kubernetes-apt-keyring.gpg
-    state: present
-  become: yes
-
-- name: Add Kubernetes apt repository
-  ansible.builtin.apt_repository:
-    repo: "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] {{ k8s_repo }} /"
-    state: present
-    filename: kubernetes
-  become: yes
-
-- name: Install kubelet and kubeadm
-  ansible.builtin.apt:
-    name: "{{ k8s_packages }}"
-    state: present
-    update_cache: true
-  become: yes
-
-- name: Hold kubelet and kubeadm packages
-  ansible.builtin.command:
-    cmd: apt-mark hold {{ item }}
-  loop: "{{ k8s_packages }}"
-  register: _kube_hold
-  become: yes
-  changed_when: "'already set on hold' not in _kube_hold.stdout"

ansible/kubernetes/roles/k8s_node/tasks/system.yml

@@ -1,51 +0,0 @@
----
-- name: Disable swap
-  ansible.builtin.shell:
-    cmd: swapoff -a
-  become: yes
-  changed_when: false
-
-- name: Comment out swap entries in fstab
-  ansible.builtin.replace:
-    path: /etc/fstab
-    regexp: '^(\s*[^#\s][^\s]*\s+swap\s+.*)$'
-    replace: '# \1'
-  become: yes
-
-- name: Load overlay kernel module
-  ansible.builtin.shell:
-    cmd: modprobe overlay
-  become: yes
-  changed_when: false
-  failed_when: false
-
-- name: Load br_netfilter kernel module
-  ansible.builtin.shell:
-    cmd: modprobe br_netfilter
-  become: yes
-  changed_when: false
-
-- name: Persist kernel modules
-  ansible.builtin.copy:
-    dest: /etc/modules-load.d/k8s.conf
-    content: |
-      overlay
-      br_netfilter
-    mode: "0644"
-  become: yes
-
-- name: Configure kernel network settings
-  ansible.builtin.copy:
-    dest: /etc/sysctl.d/k8s.conf
-    content: |
-      net.bridge.bridge-nf-call-iptables  = 1
-      net.bridge.bridge-nf-call-ip6tables = 1
-      net.ipv4.ip_forward                 = 1
-    mode: "0644"
-  become: yes
-
-- name: Apply kernel network settings
-  ansible.builtin.command:
-    cmd: sysctl --system
-  become: yes
-  changed_when: false

ansible/opnsense/inventory.yml

@@ -16,5 +16,4 @@ all:
             cerbere-head1:
               ansible_host: 192.168.88.2
               main_role: MASTER
-              hypervisor: Proxmox
-              proxmox_vmid: 122
+              hypervisor: Proxmox

ansible/opnsense/requirements.yml

@@ -1,4 +0,0 @@
----
-collections:
-  - name: community.proxmox
-    version: "2.0.0"

ansible/opnsense/update_opnsense_ha_cluster.yml

@@ -46,59 +46,47 @@
             password: "{{ opnsense_api_secret }}"
             force_basic_auth: true
             validate_certs: false
-          register: _firmware_status 
-          until: _firmware_status.json.status_msg != "Firmware status requires to check for update first to provide more information."
-          retries: 10
-          delay: 2
+          register: _firmware_status
+          until: _firmware_status.json.status != 'none'
+          retries: 12
+          delay: 5
 
-        - name: Store node facts
+        - name: Store firmware facts
           ansible.builtin.set_fact:
-            firmware_action: "{{ opnsense_action | default('update')}}"
             firmware_status: "{{ _firmware_status.json.status }}"
-            firmware_status_msg: "{{ _firmware_status.json.status_msg }}"
-            firmware_current_version: "{{ _firmware_status.json.product.product_version | default('unknown') }}"
-            firmware_product_series: "{{ _firmware_status.json.product.product_series | default('unknown') }}"
-            firmware_update_version: "{{ _firmware_status.json.upgrade_packages | selectattr('name', 'equalto', 'opnsense') | map(attribute='new_version') | first | default('') }}"
+            firmware_current_version: "{{ _firmware_status.json.product_version | default('unknown') }}"
+            firmware_new_version: "{{ _firmware_status.json.upgrade_packages | selectattr('name', 'equalto', 'opnsense') | map(attribute='new_version') | first | default('') }}"
             firmware_upgrade_version: "{{ _firmware_status.json.upgrade_major_version | default('unknown') }}"
-            firmware_upgrade_message: "{{ _firmware_status.json.upgrade_major_message | regex_replace('<[^>]+>', ' ') | regex_replace('\\s{2,}', ' ') | trim | default('') }}"
-            firmware_up_to_date: "{{ _firmware_status.json.status_msg == 'There are no updates available on the selected mirror.' }}"                   
-            needs_reboot: "{{ (_firmware_status.json.upgrade_needs_reboot == '1') if _firmware_status.json.status == 'upgrade' else (_firmware_status.json.needs_reboot == '1') }}"
+            needs_reboot: "{{ true if _firmware_status.json.needs_reboot == '1' else false }}"
             total_vips: "{{ _vip_status.json.rowCount }}"
             mismatched_vips: "{{ _vip_status.json.rows | rejectattr('status', 'equalto', main_role) | list | length }}"
             in_maintenance: "{{ _vip_status.json.carp.maintenancemode }}"
+            # update_result: "{{ 'pending' if _firmware_status.json.status == 'update' else 'skipped' }}"
 
-        - name: Resolve update-vs-upgrade specifics
-          ansible.builtin.set_fact:
-            firmware_target_kind: "{{ 'series' if firmware_status == 'upgrade' else 'version' }}"
-            firmware_target_value: "{{ firmware_upgrade_version if firmware_status == 'upgrade' else firmware_update_version }}"
-
-        - name: Backup already updated
+        - name: Skip backup update
           ansible.builtin.set_fact:
             skip_update: true
-            firmware_status: "skipped"
           delegate_to: "{{ groups['opnsense_backup'][0] }}"
           delegate_facts: true
           run_once: true
-          when: >-
-            (hostvars[groups['opnsense_backup'][0]].firmware_current_version
-            if hostvars[groups['opnsense_master'][0]].firmware_target_kind == 'version'
-            else hostvars[groups['opnsense_backup'][0]].firmware_product_series)
-            == hostvars[groups['opnsense_master'][0]].firmware_target_value
+          when: hostvars[groups['opnsense_backup'][0]].firmware_current_version == hostvars[groups['opnsense_master'][0]].firmware_new_version  
 
-        - name: Skip node {{ firmware_action }}
-          ansible.builtin.set_fact:
-            skip_update: true
-            firmware_status: "skipped"
-          when: firmware_up_to_date or firmware_action != firmware_status
-        
         - name: "Assert"
           ansible.builtin.assert:
             that: 
+              - firmware_status == 'update'
               - not in_maintenance
               - mismatched_vips == 0
               - total_vips > 0
             quiet: true
 
+        - name: Display firmware status
+          ansible.builtin.debug:
+            msg: >-
+              {{ inventory_hostname }} ({{ main_role }}): {{ firmware_status }}
+              ({{ firmware_current_version }}
+              {% if firmware_new_version %} → {{ firmware_new_version }} (reboot: {{ needs_reboot }}){% endif %})
+
       rescue:
 
         - name: Send Ntfy notification
@@ -110,19 +98,24 @@
             force_basic_auth: true
             body: |
               Current version: {{ firmware_current_version }}
-              {% if in_maintenance %}
+              {% if firmware_status != "update" %}
+                {% if firmware_status == "upgrade" %}
+              Upgrade to {{ firmware_upgrade_version }} available, relaunch the playbook manually to proceed.
+                {% else %}
+              No updates available on the cluster.
+                {% endif %}
+              {% elif in_maintenance %}
               The node {{ inventory_hostname }} is in CARP maintenance mode.
               {% elif mismatched_vips > 0 %}
               Some VIP are not {{ main_role }} on {{ ansible_host }}.
-              {% elif total_vips == 0 %}
-              No VIPs are managed.
               {% else %}
-              An error occured during the {{ firmware_action }}.
+              No VIPs are managed.
               {% endif %}
             headers:
-              Title: "OPNsense {{ firmware_action }} report"
+              Title: "OPNsense update aborted"
               Priority: "default"
               Tags: "x"
+          run_once: true
           when: ntfy_url is defined
 
         - ansible.builtin.fail:
@@ -131,12 +124,10 @@
 
 - name: "Phase 2 | Update backup node (TrueNAS, no snapshot)"
   hosts: opnsense_backup
-  any_errors_fatal: true
   gather_facts: false
   tasks:
 
-    - when: not skip_update | default(false)
-      block:
+    - block:
 
         - name: Enable CARP maintenance mode
           ansible.builtin.uri:
@@ -149,16 +140,16 @@
           register: _enable_maintenance
           changed_when: _enable_maintenance.status == "ok"
 
-        - name: Trigger firmware {{ firmware_action }}
+        - name: Trigger firmware update
           ansible.builtin.uri:
-            url: "https://{{ opnsense_host }}/api/core/firmware/{{ firmware_action }}"
+            url: "https://{{ opnsense_host }}/api/core/firmware/update"
             method: POST
             user: "{{ opnsense_api_key }}"
             password: "{{ opnsense_api_secret }}"
             force_basic_auth: true
             validate_certs: false
 
-        - name: Check if the {{ firmware_action }} is running
+        - name: Wait for packages to update
           ansible.builtin.uri:
             url: "https://{{ opnsense_host }}/api/core/firmware/running"
             method: GET
@@ -167,266 +158,357 @@
             force_basic_auth: true
             validate_certs: false
           register: _update_running
-          until: _update_running.json.status == 'busy'
-          retries: 2
-          delay: 2
-
-        - name: Wait for node to reboot after the {{ firmware_action }}
-          ansible.builtin.wait_for:
-            host: "{{ ansible_host }}"
-            port: "{{ opnsense_https_port }}"
-            state: stopped
-            timeout: 3600
-          when: needs_reboot
-
-        - name: Wait for node to come back online
-          ansible.builtin.wait_for:
-            host: "{{ ansible_host }}"
-            port: "{{ opnsense_https_port }}"
-            state: started
-            timeout: 5400
-            delay: 30
-          when: needs_reboot
-
-        - name: Check firmware version
-          ansible.builtin.uri:
-            url: "https://{{ opnsense_host }}/api/core/firmware/status"
-            method: GET
-            user: "{{ opnsense_api_key }}"
-            password: "{{ opnsense_api_secret }}"
-            force_basic_auth: true
-            validate_certs: false
-          register: _post_firmware_status
-          until: _post_firmware_status.json.product['product_' ~ firmware_target_kind] | default('unknown') == firmware_target_value
-          retries: 240
+          until: _update_running.json.status != 'busy'
+          retries: 300
           delay: 15
 
-        - name: Check VIP status 
-          ansible.builtin.uri:
-            url: "https://{{ opnsense_host }}/api/diagnostics/interface/get_vip_status"
-            method: GET
-            user: "{{ opnsense_api_key }}"
-            password: "{{ opnsense_api_secret }}"
-            force_basic_auth: true
-            validate_certs: false
-          register: _carp_post_udapte
+#     - name: Wait for node to go offline
+#       ansible.builtin.wait_for:
+#         host: "{{ ansible_host }}"
+#         port: 443
+#         state: stopped
+#         timeout: 60
+#       delegate_to: localhost
 
-        - name: Disable CARP maintenance mode
-          ansible.builtin.uri:
-            url: "https://{{ opnsense_host }}/api/diagnostics/interface/carp_status/maintenance"
-            method: POST
-            user: "{{ opnsense_api_key }}"
-            password: "{{ opnsense_api_secret }}"
-            force_basic_auth: true
-            validate_certs: false
-          when: _carp_post_udapte.json.carp.maintenancemode
-          ignore_errors: true
+#     - name: Wait for node to come back online
+#       ansible.builtin.wait_for:
+#         host: "{{ ansible_host }}"
+#         port: 443
+#         state: started
+#         timeout: 300
+#         delay: 30
+#       delegate_to: localhost
 
-      rescue:
+#     - name: Wait for API to be fully responsive
+#       ansible.builtin.uri:
+#         url: "https://{{ opnsense_host }}/api/core/firmware/status"
+#         method: GET
+#         user: "{{ opnsense_api_key }}"
+#         password: "{{ opnsense_api_secret }}"
+#         force_basic_auth: true
+#         validate_certs: false
+#       register: _post_status
+#       until: _post_status.status == 200
+#       retries: 20
+#       delay: 10
 
-        - name: Send Ntfy notification
-          ansible.builtin.uri:
-            url: "{{  ntfy_url }}/{{ ntfy_topic }}"
-            method: POST
-            user: "{{ ntfy_user }}"
-            password: "{{ lookup('env', 'NTFY_PASSWORD') }}"
-            force_basic_auth: true
-            body: |
-              Current version: {{ firmware_current_version }}
-              {{ firmware_action | capitalize }} to {{ firmware_target_value }} failed on {{ inventory_hostname }} ({{ main_role }}).
-              No snapshot have been taken before the {{ firmware_action }}.
-            headers:
-              Title: "OPNsense {{ firmware_action }} failed on {{ inventory_hostname }}"
-              Priority: "high"
-              Tags: "x"
-          when: ntfy_url is defined
+#     - name: Health check — no remaining updates
+#       ansible.builtin.assert:
+#         that:
+#           - _post_status.json.status == 'none'
+#         fail_msg: >-
+#           Post-update firmware status is '{{ _post_status.json.status }}'
+#           on {{ inventory_hostname }}. Expected 'none'.
 
-        - ansible.builtin.fail:
-            msg: "Update failed"
+#     - name: Health check — CARP role is BACKUP
+#       ansible.builtin.uri:
+#         url: "https://{{ opnsense_host }}/api/carp/status/get"
+#         method: GET
+#         user: "{{ opnsense_api_key }}"
+#         password: "{{ opnsense_api_secret }}"
+#         force_basic_auth: true
+#         validate_certs: false
+#       register: _carp_post
+#       failed_when: _carp_post.json.status | upper not in ['BACKUP', 'MAINTENANCE']
 
-- name: "Phase 3 | Update master node (Proxmox, with snapshot)"
-  hosts: opnsense_master
-  any_errors_fatal: true
-  gather_facts: false
-  vars:
-    proxmox_snap_name: "preupdate-{{ now().strftime('%Y%m%d') }}"
+#     - name: Disable CARP maintenance mode
+#       ansible.builtin.uri:
+#         url: "https://{{ opnsense_host }}/api/carp/status/maintenancemode"
+#         method: POST
+#         user: "{{ opnsense_api_key }}"
+#         password: "{{ opnsense_api_secret }}"
+#         force_basic_auth: true
+#         validate_certs: false
+#         body_format: json
+#         body: {}
 
-  tasks:
-
-    - when: not skip_update | default(false)
-      block:
-
-        - name: Take Proxmox VM snapshot
-          community.proxmox.proxmox_snap:
-            vmid: "{{ proxmox_vmid }}"
-            state: present
-            snapname: "{{ proxmox_snap_name }}"
-            description: "Pre-firmware-{{ firmware_action }}: {{ firmware_current_version }} → {{ firmware_target_value }}"
-
-        - name: Enable CARP maintenance mode
-          ansible.builtin.uri:
-            url: "https://{{ opnsense_host }}/api/diagnostics/interface/carp_status/maintenance"
-            method: POST
-            user: "{{ opnsense_api_key }}"
-            password: "{{ opnsense_api_secret }}"
-            force_basic_auth: true
-            validate_certs: false
-          register: _enable_maintenance
-          changed_when: _enable_maintenance.status == "ok"
-
-        - name: Wait for CARP failover to settle
-          ansible.builtin.pause:
-            seconds: 5
-
-        - name: Trigger firmware {{ firmware_action }}
-          ansible.builtin.uri:
-            url: "https://{{ opnsense_host }}/api/core/firmware/{{ firmware_action }}"
-            method: POST
-            user: "{{ opnsense_api_key }}"
-            password: "{{ opnsense_api_secret }}"
-            force_basic_auth: true
-            validate_certs: false
-
-        - name: Check if the {{ firmware_action }} is running
-          ansible.builtin.uri:
-            url: "https://{{ opnsense_host }}/api/core/firmware/running"
-            method: GET
-            user: "{{ opnsense_api_key }}"
-            password: "{{ opnsense_api_secret }}"
-            force_basic_auth: true
-            validate_certs: false
-          register: _update_running
-          until: _update_running.json.status == 'busy'
-          retries: 2
-          delay: 2
-
-        - name: Wait for node to reboot after the {{ firmware_action }}
-          ansible.builtin.wait_for:
-            host: "{{ ansible_host }}"
-            port: "{{ opnsense_https_port }}"
-            state: stopped
-            timeout: 3600
-          when: needs_reboot
-
-        - name: Wait for node to come back online
-          ansible.builtin.wait_for:
-            host: "{{ ansible_host }}"
-            port: "{{ opnsense_https_port }}"
-            state: started
-            timeout: 5400
-            delay: 30
-          when: needs_reboot
-
-        - name: Check firmware version
-          ansible.builtin.uri:
-            url: "https://{{ opnsense_host }}/api/core/firmware/status"
-            method: GET
-            user: "{{ opnsense_api_key }}"
-            password: "{{ opnsense_api_secret }}"
-            force_basic_auth: true
-            validate_certs: false
-          register: _post_firmware_status
-          until: _post_firmware_status.json.product['product_' ~ firmware_target_kind] | default('unknown') == firmware_target_value
-          retries: 240
-          delay: 15
-
-        - name: Check VIP status 
-          ansible.builtin.uri:
-            url: "https://{{ opnsense_host }}/api/diagnostics/interface/get_vip_status"
-            method: GET
-            user: "{{ opnsense_api_key }}"
-            password: "{{ opnsense_api_secret }}"
-            force_basic_auth: true
-            validate_certs: false
-          register: _carp_post_udapte
-
-        - name: Disable CARP maintenance mode
-          ansible.builtin.uri:
-            url: "https://{{ opnsense_host }}/api/diagnostics/interface/carp_status/maintenance"
-            method: POST
-            user: "{{ opnsense_api_key }}"
-            password: "{{ opnsense_api_secret }}"
-            force_basic_auth: true
-            validate_certs: false
-          when: _carp_post_udapte.json.carp.maintenancemode
-          ignore_errors: true
-
-      rescue:
-
-        - name: Rollback VM to the pre-{{ firmware_action }} snapshot
-          community.proxmox.proxmox_snap:
-            vmid: "{{ proxmox_vmid }}"
-            state: rollback
-            snapname: "{{ proxmox_snap_name }}"
-          register: _proxmox_snapshot_rollback
-          ignore_errors: true
-
-        - name: Send Ntfy notification
-          ansible.builtin.uri:
-            url: "{{  ntfy_url }}/{{ ntfy_topic }}"
-            method: POST
-            user: "{{ ntfy_user }}"
-            password: "{{ lookup('env', 'NTFY_PASSWORD') }}"
-            force_basic_auth: true
-            body: |
-              Current version: {{ firmware_current_version }}
-              {{ firmware_action | capitalize }} to {{ firmware_target_value }} failed on {{ inventory_hostname }} ({{ main_role }}).
-              A snapshot have been taken before the {{ firmware_action }} and the VM has been rollback.
-            headers:
-              Title: "OPNsense {{ firmware_action }} failed on {{ inventory_hostname }}"
-              Priority: "high"
-              Tags: "x"
-          when: ntfy_url is defined
-
-        - ansible.builtin.fail:
-            msg: "Update aborted"
+#     - name: Record backup update result
+#       ansible.builtin.set_fact:
+#         update_result: "success"
+#         fw_post_version: "{{ _post_status.json.product_version | default('unknown') }}"
 
 
-- name: "Phase 4 | Notification"
-  hosts: localhost
-  gather_facts: false
-  tasks:
+# ###############################################################
+# - name: "Phase 3 | Update master node (Proxmox, with snapshot)"
+#   hosts: opnsense_master
+#   gather_facts: false
 
-    - name: Send Ntfy notification
-      vars:
-        master: "{{ groups['opnsense_master'][0] }}"
-        backup: "{{ groups['opnsense_backup'][0] }}"
-        m: "{{ hostvars[master] }}"
-        b: "{{ hostvars[backup] }}"
-        same_operation: "{{ m.firmware_status == b.firmware_status }}"
+#   vars:
+#     snap_name: "preupdate-{{ now().strftime('%Y%m%d-%H%M') }}"
+#     _pve_base: "https://{{ proxmox_api_host }}/api2/json/nodes/{{ proxmox_node }}/qemu/{{ proxmox_vmid }}"
+#     _pve_auth:
+#       Authorization: "PVEAPIToken={{ proxmox_api_token_id }}={{ proxmox_api_token_secret }}"
 
-      ansible.builtin.uri:
-        url: "{{ ntfy_url }}/{{ ntfy_topic }}"
-        method: POST
-        user: "{{ ntfy_user }}"
-        password: "{{ lookup('env', 'NTFY_PASSWORD') }}"
-        force_basic_auth: true
-        body: |
-          {% if same_operation %}
-          {% if m.skip_update | default(false) %}
-          Both nodes already on {{ m.firmware_current_version }}, no action taken.
-          {% else %}
-          OPNsense cluster: {{ m.firmware_current_version }} → {{ m.firmware_target_value }} ({{ m.firmware_status }})
-          {% endif %}
-          {% else %}
-          {% if m.skip_update | default(false) %}
-          MASTER ({{ master }}): already on {{ m.firmware_current_version }}, no action taken.
-          {% else %}
-          MASTER ({{ master }}): {{ m.firmware_current_version }} → {{ m.firmware_target_value }} ({{ m.firmware_status }})
-          {% endif %}
-          {% if b.skip_update | default(false) %}
-          BACKUP ({{ backup }}): already on {{ b.firmware_current_version }}, no action taken.
-          {% else %}
-          BACKUP ({{ backup }}): {{ b.firmware_current_version }} → {{ b.firmware_target_value }} ({{ b.firmware_status }})
-          {% endif %}
-          {% endif %}
-        headers:
-          Title: >-
-            {%- if m.skip_update | default(false) and b.skip_update | default(false) -%}
-            OPNsense cluster - no {{m.firmware_action }} available
-            {%- else -%}
-            OPNsense cluster {{m.firmware_action }} completed
-            {%- endif -%}
-          Priority: "{{ 'min' if (m.skip_update | default(false) and b.skip_update | default(false)) else 'default' }}"
-          Tags: "{{ 'information_source' if (m.skip_update | default(false) and b.skip_update | default(false)) else 'white_check_mark' }}"
-      when: ntfy_url is defined
+#   tasks:
+#     - name: Skip — no minor updates available on master
+#       ansible.builtin.meta: end_host
+#       when: firmware_status != 'update'
+
+#     - name: Verify master node CARP role
+#       ansible.builtin.uri:
+#         url: "https://{{ opnsense_host }}/api/carp/status/get"
+#         method: GET
+#         user: "{{ opnsense_api_key }}"
+#         password: "{{ opnsense_api_secret }}"
+#         force_basic_auth: true
+#         validate_certs: false
+#       register: _carp
+#       failed_when: _carp.json.status | upper != 'MASTER'
+
+#     # Snapshot BEFORE maintenance mode — the restore point should be
+#     # a fully functioning MASTER node, not one mid-failover
+#     - name: Take Proxmox VM snapshot
+#       community.general.proxmox_snap:
+#         api_host: "{{ proxmox_api_host }}"
+#         api_token_id: "{{ proxmox_api_token_id }}"
+#         api_token_secret: "{{ proxmox_api_token_secret }}"
+#         vmid: "{{ proxmox_vmid }}"
+#         state: present
+#         snapname: "{{ snap_name }}"
+#         description: "Pre-firmware-update: {{ firmware_current_version }} → {{ firmware_new_version }}"
+#         timeout: 60
+#       delegate_to: localhost
+
+#     - name: Enable CARP maintenance mode (triggers controlled failover to backup)
+#       ansible.builtin.uri:
+#         url: "https://{{ opnsense_host }}/api/carp/status/maintenancemode"
+#         method: POST
+#         user: "{{ opnsense_api_key }}"
+#         password: "{{ opnsense_api_secret }}"
+#         force_basic_auth: true
+#         validate_certs: false
+#         body_format: json
+#         body: {}
+
+#     - name: Wait for CARP failover to settle
+#       ansible.builtin.pause:
+#         seconds: 10
+
+#     # Do not proceed if the backup has not actually taken over.
+#     # A failed failover means no traffic coverage during master update.
+#     - name: Confirm backup node is now MASTER
+#       ansible.builtin.uri:
+#         url: "https://{{ hostvars[groups['opnsense_backup'][0]].ansible_host }}/api/carp/status/get"
+#         method: GET
+#         user: "{{ opnsense_api_key }}"
+#         password: "{{ opnsense_api_secret }}"
+#         force_basic_auth: true
+#         validate_certs: false
+#       register: _backup_carp
+#       delegate_to: localhost
+#       failed_when: _backup_carp.json.status | upper != 'MASTER'
+
+#     - name: Update and verify
+#       block:
+
+#         - name: Trigger firmware update
+#           ansible.builtin.uri:
+#             url: "https://{{ opnsense_host }}/api/core/firmware/update"
+#             method: POST
+#             user: "{{ opnsense_api_key }}"
+#             password: "{{ opnsense_api_secret }}"
+#             force_basic_auth: true
+#             validate_certs: false
+#             body_format: json
+#             body: {}
+
+#         - name: Wait for packages to install
+#           ansible.builtin.uri:
+#             url: "https://{{ opnsense_host }}/api/core/firmware/running"
+#             method: GET
+#             user: "{{ opnsense_api_key }}"
+#             password: "{{ opnsense_api_secret }}"
+#             force_basic_auth: true
+#             validate_certs: false
+#           register: _update_running
+#           until: _update_running.json.status != 'running'
+#           retries: 30
+#           delay: 15
+
+#         - name: Reboot node
+#           ansible.builtin.uri:
+#             url: "https://{{ opnsense_host }}/api/core/firmware/reboot"
+#             method: POST
+#             user: "{{ opnsense_api_key }}"
+#             password: "{{ opnsense_api_secret }}"
+#             force_basic_auth: true
+#             validate_certs: false
+#             body_format: json
+#             body: {}
+#           ignore_errors: true
+
+#         - name: Wait for node to go offline
+#           ansible.builtin.wait_for:
+#             host: "{{ ansible_host }}"
+#             port: 443
+#             state: stopped
+#             timeout: 60
+#           delegate_to: localhost
+
+#         - name: Wait for node to come back online
+#           ansible.builtin.wait_for:
+#             host: "{{ ansible_host }}"
+#             port: 443
+#             state: started
+#             timeout: 300
+#             delay: 30
+#           delegate_to: localhost
+
+#         - name: Wait for API to be fully responsive
+#           ansible.builtin.uri:
+#             url: "https://{{ opnsense_host }}/api/core/firmware/status"
+#             method: GET
+#             user: "{{ opnsense_api_key }}"
+#             password: "{{ opnsense_api_secret }}"
+#             force_basic_auth: true
+#             validate_certs: false
+#           register: _post_status
+#           until: _post_status.status == 200
+#           retries: 20
+#           delay: 10
+
+#         - name: Health check — no remaining updates
+#           ansible.builtin.assert:
+#             that:
+#               - _post_status.json.status == 'none'
+#             fail_msg: >-
+#               Post-update firmware status is '{{ _post_status.json.status }}'.
+#               Expected 'none'. Triggering rollback.
+
+#         # After maintenance mode failover, master should rejoin as BACKUP
+#         - name: Health check — CARP role is BACKUP
+#           ansible.builtin.uri:
+#             url: "https://{{ opnsense_host }}/api/carp/status/get"
+#             method: GET
+#             user: "{{ opnsense_api_key }}"
+#             password: "{{ opnsense_api_secret }}"
+#             force_basic_auth: true
+#             validate_certs: false
+#           register: _carp_post
+#           failed_when: _carp_post.json.status | upper not in ['BACKUP', 'MAINTENANCE']
+
+#         - name: Record master update result
+#           ansible.builtin.set_fact:
+#             update_result: "success"
+#             fw_post_version: "{{ _post_status.json.product_version | default('unknown') }}"
+
+#       rescue:
+#         # Snapshot was taken before maintenance mode, so the restored VM
+#         # will come back as a healthy MASTER on the previous version.
+#         # Backup (already updated) continues serving traffic during rollback.
+
+#         - name: "ROLLBACK | Stop VM"
+#           ansible.builtin.uri:
+#             url: "{{ _pve_base }}/status/stop"
+#             method: POST
+#             headers: "{{ _pve_auth }}"
+#             validate_certs: false
+#           delegate_to: localhost
+
+#         - name: "ROLLBACK | Wait for VM to stop"
+#           ansible.builtin.uri:
+#             url: "{{ _pve_base }}/status/current"
+#             method: GET
+#             headers: "{{ _pve_auth }}"
+#             validate_certs: false
+#           register: _vm_state
+#           until: _vm_state.json.data.status == 'stopped'
+#           retries: 12
+#           delay: 5
+#           delegate_to: localhost
+
+#         - name: "ROLLBACK | Restore snapshot {{ snap_name }}"
+#           ansible.builtin.uri:
+#             url: "{{ _pve_base }}/snapshot/{{ snap_name }}/rollback"
+#             method: POST
+#             headers: "{{ _pve_auth }}"
+#             validate_certs: false
+#           delegate_to: localhost
+
+#         # Proxmox rollback tasks run async; wait for completion
+#         - name: "ROLLBACK | Wait for snapshot restore to complete"
+#           ansible.builtin.pause:
+#             seconds: 30
+
+#         - name: "ROLLBACK | Start VM"
+#           ansible.builtin.uri:
+#             url: "{{ _pve_base }}/status/start"
+#             method: POST
+#             headers: "{{ _pve_auth }}"
+#             validate_certs: false
+#           delegate_to: localhost
+
+#         - name: "ROLLBACK | Wait for VM to come back online"
+#           ansible.builtin.wait_for:
+#             host: "{{ ansible_host }}"
+#             port: 443
+#             state: started
+#             timeout: 300
+#             delay: 30
+#           delegate_to: localhost
+
+#         - name: "ROLLBACK | Record result"
+#           ansible.builtin.set_fact:
+#             update_result: "rolled_back"
+#             fw_post_version: "{{ firmware_current_version }}"
+
+#       always:
+#         # Best-effort: disable maintenance mode whether update succeeded or rolled back.
+#         # On rollback, the restored snapshot predates maintenance mode so this
+#         # is effectively a no-op, but run it anyway as a safety net.
+#         - name: Disable CARP maintenance mode
+#           ansible.builtin.uri:
+#             url: "https://{{ opnsense_host }}/api/carp/status/maintenancemode"
+#             method: POST
+#             user: "{{ opnsense_api_key }}"
+#             password: "{{ opnsense_api_secret }}"
+#             force_basic_auth: true
+#             validate_certs: false
+#             body_format: json
+#             body: {}
+#           ignore_errors: true
+
+
+# ###############################################################
+# # Phase 4 targets localhost — it runs even if Phase 3 failed,
+# # because localhost was never marked as a failed host.
+# - name: "Phase 4 | Send ntfy notification"
+#   hosts: localhost
+#   gather_facts: false
+
+#   tasks:
+#     - name: Build report
+#       ansible.builtin.set_fact:
+#         _report: |
+#           {% for host in groups['opnsense'] %}
+#           {% set h = hostvars[host] %}
+#           {{ host }}
+#             fw check : {{ h.firmware_status | default('unknown') }}
+#             result   : {{ h.update_result | default('unknown') }}
+#             version  : {{ h.fw_post_version | default(h.firmware_current_version | default('?')) }}
+#           {% endfor %}
+#         _any_issue: >-
+#           {{ groups['opnsense']
+#              | map('extract', hostvars, 'update_result')
+#              | select('in', ['rolled_back', 'pending'])
+#              | list
+#              | length > 0 }}
+
+#     - name: Send ntfy notification
+#       ansible.builtin.uri:
+#         url: "{{ ntfy_url }}/{{ ntfy_topic }}"
+#         method: POST
+#         headers:
+#           Authorization: "Bearer {{ ntfy_token }}"
+#           Title: "OPNsense HA Update Report"
+#           Priority: "{{ 'high' if _any_issue else 'default' }}"
+#           Tags: "{{ 'warning' if _any_issue else 'white_check_mark' }},shield"
+#           Content-Type: "text/plain"
+#         body: "{{ _report }}"
+
+#     - name: Fail playbook if any node was rolled back or did not complete
+#       ansible.builtin.fail:
+#         msg: >-
+#           One or more nodes did not update successfully.
+#           Check ntfy report and Ansible output for details.
+#       when: _any_issue | bool

ansible/proxmox/cloudinit_template.yml

@@ -1,120 +0,0 @@
----
-- name: Create or update a cloud-init Ubuntu template
-  hosts: nodes
-  become: true
-  vars:
-    template_vmid: 900
-    template_os: ubuntu
-    os_release: noble
-
-    template_name: "{{template_os}}-{{ os_release }}-cloud"
- 
-    image_filename: "{{ os_release }}-server-cloudimg-amd64.img"
-    image_url: "https://cloud-images.ubuntu.com/{{ os_release }}/current/{{ image_filename }}"
-    image_dest: "/var/lib/vz/template/iso/{{ image_filename }}"
- 
-    iso_storage: local
-    vm_storage: ceph-workload
- 
-    template_memory: 2048
-    template_cores: 1
- 
-  tasks:
-    - name: Check whether the template VMID already exists
-      ansible.builtin.command: qm status {{ template_vmid }}
-      register: template_status
-      changed_when: false
-      failed_when: false
-
-    - name: Check if the cloud image file exists
-      ansible.builtin.stat:
-        path: "{{ image_dest }}"
-      register: image_exists
-      changed_when: false
-      failed_when: false
-
-    - name: Set candidate information
-      ansible.builtin.set_fact:
-        has_image_file: "{{ image_exists.stat.exists }}"
-        root_free_space: "{{ (ansible_facts['mounts'] | selectattr('mount', 'equalto', '/') | first).size_available }}"
-
-    - name: Select target host
-      ansible.builtin.set_fact:
-        target_host: >-
-          {{ (ansible_play_hosts
-              | map('extract', hostvars)
-              | selectattr('has_image_file', 'equalto', true)
-              | sort(attribute='root_free_space', reverse=true)
-              | map(attribute='inventory_hostname')
-              | first)
-              | default(ansible_play_hosts
-              | map('extract', hostvars)
-              | sort(attribute='root_free_space', reverse=true)
-              | map(attribute='inventory_hostname')
-              | first,
-              true) }}
-      run_once: true
-
-    - name: Download the current Ubuntu cloud image
-      ansible.builtin.get_url:
-        url: "{{ image_url }}"
-        dest: "{{ image_dest }}"
-        force: false 
-        mode: "0644"
-      register: image_download
-      when: inventory_hostname == target_host
-
-    - name: Destroy existing template if present
-      ansible.builtin.command: qm destroy {{ template_vmid }} --purge
-      when: 
-        - template_status.rc == 0
-        - image_download.changed
- 
-    - name: Rebuild the template
-      when: 
-        - inventory_hostname == target_host
-        - image_download.changed or 
-          template_status.rc != 0
-      block:
- 
-        - name: Create the base VM shell
-          ansible.builtin.command: >
-            qm create {{ template_vmid }}
-            --memory {{ template_memory }}
-            --core {{ template_cores }}
-            --net0 virtio,bridge=vmbr0
-            --scsihw virtio-scsi-pci
-            --bios ovmf
-            --machine q35
-            --efidisk0 {{ vm_storage }}:0,pre-enrolled-keys=0
-            --name {{ template_name }}
- 
-        - name: Import the cloud image as the primary disk
-          ansible.builtin.command: qm set {{ template_vmid }} --scsi0 {{ vm_storage }}:0,import-from={{ image_dest }}
- 
-        - name: Attach a cloud-init drive
-          ansible.builtin.command: qm set {{ template_vmid }} --scsi1 {{ vm_storage }}:cloudinit
- 
-        - name: Set boot order to the primary disk
-          ansible.builtin.command: qm set {{ template_vmid }} --boot order=scsi0
- 
-        - name: Add serial console for cloud-init consoles
-          ansible.builtin.command: qm set {{ template_vmid }} --serial0 socket --vga serial0
- 
-        - name: Convert the VM to a template
-          ansible.builtin.command: qm template {{ template_vmid }}
-
-        - name: Send notification
-          ansible.builtin.uri:
-            url: "{{  ntfy_url }}/{{ ntfy_topic }}"
-            method: POST
-            user: "{{ ntfy_user }}"
-            password: "{{ lookup('env', 'NTFY_PASSWORD') }}"
-            force_basic_auth: true
-            body: |
-              The {{ template_os | capitalize }} {{ os_release }} cloud-init template has been successfully updated on {{ inventory_hostname }}.
-            headers:
-              Title: "Cloud-init template updated"
-              Priority: "min"
-              Tags: "white_check_mark"
-          when: ntfy_url is defined

ansible/proxmox/update_proxmox.yml

@@ -94,33 +94,12 @@
             retries: 60
             delay: 15
 
-          - name: Pause 15s 
-            ansible.builtin.pause:
-              seconds: 15
-
           - name: Update packages
             ansible.builtin.apt:
               upgrade: full
               autoremove: true
               autoclean: true
-
-          - name: Check whether Proxmox subscription nag is already patched
-            ansible.builtin.shell:
-              cmd: >
-                grep -Fq
-                'return (typeof orig_cmd === "function" && (orig_cmd(), true));'
-                /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js
-            register: proxmox_nag_patch
-            changed_when: false
-            failed_when: false
-
-          - name: Patch Proxmox subscription nag
-            ansible.builtin.lineinfile:
-              path: /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js
-              insertafter: 'checked_command: function (orig_cmd) {'
-              line: '            return (typeof orig_cmd === "function" && (orig_cmd(), true));'
-            when: proxmox_nag_patch.rc != 0
-
+              
           - name: Disable Ceph rebalancing
             ansible.builtin.command: ceph osd set noout
 

kubernetes/cilium/bgp.yaml

@@ -1,49 +0,0 @@
----
-apiVersion: cilium.io/v2
-kind: CiliumBGPPeerConfig
-metadata:
-  name: bgp-cilium-peers
-spec:
-  timers:
-    holdTimeSeconds: 9
-    keepAliveTimeSeconds: 3
-  gracefulRestart:
-    enabled: true
-    restartTimeSeconds: 15
-  families:
-    - afi: ipv4
-      safi: unicast
-      advertisements:
-        matchLabels:
-          advertise: "bgp"
----
-apiVersion: cilium.io/v2
-kind: CiliumBGPClusterConfig
-metadata:
-  name: bgp-opnsense
-spec:
-  nodeSelector:
-    matchLabels:
-      kubernetes.io/role: "worker"
-  bgpInstances:
-  - name: "bgp-cilium-cluster"
-    localASN: 64513 # Cilium ASN
-    peers:
-    - name: "opnsense"
-      peerASN: 64512 # OPNsense ASN
-      peerAddress: 192.168.66.1  # OPNsense IP on the Lab network
-      peerConfigRef:
-        name: "bgp-cilium-peers"
----
-apiVersion: cilium.io/v2
-kind: CiliumBGPAdvertisement
-metadata:
-  name: bgp-advertisement
-  labels:
-    advertise: bgp
-spec:
-  advertisements:
-    - advertisementType: "Service"
-      service:
-        addresses:
-          - LoadBalancerIP

opnsense/10-wan

@@ -84,7 +84,7 @@ if ($type === "MASTER") {
         write_config("disable interface '$ifkey' due CARP event '$type'", false);
         interface_configure(false, $ifkey, false, false);
         // Remove WAN default gateway
-        mwexecf("/sbin/route delete default");
+        mwexec("/sbin/route delete default");
         foreach ($config['OPNsense']['Gateways']['gateway_item'] as $gw) {
             if ($gw['name'] === $gw_name) {
                 $gw_ip = $gw['gateway'];
@@ -92,9 +92,9 @@ if ($type === "MASTER") {
             }
         }
         // Shutdown WAN interface
-        mwexecf("/sbin/ifconfig {$real_if} down")
+        mwexec("/sbin/ifconfig {$real_if} down")
         // Add fallback default gateway
-        mwexecf("/sbin/route add default {$gw_ip}");
+        mwexec("/sbin/route add default {$gw_ip}");
         // Create lock file
         file_put_contents($lock_file, time());
         sleep($lock_default_age);

terraform/modules/pve_vm/variables.tf

@@ -12,7 +12,7 @@ variable "node_datastore" {
 variable "vm_template" {
   description = "Template of the VM"
   type        = string
-  default     = "ubuntu-noble-cloud"
+  default     = "ubuntu-cloud"
 }
 
 variable "vm_name" {

terraform/projects/kubernetes_vms/README.md

@@ -1,93 +0,0 @@
-# Terraform – Kubernetes VMs
-
-Provisions the Kubernetes node VMs on Proxmox for the homelab Kubernetes cluster.
-
-## Overview
-
-This Terraform project clones the `ubuntu-noble-cloud` template into virtual machines
-on the proxmox nodes, following a capacity-aware placement algorithm:
-
-1. **Control plane (master)** VMs are spread one per node, on the nodes with the most
-   free memory that still fit the role's `ram`.
-2. **Pack roles (worker)** are water-filled over the remaining capacity, most-available
-   node first, without a per-node limit.
-
-Each VM is tagged with the environment (`vm_env`) and registered in a Terraform-backed
-Ansible inventory (via the `ansible` provider), so the `k8s_node` playbook can target
-them by role group.
-
-## Placement details
-
-- Online Proxmox nodes are ordered by available memory (most available first, node name
-  as tie-break).
-- The control-plane role must fit entirely on distinct nodes with `free_mib >= ram`.
-- Workers are packed into residual capacity (`free_mib - ram_per_cp_vm_on_node`),
-  again most-available first.
-- Naming: `kub-<env>-m<NN>` for masters and `kub-<env>-w<NN>` for workers, e.g.
-  `kub-prod-m01`, `kub-prod-w01`.
-
-Deployment is blocked by `check` blocks when the fleet cannot be placed:
-
-- no online node is available,
-- not enough nodes have the free memory required for the control-plane role,
-- residual capacity cannot fit all worker VMs.
-
-## Prerequisites
-
-- Proxmox with:
-  - a cloneable template named `ubuntu-noble-cloud` (from the `pve_vm` module),
-  - a `ceph-workload` datastore,
-  - VLAN bridge `vlan<id>` (default `vlan66`),
-  - an API token and an SSH user/password with rights to create VMs on the nodes.
-- Terraform with the `bpg/proxmox` and `ansible/ansible` providers (init downloads them).
-
-## Input variables
-
-| Variable                     | Type     | Required | Default                      | Description                                        |
-| ---------------------------- | -------- | -------- | ---------------------------- | -------------------------------------------------- |
-| `proxmox_endpoint`           | string   | yes      | –                            | Proxmox API URL                                    |
-| `proxmox_api_token`          | string   | yes      | –                            | Proxmox API token (secret)                         |
-| `proxmox_ssh_username`       | string   | yes      | –                            | Proxmox SSH user for placement (secret)            |
-| `proxmox_ssh_password`       | string   | yes      | –                            | Proxmox SSH password (secret)                      |
-| `vm_env`                     | string   | no       | `"prod"`                     | Environment, part of the VM name and tags          |
-| `master_count`               | number   | no       | `null` → `vm_attr` default   | Number of master VMs                               |
-| `master_cpu`                 | number   | no       | `null` → `vm_attr` default   | vCPU cores for masters                             |
-| `master_ram`                 | number   | no       | `null` → `vm_attr` default   | Master RAM in **GiB** (converted to MiB internally)|
-| `master_vlan`                | number   | no       | `null` → `vm_attr` default   | VLAN for masters                                   |
-| `worker_count`               | number   | no       | `null` → `vm_attr` default   | Number of worker VMs                               |
-| `worker_cpu`                 | number   | no       | `null` → `vm_attr` default   | vCPU cores for workers                             |
-| `worker_ram`                 | number   | no       | `null` → `vm_attr` default   | Worker RAM in **GiB**                              |
-| `worker_vlan`                | number   | no       | `null` → `vm_attr` default   | VLAN for workers                                   |
-| `vm_attr`                    | map(… )  | no       | see below                    | Full per-role VM defaults                          |
-
-The `vm_attr` map holds the fallback attributes used when the corresponding
-`master_*`/`worker_*` override is `null`:
-
-```hcl
-vm_attr = {
-  master = { ram = 2048, cpu = 2, vlan = 66, count = 3 }
-  worker = { ram = 4096, cpu = 2, vlan = 66, count = 3 }
-}
-```
-
-> `ram` values inside `vm_attr` are MiB; the `master_ram` / `worker_ram` overrides are
-> set in GiB and converted internally (`× 1024`). The placement math always works in MiB.
-
-## Usage
-
-```bash
-terraform init
-terraform plan
-terraform apply
-```
-
-The `vm_ip` output maps VM name to assigned IP and feeds the Ansible inventory
-(`inventories/`) of the `k8s_node` playbook.
-
-## File reference
-
-| File         | Purpose                                                   |
-| ------------ | --------------------------------------------------------- |
-| `main.tf`    | Placement algorithm, `pve_vm` module wiring, checks       |
-| `variables.tf` | Clusters, Proxmox credentials, per-role defaults/overrides |
-| `provider.tf`  | `proxmox` and `ansible` provider configuration          |

terraform/projects/kubernetes_vms/main.tf

@@ -1,198 +0,0 @@
-module "pve_vm" {
-  source    = "../../modules/pve_vm"
-  for_each  = local.vm_list
-  node_name = each.value.node_name
-  vm_name   = each.key
-  vm_cpu    = each.value.vm_cpu
-  vm_ram    = each.value.vm_ram
-  vm_vlan   = each.value.vm_vlan
-  vm_tags   = ["${var.vm_env}"]
-}
-
-locals {
-  node_stats = [
-    for i, name in data.proxmox_virtual_environment_nodes.pve_nodes.names : {
-      name     = name
-      free_mib = floor(data.proxmox_virtual_environment_nodes.pve_nodes.memory_available[i] / 1024 / 1024)
-    }
-    if data.proxmox_virtual_environment_nodes.pve_nodes.online[i]
-  ]
-
-  max_free_mib = length(local.node_stats) > 0 ? max([for node in local.node_stats : node.free_mib]...) : 0
-
-  # Online nodes ordered by available memory, most available first (name as tie-break)
-  node_order = [
-    for key in sort([
-      for node in local.node_stats : "${format("%012d", local.max_free_mib - node.free_mib)}|${node.name}"
-    ]) : split("|", key)[1]
-  ]
-
-  node_free = { for node in local.node_stats : node.name => node.free_mib }
-
-  # Per-role simple overrides merged over vm_attr defaults (null keeps the default)
-  role_overrides = {
-    for role, cfg in var.vm_attr : role => lookup({
-      "master" = { count = var.master_count, ram = var.master_ram != null ? var.master_ram * 1024 : null, cpu = var.master_cpu, vlan = var.master_vlan }
-      "worker" = { count = var.worker_count, ram = var.worker_ram != null ? var.worker_ram * 1024 : null, cpu = var.worker_cpu, vlan = var.worker_vlan }
-    }, role, {})
-  }
-
-  # Internal placement rule: the control-plane role (master) sits one per node, pack roles are unlimited
-  cp_role_name      = "master"
-  role_max_per_node = { for role, cfg in var.vm_attr : role => role == local.cp_role_name ? 1 : 0 }
-
-  effective_vm_attr = {
-    for role, cfg in var.vm_attr : role => merge(cfg, {
-      for attr, value in local.role_overrides[role] : attr => value if value != null
-    }, { max_per_node = local.role_max_per_node[role] })
-  }
-
-  # Control plane (max_per_node = 1): one VM per node, only on nodes with enough free memory
-  cp_role = [for role, cfg in local.effective_vm_attr : role if cfg.max_per_node == 1][0]
-  cp_attr = local.effective_vm_attr[local.cp_role]
-
-  cp_feasible = [
-    for node in local.node_order : node
-    if local.node_free[node] >= local.cp_attr.ram
-  ]
-
-  cp_nodes = length(local.cp_feasible) >= local.cp_attr.count ? [
-    for i in range(local.cp_attr.count) : local.cp_feasible[i]
-  ] : []
-
-  cp_hosts = toset(local.cp_nodes)
-
-  # Residual capacity per node after control-plane VMs are reserved
-  residual_mib = {
-    for node in local.node_order :
-    node => local.node_free[node] - (contains(local.cp_hosts, node) ? local.cp_attr.ram : 0)
-  }
-
-  max_residual_mib = length(local.node_order) > 0 ? max([for node in local.node_order : local.residual_mib[node]]...) : 0
-
-  # Remaining roles (max_per_node != 1): pack nodes with residual capacity, most available first
-  pack_roles = [for role, cfg in local.effective_vm_attr : role if cfg.max_per_node != 1]
-
-  pack_slots = {
-    for role in local.pack_roles : role => {
-      for node in local.node_order :
-      node => min(
-        floor(local.residual_mib[node] / local.effective_vm_attr[role].ram),
-        local.effective_vm_attr[role].max_per_node == 0
-        ? floor(local.residual_mib[node] / local.effective_vm_attr[role].ram)
-        : local.effective_vm_attr[role].max_per_node
-      )
-    }
-  }
-
-  pack_precedence = {
-    for role in local.pack_roles : role => [
-      for key in sort([
-        for node in local.node_order :
-        "${format("%012d", local.max_residual_mib - local.residual_mib[node])}|${node}"
-        if local.pack_slots[role][node] >= 1
-      ]) : split("|", key)[1]
-    ]
-  }
-
-  pack_slots_list = {
-    for role in local.pack_roles : role =>
-    [for node in local.pack_precedence[role] : local.pack_slots[role][node]]
-  }
-
-  # Water-fill: each node takes VMs until it runs out of capacity or demand is met
-  pack_placed = {
-    for role in local.pack_roles : role => [
-      for i in range(length(local.pack_precedence[role])) :
-      min(local.pack_slots_list[role][i], max(0, local.effective_vm_attr[role].count - try(sum([for j in range(i) : local.pack_slots_list[role][j]]), 0)))
-    ]
-  }
-
-  pack_prefix = {
-    for role in local.pack_roles : role => [
-      for i in range(length(local.pack_placed[role])) :
-      try(sum([for j in range(i) : local.pack_placed[role][j]]), 0)
-    ]
-  }
-
-  cp_placements = [
-    for i in range(length(local.cp_nodes)) : {
-      vm_name = "kub-${var.vm_env}-${substr(local.cp_role, 0, 1)}${format("%01d", i + 1)}"
-      vm_role = local.cp_role
-      vm_node = local.cp_nodes[i]
-    }
-  ]
-
-  pack_placements = flatten([
-    for role in local.pack_roles : [
-      for p in range(length(local.pack_precedence[role])) : [
-        for k in range(local.pack_placed[role][p]) : {
-          vm_name = "kub-${var.vm_env}-${substr(role, 0, 1)}${format("%01d", local.pack_prefix[role][p] + k + 1)}"
-          vm_role = role
-          vm_node = local.pack_precedence[role][p]
-        }
-      ]
-    ]
-  ])
-
-  vm_list = {
-    for vm in concat(local.cp_placements, local.pack_placements) :
-    vm.vm_name => {
-      node_name = vm.vm_node
-      vm_cpu    = local.effective_vm_attr[vm.vm_role].cpu
-      vm_ram    = local.effective_vm_attr[vm.vm_role].ram
-      vm_vlan   = local.effective_vm_attr[vm.vm_role].vlan
-      vm_role   = vm.vm_role
-    }
-  }
-
-  roles = toset([for vm in local.vm_list : vm.vm_role])
-}
-
-data "proxmox_virtual_environment_nodes" "pve_nodes" {}
-
-check "proxmox_nodes" {
-  assert {
-    condition     = length(local.node_order) > 0
-    error_message = "No online Proxmox node available for deployment."
-  }
-}
-
-check "control_plane_capacity" {
-  assert {
-    condition     = length(local.cp_nodes) == local.cp_attr.count
-    error_message = "Cannot place ${local.cp_attr.count} ${local.cp_role} VM(s): only ${length(local.cp_feasible)} online node(s) have at least ${local.cp_attr.ram} MiB free memory."
-  }
-}
-
-check "pack_capacity" {
-  assert {
-    condition = length(local.pack_roles) == 0 ? true : alltrue([
-      for role in local.pack_roles :
-      try(sum(local.pack_placed[role]), 0) == local.effective_vm_attr[role].count
-    ])
-    error_message = "Cannot place all pack VMs: ${join(", ", [
-      for role in local.pack_roles :
-      "${role} needs ${local.effective_vm_attr[role].count} VM(s) (${local.effective_vm_attr[role].ram} MiB each), residual free capacity fits only ${try(sum(local.pack_placed[role]), 0)} VM(s)."
-      if try(sum(local.pack_placed[role]), 0) != local.effective_vm_attr[role].count
-    ])}"
-  }
-}
-
-output "vm_ip" {
-  value = { for k, v in module.pve_vm : k => v.vm_ip }
-}
-
-resource "ansible_group" "vm_groups" {
-  for_each = local.roles
-  name     = each.key
-}
-
-resource "ansible_host" "vm_hosts" {
-  for_each = module.pve_vm
-  name     = each.key
-  variables = {
-    ansible_host = each.value.vm_ip
-  }
-  groups = [local.vm_list[each.key].vm_role]
-}

terraform/projects/kubernetes_vms/provider.tf

@@ -1,21 +0,0 @@
-terraform {
-  required_providers {
-    proxmox = {
-      source = "bpg/proxmox"
-    }
-    ansible = {
-      source = "ansible/ansible"
-    }
-  }
-}
-
-provider "proxmox" {
-  endpoint  = var.proxmox_endpoint
-  api_token = var.proxmox_api_token
-  insecure  = false
-  ssh {
-    agent       = false
-    username    = var.proxmox_ssh_username
-    password    = var.proxmox_ssh_password
-  }
-}

terraform/projects/kubernetes_vms/variables.tf

@@ -1,135 +0,0 @@
-variable "proxmox_endpoint" {
-  description = "Proxmox URL endpoint"
-  type        = string
-}
-
-variable "proxmox_api_token" {
-  description = "Proxmox API token"
-  type        = string
-  sensitive   = true
-}
-
-variable "proxmox_ssh_username" {
-  description = "Proxmox SSH username"
-  type        = string
-  sensitive   = true
-}
-
-variable "proxmox_ssh_password" {
-  description = "Proxmox SSH password"
-  type        = string
-  sensitive   = true
-}
-
-variable "vm_attr" {
-  description = "VM attributes per role. The control-plane role (master) is placed one per node; other roles are packed without a per-node limit."
-  type = map(object({
-    ram   = number
-    cpu   = number
-    vlan  = number
-    count = number
-  }))
-  default = {
-    "master" = { ram = 2048, cpu = 2, vlan = 66, count = 3 }
-    "worker" = { ram = 4096, cpu = 2, vlan = 66, count = 3 }
-  }
-
-  validation {
-    condition     = alltrue([for v in values(var.vm_attr) : v.count >= 1])
-    error_message = "Each role must deploy at least 1 VM."
-  }
-}
-
-variable "master_count" {
-  description = "Master role VM count override. null keeps the vm_attr default."
-  type        = number
-  default     = null
-
-  validation {
-    condition     = var.master_count == null || var.master_count >= 1
-    error_message = "master_count must be null or at least 1."
-  }
-}
-
-variable "master_cpu" {
-  description = "Master role CPU override. null keeps the vm_attr default."
-  type        = number
-  default     = null
-
-  validation {
-    condition     = var.master_cpu == null || var.master_cpu >= 1
-    error_message = "master_cpu must be null or at least 1."
-  }
-}
-
-variable "master_ram" {
-  description = "Master role RAM override in GiB (converted to MiB internally). null keeps the vm_attr default."
-  type        = number
-  default     = null
-
-  validation {
-    condition     = var.master_ram == null || var.master_ram >= 1
-    error_message = "master_ram must be null or at least 1 GiB."
-  }
-}
-
-variable "master_vlan" {
-  description = "Master role VLAN override. null keeps the vm_attr default."
-  type        = number
-  default     = null
-
-  validation {
-    condition     = var.master_vlan == null || var.master_vlan >= 1
-    error_message = "master_vlan must be null or at least 1."
-  }
-}
-
-variable "worker_count" {
-  description = "Worker role VM count override. null keeps the vm_attr default."
-  type        = number
-  default     = null
-
-  validation {
-    condition     = var.worker_count == null || var.worker_count >= 1
-    error_message = "worker_count must be null or at least 1."
-  }
-}
-
-variable "worker_cpu" {
-  description = "Worker role CPU override. null keeps the vm_attr default."
-  type        = number
-  default     = null
-
-  validation {
-    condition     = var.worker_cpu == null || var.worker_cpu >= 1
-    error_message = "worker_cpu must be null or at least 1."
-  }
-}
-
-variable "worker_ram" {
-  description = "Worker role RAM override in GiB (converted to MiB internally). null keeps the vm_attr default."
-  type        = number
-  default     = null
-
-  validation {
-    condition     = var.worker_ram == null || var.worker_ram >= 1
-    error_message = "worker_ram must be null or at least 1 GiB."
-  }
-}
-
-variable "worker_vlan" {
-  description = "Worker role VLAN override. null keeps the vm_attr default."
-  type        = number
-  default     = null
-
-  validation {
-    condition     = var.worker_vlan == null || var.worker_vlan >= 1
-    error_message = "worker_vlan must be null or at least 1."
-  }
-}
-
-variable "vm_env" {
-  description = "VM environment"
-  type        = string
-  default     = "prod"
-}

terraform/projects/simple-vm-telmate/main.tf

@@ -6,7 +6,7 @@ resource "proxmox_vm_qemu" "cloudinit-test" {
 
   target_node = "zenith"
 
-  clone = "ubuntu-noble-cloud"
+  clone = "ubuntu-cloud"
 
   agent = 0
 

terraform/projects/simple-vm/variables.tf

@@ -23,7 +23,7 @@ variable "node_datastore" {
 variable "vm_template" {
   description = "Template of the VM"
   type        = string
-  default     = "ubuntu-noble-cloud"
+  default     = "ubuntu-cloud"
 }
 
 variable "vm_name" {