diff --git a/content/post/deploy kub application.md b/content/post/deploy kub application.md index fdb9b33..2eb691c 100644 --- a/content/post/deploy kub application.md +++ b/content/post/deploy kub application.md @@ -4,7 +4,14 @@ title: Template description: date: draft: true -tags: +tags: + - kubernetes + - helm + - bgp + - opnsense + - cilium + - nginx-ingress-controller + - cert-manager categories: --- @@ -85,67 +92,416 @@ With Cilium + BGP, you get: ### BGP Setup -BGP is +By default, BGP is disabled by default, both on my OPNsense router and in Cilium. Let’s enable it on both ends. #### On OPNsense -Following the [OPNsense BGP documentation](https://docs.opnsense.org/manual/dynamic_routing.html#bgp-section), to enable BGP, I need to install a plugin. Go to `System` > `Firmware` > `Plugins` and install the `os-frr` plugin: -![opnsense-add-os-frr-plugin.png](img/opnsense-add-os-frr-plugin.png) +According to the [official OPNsense documentation](https://docs.opnsense.org/manual/dynamic_routing.html#bgp-section), enabling BGP requires installing a plugin. -First, enable the plugin in the `Routing` > `General`: -![opnsense-enable-routing-frr-plugin.png](img/opnsense-enable-routing-frr-plugin.png) +Head to `System` > `Firmware` > `Plugins` and install the `os-frr` plugin: +![ ](img/opnsense-add-os-frr-plugin.png) +Install `os-frr` plugin in OPNsense -Then, go to the `BGP` section, enable it in the `General` tab by ticking the box. Set the BGP AS Number, I set `64512` as it is the first in the AS (autonomous System) private range, you can find the ranges [here](https://en.wikipedia.org/wiki/Autonomous_system_(Internet)#ASN_Table): -![opnsense-enable-bgp.png](img/opnsense-enable-bgp.png) +Once installed, enable the plugin under `Routing` > `General`: +![ ](img/opnsense-enable-routing-frr-plugin.png) +Enable routing in OPNsense -Now create the neighbors, I will add the 3 workers, I don't add the masters as they won't run any workload. I set the node IP in the `Peer-IP` field. For the `Remote AS`, I use the same number for all the node: `64513`, I set the Interface name in `Update-Source Interface`, which is `Lab`. Finally I tick the box for `Next-Hop-Self`: -![opnsense-bgp-create-neighbor.png](img/opnsense-bgp-create-neighbor.png) +Then navigate to the `BGP` section. In the **General** tab: +- Tick the box to enable BGP. +- Set your **BGP ASN**. I used `64512`, the first private ASN from the reserved range (see [ASN table](https://en.wikipedia.org/wiki/Autonomous_system_\(Internet\)#ASN_Table)): +![ ](img/opnsense-enable-bgp.png) +General BGP configuration in OPNsense -Finally, my neighbor list look like this: -![opnsense-bgp-nieghbor-list.png](img/opnsense-bgp-nieghbor-list.png) +Now create your BGP neighbors. I’m only peering with my **worker nodes** (since only they run workloads). For each neighbor: +- Set the node’s IP in `Peer-IP` +- Use `64513` as the **Remote AS** (Cilium’s ASN) +- Set `Update-Source Interface` to `Lab` +- Tick `Next-Hop-Self`: +![ ](img/opnsense-bgp-create-neighbor.png) +BGP neighbor configuration in OPNsense +Here’s how my neighbor list looks once complete: +![ ](img/opnsense-bgp-nieghbor-list.png) +BGP neighbor list + +Don’t forget to create a firewall rule allowing BGP (port `179/TCP`) from the **Lab** VLAN to the firewall: +![ ](img/opnsense-create-firewall-rule-bgp-peering.png) +Allow TCP/179 from Lab to OPNsense #### In Cilium -### Deploying a LoadBalancer with BGP -#### Using an IP Address -#### Using a URL +I already had Cilium installed and couldn’t find a way to enable BGP with the CLI, so I simply reinstalled it with the BGP option: + +```bash +cilium uninstall +cilium install --set bgpControlPlane.enabled=true +``` + +Next, I want only **worker nodes** to establish BGP peering. I add a label to each one for the future `nodeSelector`: +```bash +kubectl label node apex-worker node-role.kubernetes.io/worker="" +kubectl label node vertex-worker node-role.kubernetes.io/worker="" +kubectl label node zenith-worker node-role.kubernetes.io/worker="" +``` +```plaintext +NAME STATUS ROLES AGE VERSION +apex-master Ready control-plane 5d4h v1.32.7 +apex-worker Ready worker 5d1h v1.32.7 +vertex-master Ready control-plane 5d1h v1.32.7 +vertex-worker Ready worker 5d1h v1.32.7 +zenith-master Ready control-plane 5d1h v1.32.7 +zenith-worker Ready worker 5d1h v1.32.7 +``` + +For the entire BGP configuration, I need: +- **CiliumBGPClusterConfig**: BGP settings for the Cilium cluster, including its local ASN and its peer +- **CiliumBGPPeerConfig**: Sets BGP timers, graceful restart, and route advertisement settings. +- **CiliumBGPAdvertisement**: Defines which Kubernetes services should be advertised via BGP. +- **CiliumLoadBalancerIPPool**: Configures the range of IPs assigned to Kubernetes LoadBalancer services. + +```yaml +--- +apiVersion: cilium.io/v2alpha1 +kind: CiliumBGPClusterConfig +metadata: + name: bgp-cluster +spec: + nodeSelector: + matchLabels: + node-role.kubernetes.io/worker: "" # Only for worker nodes + bgpInstances: + - name: "cilium-bgp-cluster" + localASN: 64513 # Cilium ASN + peers: + - name: "pfSense-peer" + peerASN: 64512 # OPNsense ASN + peerAddress: 192.168.66.1 # OPNsense IP + peerConfigRef: + name: "bgp-peer" +--- +apiVersion: cilium.io/v2alpha1 +kind: CiliumBGPPeerConfig +metadata: + name: bgp-peer +spec: + timers: + holdTimeSeconds: 9 + keepAliveTimeSeconds: 3 + gracefulRestart: + enabled: true + restartTimeSeconds: 15 + families: + - afi: ipv4 + safi: unicast + advertisements: + matchLabels: + advertise: "bgp" +--- +apiVersion: cilium.io/v2alpha1 +kind: CiliumBGPAdvertisement +metadata: + name: bgp-advertisement + labels: + advertise: bgp +spec: + advertisements: + - advertisementType: "Service" + service: + addresses: + - LoadBalancerIP + selector: + matchExpressions: + - { key: somekey, operator: NotIn, values: [ never-used-value ] } +--- +apiVersion: "cilium.io/v2alpha1" +kind: CiliumLoadBalancerIPPool +metadata: + name: "dmz" +spec: + blocks: + - start: "192.168.55.20" # LB Range Start IP + stop: "192.168.55.250" # LB Range End IP +``` + +Apply it: +```bash +kubectl apply -f bgp.yaml + +ciliumbgpclusterconfig.cilium.io/bgp-cluster created +ciliumbgppeerconfig.cilium.io/bgp-peer created +ciliumbgpadvertisement.cilium.io/bgp-advertisement created +ciliumloadbalancerippool.cilium.io/dmz created +``` + +If everything works, you should see the BGP sessions **established** with your workers: +```bash +cilium bgp peers + +Node Local AS Peer AS Peer Address Session State Uptime Family Received Advertised +apex-worker 64513 64512 192.168.66.1 established 6m30s ipv4/unicast 1 2 +vertex-worker 64513 64512 192.168.66.1 established 7m9s ipv4/unicast 1 2 +zenith-worker 64513 64512 192.168.66.1 established 6m13s ipv4/unicast 1 2 +``` + +### Deploying a `LoadBalancer` Service with BGP + +Let’s quickly validate that the setup works by deploying a test `Deployment` and `LoadBalancer` `Service`: +```yaml +--- +apiVersion: v1 +kind: Service +metadata: + name: test-lb +spec: + type: LoadBalancer + ports: + - port: 80 + targetPort: 80 + protocol: TCP + name: http + selector: + svc: test-lb +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginx +spec: + selector: + matchLabels: + svc: test-lb + template: + metadata: + labels: + svc: test-lb + spec: + containers: + - name: web + image: nginx + imagePullPolicy: IfNotPresent + ports: + - containerPort: 80 + readinessProbe: + httpGet: + path: / + port: 80 +``` + +Check if it gets an external IP: +```bash +kubectl get services test-lb + +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +test-lb LoadBalancer 10.100.167.198 192.168.55.20 80:31350/TCP 169m +``` + +The service got the first IP from our defined pool: `192.168.55.20`. + +Now from any device on the LAN, try to reach that IP on port 80: +![Test LoadBalancer service with BGP](img/k8s-test-loadbalancer-service-with-bgp.png) + +✅ Our pod is reachable through BGP-routed `LoadBalancer` IP, first step successful! --- ## Kubernetes Ingress -TODO add why we need service -### What is a Kubernetes Ingress +We managed to expose a pod externally using a `LoadBalancer` service and a BGP-assigned IP address. This approach works great for testing, but it doesn't scale well. -explain what is an Ingress and its purpose +Imagine having 10, 20, or 50 different services, would I really want to allocate 50 IP addresses, and clutter my firewall and routing tables with 50 BGP entries? Definitely not. -### How Ingress Work +That’s where **Ingress** kicks in. +### What Is a Kubernetes Ingress? +A **Kubernetes Ingress** is an API object that manages **external access to services** in a cluster, typically HTTP and HTTPS, all through a single entry point. + +Instead of assigning one IP per service, you define routing rules based on: +- **Hostnames** (`app1.vezpi.me`, `blog.vezpi.me`, etc.) +- **Paths** (`/grafana`, `/metrics`, etc.) + +With Ingress, I can expose multiple services over the same IP and port (usually 443 for HTTPS), and Kubernetes will know how to route the request to the right backend service. + +Here is an example of a simple `Ingress`, routing traffic of `test.vezpi.me` to the `test-lb` service on port 80: +```yaml --- -## Ingress Controller +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: test-ingress +spec: + rules: + - host: test.vezpi.me + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: test-lb + port: + number: 80 +``` -### What is an Ingress Controller +### Ingress Controller -explain what is an Ingress Controller and its purpose +On its own, an Ingress is just a set of routing rules. It doesn’t actually handle traffic. To bring it to life, I need an **Ingress Controller** which will: +- Watches the Kubernetes API for `Ingress` resources. +- Opens HTTP(S) ports on a `LoadBalancer` or `NodePort` service. +- Routes traffic to the correct `Service` based on the `Ingress` rules. -### Which Ingress Controller to Use +Think of it as a reverse proxy (like NGINX or Traefik), but integrated with Kubernetes. -comparison between ingress controller -which one I picked and why +Since I’m looking for something simple, stable, well-maintained, and with a large community, I went with **NGINX Ingress Controller**. ### Install NGINX Ingress Controller -detail installation of NGINX Ingress Controller -verify ingress controller service +I install it using Helm, I set `controller.ingressClassResource.default=true` to define `nginx` as default for all my future ingresses: +```bash +helm install ingress-nginx \ + --repo=https://kubernetes.github.io/ingress-nginx \ + --namespace=ingress-nginx \ + --create-namespace ingress-nginx \ + --set controller.ingressClassResource.default=true +``` +```plaintext +NAME: ingress-nginx +LAST DEPLOYED: Wed Jul 23 15:44:47 2025 +NAMESPACE: ingress-nginx +STATUS: deployed +REVISION: 1 +TEST SUITE: None +NOTES: +The ingress-nginx controller has been installed. +It may take a few minutes for the load balancer IP to be available. +You can watch the status by running 'kubectl get service --namespace ingress-nginx ingress-nginx-controller --output wide --watch' + +An example Ingress that makes use of the controller: + apiVersion: networking.k8s.io/v1 + kind: Ingress + metadata: + name: example + namespace: foo + spec: + ingressClassName: nginx + rules: + - host: www.example.com + http: + paths: + - pathType: Prefix + backend: + service: + name: exampleService + port: + number: 80 + path: / + # This section is only required if TLS is to be enabled for the Ingress + tls: + - hosts: + - www.example.com + secretName: example-tls + +If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided: + + apiVersion: v1 + kind: Secret + metadata: + name: example-tls + namespace: foo + data: + tls.crt: + tls.key: +``` + +My NGINX Ingress Controller is now installed and its service picked the 2nd IP in the load balancer range, `192.168.55.21`: +```bash +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR +ingress-nginx-controller LoadBalancer 10.106.236.13 192.168.55.21 80:31195/TCP,443:30974/TCP 75s app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx +``` + +>💡 I want to make sure my controller will always pick the same IP. + +I will create 2 separate pools, one dedicated for the Ingress Controller with only one IP, and another one for anything else. +```yaml +--- +apiVersion: "cilium.io/v2alpha1" +kind: CiliumLoadBalancerIPPool +metadata: + name: "ingress-nginx" +spec: + blocks: + - cidr: "192.168.55.55/32" # Ingress Controller IP + serviceSelector: + matchLabels: + app.kubernetes.io/name: ingress-nginx + app.kubernetes.io/component: controller +--- +apiVersion: "cilium.io/v2alpha1" +kind: CiliumLoadBalancerIPPool +metadata: + name: "default" +spec: + blocks: + - start: "192.168.55.100" # LB Start IP + stop: "192.168.55.250" # LB Stop IP + serviceSelector: + matchExpressions: + - key: app.kubernetes.io/name + operator: NotIn + values: + - ingress-nginx +``` + +After replacing the previous pool by these two, my Ingress Controller got the desired IP `192.168.55.55` and my `test-lb` service picked the first one `192.168.55.100` in the new range as expected. +```bash +NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +default test-lb LoadBalancer 10.100.167.198 192.168.55.100 80:31350/TCP 6h34m +ingress-nginx ingress-nginx-controller LoadBalancer 10.106.236.13 192.168.55.55 80:31195/TCP,443:30974/TCP 24m +``` + ### Associate a Service to an Ingress +Now let’s wire up a service to this controller. -oneline to explain how to use https +We transform our `LoadBalancer` service to a standard `ClusterIP` and add a minimal Ingress definition to expose my test pod over HTTP: +```yaml +--- +apiVersion: v1 +kind: Service +metadata: + name: test-lb +spec: + ports: + - port: 80 + targetPort: 80 + protocol: TCP + name: http + selector: + svc: test-lb +--- +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: test-ingress +spec: + rules: + - host: test.vezpi.me + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: test-lb + port: + number: 80 +``` + + +![Pasted_image_20250803215654.png](img/Pasted_image_20250803215654.png) --- ## Secure Connection with TLS -to use https +oneline to explain how to use https ### Cert-Manager diff --git a/static/img/Pasted_image_20250803215654.png b/static/img/Pasted_image_20250803215654.png new file mode 100644 index 0000000..a807534 Binary files /dev/null and b/static/img/Pasted_image_20250803215654.png differ diff --git a/static/img/k8s-test-loadbalancer-service-with-bgp.png b/static/img/k8s-test-loadbalancer-service-with-bgp.png new file mode 100644 index 0000000..22e7d11 Binary files /dev/null and b/static/img/k8s-test-loadbalancer-service-with-bgp.png differ diff --git a/static/img/opnsense-create-firewall-rule-bgp-peering.png b/static/img/opnsense-create-firewall-rule-bgp-peering.png new file mode 100644 index 0000000..00457a5 Binary files /dev/null and b/static/img/opnsense-create-firewall-rule-bgp-peering.png differ