Auto-update blog content from Obsidian: 2025-08-24 07:40:43

content/post/opnsense-crash.md

@@ -1,35 +1,37 @@
 ---
-slug: 
+slug:
 title: Template
-description: 
-date: 
+description:
+date: 2025-08-22
 draft: true
-tags: 
+tags:
+  - opnsense
 categories:
+  - homelab
 ---
 ## Intro
 
 This week, I experienced my first real problem on my homelab, which caused my whole home network to go down. 
 
-My OPNsense router crashed and after trying to recover , I finally had to reinstall it from scratch and restore almost all the configuration, thanks to a single XML file. In that story, I will tell you what happened, what I did to recover and what I shouldn't have done.
+My OPNsense router crashed and, after several failed recovery attempts, I finally had to reinstall it from scratch. Luckily, almost all of the configuration came back thanks to a single XML file. In that story, I will tell you what happened, what I did to recover and what I shouldn't have done.
 
-This kind of exercise is the worst thing you want to happen because it's never funny to have everything go boom, but this is, by far, the best way to learn.
+This kind of exercise is the worst thing you want to happen because it's never funny to have everything go boom, but this is by far the best way to learn.
 
 ## The Calm Before the Storm
 
-My OPNsense box had been running smoothly for months. Router, firewall, DNS, DHCP, VLANs, VPN, reverse proxy and even UniFi controller, all the pieces of my homelab network ran through it, but not only, it is also serving internet at home.
+My OPNsense box had been running smoothly for months. Router, firewall, DNS, DHCP, VLANs, VPN, reverse proxy and even UniFi controller: all the pieces of my homelab run through it. but not only, it is also serving internet at home.
 
-This is the heart of my network, I barely can't do anything without it now. I have detailed how this is working in my [Homelab]({{< ref "page/homelab" >}}) section. It was “just working,” and I wasn’t worried about it. I felt confident, its backup was living only inside the machine...
+This box is the heart of my network, without it, I can hardly do anything. I have detailed how this is working in my [Homelab]({{< ref "page/homelab" >}}) section. It was “just working,” and I wasn’t worried about it. I felt confident, its backup was living only inside the machine...
 
 Maybe too confident.
 
 ## The Unexpected Reboot
 
-Out of nowhere, the box rebooted by itself just before midnight. By chance, I was just passing by my rack on my way to bed. I knew the box rebooted because I heard its little beep it is doing when the machine start.
+Out of nowhere, the box rebooted by itself just before midnight. By chance, I was just passing by my rack on my way to bed. I knew it had rebooted because I heard its little startup beep.
 
 I wondered why the router restarted without my will. In my bed, I quickly checked if internet was working, and it was. But none of my services were available, my home automation or even this blog. I was tired, I would fix that the next day...
 
-In the morning, looking at the logs, I found the culprit
+In the morning, looking at the logs, I found the culprit:
 ```
 panic: double fault
 ```
@@ -48,7 +50,7 @@ caching certificate: decoding certificate metadata: unexpected end of JSON input
 
 It turned out that one of the cached certificates had been corrupted during the crash. Deleting its cache folder fixed Caddy, and suddenly all my HTTPS services were back online.
 
-I thought I had dodged the bullet. I didn't investigate much on the root cause analysis, the kernel logs were polluted by one of the interfaces flapping, I thought it was just a bug, instead, I checked for any updates, my first mistake.
+I thought I had dodged the bullet. I didn't investigate much on the root cause analysis: the kernel logs were polluted by one of the interfaces flapping, I thought it was just a bug. Instead, I went ahead and checked for updates, my first mistake.
 
 My OPNsense instance was in version 25.1, and the newer 25.7 was available. Let's upgrade it, yay!
 
@@ -64,7 +66,7 @@ Clicking the `Download configuration` button, I downloaded the current `config.x
 
 ## Filesystem Corruption
 
-I decided to recover the pkg database the worst possible way, I backed up the `/var/db/pkg` folder and I tried to `bootstrap` it.
+I decided to recover the pkg database the worst possible way, I backed up the `/var/db/pkg` folder and I tried to `bootstrap` it:
 ```bash
 cp -a /var/db/pkg /var/db/pkg.bak
 pkg bootstrap -f
@@ -73,17 +75,10 @@ pkg bootstrap -f
 The package management tool is not yet installed on your system.
 Do you want to fetch and install it now? [y/N]: y
 Bootstrapping pkg from https://pkg.opnsense.org/FreeBSD:14:amd64/25.7/latest, please wait...
-Verifying signature with trusted certificate pkg.opnsense.org.20250710... done
-Installing pkg-1.19.2_5...
-Extracting pkg-1.19.2_5:  13%
+[...]
 pkg-static: Fail to extract /usr/local/lib/libpkg.a from package: Write error
-Extracting pkg-1.19.2_5: 100%
-
 Failed to install the following 1 package(s): /tmp//pkg.pkg.scQnQs
-Bootstrapping pkg from https://opn-repo.routerperformance.net/repo/FreeBSD:14:amd64, please wait...
-pkg: Attempted to fetch https://opn-repo.routerperformance.net/repo/FreeBSD:14:amd64/Latest/pkg.pkg
-pkg: Attempted to fetch https://opn-repo.routerperformance.net/repo/FreeBSD:14:amd64/Latest/pkg.txz
-pkg: Error: Not Found
+[...]
 A pre-built version of pkg could not be found for your system.
 ```
 
@@ -123,6 +118,8 @@ I discovered the utility `opnsense-bootstrap`, which promises to reinstall all p
 - Remove all installed packages.
 - Fresh 25.7 base system and kernel will be downloaded and installed.
 - All standard OPNsense packages will be reinstalled.
+
+Wonderful!
 ```
 opnsense-bootstrap
 ```
@@ -130,7 +127,7 @@ opnsense-bootstrap
 This utility will attempt to turn this installation into the latest OPNsense 25.7 release. All packages will be deleted, the base system and kernel will be replaced, and if all went well the system will automatically reboot. Proceed with this action? [y/N]:
 ```
 
-I pressed `y`. This started well, but then... no more signal -> no more internet.
+I pressed `y`. This started well, but then... no more signal -> no more internet. I thought this bootstrap would save me. Instead, it buried me.
 
 🙈 Oops.
 
@@ -150,7 +147,7 @@ bad dir ino … mangled entry
 Input/output error
 ```
 
-The disk is in a bad shape, I can't do anything more for that instance, I'd better start from scratch now, I have backup, haven't it? (lol)
+The disk is in a bad shape, at this point, I couldn’t save the install anymore. Time to start from scratch. Luckily, I had a backup… right?
 
 I downloaded the latest OPNsense ISO (v25.7) and put it into a USB stick. I reinstall OPNsense and overwrite the current installation, I kept everything as default.
 
@@ -158,15 +155,17 @@ I downloaded the latest OPNsense ISO (v25.7) and put it into a USB stick. I rein
 
 OPNsense keeps the whole configuration in a single file: `/conf/config.xml`. That file was my lifeline.
 
-I copied the `config.xml`file saved earlier into the USB stick. When plugged into the fresh OPNsense box, I overwrite the file with this one:
+I copied the `config.xml`file saved earlier into the USB stick. When plugged into the fresh OPNsense box, I overwrite the file:
 ```bash
 mount -t msdosfs /dev/da0s1 /mnt
 cp /mnt/config.xml /conf/config.xml
 ```
 
-I placed the router back in the rack, powered it on and crossed my fingers. beep!
+I placed the router back in the rack, powered it on and crossed my fingers... *beep!* 🎉
 
-The DHCP gave me an address, good start. I could reach its URL, awesome. My configuration is here, almost everything, but the plugins. I can't install them right away because they need another update, let's update it!
+The DHCP gave me an address, good start. I could reach its URL, awesome. My configuration is here, almost everything but the plugins, as expected. I can't install them right away because they need another update, let's update it!
+
+This single XML file is the reason I could rebuild my router without losing my sanity
 
 DNS is KO because the AdGuard Home plugin is not installed, I temporary set the system DNS to `1.1.1.1`
 
@@ -176,13 +175,13 @@ During that upgrade, the system threw errors again… and then rebooted itself.
 
 I can officially say that my NVMe drive is dead.
 
-🪦 Rest in peace.
+🪦 Rest in peace, thank you for your great services.
 
-By chance, I have an unused NVMe Kingston drive of 512GB which was deliver with that box. I never used it because I preferred to use the one I was using before in my Vertex server.
+Luckily, I had a spare 512GB Kingston NVMe that came with that box. I never used it because I preferred to reuse the one inside my *Vertex* server.
 
-I redo the same steps to reinstall OPNsense on that disk. I could finally update OPNsense to 25.7.1 and reinstall all the official plugins that I was using. 
+I redo the same steps to reinstall OPNsense on that disk and this time everything worked: I could finally update OPNsense to 25.7.1 and reinstall all the official plugins that I was using. 
 
-To install custom plugins (AdGuard Home and Unifi), I had to add the custom repository `/usr/local/etc/pkg/repos/mimugmail.conf` (documentation [here](https://www.routerperformance.net/opnsense-repo/)) 
+To install custom plugins (AdGuard Home and Unifi), I had to add the custom repository in `/usr/local/etc/pkg/repos/mimugmail.conf` (documentation [here](https://www.routerperformance.net/opnsense-repo/)) 
 ```json
 mimugmail: {
   url: "https://opn-repo.routerperformance.net/repo/${ABI}",
@@ -193,45 +192,32 @@ mimugmail: {
 
 After a final reboot, the router is almost ready, but I still don't have DNS services. This is because AdGuard Home is not configured.
 
-⚠️ Custom plugin configuration is not saved within the standard backup in `config.xml`, which makes sense. As this is the only file I saved, I don't have any backup configuration for these plugins.
+⚠️ Custom plugin configuration is not saved within the backup in `config.xml`.
 
-Reconfigure AdGuard Home is pretty straight forward, finally my DNS is working and everything is back to nominal, except the UniFi controller/
+Reconfigure AdGuard Home is pretty straight forward, finally my DNS is working and everything is back to nominal... except the UniFi controller.
 
 ## Lessons Learned the Hard Way
 
-OPNsense Backups
-
-After a crash, healthcheck
-
-- **Don’t reuse old hardware for critical services.** That NVMe was living on borrowed time.
-    
-- **Always trust but verify storage.** Run `smartctl`, run `fsck`, and don’t ignore write errors.
-    
-- **`config.xml` is the crown jewel.** With it, a full reinstall is almost painless. Without it, I would have been rebuilding from scratch.
-    
-- **Custom plugin configs are not in config.xml.** If you rely on AdGuard, UniFi, etc., back them up separately.
-    
-- **Know when to stop repairing.** I wasted hours trying to nurse a dead disk. Installing on new hardware fixed everything in minutes.
-
-
-What I did wrong (and why it hurt)
-
-What I should have done differently
-
-The single most important file in OPNsense
-
-Why keeping off-box backups matters
+- **Backups matter**: I always found myself thinking backups are not relevant... until you need to restore and it's too late.
+- **Keep backups off the box**: I was lucky to get the `config.xml` before my disk die, I would have a really hard time to fully recover.
+- **Healthcheck after a crash**: Do not ignore a kernel panic.
+- **I/O errors = red flag**: I should have stopped trying to repair. I lost hours fighting a dead disk.
+- **Custom plugin configs aren’t include**d: OPNsense configuration and its official plugin are saved into the backups, this is not the case for the others.
+- **My router is a SPOF** (*single point of failure*): In my homelab, I wanted to have most of my elements highly available, I need to find a better solution.
 
 ## Moving Forward
 
-My new backup strategy
+I really need to think on my backup strategy. I'm too lazy and always keep it for later, until it is too late. It's been a long time since I've been struck by a hardware failure. When it strikes, it hurts.
 
-Plans to improve reliability in my homelab
-
-Final thoughts: sometimes starting fresh is the cleanest fix
+Initially I wanted my router to be in its own hardware because I thought it was safe, I was damn wrong. I will think on a solution to virtualize OPNsense in Proxmox to have it highly available, a great project in perspective!
 
 ## Conclusion
 
-How this failure taught me more than a normal upgrade ever could
+My OPNsense router went from a random reboot to a dead disk, with a rollercoaster of troubleshooting. In the end, I'm happy with the result, it taught me more than any smooth upgrade ever could.
 
-Encouragement for others to prepare before disaster strikes
+If you run OPNsense (or any router), remember this:  
+**Keep a backup off the box.**
+
+Because when things go wrong, and eventually they will, that backup can save your homelab. Thanks to that one little XML file, what could have been a complete disaster ended as just a painful, but very educational, weekend.
+
+Stay safe, make backups.