Auto-update blog content from Obsidian: 2025-08-22 09:21:52
All checks were successful
Blog Deployment / Check-Rebuild (push) Successful in 6s
Blog Deployment / Build (push) Has been skipped
Blog Deployment / Deploy-Staging (push) Successful in 11s
Blog Deployment / Test-Staging (push) Successful in 2s
Blog Deployment / Merge (push) Successful in 9s
Blog Deployment / Deploy-Production (push) Successful in 10s
Blog Deployment / Test-Production (push) Successful in 3s
Blog Deployment / Clean (push) Has been skipped
Blog Deployment / Notify (push) Successful in 3s
All checks were successful
Blog Deployment / Check-Rebuild (push) Successful in 6s
Blog Deployment / Build (push) Has been skipped
Blog Deployment / Deploy-Staging (push) Successful in 11s
Blog Deployment / Test-Staging (push) Successful in 2s
Blog Deployment / Merge (push) Successful in 9s
Blog Deployment / Deploy-Production (push) Successful in 10s
Blog Deployment / Test-Production (push) Successful in 3s
Blog Deployment / Clean (push) Has been skipped
Blog Deployment / Notify (push) Successful in 3s
This commit is contained in:
92
content/post/opnsense-crash.md
Normal file
92
content/post/opnsense-crash.md
Normal file
@@ -0,0 +1,92 @@
|
|||||||
|
---
|
||||||
|
slug:
|
||||||
|
title: Template
|
||||||
|
description:
|
||||||
|
date:
|
||||||
|
draft: true
|
||||||
|
tags:
|
||||||
|
categories:
|
||||||
|
---
|
||||||
|
## Intro
|
||||||
|
|
||||||
|
Why I’m sharing this story
|
||||||
|
|
||||||
|
A quick disclaimer: not everything went smoothly
|
||||||
|
|
||||||
|
## The Calm Before the Storm
|
||||||
|
|
||||||
|
My OPNsense setup before the incident
|
||||||
|
|
||||||
|
Confidence in my homelab
|
||||||
|
|
||||||
|
## The Unexpected Reboot
|
||||||
|
|
||||||
|
How I first noticed something was wrong
|
||||||
|
|
||||||
|
The kernel panic that started it all
|
||||||
|
|
||||||
|
## First Troubleshooting Attempts
|
||||||
|
|
||||||
|
Caddy refusing to start
|
||||||
|
|
||||||
|
Digging into logs and finding corrupted certs
|
||||||
|
|
||||||
|
Quick fixes to restore services
|
||||||
|
|
||||||
|
## Filesystem Corruption
|
||||||
|
|
||||||
|
Upgrade attempt from 25.1 to 25.7 gone wrong
|
||||||
|
|
||||||
|
pkg errors and a broken package database
|
||||||
|
|
||||||
|
Discovering the dirty filesystem with fsck
|
||||||
|
|
||||||
|
## When Things Got Worse
|
||||||
|
|
||||||
|
Broken bootstrap and missing core components
|
||||||
|
|
||||||
|
Realizing the system was half-upgraded and unstable
|
||||||
|
|
||||||
|
The tough decision: reinstall vs. repair
|
||||||
|
|
||||||
|
## Starting Over the Hard Way
|
||||||
|
|
||||||
|
Pulling the box out of the rack
|
||||||
|
|
||||||
|
Preparing the installer USB
|
||||||
|
|
||||||
|
Fresh install of OPNsense 25.7
|
||||||
|
|
||||||
|
## The Lifesaver: config.xml
|
||||||
|
|
||||||
|
Copying my configuration from backup
|
||||||
|
|
||||||
|
Restoring the system to its former self
|
||||||
|
|
||||||
|
Which services came back instantly
|
||||||
|
|
||||||
|
Which ones didn’t (goodbye UniFi controller backup…)
|
||||||
|
|
||||||
|
## Lessons Learned the Hard Way
|
||||||
|
|
||||||
|
What I did wrong (and why it hurt)
|
||||||
|
|
||||||
|
What I should have done differently
|
||||||
|
|
||||||
|
The single most important file in OPNsense
|
||||||
|
|
||||||
|
Why keeping off-box backups matters
|
||||||
|
|
||||||
|
## Moving Forward
|
||||||
|
|
||||||
|
My new backup strategy
|
||||||
|
|
||||||
|
Plans to improve reliability in my homelab
|
||||||
|
|
||||||
|
Final thoughts: sometimes starting fresh is the cleanest fix
|
||||||
|
|
||||||
|
## Conclusion
|
||||||
|
|
||||||
|
How this failure taught me more than a normal upgrade ever could
|
||||||
|
|
||||||
|
Encouragement for others to prepare before disaster strikes
|
Reference in New Issue
Block a user