The Deep Audit

Methodology

The approach was straightforward but thorough: take every factual claim in the infrastructure document and verify it. CPU models, RAM amounts, disk SMART status, BIOS versions, kernel versions, service versions, firewall rules, DHCP scopes, DNS configurations, VLAN assignments, backup jobs, cloud sync tasks, Tailscale nodes, Docker containers, NFS exports, port mappings, and more. If the document said it, we checked it.

This meant SSH sessions into all seven hosts: the router, both Proxmox hypervisors, the TrueNAS storage server, both application VMs, and the VPS. The audit also hit external endpoints to verify public-facing services were responding correctly.

What Matched

The good news: the vast majority of the document was accurate. All six VMs were running. All 13 Docker containers were healthy. The five-VLAN network architecture was correctly configured with proper DHCP scopes and firewall zones. All NFS exports had the host restrictions applied during the January hardening. The Backblaze B2 offsite backup tasks were running on schedule, with four completed and two large initial syncs still in progress. Fail2Ban on the VPS had already banned over 250 malicious IPs since installation. All external URLs were responding with correct content and security headers.

Disk SMART checks passed across all drives. The enterprise SAS drives in the TrueNAS server continued to show clean health. Backup jobs were running, with 39 successful backups on file across both Proxmox hosts.

What Didn't

The audit uncovered several discrepancies, ranging from cosmetic to significant:

• collectd was not running on the router. The monitoring data flow diagram showed router metrics flowing to Grafana, but collectd had actually been crashing silently. The process manager had given up restarting it after repeated failures. This meant the router monitoring dashboards were showing stale data with nobody noticing.
• Storage metrics had drifted significantly. The SSD thin pool documented at 70% was actually at 65%. The Tailscale VM disk documented at 83% was down to 53%. The Home Assistant disk documented at 84% had dropped to 75%. InfluxDB data had more than doubled. None of these were problems, but the document was out of date.
• One VM had no SSH key deployed. The metrics server couldn't be accessed via SSH key authentication because nobody had ever added the authorized key. It had been set up before I standardized SSH access.
• A USB device was misidentified. The document listed an ESP32 connected via USB, but lsusb showed it was actually a Nabu Casa Zigbee coordinator. A different USB device listed as connected wasn't detected at all, likely unplugged at some point.

The collectd Crash Investigation

This was the most interesting fix of the audit. The router's collectd service had been crashing repeatedly, with the process manager logging eight consecutive crashes before giving up entirely. The service was dead, and since nobody was alerting on it, nobody knew.

The root cause turned out to be three problematic plugins. The processes plugin required kernel TASKSTATS support that wasn't compiled into the OpenWrt kernel. The tcpconns plugin was throwing netlink errors. And rrdtool was configured but unnecessary since the data was being shipped to InfluxDB anyway.

The fix was to create a simplified configuration with only the stable, useful plugins: CPU, memory, load, network interfaces, thermal, uptime, connection tracking, DHCP leases, and WiFi statistics. After restarting with the lean config, collectd ran stable. I verified data was flowing end-to-end: router collectd to Telegraf to InfluxDB to Grafana dashboards. The monitoring gap was closed.

This was a perfect example of why you need alerting, not just monitoring. Having beautiful Grafana dashboards doesn't help if you don't notice when they stop updating.

Tailscale Cleanup

The audit revealed two stale phone entries in the Tailscale network from devices I no longer use. These were removed via the Tailscale API, bringing the mesh network from nine nodes down to seven. I also verified the subnet routing configuration: the primary router and backup were correctly designated with no conflicts.

Docker Update Automation

With 13 containers running across five Docker Compose projects, manually checking for updates was tedious and easily forgotten. I deployed an automated update script that runs weekly via a systemd timer. It pulls new images for each compose project, recreates containers if updates are available, and prunes old images to prevent disk bloat. Everything is logged for review after each run.

Nextcloud Sync Troubleshooting

A separate issue surfaced when Obsidian's sync plugin started throwing 403 Forbidden errors against Nextcloud. The root cause was subtle: a new Obsidian plugin had created a local directory that didn't exist on the Nextcloud server, and Nextcloud returned 403 (not 404) when the sync tried to upload files to a nonexistent path. Creating the missing directory and running Nextcloud's file scanner resolved it.

This also revealed that Nextcloud's internal log file hadn't been updating since May 2025. The Apache access logs were the reliable source for troubleshooting, which I documented for next time.

Key Feedback

The reviewers were broadly positive about the VLAN segmentation, the offsite backup strategy, and the overall architecture. But they found real gaps:

• SSH was still accepting passwords. Most hosts still allowed password-based SSH login alongside key authentication. Two reviewers pointed out this left a brute-force attack surface open on every host in the infrastructure.
• No swap on the main application server. The VM running 13 Docker containers had no swap configured. Under heavy load, it would OOM-kill processes with no graceful degradation path.
• Home Assistant sprawl. Two reviewers noted the smart home setup had hundreds of entities but very few active automations, with many sensors showing as unavailable. The kindest description was "a setup that got built out ambitiously and then partially abandoned."

The feedback led to a complete reorganization of my priority list into four tiers, from immediate quick wins to items that could wait for a planned house move.

Closing the Password Door

Prompted by the reviewers' feedback, I conducted a full SSH authentication audit across all seven hosts. The results were worse than expected: five of seven hosts still accepted password-based SSH logins. Three of those also allowed root login with a password. My SSH key was deployed everywhere and working, but the password fallback left every host vulnerable to brute-force attacks.

The fix was systematic. On each Linux host, I disabled password authentication and restricted root login to key-only access. The router required a different approach since it uses Dropbear instead of OpenSSH, but the end result was the same. After each change, I verified I could still connect with my key before moving to the next host. TrueNAS was the only host that was already correctly configured as key-only.

The result: every host in the infrastructure now requires key-based SSH authentication. No passwords accepted anywhere. I also exported a backup of my SSH keys for secure storage in a password manager, since losing that key would now mean console-only access to everything.

Preventing an OOM Crisis

The main application server was running 13 Docker containers on under 8 GB of RAM with zero swap space. Under normal load, it sat at about 85% memory usage. A burst of activity from Immich's machine learning service or Collabora Online could push it over the edge, triggering the OOM killer and potentially corrupting database state.

I added a 4 GB swap file to give the system breathing room. Swap isn't a replacement for adequate RAM, but it's the difference between graceful degradation and sudden process death. I also cleaned up a stale swap partition reference from the original Debian installation that was pointing at a partition that no longer existed.

Nextcloud Upload Limits

Nextcloud's PHP configuration still had default upload limits: 2 MB maximum file size. This made it essentially useless for anything larger than a small document. I raised the limits to 2 GB and increased the memory and timeout settings to handle large file transfers. A simple fix that should have been caught during initial setup.

Tailscale Key Expiry

The backup Pi-hole's Tailscale key was set to expire in two weeks. If it had expired unnoticed, every device on the Tailscale mesh using it as a DNS resolver would have lost DNS resolution. I rotated the key and disabled automatic expiry on both Tailscale nodes to prevent this from becoming a recurring maintenance burden.

Documentation Consolidation

I merged the separate smart home documentation into the main infrastructure document, expanding it from a summary into a comprehensive reference covering all Home Assistant areas, integrations, automations, and known issues. Having everything in one document eliminates the drift that inevitably happens when related information lives in separate files.