{"id":21,"date":"2026-03-11T12:50:58","date_gmt":"2026-03-11T20:50:58","guid":{"rendered":"https:\/\/satchnet.io\/?p=21"},"modified":"2026-03-19T05:36:11","modified_gmt":"2026-03-19T13:36:11","slug":"the-day-i-decided-my-homelab-needed-therapy-a-notification-infrastructure-journey","status":"publish","type":"post","link":"https:\/\/satchnet.io\/?p=21","title":{"rendered":"The Day I Decided My Homelab Needed Therapy (A Notification Infrastructure Journey)"},"content":{"rendered":"\n<p><em>Or: How I Learned to Stop Worrying and Love SMTP<\/em><\/p>\n\n\n\n<!--more-->\n\n\n\n<h2 class=\"wp-block-heading\">The Problem Nobody Asked Me to Solve<\/h2>\n\n\n\n<p>So there I was, traveling for work, when I had a thought that would consume my entire evening: &#8220;My homelab notifications aren&#8217;t <em>elegant<\/em> enough.&#8221;<\/p>\n\n\n\n<p>You know you&#8217;ve gone too deep into homelab territory when you start critiquing the <em>aesthetics<\/em> of your alert delivery system. Most people get a text when their server is on fire. I apparently needed a <em>philosophy<\/em>.<\/p>\n\n\n\n<p>I had Gotify. It worked. Proxmox and TrueNAS were happily shouting at it. But I&#8217;d discovered ntfy, and like a middle-aged man buying a sports car, I decided I needed to replace something that was working perfectly fine with something shinier.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The Planning Phase (Airport WiFi Psychology)<\/h2>\n\n\n\n<p>While my coworkers were watching Netflix on the plane, I was sketching notification architecture diagrams in my notes app. This is what homelabbing does to you. You become the person who finds SMTP routing <em>genuinely exciting<\/em>.<\/p>\n\n\n\n<p>&#8220;I&#8217;ll just add ntfy,&#8221; I thought. &#8220;Quick weekend project.&#8221;<\/p>\n\n\n\n<p>Narrator: <em>It was not a quick weekend project.<\/em><\/p>\n\n\n\n<p>But then Claude (my AI buddy who I probably talk to more than my actual friends) asked the fatal question: &#8220;What about Apprise?&#8221;<\/p>\n\n\n\n<p>Oh no.<\/p>\n\n\n\n<p>Now we weren&#8217;t just replacing Gotify with ntfy. We were building a <em>notification orchestration platform<\/em>. Because apparently, I&#8217;m the kind of person who needs enterprise-grade alerting for a homelab that primarily serves cat photos via Immich.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The &#8220;While I&#8217;m At It&#8221; Cascade<\/h2>\n\n\n\n<p>You know that thing where you go to fix a light bulb and end up rewiring the entire house? Yeah.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>&#8220;I&#8217;ll deploy ntfy&#8221; became &#8220;I&#8217;ll deploy ntfy + Apprise + Mailrise&#8221;<\/li>\n\n\n\n<li>Which became &#8220;In a new dedicated LXC&#8221;<\/li>\n\n\n\n<li>Which became &#8220;With proper Docker Compose&#8221;<\/li>\n\n\n\n<li>Which became &#8220;And expose it securely to the internet&#8221;<\/li>\n\n\n\n<li>Which became &#8220;And migrate all my services&#8221;<\/li>\n\n\n\n<li>Which became <em>checks watch<\/em> &#8220;&#8230;and there goes my entire evening&#8221;<\/li>\n<\/ul>\n\n\n\n<p>The mission creep was real, folks.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The Great Memory Incident<\/h2>\n\n\n\n<p>So I spun up the new Debian 13 LXC, gave it a whole 2GB of RAM (luxury!), and fired up the stack.<\/p>\n\n\n\n<p>Everything started up beautifully. Containers running, logs looking good, and then&#8230;<\/p>\n\n\n\n<p><em>::LXC immediately maxes out CPU and RAM and becomes completely unresponsive::<\/em><\/p>\n\n\n\n<p>Me, frantically checking Proxmox stats: &#8220;98.19% CPU, 99.97% RAM used&#8230; WHAT THE HELL IS APPRISE DOING?!&#8221;<\/p>\n\n\n\n<p>For a brief, terrifying moment, I thought Apprise was mining Bitcoin or trying to achieve sentience. The LXC was so locked up I couldn&#8217;t even SSH in.<\/p>\n\n\n\n<p>From the Proxmox host, I tried the graceful approach:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>pct stop 125 --force\n<\/code><\/pre>\n\n\n\n<p>Proxmox: &#8220;Unknown option: force&#8221;<\/p>\n\n\n\n<p>Right. Because when would you ever need to <em>force<\/em> stop a container? \/s<\/p>\n\n\n\n<p>Eventually got it killed, doubled the resources (because throwing RAM at problems is the homelab way), and discovered&#8230; Apprise was using 3.3GB of memory on startup.<\/p>\n\n\n\n<p><strong>3.3 GIGABYTES. FOR A NOTIFICATION ROUTER.<\/strong><\/p>\n\n\n\n<p>Turns out it was some weird startup anomaly. Restarted it, and Apprise settled down to a totally reasonable 2MB. But for a hot minute there, I thought my notification system was preparing to launch Skynet.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The SMTP Detective Story<\/h2>\n\n\n\n<p>With the stack finally stable, I started testing. Sent a notification via Apprise to ntfy and&#8230; nothing.<\/p>\n\n\n\n<p>Logs showed Apprise received it. Logs showed it sent to ntfy. But ntfy? Crickets.<\/p>\n\n\n\n<p>Ten minutes of head-scratching later, I realized: I was testing against the internal IP but viewing the external domain in my browser. The notifications were going to the <em>internal<\/em> ntfy, but I was watching the <em>external<\/em> ntfy through NPM.<\/p>\n\n\n\n<p><em>Facepalm.<\/em><\/p>\n\n\n\n<p>Fixed the NPM config, refreshed the browser, and BOOM &#8211; there were all my test notifications, patiently waiting like dogs at the door.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The Mailrise Revelation<\/h2>\n\n\n\n<p>Then came Mailrise testing. This was supposed to be the easy part. SMTP goes in, notifications come out. You can&#8217;t explain that.<\/p>\n\n\n\n<p>Except Mailrise kept rejecting everything:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ERROR: Recipient is not configured: root@localhost\nERROR: Recipient is not configured: critical@mydomain.com\nERROR: Recipient is not configured: literally-anything@anywhere.com\n<\/code><\/pre>\n\n\n\n<p>I had configured routing! I had a <code>default<\/code> catch-all! What more did it want?!<\/p>\n\n\n\n<p>Turns out Mailrise&#8217;s &#8220;default&#8221; isn&#8217;t actually a catch-all. It&#8217;s more like a &#8220;default suggestion that we&#8217;ll completely ignore.&#8221;<\/p>\n\n\n\n<p>The fix? Just&#8230; explicitly configure every recipient email address. No wildcards, no clever routing, just good old-fashioned manual labor.<\/p>\n\n\n\n<p>Once I figured that out, the beautiful moment arrived:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>swaks --to notifications@mydomain.com --body \"Test from Mailrise SMTP\"\n<\/code><\/pre>\n\n\n\n<p>And on my phone, 3,000 miles away from my homelab:<\/p>\n\n\n\n<p>\ud83d\udcf1 <em>ding<\/em> &#8220;Test from Mailrise SMTP&#8221;<\/p>\n\n\n\n<p><em>Chef&#8217;s kiss.<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The TrueNAS Plot Twist<\/h2>\n\n\n\n<p>TrueNAS Scale has moved the email settings in Electric Eel (version 25.10), and apparently nobody at iXsystems thought to tell anyone where.<\/p>\n\n\n\n<p>&#8220;Just go to System \u2192 Email,&#8221; they said.<\/p>\n\n\n\n<p>There is no System \u2192 Email.<\/p>\n\n\n\n<p>&#8220;Try Alert Settings!&#8221;<\/p>\n\n\n\n<p>No email config there.<\/p>\n\n\n\n<p>&#8220;General Settings?&#8221;<\/p>\n\n\n\n<p>Getting warmer&#8230;<\/p>\n\n\n\n<p>Turns out it&#8217;s hidden in a &#8220;Email Options&#8221; dialog that you access by clicking a button that&#8217;s in a place I would <em>never<\/em> look first. It&#8217;s like iXsystems hired the same UX designer who did the Windows Control Panel.<\/p>\n\n\n\n<p>But I found it, configured it, sent a test and&#8230;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ERROR: Recipient is not configured: myemail@gmail.com\n<\/code><\/pre>\n\n\n\n<p>TrueNAS was sending test emails to MY personal email, not to the configured recipient. Because of course it was.<\/p>\n\n\n\n<p>Had to override the recipient in Alert Settings, because TrueNAS apparently believes in a separation of church and state when it comes to email configuration.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The Mobile Moment<\/h2>\n\n\n\n<p>After hours of debugging, configuring, and questioning my life choices, I installed the ntfy app on my phone.<\/p>\n\n\n\n<p>Added my server: <code>https:\/\/ntfy.mydomain.com<\/code> Entered my credentials. Subscribed to topics.<\/p>\n\n\n\n<p>And there they were. All my test notifications. From hours of debugging. Staring at me like a digital graveyard of my evening.<\/p>\n\n\n\n<p>But then I sent a fresh test from Proxmox and&#8230;<\/p>\n\n\n\n<p><em>::Phone buzzes in my hand::<\/em><\/p>\n\n\n\n<p><strong>IT WORKED.<\/strong><\/p>\n\n\n\n<p>I was in a hotel room, 3,000 miles from my homelab, on cellular data, and my Proxmox cluster was notifying me in real-time.<\/p>\n\n\n\n<p>I may have done a small victory dance. Don&#8217;t judge me.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The Scoreboard<\/h2>\n\n\n\n<p><strong>What I started with:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Gotify (working perfectly fine)<\/li>\n\n\n\n<li>Two services configured (Proxmox, TrueNAS)<\/li>\n\n\n\n<li>Local-only notifications<\/li>\n<\/ul>\n\n\n\n<p><strong>What I ended with:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A dedicated Debian 13 LXC<\/li>\n\n\n\n<li>Three containerized services (ntfy, Apprise, Mailrise)<\/li>\n\n\n\n<li>Nginx Proxy Manager reverse proxy with SSL<\/li>\n\n\n\n<li>Websocket support (because real-time is <em>important<\/em>)<\/li>\n\n\n\n<li>Proxmox VE notifications \u2705<\/li>\n\n\n\n<li>Proxmox Backup Server notifications \u2705<\/li>\n\n\n\n<li>TrueNAS Scale notifications \u2705<\/li>\n\n\n\n<li>Mobile app configured and working externally \u2705<\/li>\n\n\n\n<li>Gotify still running (because I&#8217;m not <em>crazy<\/em>)<\/li>\n\n\n\n<li>A notification architecture that could serve a small enterprise<\/li>\n\n\n\n<li>An evening I&#8217;ll never get back<\/li>\n\n\n\n<li>Zero regrets<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Lessons Learned<\/h2>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>&#8220;While I&#8217;m at it&#8221; is a dangerous phrase<\/strong> in homelab contexts<\/li>\n\n\n\n<li><strong>Apprise briefly becoming a RAM monster<\/strong> is apparently normal<\/li>\n\n\n\n<li><strong>Mailrise&#8217;s &#8220;default&#8221; is a lie<\/strong><\/li>\n\n\n\n<li><strong>TrueNAS hides email settings<\/strong> like they&#8217;re state secrets<\/li>\n\n\n\n<li><strong>Docker Compose V2 will yell at you<\/strong> about the version attribute<\/li>\n\n\n\n<li><strong>NPM websockets are critical<\/strong> and easy to forget<\/li>\n\n\n\n<li><strong>The satisfaction of seeing your first external notification<\/strong> is worth every frustrating minute<\/li>\n<\/ol>\n\n\n\n<h2 class=\"wp-block-heading\">The Real Victory<\/h2>\n\n\n\n<p>You know what the best part is? Now when my TrueNAS has a disk failure at 3 AM, I&#8217;ll get a beautifully routed, SSL-encrypted, authenticated notification delivered via SMTP \u2192 Mailrise \u2192 ntfy \u2192 my phone.<\/p>\n\n\n\n<p>Is this over-engineered? Absolutely.<\/p>\n\n\n\n<p>Did I need to spend an entire evening on this? Definitely not.<\/p>\n\n\n\n<p>Would I do it again?<\/p>\n\n\n\n<p><em>::Looks at the 20+ other services that still use Gotify::<\/em><\/p>\n\n\n\n<p>&#8230;ask me after I&#8217;ve had some bourbon.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Epilogue: The Gotify Question<\/h2>\n\n\n\n<p>I still have Gotify running. Both systems are sending notifications in parallel now, like some kind of redundant notification democracy.<\/p>\n\n\n\n<p>Eventually, I&#8217;ll decommission Gotify. Eventually.<\/p>\n\n\n\n<p>But today is not that day.<\/p>\n\n\n\n<p>Because homelabbers don&#8217;t <em>remove<\/em> old infrastructure. We just add new layers on top and call it &#8220;high availability.&#8221;<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p><strong>Stack deployed:<\/strong> ntfy + Apprise + Mailrise<br><strong>Services migrated:<\/strong> 3<br><strong>Hours invested:<\/strong> Let&#8217;s not talk about it<br><strong>Services still to migrate:<\/strong> ~22<br><strong>Will I actually migrate them all:<\/strong> Ask me in 6 months<br><strong>Was it worth it:<\/strong> <em>::notification buzzes on phone::<\/em> &#8230;yeah, it was worth it.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p><em>If you enjoyed this journey into notification infrastructure madness, check out the others where I document other homelab adventures, like <a href=\"https:\/\/satchnet.io\/?p=6\">&#8220;The Great 3 AM Network Mystery&#8221;<\/a> where I spent three weeks debugging a problem I created six months earlier while drunk.<\/em><\/p>\n\n\n\n<p><em>Homelab: Because sometimes the journey is more important than the destination. Or the budget. Or your free time. Or your sanity.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Or: How I Learned to Stop Worrying and Love SMTP<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[],"class_list":["post-21","post","type-post","status-publish","format-standard","hentry","category-monitoring"],"_links":{"self":[{"href":"https:\/\/satchnet.io\/index.php?rest_route=\/wp\/v2\/posts\/21","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/satchnet.io\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/satchnet.io\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/satchnet.io\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/satchnet.io\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=21"}],"version-history":[{"count":4,"href":"https:\/\/satchnet.io\/index.php?rest_route=\/wp\/v2\/posts\/21\/revisions"}],"predecessor-version":[{"id":39,"href":"https:\/\/satchnet.io\/index.php?rest_route=\/wp\/v2\/posts\/21\/revisions\/39"}],"wp:attachment":[{"href":"https:\/\/satchnet.io\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=21"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/satchnet.io\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=21"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/satchnet.io\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=21"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}