monitoring

2026-06-20

Monitoring without 3rd parties / cloud

Scenario: you have 2 independent homelabs. Seperate URLs, internet connection, hardware...

You want to be informed if one goes down, without using any external services.

Naive plan:

Each server has a script to occasionally "Ping" the other. if the ping fails, send out a notification to it's owner's cell phone.

Problems:

First of all: how do you send notifications to a phone? usually using a third party.

We solve this with a self-hosted push service, such as ntfy.sh. this one comes with a handy app (fully open source too).

Second: if one of the scripts fails but the pings still succeed, we have no way of knowing that - our monitoring could silently go down.

Next plan: Heartbeat

To ensure we know if one of the scripts stops monitoring, each script will have to "write" some signal onto the other server confirming it's been active recently. then the script on the other end can read that signal, and if it's been too long, alert that something's wrong.

For sending these signals, i considered a few options (ssh, a small http api) but to avoid running extra services, we can use ntfy.sh itself.

We set up 2 "topics" on each ntfy.sh server: One for actual alerts if something's wrong (this one we subscribe to on the phone) and one for the heartbeat. Each script will post something to the other server's heartbeat topic, and each script will check it's own server for a recent message to know the other script is still working.

Script pseudocode

# first send our heartbeat message
send_heartbeat(otherserver)

# then check health of other server
query_status(otherserver)

# query our server if heartbeat arrived recently
if query_last_heartbeat_time (localserver) > max_time_delay:
    send_alert(mobilephone, "heartbeat gone")

# if ANY steps fail, alert:
if <error>:
    send_alert(mobilephone)