I tend to find myself obsessing over specific topics and projects. Hell bent on my goal.

This time the goal was quieter. File Orbit identity runs on AT Proto. Sign up and recovery still need email. The mailbox went quiet for the wrong reason. SMTP was timing out. Tranquil was logging Failed to send comms. And Observatory stayed green.

The Identity health check was not lying about what it knew. The HTTP surface was up. It just could not see a send path that only fails inside a worker. Green dots feel great until they become a false sense of security.

That gap pushed me to treat observability less like dashboards you open when something hurts, and more like tests that keep running after ship.

TLDR

I wired log failures into the same place operators already trust for uptime. Grafana finds the bad (and good) log lines. A tiny probe bridge turns that into JSON Gatus can poll. Gatus owns the status dots and Telegram pages. One pager. No second alert carnival from Grafana.

The first signal is Identity Email. Down on send failure. Up only on a real send success. Idle after failure stays red on purpose.

Observability as TDD (production is the suite)

In TDD you write the failing assertion first, then the code that makes it pass. For ops I wanted the same shape.

  1. 1.

    Name the failure in logs with a fingerprint we refuse to "fix" by renaming quietly (Failed to send comms / Comms sent successfully).

  2. 2.

    Assert on that fingerprint with LogQL in Grafana Alerting, provisioned in git, not click ops.

  3. 3.

    Close the loop where we already look. Status dots and Telegram.

The "test" is not a Vitest case. It is a continuous claim. If identity email cannot send, Identity Email is red and someone gets paged. If that claim is false, the system is lying the same way an untested module lies.

Recovery is part of the assertion too. Emptying a failure window must not flip email back to green. Quiet after failure can mean SMTP is still dead and nobody tried. We only mark up when we see a real success line. That forces honest debug logging on the send path so success can reach Loki (our log aggregator). Another production invariant, not a wish.

Three planes, one pager

File Orbit already had two planes.

Log event stream. stdout to journald (server logs) → Vector (logs drain off each box) → Loki (label indexed log store) → Grafana (our neat dashboard, and alerts). Staff only behind admin forward auth.

Synthetic uptime. Gatus YAML in git to public Observatory (FileOrbits status page) and Telegram.

The missing plane was log derived signals. Failures that never show up as "port closed." Grafana is good at LogQL. Gatus is good at health and paging. Neither speaks the other's language.

So I added a small portable probe bridge (grafana-gatus-bridge). Grafana POSTs authenticated up or down. Gatus GETs JSON. File Orbit keeps those rows in managed Postgres next to incident narratives, behind an operations API, with status Caddy proxying /probes/*. Writes need a Bearer secret and a Postgres RLS session role the webhook assumes. Defense in depth for a public write surface which will help me sleep at night if someone gets access to the alert system they can't rick roll the whole database.

Grafana does not page Telegram. One pager, and public facing status view. Gatus owns the noise budget. Incident reports stays entered by me.

For my fellow nerds...sorry normies

Tech stack

Tranquil PDS / SMTP
Where send succeeds or times out

Vector → Loki → Grafana
Streams plus LogQL evaluation

grafana-gatus-bridge
The sticky note between alerts and uptime. Store agnostic. Memory for the binary. Postgres adapter for File Orbit.

Postgres status_probe_signal
Durable probe state next to status_incident

Gatus + Observatory + Telegram
Dots, history, pages

I wanted this to be more than "fix SMTP and move on." Portable seam. Git as source of truth for rules and endpoints. Sticky fail and recover semantics. Audience honesty. Observatory is ops and tech friendly. Bridged signals live there with the synthetics.

Fixing the mailbox is necessary. Teaching the product to notice the next silent failure is the long term return.

Until next time, peace nerds and normies!

~ Triforce