Skip to content

Self-hosting SilenceWatch

{/* Generated from docs/self-hosting.md by scripts/prepare.mjs. Edit that file. */}

Everything here works on one small VPS. The whole system is a Node process and a PostgreSQL database; there is nothing else to install, scale or watch.

  • Docker with Compose v2
  • 1 vCPU and 1 GB of RAM handles thousands of checks comfortably
  • PostgreSQL 14 or later (16 recommended). The Compose file brings its own
Terminal window
git clone https://github.com/liliang-dev/SilenceWatch.git
cd SilenceWatch
cp .env.example .env

Fill in at least:

Terminal window
SECRET_KEY=$(openssl rand -hex 32) # all signing keys derive from this
POSTGRES_PASSWORD=$(openssl rand -hex 24)
BASE_URL=https://watch.example.com # what your users actually type
ADMIN_EMAIL=you@example.com # your address: the administrator account

Then:

Terminal window
docker compose up -d

Then create your account with the one-time link the server prints in its log (docker compose logs silencewatch): it is the only way to create the account of ADMIN_EMAIL, works once and expires after 30 minutes, and a new one is printed at every start until the account exists. That account is the administrator’s, the only one that sees the instance’s statistics (Settings › Statistics).

People sign in with a passkey by default, which browsers only accept on https:// or localhost: on a plain-http:// address set AUTH_MODE=password, or the server refuses to start and says why. Anyone else may then sign up while SIGNUP_ENABLED=true; once your team is in, set SIGNUP_ENABLED=false and restart, and no account but the administrator’s can be created any more. authentication.md has the sign-in methods, the administrator account and what to do when someone is locked out.

Every setting is an environment variable. The server validates them at startup and refuses to boot on an unsafe or incoherent configuration rather than running in a degraded state you would discover during an incident.

Four values. With any of them missing the instance does not start, and says which one:

Variable Set it to
SECRET_KEY openssl rand -hex 32. Session cookies are signed with it, so rotating it signs everyone out, and the notification channels’ webhook URLs and secrets are encrypted under it: after rotating, run OLD_SECRET_KEY=<previous> silencewatch reseal (see Authentication). Keep a copy of it with your database backups: a backup is unreadable for the channels without it
POSTGRES_PASSWORD openssl rand -hex 24. Compose builds DATABASE_URL from it
SMTP_URL smtp://user:password@relay.example.com:587. Or a POSTMARK_TOKEN / BREVO_API_KEY with the matching EMAIL_PROVIDER
ADMIN_EMAIL Your own address. It names the administrator, whose account sees the instance’s statistics and can only be created with the one-time link printed in the server log

SMTP_URL is in that list because docker-compose.yml defaults EMAIL_PROVIDER to smtp, and a mail provider with no transport is a configuration that would accept alerts and drop them. Nothing else on this page has to be set.

One more you should set anyway, whose default is only right on a laptop:

Variable Default Why it matters
BASE_URL http://localhost:8080 Ping URLs and alert links are built from it, so a wrong value hands out links nobody can follow

Which makes the smallest genuinely usable .env four lines:

Terminal window
SECRET_KEY=…
POSTGRES_PASSWORD=…
SMTP_URL=smtp://user:password@relay.example.com:587
BASE_URL=https://watch.example.com

DATABASE_URL is deliberately not in that list. The server does require it, but docker-compose.yml and docker-stack.yml both assemble it from POSTGRES_PASSWORD — you only set it yourself when pointing at a PostgreSQL the Compose file did not start.

EMAIL_PROVIDER=console is not usable in production. It prints alerts to the log instead of sending them, and the published image sets NODE_ENV=production, where the server refuses to boot with it.

Variable Default Meaning
EMAIL_PROVIDER console console, smtp, postmark, brevo
SMTP_URL — smtp://user:pass@host:587 (STARTTLS required) or smtps://…:465
POSTMARK_TOKEN / BREVO_API_KEY — API token for the matching provider
EMAIL_FROM, EMAIL_FROM_NAME — Sender identity
NOTIFICATION_MAX_ATTEMPTS 6 Retries before a delivery is abandoned
NOTIFICATION_TIMEOUT_MS 10000 Timeout for each outbound alert
ALLOW_PRIVATE_NOTIFICATION_TARGETS false Allow alerts to reach private addresses
ALERT_MAX_PER_CHANNEL_PER_HOUR 0 Alerts one channel may deliver per hour; 0 is no limit
TEST_ALERT_COOLDOWN_SECONDS 60 Gap between two test alerts from one channel; 0 disables it
TEST_ALERT_MAX_PER_HOUR 20 Test alerts one account may send per hour; 0 disables it

A test alert (the button on a channel) sends a real message to whatever address or URL the channel points at, so it is limited: one every TEST_ALERT_COOLDOWN_SECONDS per channel, and TEST_ALERT_MAX_PER_HOUR per account. Both answer 429 with the time to wait. ALERT_MAX_PER_CHANNEL_PER_HOUR bounds the real ones: past it an alert is not sent and is marked as refused (with the reason) rather than retried, which keeps a check that keeps flapping, or a channel pointed at someone else’s mailbox, from sending without end. Leave it at 0 unless you run SilenceWatch for other people.

console prints alerts to the log instead of sending them, which is useful in development and unacceptable in production — the server refuses to start with it when NODE_ENV=production.

Do not self-host SMTP for alerts. Deliverability is a reputation game you have no reason to play; the day an alert matters is the day you do not want it in a spam folder. Relay through a provider or through a relay you already trust.

How people sign in is AUTH_MODE (passkey, password or full), with its own settings: all of them are on authentication.md, together with what the server checks at startup.

Variable Default Meaning
SIGNUP_ENABLED true Whether anyone may create an account. When false, the administrator’s is the only one that can be
SIGNUP_BLOCK_DISPOSABLE_EMAIL false Reject known throwaway mailbox domains
SIGNUP_BLOCKED_EMAIL_DOMAINS — Extra domains to reject, comma-separated
SIGNUP_MAX_PER_NETWORK_PER_HOUR 0 Accounts per hour per network prefix; 0 disables it
SIGNUP_MAX_ACCOUNTS_PER_ADDRESS 0 Accounts that may exist from one connection; 0 disables it

The last four rows are off by default and stay that way for most self-hosters. If your instance is on a private network, or you set SIGNUP_ENABLED=false once the team is in, you have already solved the problem these settings address — none of them is worth turning on.

They exist for instances whose sign-up form is reachable by strangers. abuse-prevention.md explains what each one actually buys, with measurements, including the ones that are not worth what people expect.

A self-hosted SilenceWatch has no limits, and nothing here needs setting. QUOTAS_ENABLED is off, user.plan stays null, and null means unlimited on every axis. This section exists because the hosted deployment runs the same code; it is documented so you can see that it does, and so you can use it if you run SilenceWatch for other people.

Variable Default Meaning
QUOTAS_ENABLED false Master switch. Off means unlimited, always
DEFAULT_PLAN free Plan given to a new account when quotas are on
PLAN_LIMITS {} JSON: what each plan name is allowed
QUOTA_RECONCILE_INTERVAL_MS 300000 How often accounts are matched against their plan
Terminal window
QUOTAS_ENABLED=true
PLAN_LIMITS='{
"free": {"checks": 10, "projects": 3, "channelsPerProject": 3, "retentionDays": 7},
"supporter": {"checks": 10, "projects": 3, "channelsPerProject": 3, "retentionDays": 30},
"pro": {"checks": 100, "projects": 20, "retentionDays": 90},
"max": {"checks": 1000}
}'

An omitted key is unlimited: max above has no project ceiling and no retention cap. An unknown plan name is also unlimited — a typo in this JSON, or a plan renamed on the billing side, should briefly give someone too much rather than lock a paying customer out of their own monitoring.

Which plan an account is on is the plan column on user. Something has to write it: by hand, from your own billing, or with the Stripe integration below. The reconciler picks the change up within QUOTA_RECONCILE_INTERVAL_MS.

Checks are counted across every project the account owns, so a second project does not reset the allowance. On a downgrade the excess checks are paused — newest first, deliberately paused checks untouched, and the account emailed the list — and they resume on their own when the account moves back under its limit.

Also off, also invisible, and not needed to self-host. With BILLING_ENABLED unset no route answers, the Settings page has no subscription tab and Stripe is never contacted. It exists for the hosted service, and is documented for the same reason as the quotas above: it is the same code.

When on, an account chooses a plan by paying for it on Stripe’s own page. Stripe holds the card, the invoices and the prices; this side keeps only which Stripe customer an account is, and what its subscription last said. It needs quotas, because a subscription does nothing but pick a plan.

Variable Default Meaning
BILLING_ENABLED false Master switch. Needs QUOTAS_ENABLED
STRIPE_SECRET_KEY — A secret or restricted key (sk_…, rk_…)
STRIPE_WEBHOOK_SECRET — Signing secret of the webhook endpoint (whsec_…)
STRIPE_PLANS {} JSON: for each paid plan, its Stripe price and the amount shown
BILLING_CURRENCY eur Currency of the prices, lower case
BILLING_AUTOMATIC_TAX false Let Stripe Tax compute the tax on each payment
BILLING_TERMS_URL Address of your terms of sale. When set, Stripe’s payment page asks the buyer to accept them before paying (Stripe also needs the address in the Dashboard, under Settings, Public details)
BILLING_TERMS_URL_FR The French page, used when the buyer’s language is French; without it BILLING_TERMS_URL serves everyone
BILLING_PROFESSIONAL_ONLY false The same box also states that the buyer acts for their business. Needs BILLING_TERMS_URL
STRIPE_API_URL https://api.stripe.com Only worth changing to point a test at a stand-in
Terminal window
BILLING_ENABLED=true
STRIPE_SECRET_KEY=sk_live_…
STRIPE_WEBHOOK_SECRET=whsec_…
STRIPE_PLANS='{
"pro": {"price": "price_…", "amount": 499},
"business": {"price": "price_…", "amount": 1999}
}'

An account with no plan is unlimited, and so is every account created before quotas were turned on. To put them on the default plan, once your users have been told: UPDATE "user" SET plan = 'free' WHERE plan IS NULL;. Take that seriously: the reconciler then pauses the checks above the plan’s limit within minutes, and the purge deletes history older than its retention.

Every plan named in STRIPE_PLANS has to exist in PLAN_LIMITS, and none can be DEFAULT_PLAN: that is the plan an account falls back to. The amount is only displayed; what a customer pays is the Stripe price. The server refuses to boot if any of this is missing or inconsistent.

In Stripe, create one product with a monthly price per paid plan, set up the customer portal (payment method, invoices, plan changes between those prices, cancellation), and add a webhook endpoint at https://<your host>/api/v1/billing/webhook for the events customer.subscription.created, customer.subscription.updated, customer.subscription.deleted, customer.subscription.paused and customer.subscription.resumed. Its signing secret is STRIPE_WEBHOOK_SECRET. An endpoint set up with the first three keeps working: the last two only matter for a subscription that Stripe pauses (a trial that ended with no payment method), and without them such an account would keep its plan until the next change.

How an event becomes a plan: active and trialing give the plan of the price; past_due keeps it, since Stripe retries the payment before giving up; canceled unpaid and paused send the account back to DEFAULT_PLAN (and resumed gives the plan back), and the reconciler pauses what no longer fits. A price that is not in STRIPE_PLANS changes nothing. Events are applied once whatever the number of deliveries, and an older one never undoes a newer one. Deleting an account first ends its subscription, and is refused if Stripe cannot be reached, so that no card is charged for an account that is gone.

Variable Default Meaning
AUDIT_RETENTION_DAYS 365 How long the audit trail is kept

The audit trail records sign-ins and failed sign-ins, password changes, passkeys and recovery codes, API key and alert channel changes, check deletions, ping-key rotations and quota pauses. It is readable in Settings → Security activity by project admins, and never contains a token, a key or a channel’s configuration.

A leaked ping URL no longer means recreating the check: Check → ⋯ → Rotate ping URL issues a new one and keeps the history. Every job still calling the old URL will be reported as down, which is the point — update them first.

Variable Default Meaning
DETECTION_INTERVAL_MS 10000 How often the detection loop runs
DETECTION_BATCH_SIZE 200 Checks claimed per batch
PING_RETENTION_DAYS 90 Ping history kept (per-project override available)
PURGE_CRON 17 3 * * * When the purge runs, in UTC
PING_RATE_LIMIT_PER_MINUTE 120 Per ping key, so a looping job cannot saturate the database
PING_BODY_MAX_BYTES 10000 Ping bodies are truncated to this

Set TRUST_PROXY only when a proxy really is in front, and set it to the proxy’s address or CIDR rather than true:

Terminal window
TRUST_PROXY=10.0.0.0/8

A bare true trusts the hop count from anything that can reach the server, so anyone bypassing the proxy can claim any client address they like — and every per-address control (rate limits, sign-up velocity, the audit trail) is only as good as that value. Production refuses to boot on TRUST_PROXY=true for exactly this reason.

Getting it wrong in either direction is otherwise silent, so the server samples its first requests and says so in the log: trusting a proxy whose header never arrives, or refusing to trust one whose header always does. The second is the quieter failure — it throttles the entire internet as if it were one visitor.

A minimal nginx front:

location / {
proxy_pass http://127.0.0.1:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}

docker compose up -d serves plain HTTP on PORT and nothing else. That is deliberate: plenty of instances sit behind an nginx, a Traefik or a company load balancer that already terminates TLS, and a certificate this side of it would be one more thing to renew for no gain.

If nothing is in front, Compose can bring its own:

Terminal window
docker compose --profile tls up -d

That adds Caddy, which obtains a certificate and renews it without being asked. The profile is why the plain path is unaffected — without --profile tls the service does not exist, and docker compose up -d starts exactly what it always did.

It needs two things that are not Docker’s to give: SILENCEWATCH_DOMAIN must resolve to this machine, and ports 80 and 443 must be reachable from the internet. Both are how Caddy proves the name is yours; a firewall or a DNS record still pointing at a parking page fails certificate issuance, not the container.

Four settings move together, and three of the four are easy to forget:

Terminal window
SILENCEWATCH_DOMAIN=status.example.com # the name on the certificate
BASE_URL=https://status.example.com # or every ping URL you hand out is wrong
BIND_ADDRESS=127.0.0.1 # stop publishing 8080 to the world
TRUST_PROXY=172.28.0.0/16 # the Compose network Caddy speaks from

BIND_ADDRESS matters more than it looks. Left at the default the application’s own port stays open beside the certificate — a plain-HTTP way around it, on which requests arrive with no X-Forwarded-For at all.

TRUST_PROXY takes the network rather than true, for the reason above: in production a bare true is refused. The Compose network is given a fixed subnet precisely so it can be named here; change it with DOCKER_SUBNET if 172.28.0.0/16 collides with something on your host.

Under Swarm there is no profile — Caddy is part of docker-stack.yml and always deployed, since a swarm reached over the internet wants TLS anyway. The subnet variable is SWARM_SUBNET, defaulting to 10.20.0.0/16.

A second name for a static site (the hosted service’s layout)

Section titled “A second name for a static site (the hosted service’s layout)”

The hosted service runs the application on app.silencewatch.com and serves its showcase site and documentation (site/) from the bare domain. The stack does the same when it is given a second name, and does nothing about it when it is not:

Terminal window
SILENCEWATCH_DOMAIN=app.example.com # the application — as above
SILENCEWATCH_SITE_DOMAIN=example.com # optional: a static site on this name
BASE_URL=https://app.example.com # still the application's address

With SILENCEWATCH_SITE_DOMAIN set, Caddy also serves the site volume on that name, redirects www. to it, and sends the addresses the application used to answer there (/p/…, /api/…, /login and the rest) to the application with a 308, so a bookmark, a verification email sent before the move or an old crontab still lands. Unset, none of this exists: no second block, no redirect, and the volume is never read. Both names must resolve to the swarm (add www. too), with 80 and 443 reachable.

The volume is filled by .github/workflows/site.yml on every change to site/ or docs/ that reaches main: each build is unpacked beside the others and a symlink, live, is moved to it in one rename, so a visitor never sees half a site. The last three are kept; going back is ln -sfn <older> live inside the volume. That job connects to the same host as the release deploy, so it needs the swarm to be a single node, or the SSH host to be the node Caddy is pinned to — the volume is local to it, like Caddy’s certificates.

BIND_ADDRESS has no Swarm equivalent: the routing mesh publishes a port on every node and cannot be told to bind one address. The application’s port is still published there, because the deployment check reads /health through it, on 127.0.0.1.

For information, the ports that have to be open to the internet:

Port For
22/tcp (or your SSH port) SSH: administration, and the release and site deploys
80/tcp HTTP: Caddy’s certificate challenge, and the redirect to HTTPS
443/tcp HTTPS: the application and the site

Nothing else needs to be reachable from outside: the application’s port (8080) is only read from the server itself.

Two things on the host get in the way. Check both before the first deploy:

  • Nothing else may listen on 80 or 443. Caddy publishes them on the node itself, so a web server that came with the machine’s image (nginx, Apache) stops it from starting: failed to bind host port 0.0.0.0:80/tcp: address already in use. Find it, then stop it if nothing of yours depends on it:

    Terminal window
    sudo ss -ltnp | grep -E ':(80|443) ' # should print nothing
    sudo systemctl disable --now nginx # or apache2, whichever it is
  • Do not turn on UFW on a Swarm node. It does not filter the ports Docker publishes, which bypass it, so it protects less than it seems to. Turning it on also rewrites the host’s firewall under Docker. Filter at your provider’s network firewall instead, with the ports above. If the containers stop reaching each other after a change to the host’s firewall (the application logging Can't reach database server at db:5432 while the database is up), sudo systemctl restart docker puts Docker’s own rules back.

SilenceWatch cannot monitor itself. A monitoring server that dies quietly is the one outage there is no recovering from: everything looks green because nothing is looking.

Set OUTBOUND_HEARTBEAT_URL to a dead man’s switch owned by someone else — Healthchecks.io, Cronitor, or a colleague’s SilenceWatch:

Terminal window
OUTBOUND_HEARTBEAT_URL=https://hc-ping.com/<your-uuid>

The server pings it every minute, but only while the detection loop has completed a pass recently. If detection stalls while HTTP keeps answering — the failure mode a naive uptime check misses entirely — the heartbeat stops and the external watchdog fires.

/health follows the same rule: it returns 503 when the detection loop has stalled, so an orchestrator restarts a process that is technically alive and practically useless.

Everything lives in PostgreSQL. Nothing is stored on disk by the application.

Terminal window
docker compose exec db pg_dump -U silencewatch silencewatch | gzip > silencewatch-$(date +%F).sql.gz

Restore:

Terminal window
gunzip -c silencewatch-2026-01-15.sql.gz | docker compose exec -T db psql -U silencewatch silencewatch

And — this being what the product is for — monitor the backup job with a check.

Edit the image tag in docker-compose.yml to the release you want, then:

Terminal window
docker compose pull
docker compose up -d

The tag is pinned rather than latest on purpose: a restart should not change which version is running, and a bug report needs a version to name. Published images are listed under Releases.

Working from a checkout instead, with the build stanza uncommented:

Terminal window
git pull
docker compose build --pull
docker compose up -d

Migrations are applied at startup. They are additive by design; when a release needs a destructive change, the notes say so and give the steps.

Docker Swarm, and upgrading without downtime

Section titled “Docker Swarm, and upgrading without downtime”

Compose restarts the container, so an upgrade is a short outage. Swarm can start the new version, wait for it to report healthy, and only then stop the old one. docker-stack.yml in the repository root is that deployment.

It is not docker-compose.yml with a deploy: block added. docker stack deploy silently ignores depends_on, restart, and the short form of tmpfs, so the two files differ where it matters — a read-only container whose /tmp mount was dropped does not start at all.

Two replicas, order: start-first, and the image’s own HEALTHCHECK. Swarm brings a new task up, waits for /health, shifts traffic, and stops the old one; with a single replica there is still a moment when the only healthy task is the one being replaced.

Running two instances is safe here by design rather than by luck: the detection loop and the alert queue claim rows with FOR UPDATE SKIP LOCKED, so they share work instead of alerting twice, and the ingest cache is invalidated across instances with LISTEN/NOTIFY.

The precondition is that migrations stay additive. During the changeover the old and new versions run against the same schema for a few seconds. That holds for every release whose notes do not say otherwise — when one does, deploy it as a brief planned outage instead.

If the new version never reports healthy, Swarm puts the old one back on its own (failure_action: rollback), and the deploy job fails on the version check rather than reporting a success that did not happen.

Terminal window
docker swarm init # if it is not already one
docker node update --label-add silencewatch.db=true "$(docker node ls -q)"
sudo install -d -m 750 /opt/silencewatch
sudo cp .env /opt/silencewatch/.env # SECRET_KEY, POSTGRES_PASSWORD, BASE_URL…
sudo chmod 600 /opt/silencewatch/.env

The node label is not optional. Without it a reschedule would start PostgreSQL on another machine against an empty local volume — an instance that looks healthy and has lost every check, ping and account. For the same reason the database updates stop-first: two PostgreSQL processes on one data directory corrupt it, so it takes a brief pause where the application does not.

Deploy by hand with:

Terminal window
set -a; . /opt/silencewatch/.env; set +a
SILENCEWATCH_VERSION=0.1.0 docker stack deploy -c docker-stack.yml silencewatch

.github/workflows/release.yml runs the same deployment over SSH once a tag’s image has been published and smoke tested. Tag a release, and the swarm is on it a couple of minutes later without anyone logging in.

This is for a fork you operate. It is not needed to self-host: the manual docker stack deploy above is the same operation, run when you choose.

Terminal window
ssh-keygen -t ed25519 -C "github-actions@silencewatch" \
-f ~/.ssh/silencewatch_deploy -N ""

-N "" means no passphrase, because CI cannot type one. That is exactly why this is a dedicated key rather than yours: it can be revoked on its own, and it never protected anything else.

Terminal window
sudo adduser --disabled-password --gecos "" deploy
sudo usermod -aG docker deploy
sudo install -d -m 700 -o deploy -g deploy /home/deploy/.ssh
sudo tee /home/deploy/.ssh/authorized_keys <<EOF
restrict,pty $(cat ~/.ssh/silencewatch_deploy.pub)
EOF
sudo chown deploy:deploy /home/deploy/.ssh/authorized_keys
sudo chmod 600 /home/deploy/.ssh/authorized_keys

restrict disables port forwarding, agent forwarding and X11, none of which a deployment needs.

Membership of the docker group is equivalent to root on that machine. The daemon runs as root and will mount any path on the host for you. This is inherent to deploying over SSH rather than a weakness of this setup, but it sets the value of the key: treat it as a root credential, keep it out of any other system, and revoke it by deleting the line from authorized_keys.

Secret What to paste Where it comes from
SWARM_SSH_HOST the host alone, e.g. silencewatch.com no user@, no port
SWARM_SSH_USER deploy the user created above
SWARM_SSH_KEY the private key, in full cat ~/.ssh/silencewatch_deploy
SWARM_SSH_KNOWN_HOSTS the host’s fingerprints ssh-keyscan -H <host>

Two things that trip people up. SWARM_SSH_KEY is the file without the .pub suffix — the .pub one belongs on the server — and it must include the -----BEGIN/-----END lines. ssh-keyscan prints one line per key type: paste all of them.

The last secret is what makes the connection safe. Without a pinned host key the alternative is StrictHostKeyChecking=no, which hands the deploy key, and every command that follows it, to whatever answers on that address. The job refuses to run if the value is empty rather than falling back to trusting anything.

Settings → Environments → production → Add environment secret.

Repository secrets work too, but scoping them to the environment means only the job that targets it can read them — not the build job, and not a workflow added later. production is also where a required reviewer goes if releases should stop shipping unattended.

If the env file is not at /opt/silencewatch/.env, set a repository variable (not a secret) named SWARM_ENV_FILE to its path. The values inside it stay on the server: CI reads their names from the stack file and never sees them.

Terminal window
ssh -i ~/.ssh/silencewatch_deploy deploy@<host> \
'docker version --format "{{.Server.Version}}" && docker node ls'

A version and a node list means the four secrets will work. A password prompt means authorized_keys is not being read — usually its permissions, or those of /home/deploy/.ssh.

The workflow connects on port 22. A different port needs -p adding to the two ssh invocations in release.yml.

For a support thread, produce a bundle:

Terminal window
docker compose exec silencewatch node packages/server/dist/diagnostics/support-bundle.js /tmp/bundle.json
docker compose cp silencewatch:/tmp/bundle.json .

It contains versions, the effective configuration as set/not-set rather than values, table sizes, index presence, migration state and health — no secrets, no ping keys, no email addresses. Read it before sending it anyway.

Run more than one instance when a single process is no longer enough. Nothing needs changing: the detection loop and the alert queue claim rows with FOR UPDATE SKIP LOCKED, so instances share work without duplicating alerts, and the ingestion cache is invalidated across instances through PostgreSQL LISTEN/NOTIFY.

Beyond that the database is the limit. In order: raise INGEST_POOL_MAX, put the ping table on faster storage, then lower PING_RETENTION_DAYS.