Applies to: Every federation site that wants automated, API-accessible health monitoring without requiring SSH from the coordinator.
Working implementation: wf/ops/templates/services/site-server/ Live endpoint: https://wf.klopfenstein.org/status.json Coordinator integration: wip/ops/scripts/netstack-status.js Federation dashboard: wip.2cld.net/docs/status/ (reads all site endpoints) Related issue: wip#16 — reactive alerting
A site should report its own health. The coordinator (wip) reads the report via HTTPS — no SSH, no VPN dependency, no key management.
The site-status service runs locally on the site, checks local infrastructure, and publishes a JSON status document via the site’s existing Cloudflare tunnel. The coordinator reads it with a single curl.
┌─── Site (e.g., wf) ────────────────────────────────────────┐
│ │
│ checks.yml ──→ checker (Python, Docker) │
│ │ │
│ ├──→ /srv/output/status.json (API) │
│ └──→ /srv/output/health.json (self-check) │
│ │
│ nginx (Docker, :8600) │
│ │ │
│ ▼ │
│ Cloudflare tunnel route │
│ (site.domain.org/status.json, site.domain.org/health) │
│ │
└─────────────────────────────────────────────────────────────┘
│
│ HTTPS (public internet)
▼
┌─── Coordinator (wip/nsdockerhv) ───────────────────────────┐
│ │
│ curl -s https://site.domain.org/status.json │
│ → parse JSON → compare with previous state → alert │
│ │
│ netstack-status.js --write-state --alert │
│ → .local/.monitor-state.json (state tracking) │
│ → .local/.alerts (on critical transition) │
│ │
└─────────────────────────────────────────────────────────────┘
checks.yml — What to checkDeclarative config listing all health checks for the site:
site: wf
site_name: "Winfield"
checks:
- name: "cg2 (Proxmox + ZFS)"
type: ping
target: "192.168.9.3"
tier: operational
goal: "Media distribution"
- name: "Plex (wfMedia)"
type: http
url: "http://192.168.9.3:32400/identity"
expect_status: 200
tier: operational
goal: "Media distribution"
- name: "sg Synology (known DOWN)"
type: ping
target: "192.168.9.2"
tier: cold
known_down: true
issue: "https://github.com/2cld/wf/issues/2"
- name: "wf-cg boot disk"
type: disk
path: "/"
alert_below_gb: 5
tier: operational
goal: "Compute health"
Check types:
| Type | What it does | Required fields |
|---|---|---|
ping |
ICMP ping to target IP | target |
http |
HTTP GET, verify status code | url, expect_status |
command |
Run shell command, check output | command, expect_contains |
disk |
Check free space on path | path, alert_below_gb |
Special fields:
known_down: true — suppresses alerts for intentionally-offline nodesissue — links to the tracking issue for the known-down statetier — operational (alert if down), cold (informational only)goal — maps to site goals per site-status-page-patternchecker.py — Status check loopPython service that:
checks.ymlok / warning / degraded)status.json atomically (tmp + rename)health.json (self-check)CHECK_INTERVAL seconds (default: 300 = 5 min)Environment variables:
| Var | Default | Purpose |
|---|---|---|
SITE_CODE |
wf |
Site identifier in output |
CONFIG_FILE |
/config/checks.yml |
Path to check definitions |
OUTPUT_DIR |
/data |
Where to write results |
CHECK_INTERVAL |
300 |
Seconds between check runs |
| Endpoint | Purpose | Update frequency |
|---|---|---|
/status.json |
Machine-readable health checks | Every 5 min |
/health or /health.json |
Container self-check (is the service alive?) | Every 5 min |
docker-compose.yml — Deploy togetherservices:
checker:
build: ./checker
container_name: site-status-checker
restart: unless-stopped
network_mode: host
volumes:
- status_data:/data
- ./checks.yml:/config/checks.yml:ro
environment:
- CHECK_INTERVAL=300
- SITE_CODE=wf
- OUTPUT_DIR=/data
- CONFIG_FILE=/config/checks.yml
web:
image: nginx:alpine
container_name: site-status-web
restart: unless-stopped
ports:
- "8600:80"
volumes:
- status_data:/usr/share/nginx/html:ro
- ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
volumes:
status_data:
status.json){
"site": "wf",
"site_name": "Winfield",
"timestamp": "2026-08-06T22:37:42.131000+00:00",
"status": "ok",
"check_interval_seconds": 300,
"checks": [
{
"name": "cg2 (Proxmox + ZFS)",
"type": "ping",
"status": "ok",
"message": "0.301ms",
"tier": "operational",
"goal": "Media distribution"
}
],
"check_count": 6,
"ok_count": 4,
"error_count": 0,
"warning_count": 0,
"known_down_count": 2
}
Status values:
| Value | Meaning | Coordinator action |
|---|---|---|
ok |
All operational checks passing | None |
warning |
Non-critical degradation | Log, surface in review |
degraded |
Operational check failing | Alert (per escalation levels) |
error |
Checker itself failed | Alert — checker container down |
known_down |
Intentionally offline | Ignore |
STATUS_JSON=$(curl -s --max-time 10 https://site.domain.org/status.json)
OVERALL=$(echo "$STATUS_JSON" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d['status'])")
Compare timestamp to now. If > 10 minutes old (CHECK_INTERVAL × 2), the checker may be down:
from datetime import datetime, timezone
age_min = (datetime.now(timezone.utc) - dt).total_seconds() / 60
stale = age_min > 10
The coordinator (netstack-status.js) runs every 30 min and:
.local/.monitor-state.json).local/.alertswf/ops/templates/services/site-status/checks.yml for site infrastructure (nodes, services, disks)docker compose up -d --buildsite.domain.org → http://localhost:8600wip-daily-cron.sh federation section)curl -s https://site.domain.org/status.json | python3 -m json.toolA static HTML/JS page at wip.2cld.net/docs/status/ consumes the status.json endpoints from all sites and renders a unified view:
Access-Control-Allow-Origin: *) enable browser fetchSource: wip/docs/status/index.html Issue: netstack#16
| Site | Status | Endpoint | Repo | Deploy Runbook |
|---|---|---|---|---|
| wf | ✅ Live | https://wf.klopfenstein.org/status.json | 2cld/wf | — |
| cf | 🔄 PR#5 | https://cf.2cld.net/status.json (pending) | 2cld/cf PR#5 | ops-cf-status-deploy.md |
| sl | Planned | https://sl.2cld.net/status.json | 2cld/sl | TBD |