| Status: Adopted (wf validated 2026-08-13, cf PR open 2026-08-16) | Goal: All sites by Q4 2026 |
SSH-based monitoring from the coordinator (nsdockerhv/wip) to site nodes is unreliable:
Result: “Permission denied” errors in daily cron logs for weeks/months with no fix.
Each site generates its own status.json locally, then publishes it via a Cloudflare tunnel at a public HTTPS endpoint. The coordinator reads it with a simple curl or fetch().
┌─────────────────────────────────────────────────────┐
│ SITE (local) │
│ │
│ checks.yml → site-status-checker → status.json │
│ │ │
│ nginx (site-web) ← serves status.json + /health │
│ │ │
│ cloudflared tunnel ← exposes to internet │
└─────────┼───────────────────────────────────────────┘
│ HTTPS (public, CF Access bypass on /status.json)
▼
┌─────────────────────────────────────────────────────┐
│ COORDINATOR (wip on nsdockerhv) │
│ │
│ curl https://site.domain/status.json │
│ │ │
│ netstack-status.js / storage-status.js │
│ │ │
│ .monitor-state.json → morning-update → README │
└─────────────────────────────────────────────────────┘
│
│ HTTPS (public)
▼
┌─────────────────────────────────────────────────────┐
│ FEDERATION DASHBOARD (wip.2cld.net/docs/status/) │
│ │
│ Static HTML + JS (GitHub Pages) │
│ Fetches status.json from each site │
│ Renders unified status view │
│ Auto-refreshes every 60s │
│ │
└─────────────────────────────────────────────────────┘
Publish status per site, not per service: one /status.json and /health on the site’s hostname (e.g. cf.cat9.me). The checker probes each service from inside, so the public view is a single endpoint. Every other hostname stays fully behind Cloudflare Access (access pattern).
| Path | Required keys | Notes |
|---|---|---|
/status.json |
site, site_name, timestamp (ISO 8601 UTC), status (ok/degraded/error), check_interval_seconds, checks[] (name, type, status, message, tier, goal), counts |
read-only, no actions |
/health |
service, site, status, timestamp, check_interval_seconds |
liveness of the status service only; ok even when /status.json says degraded |
Rules, all enforced by verify-access.sh on every bypass path (cf ops/services/site-status/):
messagetimestamp no older than 3 x check_interval_seconds, so a dead checker serving an old file turns redVerified 2026-11-01: cf and wip match this contract (same checker output).
Status is public, data is private. The status endpoint reports health (ok/error/warning) and capacity (GB free). It does NOT expose file contents, credentials, or user data.
No SSH for monitoring. SSH is for administration and backup. Monitoring reads a published artifact.
No ZeroTier dependency for monitoring. Cloudflare tunnels provide the transport. ZT is only needed for backup transfers and SSH admin.
Staleness is the alert. If status.json is older than 10 minutes, the checker is down. The coordinator doesn’t need to know why — stale = problem.
Site autonomy. Each site owns its checks.yml. Adding a new check is a git push, not a coordinator config change.
Federation dashboard reads the same endpoints. The web UI at wip.2cld.net/docs/status/ fetches each site’s status.json directly — no server-side aggregation needed.
| Component | Container | Purpose |
|---|---|---|
| site-status-checker | site-status-checker or cf-status-checker |
Reads checks.yml, runs checks every 5 min, writes status.json |
| site-web | nginx | Serves status.json at /status.json and /health |
| cloudflared | cloudflared |
Tunnels site-web to public internet |
| Endpoint | Auth | Purpose |
|---|---|---|
https://{site}.domain/status.json |
None (CF Access bypass) | Machine-readable health checks |
https://{site}.domain/health |
None (CF Access bypass) | Simple container alive check |
status.json schema{
"site": "wf",
"site_name": "Winfield",
"timestamp": "2026-08-13T15:47:20.123Z",
"status": "ok|warning|degraded|error",
"check_interval_seconds": 300,
"check_count": 12,
"ok_count": 10,
"error_count": 1,
"warning_count": 0,
"known_down_count": 1,
"checks": [
{
"name": "descriptive name",
"type": "ping|http|command|disk",
"status": "ok|error|warning|known_down",
"message": "human-readable result",
"tier": "operational|cold|glacial",
"goal": "which contract goal this serves"
}
]
}
| Site | Local Checker | Public Endpoint | Coordinator Reads | Dashboard | Status |
|---|---|---|---|---|---|
| wf | ✅ | ✅ wf.klopfenstein.org/status.json |
✅ | ✅ | Done (2026-08-13) |
| cf | ✅ PR#5 | Pending deploy | ❌ (runs locally) | Pending | In Progress (2026-08-16) |
| sl | ❌ | ❌ | ❌ (SSH to WSL) | Pending | Needs site-server deploy |
cf: Merge PR#5, deploy Docker stack, add CF tunnel route to cf.2cld.net. Deploy runbook: wip/docs/ops-cf-status-deploy.md
sl: Deploy site-server container on slwin11ops WSL (or dedicated LXC). Configure checks.yml for F: drive, docker services, tunnel health. Expose via sl’s Cloudflare tunnel at sl.2cld.net/status.json.
Federation dashboard: Live at wip.2cld.net/docs/status/ — fetches all site endpoints, renders unified view. Code in wip/docs/status/index.html.
Coordinator migration: Once all sites publish, remove SSH-based checks from wip-daily-cron.sh, storage-status.js, and netstack-status.js. The public endpoint becomes the single source of truth.
Before (SSH proxy, daily failures):
ghadmin@10.147.17.165: Permission denied (publickey,password,keyboard-interactive).
❌ site-server: Command failed: ssh -o BatchMode=yes ...
After (public endpoint, instant):
--- WF (Winfield) ---
✅ devwin10 (10.147.17.165) — online
--- site-server (https://wf.klopfenstein.org/status.json) ---
✅ site-server: ok
📊 status.json: error (10/12 ok) @ 2026-08-13T15:47:20
✅ Mikrotik router: 0.345ms
✅ cg2 (Proxmox + ZFS): 0.302ms
...
No SSH. No ZT dependency. No Permission denied. Sub-second response.