netstack

Pattern: Access Monitoring & Authentication Audit (Security Layers 3-5)

Category: docs/ops/security/ Purpose: Detective security — who accessed what, when, and was it expected? Catch unauthorized access, stale keys, and anomalous behavior. Prerequisite patterns:

Security Onion Model

Layer 0: Network boundary    (ZeroTier membership — who CAN reach)
Layer 1: Authentication      (SSH keys — who IS on the node)
Layer 2: Authorization       (sudoers — what they CAN do)
Layer 3: Audit trail         (logs — what they DID do)           ← THIS DOC
Layer 4: Behavioral baseline (patterns — was this EXPECTED)      ← THIS DOC
Layer 5: Alerting            (deviation — flag the UNEXPECTED)   ← THIS DOC

Part 1: Access Matrix

This table defines the AUTHORIZED access paths. Anything not listed here is unexpected.

SSH Keys (outbound from nsdockerhv)

Key Fingerprint Identity Targets Purpose Schedule
id_backup SHA256:KDGH24XN... nsub2404hv-backup CT, sl, wf, cat9fin Backup cron (SCP) Daily 2 AM
id_rsa SHA256:UqoEjZmH... nsadmin@horseoff.com github.com (as horseoffcom) Interactive git push On-demand

SSH Keys (inbound to nsdockerhv)

Source Key comment Authorized for user Purpose
CyberTruck cfbu-backup@cybertruck nsadmin (legacy — review if still needed)

SSH Targets (per site contract)

Target IP (ZT) Port User Key Access from nsdockerhv
CyberTruck 10.147.17.219 22 ghadmin id_backup Yes (backup-daily.sh)
cat9fin 10.147.17.218 22 ghadmin id_backup Yes (backup-daily.sh)
slwin11ops 10.147.17.94 22 ghadmin id_backup Yes (backup + monitoring)
slwin11ops WSL 10.147.17.94 2020 ghadmin id_backup Yes (sl-status)
devwin10 10.147.17.165 22 ghadmin default Yes (wf monitoring)

Part 2: Expected Automated Access (Behavioral Baseline)

These SSH connections are EXPECTED at specific times. Anything outside this pattern is a deviation.

Time Source Target User Purpose Script
02:00 daily nsdockerhv CyberTruck :22 ghadmin Pull hwpc-rp sync backup-daily.sh
02:00 daily nsdockerhv slwin11ops :22 ghadmin Push Docker backups backup-daily.sh
02:00 daily nsdockerhv devwin10 :22 buadmin Push Docker backups (wf leg) backup-daily.sh
03:00 Sun nsdockerhv CyberTruck :22 ghadmin Push wip-creds backup backup-wip-credentials.sh
03:00 Sun nsdockerhv slwin11ops :22 ghadmin Push wip-creds backup backup-wip-credentials.sh
05:30 daily nsdockerhv slwin11ops :22 ghadmin sl disk check (wip-daily-cron) wip-daily-cron.sh
05:30 daily nsdockerhv devwin10 :22 ghadmin wf status check wip-daily-cron.sh
05:30 daily nsdockerhv devwin10 :22 ghadmin site-server proxy (192.168.9.9) netstack-status.js
On-demand nsdockerhv any ZT node ghadmin Interactive troubleshooting Manual (site-admin)

What “unexpected” looks like:

Part 3: Auth Log Collection

Linux (nsdockerhv, LXCs)

# Recent auth events
grep "sshd" /var/log/auth.log | tail -20

# Failed attempts (Layer 5 signal)
grep "Failed password\|Invalid user\|Connection closed by.*preauth" /var/log/auth.log | tail -10

# Accepted keys (Layer 3 audit trail)
grep "Accepted publickey" /var/log/auth.log | tail -10

# Enable key fingerprint logging (one-time):
# /etc/ssh/sshd_config: LogLevel VERBOSE
# systemctl restart sshd

Windows (slwin11ops, devwin10, CyberTruck)

# Failed logons (Event 4625) — last 24h
Get-WinEvent -FilterHashtable @{LogName='Security';Id=4625;StartTime=(Get-Date).AddDays(-1)} | Select TimeCreated,Message | Format-List

# Successful logons (Event 4624) — last 24h
Get-WinEvent -FilterHashtable @{LogName='Security';Id=4624;StartTime=(Get-Date).AddDays(-1)} | Where-Object { $_.Message -match "Logon Type:\s+10|Logon Type:\s+3" } | Select TimeCreated

# OpenSSH log (if sshd logging configured)
Get-Content C:\ProgramData\ssh\logs\sshd.log -Tail 20

Part 4: Audit Script (audit-ssh-keys.sh)

Run periodically to detect stale or unauthorized keys:

#!/bin/bash
# audit-ssh-keys.sh — Compare deployed authorized_keys against expected list
# Run: manually during weekly review, or add to wip-daily-cron.sh

# Expected keys (from this pattern doc — source of truth)
declare -A EXPECTED_KEYS
EXPECTED_KEYS["nsadmin@nsdockerhv"]="cfbu-backup@cybertruck"
EXPECTED_KEYS["wip@nsdockerhv"]=""  # no inbound expected

echo "=== SSH Key Audit ==="

for user_host in "${!EXPECTED_KEYS[@]}"; do
  user="${user_host%%@*}"
  host="${user_host##*@}"
  auth_file="/home/$user/.ssh/authorized_keys"
  
  if [ -f "$auth_file" ]; then
    actual=$(awk '{print $NF}' "$auth_file" | sort | tr '\n' ',' | sed 's/,$//')
    expected="${EXPECTED_KEYS[$user_host]}"
    
    if [ "$actual" = "$expected" ]; then
      echo "  OK $user@$host: $actual"
    else
      echo "  ⚠️  $user@$host: MISMATCH"
      echo "    Expected: $expected"
      echo "    Actual:   $actual"
    fi
  else
    echo "  -- $user@$host: no authorized_keys file"
  fi
done

# Check for unexpected users with .ssh directories
echo ""
echo "=== Users with .ssh dirs ==="
find /home -maxdepth 2 -name ".ssh" -type d 2>/dev/null | while read dir; do
  user=$(echo "$dir" | cut -d/ -f3)
  keys=$(wc -l < "$dir/authorized_keys" 2>/dev/null || echo 0)
  echo "  $user: $keys authorized key(s)"
done

Part 5: Integration with wip-daily-cron.sh

Add to the daily cron report (quick check, not full audit):

# --- Auth Health Check ---
echo "## Auth Health"

# Failed SSH attempts in last 24h
FAILED=$(grep -c "Failed\|Invalid user" /var/log/auth.log 2>/dev/null || echo 0)
if [ "$FAILED" -gt 10 ]; then
  echo "  ❌ $FAILED failed SSH attempts (last 24h) — investigate"
elif [ "$FAILED" -gt 0 ]; then
  echo "  ⚠️ $FAILED failed SSH attempts (last 24h)"
else
  echo "  ✅ No failed SSH attempts"
fi

# Check if any unexpected keys were added
EXPECTED_KEY_COUNT=1  # nsadmin should have 1 key
ACTUAL=$(wc -l < /home/nsadmin/.ssh/authorized_keys 2>/dev/null || echo 0)
if [ "$ACTUAL" -ne "$EXPECTED_KEY_COUNT" ]; then
  echo "  ⚠️ nsadmin authorized_keys: $ACTUAL keys (expected $EXPECTED_KEY_COUNT)"
else
  echo "  ✅ authorized_keys: $ACTUAL key(s) (expected)"
fi

Part 6: Response Playbook

Signal Severity Action
Failed attempts < 5/day INFO Log, no action (noise from port scanners)
Failed attempts > 10/day WARNING Check source IPs, verify ZT boundary intact
Failed attempts > 50/day CRITICAL Enable fail2ban, investigate source, alert site-admin
Unknown key in authorized_keys CRITICAL Identify owner immediately, revoke if unknown
SSH at unexpected time WARNING Correlate with cron schedule, verify if manual admin work
Root login attempt CRITICAL Root SSH should be disabled — if attempt succeeded, incident response
Key fingerprint mismatch on target WARNING Target node may have been rebuilt without updating known_hosts

Part 7: Key Rotation Schedule

Key Rotation trigger Procedure
id_backup (backup-agent) On compromise, or annually Generate new ed25519, deploy pub to all targets, update DR USB
id_rsa (nsadmin interactive) On compromise Generate new, register on GitHub (horseoffcom)
GitHub PAT (GITHUB_HOWIP_API) On expiry or compromise gh auth login device flow, update .env
Gitea token On compromise Regenerate at gitea.cat9.me/user/settings/applications
ZeroTier API token On compromise Regenerate at my.zerotier.com

Quarterly check (Sunday weekly review, once per quarter):

Implementation Phases

Phase 1: Immediate (today)

Phase 2: Short-term (next month)

Phase 3: Medium-term (quarterly)