Skip to content

How to recover from a full outage

Symptoms

  • ping 147.93.186.68 fails, or
  • Host boots but no HTTPS services answer, or
  • Multiple interdependent stacks failed after reboot/disk pressure

Steps

  1. Confirm reachability from more than one network path:
    bash
    ping -c 3 147.93.186.68
  2. Out-of-band console via the hosting provider if SSH is dead.
  3. Boot errors:
    bash
    journalctl -b -p err --no-pager | tail -100
  4. Disk full (common silent killer):
    bash
    df -h
    docker system df
    Clear safe space: old logs, docker image prune, /tmp junk — not application bind mounts under ~/nextcloud, ~/puter, etc.
  5. Failed units:
    bash
    systemctl --failed
    systemctl restart docker caddy
  6. Containers:
    bash
    docker ps -a
    # restart stacks intentionally, one at a time
  7. If filesystem corruption is evident, stop writes and restore from backup (restic how-to).

Verify

bash
systemctl is-active caddy docker
docker ps
curl -sI https://cloud.cadenceworks.tech | head -5

Northeast Systems · Aux Server · Infra Server