Server Health Check: 13 Checks and a Linux Script

A server health check answers one question: is anything on this machine about to cause an outage? On a Linux server that means load, memory, disk space, failed services, pending updates, the SSH and firewall settings, and the containers that run your apps.

This guide gives you the command for each check, what a bad result looks like, and one script that runs them all and exits non-zero when something needs you. Then it runs the script on a schedule, sends failures to a webhook, and covers the one failure a server can’t report itself. The commands work on Ubuntu 24.04 and Debian 12, and most work on any distribution with systemd.

For monitoring concepts and tool choices, see the Linux server monitoring and uptime guide.

The checklist

Run these as root, or with sudo, on the server.

A few of these catch people out.

• Load is per core. A load of 4 is fine on 8 cores and a problem on 2. Compare it with nproc, not with a fixed number.
• Used memory looks high on a healthy server. Linux fills spare memory with cache and gives it back on demand. Read the available column, not used.
• Inodes run out before space does. Millions of small files, such as sessions or cache entries, can fill the inode table on a disk that is half empty. Writes then fail with No space left on device.
• sshd -T shows the settings in force. Reading /etc/ssh/sshd_config misses files in sshd_config.d/ that override it.
• Exited (0) is not a crash. A one-off container that finished its job exits with code 0. Any other code means the process failed.

A health check script

The script runs every check in the table and prints one line per result. It exits 0 when everything is OK, 1 when there is at least one warning, and 2 when at least one check fails. The thresholds are environment variables, so you can change them per server without editing the file.

#!/usr/bin/env bash
# Server health check. Prints one line per check.
# Exit code: 0 all OK, 1 at least one WARN, 2 at least one FAIL.
set -uo pipefail

DISK_WARN=${DISK_WARN:-80}
DISK_FAIL=${DISK_FAIL:-90}
MEM_WARN=${MEM_WARN:-10}   # warn below this % of memory available
LOAD_WARN=${LOAD_WARN:-1.5} # warn above this 5-minute load per CPU core

status=0
ok()   { printf 'OK    %s\n' "$1"; }
warn() { printf 'WARN  %s\n' "$1"; (( status < 1 )) && status=1; }
fail() { printf 'FAIL  %s\n' "$1"; status=2; }
has()  { command -v "$1" >/dev/null 2>&1; }

# Load, per CPU core
cores=$(nproc)
load5=$(cut -d' ' -f2 /proc/loadavg)
if awk -v l="$load5" -v c="$cores" -v t="$LOAD_WARN" 'BEGIN { exit !(l / c > t) }'; then
  warn "load: $load5 over 5 minutes on $cores cores"
else
  ok "load: $load5 over 5 minutes on $cores cores"
fi

# Memory available
mem_pct=$(awk '/^MemTotal/ { t = $2 } /^MemAvailable/ { a = $2 } END { printf "%d", a * 100 / t }' /proc/meminfo)
if (( mem_pct < MEM_WARN )); then
  warn "memory: ${mem_pct}% available"
else
  ok "memory: ${mem_pct}% available"
fi

# Disk space and inodes, per real filesystem
while read -r use inode mount; do
  use=${use%\%}
  inode=${inode%\%}
  [[ $inode == "-" ]] && inode=0
  if (( use >= DISK_FAIL || inode >= DISK_FAIL )); then
    fail "disk $mount: ${use}% used, ${inode}% inodes"
  elif (( use >= DISK_WARN || inode >= DISK_WARN )); then
    warn "disk $mount: ${use}% used, ${inode}% inodes"
  else
    ok "disk $mount: ${use}% used, ${inode}% inodes"
  fi
done < <(df -x tmpfs -x devtmpfs -x overlay -x squashfs --output=pcent,ipcent,target | tail -n +2)

# Failed systemd units
failed=$(systemctl list-units --state=failed --no-legend --plain | awk '{ print $1 }' | paste -sd' ' -)
if [[ -n $failed ]]; then
  fail "systemd: failed units: $failed"
else
  ok "systemd: no failed units"
fi

# Pending updates and reboots (Debian and Ubuntu)
if has apt-get; then
  updates=$(apt-get -s -o Debug::NoLocking=true upgrade 2>/dev/null | grep -c '^Inst')
  if (( updates > 0 )); then warn "updates: $updates packages pending"; else ok "updates: none pending"; fi
fi
if [[ -f /var/run/reboot-required ]]; then
  warn "reboot: required"
fi

# Time sync
if [[ $(timedatectl show -p NTPSynchronized --value 2>/dev/null) == "yes" ]]; then
  ok "time: synchronised"
else
  warn "time: not synchronised"
fi

# SSH root login (needs root to read the effective config)
if has sshd && (( EUID == 0 )); then
  root_login=$(sshd -T 2>/dev/null | awk '$1 == "permitrootlogin" { print $2 }')
  if [[ $root_login == "no" ]]; then ok "ssh: root login off"; else warn "ssh: root login is ${root_login:-unknown}"; fi
fi

# Fail2Ban and firewall
if systemctl is-active --quiet fail2ban; then ok "fail2ban: active"; else warn "fail2ban: not active"; fi
if has ufw && (( EUID == 0 )); then
  if ufw status | grep -q '^Status: active'; then ok "firewall: active"; else warn "firewall: inactive"; fi
fi

# Docker containers
if has docker; then
  if ! containers=$(docker ps -a --format '{{.Names}} {{.Status}}' 2>/dev/null); then
    fail "docker: daemon not reachable"
  else
    bad=$(grep -E 'unhealthy|Restarting|Exited \([1-9][0-9]*\)' <<<"$containers" | awk '{ print $1 }' | paste -sd' ' -)
    if [[ -n $bad ]]; then fail "docker: $bad"; else ok "docker: no unhealthy, restarting or crashed containers"; fi
  fi
fi

exit "$status"

Install it and run it once:

sudo install -m 755 health-check.sh /usr/local/bin/health-check.sh
sudo health-check.sh; echo "exit $?"

Example output on a server with one crashed container and pending updates:

OK    load: 0.42 over 5 minutes on 4 cores
OK    memory: 63% available
OK    disk /: 41% used, 7% inodes
OK    systemd: no failed units
WARN  updates: 14 packages pending
OK    time: synchronised
OK    ssh: root login off
OK    fail2ban: active
OK    firewall: active
FAIL  docker: worker
exit 2

Checks that don’t apply are skipped. A server without Docker gets no Docker line, and the SSH and firewall lines need root. Disks of type tmpfs and overlay are left out, because Docker’s overlay mounts repeat the root disk.

Run it on a schedule and get alerts

A check you have to remember to run is a check you won’t run. This wrapper runs the script, posts the failures to a webhook, and pings a heartbeat URL on every run:

#!/usr/bin/env bash
# Runs the health check, posts failures to a webhook, pings a heartbeat on every run.
set -uo pipefail

report=$(/usr/local/bin/health-check.sh)
code=$?

if (( code == 2 )); then
  text=$(printf 'Health check failed on %s:\n%s' "$(hostname)" "$(grep '^FAIL' <<<"$report")")
  curl -fsS -m 10 -H 'Content-Type: application/json' \
    -d "$(jq -n --arg text "$text" '{text: $text}')" "$HEALTH_WEBHOOK_URL"
fi

curl -fsS -m 10 -o /dev/null "$HEALTH_HEARTBEAT_URL"

It alerts only on FAIL. Warnings such as pending updates can stay true for days, and an alert every ten minutes for them trains you to ignore the channel. Read the warnings from a manual run, or from a daily report.

The payload is {"text": "..."}, which Slack incoming webhooks accept. For Discord, add /slack to the end of the webhook URL. The wrapper needs jq and curl (sudo apt install jq curl).

Schedule it with a file in /etc/cron.d:

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HEALTH_WEBHOOK_URL=https://hooks.example.com/your-webhook
HEALTH_HEARTBEAT_URL=https://heartbeat.example.com/your-check
*/10 * * * * root /usr/local/bin/health-alert.sh >/dev/null 2>&1

Set the PATH line. Cron’s default PATH leaves out /usr/sbin, where sshd and ufw live, and the script then skips those checks without a word.

sudo install -m 755 health-alert.sh /usr/local/bin/health-alert.sh
sudo chmod 644 /etc/cron.d/health-check

The check a server can’t run on itself

Every check above runs on the server. When the server is down, out of memory, or cut off from the network, nothing runs and nothing alerts. A silent channel then looks the same as a healthy server.

The heartbeat covers that gap. The wrapper calls HEALTH_HEARTBEAT_URL on every run, and a heartbeat service on another machine alerts you when the pings stop. Any uptime or cron-monitoring service with a heartbeat (also called a dead man’s switch) works. Set its grace period a little longer than the schedule, such as 15 minutes for a 10-minute cron, so one slow run doesn’t page you.

For a web app, add an HTTP check from outside as well. A server can pass every check here while the app returns errors.

Check a fleet without the scripts

The script works for one server. With five, you are maintaining five cron files, five sets of thresholds and a webhook that tells you nothing until something breaks.

Serversinc runs these checks on every server it manages. The agent reports every 5 minutes, and a server that misses its heartbeat for 12 minutes raises a critical alert. Disk, CPU and memory raise a warning when they stay above 80% for 10 minutes, so a short spike doesn’t page you. Hardening checks (SSH root login, Fail2Ban, automatic updates, time sync and the firewall) run daily, and some have a fix you can apply from the dashboard. A container that stops unexpectedly raises an alert, and alerts reach email, Slack or a webhook.

See Server health and metrics and the alert reference for every check and threshold. For containers in particular, How to Monitor Docker Containers with Real-time Alerts shows an event-driven approach that alerts the moment a container stops.