Cron job monitor + healthcheck.io ping

Cron silent failure adalah mimpi buruk — backup harian gak jalan 3 minggu, baru ketahuan saat butuh restore. Healthchecks.io (atau cronitor) provide dead-man switch — kirim ping setiap run, kalau gak ping selama X waktu, alert. Snippet ini wrapper bash yang ping start/success/fail + log capture.

Kode

#!/usr/bin/env bash
# /usr/local/bin/run_with_healthcheck.sh
# Usage: run_with_healthcheck.sh <healthcheck-uuid> <command...>

set -euo pipefail

readonly HC_UUID="${1:?Pakai: $0 <hc-uuid> <command...>}"
shift
readonly CMD=("$@")

readonly HC_BASE="${HC_BASE:-https://hc-ping.com}"
readonly HC_URL="${HC_BASE}/${HC_UUID}"
readonly LOG_FILE="/var/log/cron_jobs/${HC_UUID}.log"
readonly MAX_LOG_SIZE_KB=100  # Truncate kalau lebih besar

mkdir -p "$(dirname "${LOG_FILE}")"

hc_ping() {
  local endpoint="$1"
  local body="${2:-}"

  # Retry dengan backoff — tahan network flap
  for attempt in 1 2 3 4 5; do
    if curl --fail --silent --show-error --max-time 10 \
        --data-binary @- \
        --user-agent "cron-monitor/1.0" \
        "${HC_URL}${endpoint}" <<< "${body}" > /dev/null; then
      return 0
    fi
    sleep "$((attempt * 2))"
  done

  # All retry gagal — log lokal saja
  echo "[$(date -Is)] WARN: hc ping ${endpoint} gagal setelah 5 retry" \
    >> "${LOG_FILE}"
  return 0  # Jangan fail job hanya karena monitoring fail
}

main() {
  local start_time
  start_time=$(date +%s)

  # 1. Ping start
  hc_ping "/start"

  # 2. Jalankan command, capture output ke log + stdout
  local exit_code=0
  local tmp_log
  tmp_log=$(mktemp)

  if "${CMD[@]}" > >(tee -a "${tmp_log}") 2>&1; then
    exit_code=0
  else
    exit_code=$?
  fi

  local duration=$(( $(date +%s) - start_time ))

  # Truncate output kalau besar (healthchecks limit 100KB)
  local truncated_log
  if [ "$(wc -c < "${tmp_log}")" -gt "$((MAX_LOG_SIZE_KB * 1024))" ]; then
    truncated_log=$(tail -c "$((MAX_LOG_SIZE_KB * 1024))" "${tmp_log}")
    truncated_log=$'[...log truncated...]\n'"${truncated_log}"
  else
    truncated_log=$(cat "${tmp_log}")
  fi

  # Append ke local log (rotate via logrotate terpisah)
  {
    echo "===== Run at $(date -Is) — exit=${exit_code} — duration=${duration}s ====="
    echo "${truncated_log}"
    echo ""
  } >> "${LOG_FILE}"

  # 3. Ping end (success atau fail)
  if [ "${exit_code}" -eq 0 ]; then
    hc_ping "" "${truncated_log}"
  else
    hc_ping "/fail" "Exit code: ${exit_code}\nDuration: ${duration}s\n\n${truncated_log}"
  fi

  rm -f "${tmp_log}"
  return "${exit_code}"
}

main

# /etc/logrotate.d/cron-jobs — rotate log harian
/var/log/cron_jobs/*.log {
  daily
  rotate 14
  compress
  delaycompress
  missingok
  notifempty
  copytruncate
}

Pemakaian

# /etc/cron.d/backup_db
# Pakai wrapper untuk job backup harian
# UUID dari healthchecks.io dashboard
0 2 * * * postgres /usr/local/bin/run_with_healthcheck.sh \
  c5e3f7a1-b2d4-4e8f-9a1b-2c3d4e5f6a7b \
  /opt/backup/pg_backup_nightly.sh

# Job lain: sync produk dari supplier setiap 6 jam
0 */6 * * * www-data /usr/local/bin/run_with_healthcheck.sh \
  9f8a7b6c-5d4e-3f2a-1b0c-9d8e7f6a5b4c \
  /opt/sync/produk_supplier.sh

# Manual test wrapper
/usr/local/bin/run_with_healthcheck.sh c5e3f7a1-... echo "Hello dari cron"

# Cek log
tail -20 /var/log/cron_jobs/c5e3f7a1-*.log

# Cek di dashboard healthchecks.io
# https://healthchecks.io/checks/c5e3f7a1-b2d4-4e8f-9a1b-2c3d4e5f6a7b/details/

# healthchecks.io schedule config (set lewat UI atau API)
schedule: "0 2 * * *"     # Cron expression
grace_period: 1h          # Toleransi delay
notification_channels:
  - slack: "#alerts-ops"
  - email: "ops@perusahaan.id"
  - webhook: "https://incident.io/webhook/..."

Kapan dipakai

• Backup harian database — kritikal kalau diam-diam gagal.
• Sync data dari supplier / external API.
• Renewal SSL certificate via Let’s Encrypt.
• Cleanup job (rotate log, prune Docker, expire session).
• Report generator (PDF laporan keuangan, dashboard cache warm).

Catatan

• set -euo pipefail di wrapper — kalau curl gagal, ada visibility. Tanpa pipefail, error di tengah pipe bisa hilang.
• mktemp + tee capture output sambil tetap mengalir ke stdout. Berguna untuk debugging real-time + log eksternal.
• Truncate log 100KB — healthchecks.io batasi body 100KB per ping. Tail bagian akhir biasanya yang ada error.
• Don’t fail job karena monitor fail — kalau healthchecks.io down, jangan bikin backup job ikut fail. Network error di ping di-suppress.
• UUID per job — jangan reuse UUID. Setiap cron job punya endpoint dan schedule sendiri.
• Grace period > job duration — kalau backup biasanya 30 menit, grace 1 jam aman.

Healthchecks.io free tier 20 check cukup buat startup kecil. Untuk enterprise, self-host versi open source (Docker image official) di cluster monitoring kamu sendiri.