Promtail + Loki log shipping setup

ELK keren tapi butuh CPU/RAM 10x lebih banyak dari Loki untuk volume sama. Untuk Indonesia startup yang hemat infra, Loki + Promtail combo lebih realistic. Snippet ini setup Promtail baca log Docker container + journald systemd, push ke Loki, dengan label discipline yang gak meledak.

Kode

# /etc/promtail/promtail.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: http://loki.observability.svc.cluster.local:3100/loki/api/v1/push
    tenant_id: tokopedia-prod

# Limit untuk mencegah Promtail OOM
limits_config:
  readline_rate: 10000
  readline_burst: 20000

scrape_configs:
  # ==========================================
  # Docker container — auto-discover via label
  # ==========================================
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 30s
        filters:
          - name: label
            values: ["promtail.enable=true"]

    relabel_configs:
      # Ambil nama container sebagai 'container' label
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.+)'
        target_label: 'container'

      # Ambil service name dari label compose
      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
        target_label: 'service'

      # Ambil env (production/staging) dari label
      - source_labels: ['__meta_docker_container_label_env']
        target_label: 'env'

      # IMPORTANT: drop label yang high cardinality
      - regex: '__meta_docker_container_id'
        action: labeldrop

    pipeline_stages:
      # Parse JSON log dari Pino / structured logger
      - json:
          expressions:
            level: level
            msg: msg
            trace_id: trace_id

      # Promote 'level' jadi label (low cardinality: info/warn/error)
      - labels:
          level:

      # Output: format output ke "level=info msg=..." untuk grep
      - output:
          source: msg

  # ==========================================
  # Systemd journald — system service log
  # ==========================================
  - job_name: journal
    journal:
      max_age: 12h
      path: /var/log/journal
      labels:
        job: systemd

    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'
      - source_labels: ['__journal__hostname']
        target_label: 'host'
      # Drop unit yang gak penting biar gak noise
      - source_labels: ['unit']
        regex: '(systemd-tmpfiles-setup|cron|networkd)\.service'
        action: drop

  # ==========================================
  # Nginx access log — parse jadi label efisien
  # ==========================================
  - job_name: nginx_access
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx
          __path__: /var/log/nginx/access.log

    pipeline_stages:
      - regex:
          expression: '^(?P<remote>\S+) - (?P<user>\S+) \[(?P<timestamp>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) [^"]+" (?P<status>\d+) (?P<size>\d+)'

      # Method jadi label (8 value max)
      - labels:
          method:
          status:

      - timestamp:
          source: timestamp
          format: "02/Jan/2006:15:04:05 -0700"

#!/usr/bin/env bash
# install_promtail.sh — install Promtail di Ubuntu / Debian

set -euo pipefail

readonly VERSION="${PROMTAIL_VERSION:-2.9.7}"
readonly ARCH="$(dpkg --print-architecture)"

# Download binary
cd /tmp
curl -fsSL "https://github.com/grafana/loki/releases/download/v${VERSION}/promtail-linux-${ARCH}.zip" -o promtail.zip
unzip -o promtail.zip
sudo mv "promtail-linux-${ARCH}" /usr/local/bin/promtail
sudo chmod +x /usr/local/bin/promtail

# Permission: butuh akses ke Docker socket dan journald
sudo usermod -a -G docker promtail 2>/dev/null || true
sudo usermod -a -G systemd-journal promtail 2>/dev/null || true

# Direktori state
sudo mkdir -p /var/lib/promtail
sudo chown promtail:promtail /var/lib/promtail

# Systemd unit
sudo tee /etc/systemd/system/promtail.service > /dev/null <<'EOF'
[Unit]
Description=Promtail log shipper
After=network-online.target

[Service]
Type=simple
User=promtail
Group=promtail
ExecStart=/usr/local/bin/promtail -config.file=/etc/promtail/promtail.yaml
Restart=on-failure
RestartSec=10s

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now promtail

echo "Promtail running. Cek status:"
sudo systemctl status promtail --no-pager

Pemakaian

# Validasi config sebelum apply
promtail -config.file=/etc/promtail/promtail.yaml -dry-run

# Cek log discovery — Docker container apa yang ke-scrape
curl -s http://localhost:9080/api/v1/targets | jq '.data.active[] | {labels, lastError}'

# Tail Promtail sendiri (kalau ada error config)
sudo journalctl -u promtail -f

# docker-compose.yml — label container supaya ke-scrape Promtail
services:
  api-tokopedia:
    image: ghcr.io/tokopedia/api:v2
    labels:
      promtail.enable: "true"
      env: "production"
    logging:
      driver: json-file
      options:
        max-size: "100m"
        max-file: "5"

# Query LogQL di Grafana Explore
# 1. Cari error semua API service
{env="production", level="error"} |~ "panic|fatal"

# 2. Hitung error rate per service
sum by (service) (rate({env="production", level="error"}[5m]))

# 3. Cari log specific trace
{env="production"} |~ "trace_id=5f3b8a2c1d9e4f6a7b8c9d0e"

Kapan dipakai

• Replace ELK / Splunk untuk cost reduction.
• Multi-tenant logging cluster (per project, per env).
• Pair dengan Grafana untuk single pane of glass (log + metric + trace).
• Kubernetes cluster yang sudah deploy Promtail DaemonSet.

Catatan

• Label cardinality — total unique combinasi label hindari di atas 100rb per stream. Trace_id, request_id, user_id JANGAN jadi label. Taruh di body log.
• labeldrop untuk drop high cardinality label yang Docker auto-attach (container_id 64-char hex).
• Pipeline stages order matter: json → labels → timestamp. Salah urutan, parse gagal.
• clients[].tenant_id untuk multi-tenant Loki. Tanpa ini, log nyampur di default tenant.
• positions file persist offset baca. Jangan hapus saat troubleshoot — bisa re-ship semua log dari awal.
• Limit readline_rate mencegah Promtail consume CPU 100% kalau ada app spam log.

Loki bukan untuk full-text search dokumen. Body log boleh besar, tapi query LogQL pakai filter expression linear scan. Untuk search free-text dengan throughput tinggi, tetap pakai Elasticsearch.