Observability Stack: Grafana + Loki + Tempo 6 Bulan

Enam bulan lalu, klien enterprise yang saya advise (logistics + e-commerce, ~3.000 RPS aggregate, 64 microservice di K8s) decide migrate observability dari mixed setup (Datadog APM + Splunk log + Prometheus metric) ke unified Grafana stack: Grafana + Loki + Tempo + Mimir. Cost target: < $1k/bulan dari $6.8k sebelumnya.

Setelah 6 bulan production stable, share angka, architecture, dan operational lessons.

Konteks workload

• App: logistics + e-commerce, marketplace + fulfillment.
• Skala: 64 microservice di EKS Kubernetes, 3.000 RPS aggregate peak.
• Tim engineering: 28 engineer + 4 SRE.
• Compliance: ISO 27001, PCI-DSS untuk e-commerce side.
• Existing observability cost: Datadog $4.200/mo + Splunk Cloud $2.100/mo + Prometheus self-host $500/mo = $6.800/mo total.

Pain point:

1. Cost growth: 28% YoY, tidak proportional dengan traffic growth.
2. Tool fragmentation: 3 UI, 3 query language (DD APM, SPL, PromQL), 3 alerting config.
3. Cross-correlation manual: trace di DD → log di Splunk → metric di Grafana = 3 tab + manual time alignment.
4. Vendor lock-in: kontrak DD 1 tahun, susah migrate piece-by-piece.

Strategi: unified Grafana stack

LGTM stack (Loki, Grafana, Tempo, Mimir) — semua complement, single UI, query language sama-sama LogQL/TempoQL/PromQL tapi seamlessly correlate di Grafana.

Architecture:

[64 microservices]
   ↓ stdout/stderr      ↓ OTLP              ↓ /metrics
[Promtail per-node]   [OTel Collector]   [Prometheus Agent]
   ↓                    ↓                   ↓
[Loki distributor]   [Tempo distributor] [Mimir distributor]
   ↓                    ↓                   ↓
[Loki ingester]      [Tempo ingester]    [Mimir ingester]
   ↓                    ↓                   ↓
[Object storage MinIO on-prem (data residency BI compliance)]
   ↓
[Grafana 11 - unified UI]

Object storage: MinIO 6-node, ~256TB usable, hosted di colo Cyber Cibubur (data residency mandatory).

Loki: log aggregation

Promtail DaemonSet collect from container stdout. Setup:

positions:
  filename: /run/promtail/positions.yaml

clients:
  - url: http://loki-distributor:3100/loki/api/v1/push
    batchwait: 1s
    batchsize: 1048576

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    pipeline_stages:
      - cri: {}
      - json:
          expressions:
            level: level
            msg: msg
            trace_id: trace_id
      - labels:
          level:
      - timestamp:
          source: timestamp
          format: RFC3339Nano

trace_id extracted as field (not label) — high cardinality, label akan explode.

Label discipline

Critical: Loki label cardinality affects performance directly.

Allowed labels (low cardinality):

• namespace (~10 values)
• app (~64 values)
• container_name (~80 values)
• pod (~340 values, watch carefully)
• level (4 values: debug, info, warn, error)
• cluster (3 values)

Forbidden labels:

• trace_id, request_id, user_id, tenant_id (high cardinality → use fields)

Cost & sizing

Log volume: ~8 TB/month ingested, ~2.4 TB/month stored (zstd compression ratio ~3.3x).

Loki cluster di EKS:

• 3 distributor (CPU-bound)
• 6 ingester (memory + write disk)
• 3 querier (CPU-bound saat query)
• 2 query-frontend
• 2 compactor

Total Loki cost: ~$280/month (EC2) + $48/month (MinIO storage proportional).

Tempo: distributed tracing

OTel Collector Agent (DaemonSet) + Gateway (deployment). Lihat juga setup detail di OpenTelemetry microservices Jakarta SaaS.

Tail-based sampling agresif:

• Error trace: 100%
• Slow trace (> 800ms): 100%
• High-value business flow (payment, order): 100%
• Baseline probabilistic: 8%

Effective: ~14% trace persisted. ~5.8 TB/month raw, ~1.4 TB/month stored.

Tempo cluster:

• 3 distributor
• 6 ingester
• 3 querier
• 2 compactor
• 2 metrics-generator (untuk RED metric dari trace)

Cost Tempo: ~$240/month (EC2) + $28/month (MinIO).

Mimir: long-term Prometheus storage

Existing Prometheus self-host, tapi retention cuma 15 hari (disk-limited). Mimir = scalable, long-term Prometheus-compatible.

Setup: Prometheus Agent mode (no local TSDB), remote_write ke Mimir.

remote_write:
  - url: http://mimir-gateway:8080/api/v1/push
    write_relabel_configs:
      - source_labels: [__name__]
        regex: 'go_.*'
        action: drop  # drop noisy Go runtime metric

Mimir retention 13 bulan (compliance audit). Storage: ~380 GB/month, compressed.

Mimir cluster:

• 3 distributor
• 6 ingester
• 3 querier
• 3 store-gateway
• 2 compactor

Cost Mimir: ~$180/month (EC2) + $14/month (MinIO).

Grafana: unified UI

Grafana 11.4 self-host:

• Dashboard datasource: Loki, Tempo, Mimir.
• Alerting: Grafana managed alerting (unified, replace Alertmanager standalone).
• Authentication: SSO via Google Workspace + RBAC per team.

Datasource correlation:

datasources:
  - name: Tempo
    type: tempo
    url: http://tempo-query-frontend:3100
    jsonData:
      tracesToLogsV2:
        datasourceUid: 'loki'
        filterByTraceID: true
        customQuery: true
        query: '{cluster="prod"} |= "${__trace.traceId}"'
      tracesToMetrics:
        datasourceUid: 'mimir'
        queries:
          - name: 'Service RPS'
            query: 'sum(rate(http_requests_total{service="${__trace.service}"}[5m]))'
  
  - name: Loki
    type: loki
    jsonData:
      derivedFields:
        - name: trace_id
          matcherRegex: 'trace_id=(\w+)'
          url: '$${__value.raw}'
          datasourceUid: 'tempo'

Workflow incident:

1. Alert fire → buka di Grafana.
2. Klik “View related traces” → buka Tempo trace.
3. Klik trace span → “View logs in time range” → Loki dengan trace_id filter.
4. Klik metric → drill down via PromQL di Mimir.

Single UI, zero context switch. Inilah benefit terbesar unified stack.

Hasil 6 bulan

Metric	Sebelum (DD + Splunk + Prom)	Setelah (LGTM)
Cost monthly	$6.800	$790
MTTR (incident)	30 min avg	12 min avg
Trace coverage	64% (DD APM)	96% (OTel)
Log retention	30 hari (Splunk)	60 hari (Loki)
Metric retention	90 hari	395 hari (Mimir)
Vendor lock-in	high	low
UI used by team	3 (DD, Splunk, Grafana)	1 (Grafana)
Alert fatigue (false positive %)	32%	14%

Cost saving $6k/month = $72k/year. ROI engineering investment (sekitar 6 bulan × 1.5 FTE = ~$120k) tercapai di bulan ke-20.

Yang break

1. Loki ingester OOM saat traffic spike

Bulan ke-2, Lebaran traffic spike 4x baseline. Loki ingester memory pressure, OOM. Log ingestion lag 12 menit, beberapa log lost.

Fix:

• HPA Loki ingester berdasar memory utilization (target 70%, max replica 18).
• Persistent volume per ingester (sebelumnya ephemeral, kehilangan WAL saat OOM).
• Pre-scale untuk known traffic event (Lebaran, year-end sale).

2. Tempo trace gap saat ingester restart

Tempo ingester rolling restart untuk upgrade. Selama restart window, trace yang in-flight di-flushed tapi not yet compacted = di-lost.

Fix:

• Ingester graceful shutdown lebih lama (default 30s → 120s).
• Pre-stop hook untuk drain trace.
• Schedule upgrade di low-traffic window (Sabtu pagi 06:00-08:00).

3. Mimir cardinality explosion

Salah satu service tambah metric label http_route yang tidak normalized — /users/12345, /users/12346, dst. 4 juta unique series dalam 2 hari. Mimir ingester struggle.

Fix:

• Audit metric label di SDK level — wajib normalize path.
• Alert Mimir cortex_ingester_memory_series growth rate.
• Tracker tool “top-10 high cardinality metric” di Grafana dashboard.

4. Grafana alert duplication

Migrasi dari Alertmanager standalone ke Grafana managed alerting. Konfigurasi overlap → alert duplicated, 2x notification per incident.

Fix: phased migration. Disable Alertmanager standalone setelah Grafana alert fully tested.

5. MinIO disk fill at 78%

Erasure coding EC:4+2 dengan 6-node, kalau disk fill > 80%, write performance degrade significantly. Hari ke-127, disk fill 78%.

Fix:

• Audit retention: drop log debug yang volume besar, retain hanya level=info ke atas.
• Tighten Tempo sampling (probabilistic dari 10% ke 8%).
• Pre-emptive expansion: add 2 node MinIO sebelum hit 80%.

6. Query timeout untuk long time range

Grafana query Loki untuk 30 hari time range → timeout (default 1 menit). Investigation incident yang span hari frustrating.

Fix:

• Split-by-interval di query (Loki split query per 1 jam chunk, parallel).
• Increase query_timeout ke 5 menit untuk investigation use case.
• Pre-materialized query result untuk common pattern via Grafana Scheduled Reports.

Operational ownership

LGTM stack butuh dedicated ownership. RACI matrix:

Component	Responsible	Accountable	Consulted	Informed
Loki	SRE-team	SRE-lead	Platform-lead	All eng
Tempo	SRE-team	SRE-lead	Platform-lead	All eng
Mimir	SRE-team	SRE-lead	Platform-lead	All eng
Grafana	SRE-team	Platform-lead	Eng-leads	All eng
MinIO storage	Infra-team	Platform-lead	SRE-lead	All eng

Without clear ownership, observability stack jadi orphan tool yang accumulate cruft.

Kapan saya tidak rekomendasi LGTM self-host

1. Tim < 8 engineer dengan 0 SRE: operational overhead Loki + Tempo + Mimir bukan trivial.
2. Traffic < 200 RPS aggregate: Datadog Starter cukup, saving self-host marginal.
3. Compliance audit lebih nyaman dengan vendor: SOC 2 vendor have pre-audited setup.
4. No on-prem requirement: kalau cloud OK, Grafana Cloud managed lebih simple ops.

Kapan LGTM self-host menang

1. Data residency Indonesia mandatory (BI, ISO 27001).
2. Tim observability dedicated (≥ 2 FTE).
3. Workload > 1k RPS aggregate (cost benefit material).
4. Multi-cluster / multi-region setup (vendor cost scale ugly).

Verdict

Grafana + Loki + Tempo + Mimir untuk enterprise Indonesia dengan compliance + cost concern: solusi yang work setelah 6 bulan tuning. $6k/month saving signifikan, plus zero vendor lock-in dan unified UI yang signifikan turunkan MTTR.

Tapi: bukan plug-and-play. Sizing capacity butuh empirical (mulai konservatif, scale up). Operational ownership clear adalah prasyarat. Tim observability butuh investment training.

Bukan magic. 6 bulan migration termasuk dual-stack period 3 bulan. Tapi 18 bulan steady-state setelah: payback engineering investment + free Anda dari vendor lock-in dynamic yang akan terus tekan budget.

Lihat juga Prometheus Grafana SaaS Jakarta setup untuk konteks SMB-scale dan OpenTelemetry microservices Jakarta untuk dalam tentang OTel layer.