Event Sourcing + Saga Pattern: Microservices Payment

Dua tahun lalu, klien payment processor Indonesia (B2B settlement untuk e-commerce + marketplace, ~480k transaksi/hari, total value Rp 12 miliar/hari) butuh re-arsitektur payment flow yang sebelumnya monolith dengan distributed transaction (XA + 2PC) yang prone to deadlock.

Pilih event sourcing + saga pattern. Setelah 18 bulan production, share angka, decisions, dan lessons.

Konteks payment flow

Payment flow lengkap untuk single transaction:

1. Order accepted di e-commerce → POST ke payment service.
2. Risk check: fraud scoring (sync, <200ms timeout).
3. Reserve funds: hold di account customer.
4. Settle: transfer ke merchant account.
5. Notify: customer + merchant + e-commerce platform.
6. Update ledger: double-entry bookkeeping (mandatory audit).

Failure mode mana saja bisa terjadi. Sebelumnya pakai XA transaction yang ENVELOPE 1-6, deadlock saat partner bank slow, throughput cap ~80 TPS.

Target arsitektur baru:

• Throughput 1.000+ TPS sustained
• Audit trail lengkap (BI compliance)
• Failure recovery automated
• Cross-service consistency eventual but verifiable

Event sourcing architecture

Core idea: instead of storing current state, store all events. Current state = projection from events.

Event store: Postgres dengan tabel append-only:

CREATE TABLE event_store (
    event_id UUID PRIMARY KEY,
    aggregate_id UUID NOT NULL,
    aggregate_type TEXT NOT NULL,
    event_type TEXT NOT NULL,
    event_version INT NOT NULL,
    event_data JSONB NOT NULL,
    metadata JSONB NOT NULL,
    sequence_number BIGSERIAL NOT NULL,
    occurred_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_aggregate ON event_store(aggregate_id, sequence_number);
CREATE INDEX idx_type_time ON event_store(event_type, occurred_at);

Event per transaction (example):

[
  {"event_type": "PaymentInitiated", "data": {"amount": 250000, "merchant": "m_123", "customer": "c_456"}},
  {"event_type": "RiskCheckPassed", "data": {"score": 12, "threshold": 50}},
  {"event_type": "FundsReserved", "data": {"reference": "hold_789"}},
  {"event_type": "Settlement Initiated", "data": {"settlement_id": "s_001"}},
  {"event_type": "SettlementCompleted", "data": {"bank_ref": "BCA-202607180012345"}},
  {"event_type": "NotificationSent", "data": {"channels": ["sms", "email"]}},
  {"event_type": "LedgerEntryRecorded", "data": {"entry_id": "le_004"}}
]

Replay events di-order = reconstruct payment state. Audit query: “tunjukkan history transaksi T” = SELECT event WHERE aggregate_id = T ORDER BY sequence_number.

Projection: current state untuk read

Event store too slow untuk lookup current state real-time. Projection = denormalized read model.

CREATE TABLE payment_current_state (
    payment_id UUID PRIMARY KEY,
    status TEXT NOT NULL,
    amount NUMERIC NOT NULL,
    merchant_id UUID NOT NULL,
    customer_id UUID NOT NULL,
    last_event_sequence BIGINT NOT NULL,
    updated_at TIMESTAMPTZ NOT NULL
);

Projection worker:

@Component
public class PaymentProjector {
    
    @EventHandler  // Spring custom event handler dari event_store
    public void on(PaymentInitiated event) {
        jdbc.update("""
            INSERT INTO payment_current_state 
            (payment_id, status, amount, merchant_id, customer_id, last_event_sequence, updated_at)
            VALUES (?, 'INITIATED', ?, ?, ?, ?, now())
            """, event.aggregateId(), event.amount(), event.merchant(), event.customer(), event.sequence());
    }
    
    @EventHandler
    public void on(SettlementCompleted event) {
        jdbc.update("""
            UPDATE payment_current_state 
            SET status = 'COMPLETED', last_event_sequence = ?, updated_at = now()
            WHERE payment_id = ? AND last_event_sequence < ?
            """, event.sequence(), event.aggregateId(), event.sequence());
    }
}

Idempotency via last_event_sequence: kalau event reprocessed, skip kalau sequence sama atau lebih kecil.

Saga: orchestration untuk payment flow

Saga = workflow yang manage multi-step transaction dengan compensating action.

Saya pilih orchestration (vs choreography) untuk payment flow karena:

• Visible ke business stakeholder (single source of truth flow).
• Easier debug (single coordinator yang track state).
• Compensating logic centralized.

Implementation pakai workflow engine (Temporal.io self-host):

@WorkflowInterface
public interface PaymentSagaWorkflow {
    @WorkflowMethod
    PaymentResult execute(PaymentRequest request);
}

public class PaymentSagaWorkflowImpl implements PaymentSagaWorkflow {
    
    private final RiskActivity risk = Workflow.newActivityStub(RiskActivity.class,
        ActivityOptions.newBuilder()
            .setStartToCloseTimeout(Duration.ofSeconds(5))
            .setRetryOptions(RetryOptions.newBuilder().setMaximumAttempts(2).build())
            .build());
    
    private final FundsActivity funds = Workflow.newActivityStub(FundsActivity.class, ...);
    private final SettlementActivity settlement = Workflow.newActivityStub(SettlementActivity.class, ...);
    private final NotificationActivity notif = Workflow.newActivityStub(NotificationActivity.class, ...);
    private final LedgerActivity ledger = Workflow.newActivityStub(LedgerActivity.class, ...);
    
    @Override
    public PaymentResult execute(PaymentRequest req) {
        // Step 1: Risk check
        RiskResult riskResult = risk.check(req);
        if (riskResult.score() > 50) {
            return PaymentResult.rejected("HIGH_RISK");
        }
        
        // Step 2: Reserve funds
        ReservationResult reservation = funds.reserve(req.customerId(), req.amount());
        
        try {
            // Step 3: Settlement
            SettlementResult settle = settlement.execute(req, reservation);
            
            // Step 4: Ledger entry (CRITICAL: must succeed, retry indefinitely if needed)
            ledger.recordWithRetry(settle);
            
            // Step 5: Notification (best-effort, async)
            Async.procedure(notif::send, req.customerId(), settle);
            
            return PaymentResult.success(settle.bankRef());
            
        } catch (SettlementFailedException e) {
            // Compensating action: release funds
            funds.release(reservation.holdId());
            ledger.recordReversal(reservation, e.reason());
            return PaymentResult.failed(e.reason());
        }
    }
}

Temporal.io guarantees:

• Workflow execution durable (state in DB).
• Activity retries automatic dengan backoff.
• Compensating action explicit.
• Replay untuk recovery (kalau service restart).

Choreography untuk loose-coupled flow

Notification + analytics flow tidak butuh orchestration. Pakai Kafka choreography:

PaymentService → publish "PaymentCompleted" event
    ↓
[NotificationService]   [AnalyticsService]   [FraudRetrospectiveService]
   (consume, send SMS)   (update dashboard)   (update fraud model)

Each consumer independent. Failure di satu tidak block lain.

Yang dipilih per use case

Flow	Pattern
Payment processing	Orchestration (Temporal)
Refund processing	Orchestration
Settlement reconciliation	Orchestration
Notification dispatch	Choreography (Kafka)
Analytics & reporting	Choreography
Audit log shipping	Choreography
Fraud model retraining	Choreography

Mixed approach. Critical money-movement = orchestration (visibility + control). Side effect = choreography (decouple).

Hasil 18 bulan

Metric	Sebelum (monolith XA)	Setelah (ES + Saga)
Throughput peak	80 TPS	1.240 TPS
Latency p99	4.2s	1.8s
Saga completion rate	n/a	99.97%
Distributed deadlock	2-3/hari	0
Audit trail completeness	partial (log-based)	100% (event-sourced)
DR replay time (full event store)	n/a	38 menit
Engineering team	6	9
Operational complexity	medium	high

Throughput naik 15x karena: no XA transaction overhead, parallelizable saga steps, async event distribution.

Latency naik (lebih bagus) karena: pipelining step yang bisa paralel, no blocking lock cross-service.

Yang break

1. Event store schema migration

Event versioning critical. Event v1 PaymentInitiated punya field amount integer (sen). Bug awal: di v2 kami change ke BigDecimal. Replay event v1 break karena projection expect BigDecimal.

Fix: upcast saat read.

public class EventUpcastingService {
    
    public Event upcast(Event event) {
        if ("PaymentInitiated".equals(event.type()) && event.version() == 1) {
            JsonNode data = event.data();
            // v1: amount as integer cents, v2: amount as decimal rupiah
            BigDecimal amountRupiah = new BigDecimal(data.get("amount").asLong())
                .divide(BigDecimal.valueOf(100));
            ((ObjectNode) data).put("amount", amountRupiah);
            return event.withVersion(2);
        }
        return event;
    }
}

Upcasting jadi mandatory: setiap event version, ada upcaster ke version terbaru.

2. Projection out-of-order

Multiple projection worker consume parallel dari event_store. Worker A faster than B → projection inconsistent saat read between checkpoints.

Fix:

• Single projection worker per aggregate type (single-threaded per type).
• Sequence_number monotonic check sebelum apply.
• Read after write: client tolerate eventual consistency atau read from event store langsung untuk strong consistency.

3. Saga timeout dilemma

Activity timeout 5 detik. Bank partner sometimes slow 8-12 detik. Activity timeout → saga compensate → tapi sebenarnya bank still process.

Result: customer charged twice (saga retry create new bank request).

Fix:

• Idempotency key per request to bank.
• Bank-side request deduplication (bank-specific, kami invest 6 minggu integration dengan 3 bank partner).
• Saga timeout naik ke 30 detik untuk activity bank call, dengan circuit breaker untuk protect.

After fix: double-charge incident dari 4-6/bulan ke 0.

4. Event store size growth

18 bulan running, event_store table 480GB. Postgres maintenance (vacuum, index rebuild) jadi expensive.

Fix:

• Partition by month: event_store_2026_01, event_store_2026_02, …
• Cold partition (> 18 bulan) archive ke object storage (S3/MinIO).
• Active query window: 18 bulan, archive query via separate API.

Active partition size jadi manageable ~50GB per 6 bulan.

5. Compensating action incomplete

Bulan ke-4 incident: settlement gagal di step 3 setelah reserve funds, compensating “release funds” gagal karena bug. Funds stuck in hold selama 4 jam sebelum manual cleanup.

Fix:

• Compensating action critical: punya retry indefinite + alert kalau gagal > 3x.
• Manual override runbook (DBA can release manually via SQL with audit trail).
• Periodic reconciliation: scan held funds > 1 jam tanpa active saga, alert.

6. Replay storm setelah outage

Setelah outage Temporal cluster 8 menit, semua saga restart concurrent. ~2.400 saga replay simultan, overload downstream service.

Fix:

• Temporal rate limit replay (default 1000/sec, kami throttle ke 200/sec).
• Downstream service: increase rate limit acceptance untuk known replay scenario.
• Circuit breaker di saga: kalau downstream error rate > 30%, slow down.

Cost & ops

Infrastructure cost (Postgres event store + projection + Temporal cluster):

• Postgres event store: Hetzner CCX43 × 3 (1 primary + 2 replica) = €270/month
• Temporal cluster: 3 node CCX23 = €120/month
• Kafka cluster (untuk choreography): MSK 3-broker = $1.400/month ≈ Rp 22 juta
• Total: ~Rp 28 juta/month

Sebelumnya monolith XA: Rp 18 juta/month. Cost naik 56%, tapi throughput 15x. Cost per transaction turun signifikan.

Ops bandwidth: 1.5 FTE SRE dedicated (event store backup, Temporal upgrade, projection lag monitoring).

Verdict

Event sourcing + saga pattern itu power tool, bukan default architecture. Worth untuk:

• Audit trail mandatory (financial, healthcare, fintech).
• Cross-service consistency dengan compensating semantics.
• Throughput tinggi yang XA tidak bisa scale.
• Tim engineering > 8 dengan domain expert.

Tidak worth untuk:

• CRUD biasa dengan audit log file-based cukup.
• Tim < 6 engineer.
• Domain yang flow-nya sederhana 2-3 step.

Bukan magic. Event versioning, replay storm, projection lag — banyak failure mode unik yang harus dihandle. Tapi untuk payment processor Indonesia dengan compliance BI dan throughput target: arsitektur ini yang scale.

Lihat juga Kafka vs NATS Jetstream untuk konteks broker pilihan di choreography, dan Postgres sharding vertikal untuk konteks data layer scaling.