Designing a Self-Healing Payment Pipeline: An Event-Driven Go & Kafka Proof of Concept

By Michael Mifsud

Asynchronous vs. Synchronous Payment Processing

Traditional monolithic payment processing often relies on synchronous service-to-service communication. Under a synchronous model, Service A (e.g., a checkout UI) directly calls Service B (e.g., a ledger or clearing system) over HTTP or gRPC. If Service B is slow, undergoing maintenance, or suffering from network volatility, the entire client experience blocks, resulting in connection timeouts, transaction drift, or double-charge risks.

An Event-Driven Architecture (EDA) solves these bottlenecks by decoupling the write path from the read path. Instead of calling downstream systems directly, the entry point publishes an event to a durable message broker and immediately acknowledges receipt to the user. Downstream workers process the event asynchronously at their own speed.

This article dissects Dojo Payment POC, a distributed proof of concept built with Go, Apache Kafka, and Docker. This project acts as an architectural blueprint for a resilient payment ingestion and processing engine designed around concurrency, infrastructure self-healing, and shift-left test safety.

Architectural Blueprint

The system decouples responsibilities between an ingestion server, a durable Kafka event broker, a background transaction consumer, and a real-time visualization frontend.

Consumer ServiceKafka (Broker)Producer ServiceUser (Browser)Consumer ServiceKafka (Broker)Producer ServiceUser (Browser)loop[Async Processing]loop[Polling]POST /pay {"amount": 50.00}Publish to "transactions"200 OK (Ack)Consume MessageJSON PayloadProcess & Save to MemoryGET /paymentsReturn List [ {50.00, PROCESSED} ]

The system operates across three separate, containerized layers:

1. Non-Blocking Ingestion: The Producer Service

To handle massive concurrent transaction volume, the ingestion gate must remain as lightweight and high-throughput as possible. In the Go-based Producer, this is accomplished by offloading heavy processing from the HTTP request-response thread.

The producer exposes a single POST /pay REST endpoint. When a payment event arrives, the handler decodes the payload, serializes it to a compact JSON string, and pushes it asynchronously to the transactions topic using the IBM/sarama SyncProducer client.

Reusable Interface Design for Testability

To ensure the core business logic remains testable without requiring a live, external Kafka broker, the Producer decouples publishing actions behind a BrokerInterface:

type BrokerInterface interface {
	Publish(topic string, message []byte) error
}

type PaymentEvent struct {
	Amount   float64 `json:"amount"`
	Currency string  `json:"currency"`
	Status   string  `json:"status"`
}

type KafkaBroker struct {
	producer sarama.SyncProducer
}

func (k *KafkaBroker) Publish(topic string, message []byte) error {
	msg := &sarama.ProducerMessage{
		Topic: topic,
		Value: sarama.ByteEncoder(message),
	}
	_, _, err := k.producer.SendMessage(msg)
	return err
}

By abstracting broker actions behind the interface, unit tests can run in complete isolation by supplying a MockBroker that records calls without establishing real network sockets.

Once verified, the Producer binds the real implementation globally:

func SendPayment(broker BrokerInterface, amount float64, currency string) error {
	payment := PaymentEvent{
		Amount:   amount,
		Currency: currency,
		Status:   "PENDING",
	}

	bytes, err := json.Marshal(payment)
	if err != nil {
		return err
	}

	return broker.Publish("transactions", bytes)
}

2. Asynchronous Consumption: The Consumer Service

Once published to Kafka, events sit durably on disk until consumed. The Go Consumer service acts as a background processing engine.

Because the system is asynchronous, the consumer operates two major subsystems concurrently using Go’s built-in scheduler and concurrency primitives (goroutines):

  1. The Background Listener: An infinite event loop subscribing to partition 0 of the transactions topic, reading raw byte buffers, and marshaling them into standard schemas.
  2. The Read-Path API: An isolated HTTP server listening on :8081 to serve the transaction ledger history to the frontend dashboard.

Thread-Safe In-Memory State

To prevent race conditions between the concurrent background event listener writing new transactions to the database and the HTTP thread reading from it, the database ledger uses Go’s synchronization lockers (sync.RWMutex).

var (
	paymentHistory []PaymentEvent
	mutex          sync.RWMutex
)

// In the background Kafka consumption goroutine
for {
	select {
	case msg := <-partitionConsumer.Messages():
		event, err := HandleMessage(msg.Value)
		if err == nil {
			mutex.Lock()
			event.Status = "PROCESSED"
			paymentHistory = append(paymentHistory, *event)
			mutex.Unlock()
			fmt.Printf("💰 Processed: %.2f %s\n", event.Amount, event.Currency)
		}
	}
}

// In the HTTP thread servicing dashboard polling
http.HandleFunc("/payments", func(w http.ResponseWriter, r *http.Request) {
	enableCors(&w)
	mutex.RLock()
	defer mutex.RUnlock()
	json.NewEncoder(w).Encode(paymentHistory)
})

By utilizing a Reader/Writer Mutex, multiple API requests can read from the ledger simultaneously (mutex.RLock) without locking each other out, while ensuring only one background consumer can modify the ledger at a time (mutex.Lock).

3. Resilience and Self-Healing Infrastructure

A key requirement of modern fintech infrastructure is autonomous operational recovery. If a network blip occurs or the message broker is temporarily restarted, the services should not panic and die permanently.

The Dojo POC ensures self-healing behavior across its Docker environment through two mechanisms:

KRaft-based Zero-ZooKeeper Architecture

Traditional Kafka deployments required managing a separate Apache ZooKeeper cluster for consensus and partition metadata management, which increased resource footprint and operational failure points.

This infrastructure uses modern KRaft mode, combining controller and broker coordination within a single, unified Kafka service:

kafka:
  image: apache/kafka:latest
  ports:
    - "9092:9092"
  environment:
    KAFKA_NODE_ID: 1
    KAFKA_PROCESS_ROLES: 'broker,controller'
    KAFKA_LISTENERS: 'PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093'
    KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka:9092'
    KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:9093'
    KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'

Automatic Restart Policies

Both Go binaries rely on Docker Compose’s orchestrator to manage lifecycle recovery. Under a dependency start race condition (such as when a Go service boots faster than Kafka is fully initialized), the service will panic and fail.

By registering restart: always on all application services, the Docker daemon automatically handles container restarts with incremental backoffs, allowing services to self-heal once the broker is healthy:

producer:
  build:
    context: .
    dockerfile: Dockerfile
  ports:
    - "8090:8080"
  depends_on:
    - kafka
  restart: always

consumer:
  build:
    context: .
    dockerfile: Dockerfile
  ports:
    - "8081:8081"
  depends_on:
    - kafka
  restart: always

4. “Shift-Left” Integration Verification

To prevent regressions in distributed pipelines, integration testing should not wait for staging or QA deployments. It must happen at compile time inside the automated CI pipeline.

This codebase features a complete End-to-End Integration Verification Harness (run_integration_test.sh) run automatically within every GitLab CI/CD execution.

The Integration Test Stage

The GitLab runner utilizes a Docker-in-Docker (dind) service to build container images and run the full Docker Compose stack. It then executes the integration script, which acts as a verification loop:

# Start the full container network in detached mode
docker compose up -d --build

# Wait loop: Check logs for successful processing signatures
echo "⏳ Waiting for payment transaction..."
for i in $(seq 1 30); do
    if docker compose logs consumer | grep -q "💰"; then
        echo "✅ SUCCESS: Payment received by Consumer!"
        docker compose down
        exit 0
    fi
    sleep 2
done

echo "❌ FAILURE: Timed out waiting for payment."
docker compose logs
docker compose down
exit 1

The script polls the container logs for the specific processing signature emoji (💰) output by the consumer daemon upon completing a transaction. If the signature is not detected within 60 seconds, the script shuts down the stack and exits with a non-zero exit code (1), immediately failing the CI/CD pipeline.

5. Production Readiness Roadmap

As a proof of concept, this system successfully demonstrates the mechanics of distributed EDA pipelines. However, transitioning this system to a high-volume fintech environment requires resolving specific production constraints: