Kafka and event-driven systems, engineered for correctness.
Topics, outbox patterns, idempotent consumers and replay — the event backbone under our own platforms, built by engineers who run it.
- Kafka
- Kafka Connect
- Debezium
- Schema Registry
- Kafka Streams
- Flink
Enterprise integration often fails quietly.
A system calls another, the call times out, one side has written and the other has not, and nobody notices until the month-end reconciliation. Kafka changes the shape of the problem: a producer writes a fact once to a durable, ordered log, and every consumer reads it at its own pace, from the position it last acknowledged. A slow system no longer stalls a fast one, an outage becomes a backlog instead of a loss, and a bug can be fixed by replaying the log rather than re-keying the data.
It is part of the stack under our own platforms. A points transaction in Loyalty & Retention, a rated sale in Sales Performance, a load status in Order & Supply Management — each is the kind of fact that several services need to react to, and an event log lets reporting read it without touching the system of record. The integration estates we build for clients use the same patterns.
When we don’t: a request that needs an answer now is an API, not an event. A team with two services and one database should not install a distributed log to talk to itself. And a simple task queue — emails, thumbnails — is better served by RabbitMQ or a cloud queue than by a Kafka cluster.
Six things we build on Kafka.
Event backbones between systems
Domain events with owned topics, agreed schemas and a registry, so ERP, CRM, billing and the platforms in between exchange facts instead of nightly files.
The outbox pattern by default
The event is written in the same PostgreSQL transaction as the business change and relayed to Kafka by Debezium or a relay — which removes the dual-write race that loses messages.
Change data capture from legacy stores
Debezium on Oracle, SQL Server, MySQL or PostgreSQL write-ahead logs turns a system you cannot change into an event source — the standard first move in a legacy re-engineering programme.
Idempotent consumers and stream processors
Consumers in Go, Java or Node.js with explicit offset commits, deduplication keys, dead-letter topics and back-pressure; Kafka Streams or Flink where aggregation over windows is the job.
Real-time pipelines for data and AI
Streams into the warehouse and feature stores for fraud, operations and personalisation use cases where yesterday’s batch is too late — with Data Engineering owning the models downstream.
Operating the cluster
Managed Kafka (MSK, Confluent Cloud, Event Hubs) or Strimzi on Kubernetes, with partition strategy, retention, quotas, monitoring and the replay tooling that makes the log useful in an incident.
When to choose it.
Choose Kafka when several systems need to react to the same facts, when the producer and consumers scale or fail independently, when you need the history — replay, audit, late-joining consumers — and when volume is high enough that per-message brokering becomes the bottleneck. It fits integration estates with more than a handful of systems, event-sourced or CQRS designs, and operational analytics that cannot wait for the nightly batch. It is also the honest answer to the question “how do we get data out of the mainframe without touching it”.
Look elsewhere when the interaction is a request that needs a synchronous response — that is REST or gRPC. Look elsewhere when you have a handful of background jobs and no ordering or replay requirement; a queue is simpler to run and to reason about. And be cautious when nobody in the organisation will own the cluster, the schemas and the consumer lag: an unowned Kafka is a very durable place to lose track of data.
One transaction, one event, no dual write.
begin;
insert into points_ledger (member_id, amount, reason, idempotency)
values ($1, $2, 'flight-earn', $3);
-- the event is part of the same commit as the business fact
insert into outbox (aggregate, aggregate_id, type, payload)
values ('member', $1, 'PointsAwarded', $4);
commit;
-- Debezium reads the outbox from the WAL and publishes to Kafka:
-- topic: loyalty.member.events key: member_id header: idempotency
-- consumers dedupe on the idempotency key and commit offsets after the write
+ create index on outbox (published_at) where published_at is null;An event backbone the way we hand it over.
What the event flows we hand over have.
- No dual writes. Every event that must not be lost leaves the producer through an outbox table or CDC. A service that writes to its database and then publishes to Kafka in a second step fails our review.
- Idempotent consumers keyed on a business identifier, with offsets committed only after the side effect is durable — so at-least-once delivery produces exactly-once outcomes.
- Schemas in a registry (Avro or Protobuf, JSON Schema where the ecosystem needs it), with compatibility rules enforced in CI and a written ownership record per topic.
- Partition keys chosen for ordering, not convenience. Events for one member, one load or one plan land on one partition; hot keys are looked for in load tests, not in production.
- Dead-letter topics with a replay procedure, retry budgets with exponential back-off, and alerts on dead-letter volume and consumer lag.
- Traces across the log. OpenTelemetry context propagated in headers, so one request can be followed from the API through three consumers to the warehouse.
- Retention and residency decided per topic, with personal data mapped for GDPR / KVKK, encrypted in transit and at rest, and ACLs per service principal — no shared credentials.
- Documented replay and disaster procedures — managed flows have a written, rehearsed way to reprocess a time range and to rebuild a consumer’s state from the log.
What usually ships alongside the topics.
- An event catalogue — topics, owners, schemas, keys, retention — kept in the repository next to the registry
- Outbox and CDC connectors configured as code, with the Debezium or relay deployment
- Consumer templates for your languages with idempotency, dead-lettering and tracing built in
- Terraform for the managed cluster or the Strimzi operator, quotas and ACLs
- Lag, throughput and dead-letter dashboards with alert policies exported as code
- A replay and incident runbook, rehearsed with your team in a game day
Frequently asked questions
Neighbouring stacks and the services that use them.
Have two systems that disagree about the same fact?
Tell us which systems and what volume. A Kafka engineer reads it and replies in writing with how we would approach it.