How to Build Real-Time Change Data Capture Pipelines

Most teams do not start with a data capture pipeline problem. They start with a product problem that quietly turns into a data problem.

A customer updates an address, your warehouse sees it six hours later, your fraud model scores against stale data, and your “real-time” dashboard turns out to mean “after the next batch job.” Change data capture, or CDC, fixes that by turning database writes into a stream of inserts, updates, and deletes that downstream systems can consume almost immediately. PostgreSQL does this from its write-ahead log through logical replication, Debezium turns those changes into structured events, Kafka gives you a durable log to fan them out, and stream processors or sinks move them into search indexes, caches, warehouses, and services.

We pulled this guide together from the systems people actually run in production, not from architecture-diagram theater. Martin Kleppmann, distributed systems researcher and author, has long argued that CDC is powerful because it lets applications subscribe to everything written to a database, which makes it useful for indexes, caches, recommendations, and replication. Gunnar Morling, Debezium maintainer, has pushed the outbox pattern because direct table CDC often leaks internal schema changes to consumers, while an outbox gives you a cleaner contract. Kafka documentation makes the other half of the point: delivery guarantees are a design choice, not a checkbox, and exactly-once needs specific producer, broker, and consumer behavior to hold up under failure. Put those together and the practical lesson is clear: the hard part is not getting changes out of a database; it is preserving ordering, contracts, and recovery semantics when the rest of the stack misbehaves.

Pick the right CDC shape before you pick tools

There are three common ways to build CDC pipelines, and they solve different problems.

PostgreSQL logical replication uses a publish-subscribe model, which is excellent when you want row-level changes from selected tables. Debezium’s PostgreSQL connector can take an initial snapshot and then continue streaming row changes, which makes it a practical bridge from an existing OLTP database into Kafka. Flink CDC adds end-to-end integration features like schema evolution and exactly-once semantics for cases where you need to process the stream, not just move it. Snowflake, by contrast, has Streams and Tasks for warehouse-side CDC workflows, which is great when your destination is the warehouse and not a broader event platform.

That choice matters more than most teams admit. If your only requirement is “keep one analytics table fresh,” a warehouse-native path is often enough. If you need one source database to update search, cache invalidation, fraud scoring, and three downstream services at once, you want a durable event log in the middle.

Build around the source of truth, not around ETL convenience

A real CDC pipeline starts at the transaction log, because polling tables is the fastest route to duplicate reads, missed deletes, and ugly load on the primary database. PostgreSQL’s logical replication is designed for fine-grained replication, and Debezium reads from that layer rather than repeatedly scanning business tables. That is why log-based CDC usually wins on freshness and source impact.

You also need to decide whether you are exposing raw table mutations or business events. Raw CDC is simple and honest, but it couples consumers to your schema. The outbox pattern adds a dedicated outbox table in the same local transaction as the business write, then Debezium captures the outbox row and publishes a cleaner event. Instead of shipping internal row changes to everyone, you publish a business-level message with a stable contract. That extra table feels annoying right up until your application team renames a column and five consumers break at once.

Here is the practical rule: use raw table CDC for internal analytics and replication where downstream systems can tolerate schema coupling. Use an outbox when you are publishing domain events to other services.

Design for ordering, replay, and failure on day one

This is where toy pipelines die.

Kafka’s delivery model can be at-most-once, at-least-once, or exactly-once, depending on how you configure producers, brokers, and consumers. Idempotent producers help prevent duplicate writes, transactions let you atomically write records across partitions and commit offsets, and consumers must use the right isolation behavior to avoid reading uncommitted transactional data. Exactly-once is not magic, it is coordinated plumbing.

For CDC specifically, you need four guarantees to be explicit:

1. Position tracking: store source offsets or WAL positions durably.
2. Partitioning strategy: key records so related entity changes stay ordered.
3. Idempotent sinks: downstream writers must tolerate retries.
4. Replay path: be able to rebuild a target from a snapshot plus the log.

Debezium supports initial snapshots and then ongoing streaming, which is the usual answer to “how do I bootstrap this without downtime?” Flink CDC similarly treats snapshot and incremental phases as part of one stateful pipeline. That matters because the first production incident is rarely “we stopped consuming new changes.” It is “we need to rebuild one sink from scratch without pausing everything else.”

A simple worked example makes the economics obvious. Suppose your orders table changes at 5,000 rows per second, with an average serialized CDC event size of 1 KB. That is about 5 MB per second, roughly 18 GB per hour before compression and replication. If your broker replication factor is 3, your raw storage footprint jumps to about 54 GB per hour. Suddenly retention, compaction, and replay windows are not academic settings, they are budget decisions.

Use this reference architecture when you need a pipeline that survives contact with reality

The most practical default stack for many teams is PostgreSQL, Debezium, Kafka, and either a lightweight sink layer or Flink for transformations. PostgreSQL publishes logical changes, Debezium captures them, Kafka holds and fans them out, and downstream consumers update search, cache, warehouse, and operational views.

Here is how to build it.

Step 1: Enable database-level CDC correctly

For PostgreSQL, set up logical replication, create the right publication, and manage replication slots carefully. Logical replication is native, selective, and production-grade, but replication slots can retain WAL if consumers fall behind. In other words, backpressure is not just a downstream problem, it can become a database storage problem.

This is also where many teams make their first serious mistake: they test on a noncritical table, watch rows appear in Kafka, and call it done. In production you need to know which tables are included, how primary keys behave on updates, what deletes look like downstream, and how long the source can tolerate a stalled slot.

Step 2: Snapshot once, then stream continuously

Debezium can snapshot existing rows and then continue streaming row-level changes, which is the cleanest way to avoid a separate bulk-load path for many use cases. Its PostgreSQL connector documentation also calls out operational caveats, including limitations around schema changes during incremental snapshots. That warning is exactly the kind of thing teams discover at 2 a.m. if they skip the docs.

The design choice here is whether you want a blocking first snapshot, an incremental snapshot, or a separate historical backfill. For a small to medium table set, one connector-managed snapshot is usually the least painful path. For very large tables, you may want chunked backfill plus live tailing so you do not hold your breath for six hours waiting for “real-time” to begin.

Step 3: Normalize event contracts before consumers multiply

If you stream raw row changes directly, you are publishing your schema. Sometimes that is fine. Often it is a trap.

Debezium’s outbox event router exists because the outbox pattern gives you a stable event envelope while keeping atomicity with the business write. Instead of asking every consumer to interpret before-and-after row state, tombstones, and internal table semantics, you publish events like OrderPlaced or CustomerEmailUpdated with explicit IDs and payloads. That makes replay, versioning, and team boundaries much saner.

My bias is simple: if more than one service will consume the stream, invest in event contracts early. Nothing ages worse than “temporary” raw CDC that became a de facto public API.

Step 4: Add stream processing only when you actually need it

Flink CDC becomes compelling when the pipeline has to do more than copy data. It is a strong fit for full database sync, schema evolution, transformations, and exactly-once workflows. It is overkill for “copy this table into one warehouse table.”

A useful litmus test is whether your target needs the same shape as the source. If yes, use sinks and keep life simple. If no, and especially if you need multi-stream joins or stateful enrichment, a stream processor earns its keep.

Step 5: Build the boring operational rails that save you later

Kafka Connect supports dead letter queues and different failure strategies. Use them. A malformed record should not force a choice between silent data loss and total connector failure. Likewise, monitor lag, replication slot growth, connector restarts, transaction aborts, and sink apply latency as first-class metrics.

This part is never glamorous, which is why it causes the most pain. A CDC pipeline is only “real-time” if you can prove how stale it is right now.

The tricky parts nobody puts on the whiteboard

Schema evolution is the obvious one. Debezium and Flink CDC both support schema-aware workflows, but support does not mean zero friction. Column additions are easy compared with type changes, renames, or semantic shifts that require downstream backfills.

Deletes are another classic footgun. Some targets want tombstones, some want hard deletes, some want soft-delete flags, and some quietly ignore deletes and give you immortal rows. Decide the delete contract up front.

Then there is backpressure. Kafka can absorb some mismatch between source and sink, which is one reason it is so useful in the middle of CDC architectures. But buffering is not the same thing as problem solved. If downstream consumers stay slow, retention windows shrink, replay costs rise, and source-side replication lag can start to bite.

Finally, there is the uncomfortable truth that CDC is not always the right abstraction. Sometimes what you really need is a governed event or data product, not a firehose of table mutations.

FAQ

When should you use CDC instead of polling?

Use CDC when freshness matters, deletes matter, or source load matters. Polling is still acceptable for low-frequency, low-stakes sync jobs, but log-based CDC is usually more efficient and accurate for operational systems.

Can CDC pipelines be exactly-once?

They can be exactly-once within defined boundaries, but only if the whole path participates. Kafka supports exactly-once semantics through idempotence and transactions, and stream processors like Flink build on checkpointing and transactional sinks. In practice, many production pipelines are effectively at-least-once plus idempotent consumers, because that is simpler and often good enough.

Should you stream raw table changes or business events?

For internal analytics, raw table CDC can be fine. For service-to-service integration, use an outbox and publish business events. That gives you a better contract and reduces breakage from schema churn.

Is Kafka mandatory?

No. If your use case is mostly warehouse sync, database-native CDC into a target platform such as Snowflake Streams and Tasks can be enough. Kafka becomes valuable when you need replay, multiple independent consumers, retention, and fan-out.

Honest Takeaway

The cleanest way to build a real-time CDC pipeline is to think like a distributed systems engineer, not like an ETL operator. Start from the database log, decide whether you are publishing raw mutations or stable events, preserve ordering and replay, and make downstream writes idempotent. Everything else is an implementation detail.

If you only remember one thing, remember this: the pipeline is not “done” when rows appear downstream. It is done when you can survive a restart, a schema change, a slow consumer, and a full replay without guessing what happened.

Sumit Kumar

Senior Software Engineer with a passion for building practical, user-centric applications. He specializes in full-stack development with a strong focus on crafting elegant, performant interfaces and scalable backend solutions. With experience leading teams and delivering robust, end-to-end products, he thrives on solving complex problems through clean and efficient code.