You can carve a monolith into fifty microservices and still ship a system that behaves like a flaky distributed monolith. The difference is that now every inconsistency shows up as a cross service incident instead of a single database lock. Teams often jump to microservices for autonomy or performance, then discover they never agreed on how data should stay correct across boundaries. Incidents start to sound the same: double charges, lost updates, ghost orders, or dashboards that never match production reality. Before you add another service, you need to be able to articulate a handful of hard data consistency decisions with real precision. That is the real architecture work.
1. What is the system of record for each piece of data?
The first consistency decision is ownership. For every field that matters, you should be able to answer which service is the system of record and which ones are holding a cached or derived copy. I have seen large retail platforms where inventory counts lived in three different databases with no clear owner. During peak sale events, one path decremented inventory synchronously while another relied on a lagging event stream. The result was oversells and manual reconciliation that took weeks. The root cause was not the message broker but the fact that ownership was never made explicit.
Once you define the system of record, you inherit responsibilities. The owning service must expose change events, manage invariants, and provide queries that other services should not reimplement on their own stores. Replicas become consumers of a well defined data contract rather than random joins on whatever table is easy. This is also where you decide whether you will push updates to consumers or let them pull at their own cadence. Stronger clarity on ownership does not eliminate duplication, but it allows you to reason about which copy wins when the world gets weird.
2. How consistent do reads need to be for each use case?
Not every read in your system needs strong consistency, but some absolutely do. The problem is that teams rarely write this down. A classic pattern is a user profile service in front of a separate analytics or recommendation store. Product wants profile changes to appear instantly in the UI, but they are fine if recommendation updates lag by a few minutes. If you do not capture that difference explicitly, you end up over engineering some paths and under engineering others. In a payments flow, the tolerance is much lower. A user should not see a refunded charge appear as pending for hours because a service relied on eventually consistent reads from a replica that can drift under load.
A useful move is to label key read paths with their required guarantees. Do you need read your writes for a given user after an update? Is monotonic reads enough across a session? Is it acceptable for an admin dashboard to be a few minutes behind? At one fintech, we tagged endpoints with a staleness budget in seconds. It forced honest conversations about whether we could safely read from replicas, caches, or materialized views. When you know which reads can be stale and by how much, you can scale aggressively where it is safe and pay for stronger consistency only where it is necessary.
3. How will you maintain invariants across service boundaries?
The third decision is about invariants that cross service lines. Things like “an order cannot be shipped if payment authorization has not succeeded” or “a user cannot have two active primary payment methods.” In a monolith, these invariants often live in a single transaction. Once you split services, you have to choose how to preserve them. I have seen teams try to keep cross service invariants by reaching into each other’s databases, which works until it does not and leaves you with tight coupling and no clear migration path. A better answer usually involves sagas, outbox patterns, or domain events that encode the invariant as a state machine rather than a single ACID boundary.
Consider a simple order pipeline with Order, Payment, and Fulfillment services. You can insist on a distributed transaction across the three, but you will pay in complexity, latency, and operational fragility. Or you can model the process as a saga where each service performs a local transaction and emits an event, with compensating actions on failure. The consistency decision is not “eventual vs strong” in the abstract. It is “which invariants must never be violated, and what is the acceptable window where the world might look temporarily inconsistent while the saga completes.” Senior engineers make that window explicit and build observability around it instead of pretending the system behaves like a single database.
4. What happens under failure, retries, and duplicates?
Your consistency story is only as good as its behavior when the network misbehaves. The fourth decision is about failure semantics and how you handle retries, duplicates, and reordering. Any serious system with queues, brokers, or streaming platforms will redeliver messages. I have seen Kafka consumers “fix” a bug by enabling at most once delivery through misconfiguration, which made some incidents disappear and created much more serious silent data loss. In an ecommerce system, a poorly designed retry mechanism took p95 latency on a checkout API from 120 ms to 650 ms because the service issued duplicate writes to downstream systems that then hit unique constraints and locked rows.
The healthy pattern is to assume at least once delivery and design idempotent handlers. That often means idempotency keys, natural business identifiers, and version checks at the storage layer. You need to decide where in the stack you will deduplicate and what your policy is for conflicting writes. Do you accept last write wins, require explicit versioning, or reject late updates with an error the client must resolve? Failure paths need just as much design attention as happy paths. If the only thing preventing double booking a hotel room is the hope that retries never collide, you do not have a data consistency strategy, you have a lucky streak.
5. How will you evolve schemas and migrate data across services?
The last decision is about time, not just space. Once you have multiple services owning and consuming data, schema evolution becomes a core part of your consistency model. I have seen organizations freeze feature work for months because a “simple” change to a user entity broke eight downstream services that had been deserializing internal fields they were never supposed to touch. Another common pattern is a long running migration to split a monolithic table into per service stores without a clear plan for dual writes or backfill ordering. The result is subtle data skew that only shows up when support tickets pile up.
You need an explicit strategy for compatibility and migration. That includes versioned schemas or events, guarantees about backward and forward compatibility for a certain window, and a runbook for dual writing and backfills. Techniques like the outbox pattern plus change data capture can help you keep new stores in sync while you cut traffic over incrementally. The key consistency decision is “who can change what, when, and how do we ensure every consumer either understands the new shape or safely ignores it.” Without that, every schema change becomes a high risk operation, and teams avoid evolving the model entirely, which is its own form of inconsistency with the domain.
Microservices do not magically improve data consistency. They simply force you to confront it in more places. Before adding another service, you should be able to answer five questions with confidence: who owns which data, how fresh reads must be, how cross service invariants hold, what happens under failure and retries, and how schemas evolve safely. These are not academic concerns. They are the difference between a system that scales with predictable behavior and one that accumulates subtle, business critical bugs. Get these decisions clear, and the rest of your architecture has a chance to age gracefully.
Kirstie a technology news reporter at DevX. She reports on emerging technologies and startups waiting to skyrocket.
