Performance incidents rarely fail because of missing dashboards. They fail because the investigation path is unclear, the signal is buried, and the system behaves in ways your mental model does not predict. If you have ever been on a call where ten senior engineers are staring at Grafana, debating whether it is the database, the cache, or “something in Kubernetes,” you know the feeling. The problem is rarely a lack of effort. It is usually a set of upstream mistakes that quietly make performance investigations exponentially harder than they need to be.
Below are nine of the most common mistakes I see in production systems. Every one of them is avoidable. None of them is trivial.
1. Treating observability as an afterthought instead of a design constraint
If you bolt on metrics and tracing after the system is already in production, you are reverse-engineering intent from side effects. That is a losing strategy under pressure.
In one high volume e commerce platform built on Kubernetes and Kafka, we inherited a service mesh with basic HTTP metrics but no domain-level instrumentation. When checkout latency spiked from 180 ms p95 to over 900 ms during peak traffic, we could see CPU, memory, and request rates. We could not see cart validation, pricing calls, or fraud checks as distinct spans. We spent hours triangulating what a few well-placed spans would have made obvious in minutes.
Designing for observability means defining:
- Service level objectives and error budgets
- Key business and system metrics
- Trace boundaries and correlation IDs
- Log structure and cardinality limits
If you do this early, performance investigations become hypothesis driven. If you do not, they become archaeology.
2. Ignoring request context propagation across service boundaries
Distributed systems fail in distributed ways. If your correlation IDs do not cross async boundaries, message queues, and background jobs, your trace graph is fiction.
Teams often instrument HTTP layers but forget that the slow path actually runs through a Kafka consumer, a background worker, and a cache refresh job. When context propagation breaks, you see a 2 second API call but cannot tie it to the downstream batch write that saturated your database.
In one microservices environment, simply enforcing end to end propagation of trace IDs across HTTP, gRPC, and Kafka reduced mean time to isolate performance regressions by more than half. Not because the system got faster. Because the narrative got coherent.
You cannot optimize what you cannot narrate.
3. Over-indexing on infrastructure metrics while ignoring application behavior
A CPU at 30 percent does not mean your system is healthy. Memory at 70 percent does not mean it is not. Infrastructure metrics tell you about resource saturation, not about contention, lock amplification, or inefficient algorithms.
I have seen teams chase Kubernetes node scaling because pods were “under pressure,” when the real issue was an N plus one query pattern in a hot code path. The database was fine. The cluster was fine. The ORM was not.
Performance investigations should move from outside in:
- User-facing latency and error rates
- Service level timings and dependencies
- Code-level hotspots and query plans
- Infrastructure saturation
If you start at layer four, you will often end up tuning the wrong thing.
4. Failing to capture baselines before making changes
Without a baseline, every change feels like progress. Or regression. You cannot tell which.
In a payments service handling tens of thousands of transactions per minute, we ran a series of “optimizations” that claimed to reduce latency. The average improved slightly, but p99 degraded by nearly 40 percent under load. We only caught it because we had historical latency histograms and load test profiles.
Performance is distributional, not average-based. When you change thread pools, garbage collector settings, or connection pool sizes, capture:
- p50, p95, p99 latency
- Throughput under representative load
- Error rates and timeouts
- Resource utilization patterns
Without those, your investigation becomes anecdotal. Senior engineers know anecdotes do not survive scale.
5. Optimizing in production without controlled reproduction
If you cannot reproduce the issue in a controlled environment, you are experimenting on your customers.
There are cases where this is unavoidable. But more often, the real mistake is not investing in realistic load testing and staging environments. Netflix’s chaos engineering practices did not emerge from boredom. They emerged from the recognition that complex distributed systems fail in nonlinear ways, and you need safe environments to surface those behaviors.
Even modest investments help. A staging cluster with production like data volume. Synthetic traffic that mirrors real usage patterns. Feature flags that isolate code paths. When performance degrades, you can then validate hypotheses before pushing speculative fixes.
Production should confirm your theory, not serve as your lab.
6. Letting high cardinality metrics explode your signal
Observability platforms are powerful until they are not. High cardinality labels such as user ID, session ID, or dynamic request parameters can silently degrade your monitoring pipeline.
I have seen Prometheus clusters buckle under label explosion because engineers tagged every metric with customer specific identifiers. Scrape times increased. Queries slowed. During an incident, dashboards timed out just as we needed them most.
High cardinality is sometimes justified. But it must be deliberate. Ask yourself whether you need per user metrics or whether percentile distributions and sampled traces suffice. The wrong choice turns your observability stack into another bottleneck.
Performance investigations are hard enough. Do not make your tooling part of the problem.
7. Treating caching as a silver bullet
Caching can mask structural performance problems until traffic grows enough to punch through the cache layer.
A content API once showed stellar latency under most workloads. The team proudly pointed to a Redis cache hit rate above 95 percent. Then a marketing campaign triggered a surge of cache misses on new content keys. The underlying database queries were poorly indexed and performed full table scans. Latency spiked by an order of magnitude.
Caching works best when the underlying system is already reasonably efficient. Otherwise, you are building a brittle performance illusion. During investigations, always ask: what happens on a cold cache? What happens when keys churn? What is the write amplification?
If the answers are uncomfortable, the cache is not your fix. It is your blindfold.
8. Ignoring tail latency and focusing on averages
Your users do not experience the mean. They experience the worst case that affects them.
Google’s SRE guidance on tail latency emphasizes that even small increases in p99 can materially degrade user experience at scale. In fan-out architectures, tail latency compounds. Ten downstream calls, each with a p99 of 200 ms, do not yield a 200 ms response. They can produce seconds of delay.
In one API gateway service, average latency was stable at 120 ms. But p99 climbed to over 1.5 seconds during peak load because of lock contention in a shared rate limiter. The mean looked healthy. The tails were on fire.
If your dashboards do not prominently feature high percentiles, your investigations will miss the story that matters.
9. Decoupling performance from architectural decision making
The most expensive performance problems are architectural, not tactical.
Choosing synchronous service for service calls across five layers. Centralizing all writes through a single database instance. Ignoring data locality in globally distributed systems. These are not tuning issues. They are design decisions.
I once worked on a platform where every user action triggered a cascade of synchronous calls across seven services. Each team optimized its slice. The system still struggled under load because the critical path was simply too long. The real fix required collapsing services and introducing asynchronous workflows. That was an architectural shift, not a JVM flag tweak.
Performance investigations get dramatically harder when the root cause lives in the shape of the system. At that point, you are not debugging code. You are renegotiating architecture.
Final thoughts
Performance work is rarely glamorous. It is patient, methodical, and often humbling. But when you design for observability, propagate context, respect tail latency, and treat architecture as a first-class performance concern, investigations become tractable instead of chaotic. Systems will always surprise you. The goal is not to eliminate that complexity. It is to make it legible when it matters most.
Kirstie a technology news reporter at DevX. She reports on emerging technologies and startups waiting to skyrocket.
