You usually discover you need better scaling in Kubernetes at the worst possible moment. Latency creeps up. A batch job lands unexpectedly. Traffic doubles after a launch. Suddenly, pods are stuck in Pending, or worse, everything is technically “Running,” but the user experience is deteriorating.
In plain terms, scaling Kubernetes clusters means adjusting how many workloads you run and how much infrastructure backs them. That includes scaling pods horizontally, resizing them vertically, and adding or removing nodes so everything fits. Kubernetes can automate most of this, but only if you feed it accurate signals. If your resource requests are wrong, your metrics are noisy, or your disruption rules are overly strict, autoscaling amplifies those mistakes.
This guide is the practitioner’s version of cluster scaling. No magic. No hype. Just what actually moves the needle when you operate real production systems.
What experienced operators agree on
When you study how large teams run Kubernetes in production, a few consistent patterns emerge.
Sandeep Dinesh, Developer Advocate at Google Cloud, has long emphasized that clusters appear stable without well-defined resource requests and limits until they suddenly are not. The scheduler relies on resource requests to place workloads safely. Limits prevent noisy neighbors from destabilizing nodes. As teams grow and services multiply, missing resource contracts become operational debt that surfaces as instability.
On the infrastructure side, Robert Northard, AWS Container Specialist SA, and Carlos Manzanedo Rueda, AWS Efficient Compute Leader, highlight that modern node provisioning tools like Karpenter react to unschedulable pods, not to abstract utilization graphs. They aggregate pod requests and scheduling constraints to decide what infrastructure to launch. They do not look at real-time CPU utilization or pod limits when making provisioning decisions. That makes them fast and efficient, but unforgiving when requests are misconfigured.
Upstream Kubernetes documentation reinforces the same mechanism. Node autoscalers add capacity when pods cannot be scheduled. If everything fits, even if nodes are running hot, the autoscaler does nothing.
Taken together, the lesson is simple: scaling is driven by scheduling mechanics and resource definitions, not dashboard aesthetics. If your YAML lies, your scaling lies.
Understand your scaling levers before you touch them
Kubernetes offers multiple scaling mechanisms. Each solves a different class of problem.
You can scale:
- Pods horizontally by increasing replicas
- Pods vertically scale by increasing their resource requests
- Nodes horizontally by adding more machines
- Infrastructure efficiency by consolidating workloads
The most common tools in production clusters include:
| Lever | What it scales | Trigger signal | Best for | Common pitfall |
|---|---|---|---|---|
| HPA | Pod replicas | CPU, memory, custom metrics | Request-driven services | Poor metrics, wrong requests |
| VPA | Pod requests | Observed usage trends | Rightsizing steady workloads | Disruptive restarts |
| Cluster Autoscaler | Node count in node groups | Unschedulable pods | Stable node pools | Blocked scale-down |
| Karpenter | Dynamically provisioned nodes | Unschedulable pods plus constraints | Flexible capacity | Inaccurate requests |
| KEDA | Pod replicas | External event triggers | Burst and scale-to-zero | Flapping from bad triggers |
The key distinction is this: HPA and KEDA change how many pods you want. Cluster Autoscaler and Karpenter determine whether the cluster can fit them. VPA changes how large those pods are.
When teams mix these without understanding the boundaries, scaling becomes unpredictable.
Step 1: Fix resource requests and limits first
Almost every scaling issue traces back to resource configuration.
Kubernetes schedules pods based on resource requests, not actual usage. If you omit requests or set them inaccurately, scheduling decisions and autoscaling decisions both degrade.
Two principles consistently hold up in production:
- Define CPU and memory requests for every container.
- Be conservative with CPU limits for latency-sensitive services.
- Be strict with memory where OOM risk is real.
Memory is not compressible. When you exceed a memory limit, the container dies. CPU can throttle, but memory kills.
A worked example with numbers
Imagine an API service running 60 pods.
Each pod requests:
- 500m CPU
- 1Gi memory
Total requested:
- 30 vCPU
- 60Gi memory
Now, assume you use 8 vCPU and 32Gi memory nodes. After reserving system overhead, you may have roughly 30Gi usable memory per node.
With 1Gi per pod, each node fits about 30 pods by memory. You need 2 nodes.
Now someone “plays it safe” and doubles memory requests to 2Gi.
Total memory requested becomes 120Gi.
Now each node fits roughly 15 pods. You need 4 nodes instead of 2.
If actual usage never exceeds 900Mi per pod, you just doubled infrastructure cost without improving stability.
Autoscalers did exactly what you asked. The mistake was upstream.
Scaling begins with honest resource accounting.
Step 2: Make horizontal pod scaling predictable
Horizontal Pod Autoscaler adjusts replica count based on observed metrics. Most teams start with CPU utilization because it is simple.
But two realities matter.
First, CPU-based HPA compares usage to the requested CPU. If requests are inflated, HPA under-scales. If requests are tiny, HPA overreacts.
Second, CPU is often a lagging indicator. For APIs, queue depth, request concurrency, or p95 latency may provide better signals than raw CPU.
A pragmatic production setup looks like this:
- Use CPU HPA as a baseline safety mechanism.
- Add one business-relevant metric once instrumentation is trustworthy.
- Set realistic minimum and maximum replicas.
Scaling should feel boring. If replicas oscillate wildly, your metrics or stabilization windows need tuning.
Step 3: Add node scaling with clear expectations
Cluster Autoscaler increases node count when pods are unschedulable. It reduces nodes when they can be drained safely.
It does not add nodes because CPU graphs look high. It reacts to scheduling failures.
Two issues commonly block scale-down:
Pod disruption constraints. If a Pod Disruption Budget is too strict, nodes cannot drain.
Unevictable pods. Certain annotations or workload types prevent eviction, which blocks node removal.
If scale-down never happens, check disruption policies, affinity rules, and daemonset overhead before blaming the autoscaler.
Many teams also benefit from separating workload types into distinct node pools. Long-running services tolerate disruption differently from bursty jobs. Mixing them complicates autoscaling behavior.
Step 4: Use dynamic provisioning when flexibility matters
If traditional node groups feel rigid, dynamic provisioning tools like Karpenter offer more flexibility.
Instead of scaling predefined node groups, dynamic provisioners evaluate unschedulable pods and create infrastructure that satisfies their aggregate resource and scheduling constraints.
This approach shines when:
- Workloads vary significantly in size
- You use multiple instance families
- You want aggressive cost optimization
However, dynamic consolidation introduces disruption. If workloads lack proper disruption budgets or restart tolerance, consolidation can create instability.
Again, everything flows back to resource accuracy and workload design.
Step 5: Rightsize continuously with vertical autoscaling
Vertical Pod Autoscaler analyzes historical usage and recommends better resource requests.
The safest rollout strategy is:
- Run in recommendation mode first.
- Compare recommendations against actual production metrics.
- Apply gradually to selected services.
In some managed environments, VPA can react to OOM events by increasing memory recommendations after a crash, which improves resilience over time. But blindly enabling automatic updates across all workloads can cause avoidable restarts.
Use VPA as a calibration tool, not a blanket fix.
Think of scaling as two questions:
- How many replicas do you need?
- How large should each replica be?
Horizontal scaling answers the first. Vertical scaling informs the second.
Step 6: Handle bursts and scale-to-zero carefully
Event-driven scaling systems like KEDA enable scaling based on external triggers such as queue depth or cloud service metrics. They are powerful for workloads that do not need constant capacity.
The implementation pattern typically involves defining a scaling object that connects event triggers to replica counts. Under the hood, this often integrates with HPA behavior.
Practical guidance:
- Match polling intervals to real workload patterns.
- Prefer backlog size over raw event rate.
- Test failure scenarios, including missing metrics and extreme spikes.
Event-driven scaling is excellent for cost control, but misconfigured triggers can create replica oscillation.
FAQ
Why didn’t my cluster scale up even though nodes were hot?
Because node autoscalers respond to unschedulable pods, not utilization percentages. If pods still fit, no new nodes are added.
Why won’t it scale down?
Strict disruption policies, unevictable pods, or heavy daemonset overhead often prevent node draining.
Should I use Cluster Autoscaler or a dynamic provisioner?
If workloads are predictable and node pools are stable, traditional autoscaling works well. If workload diversity and cost optimization are priorities, dynamic provisioning may offer better flexibility.
Do requests really matter that much?
Yes. Requests drive scheduling. Scheduling drives autoscaling. Everything else builds on that foundation.
Honest Takeaway
Scaling Kubernetes clusters is not about stacking more autoscalers onto the cluster. It is about building a clean feedback loop between workload demands and infrastructure supply.
Start with accurate requests. Add horizontal scaling using metrics you trust. Introduce node scaling with realistic disruption policies. Layer in rightsizing and event-driven scaling once the fundamentals are stable.
Most production scaling problems are not caused by Kubernetes being complex. They are caused by treating resource definitions casually.
If you fix that, scaling becomes predictable instead of dramatic.
Rashan is a seasoned technology journalist and visionary leader serving as the Editor-in-Chief of DevX.com, a leading online publication focused on software development, programming languages, and emerging technologies. With his deep expertise in the tech industry and her passion for empowering developers, Rashan has transformed DevX.com into a vibrant hub of knowledge and innovation. Reach out to Rashan at [email protected]
