Kubernetes node autoscaling: Handling unpredictable traffic spikes without manual intervention

The Problem

A real-time data platform handled unpredictable traffic: election updates, breaking news, market announcements created sudden spikes of millions of requests in minutes. The team tried Horizontal Pod Autoscaler (HPA) to scale replicas, but it failed.

Why? HPA scaled pods, but the cluster had no room to place them. New pods sat “Pending” forever because nodes were full. Traffic queued, clients timed out. The problem wasn’t “how many pods” — it was “how many nodes.”

The Challenge

Traditional autoscaling approaches all failed:

• HPA alone doesn’t work — scales pod count, but cluster has no capacity
• Manual node scaling — defeats the purpose of Kubernetes, doesn’t predict spikes
• Vertical Pod Autoscaler — changes request sizes, doesn’t solve the “no room” problem
• Overprovisioning — buying extra nodes for “just in case” is wasteful and expensive
• Unpredictability — traffic spikes are random; can’t predict when they’ll happen

The Solution

We implemented Cluster Autoscaler with proper pod request sizing:

Phase 1: Pod Request Sizing

• Analyzed actual resource usage across all workloads
• Set CPU/memory requests based on 95th percentile observed usage (not guesses)
• Removed over-provisioning that was wasting 60% of capacity
• Configured pod disruption budgets for safe scaling down

Phase 2: Cluster Autoscaler

• Deployed Cluster Autoscaler to monitor for pending pods
• Configured auto-add nodes when pods can’t be scheduled
• Set auto-remove nodes when they’re idle
• Integrated with cloud provider scaling groups (AWS, GCP, Azure)

Phase 3: Scaling Groups & Instance Types

• Created separate node pools for different workload types
• Reserved instances for baseline capacity (always needed)
• Spot instances for burstable workloads (handles spikes cost-effectively)
• Mixed fleet strategy to reduce interruption risk

Phase 4: Monitoring & Safety

• Alert on pending pods (indicates insufficient capacity)
• Track scale-up/scale-down events and timing
• Test failover for Spot instance interruptions
• Document scaling behavior for team

Results

Traffic Handling:

• ✅ Traffic spikes automatically scale cluster (no pending pods)
• ✅ Scale-up time: pods running within 2-3 minutes of spike start
• ✅ Scale-down time: 10 minutes after spike ends (graceful)
• ✅ Zero request timeouts due to capacity (previously common)

Operational:

• ✅ No manual node scaling required
• ✅ Scaling is predictable and auditable
• ✅ Team confidence in handling unexpected traffic
• ✅ Simple configuration (yaml-based, no custom code)

Financial:

• ✅ Cost reduced vs manual over-provisioning
• ✅ Spot instances reduce baseline cost 60-70%
• ✅ Reserved instances provide stable baseline
• ✅ No “just in case” over-provisioning wasting money

Key Takeaways

1. HPA solves pod-level scaling, not cluster-level capacity — you need both
2. Correct pod requests are foundation — without them, autoscaling can’t work properly
3. Cluster Autoscaler is simple and effective — open source, cloud-agnostic, just watches for pending pods
4. Spot instances are safe — with proper disruption budgets and mixed fleet strategy
5. Unpredictable workloads still autoscale well — if the system reacts to actual demand (pending pods), not predicted demand