Kubernetes Cost Mistakes: Why Your K8s Bill Is 3x What It Should Be

The Scale of the Problem

Research from the FinOps Foundation consistently shows that Kubernetes clusters run at 20-35% average utilization. That means 65-80% of the compute capacity you're paying for is sitting idle. For a mid-size enterprise running 50 nodes at $500/month each, that's $16,000-$20,000/month in waste.

Mistake #1: Over-Provisioning Requests and Limits

The most expensive mistake: developers set CPU and memory requests based on fear, not data. They request 2 CPU cores and 4GB RAM for a service that actually uses 0.3 cores and 512MB. The scheduler reserves those resources, even though they'll never be used.

# What developers write (fear-based):
resources:
  requests:
    cpu: "2000m"      # 2 full cores reserved
    memory: "4Gi"     # 4GB reserved
  limits:
    cpu: "4000m"
    memory: "8Gi"

# What the service actually uses:
#   CPU: avg 280m, p99 620m
#   Memory: avg 480Mi, peak 720Mi

# The fix (data-driven):
resources:
  requests:
    cpu: "350m"       # ~1.25x average usage
    memory: "600Mi"   # ~1.25x average usage
  limits:
    cpu: "1000m"      # ~1.5x p99
    memory: "1Gi"     # ~1.4x peak

Mistake #2: Non-Production Clusters Running 24/7

Your production cluster needs to run 24/7. Your dev, staging, and QA clusters do not. Engineers work ~10 hours/day, 5 days/week. Those dev clusters run 168 hours/week but are actively used for ~50 hours.

• Schedule non-prod clusters to scale down at 7pm and scale up at 7am weekdays
• Ephemeral preview environments — spin up per pull request, tear down on merge
• Namespace quotas — prevent developers from running 20 replicas in dev
• Savings: typically 50-60% of non-production compute costs

Mistake #3: Wrong Node Sizing

The choice of node instance type dramatically affects packing efficiency. Too large, and you waste capacity. Too small, and Kubernetes daemon overhead (kubelet, kube-proxy, OS) eats a disproportionate share.

Mistake #4: Ignoring Cross-AZ Data Transfer

In AWS, inter-AZ data transfer costs $0.01/GB in each direction. If your microservices are spread across 3 AZs and chatty, this adds up fast. A service exchanging 10GB/hour with another service in a different AZ costs $144/day in data transfer alone.

The Fix

• Use topology-aware routing (topology.kubernetes.io/zone) to prefer same-AZ communication
• Co-locate chatty services in the same AZ using pod affinity rules
• Compress gRPC/HTTP payloads between services
• Monitor cross-AZ traffic with VPC Flow Logs

Mistake #5: Persistent Volume Sprawl

Persistent Volumes (PVs) are provisioned and never cleaned up. Developers create 100GB volumes for a 2GB database "just in case," and when the pod is deleted, the PV remains — still being billed.

• Audit PVs monthly — delete unattached volumes
• Use reclaimPolicy: Delete for non-critical workloads
• Right-size volumes and use volume expansion instead of over-provisioning
• Use cheaper storage classes (gp3 instead of io2) where IOPS aren't critical

Mistake #6: Not Using Spot/Preemptible Instances

Spot instances are 60-90% cheaper than on-demand. For stateless, fault-tolerant workloads (web servers, batch processing, CI/CD), there's no reason not to use them. Yet most K8s clusters run 100% on-demand.

Mistake #7: Observability Costs Exceeding Compute

In some organizations, Datadog, New Relic, or Splunk bills exceed the actual infrastructure costs. Logging every request at full fidelity, retaining metrics for 13 months, and tracing 100% of requests is expensive and usually unnecessary.

• Sample traces — trace 10% of requests, 100% of errors
• Reduce log verbosity — INFO in production, DEBUG only in dev
• Shorten retention — 30 days for logs, 90 days for metrics, 7 days for traces
• Switch to open-source — OpenTelemetry + Grafana + Loki saves 60-80% vs commercial APM

The K8s Cost Optimization Playbook

A phased approach to reducing Kubernetes spend by 40-60%:

Week 1-2: Quick Wins (20-30% savings)

1. Install VPA in recommendation mode on all workloads
2. Schedule non-prod clusters to shut down nights/weekends
3. Delete unattached PersistentVolumes and orphaned LoadBalancers
4. Switch to gp3 storage class where applicable

Month 1: Right-Sizing (15-25% additional savings)

1. Apply VPA recommendations to resource requests/limits
2. Optimize node instance types for workload profiles
3. Implement Cluster Autoscaler or Karpenter for dynamic scaling
4. Enable topology-aware routing

Month 2-3: Advanced Optimization (10-15% additional savings)

1. Migrate stateless workloads to spot instances (50-70% of fleet)
2. Implement HPA with custom metrics (CPU/memory + request latency)
3. Right-size observability stack and sampling rates
4. Purchase reserved capacity for baseline workloads (1-year or 3-year savings plans)