Running Kubernetes in Production: Best Practices and Pitfalls to Avoid

Kubernetes has become the industry standard for container orchestration, with over 96% of organizations using or evaluating it. However, running Kubernetes in production requires careful planning around security, reliability, observability, and cost management. Many teams learn these lessons the hard way through production incidents. This guide distills best practices gathered from running Kubernetes clusters at scale.

Cluster Architecture and Setup

Start with the right foundation. Use managed Kubernetes services (EKS, GKE, AKS) to reduce operational overhead. Run at least 3 control plane nodes for high availability. Use multiple availability zones for worker nodes. Separate workloads using namespaces and implement resource quotas to prevent noisy-neighbor problems. Use node pools with different instance types for varying workload requirements.

Resource Management

Proper resource management is critical for both reliability and cost control. Always set resource requests and limits for CPU and memory. Requests determine scheduling and should match typical usage. Limits prevent runaway containers and should be set with headroom for spikes. Use the Vertical Pod Autoscaler to right-size resource requests based on actual usage data.

• Requests: Set based on actual P50 usage. Determines scheduling decisions.
• Limits: Set at 2-3x requests. Prevents resource starvation.
• Pod Disruption Budgets: Define how many pods can be unavailable during disruptions.
• Priority classes: Ensure critical workloads get resources first during contention.

Security Hardening

Pod Security

Never run containers as root. Use read-only root filesystems. Drop unnecessary Linux capabilities. Implement Pod Security Standards (Restricted or Baseline profiles). Use network policies to restrict pod-to-pod communication to only what is necessary. Scan container images for vulnerabilities and enforce image signing.

RBAC and Access Control

Implement role-based access control with the principle of least privilege. Create specific roles for different team functions (developers, operators, security). Use service accounts with minimal permissions for workloads. Integrate with your identity provider for authentication. Audit access regularly and remove unused permissions.

Secrets Management

Kubernetes secrets are base64-encoded, not encrypted. Use external secrets managers (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault) with the External Secrets Operator for sensitive data. Enable encryption at rest for etcd. Never store secrets in container images, environment variables, or Git repositories.

Networking Best Practices

Choose a CNI plugin that fits your requirements. Calico offers network policies and good performance. Cilium provides eBPF-based networking with advanced observability. Use service meshes like Istio or Linkerd for mTLS, traffic management, and observability in complex architectures. Implement ingress controllers with proper SSL termination and rate limiting.

Observability and Monitoring

Implement comprehensive monitoring with Prometheus for metrics, Grafana for dashboards, Loki or Elasticsearch for logs, and Jaeger for distributed tracing. Monitor cluster-level metrics (node health, resource utilization) and application-level metrics (latency, error rates, throughput). Set up alerts for critical conditions and establish on-call procedures for incident response.

Backup and Disaster Recovery

Back up etcd regularly and test restoration procedures. Use Velero for cluster-state backups, including persistent volumes. Document your disaster recovery process and practice it quarterly. Maintain infrastructure-as-code definitions so you can recreate clusters from scratch if needed. Consider multi-cluster strategies for critical workloads.

Cost Optimization

Kubernetes cost optimization requires visibility into resource consumption. Use tools like Kubecost or OpenCost to track spending per namespace, label, or team. Right-size resource requests to avoid waste. Use spot/preemptible nodes for fault-tolerant workloads. Implement cluster autoscaling to match capacity with demand. Schedule non-critical workloads during off-peak hours.

Let Bitropix Help

At Bitropix, our platform engineering team designs, deploys, and manages production Kubernetes environments for organizations of all sizes. Whether you are migrating existing workloads to Kubernetes or building cloud-native applications from scratch, we bring the expertise needed to do it right. Contact us to discuss your Kubernetes journey.