Checklist for Kubernetes in Production: Best Practices for SREs

Kubernetes has become the backbone of modern distributed and microservices applications, due to its scalability and out-of-the-box automation capabilities. However, with these powerful capabilities comes quite a bit of complexity that often poses significant challenges, especially for the team tasked with operating production environments.

For SREs managing high-scale Kubernetes operations, ensuring stability and efficiency isn’t impossible. There are some good and replicable practices that can help streamline this significantly. As an SRE at Firefly, who has managed many large-scale production K8s environments, I’ve crystallized these practices into a checklist to help SREs manage their K8s-Ops effectively.

The Kubernetes Production Checklist

Managing Kubernetes in production is no small feat. After navigating challenges, failures, and production incidents across many clusters, this checklist has been created to address the most common root causes of Kubernetes instability. By adhering to these practices, you can mitigate the majority of issues that lead to downtime, performance bottlenecks, and unexpected costs. The practices’ areas are:

• Resource Management: The practice of properly defining requests and limits for workloads.
• Workload Placement: Using selectors, affinities, taints, and tolerations to optimize scheduling.
• High Availability: Ensuring redundancy with topology spread constraints and pod disruption budgets.
• Health Probes: Configuring liveness, readiness, and startup probes to monitor application health.
• Persistent Storage: Establishing reclaim policies for stateful applications.
• Observability and Monitoring: Building robust monitoring systems with alerts and logs.
• GitOps Automation: Using declarative configurations and version control for consistency.
• Cost Optimization: Leveraging quotas, spot instances, and proactive cost management.
• Avoiding Common Pitfalls: Preventing issues with image tags and planning for node maintenance.

This may seem like a long and daunting list, but with today’s DevOps and GitOps practices, we can automate most of the complexity. Once you have an organized checklist it is much easier to add consistency and efficiency to your entire Kubernetes operation. Below we’ll dive into each one of these categories.

Resource Management

Resource allocation is the foundation of Kubernetes stability. Requests define the minimum resources needed for a pod to run, while limits cap the maximum it can consume. Without proper limits, some pods can monopolize node resources, causing others to crash. Conversely, over-restricting resources can lead to throttling, where applications perform sluggishly.

For critical applications, best practices dictate aligning requests and limits to ensure a guaranteed Quality of Service (QoS). Use Kubernetes tools like kubectl describe pod to monitor pod behavior and adjust configurations proactively.

While there are entire blog posts focused just on monitoring pod behavior, a good way to do this is using kubectl describe pod to identify resource issues. You would next inspect the output for details such as resource requests and limits, and events like OOMKilled (Out of Memory) or CrashLoopBackOff, and node scheduling details. Finally you would diagnose the issue and decide if the container is running out of memory because its workload exceeds the defined limits, and adjust the configurations accordingly.

Proactively identifying and resolving resource allocation issues, helps prevent operational disruptions.

Workload Placement

Workload placement determines how effectively your resources are used and whether critical applications are isolated from less important ones.

Node Selectors and Affinities

Assigning workloads to specific nodes based on labels helps optimize resource utilization and ensure workloads run on the most appropriate hardware. Assigning specific nodes is especially important for workloads with specialized requirements, because it prevents resource contention and enhances application performance.

For example, assigning GPU-intensive pods to GPU-enabled nodes ensures these workloads can leverage the required hardware accelerators without impacting other workloads on general-purpose nodes. Similarly, by using node labels to group nodes with high memory or fast storage, applications that need these capabilities can be efficiently scheduled without unnecessary conflicts.

Additionally, node affinities allow for more granular placement rules, such as preferring workloads to run on certain nodes, while still permitting scheduling flexibility. This approach ensures that Kubernetes schedules pods in a way that aligns with both operational priorities and resource availability.

Taints and Tolerations

Using taints and tolerations helps maintain workload isolation by preventing non-critical applications from running on nodes reserved for high-priority or specialized workloads. This tool ensures that critical applications have uninterrupted access to the resources they require, minimizing the risk of performance degradation caused by resource contention.

For instance, applying a taint to nodes designated for database workloads restricts those nodes to handle only workloads with tolerations for that taint. Applying a taint prevents general-purpose or less critical applications from consuming resources on these reserved nodes, ensuring databases operate consistently and without interruptions.

By implementing taints and tolerations strategically, Kubernetes clusters can achieve greater reliability and predictability, especially for workloads with stringent performance or availability requirements.

High Availability

High availability ensures services remain operational despite failures or maintenance, here are several factors that impact your availability in Kubernetes environments.

Topology Spread Constraints

Distributing workloads evenly across zones or nodes using topology spread constraints helps ensure high availability and resilience in the event of failures. This approach minimizes the risk of overloading a single zone or node, maintaining consistent performance and availability even during disruptions.

For example, in a multi-zone setup, configuring topology spread constraints ensures that pods are balanced across all available zones. This way, if one zone becomes unavailable due to a failure or maintenance, the remaining zones can continue handling the workload without a significant impact on application availability or performance.

By leveraging topology spread constraints, Kubernetes can enforce even distribution policies, reducing single points of failure and enhancing the reliability of services in production environments.

Pod Disruption Budgets (PDBs)

Setting Pod Disruption Budgets (PDBs) helps maintain service continuity by controlling the number of pods that can be disrupted during events such as updates, node maintenance, or failures. With PDBs, critical workloads remain operational and available, even when disruptions occur.

For instance, configuring a PDB for a deployment running three replicas might specify that at least two pods must remain available at all times. This configuration prevents Kubernetes from evicting too many pods simultaneously, ensuring the application continues serving requests and meeting availability requirements.

By using PDBs, organizations can strike a balance between operational flexibility (e.g., rolling updates or scaling nodes) and application reliability, making them a crucial tool for maintaining stability in production environments.

Health Probes

Kubernetes health probes play a critical role in automating container lifecycle management, keeping applications responsive and functional under varying conditions. These probes help Kubernetes detect and resolve issues with containers automatically, reducing downtime and operational overhead. There are 3 types of probes:

• Liveness Probes: These probes check if a container is stuck or has stopped responding due to internal errors. When a liveness probe fails, Kubernetes restarts the container to restore functionality. Liveness probes are particularly useful for long-running applications that might encounter memory leaks or deadlocks over time.
• Readiness Probes: These probes verify if a container is ready to handle incoming traffic. For instance, Kubernetes uses readiness probes to delay traffic routing to a pod until it has fully initialized and is prepared to serve requests. Readiness probes provide a smooth user experience by preventing failed requests during startup or configuration changes.
• Startup Probes: Designed for applications with long initialization times, such as Elasticsearch or other stateful workloads, startup probes prevent premature health checks from failing during startup. By allowing these applications sufficient time to initialize, Kubernetes avoids unnecessary restarts or disruptions caused by incomplete readiness or liveness evaluations.

Together, these probes ensure that applications running in Kubernetes remain healthy, scalable, and ready to meet user demands with minimal manual intervention.

Persistent Storage

Stateful workloads demand reliable and consistent storage strategies to ensure data integrity and availability across container lifecycles. Kubernetes provides persistent volumes (PVs) with configurable reclaim policies that determine how storage is managed when associated pods are terminated or removed:

• Retain Policy: The retain policy is ideal for critical applications like databases, where data persistence is essential. With this policy, data stored in a persistent volume remains intact even if the associated pod is deleted. By using a retain policy, critical data can be accessed and restored when needed, providing stability and continuity for stateful applications.
• Delete Policy: The delete policy is suited for temporary workloads where data does not need to persist beyond the lifecycle of the workload. For instance, a log processing pipeline that generates intermediate files can use this policy to automatically clean up storage after completion, preventing unnecessary resource consumption.

By aligning the reclaim policy with workload requirements, Kubernetes ensures efficient use of storage resources while maintaining the reliability needed for both critical and transient applications.

Observability and Monitoring

• Robust observability is essential for detecting and resolving issues in Kubernetes environments before they escalate into critical failures. By implementing comprehensive monitoring and logging systems, teams can gain actionable insights into cluster performance and maintain operational stability.
• Prometheus and Grafana: Prometheus serves as a powerful time-series database for collecting metrics from Kubernetes components, while Grafana provides intuitive visualizations of these metrics. Together, they enable teams to monitor cluster health in real-time and identify trends or anomalies that may require attention. For instance, a spike in CPU usage across nodes can be visualized in Grafana dashboards, prompting proactive scaling.
• Critical Alerts: Configuring alerts ensures that key issues, such as node memory pressure, insufficient disk space, or pods stuck in crash loops, are flagged immediately. Tools like Prometheus Alertmanager or Grafana Loki can send notifications to on-call engineers. Additionally, using commands like kubectl top allows teams to identify and address resource bottlenecks at the node or pod level.
• Log Retention: Retaining logs for post-incident analysis is crucial for understanding root causes and preventing recurrences. Tools like Loki or Elasticsearch can aggregate and store logs, making them easily searchable during debugging. For example, when investigating a pod crash, logs can reveal the exact error that caused the failure, enabling targeted fixes.

By integrating these observability practices, Kubernetes operators can maintain high availability, optimize performance, and respond swiftly to unexpected incidents.

GitOps Automation

GitOps, a declarative approach to managing infrastructure and applications using Git as the single source of truth, is often leveraged for Kubernetes systems for automated processes that continuously reconcile the desired and actual states of a system. This is because GitOps introduces a streamlined and reliable approach to Kubernetes operations by leveraging automation and version control to manage infrastructure and application configurations. This methodology ensures consistency, simplifies deployments, and facilitates rapid recovery in the event of failures.

• Declarative Configurations: In GitOps, all Kubernetes resources are defined as code, ensuring they are versioned, auditable, and reproducible. Tools like Helm charts or Kustomize allow you to create modular and reusable deployment templates, making it easier to manage complex configurations and scale applications effectively – and tools like Firefly help codify your resources rapidly.
• Version Control: By storing configurations in a Git repository, GitOps provides a single source of truth for your cluster. This setup allows you to track changes, implement code reviews, and roll back to a previous state if a deployment fails. For instance, if an update introduces a bug, reverting to a stable configuration becomes a simple git revert operation.
• Reconciliation: Tools like ArgoCD or Flux continuously monitor the Git repository and the cluster, ensuring that the desired state defined in Git matches the actual state of the cluster. If discrepancies are detected (e.g., a manual change in the cluster), these tools automatically apply corrections to restore compliance. This self-healing capability reduces manual intervention and enforces consistency across environments.

GitOps not only simplifies Kubernetes management but also fosters a culture of automation and transparency, enabling teams to deploy with confidence and maintain stability in production environments.

Cost Optimization

Efficient cost management is a critical aspect of Kubernetes operations, especially in large-scale environments where resource usage can escalate quickly. When running Kubernetes operations on public clouds, it is a good idea to employ strategic cost optimization techniques, so that organizations can reduce expenses without compromising reliability or performance.

• Spot Instances: Spot instances are a cost-effective solution for non-critical workloads like batch processing or CI/CD pipelines. These instances are significantly cheaper than on-demand instances but come with the risk of being terminated if capacity is needed elsewhere. Therefore, spot instances should be avoided for critical applications, such as databases or stateful services, where disruptions could impact operations.
• Reserved Instances and Committed Use Discounts: For workloads that require long-term stability, leveraging reserved instances (AWS RIs, Azure Reserved VM Instances) or committed use discounts (Google Cloud Committed Use Contracts) provides significant savings over on-demand pricing. By committing to a specific amount of compute capacity for a fixed period (such as one year or even longer), organizations can optimize costs for predictable, long-running workloads such as databases, stateful applications, and core business services.
• Quotas and Limits: Resource quotas and limits help control resource allocation within namespaces, ensuring that workloads do not exceed defined thresholds. For example, setting a CPU or memory limit in a testing namespace prevents developers from unintentionally overloading the cluster, which could lead to unnecessary costs. These configurations also encourage teams to optimize their workloads and use resources judiciously.
• Cloud Cost Alerts: Monitoring cloud usage is essential to catch unexpected cost spikes early. Setting up alerts for key metrics like excessive resource consumption, unoptimized storage, or prolonged idle workloads can help teams take immediate corrective actions. Many cloud providers and Kubernetes monitoring tools integrate cost-tracking features that provide detailed insights into resource utilization and associated costs.

By implementing these cost optimization strategies, teams can effectively manage their Kubernetes environments while staying within budget, ensuring that operational efficiency aligns with financial goals.

Avoiding Common Pitfalls

In Kubernetes, seemingly minor missteps can lead to significant operational challenges. By proactively addressing common pitfalls, teams can maintain stability, predictability, and resilience in production environments.

• Avoid “Latest” Tags: Using the latest image tag for container deployments may seem convenient, but it introduces unpredictability. When the latest tag is updated, Kubernetes might pull a new version of the container without notice, leading to version mismatches or unintended behavior. Instead, always use specific, versioned image tags (e.g., v1.2.3) to ensure consistency and traceability in deployments. This approach also simplifies debugging, as teams can identify exactly which version of the application is running.
• Node Maintenance: Regular node maintenance is essential for applying updates, scaling, or resolving hardware issues, but it must be carefully planned to avoid disruptions. Use Kubernetes pod eviction strategies to manage workloads during maintenance:

  • Run kubectl cordon <node-name> to mark a node as unschedulable, preventing new pods from being assigned to it.
  • Use kubectl drain <node-name> to safely evict running pods and migrate them to other nodes in the cluster. These commands ensure workloads are redistributed without downtime, maintaining service continuity during upgrades or repairs.

By avoiding these common pitfalls, teams can ensure that their Kubernetes clusters remain stable and predictable, even as they evolve to meet changing business needs.

An SRE’s Kubernetes Roadmap

Kubernetes provides immense power and flexibility, but success in production demands thorough preparation. This list was built to address some of the core areas that frequently cause instability, downtime, and inefficiencies in Kubernetes clusters, offering actionable steps to mitigate these challenges. By following these practices, teams can transform Kubernetes into a reliable, efficient, and cost-effective platform.

Running Kubernetes in production requires both technical expertise and meticulous planning, and a good checklist serves as an excellent tool for SREs to serve as a roadmap, covering critical areas from resource management to cost optimization. By following these guidelines and leveraging Kubernetes’ native tools, teams can build resilient, efficient, and scalable environments.