エピソード

  • How Kubernetes Watch Events Overload the API Server

    How Kubernetes Watch Events Overload the API Server
    2026/07/21
    In this episode of DevOps Daily, Lucas and Luna dive into a notoriously overlooked Kubernetes failure mode—API server overload caused by excessive watch events. They break down how controllers, informers, and custom operators all lean on watch-based architectures, and why a single misconfigured watch can cascade into cluster-wide latency or even outages. Through the lens of a real incident at a mid-sized fintech, they explain what happened when a poorly tuned deployment triggered millions of watch events per minute, flooded etcd, and brought the control plane to its knees. Lucas outlines the specific metrics to monitor (watch event rate, request latency, etcd db size) and the architectural patterns—like using DeltaFIFO, filtering with field selectors, and throttling informer resync intervals—that prevent these meltdowns. Luna brings her own war story about a CI/CD operator that hammered the API server during a Helm release, and the two discuss how to audit your cluster for watch-heavy workloads. By the end, listeners will know exactly how to diagnose and prevent one of the most common yet invisible causes of Kubernetes instability. #Kubernetes #APIServer #WatchEvents #Etcd #K8sOperators #InformerPattern #ClusterReliability #DevOps #CI/CD #Fintech #SiteReliabilityEngineering #K8sPerformance #ControlPlane #DeltaFIFO #FieldSelectors #ContainerOrchestration #FexingoBusiness #BusinessPodcast Keep every episode free: buymeacoffee.com/fexingo
    続きを読む 一部表示
    10 分
  • How Kubernetes Vertical Pod Autoscaler Recommends Wrong Requests

    How Kubernetes Vertical Pod Autoscaler Recommends Wrong Requests
    2026/07/21
    Kubernetes Vertical Pod Autoscaler (VPA) is supposed to right-size container resource requests automatically. But in practice, VPA's recommender often suggests CPU and memory values that lead to pod evictions or wasted capacity. Drawing on real incidents from a mid-2026 production cluster running 50 microservices, Lucas and Luna unpack why VPA's default OOM-aware policy can misinterpret short-lived memory spikes, how its recommendation window of 8 days lags behind traffic patterns, and why the 'lower bound' mode sometimes recommends requests below actual usage. They walk through a specific case where VPA recommended 512 MiB of memory for a Go service that actually needed 1.2 GiB under peak load, causing repeated OOMKills. The episode closes with practical mitigations: setting custom OOM-scoring thresholds, using VPA in 'initial' mode for batch workloads, and combining VPA with Horizontal Pod Autoscaler using resource metrics. Listeners come away with a clear mental model of VPA's blind spots and how to compensate for them. #Kubernetes #VPA #VerticalPodAutoscaler #ResourceRequests #OOMKill #K8sAutoscaling #ContainerSizing #GoService #PodEviction #HPA #ResourceMetrics #ClusterOps #PerformanceTuning #DevOps #Technology #FexingoBusiness #BusinessPodcast #DevOpsDaily Keep every episode free: buymeacoffee.com/fexingo
    続きを読む 一部表示
    10 分
  • How Kubernetes StatefulSet Rolling Update Breaks Persistent Volume Claims

    How Kubernetes StatefulSet Rolling Update Breaks Persistent Volume Claims
    2026/07/20
    In this episode of DevOps Daily, Lucas and Luna dive into a subtle but destructive Kubernetes pitfall: StatefulSet rolling updates that orphan PersistentVolumeClaims when the pod template changes. Using a real-world incident at a mid-sized fintech, they walk through how a simple image update triggered PVC detachment, stalled the rollout, and left the cluster in an inconsistent state. They explain the underlying mechanics — how StatefulSet's pod identity logic interacts with PVC templates, why the 'OnDelete' update strategy buys time but doesn't fix the root cause, and what monitoring signals to watch for. Lucas shares a concrete remediation: pre-binding PVCs via volumeClaimTemplates with stable names, and using a custom operator to handle canary updates safely. Listeners will learn one specific configuration pattern to prevent this silent failure, plus a diagnostic command to check for orphaned PVCs after any StatefulSet change. This is a focused, practical episode for anyone running stateful workloads on Kubernetes. #Kubernetes #StatefulSet #PersistentVolumeClaim #RollingUpdate #DevOps #CloudNative #StatefulWorkloads #K8sFailure #PodIdentity #VolumeClaimTemplates #OnDeleteStrategy #Fintech #IncidentResponse #KubernetesTroubleshooting #CI/CD #Technology #FexingoBusiness #DevOpsPodcast Keep every episode free: buymeacoffee.com/fexingo
    続きを読む 一部表示
    9 分
  • How Kubernetes Service Mesh mTLS Rotation Causes Pod Restarts

    How Kubernetes Service Mesh mTLS Rotation Causes Pod Restarts
    2026/07/20
    In this episode of DevOps Daily, Lucas and Luna dive into a subtle but dangerous issue in Kubernetes service meshes: automatic mTLS certificate rotation. When the Istio or Linkerd control plane rotates certificates, badly configured sidecar proxies can restart mid-request, causing connection drops, retry storms, and degraded latency. The hosts walk through a real incident at a mid-size fintech where a 24-hour certificate rotation window triggered cascading failures across 200 microservices. They explain the root cause—a mismatch between the certificate refresh interval and the envoy hot-restart mechanism—and the fix: aligning secrets mount propagation with proxy startup probes. Listeners get a concrete checklist to audit their own mesh configurations before the next rotation hits. #Kubernetes #ServiceMesh #mTLS #Istio #Linkerd #CertificateRotation #EnvoyProxy #CI/CD #DevOps #SiteReliabilityEngineering #CloudNative #SidecarProxy #PodRestarts #Security #Technology #FexingoBusiness #BusinessPodcast #DevOpsDaily Keep every episode free: buymeacoffee.com/fexingo
    続きを読む 一部表示
    10 分
  • How Kubernetes PodPreset Drove Operator Config Drift

    How Kubernetes PodPreset Drove Operator Config Drift
    2026/07/19
    In this episode of DevOps Daily with Fexingo, Lucas and Luna dig into a forgotten Kubernetes feature: PodPreset. Once hailed as a way to inject configuration into pods automatically, PodPreset was deprecated in 2022 and removed in Kubernetes 1.25. But many clusters still carry its legacy — and that legacy causes config drift that operators never see coming. Lucas explains how PodPreset worked, why it was retired in favor of admission webhooks and Operators, and how leftover PodPreset admission controller configurations can silently mutate pod specs. He walks through a real incident where a team spent weeks debugging mysterious container restarts, only to trace the root cause to an old PodPreset definition. Luna pushes on whether the Kubernetes ecosystem moves too fast for production stability. The episode ends with a forward-looking question: as Kubernetes sheds beta APIs and legacy features every release, are operators keeping up with the removal notices buried in release notes? If today's tech conversation gave you something usable, support the show at buy me a coffee dot com slash fexingo. #Kubernetes #PodPreset #ConfigDrift #DevOps #CloudNative #K8s #AdmissionWebhook #OperatorPattern #KubernetesDeprecation #KubernetesMigration #KubernetesReleaseNotes #KubernetesBetaAPI #KubernetesConfigManagement #CI/CD #Technology #FexingoBusiness #BusinessPodcast #DevOpsDaily Keep every episode free: buymeacoffee.com/fexingo
    続きを読む 一部表示
    10 分
  • How Kubernetes PriorityClass Starvation Blocks Node-Level Scheduling

    How Kubernetes PriorityClass Starvation Blocks Node-Level Scheduling
    2026/07/19
    In this episode of DevOps Daily with Fexingo, Lucas and Luna dive into a subtle but severe Kubernetes anti-pattern: PriorityClass starvation at the node level. They break down a real-world case where a team set a critical batch job's priority too high, causing it to preempt essential daemonsets and system pods, leading to node instability and failed scheduling. Lucas explains how the Kubernetes scheduler's preemption logic interacts with kubelet's resource allocation, why setting priorities without understanding the full hierarchy can backfire, and the three metrics you should monitor — preemption counts, pod eviction rates, and node pressure conditions — to catch starvation before it takes down your cluster. Luna pushes on whether kubectl describe node gives enough information, and they walk through a practical audit using PriorityClass objects and pod spec priority fields. If you've ever wondered why nodes look healthy but pods won't land, this episode gives you the debugging framework. No fluff, just the edge cases that break production. #Kubernetes #PriorityClass #PodPreemption #Scheduling #DevOps #CloudNative #NodeStarvation #ClusterReliability #kubelet #BatchJobs #DaemonSet #ResourceManagement #KubernetesScheduler #PodEviction #NodePressure #Technology #FexingoBusiness #BusinessPodcast Keep every episode free: buymeacoffee.com/fexingo
    続きを読む 一部表示
    11 分
  • Why Kubernetes Ingress Controller Sticky Sessions Break Under Rolling Updates

    Why Kubernetes Ingress Controller Sticky Sessions Break Under Rolling Updates
    2026/07/18
    In this episode of DevOps Daily, Lucas and Luna investigate a deceptively common Kubernetes failure: when ingress controller session affinity (sticky sessions) silently breaks during rolling updates. They walk through a real incident at a mid-size e-commerce platform where users were suddenly logged out mid-session after a routine deployment. Lucas explains the mechanics — how NGINX ingress controller uses Pod IP hashing by default, and how a new replica set flips the hash mapping, redirecting users to pods that have never seen their session data. Luna pushes on whether the fix is a readiness probe tweak, an external session store, or a different ingress controller. They also discuss when to use sticky sessions at all, and why session affinity is often a workaround for a deeper architectural issue. Concrete, specific, and immediately applicable for anyone running Kubernetes in production. #Kubernetes #IngressController #StickySessions #SessionAffinity #RollingUpdate #NGINXIngress #DevOps #SiteReliabilityEngineering #PodLifecycle #SessionManagement #K8sNetworking #ProductionIncident #ECommerce #LoadBalancing #ReadinessProbe #Technology #FexingoBusiness #BusinessPodcast Keep every episode free: buymeacoffee.com/fexingo
    続きを読む 一部表示
    9 分
  • How Kubernetes Audit Logs Expose Silent Security Threats

    How Kubernetes Audit Logs Expose Silent Security Threats
    2026/07/18
    In this episode of DevOps Daily, Lucas and Luna dive deep into Kubernetes audit logs—a powerful but often overlooked tool for detecting silent security threats. They walk through a real-world scenario where a cluster was compromised via a misconfigured webhook, and the only clue was buried in the audit log. They explain how to enable audit logging, what to look for (like unexpected impersonation attempts or suspicious RBAC escalations), and why most teams don't parse these logs until it's too late. The hosts also discuss how audit logs can be integrated with SIEM tools for automated alerting, and share a practical tip: watch for repeated 403 errors—they might indicate an attacker probing your cluster. If you're running Kubernetes in production, this episode will change how you think about observability. #Kubernetes #AuditLogs #Security #CloudNative #K8s #DevOps #SIEM #RBAC #ThreatDetection #Webhook #Infrastructure #Observability #Technology #FexingoBusiness #BusinessPodcast #DevOpsDaily #Podcast #CyberSecurity Keep every episode free: buymeacoffee.com/fexingo
    続きを読む 一部表示
    11 分