A one-line config change, swapping a pinned AMI version to latest, looks like a security win. Your nodes will always run the newest, most patched AMI. Except when they won't.

This post walks through how a change meant to track the latest EKS-optimized AMI can leave your most critical nodes stuck on an old, unpatched version. There are no alerts, so nothing looks wrong until you check node versions by hand.

It covers why this happens, why the config that causes it exists for good reason, and what options you have to get both security and stability.

Prerequisites

If you're already familiar with Karpenter, NodePools, and disruption budgets, skip ahead to The harmless-looking commit

Karpenter is a Kubernetes node autoscaler for AWS. Unlike traditional managed nodegroups with fixed Auto Scaling Groups, Karpenter watches for pods that can't be scheduled and provisions right-sized EC2 instances on the fly. When those nodes are no longer needed, it removes them.

Key concepts:

• NodePool: defines what kind of nodes can be created (instance types, capacity type, zones) and how they can be disrupted (removed/replaced).

• EC2NodeClass: defines the AMI, networking, and boot configuration for nodes.

• NodeClaim: Karpenter's record of a provisioned node.

• Drift: when a running node no longer matches the desired spec e.g. the AMI has changed. Karpenter detects this automatically.

• Disruption: any action that removes or replaces a node, including consolidation, drift replacement, and expiry.

Why multiple node groups exist

A typical EKS cluster has multiple tiers of compute:

The separation exists because different workloads have different reliability requirements. A batch job can handle being rescheduled. A CI pipeline halfway through a deployment cannot.

Disruption budgets

Karpenter's disruption budgets control how many nodes it can disrupt at once, and for what reason:

disruption:
  budgets:
    - nodes: "0"
      reasons:
      - "Underutilized"
      - "Drifted"
    - nodes: "20%"
      reasons:
      - "Empty"

This says:

• Zero nodes may be disrupted for being underutilized or drifted (complete protection).

• 20% of nodes may be disrupted if they're completely empty (safe: no workloads affected).

Why budget=0 for Drifted makes sense

For the ondemand pool, blocking drift replacement protects against:

1. Mid-flight CI/CD failures: if a runner is mid-pipeline and Karpenter drains the node, the job fails.

2. Loss of monitoring: if the observability stack gets restarted, you lose visibility during the transition.

3. Uncontrolled timing: AWS releases AMIs at unpredictable times. Budget=0 means you choose when to take disruption.

4. Cascading failures: if PodDisruptionBudgets aren't perfectly configured, node replacement can cause more disruption than expected.

The intent is clear: ondemand nodes should only be disrupted when a human decides it's safe.

The harmless-looking commit

Consider this one-line change to the EC2NodeClass config:

- amiVersion: v20251209
+ amiVersion: latest

This reads as a straightforward security improvement. Instead of being stuck on a pinned AMI that requires a manual PR to bump, the cluster now always targets the latest EKS-optimized AMI. New nodes will always be secure.

The problem shows up on the nodes that are already running.

What actually happens

Here's a concrete timeline of how this plays out:

• April 20, 12:00 PM - Workload scheduled on ondemand pool. Karpenter launches a node with AMI v20251209 (the latest at the time), running kubelet v1.34.2.

• June 5, 12:30 PM - Config deployed: amiVersion changed from pinned v20251209 to latest. At this point, latest resolves to AMI v20260526 (kubelet v1.34.8).

• June 5, 12:31 PM - Karpenter detects the mismatch. Node AMI (v20251209) != latest AMI (v20260526). Node marked Drifted: True with reason AMIDrift.

• June 5, 12:32 PM - Karpenter checks the disruption budget. Budget = 0 for Drifted. Replacement blocked.

• June 5, 12:32 PM onward - Node stays on the old AMI. 46 days behind on security patches. No alert, no notification.

Generalizing this, the flow is:

Fig 1: The same drift is fixed in the default pool and blocked in the ondemand pool.

• For the default pool (permissive budget), Karpenter replaces the nodes immediately. They get the new AMI. No problem.

• For the ondemand pool (nodes: "0" for Drifted), Karpenter marks the node as drifted but cannot replace it. The node stays on the old AMI indefinitely.

The result: a node running your most critical workloads is permanently stuck on an old, unpatched AMI. The exact opposite of what "latest" was supposed to achieve.

Why this isn't a one-time problem

AWS releases new EKS-optimized AMIs every few weeks for security patches and kubelet updates. Each release triggers the same cycle:

Fig 2: Every AMI release repeats the cycle, so the protected node falls further behind.

1. Shifts what latest resolves to.

2. Causes Karpenter to mark all existing ondemand nodes as drifted.

3. The budget blocks replacement.

4. The node falls further behind on patches.

As long as the ondemand node has running workloads (which it always will: that's why it exists), it will never become "Empty" and never be eligible for the only allowed disruption reason.

The node you're protecting the most is the node that falls furthest behind on security patches.

The contradiction

The configuration now says two contradictory things:

• "Always target the latest AMI" → security intent

• "Never replace a node for drift" → stability intent

For new nodes created during scale-up, latest works: they get the newest AMI. But existing nodes, provisioned before the latest AMI release, stay on whatever AMI they launched with.

You get latest on creation, fixed after that.

What manual intervention looks like

To actually update the stuck node, someone must:

Fig 3: Without automation, someone repeats this loop after every AMI release.

1. Patch the NodePool to temporarily allow drift replacement.

2. Wait for Karpenter to provision a new node, drain the old one, and terminate it.

3. Revert the patch to restore the protective budget.

This is the same amount of effort as the old pinned approach (bump version in a PR, deploy), but without the clear trigger of a version change in git. The drift builds up without a signal: you only notice when you inspect node versions and find one lagging weeks or months behind.

Approaches to consider

Pin the AMI version (the original approach)

Keep amiVersion set to a specific version e.g. v20260526. Bump it deliberately via PR when ready.

Latest + budget=0 (the current trap)

Keep amiVersion: latest with nodes: "0" for Drifted.

Latest + scheduled maintenance window

Keep amiVersion: latest but add a time-based budget that allows drift replacement during off-peak hours:

budgets:
  - nodes: "0"
    reasons: ["Drifted"]
  - nodes: "1"
    reasons: ["Drifted"]
    schedule: "0 2 * * MON"
    duration

Outside the window: fully protected. During the window: Karpenter replaces one drifted node at a time.

Latest + budget=1 (fully automatic)

Keep amiVersion: latest with nodes: "1" for Drifted. Karpenter replaces immediately.

The takeaway

amiVersion: latest is not a complete security solution for node pools with strict disruption budgets. It only guarantees that new nodes are on the latest AMI. For existing nodes, the disruption budget determines whether they actually get updated.

Before switching a pinned AMI to latest, ask one question: can Karpenter actually replace the drifted node? If the budget says no, you've written a config that claims to patch your most critical nodes but never will.

If you liked this post, you might find these interesting too

• Why your EKS Auto Mode NodeClass keeps reverting to default

• Gotchas to avoid when using spot instances in production

• GitHub runners basics and self-hosted runner setup