Services

Resources

Company

Why Karpenter isn’t updating your node AMIs

Why Karpenter isn’t updating your node AMIs

Why Karpenter isn’t updating your node AMIs

Why Karpenter isn’t updating your node AMIs

In an EKS cluster where Karpenter creates EC2 nodes, a common best practice is to set Karpenter’s AMI version to latest. Karpenter then uses the updated EKS-optimized AMI and its security patches when it creates a node.

But Karpenter checks this setting only when creating a node. To update a running node, it must replace the node and move its workloads. If your safety settings block replacement, the node remains unpatched.

This is easy to miss because the configuration looks correct and Karpenter sends no alert. Critical nodes can remain unpatched while new nodes run the latest AMI. This post explains how that happens and compares four ways to balance AMI updates against workload disruption.

Prerequisites

If you’re already familiar with Karpenter, NodePools, and disruption budgets, skip ahead to How disruption budgets protect workloads.

Karpenter is a Kubernetes node autoscaler for AWS. Unlike a managed node group backed by a fixed Auto Scaling Group, Karpenter watches for pods that cannot be scheduled and creates suitable EC2 instances as needed. It removes those nodes when they are no longer required.

This post uses the following Karpenter concepts:

  • A NodePool defines which nodes Karpenter can create, including their instance types, capacity types, and availability zones. It also controls how Karpenter may disrupt them.

  • An EC2NodeClass defines the AMI, networking, and boot configuration for those nodes.

  • A NodeClaim is Karpenter’s record of a provisioned node.

  • Drift means that a running node no longer matches its desired configuration. An AMI change is one cause of drift.

  • Disruption is any action that removes or replaces a node. It includes consolidation, drift replacement, and expiry.

Why clusters use different groups of nodes

A typical EKS cluster separates its node groups based on workload reliability requirements. A batch job can usually tolerate being rescheduled, but a CI runner halfway through a deployment cannot.

Tier

Managed by

Role

Managed node group

AWS EKS

The cluster’s fixed bootstrap node group. It exists before Karpenter starts and runs system pods such as Karpenter, the CNI, and CoreDNS.

Karpenter default pool

Karpenter

A flexible group for general workloads. Karpenter may replace its nodes when they are underutilized, drifted, or expired.

Karpenter ondemand pool

Karpenter

A protected group for critical workloads. It runs services such as CI/CD runners and monitoring on on-demand instances with strict disruption controls.

How disruption budgets protect workloads

Karpenter’s disruption budgets control how many nodes it may disrupt at once and which reasons permit that disruption:

disruption:
  budgets:
    - nodes: "0"
      reasons:
      - "Underutilized"
      - "Drifted"
    - nodes: "20%"
      reasons:
      - "Empty"

The two rules apply to different situations:

  • nodes: "0" keeps underutilized nodes with spare capacity running. It also keeps drifted nodes running when their AMI or other settings no longer match the configured values.

  • nodes: "20%" for Empty means Karpenter can remove up to 20% of nodes that have no workloads running on them. Because these nodes are empty, Karpenter does not need to move any workloads first.

The zero-node rule blocks replacement until the team temporarily changes the budget during a maintenance window.

The harmless-looking commit

Consider this one-line change to the EC2NodeClass configuration:

- amiVersion: v20251209
+ amiVersion: latest

With the pinned setting, Karpenter keeps using v20251209 for new nodes until the team changes the version in a pull request. With latest, Karpenter selects the newest EKS-optimized AMI available when it creates a node.

This selection applies only to new nodes. A running node receives the new AMI only after Karpenter replaces it.

Upgrade timeline

Consider this timeline:

  • April 20, 12:00 PM

    • A workload needs a node in the ondemand pool.

    • Karpenter creates the node with AMI v20251209, the latest version at the time.

    • The AMI contains kubelet v1.34.2.

  • June 5, 12:30 PM

    • The team deploys the configuration change.

    • amiVersion changes from v20251209 to latest.

    • The latest AMI is now v20260526, which contains kubelet v1.34.8.

  • June 5, 12:31 PM

    • Karpenter compares the running node with the configured AMI.

    • The versions differ, so Karpenter marks the NodeClaim as Drifted: True with reason AMIDrift.

  • June 5, 12:32 PM

    • Karpenter checks the disruption budget.

    • The budget allows zero replacements for Drifted nodes, so the old node stays in place.

  • After June 5

    • The node remains on AMI v20251209, which is 46 days older than v20260526.

    • Karpenter sends no alert or notification about the blocked replacement.

Flowchart: after amiVersion is set to latest, a node not on the latest AMI is marked drifted. In the default pool it is replaced with the new AMI; in the ondemand pool, a drift budget of 0 blocks replacement and the node stays on the old AMI.
Fig 1: The same drift is fixed in the default pool and blocked in the ondemand pool.

This cluster has two Karpenter pools with different replacement rules. The default pool runs general workloads and allows Karpenter to replace drifted nodes. The ondemand pool runs critical workloads and sets its drift budget to zero.

When the AMI changes, Karpenter replaces drifted nodes in the default pool with nodes that use the new AMI. It also detects drift in the ondemand pool, but the zero-node budget keeps those nodes on the AMI they had at launch.

The security change therefore updates new nodes and replaceable nodes, but it does not update protected nodes that are already running.

Each AMI release increases the gap

AWS releases new EKS-optimized AMIs every few weeks with security patches and kubelet updates. Each release changes what latest resolves to, and Karpenter marks existing ondemand nodes as drifted again. The disruption budget still blocks their replacement.

A loop: AWS releases a new AMI, latest resolves to it, Karpenter detects the mismatch and marks the node drifted, a budget of 0 blocks replacement, and the node falls one more release behind.
Fig 2: Each AMI release repeats the cycle, so the protected node falls further behind.

An ondemand node may eventually qualify for replacement under the Empty budget, but only if all its workloads leave. A node dedicated to long-running CI/CD or monitoring workloads may never become empty. It can remain on its original AMI through several releases.

latest applies when Karpenter creates a node, not throughout the node’s lifetime.

Updating a protected node by hand

Someone must temporarily change the NodePool budget to update a node that cannot become empty:

  1. Patch the NodePool to allow at least one drift replacement.

  2. Karpenter responds by creating a node with the latest AMI, moving the workloads to it, and terminating the old node. Confirm that this process finishes.

  3. Restore the zero-node budget for drifted nodes.

A loop of manual steps: notice a node on an old AMI, patch the NodePool to a drifted budget of 1, Karpenter launches a new node, the old node is drained and terminated, revert the budget to 0, and repeat after the next AMI release.
Fig 3: Without automation, the team must repeat this process after every AMI release.

This requires about the same manual work as bumping a pinned AMI in a pull request. The pinned approach at least puts the required action in Git. With latest, drift can accumulate unless someone checks the NodeClaims or compares the AMIs used by running nodes.

Four approaches to keep node AMIs updated

1. Pin the AMI version

Set amiVersion: v20260526 to pin the AMI version. Set nodes: "0" for the Drifted reason to block automatic replacement of existing nodes.

Pros

Cons

A pull request controls which AMI version the cluster targets.

Every AMI update requires a manual pull request.

Each version change has a clear Git audit trail.

The pinned version can become stale if nobody updates it.

Drift starts only after the version changes in Git.

New nodes use the stale pinned version until that change happens.

The team must temporarily raise the drift budget to replace existing nodes.

2. Use latest with a zero drift budget

Set amiVersion: latest to select the newest AMI. Set nodes: "0" for the Drifted reason to block automatic replacement of existing nodes.

Pros

Cons

New nodes receive the latest AMI.

Existing nodes do not update automatically.

The team does not need to update amiVersion for each AMI release.

Drift can build up without an alert or pull request.

Karpenter cannot cause an unexpected drift replacement.

The team must temporarily raise the drift budget to replace existing nodes.

This combination is reasonable only if the team monitors AMI drift and has a separate process for replacing protected nodes. Without that process, it is the trap described in this post.

3. Use latest with a maintenance window

Set amiVersion: latest to select the newest AMI for new nodes. Set nodes: "1" for the Drifted reason during a scheduled window so Karpenter can replace one drifted node at a time:

budgets:
  - nodes: "0"
    reasons: ["Drifted"]
  - nodes: "1"
    reasons: ["Drifted"]
    schedule: "0 2 * * MON"
    duration

The drift budget is zero outside the window, so existing drifted nodes keep running. During the window, the budget is one, so Karpenter may replace one drifted node at a time.

Pros

Cons

Existing nodes update without a manual budget change.

Replacement still disrupts workloads during the window.

The team controls when replacements can happen.

Several drifted nodes may take several windows to replace.

New nodes receive the latest AMI.

The workloads still need correct PodDisruptionBudgets.

A node can remain behind until the next maintenance window.

4. Use latest with a one-node drift budget

Set amiVersion: latest to select the newest AMI for new nodes. Set nodes: "1" for the Drifted reason with no schedule so Karpenter can replace one drifted node at any time.

Pros

Cons

Existing nodes update as soon as Karpenter detects drift.

The team does not control when replacement starts.

No manual budget change is required.

Replacement may start during peak traffic or an incident.

New nodes receive the latest AMI.

Safe rollout depends on the workloads’ PodDisruptionBudgets and rescheduling behavior.

Only one drifted node is replaced at a time.

AWS AMI release timing determines the disruption schedule.

Conclusion

Before changing a pinned AMI to latest, check your NodePool’s disruption budget:

  • amiVersion selects the AMI for new nodes.

  • The disruption budget controls whether Karpenter can replace existing nodes when their AMI drifts.

  • With nodes: "0" for Drifted, existing nodes stay on their old AMI until the team raises the budget.

Keep the AMI pinned for explicit version control, schedule a maintenance window for automatic replacement at a known time, or allow one drift replacement at any time. If you use latest with a zero drift budget, monitor drift and document the manual replacement process.

If you liked this post, you might find these interesting too

In an EKS cluster where Karpenter creates EC2 nodes, a common best practice is to set Karpenter’s AMI version to latest. Karpenter then uses the updated EKS-optimized AMI and its security patches when it creates a node.

But Karpenter checks this setting only when creating a node. To update a running node, it must replace the node and move its workloads. If your safety settings block replacement, the node remains unpatched.

This is easy to miss because the configuration looks correct and Karpenter sends no alert. Critical nodes can remain unpatched while new nodes run the latest AMI. This post explains how that happens and compares four ways to balance AMI updates against workload disruption.

Prerequisites

If you’re already familiar with Karpenter, NodePools, and disruption budgets, skip ahead to How disruption budgets protect workloads.

Karpenter is a Kubernetes node autoscaler for AWS. Unlike a managed node group backed by a fixed Auto Scaling Group, Karpenter watches for pods that cannot be scheduled and creates suitable EC2 instances as needed. It removes those nodes when they are no longer required.

This post uses the following Karpenter concepts:

  • A NodePool defines which nodes Karpenter can create, including their instance types, capacity types, and availability zones. It also controls how Karpenter may disrupt them.

  • An EC2NodeClass defines the AMI, networking, and boot configuration for those nodes.

  • A NodeClaim is Karpenter’s record of a provisioned node.

  • Drift means that a running node no longer matches its desired configuration. An AMI change is one cause of drift.

  • Disruption is any action that removes or replaces a node. It includes consolidation, drift replacement, and expiry.

Why clusters use different groups of nodes

A typical EKS cluster separates its node groups based on workload reliability requirements. A batch job can usually tolerate being rescheduled, but a CI runner halfway through a deployment cannot.

Tier

Managed by

Role

Managed node group

AWS EKS

The cluster’s fixed bootstrap node group. It exists before Karpenter starts and runs system pods such as Karpenter, the CNI, and CoreDNS.

Karpenter default pool

Karpenter

A flexible group for general workloads. Karpenter may replace its nodes when they are underutilized, drifted, or expired.

Karpenter ondemand pool

Karpenter

A protected group for critical workloads. It runs services such as CI/CD runners and monitoring on on-demand instances with strict disruption controls.

How disruption budgets protect workloads

Karpenter’s disruption budgets control how many nodes it may disrupt at once and which reasons permit that disruption:

disruption:
  budgets:
    - nodes: "0"
      reasons:
      - "Underutilized"
      - "Drifted"
    - nodes: "20%"
      reasons:
      - "Empty"

The two rules apply to different situations:

  • nodes: "0" keeps underutilized nodes with spare capacity running. It also keeps drifted nodes running when their AMI or other settings no longer match the configured values.

  • nodes: "20%" for Empty means Karpenter can remove up to 20% of nodes that have no workloads running on them. Because these nodes are empty, Karpenter does not need to move any workloads first.

The zero-node rule blocks replacement until the team temporarily changes the budget during a maintenance window.

The harmless-looking commit

Consider this one-line change to the EC2NodeClass configuration:

- amiVersion: v20251209
+ amiVersion: latest

With the pinned setting, Karpenter keeps using v20251209 for new nodes until the team changes the version in a pull request. With latest, Karpenter selects the newest EKS-optimized AMI available when it creates a node.

This selection applies only to new nodes. A running node receives the new AMI only after Karpenter replaces it.

Upgrade timeline

Consider this timeline:

  • April 20, 12:00 PM

    • A workload needs a node in the ondemand pool.

    • Karpenter creates the node with AMI v20251209, the latest version at the time.

    • The AMI contains kubelet v1.34.2.

  • June 5, 12:30 PM

    • The team deploys the configuration change.

    • amiVersion changes from v20251209 to latest.

    • The latest AMI is now v20260526, which contains kubelet v1.34.8.

  • June 5, 12:31 PM

    • Karpenter compares the running node with the configured AMI.

    • The versions differ, so Karpenter marks the NodeClaim as Drifted: True with reason AMIDrift.

  • June 5, 12:32 PM

    • Karpenter checks the disruption budget.

    • The budget allows zero replacements for Drifted nodes, so the old node stays in place.

  • After June 5

    • The node remains on AMI v20251209, which is 46 days older than v20260526.

    • Karpenter sends no alert or notification about the blocked replacement.

Flowchart: after amiVersion is set to latest, a node not on the latest AMI is marked drifted. In the default pool it is replaced with the new AMI; in the ondemand pool, a drift budget of 0 blocks replacement and the node stays on the old AMI.
Fig 1: The same drift is fixed in the default pool and blocked in the ondemand pool.

This cluster has two Karpenter pools with different replacement rules. The default pool runs general workloads and allows Karpenter to replace drifted nodes. The ondemand pool runs critical workloads and sets its drift budget to zero.

When the AMI changes, Karpenter replaces drifted nodes in the default pool with nodes that use the new AMI. It also detects drift in the ondemand pool, but the zero-node budget keeps those nodes on the AMI they had at launch.

The security change therefore updates new nodes and replaceable nodes, but it does not update protected nodes that are already running.

Each AMI release increases the gap

AWS releases new EKS-optimized AMIs every few weeks with security patches and kubelet updates. Each release changes what latest resolves to, and Karpenter marks existing ondemand nodes as drifted again. The disruption budget still blocks their replacement.

A loop: AWS releases a new AMI, latest resolves to it, Karpenter detects the mismatch and marks the node drifted, a budget of 0 blocks replacement, and the node falls one more release behind.
Fig 2: Each AMI release repeats the cycle, so the protected node falls further behind.

An ondemand node may eventually qualify for replacement under the Empty budget, but only if all its workloads leave. A node dedicated to long-running CI/CD or monitoring workloads may never become empty. It can remain on its original AMI through several releases.

latest applies when Karpenter creates a node, not throughout the node’s lifetime.

Updating a protected node by hand

Someone must temporarily change the NodePool budget to update a node that cannot become empty:

  1. Patch the NodePool to allow at least one drift replacement.

  2. Karpenter responds by creating a node with the latest AMI, moving the workloads to it, and terminating the old node. Confirm that this process finishes.

  3. Restore the zero-node budget for drifted nodes.

A loop of manual steps: notice a node on an old AMI, patch the NodePool to a drifted budget of 1, Karpenter launches a new node, the old node is drained and terminated, revert the budget to 0, and repeat after the next AMI release.
Fig 3: Without automation, the team must repeat this process after every AMI release.

This requires about the same manual work as bumping a pinned AMI in a pull request. The pinned approach at least puts the required action in Git. With latest, drift can accumulate unless someone checks the NodeClaims or compares the AMIs used by running nodes.

Four approaches to keep node AMIs updated

1. Pin the AMI version

Set amiVersion: v20260526 to pin the AMI version. Set nodes: "0" for the Drifted reason to block automatic replacement of existing nodes.

Pros

Cons

A pull request controls which AMI version the cluster targets.

Every AMI update requires a manual pull request.

Each version change has a clear Git audit trail.

The pinned version can become stale if nobody updates it.

Drift starts only after the version changes in Git.

New nodes use the stale pinned version until that change happens.

The team must temporarily raise the drift budget to replace existing nodes.

2. Use latest with a zero drift budget

Set amiVersion: latest to select the newest AMI. Set nodes: "0" for the Drifted reason to block automatic replacement of existing nodes.

Pros

Cons

New nodes receive the latest AMI.

Existing nodes do not update automatically.

The team does not need to update amiVersion for each AMI release.

Drift can build up without an alert or pull request.

Karpenter cannot cause an unexpected drift replacement.

The team must temporarily raise the drift budget to replace existing nodes.

This combination is reasonable only if the team monitors AMI drift and has a separate process for replacing protected nodes. Without that process, it is the trap described in this post.

3. Use latest with a maintenance window

Set amiVersion: latest to select the newest AMI for new nodes. Set nodes: "1" for the Drifted reason during a scheduled window so Karpenter can replace one drifted node at a time:

budgets:
  - nodes: "0"
    reasons: ["Drifted"]
  - nodes: "1"
    reasons: ["Drifted"]
    schedule: "0 2 * * MON"
    duration

The drift budget is zero outside the window, so existing drifted nodes keep running. During the window, the budget is one, so Karpenter may replace one drifted node at a time.

Pros

Cons

Existing nodes update without a manual budget change.

Replacement still disrupts workloads during the window.

The team controls when replacements can happen.

Several drifted nodes may take several windows to replace.

New nodes receive the latest AMI.

The workloads still need correct PodDisruptionBudgets.

A node can remain behind until the next maintenance window.

4. Use latest with a one-node drift budget

Set amiVersion: latest to select the newest AMI for new nodes. Set nodes: "1" for the Drifted reason with no schedule so Karpenter can replace one drifted node at any time.

Pros

Cons

Existing nodes update as soon as Karpenter detects drift.

The team does not control when replacement starts.

No manual budget change is required.

Replacement may start during peak traffic or an incident.

New nodes receive the latest AMI.

Safe rollout depends on the workloads’ PodDisruptionBudgets and rescheduling behavior.

Only one drifted node is replaced at a time.

AWS AMI release timing determines the disruption schedule.

Conclusion

Before changing a pinned AMI to latest, check your NodePool’s disruption budget:

  • amiVersion selects the AMI for new nodes.

  • The disruption budget controls whether Karpenter can replace existing nodes when their AMI drifts.

  • With nodes: "0" for Drifted, existing nodes stay on their old AMI until the team raises the budget.

Keep the AMI pinned for explicit version control, schedule a maintenance window for automatic replacement at a known time, or allow one drift replacement at any time. If you use latest with a zero drift budget, monitor drift and document the manual replacement process.

If you liked this post, you might find these interesting too

In an EKS cluster where Karpenter creates EC2 nodes, a common best practice is to set Karpenter’s AMI version to latest. Karpenter then uses the updated EKS-optimized AMI and its security patches when it creates a node.

But Karpenter checks this setting only when creating a node. To update a running node, it must replace the node and move its workloads. If your safety settings block replacement, the node remains unpatched.

This is easy to miss because the configuration looks correct and Karpenter sends no alert. Critical nodes can remain unpatched while new nodes run the latest AMI. This post explains how that happens and compares four ways to balance AMI updates against workload disruption.

Prerequisites

If you’re already familiar with Karpenter, NodePools, and disruption budgets, skip ahead to How disruption budgets protect workloads.

Karpenter is a Kubernetes node autoscaler for AWS. Unlike a managed node group backed by a fixed Auto Scaling Group, Karpenter watches for pods that cannot be scheduled and creates suitable EC2 instances as needed. It removes those nodes when they are no longer required.

This post uses the following Karpenter concepts:

  • A NodePool defines which nodes Karpenter can create, including their instance types, capacity types, and availability zones. It also controls how Karpenter may disrupt them.

  • An EC2NodeClass defines the AMI, networking, and boot configuration for those nodes.

  • A NodeClaim is Karpenter’s record of a provisioned node.

  • Drift means that a running node no longer matches its desired configuration. An AMI change is one cause of drift.

  • Disruption is any action that removes or replaces a node. It includes consolidation, drift replacement, and expiry.

Why clusters use different groups of nodes

A typical EKS cluster separates its node groups based on workload reliability requirements. A batch job can usually tolerate being rescheduled, but a CI runner halfway through a deployment cannot.

Tier

Managed by

Role

Managed node group

AWS EKS

The cluster’s fixed bootstrap node group. It exists before Karpenter starts and runs system pods such as Karpenter, the CNI, and CoreDNS.

Karpenter default pool

Karpenter

A flexible group for general workloads. Karpenter may replace its nodes when they are underutilized, drifted, or expired.

Karpenter ondemand pool

Karpenter

A protected group for critical workloads. It runs services such as CI/CD runners and monitoring on on-demand instances with strict disruption controls.

How disruption budgets protect workloads

Karpenter’s disruption budgets control how many nodes it may disrupt at once and which reasons permit that disruption:

disruption:
  budgets:
    - nodes: "0"
      reasons:
      - "Underutilized"
      - "Drifted"
    - nodes: "20%"
      reasons:
      - "Empty"

The two rules apply to different situations:

  • nodes: "0" keeps underutilized nodes with spare capacity running. It also keeps drifted nodes running when their AMI or other settings no longer match the configured values.

  • nodes: "20%" for Empty means Karpenter can remove up to 20% of nodes that have no workloads running on them. Because these nodes are empty, Karpenter does not need to move any workloads first.

The zero-node rule blocks replacement until the team temporarily changes the budget during a maintenance window.

The harmless-looking commit

Consider this one-line change to the EC2NodeClass configuration:

- amiVersion: v20251209
+ amiVersion: latest

With the pinned setting, Karpenter keeps using v20251209 for new nodes until the team changes the version in a pull request. With latest, Karpenter selects the newest EKS-optimized AMI available when it creates a node.

This selection applies only to new nodes. A running node receives the new AMI only after Karpenter replaces it.

Upgrade timeline

Consider this timeline:

  • April 20, 12:00 PM

    • A workload needs a node in the ondemand pool.

    • Karpenter creates the node with AMI v20251209, the latest version at the time.

    • The AMI contains kubelet v1.34.2.

  • June 5, 12:30 PM

    • The team deploys the configuration change.

    • amiVersion changes from v20251209 to latest.

    • The latest AMI is now v20260526, which contains kubelet v1.34.8.

  • June 5, 12:31 PM

    • Karpenter compares the running node with the configured AMI.

    • The versions differ, so Karpenter marks the NodeClaim as Drifted: True with reason AMIDrift.

  • June 5, 12:32 PM

    • Karpenter checks the disruption budget.

    • The budget allows zero replacements for Drifted nodes, so the old node stays in place.

  • After June 5

    • The node remains on AMI v20251209, which is 46 days older than v20260526.

    • Karpenter sends no alert or notification about the blocked replacement.

Flowchart: after amiVersion is set to latest, a node not on the latest AMI is marked drifted. In the default pool it is replaced with the new AMI; in the ondemand pool, a drift budget of 0 blocks replacement and the node stays on the old AMI.
Fig 1: The same drift is fixed in the default pool and blocked in the ondemand pool.

This cluster has two Karpenter pools with different replacement rules. The default pool runs general workloads and allows Karpenter to replace drifted nodes. The ondemand pool runs critical workloads and sets its drift budget to zero.

When the AMI changes, Karpenter replaces drifted nodes in the default pool with nodes that use the new AMI. It also detects drift in the ondemand pool, but the zero-node budget keeps those nodes on the AMI they had at launch.

The security change therefore updates new nodes and replaceable nodes, but it does not update protected nodes that are already running.

Each AMI release increases the gap

AWS releases new EKS-optimized AMIs every few weeks with security patches and kubelet updates. Each release changes what latest resolves to, and Karpenter marks existing ondemand nodes as drifted again. The disruption budget still blocks their replacement.

A loop: AWS releases a new AMI, latest resolves to it, Karpenter detects the mismatch and marks the node drifted, a budget of 0 blocks replacement, and the node falls one more release behind.
Fig 2: Each AMI release repeats the cycle, so the protected node falls further behind.

An ondemand node may eventually qualify for replacement under the Empty budget, but only if all its workloads leave. A node dedicated to long-running CI/CD or monitoring workloads may never become empty. It can remain on its original AMI through several releases.

latest applies when Karpenter creates a node, not throughout the node’s lifetime.

Updating a protected node by hand

Someone must temporarily change the NodePool budget to update a node that cannot become empty:

  1. Patch the NodePool to allow at least one drift replacement.

  2. Karpenter responds by creating a node with the latest AMI, moving the workloads to it, and terminating the old node. Confirm that this process finishes.

  3. Restore the zero-node budget for drifted nodes.

A loop of manual steps: notice a node on an old AMI, patch the NodePool to a drifted budget of 1, Karpenter launches a new node, the old node is drained and terminated, revert the budget to 0, and repeat after the next AMI release.
Fig 3: Without automation, the team must repeat this process after every AMI release.

This requires about the same manual work as bumping a pinned AMI in a pull request. The pinned approach at least puts the required action in Git. With latest, drift can accumulate unless someone checks the NodeClaims or compares the AMIs used by running nodes.

Four approaches to keep node AMIs updated

1. Pin the AMI version

Set amiVersion: v20260526 to pin the AMI version. Set nodes: "0" for the Drifted reason to block automatic replacement of existing nodes.

Pros

Cons

A pull request controls which AMI version the cluster targets.

Every AMI update requires a manual pull request.

Each version change has a clear Git audit trail.

The pinned version can become stale if nobody updates it.

Drift starts only after the version changes in Git.

New nodes use the stale pinned version until that change happens.

The team must temporarily raise the drift budget to replace existing nodes.

2. Use latest with a zero drift budget

Set amiVersion: latest to select the newest AMI. Set nodes: "0" for the Drifted reason to block automatic replacement of existing nodes.

Pros

Cons

New nodes receive the latest AMI.

Existing nodes do not update automatically.

The team does not need to update amiVersion for each AMI release.

Drift can build up without an alert or pull request.

Karpenter cannot cause an unexpected drift replacement.

The team must temporarily raise the drift budget to replace existing nodes.

This combination is reasonable only if the team monitors AMI drift and has a separate process for replacing protected nodes. Without that process, it is the trap described in this post.

3. Use latest with a maintenance window

Set amiVersion: latest to select the newest AMI for new nodes. Set nodes: "1" for the Drifted reason during a scheduled window so Karpenter can replace one drifted node at a time:

budgets:
  - nodes: "0"
    reasons: ["Drifted"]
  - nodes: "1"
    reasons: ["Drifted"]
    schedule: "0 2 * * MON"
    duration

The drift budget is zero outside the window, so existing drifted nodes keep running. During the window, the budget is one, so Karpenter may replace one drifted node at a time.

Pros

Cons

Existing nodes update without a manual budget change.

Replacement still disrupts workloads during the window.

The team controls when replacements can happen.

Several drifted nodes may take several windows to replace.

New nodes receive the latest AMI.

The workloads still need correct PodDisruptionBudgets.

A node can remain behind until the next maintenance window.

4. Use latest with a one-node drift budget

Set amiVersion: latest to select the newest AMI for new nodes. Set nodes: "1" for the Drifted reason with no schedule so Karpenter can replace one drifted node at any time.

Pros

Cons

Existing nodes update as soon as Karpenter detects drift.

The team does not control when replacement starts.

No manual budget change is required.

Replacement may start during peak traffic or an incident.

New nodes receive the latest AMI.

Safe rollout depends on the workloads’ PodDisruptionBudgets and rescheduling behavior.

Only one drifted node is replaced at a time.

AWS AMI release timing determines the disruption schedule.

Conclusion

Before changing a pinned AMI to latest, check your NodePool’s disruption budget:

  • amiVersion selects the AMI for new nodes.

  • The disruption budget controls whether Karpenter can replace existing nodes when their AMI drifts.

  • With nodes: "0" for Drifted, existing nodes stay on their old AMI until the team raises the budget.

Keep the AMI pinned for explicit version control, schedule a maintenance window for automatic replacement at a known time, or allow one drift replacement at any time. If you use latest with a zero drift budget, monitor drift and document the manual replacement process.

If you liked this post, you might find these interesting too

In an EKS cluster where Karpenter creates EC2 nodes, a common best practice is to set Karpenter’s AMI version to latest. Karpenter then uses the updated EKS-optimized AMI and its security patches when it creates a node.

But Karpenter checks this setting only when creating a node. To update a running node, it must replace the node and move its workloads. If your safety settings block replacement, the node remains unpatched.

This is easy to miss because the configuration looks correct and Karpenter sends no alert. Critical nodes can remain unpatched while new nodes run the latest AMI. This post explains how that happens and compares four ways to balance AMI updates against workload disruption.

Prerequisites

If you’re already familiar with Karpenter, NodePools, and disruption budgets, skip ahead to How disruption budgets protect workloads.

Karpenter is a Kubernetes node autoscaler for AWS. Unlike a managed node group backed by a fixed Auto Scaling Group, Karpenter watches for pods that cannot be scheduled and creates suitable EC2 instances as needed. It removes those nodes when they are no longer required.

This post uses the following Karpenter concepts:

  • A NodePool defines which nodes Karpenter can create, including their instance types, capacity types, and availability zones. It also controls how Karpenter may disrupt them.

  • An EC2NodeClass defines the AMI, networking, and boot configuration for those nodes.

  • A NodeClaim is Karpenter’s record of a provisioned node.

  • Drift means that a running node no longer matches its desired configuration. An AMI change is one cause of drift.

  • Disruption is any action that removes or replaces a node. It includes consolidation, drift replacement, and expiry.

Why clusters use different groups of nodes

A typical EKS cluster separates its node groups based on workload reliability requirements. A batch job can usually tolerate being rescheduled, but a CI runner halfway through a deployment cannot.

Tier

Managed by

Role

Managed node group

AWS EKS

The cluster’s fixed bootstrap node group. It exists before Karpenter starts and runs system pods such as Karpenter, the CNI, and CoreDNS.

Karpenter default pool

Karpenter

A flexible group for general workloads. Karpenter may replace its nodes when they are underutilized, drifted, or expired.

Karpenter ondemand pool

Karpenter

A protected group for critical workloads. It runs services such as CI/CD runners and monitoring on on-demand instances with strict disruption controls.

How disruption budgets protect workloads

Karpenter’s disruption budgets control how many nodes it may disrupt at once and which reasons permit that disruption:

disruption:
  budgets:
    - nodes: "0"
      reasons:
      - "Underutilized"
      - "Drifted"
    - nodes: "20%"
      reasons:
      - "Empty"

The two rules apply to different situations:

  • nodes: "0" keeps underutilized nodes with spare capacity running. It also keeps drifted nodes running when their AMI or other settings no longer match the configured values.

  • nodes: "20%" for Empty means Karpenter can remove up to 20% of nodes that have no workloads running on them. Because these nodes are empty, Karpenter does not need to move any workloads first.

The zero-node rule blocks replacement until the team temporarily changes the budget during a maintenance window.

The harmless-looking commit

Consider this one-line change to the EC2NodeClass configuration:

- amiVersion: v20251209
+ amiVersion: latest

With the pinned setting, Karpenter keeps using v20251209 for new nodes until the team changes the version in a pull request. With latest, Karpenter selects the newest EKS-optimized AMI available when it creates a node.

This selection applies only to new nodes. A running node receives the new AMI only after Karpenter replaces it.

Upgrade timeline

Consider this timeline:

  • April 20, 12:00 PM

    • A workload needs a node in the ondemand pool.

    • Karpenter creates the node with AMI v20251209, the latest version at the time.

    • The AMI contains kubelet v1.34.2.

  • June 5, 12:30 PM

    • The team deploys the configuration change.

    • amiVersion changes from v20251209 to latest.

    • The latest AMI is now v20260526, which contains kubelet v1.34.8.

  • June 5, 12:31 PM

    • Karpenter compares the running node with the configured AMI.

    • The versions differ, so Karpenter marks the NodeClaim as Drifted: True with reason AMIDrift.

  • June 5, 12:32 PM

    • Karpenter checks the disruption budget.

    • The budget allows zero replacements for Drifted nodes, so the old node stays in place.

  • After June 5

    • The node remains on AMI v20251209, which is 46 days older than v20260526.

    • Karpenter sends no alert or notification about the blocked replacement.

Flowchart: after amiVersion is set to latest, a node not on the latest AMI is marked drifted. In the default pool it is replaced with the new AMI; in the ondemand pool, a drift budget of 0 blocks replacement and the node stays on the old AMI.
Fig 1: The same drift is fixed in the default pool and blocked in the ondemand pool.

This cluster has two Karpenter pools with different replacement rules. The default pool runs general workloads and allows Karpenter to replace drifted nodes. The ondemand pool runs critical workloads and sets its drift budget to zero.

When the AMI changes, Karpenter replaces drifted nodes in the default pool with nodes that use the new AMI. It also detects drift in the ondemand pool, but the zero-node budget keeps those nodes on the AMI they had at launch.

The security change therefore updates new nodes and replaceable nodes, but it does not update protected nodes that are already running.

Each AMI release increases the gap

AWS releases new EKS-optimized AMIs every few weeks with security patches and kubelet updates. Each release changes what latest resolves to, and Karpenter marks existing ondemand nodes as drifted again. The disruption budget still blocks their replacement.

A loop: AWS releases a new AMI, latest resolves to it, Karpenter detects the mismatch and marks the node drifted, a budget of 0 blocks replacement, and the node falls one more release behind.
Fig 2: Each AMI release repeats the cycle, so the protected node falls further behind.

An ondemand node may eventually qualify for replacement under the Empty budget, but only if all its workloads leave. A node dedicated to long-running CI/CD or monitoring workloads may never become empty. It can remain on its original AMI through several releases.

latest applies when Karpenter creates a node, not throughout the node’s lifetime.

Updating a protected node by hand

Someone must temporarily change the NodePool budget to update a node that cannot become empty:

  1. Patch the NodePool to allow at least one drift replacement.

  2. Karpenter responds by creating a node with the latest AMI, moving the workloads to it, and terminating the old node. Confirm that this process finishes.

  3. Restore the zero-node budget for drifted nodes.

A loop of manual steps: notice a node on an old AMI, patch the NodePool to a drifted budget of 1, Karpenter launches a new node, the old node is drained and terminated, revert the budget to 0, and repeat after the next AMI release.
Fig 3: Without automation, the team must repeat this process after every AMI release.

This requires about the same manual work as bumping a pinned AMI in a pull request. The pinned approach at least puts the required action in Git. With latest, drift can accumulate unless someone checks the NodeClaims or compares the AMIs used by running nodes.

Four approaches to keep node AMIs updated

1. Pin the AMI version

Set amiVersion: v20260526 to pin the AMI version. Set nodes: "0" for the Drifted reason to block automatic replacement of existing nodes.

Pros

Cons

A pull request controls which AMI version the cluster targets.

Every AMI update requires a manual pull request.

Each version change has a clear Git audit trail.

The pinned version can become stale if nobody updates it.

Drift starts only after the version changes in Git.

New nodes use the stale pinned version until that change happens.

The team must temporarily raise the drift budget to replace existing nodes.

2. Use latest with a zero drift budget

Set amiVersion: latest to select the newest AMI. Set nodes: "0" for the Drifted reason to block automatic replacement of existing nodes.

Pros

Cons

New nodes receive the latest AMI.

Existing nodes do not update automatically.

The team does not need to update amiVersion for each AMI release.

Drift can build up without an alert or pull request.

Karpenter cannot cause an unexpected drift replacement.

The team must temporarily raise the drift budget to replace existing nodes.

This combination is reasonable only if the team monitors AMI drift and has a separate process for replacing protected nodes. Without that process, it is the trap described in this post.

3. Use latest with a maintenance window

Set amiVersion: latest to select the newest AMI for new nodes. Set nodes: "1" for the Drifted reason during a scheduled window so Karpenter can replace one drifted node at a time:

budgets:
  - nodes: "0"
    reasons: ["Drifted"]
  - nodes: "1"
    reasons: ["Drifted"]
    schedule: "0 2 * * MON"
    duration

The drift budget is zero outside the window, so existing drifted nodes keep running. During the window, the budget is one, so Karpenter may replace one drifted node at a time.

Pros

Cons

Existing nodes update without a manual budget change.

Replacement still disrupts workloads during the window.

The team controls when replacements can happen.

Several drifted nodes may take several windows to replace.

New nodes receive the latest AMI.

The workloads still need correct PodDisruptionBudgets.

A node can remain behind until the next maintenance window.

4. Use latest with a one-node drift budget

Set amiVersion: latest to select the newest AMI for new nodes. Set nodes: "1" for the Drifted reason with no schedule so Karpenter can replace one drifted node at any time.

Pros

Cons

Existing nodes update as soon as Karpenter detects drift.

The team does not control when replacement starts.

No manual budget change is required.

Replacement may start during peak traffic or an incident.

New nodes receive the latest AMI.

Safe rollout depends on the workloads’ PodDisruptionBudgets and rescheduling behavior.

Only one drifted node is replaced at a time.

AWS AMI release timing determines the disruption schedule.

Conclusion

Before changing a pinned AMI to latest, check your NodePool’s disruption budget:

  • amiVersion selects the AMI for new nodes.

  • The disruption budget controls whether Karpenter can replace existing nodes when their AMI drifts.

  • With nodes: "0" for Drifted, existing nodes stay on their old AMI until the team raises the budget.

Keep the AMI pinned for explicit version control, schedule a maintenance window for automatic replacement at a known time, or allow one drift replacement at any time. If you use latest with a zero drift budget, monitor drift and document the manual replacement process.

If you liked this post, you might find these interesting too

In an EKS cluster where Karpenter creates EC2 nodes, a common best practice is to set Karpenter’s AMI version to latest. Karpenter then uses the updated EKS-optimized AMI and its security patches when it creates a node.

But Karpenter checks this setting only when creating a node. To update a running node, it must replace the node and move its workloads. If your safety settings block replacement, the node remains unpatched.

This is easy to miss because the configuration looks correct and Karpenter sends no alert. Critical nodes can remain unpatched while new nodes run the latest AMI. This post explains how that happens and compares four ways to balance AMI updates against workload disruption.

Prerequisites

If you’re already familiar with Karpenter, NodePools, and disruption budgets, skip ahead to How disruption budgets protect workloads.

Karpenter is a Kubernetes node autoscaler for AWS. Unlike a managed node group backed by a fixed Auto Scaling Group, Karpenter watches for pods that cannot be scheduled and creates suitable EC2 instances as needed. It removes those nodes when they are no longer required.

This post uses the following Karpenter concepts:

  • A NodePool defines which nodes Karpenter can create, including their instance types, capacity types, and availability zones. It also controls how Karpenter may disrupt them.

  • An EC2NodeClass defines the AMI, networking, and boot configuration for those nodes.

  • A NodeClaim is Karpenter’s record of a provisioned node.

  • Drift means that a running node no longer matches its desired configuration. An AMI change is one cause of drift.

  • Disruption is any action that removes or replaces a node. It includes consolidation, drift replacement, and expiry.

Why clusters use different groups of nodes

A typical EKS cluster separates its node groups based on workload reliability requirements. A batch job can usually tolerate being rescheduled, but a CI runner halfway through a deployment cannot.

Tier

Managed by

Role

Managed node group

AWS EKS

The cluster’s fixed bootstrap node group. It exists before Karpenter starts and runs system pods such as Karpenter, the CNI, and CoreDNS.

Karpenter default pool

Karpenter

A flexible group for general workloads. Karpenter may replace its nodes when they are underutilized, drifted, or expired.

Karpenter ondemand pool

Karpenter

A protected group for critical workloads. It runs services such as CI/CD runners and monitoring on on-demand instances with strict disruption controls.

How disruption budgets protect workloads

Karpenter’s disruption budgets control how many nodes it may disrupt at once and which reasons permit that disruption:

disruption:
  budgets:
    - nodes: "0"
      reasons:
      - "Underutilized"
      - "Drifted"
    - nodes: "20%"
      reasons:
      - "Empty"

The two rules apply to different situations:

  • nodes: "0" keeps underutilized nodes with spare capacity running. It also keeps drifted nodes running when their AMI or other settings no longer match the configured values.

  • nodes: "20%" for Empty means Karpenter can remove up to 20% of nodes that have no workloads running on them. Because these nodes are empty, Karpenter does not need to move any workloads first.

The zero-node rule blocks replacement until the team temporarily changes the budget during a maintenance window.

The harmless-looking commit

Consider this one-line change to the EC2NodeClass configuration:

- amiVersion: v20251209
+ amiVersion: latest

With the pinned setting, Karpenter keeps using v20251209 for new nodes until the team changes the version in a pull request. With latest, Karpenter selects the newest EKS-optimized AMI available when it creates a node.

This selection applies only to new nodes. A running node receives the new AMI only after Karpenter replaces it.

Upgrade timeline

Consider this timeline:

  • April 20, 12:00 PM

    • A workload needs a node in the ondemand pool.

    • Karpenter creates the node with AMI v20251209, the latest version at the time.

    • The AMI contains kubelet v1.34.2.

  • June 5, 12:30 PM

    • The team deploys the configuration change.

    • amiVersion changes from v20251209 to latest.

    • The latest AMI is now v20260526, which contains kubelet v1.34.8.

  • June 5, 12:31 PM

    • Karpenter compares the running node with the configured AMI.

    • The versions differ, so Karpenter marks the NodeClaim as Drifted: True with reason AMIDrift.

  • June 5, 12:32 PM

    • Karpenter checks the disruption budget.

    • The budget allows zero replacements for Drifted nodes, so the old node stays in place.

  • After June 5

    • The node remains on AMI v20251209, which is 46 days older than v20260526.

    • Karpenter sends no alert or notification about the blocked replacement.

Flowchart: after amiVersion is set to latest, a node not on the latest AMI is marked drifted. In the default pool it is replaced with the new AMI; in the ondemand pool, a drift budget of 0 blocks replacement and the node stays on the old AMI.
Fig 1: The same drift is fixed in the default pool and blocked in the ondemand pool.

This cluster has two Karpenter pools with different replacement rules. The default pool runs general workloads and allows Karpenter to replace drifted nodes. The ondemand pool runs critical workloads and sets its drift budget to zero.

When the AMI changes, Karpenter replaces drifted nodes in the default pool with nodes that use the new AMI. It also detects drift in the ondemand pool, but the zero-node budget keeps those nodes on the AMI they had at launch.

The security change therefore updates new nodes and replaceable nodes, but it does not update protected nodes that are already running.

Each AMI release increases the gap

AWS releases new EKS-optimized AMIs every few weeks with security patches and kubelet updates. Each release changes what latest resolves to, and Karpenter marks existing ondemand nodes as drifted again. The disruption budget still blocks their replacement.

A loop: AWS releases a new AMI, latest resolves to it, Karpenter detects the mismatch and marks the node drifted, a budget of 0 blocks replacement, and the node falls one more release behind.
Fig 2: Each AMI release repeats the cycle, so the protected node falls further behind.

An ondemand node may eventually qualify for replacement under the Empty budget, but only if all its workloads leave. A node dedicated to long-running CI/CD or monitoring workloads may never become empty. It can remain on its original AMI through several releases.

latest applies when Karpenter creates a node, not throughout the node’s lifetime.

Updating a protected node by hand

Someone must temporarily change the NodePool budget to update a node that cannot become empty:

  1. Patch the NodePool to allow at least one drift replacement.

  2. Karpenter responds by creating a node with the latest AMI, moving the workloads to it, and terminating the old node. Confirm that this process finishes.

  3. Restore the zero-node budget for drifted nodes.

A loop of manual steps: notice a node on an old AMI, patch the NodePool to a drifted budget of 1, Karpenter launches a new node, the old node is drained and terminated, revert the budget to 0, and repeat after the next AMI release.
Fig 3: Without automation, the team must repeat this process after every AMI release.

This requires about the same manual work as bumping a pinned AMI in a pull request. The pinned approach at least puts the required action in Git. With latest, drift can accumulate unless someone checks the NodeClaims or compares the AMIs used by running nodes.

Four approaches to keep node AMIs updated

1. Pin the AMI version

Set amiVersion: v20260526 to pin the AMI version. Set nodes: "0" for the Drifted reason to block automatic replacement of existing nodes.

Pros

Cons

A pull request controls which AMI version the cluster targets.

Every AMI update requires a manual pull request.

Each version change has a clear Git audit trail.

The pinned version can become stale if nobody updates it.

Drift starts only after the version changes in Git.

New nodes use the stale pinned version until that change happens.

The team must temporarily raise the drift budget to replace existing nodes.

2. Use latest with a zero drift budget

Set amiVersion: latest to select the newest AMI. Set nodes: "0" for the Drifted reason to block automatic replacement of existing nodes.

Pros

Cons

New nodes receive the latest AMI.

Existing nodes do not update automatically.

The team does not need to update amiVersion for each AMI release.

Drift can build up without an alert or pull request.

Karpenter cannot cause an unexpected drift replacement.

The team must temporarily raise the drift budget to replace existing nodes.

This combination is reasonable only if the team monitors AMI drift and has a separate process for replacing protected nodes. Without that process, it is the trap described in this post.

3. Use latest with a maintenance window

Set amiVersion: latest to select the newest AMI for new nodes. Set nodes: "1" for the Drifted reason during a scheduled window so Karpenter can replace one drifted node at a time:

budgets:
  - nodes: "0"
    reasons: ["Drifted"]
  - nodes: "1"
    reasons: ["Drifted"]
    schedule: "0 2 * * MON"
    duration

The drift budget is zero outside the window, so existing drifted nodes keep running. During the window, the budget is one, so Karpenter may replace one drifted node at a time.

Pros

Cons

Existing nodes update without a manual budget change.

Replacement still disrupts workloads during the window.

The team controls when replacements can happen.

Several drifted nodes may take several windows to replace.

New nodes receive the latest AMI.

The workloads still need correct PodDisruptionBudgets.

A node can remain behind until the next maintenance window.

4. Use latest with a one-node drift budget

Set amiVersion: latest to select the newest AMI for new nodes. Set nodes: "1" for the Drifted reason with no schedule so Karpenter can replace one drifted node at any time.

Pros

Cons

Existing nodes update as soon as Karpenter detects drift.

The team does not control when replacement starts.

No manual budget change is required.

Replacement may start during peak traffic or an incident.

New nodes receive the latest AMI.

Safe rollout depends on the workloads’ PodDisruptionBudgets and rescheduling behavior.

Only one drifted node is replaced at a time.

AWS AMI release timing determines the disruption schedule.

Conclusion

Before changing a pinned AMI to latest, check your NodePool’s disruption budget:

  • amiVersion selects the AMI for new nodes.

  • The disruption budget controls whether Karpenter can replace existing nodes when their AMI drifts.

  • With nodes: "0" for Drifted, existing nodes stay on their old AMI until the team raises the budget.

Keep the AMI pinned for explicit version control, schedule a maintenance window for automatic replacement at a known time, or allow one drift replacement at any time. If you use latest with a zero drift budget, monitor drift and document the manual replacement process.

If you liked this post, you might find these interesting too

Share
Share
On this page
Section
On this page

Continue reading.

Subscribe for more such content

Get the latest in software engineering best practices straight to your inbox. Subscribe now!

Subscribe for more such content

Get the latest in software engineering best practices straight to your inbox. Subscribe now!

Subscribe for more such content

Get the latest in software engineering best practices straight to your inbox. Subscribe now!