In an EKS cluster where Karpenter creates EC2 nodes, a common best practice is to set Karpenter’s AMI version to latest. Karpenter then uses the updated EKS-optimized AMI and its security patches when it creates a node.
But Karpenter checks this setting only when creating a node. To update a running node, it must replace the node and move its workloads. If your safety settings block replacement, the node remains unpatched.
This is easy to miss because the configuration looks correct and Karpenter sends no alert. Critical nodes can remain unpatched while new nodes run the latest AMI. This post explains how that happens and compares four ways to balance AMI updates against workload disruption.
Prerequisites
If you’re already familiar with Karpenter, NodePools, and disruption budgets, skip ahead to How disruption budgets protect workloads.
Karpenter is a Kubernetes node autoscaler for AWS. Unlike a managed node group backed by a fixed Auto Scaling Group, Karpenter watches for pods that cannot be scheduled and creates suitable EC2 instances as needed. It removes those nodes when they are no longer required.
This post uses the following Karpenter concepts:
A NodePool defines which nodes Karpenter can create, including their instance types, capacity types, and availability zones. It also controls how Karpenter may disrupt them.
An EC2NodeClass defines the AMI, networking, and boot configuration for those nodes.
A NodeClaim is Karpenter’s record of a provisioned node.
Drift means that a running node no longer matches its desired configuration. An AMI change is one cause of drift.
Disruption is any action that removes or replaces a node. It includes consolidation, drift replacement, and expiry.
Why clusters use different groups of nodes
A typical EKS cluster separates its node groups based on workload reliability requirements. A batch job can usually tolerate being rescheduled, but a CI runner halfway through a deployment cannot.
Tier | Managed by | Role |
|---|---|---|
Managed node group | AWS EKS | The cluster’s fixed bootstrap node group. It exists before Karpenter starts and runs system pods such as Karpenter, the CNI, and CoreDNS. |
Karpenter | Karpenter | A flexible group for general workloads. Karpenter may replace its nodes when they are underutilized, drifted, or expired. |
Karpenter | Karpenter | A protected group for critical workloads. It runs services such as CI/CD runners and monitoring on on-demand instances with strict disruption controls. |
How disruption budgets protect workloads
Karpenter’s disruption budgets control how many nodes it may disrupt at once and which reasons permit that disruption:
disruption: budgets: - nodes: "0" reasons: - "Underutilized" - "Drifted" - nodes: "20%" reasons: - "Empty"
The two rules apply to different situations:
nodes: "0"keeps underutilized nodes with spare capacity running. It also keeps drifted nodes running when their AMI or other settings no longer match the configured values.nodes: "20%"forEmptymeans Karpenter can remove up to 20% of nodes that have no workloads running on them. Because these nodes are empty, Karpenter does not need to move any workloads first.
The zero-node rule blocks replacement until the team temporarily changes the budget during a maintenance window.
The harmless-looking commit
Consider this one-line change to the EC2NodeClass configuration:
- amiVersion: v20251209 + amiVersion: latest
With the pinned setting, Karpenter keeps using v20251209 for new nodes until the team changes the version in a pull request. With latest, Karpenter selects the newest EKS-optimized AMI available when it creates a node.
This selection applies only to new nodes. A running node receives the new AMI only after Karpenter replaces it.
Upgrade timeline
Consider this timeline:
April 20, 12:00 PM
A workload needs a node in the
ondemandpool.Karpenter creates the node with AMI
v20251209, the latest version at the time.The AMI contains kubelet
v1.34.2.
June 5, 12:30 PM
The team deploys the configuration change.
amiVersionchanges fromv20251209tolatest.The latest AMI is now
v20260526, which contains kubeletv1.34.8.
June 5, 12:31 PM
Karpenter compares the running node with the configured AMI.
The versions differ, so Karpenter marks the NodeClaim as
Drifted: Truewith reasonAMIDrift.
June 5, 12:32 PM
Karpenter checks the disruption budget.
The budget allows zero replacements for
Driftednodes, so the old node stays in place.
After June 5
The node remains on AMI
v20251209, which is 46 days older thanv20260526.Karpenter sends no alert or notification about the blocked replacement.

Fig 1: The same drift is fixed in the default pool and blocked in the ondemand pool.
This cluster has two Karpenter pools with different replacement rules. The default pool runs general workloads and allows Karpenter to replace drifted nodes. The ondemand pool runs critical workloads and sets its drift budget to zero.
When the AMI changes, Karpenter replaces drifted nodes in the default pool with nodes that use the new AMI. It also detects drift in the ondemand pool, but the zero-node budget keeps those nodes on the AMI they had at launch.
The security change therefore updates new nodes and replaceable nodes, but it does not update protected nodes that are already running.
Each AMI release increases the gap
AWS releases new EKS-optimized AMIs every few weeks with security patches and kubelet updates. Each release changes what latest resolves to, and Karpenter marks existing ondemand nodes as drifted again. The disruption budget still blocks their replacement.

Fig 2: Each AMI release repeats the cycle, so the protected node falls further behind.
An ondemand node may eventually qualify for replacement under the Empty budget, but only if all its workloads leave. A node dedicated to long-running CI/CD or monitoring workloads may never become empty. It can remain on its original AMI through several releases.
latest applies when Karpenter creates a node, not throughout the node’s lifetime.
Updating a protected node by hand
Someone must temporarily change the NodePool budget to update a node that cannot become empty:
Patch the NodePool to allow at least one drift replacement.
Karpenter responds by creating a node with the latest AMI, moving the workloads to it, and terminating the old node. Confirm that this process finishes.
Restore the zero-node budget for drifted nodes.

Fig 3: Without automation, the team must repeat this process after every AMI release.
This requires about the same manual work as bumping a pinned AMI in a pull request. The pinned approach at least puts the required action in Git. With latest, drift can accumulate unless someone checks the NodeClaims or compares the AMIs used by running nodes.
Four approaches to keep node AMIs updated
1. Pin the AMI version
Set amiVersion: v20260526 to pin the AMI version. Set nodes: "0" for the Drifted reason to block automatic replacement of existing nodes.
Pros | Cons |
|---|---|
A pull request controls which AMI version the cluster targets. | Every AMI update requires a manual pull request. |
Each version change has a clear Git audit trail. | The pinned version can become stale if nobody updates it. |
Drift starts only after the version changes in Git. | New nodes use the stale pinned version until that change happens. |
The team must temporarily raise the drift budget to replace existing nodes. |
2. Use latest with a zero drift budget
Set amiVersion: latest to select the newest AMI. Set nodes: "0" for the Drifted reason to block automatic replacement of existing nodes.
Pros | Cons |
|---|---|
New nodes receive the latest AMI. | Existing nodes do not update automatically. |
The team does not need to update | Drift can build up without an alert or pull request. |
Karpenter cannot cause an unexpected drift replacement. | The team must temporarily raise the drift budget to replace existing nodes. |
This combination is reasonable only if the team monitors AMI drift and has a separate process for replacing protected nodes. Without that process, it is the trap described in this post.
3. Use latest with a maintenance window
Set amiVersion: latest to select the newest AMI for new nodes. Set nodes: "1" for the Drifted reason during a scheduled window so Karpenter can replace one drifted node at a time:
budgets: - nodes: "0" reasons: ["Drifted"] - nodes: "1" reasons: ["Drifted"] schedule: "0 2 * * MON" duration
The drift budget is zero outside the window, so existing drifted nodes keep running. During the window, the budget is one, so Karpenter may replace one drifted node at a time.
Pros | Cons |
|---|---|
Existing nodes update without a manual budget change. | Replacement still disrupts workloads during the window. |
The team controls when replacements can happen. | Several drifted nodes may take several windows to replace. |
New nodes receive the latest AMI. | The workloads still need correct PodDisruptionBudgets. |
A node can remain behind until the next maintenance window. |
4. Use latest with a one-node drift budget
Set amiVersion: latest to select the newest AMI for new nodes. Set nodes: "1" for the Drifted reason with no schedule so Karpenter can replace one drifted node at any time.
Pros | Cons |
|---|---|
Existing nodes update as soon as Karpenter detects drift. | The team does not control when replacement starts. |
No manual budget change is required. | Replacement may start during peak traffic or an incident. |
New nodes receive the latest AMI. | Safe rollout depends on the workloads’ PodDisruptionBudgets and rescheduling behavior. |
Only one drifted node is replaced at a time. | AWS AMI release timing determines the disruption schedule. |
Conclusion
Before changing a pinned AMI to latest, check your NodePool’s disruption budget:
amiVersionselects the AMI for new nodes.The disruption budget controls whether Karpenter can replace existing nodes when their AMI drifts.
With
nodes: "0"forDrifted, existing nodes stay on their old AMI until the team raises the budget.
Keep the AMI pinned for explicit version control, schedule a maintenance window for automatic replacement at a known time, or allow one drift replacement at any time. If you use latest with a zero drift budget, monitor drift and document the manual replacement process.
If you liked this post, you might find these interesting too
In an EKS cluster where Karpenter creates EC2 nodes, a common best practice is to set Karpenter’s AMI version to latest. Karpenter then uses the updated EKS-optimized AMI and its security patches when it creates a node.
But Karpenter checks this setting only when creating a node. To update a running node, it must replace the node and move its workloads. If your safety settings block replacement, the node remains unpatched.
This is easy to miss because the configuration looks correct and Karpenter sends no alert. Critical nodes can remain unpatched while new nodes run the latest AMI. This post explains how that happens and compares four ways to balance AMI updates against workload disruption.
Prerequisites
If you’re already familiar with Karpenter, NodePools, and disruption budgets, skip ahead to How disruption budgets protect workloads.
Karpenter is a Kubernetes node autoscaler for AWS. Unlike a managed node group backed by a fixed Auto Scaling Group, Karpenter watches for pods that cannot be scheduled and creates suitable EC2 instances as needed. It removes those nodes when they are no longer required.
This post uses the following Karpenter concepts:
A NodePool defines which nodes Karpenter can create, including their instance types, capacity types, and availability zones. It also controls how Karpenter may disrupt them.
An EC2NodeClass defines the AMI, networking, and boot configuration for those nodes.
A NodeClaim is Karpenter’s record of a provisioned node.
Drift means that a running node no longer matches its desired configuration. An AMI change is one cause of drift.
Disruption is any action that removes or replaces a node. It includes consolidation, drift replacement, and expiry.
Why clusters use different groups of nodes
A typical EKS cluster separates its node groups based on workload reliability requirements. A batch job can usually tolerate being rescheduled, but a CI runner halfway through a deployment cannot.
Tier | Managed by | Role |
|---|---|---|
Managed node group | AWS EKS | The cluster’s fixed bootstrap node group. It exists before Karpenter starts and runs system pods such as Karpenter, the CNI, and CoreDNS. |
Karpenter | Karpenter | A flexible group for general workloads. Karpenter may replace its nodes when they are underutilized, drifted, or expired. |
Karpenter | Karpenter | A protected group for critical workloads. It runs services such as CI/CD runners and monitoring on on-demand instances with strict disruption controls. |
How disruption budgets protect workloads
Karpenter’s disruption budgets control how many nodes it may disrupt at once and which reasons permit that disruption:
disruption: budgets: - nodes: "0" reasons: - "Underutilized" - "Drifted" - nodes: "20%" reasons: - "Empty"
The two rules apply to different situations:
nodes: "0"keeps underutilized nodes with spare capacity running. It also keeps drifted nodes running when their AMI or other settings no longer match the configured values.nodes: "20%"forEmptymeans Karpenter can remove up to 20% of nodes that have no workloads running on them. Because these nodes are empty, Karpenter does not need to move any workloads first.
The zero-node rule blocks replacement until the team temporarily changes the budget during a maintenance window.
The harmless-looking commit
Consider this one-line change to the EC2NodeClass configuration:
- amiVersion: v20251209 + amiVersion: latest
With the pinned setting, Karpenter keeps using v20251209 for new nodes until the team changes the version in a pull request. With latest, Karpenter selects the newest EKS-optimized AMI available when it creates a node.
This selection applies only to new nodes. A running node receives the new AMI only after Karpenter replaces it.
Upgrade timeline
Consider this timeline:
April 20, 12:00 PM
A workload needs a node in the
ondemandpool.Karpenter creates the node with AMI
v20251209, the latest version at the time.The AMI contains kubelet
v1.34.2.
June 5, 12:30 PM
The team deploys the configuration change.
amiVersionchanges fromv20251209tolatest.The latest AMI is now
v20260526, which contains kubeletv1.34.8.
June 5, 12:31 PM
Karpenter compares the running node with the configured AMI.
The versions differ, so Karpenter marks the NodeClaim as
Drifted: Truewith reasonAMIDrift.
June 5, 12:32 PM
Karpenter checks the disruption budget.
The budget allows zero replacements for
Driftednodes, so the old node stays in place.
After June 5
The node remains on AMI
v20251209, which is 46 days older thanv20260526.Karpenter sends no alert or notification about the blocked replacement.

Fig 1: The same drift is fixed in the default pool and blocked in the ondemand pool.
This cluster has two Karpenter pools with different replacement rules. The default pool runs general workloads and allows Karpenter to replace drifted nodes. The ondemand pool runs critical workloads and sets its drift budget to zero.
When the AMI changes, Karpenter replaces drifted nodes in the default pool with nodes that use the new AMI. It also detects drift in the ondemand pool, but the zero-node budget keeps those nodes on the AMI they had at launch.
The security change therefore updates new nodes and replaceable nodes, but it does not update protected nodes that are already running.
Each AMI release increases the gap
AWS releases new EKS-optimized AMIs every few weeks with security patches and kubelet updates. Each release changes what latest resolves to, and Karpenter marks existing ondemand nodes as drifted again. The disruption budget still blocks their replacement.

Fig 2: Each AMI release repeats the cycle, so the protected node falls further behind.
An ondemand node may eventually qualify for replacement under the Empty budget, but only if all its workloads leave. A node dedicated to long-running CI/CD or monitoring workloads may never become empty. It can remain on its original AMI through several releases.
latest applies when Karpenter creates a node, not throughout the node’s lifetime.
Updating a protected node by hand
Someone must temporarily change the NodePool budget to update a node that cannot become empty:
Patch the NodePool to allow at least one drift replacement.
Karpenter responds by creating a node with the latest AMI, moving the workloads to it, and terminating the old node. Confirm that this process finishes.
Restore the zero-node budget for drifted nodes.

Fig 3: Without automation, the team must repeat this process after every AMI release.
This requires about the same manual work as bumping a pinned AMI in a pull request. The pinned approach at least puts the required action in Git. With latest, drift can accumulate unless someone checks the NodeClaims or compares the AMIs used by running nodes.
Four approaches to keep node AMIs updated
1. Pin the AMI version
Set amiVersion: v20260526 to pin the AMI version. Set nodes: "0" for the Drifted reason to block automatic replacement of existing nodes.
Pros | Cons |
|---|---|
A pull request controls which AMI version the cluster targets. | Every AMI update requires a manual pull request. |
Each version change has a clear Git audit trail. | The pinned version can become stale if nobody updates it. |
Drift starts only after the version changes in Git. | New nodes use the stale pinned version until that change happens. |
The team must temporarily raise the drift budget to replace existing nodes. |
2. Use latest with a zero drift budget
Set amiVersion: latest to select the newest AMI. Set nodes: "0" for the Drifted reason to block automatic replacement of existing nodes.
Pros | Cons |
|---|---|
New nodes receive the latest AMI. | Existing nodes do not update automatically. |
The team does not need to update | Drift can build up without an alert or pull request. |
Karpenter cannot cause an unexpected drift replacement. | The team must temporarily raise the drift budget to replace existing nodes. |
This combination is reasonable only if the team monitors AMI drift and has a separate process for replacing protected nodes. Without that process, it is the trap described in this post.
3. Use latest with a maintenance window
Set amiVersion: latest to select the newest AMI for new nodes. Set nodes: "1" for the Drifted reason during a scheduled window so Karpenter can replace one drifted node at a time:
budgets: - nodes: "0" reasons: ["Drifted"] - nodes: "1" reasons: ["Drifted"] schedule: "0 2 * * MON" duration
The drift budget is zero outside the window, so existing drifted nodes keep running. During the window, the budget is one, so Karpenter may replace one drifted node at a time.
Pros | Cons |
|---|---|
Existing nodes update without a manual budget change. | Replacement still disrupts workloads during the window. |
The team controls when replacements can happen. | Several drifted nodes may take several windows to replace. |
New nodes receive the latest AMI. | The workloads still need correct PodDisruptionBudgets. |
A node can remain behind until the next maintenance window. |
4. Use latest with a one-node drift budget
Set amiVersion: latest to select the newest AMI for new nodes. Set nodes: "1" for the Drifted reason with no schedule so Karpenter can replace one drifted node at any time.
Pros | Cons |
|---|---|
Existing nodes update as soon as Karpenter detects drift. | The team does not control when replacement starts. |
No manual budget change is required. | Replacement may start during peak traffic or an incident. |
New nodes receive the latest AMI. | Safe rollout depends on the workloads’ PodDisruptionBudgets and rescheduling behavior. |
Only one drifted node is replaced at a time. | AWS AMI release timing determines the disruption schedule. |
Conclusion
Before changing a pinned AMI to latest, check your NodePool’s disruption budget:
amiVersionselects the AMI for new nodes.The disruption budget controls whether Karpenter can replace existing nodes when their AMI drifts.
With
nodes: "0"forDrifted, existing nodes stay on their old AMI until the team raises the budget.
Keep the AMI pinned for explicit version control, schedule a maintenance window for automatic replacement at a known time, or allow one drift replacement at any time. If you use latest with a zero drift budget, monitor drift and document the manual replacement process.
If you liked this post, you might find these interesting too
In an EKS cluster where Karpenter creates EC2 nodes, a common best practice is to set Karpenter’s AMI version to latest. Karpenter then uses the updated EKS-optimized AMI and its security patches when it creates a node.
But Karpenter checks this setting only when creating a node. To update a running node, it must replace the node and move its workloads. If your safety settings block replacement, the node remains unpatched.
This is easy to miss because the configuration looks correct and Karpenter sends no alert. Critical nodes can remain unpatched while new nodes run the latest AMI. This post explains how that happens and compares four ways to balance AMI updates against workload disruption.
Prerequisites
If you’re already familiar with Karpenter, NodePools, and disruption budgets, skip ahead to How disruption budgets protect workloads.
Karpenter is a Kubernetes node autoscaler for AWS. Unlike a managed node group backed by a fixed Auto Scaling Group, Karpenter watches for pods that cannot be scheduled and creates suitable EC2 instances as needed. It removes those nodes when they are no longer required.
This post uses the following Karpenter concepts:
A NodePool defines which nodes Karpenter can create, including their instance types, capacity types, and availability zones. It also controls how Karpenter may disrupt them.
An EC2NodeClass defines the AMI, networking, and boot configuration for those nodes.
A NodeClaim is Karpenter’s record of a provisioned node.
Drift means that a running node no longer matches its desired configuration. An AMI change is one cause of drift.
Disruption is any action that removes or replaces a node. It includes consolidation, drift replacement, and expiry.
Why clusters use different groups of nodes
A typical EKS cluster separates its node groups based on workload reliability requirements. A batch job can usually tolerate being rescheduled, but a CI runner halfway through a deployment cannot.
Tier | Managed by | Role |
|---|---|---|
Managed node group | AWS EKS | The cluster’s fixed bootstrap node group. It exists before Karpenter starts and runs system pods such as Karpenter, the CNI, and CoreDNS. |
Karpenter | Karpenter | A flexible group for general workloads. Karpenter may replace its nodes when they are underutilized, drifted, or expired. |
Karpenter | Karpenter | A protected group for critical workloads. It runs services such as CI/CD runners and monitoring on on-demand instances with strict disruption controls. |
How disruption budgets protect workloads
Karpenter’s disruption budgets control how many nodes it may disrupt at once and which reasons permit that disruption:
disruption: budgets: - nodes: "0" reasons: - "Underutilized" - "Drifted" - nodes: "20%" reasons: - "Empty"
The two rules apply to different situations:
nodes: "0"keeps underutilized nodes with spare capacity running. It also keeps drifted nodes running when their AMI or other settings no longer match the configured values.nodes: "20%"forEmptymeans Karpenter can remove up to 20% of nodes that have no workloads running on them. Because these nodes are empty, Karpenter does not need to move any workloads first.
The zero-node rule blocks replacement until the team temporarily changes the budget during a maintenance window.
The harmless-looking commit
Consider this one-line change to the EC2NodeClass configuration:
- amiVersion: v20251209 + amiVersion: latest
With the pinned setting, Karpenter keeps using v20251209 for new nodes until the team changes the version in a pull request. With latest, Karpenter selects the newest EKS-optimized AMI available when it creates a node.
This selection applies only to new nodes. A running node receives the new AMI only after Karpenter replaces it.
Upgrade timeline
Consider this timeline:
April 20, 12:00 PM
A workload needs a node in the
ondemandpool.Karpenter creates the node with AMI
v20251209, the latest version at the time.The AMI contains kubelet
v1.34.2.
June 5, 12:30 PM
The team deploys the configuration change.
amiVersionchanges fromv20251209tolatest.The latest AMI is now
v20260526, which contains kubeletv1.34.8.
June 5, 12:31 PM
Karpenter compares the running node with the configured AMI.
The versions differ, so Karpenter marks the NodeClaim as
Drifted: Truewith reasonAMIDrift.
June 5, 12:32 PM
Karpenter checks the disruption budget.
The budget allows zero replacements for
Driftednodes, so the old node stays in place.
After June 5
The node remains on AMI
v20251209, which is 46 days older thanv20260526.Karpenter sends no alert or notification about the blocked replacement.

Fig 1: The same drift is fixed in the default pool and blocked in the ondemand pool.
This cluster has two Karpenter pools with different replacement rules. The default pool runs general workloads and allows Karpenter to replace drifted nodes. The ondemand pool runs critical workloads and sets its drift budget to zero.
When the AMI changes, Karpenter replaces drifted nodes in the default pool with nodes that use the new AMI. It also detects drift in the ondemand pool, but the zero-node budget keeps those nodes on the AMI they had at launch.
The security change therefore updates new nodes and replaceable nodes, but it does not update protected nodes that are already running.
Each AMI release increases the gap
AWS releases new EKS-optimized AMIs every few weeks with security patches and kubelet updates. Each release changes what latest resolves to, and Karpenter marks existing ondemand nodes as drifted again. The disruption budget still blocks their replacement.

Fig 2: Each AMI release repeats the cycle, so the protected node falls further behind.
An ondemand node may eventually qualify for replacement under the Empty budget, but only if all its workloads leave. A node dedicated to long-running CI/CD or monitoring workloads may never become empty. It can remain on its original AMI through several releases.
latest applies when Karpenter creates a node, not throughout the node’s lifetime.
Updating a protected node by hand
Someone must temporarily change the NodePool budget to update a node that cannot become empty:
Patch the NodePool to allow at least one drift replacement.
Karpenter responds by creating a node with the latest AMI, moving the workloads to it, and terminating the old node. Confirm that this process finishes.
Restore the zero-node budget for drifted nodes.

Fig 3: Without automation, the team must repeat this process after every AMI release.
This requires about the same manual work as bumping a pinned AMI in a pull request. The pinned approach at least puts the required action in Git. With latest, drift can accumulate unless someone checks the NodeClaims or compares the AMIs used by running nodes.
Four approaches to keep node AMIs updated
1. Pin the AMI version
Set amiVersion: v20260526 to pin the AMI version. Set nodes: "0" for the Drifted reason to block automatic replacement of existing nodes.
Pros | Cons |
|---|---|
A pull request controls which AMI version the cluster targets. | Every AMI update requires a manual pull request. |
Each version change has a clear Git audit trail. | The pinned version can become stale if nobody updates it. |
Drift starts only after the version changes in Git. | New nodes use the stale pinned version until that change happens. |
The team must temporarily raise the drift budget to replace existing nodes. |
2. Use latest with a zero drift budget
Set amiVersion: latest to select the newest AMI. Set nodes: "0" for the Drifted reason to block automatic replacement of existing nodes.
Pros | Cons |
|---|---|
New nodes receive the latest AMI. | Existing nodes do not update automatically. |
The team does not need to update | Drift can build up without an alert or pull request. |
Karpenter cannot cause an unexpected drift replacement. | The team must temporarily raise the drift budget to replace existing nodes. |
This combination is reasonable only if the team monitors AMI drift and has a separate process for replacing protected nodes. Without that process, it is the trap described in this post.
3. Use latest with a maintenance window
Set amiVersion: latest to select the newest AMI for new nodes. Set nodes: "1" for the Drifted reason during a scheduled window so Karpenter can replace one drifted node at a time:
budgets: - nodes: "0" reasons: ["Drifted"] - nodes: "1" reasons: ["Drifted"] schedule: "0 2 * * MON" duration
The drift budget is zero outside the window, so existing drifted nodes keep running. During the window, the budget is one, so Karpenter may replace one drifted node at a time.
Pros | Cons |
|---|---|
Existing nodes update without a manual budget change. | Replacement still disrupts workloads during the window. |
The team controls when replacements can happen. | Several drifted nodes may take several windows to replace. |
New nodes receive the latest AMI. | The workloads still need correct PodDisruptionBudgets. |
A node can remain behind until the next maintenance window. |
4. Use latest with a one-node drift budget
Set amiVersion: latest to select the newest AMI for new nodes. Set nodes: "1" for the Drifted reason with no schedule so Karpenter can replace one drifted node at any time.
Pros | Cons |
|---|---|
Existing nodes update as soon as Karpenter detects drift. | The team does not control when replacement starts. |
No manual budget change is required. | Replacement may start during peak traffic or an incident. |
New nodes receive the latest AMI. | Safe rollout depends on the workloads’ PodDisruptionBudgets and rescheduling behavior. |
Only one drifted node is replaced at a time. | AWS AMI release timing determines the disruption schedule. |
Conclusion
Before changing a pinned AMI to latest, check your NodePool’s disruption budget:
amiVersionselects the AMI for new nodes.The disruption budget controls whether Karpenter can replace existing nodes when their AMI drifts.
With
nodes: "0"forDrifted, existing nodes stay on their old AMI until the team raises the budget.
Keep the AMI pinned for explicit version control, schedule a maintenance window for automatic replacement at a known time, or allow one drift replacement at any time. If you use latest with a zero drift budget, monitor drift and document the manual replacement process.
If you liked this post, you might find these interesting too
In an EKS cluster where Karpenter creates EC2 nodes, a common best practice is to set Karpenter’s AMI version to latest. Karpenter then uses the updated EKS-optimized AMI and its security patches when it creates a node.
But Karpenter checks this setting only when creating a node. To update a running node, it must replace the node and move its workloads. If your safety settings block replacement, the node remains unpatched.
This is easy to miss because the configuration looks correct and Karpenter sends no alert. Critical nodes can remain unpatched while new nodes run the latest AMI. This post explains how that happens and compares four ways to balance AMI updates against workload disruption.
Prerequisites
If you’re already familiar with Karpenter, NodePools, and disruption budgets, skip ahead to How disruption budgets protect workloads.
Karpenter is a Kubernetes node autoscaler for AWS. Unlike a managed node group backed by a fixed Auto Scaling Group, Karpenter watches for pods that cannot be scheduled and creates suitable EC2 instances as needed. It removes those nodes when they are no longer required.
This post uses the following Karpenter concepts:
A NodePool defines which nodes Karpenter can create, including their instance types, capacity types, and availability zones. It also controls how Karpenter may disrupt them.
An EC2NodeClass defines the AMI, networking, and boot configuration for those nodes.
A NodeClaim is Karpenter’s record of a provisioned node.
Drift means that a running node no longer matches its desired configuration. An AMI change is one cause of drift.
Disruption is any action that removes or replaces a node. It includes consolidation, drift replacement, and expiry.
Why clusters use different groups of nodes
A typical EKS cluster separates its node groups based on workload reliability requirements. A batch job can usually tolerate being rescheduled, but a CI runner halfway through a deployment cannot.
Tier | Managed by | Role |
|---|---|---|
Managed node group | AWS EKS | The cluster’s fixed bootstrap node group. It exists before Karpenter starts and runs system pods such as Karpenter, the CNI, and CoreDNS. |
Karpenter | Karpenter | A flexible group for general workloads. Karpenter may replace its nodes when they are underutilized, drifted, or expired. |
Karpenter | Karpenter | A protected group for critical workloads. It runs services such as CI/CD runners and monitoring on on-demand instances with strict disruption controls. |
How disruption budgets protect workloads
Karpenter’s disruption budgets control how many nodes it may disrupt at once and which reasons permit that disruption:
disruption: budgets: - nodes: "0" reasons: - "Underutilized" - "Drifted" - nodes: "20%" reasons: - "Empty"
The two rules apply to different situations:
nodes: "0"keeps underutilized nodes with spare capacity running. It also keeps drifted nodes running when their AMI or other settings no longer match the configured values.nodes: "20%"forEmptymeans Karpenter can remove up to 20% of nodes that have no workloads running on them. Because these nodes are empty, Karpenter does not need to move any workloads first.
The zero-node rule blocks replacement until the team temporarily changes the budget during a maintenance window.
The harmless-looking commit
Consider this one-line change to the EC2NodeClass configuration:
- amiVersion: v20251209 + amiVersion: latest
With the pinned setting, Karpenter keeps using v20251209 for new nodes until the team changes the version in a pull request. With latest, Karpenter selects the newest EKS-optimized AMI available when it creates a node.
This selection applies only to new nodes. A running node receives the new AMI only after Karpenter replaces it.
Upgrade timeline
Consider this timeline:
April 20, 12:00 PM
A workload needs a node in the
ondemandpool.Karpenter creates the node with AMI
v20251209, the latest version at the time.The AMI contains kubelet
v1.34.2.
June 5, 12:30 PM
The team deploys the configuration change.
amiVersionchanges fromv20251209tolatest.The latest AMI is now
v20260526, which contains kubeletv1.34.8.
June 5, 12:31 PM
Karpenter compares the running node with the configured AMI.
The versions differ, so Karpenter marks the NodeClaim as
Drifted: Truewith reasonAMIDrift.
June 5, 12:32 PM
Karpenter checks the disruption budget.
The budget allows zero replacements for
Driftednodes, so the old node stays in place.
After June 5
The node remains on AMI
v20251209, which is 46 days older thanv20260526.Karpenter sends no alert or notification about the blocked replacement.

Fig 1: The same drift is fixed in the default pool and blocked in the ondemand pool.
This cluster has two Karpenter pools with different replacement rules. The default pool runs general workloads and allows Karpenter to replace drifted nodes. The ondemand pool runs critical workloads and sets its drift budget to zero.
When the AMI changes, Karpenter replaces drifted nodes in the default pool with nodes that use the new AMI. It also detects drift in the ondemand pool, but the zero-node budget keeps those nodes on the AMI they had at launch.
The security change therefore updates new nodes and replaceable nodes, but it does not update protected nodes that are already running.
Each AMI release increases the gap
AWS releases new EKS-optimized AMIs every few weeks with security patches and kubelet updates. Each release changes what latest resolves to, and Karpenter marks existing ondemand nodes as drifted again. The disruption budget still blocks their replacement.

Fig 2: Each AMI release repeats the cycle, so the protected node falls further behind.
An ondemand node may eventually qualify for replacement under the Empty budget, but only if all its workloads leave. A node dedicated to long-running CI/CD or monitoring workloads may never become empty. It can remain on its original AMI through several releases.
latest applies when Karpenter creates a node, not throughout the node’s lifetime.
Updating a protected node by hand
Someone must temporarily change the NodePool budget to update a node that cannot become empty:
Patch the NodePool to allow at least one drift replacement.
Karpenter responds by creating a node with the latest AMI, moving the workloads to it, and terminating the old node. Confirm that this process finishes.
Restore the zero-node budget for drifted nodes.

Fig 3: Without automation, the team must repeat this process after every AMI release.
This requires about the same manual work as bumping a pinned AMI in a pull request. The pinned approach at least puts the required action in Git. With latest, drift can accumulate unless someone checks the NodeClaims or compares the AMIs used by running nodes.
Four approaches to keep node AMIs updated
1. Pin the AMI version
Set amiVersion: v20260526 to pin the AMI version. Set nodes: "0" for the Drifted reason to block automatic replacement of existing nodes.
Pros | Cons |
|---|---|
A pull request controls which AMI version the cluster targets. | Every AMI update requires a manual pull request. |
Each version change has a clear Git audit trail. | The pinned version can become stale if nobody updates it. |
Drift starts only after the version changes in Git. | New nodes use the stale pinned version until that change happens. |
The team must temporarily raise the drift budget to replace existing nodes. |
2. Use latest with a zero drift budget
Set amiVersion: latest to select the newest AMI. Set nodes: "0" for the Drifted reason to block automatic replacement of existing nodes.
Pros | Cons |
|---|---|
New nodes receive the latest AMI. | Existing nodes do not update automatically. |
The team does not need to update | Drift can build up without an alert or pull request. |
Karpenter cannot cause an unexpected drift replacement. | The team must temporarily raise the drift budget to replace existing nodes. |
This combination is reasonable only if the team monitors AMI drift and has a separate process for replacing protected nodes. Without that process, it is the trap described in this post.
3. Use latest with a maintenance window
Set amiVersion: latest to select the newest AMI for new nodes. Set nodes: "1" for the Drifted reason during a scheduled window so Karpenter can replace one drifted node at a time:
budgets: - nodes: "0" reasons: ["Drifted"] - nodes: "1" reasons: ["Drifted"] schedule: "0 2 * * MON" duration
The drift budget is zero outside the window, so existing drifted nodes keep running. During the window, the budget is one, so Karpenter may replace one drifted node at a time.
Pros | Cons |
|---|---|
Existing nodes update without a manual budget change. | Replacement still disrupts workloads during the window. |
The team controls when replacements can happen. | Several drifted nodes may take several windows to replace. |
New nodes receive the latest AMI. | The workloads still need correct PodDisruptionBudgets. |
A node can remain behind until the next maintenance window. |
4. Use latest with a one-node drift budget
Set amiVersion: latest to select the newest AMI for new nodes. Set nodes: "1" for the Drifted reason with no schedule so Karpenter can replace one drifted node at any time.
Pros | Cons |
|---|---|
Existing nodes update as soon as Karpenter detects drift. | The team does not control when replacement starts. |
No manual budget change is required. | Replacement may start during peak traffic or an incident. |
New nodes receive the latest AMI. | Safe rollout depends on the workloads’ PodDisruptionBudgets and rescheduling behavior. |
Only one drifted node is replaced at a time. | AWS AMI release timing determines the disruption schedule. |
Conclusion
Before changing a pinned AMI to latest, check your NodePool’s disruption budget:
amiVersionselects the AMI for new nodes.The disruption budget controls whether Karpenter can replace existing nodes when their AMI drifts.
With
nodes: "0"forDrifted, existing nodes stay on their old AMI until the team raises the budget.
Keep the AMI pinned for explicit version control, schedule a maintenance window for automatic replacement at a known time, or allow one drift replacement at any time. If you use latest with a zero drift budget, monitor drift and document the manual replacement process.
If you liked this post, you might find these interesting too
In an EKS cluster where Karpenter creates EC2 nodes, a common best practice is to set Karpenter’s AMI version to latest. Karpenter then uses the updated EKS-optimized AMI and its security patches when it creates a node.
But Karpenter checks this setting only when creating a node. To update a running node, it must replace the node and move its workloads. If your safety settings block replacement, the node remains unpatched.
This is easy to miss because the configuration looks correct and Karpenter sends no alert. Critical nodes can remain unpatched while new nodes run the latest AMI. This post explains how that happens and compares four ways to balance AMI updates against workload disruption.
Prerequisites
If you’re already familiar with Karpenter, NodePools, and disruption budgets, skip ahead to How disruption budgets protect workloads.
Karpenter is a Kubernetes node autoscaler for AWS. Unlike a managed node group backed by a fixed Auto Scaling Group, Karpenter watches for pods that cannot be scheduled and creates suitable EC2 instances as needed. It removes those nodes when they are no longer required.
This post uses the following Karpenter concepts:
A NodePool defines which nodes Karpenter can create, including their instance types, capacity types, and availability zones. It also controls how Karpenter may disrupt them.
An EC2NodeClass defines the AMI, networking, and boot configuration for those nodes.
A NodeClaim is Karpenter’s record of a provisioned node.
Drift means that a running node no longer matches its desired configuration. An AMI change is one cause of drift.
Disruption is any action that removes or replaces a node. It includes consolidation, drift replacement, and expiry.
Why clusters use different groups of nodes
A typical EKS cluster separates its node groups based on workload reliability requirements. A batch job can usually tolerate being rescheduled, but a CI runner halfway through a deployment cannot.
Tier | Managed by | Role |
|---|---|---|
Managed node group | AWS EKS | The cluster’s fixed bootstrap node group. It exists before Karpenter starts and runs system pods such as Karpenter, the CNI, and CoreDNS. |
Karpenter | Karpenter | A flexible group for general workloads. Karpenter may replace its nodes when they are underutilized, drifted, or expired. |
Karpenter | Karpenter | A protected group for critical workloads. It runs services such as CI/CD runners and monitoring on on-demand instances with strict disruption controls. |
How disruption budgets protect workloads
Karpenter’s disruption budgets control how many nodes it may disrupt at once and which reasons permit that disruption:
disruption: budgets: - nodes: "0" reasons: - "Underutilized" - "Drifted" - nodes: "20%" reasons: - "Empty"
The two rules apply to different situations:
nodes: "0"keeps underutilized nodes with spare capacity running. It also keeps drifted nodes running when their AMI or other settings no longer match the configured values.nodes: "20%"forEmptymeans Karpenter can remove up to 20% of nodes that have no workloads running on them. Because these nodes are empty, Karpenter does not need to move any workloads first.
The zero-node rule blocks replacement until the team temporarily changes the budget during a maintenance window.
The harmless-looking commit
Consider this one-line change to the EC2NodeClass configuration:
- amiVersion: v20251209 + amiVersion: latest
With the pinned setting, Karpenter keeps using v20251209 for new nodes until the team changes the version in a pull request. With latest, Karpenter selects the newest EKS-optimized AMI available when it creates a node.
This selection applies only to new nodes. A running node receives the new AMI only after Karpenter replaces it.
Upgrade timeline
Consider this timeline:
April 20, 12:00 PM
A workload needs a node in the
ondemandpool.Karpenter creates the node with AMI
v20251209, the latest version at the time.The AMI contains kubelet
v1.34.2.
June 5, 12:30 PM
The team deploys the configuration change.
amiVersionchanges fromv20251209tolatest.The latest AMI is now
v20260526, which contains kubeletv1.34.8.
June 5, 12:31 PM
Karpenter compares the running node with the configured AMI.
The versions differ, so Karpenter marks the NodeClaim as
Drifted: Truewith reasonAMIDrift.
June 5, 12:32 PM
Karpenter checks the disruption budget.
The budget allows zero replacements for
Driftednodes, so the old node stays in place.
After June 5
The node remains on AMI
v20251209, which is 46 days older thanv20260526.Karpenter sends no alert or notification about the blocked replacement.

Fig 1: The same drift is fixed in the default pool and blocked in the ondemand pool.
This cluster has two Karpenter pools with different replacement rules. The default pool runs general workloads and allows Karpenter to replace drifted nodes. The ondemand pool runs critical workloads and sets its drift budget to zero.
When the AMI changes, Karpenter replaces drifted nodes in the default pool with nodes that use the new AMI. It also detects drift in the ondemand pool, but the zero-node budget keeps those nodes on the AMI they had at launch.
The security change therefore updates new nodes and replaceable nodes, but it does not update protected nodes that are already running.
Each AMI release increases the gap
AWS releases new EKS-optimized AMIs every few weeks with security patches and kubelet updates. Each release changes what latest resolves to, and Karpenter marks existing ondemand nodes as drifted again. The disruption budget still blocks their replacement.

Fig 2: Each AMI release repeats the cycle, so the protected node falls further behind.
An ondemand node may eventually qualify for replacement under the Empty budget, but only if all its workloads leave. A node dedicated to long-running CI/CD or monitoring workloads may never become empty. It can remain on its original AMI through several releases.
latest applies when Karpenter creates a node, not throughout the node’s lifetime.
Updating a protected node by hand
Someone must temporarily change the NodePool budget to update a node that cannot become empty:
Patch the NodePool to allow at least one drift replacement.
Karpenter responds by creating a node with the latest AMI, moving the workloads to it, and terminating the old node. Confirm that this process finishes.
Restore the zero-node budget for drifted nodes.

Fig 3: Without automation, the team must repeat this process after every AMI release.
This requires about the same manual work as bumping a pinned AMI in a pull request. The pinned approach at least puts the required action in Git. With latest, drift can accumulate unless someone checks the NodeClaims or compares the AMIs used by running nodes.
Four approaches to keep node AMIs updated
1. Pin the AMI version
Set amiVersion: v20260526 to pin the AMI version. Set nodes: "0" for the Drifted reason to block automatic replacement of existing nodes.
Pros | Cons |
|---|---|
A pull request controls which AMI version the cluster targets. | Every AMI update requires a manual pull request. |
Each version change has a clear Git audit trail. | The pinned version can become stale if nobody updates it. |
Drift starts only after the version changes in Git. | New nodes use the stale pinned version until that change happens. |
The team must temporarily raise the drift budget to replace existing nodes. |
2. Use latest with a zero drift budget
Set amiVersion: latest to select the newest AMI. Set nodes: "0" for the Drifted reason to block automatic replacement of existing nodes.
Pros | Cons |
|---|---|
New nodes receive the latest AMI. | Existing nodes do not update automatically. |
The team does not need to update | Drift can build up without an alert or pull request. |
Karpenter cannot cause an unexpected drift replacement. | The team must temporarily raise the drift budget to replace existing nodes. |
This combination is reasonable only if the team monitors AMI drift and has a separate process for replacing protected nodes. Without that process, it is the trap described in this post.
3. Use latest with a maintenance window
Set amiVersion: latest to select the newest AMI for new nodes. Set nodes: "1" for the Drifted reason during a scheduled window so Karpenter can replace one drifted node at a time:
budgets: - nodes: "0" reasons: ["Drifted"] - nodes: "1" reasons: ["Drifted"] schedule: "0 2 * * MON" duration
The drift budget is zero outside the window, so existing drifted nodes keep running. During the window, the budget is one, so Karpenter may replace one drifted node at a time.
Pros | Cons |
|---|---|
Existing nodes update without a manual budget change. | Replacement still disrupts workloads during the window. |
The team controls when replacements can happen. | Several drifted nodes may take several windows to replace. |
New nodes receive the latest AMI. | The workloads still need correct PodDisruptionBudgets. |
A node can remain behind until the next maintenance window. |
4. Use latest with a one-node drift budget
Set amiVersion: latest to select the newest AMI for new nodes. Set nodes: "1" for the Drifted reason with no schedule so Karpenter can replace one drifted node at any time.
Pros | Cons |
|---|---|
Existing nodes update as soon as Karpenter detects drift. | The team does not control when replacement starts. |
No manual budget change is required. | Replacement may start during peak traffic or an incident. |
New nodes receive the latest AMI. | Safe rollout depends on the workloads’ PodDisruptionBudgets and rescheduling behavior. |
Only one drifted node is replaced at a time. | AWS AMI release timing determines the disruption schedule. |
Conclusion
Before changing a pinned AMI to latest, check your NodePool’s disruption budget:
amiVersionselects the AMI for new nodes.The disruption budget controls whether Karpenter can replace existing nodes when their AMI drifts.
With
nodes: "0"forDrifted, existing nodes stay on their old AMI until the team raises the budget.
Keep the AMI pinned for explicit version control, schedule a maintenance window for automatic replacement at a known time, or allow one drift replacement at any time. If you use latest with a zero drift budget, monitor drift and document the manual replacement process.
If you liked this post, you might find these interesting too
Share
Share
On this page
Section
On this page
In this post
section
Share
Related Content

Kong API Gateway for Gaming and Sports Betting Integrations

Modernising Observability for a Billion Dollar Revenue Gaming Platform

Migrating from AWS Glue to Self-Hosted ClickHouse: 70% Cost Reduction for a Fintech Client.

How we reduced Kafka message latency by 90% and increased throughput 4x for a $300M SaaS platform.













