Introduction

Kubernetes adoption has exploded, and most teams have gotten good at keeping workloads running in production. But diagnosing why something isn't working is still a manual process. Engineers often jump between kubectl commands, pod logs, events, manifests, and documentation before arriving at a fix.

Tools like K8sGPT significantly reduce this effort by analyzing cluster state and explaining issues in natural language using Large Language Models (LLMs). Instead of simply reporting that a Deployment is in an ImagePullBackOff state, K8sGPT can explain why the image pull failed and suggest a possible remediation.

But what if K8sGPT could go one step further and apply the fix automatically?

K8sGPT's Auto Remediation feature in operator mode attempts to automate this entire workflow using an LLM. At first glance, it might seem like K8sGPT simply sends a broken manifest to an LLM, receives an updated YAML, and applies it to the cluster. After tracing the implementation, we found the architecture is much more sophisticated.

Under the hood, the operator creates multiple Custom Resources, coordinates two independent controllers, validates AI-generated manifests, and waits for a future analysis cycle to confirm whether the issue has actually been resolved.

In this article, we'll trace that entire flow of auto remediation i.e from detecting an issue to verifying that it has been resolved.

Why Auto Remediation needs more than an LLM

Generating a YAML manifest is only one part of solving an operational problem. Before an AI-generated change can safely modify a production workload, several questions need to be answered:

• Which Kubernetes object should actually be modified?

• How do we know the AI hasn't rewritten the entire manifest?

• What happens if multiple Pods report the same issue?

• How do we verify the fix actually resolved the problem?

• Can we retry or abort the remediation safely?

These questions are why K8sGPT doesn't simply patch Kubernetes resources after analysis. Instead, it decomposes the workflow into multiple stages, each responsible for a single concern.

High-Level Architecture

The K8sGPT CR holds configuration for the analysis loop like scan interval, the AI backend to use and notification sinks.

The K8sGPT Operator periodically analyzes the cluster by querying the K8sGPT Server, which in turn talks to the configured AI provider. Each analysis result is written back as a Result CR.

Once Auto Remediation is enabled, the operator also creates a Mutation CR for every eligible Result, each one representing a proposed fix.

The proposed fix by Mutation CR is then applied to the corresponding Kubernetes resource.

On the next analysis cycle, the operator checks whether the issue is still present. If it's gone, the Result CR is cleaned up and the remediation is marked successful. If the issue persists, the Mutation CR stays in a pending state and the check repeats on the next cycle.

This loop “analyze → propose → apply → re-analyze” is what lets K8sGPT verify a fix rather than just assume it worked.

So far we've treated the operator as one unit. Next, we'll open it up.

Two Controllers, One Feature

The previous diagram shows a single "K8sGPT Operator" box but that's a simplification. Under the hood, Auto Remediation is split across two independent controllers, each owning one half of the workflow.

The K8sGPT Controller owns the analysis pipeline. It scans the cluster, queries the K8sGPT Server, and writes results as Result CRs. When Auto Remediation is enabled, it also creates a Mutation CR for every eligible Result. Initially, the Mutation stores references to the affected resource along with a snapshot of its current configuration.

The Mutation Controller owns the remediation pipeline. It evaluates each Mutation, generates a proposed by querying the K8sGPT Server, validates it against the configured similarity threshold, and then applies the change to the appropriate Kubernetes resource.

The two controllers don't coordinate through the Mutation CR alone they also share a single gRPC client to the K8sGPT Server, passed through a Go channel after the K8sGPT Controller establishes its connection to the server. So the Mutation Controller isn't opening its own connection to the server, it's reusing the same one the K8sGPT Controller already set up.

This split keeps the two pipelines loosely coupled. The K8sGPT Controller doesn't need to know whether a fix succeeded and the Mutation Controller doesn't need to know how the issue was originally detected. Each controller focuses on a single responsibility, passing work to the next stage through custom resources.

Phase 1 : Cluster Analysis

Everything begins with the Analysis step.

The operator periodically invokes the K8sGPT server, which scans the cluster and returns a list of detected issues.

Each issue is stored as a Result Custom Resource.

A Result contains:

• affected Kubernetes resource ( name, kind and parent object )

• detected errors from kubernetes

• AI-generated explanation

• lifecycle metadata

One particular implementation detail is how K8sGPT determines whether a Result has changed.

Instead of hashing the complete Result object, it hashes only the stable Kubernetes information such as the resource name, kind, and error messages. The AI-generated explanation is intentionally excluded because LLMs may describe the same underlying problem differently across analysis runs.

This prevents Results from constantly appearing as "updated", which in turn avoids duplicate notifications to sinks like Slack for a problem that hasn't actually changed.

Phase 2 : From Results to Mutations

A Result describes what is wrong.

A Mutation represents how K8sGPT intends to fix it.

Once analysis completes, the operator converts eligible Results into Mutation resources.

During this process the operator:

• fetches the current Kubernetes object

• stores its YAML as originConfiguration

• links the corresponding Result

• Initializes the Mutation state as NotStarted

One interesting optimization happens before Mutation creation.

If multiple Pods belonging to the same Deployment report identical issues, the operator creates only one Mutation instead of one per Pod. Since Kubernetes expects configuration changes to happen at the Deployment level, creating multiple identical remediations at pod level would be redundant.

Phase 3 : Mutation Controller

At this point, the K8sGPT Controller's job is finished. The Mutation Controller now reconciles every Mutation independently. Its lifecycle can be represented as a state machine.

Each state has a specific responsibility.

• NotStarted – Generate a remediation using the AI backend.

• InProgress – Validate the generated configuration and apply it.

• Completed – Check whether the Result CR has already cleared.

• Pending – Continue polling until the issue disappears.

• Successful – Confirm remediation succeeded.

• Aborted – Stop processing because the generated configuration did not meet the configured similarity threshold.

Rather than treating remediation as one long operation, K8sGPT breaks it into a series of well-defined phases. This allows the workflow to safely continue even after retries or controller restarts.

AI Prompting

One of the most surprising discoveries while tracing the implementation was that Auto Remediation doesn't involve a single AI request.

Instead, two different prompts participate in the workflow.

The first prompt asks the LLM to generate a corrected manifest based on the detected issue and the original resource configuration.

If the affected resource is a Pod managed by a Deployment, K8sGPT performs a second AI interaction. Instead of modifying the Pod directly, it asks the LLM to transform the proposed Pod fix into a Deployment update, ensuring the remediation follows Kubernetes' ownership model.

This second AI interaction is worth pausing on. The similarity check that gates remediation by comparing the AI's fix against similarityRequirement only runs once, against the Pod-level targetConfiguration. The Deployment manifest produced by the second prompt isn't evaluated against the same threshold, nor is it stored back in the Mutation's targetConfiguration.

In practice, this means the manifest actually applied to the cluster can end up different from the one that was validated, and the Mutation CR may not fully reflect what was changed.

Similarity Score

Allowing an LLM to modify production manifests requires safeguards.

Before applying a generated manifest, K8sGPT computes a similarity score between the original configuration and the AI-generated remediation using “Levenshtein distance”.

If the similarity score falls below the configured threshold, the Mutation is marked as Aborted and the remediation is not intended to be applied.

This mechanism attempts to prevent overly aggressive AI modifications while still allowing meaningful configuration changes.

How K8sGPT knows remediation succeeded

The important part of the design is how success is determined.

The Mutation Controller never checks whether a Pod is healthy or whether a Deployment became Available. Instead, it waits for the next K8sGPT analysis.

If the next analysis no longer detects the issue, the corresponding Result CR is deleted. The Mutation Controller interprets the disappearance of that Result as proof that remediation succeeded.

This creates a clean feedback loop where analysis validates remediation rather than the remediation controller implementing its own healthcheck logic.

Implementation Observations

While tracing the implementation, we came across a few interesting observations that needs to be highlighted :

1. Service and Ingress Mutations Are Created but Never Executed.

   While Service and Ingress resources are eligible for Mutation creation, execution handlers currently exist only for Pod and Deployment resources, so these Mutations are silently skipped, with no error or status update.

2. Similarity Scoring Measures Text, Not Intent.

   The similarity threshold compares YAML using Levenshtein distance, measuring character differences rather than Kubernetes object semantics. It should therefore be treated as a heuristic guardrail, not a semantic validation.

3. Deployment Remediation Uses a Second LLM Prompt.

   For Deployment-backed Pods, K8sGPT performs a second LLM prompt to generate the Deployment update. As a result, the manifest ultimately applied to the cluster can differ from the one that originally passed the similarity check.