KubeMedic: an AI agent that diagnoses failing pods, and is never allowed to fix them

ravidorairajKubernetes, ai2 days ago11 Views

I built a small tool that looks at broken Kubernetes pods and explains what is wrong. It reads the cluster, collects the evidence, and writes a report with a suggested fix. It cannot change anything. That is the whole point of the design.

Then I broke six apps on purpose and ran it against them. This post covers how it works and what actually happened.

How it works

kubectl get pods → evidence (status, events, logs, limits) → deterministic rules → LLM explains → Markdown report
  1. Collect. For each unhealthy pod it gathers the pod status, recent events, current and previous logs, the owning Deployment and the node. Anything that looks like a password, token or API key is masked before it goes anywhere.
  2. Classify with plain rules. Simple code decides what is wrong: exit code 137 with OOMKilled means out of memory, ImagePullBackOff means a bad image, FailedScheduling means the pod can’t be placed, and so on.
  3. Explain with an LLM. The model gets the findings and the evidence and writes: what’s wrong, why (with evidence), a suggested fix, and its confidence. It gets no tools. It can only write text.

It supports a local model through Ollama, or Claude through the Anthropic API.

Why it can’t fix anything

There are two separate guards, so a bug in one doesn’t open the door.

In the code. Every cluster call goes through one function that only allows read verbs:

READ_ONLY_VERBS = {"get", "describe", "logs", "top", "auth", "version", "api-resources"}

Anything else raises WriteAttemptBlocked, and there are tests for it.

In the cluster. KubeMedic logs in as its own ServiceAccount. Its ClusterRole only allows get, list and watch, and gives no access to Secrets. Here is what the cluster said when I asked what that identity can do:

get     pods         yes
list    events       yes
get     pods/log     yes
delete  pods         no
patch   deployments  no
create  pods         no
get     secrets      no

Even if the code guard had a bug, the cluster would refuse the write.

The test: six broken apps

I created a local cluster with kind and deployed six workloads, each broken in a different way:

AppWhat I brokePod status
memory-hogallocates 150 MB with a 64Mi limitOOMKilled
payments-apirequired DATABASE_URL missing, exits 1Error / CrashLoop
web-frontendimage nginx:1.99.99-doesnotexistImagePullBackOff
report-workerreferences a Secret that doesn’t existCreateContainerConfigError
catalog-svcliveness probe on port 8080, nginx listens on 80CrashLoopBackOff
ml-batchrequests 512Gi of memoryPending

Results

Run with Ollama and llama3.2:3b on my laptop:

  • The rules found the right cause for all 6. OOM_KILLED, APP_EXIT, IMAGE_PULL, CONFIG_REF_MISSING, PROBE_FAILING, UNSCHEDULABLE.
  • Speed: about 4–5 seconds per pod for the explanation, 38 seconds for the whole scan.
  • Nothing in the cluster was changed.

The explanations were useful, but not perfect:

  • It contradicted itself. In the memory-hog report it correctly quoted the 64Mi limit, then said its confidence was medium because “the specific value of the limit is not provided.”
  • Its confidence ratings were off. It rated most diagnoses “Low”, including the image tag that simply doesn’t exist, which is about as clear as evidence gets.
  • It pointed in the right direction without the final step. For catalog-svc it said to check the probe port and that the app is listening on it, which is correct. It didn’t say outright that nginx listens on 80.

This is why the rules decide what is wrong and the model only explains it. The diagnosis stays correct even when the model’s wording is shaky.

Leave a reply

Loading Next Post...
Search
Loading

Signing-in 3 seconds...

Signing-up 3 seconds...

Cart
Cart updating

ShopYour cart is currently is empty. You could visit our shop and start shopping.