When a Kubernetes Controller Silently Stops Watching One Resource Type

A platform team reprovisioned the apps. Afterwards, every newly created install — a Jupyter instance, for example — returned 404 on its domain. The pod was running. The Service had endpoints. The Ingress existed.

The domain simply did not resolve to anything, because nothing had told the ingress proxy it existed.

Symptom

New installs reachable only by internal IP, never by domain
Existing installs unaffected
kubectl get pods — healthy, Running
Ingress object present in the cluster
curl https://<new-host>/ → 404

Existing workloads working while new ones do not is the most useful clue in this class of problem: it rules out a total outage and points at a processing gap.

Diagnosis

The ingress controller in this platform annotates each Ingress with a resource-id after creating the corresponding proxy resource. Compare a new Ingress against an old one:

kubectl get ingress <new> -o yaml | grep -A2 annotations
kubectl get ingress <old> -o yaml | grep -A2 annotations

The old one carries pangolin.robustgx.com/resource-id. The new one has no annotation at all.

Controller logs confirmed it:

on_ssh_service handlers fired for the new Service
no on_ingress invocation whatsoever for the new Ingress

So the controller was alive and reacting — to one resource type. It had simply stopped watching the other.

curl https://<new-host>/  ->  404
GET /org/<org>/resources  ->  resource does not exist

The proxy side was consistent with the cluster side: no resource, no route, 404.

Root cause

The controller watches Ingresses through a Kopf ingress informer. That informer was wedged after an API server outage during the reprovision, when the server logged storage is (re)initializing and the client hit MaxRetryError against the API endpoint.

Kopf informers recover from transient failures by re-listing. This one did not: the watch had died and the re-list never happened, so the event stream for Ingress was permanently empty while the Service stream carried on normally.

That is the whole failure mode — not an expired API key, not a permission problem, not a broken controller deployment. One informer, silently dead, for hours.

The fix

kubectl -n pangolin-autodns rollout restart deployment/pangolin-autodns

On restart, Kopf re-lists every Ingress (on.resume), creates the missing proxy resources, stamps the annotations, and reconciles everything it missed. No manual patching of individual Ingresses, no data migration.

Verification

# annotation now present on the previously-stuck Ingress
kubectl get ingress <new> -o jsonpath='{.metadata.annotations}'

# the domain serves the app again
curl -sI https://<new-host>/ | head -1

# and the proxy side agrees
GET /org/<org>/resources   # resource now listed

All three confirmed, and the developer confirmed end-to-end access.

The lesson worth keeping

Restart the controller first when the symptom is “nothing happened at all”.

The distinguishing detail is the shape of the failure:

Symptom	Likely cause	Action
Annotation missing entirely, no retries in logs	informer stuck	restart the controller
Annotation missing, repeated errors in logs	API auth / permissions	fix credentials
Annotation present, but wrong or stale value	reconciliation logic	inspect the handler

If the controller is retrying and failing, you have a credentials or permissions problem. If it is not retrying and not succeeding, the event never reached it — and no amount of fixing the API key will unstick an informer.

The secondary lesson is about watch-based controllers generally: a controller can be healthy, responsive, and half-blind. /healthz returning 200 proves the process is up, not that every informer is receiving events. If a platform has per-resource-type reconciliation paths, its monitoring should assert that each one is producing events — otherwise this failure is invisible until a user reports a 404.