Cluster deleted in the Rancher UI. Entry stays in Deleting for hours. The UI keeps showing activity — “reconciling” — with no progress. And in OpenStack, the VMs are still ACTIVE, still consuming quota.

Finalizers are supposed to guarantee clean teardown. When the thing that releases the finalizer can never succeed, they guarantee the opposite.

Symptom

  • Cluster entry stuck in Deleting indefinitely
  • Rancher UI reports continuous reconciliation, no progress
  • Underlying VMs still ACTIVE in OpenStack — quota still consumed
  • Every controller cycle logs an update; nothing is actually removed

The command ladder

Each step narrows the problem. Do not skip ahead — the answer for step 5 is meaningless without step 1.

Step 1 — Which cluster objects remain, and do they carry a deletion timestamp?

export KUBECONFIG=~/kubeconfig-rancher-os.yaml

kubectl -n fleet-default get clusters.provisioning.cattle.io \
  -o custom-columns=NAME:.metadata.name,DELETING:.metadata.deletionTimestamp,FINALIZERS:.metadata.finalizers

Step 2 — Are the machines gone, or also stuck?

kubectl -n fleet-default get machines
kubectl -n fleet-default get openstackmachines

Step 3 — Is the management-level cluster still there?

kubectl get clusters.management.cattle.io

Step 4 — What finalizers are on the machines?

kubectl -n fleet-default get machines \
  -o custom-columns=NAME:.metadata.name,FINALIZERS:.metadata.finalizers

Step 5 — What finalizers are on the infrastructure machines?

kubectl -n fleet-default get openstackmachines \
  -o custom-columns=NAME:.metadata.name,FINALIZERS:.metadata.finalizers

Step 6 — Which stage is the machine stuck in?

kubectl -n fleet-default get openstackmachine <name> \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'

This is where it becomes visible: CreateJob=Unknown, reason creating machine provision job.

Step 7 — Are the provision Jobs healthy?

kubectl -n fleet-default get jobs | grep machine-provision

Step 8 — Are the VMs still alive in OpenStack?

openstack server list --name rancher-os-cluster -c ID -c Name -c Status

Yes. Orphans.

Step 9 — Which namespace hosts the CAPI controller?

kubectl get pods -A | grep -E "capi|cluster-api"

Confirm before patching anything — a wrong namespace means you are editing objects nothing watches.

Root cause

Rancher creates the provisioning Cluster
   │
   ├─> machine-provision Job ......................... FAILED (0/1) for 154m
   │      (rancher-system-agent never checked in, so the plan was never applied)
   │
   ├─> OpenstackMachine: CreateJob=Unknown  .......... "creating machine provision job"
   │      finalizer wrangler.cattle.io/machine-provision-remove never released
   │
   ├─> CAPI Machine: stage=WaitingForInfrastructureDeletion
   │      finalizer machine.cluster.x-k8s.io never released
   │
   ├─> bootstrapped finalizers on the Cluster ......... rke-cluster-remove
   │
   └─> Cluster stuck in "Deleting" forever, VMs left ACTIVE (orphans)

Two facts combine into the deadlock:

  1. The provision Job failed. Its root cause was empty server-url / cacerts Rancher settings — Rancher generated cloud-init with no CATTLE_SERVER, so rancher-system-agent aborted with [FATAL] $CATTLE_SERVER was not set and the machine plan was never applied. The Job never reached a successful state, and its retry loop is what produced the repeated VM churn.

  2. Deletion depends on that Job succeeding. The infra-machine finalizer is released by the Job’s cleanup path. Because the Job failed rather than completed, cleanup never ran — so the finalizer, and with it the machine and cluster finalizers, were held permanently. The driver could not delete the VMs either, leaving them ACTIVE in OpenStack.

The deeper lesson: a finalizer whose release condition is “the operation succeeded” deadlocks on a permanently failed operation. Nothing retries the release.

The fix — order matters

Delete the VMs first. Otherwise the cluster object disappears while orphan VMs keep consuming quota. Verify each VM name before deleting.

Step 1 — Delete the orphan VMs

source ~/rancher-openrc.sh && export PATH=~/kolla-ansible/venv/bin:$PATH

for id in <vm-id-1> <vm-id-2>; do
  NM=$(openstack server show $id -c name -f value 2>/dev/null)
  case "$NM" in
    rancher-os-cluster-cp-*) openstack server delete $id && echo "deleted: $NM" ;;
    *) echo "SKIP (not part of this cluster): $NM" ;;
  esac
done

The case guard exists so a mistyped ID can never delete the Rancher management VMs (rancher-01/02/03). Always guard destructive loops by name pattern, never by positional assumption.

Step 2 — Remove the finalizer from the infrastructure machine

for m in <suffix-1> <suffix-2>; do
  kubectl -n fleet-default patch openstackmachine rancher-os-cluster-cp-<pool>-$m \
    --type=merge -p '{"metadata":{"finalizers":[]}}'
done

Step 3 — Remove the finalizer from the CAPI machine

for m in <suffix-1> <suffix-2>; do
  kubectl -n fleet-default patch machine rancher-os-cluster-cp-<pool>-$m \
    --type=merge -p '{"metadata":{"finalizers":[]}}'
done

Remove the infrastructure finalizer first, then the machine finalizer. Doing it in the other order lets the machine controller re-add its finalizer before the infra object is released, and you get to start over.

Step 4 — Wait, then confirm the cluster vanished

sleep 45

kubectl -n fleet-default get machines
kubectl -n fleet-default get openstackmachines
kubectl -n fleet-default get clusters.provisioning.cattle.io
kubectl get clusters.management.cattle.io

Expected: No resources found for the first three, and only local for the last. Rancher releases the cluster finalizers on its own once the machines are gone — no manual patch needed.

Step 5 — Clean up the leftovers

# 5a. keypairs generated by the driver
for kp in $(openstack keypair list -c Name -f value | grep "rancher-os-cluster"); do
  openstack keypair delete "$kp" && echo "deleted keypair: $kp"
done

# 5b. per-machine secrets
kubectl -n fleet-default get secrets --no-headers \
  | awk '/rancher-os-cluster/{print $1}' \
  | while read s; do kubectl -n fleet-default delete secret "$s"; done

# 5c. failed provision jobs
kubectl -n fleet-default delete job rancher-os-cluster-cp-<pool>-<suffix>

# 5d. orphan certificate secrets (safe: no cluster remains in fleet-default)
kubectl -n fleet-default get secrets --no-headers | awk '/machine-certs/{print $1}' \
  | while read s; do kubectl -n fleet-default delete secret "$s"; done

Verification

Do not stop at “the entry is gone”. Check all four:

# nothing left in fleet-default
kubectl -n fleet-default get machines,openstackmachines,clusters.provisioning.cattle.io

# Rancher only knows about local
kubectl get clusters.management.cattle.io

# OpenStack capacity released
openstack server list --name rancher-os-cluster

# quota back to baseline
openstack quota show <project>

A cluster that disappears from Rancher while its VMs keep running is a worse outcome than one stuck in Deleting — because the visible symptom is gone.

Prevention

The real fix is upstream of the deadlock: server-url and cacerts must be set before any provisioning attempt. With those empty, every machine provision Job fails the same way, and every deletion of those machines deadlocks.

Pre-flight check before creating any downstream cluster:

kubectl get setting server-url -o jsonpath='{.value}'
kubectl get setting cacerts    -o jsonpath='{.value}'

Both must be non-empty. If server-url is blank, fix it before provisioning — otherwise you are building machines that cannot be deleted.