Fast Deploy Direct & Linked (Experimental)

This patch adds support for the Fast Deploy Direct and Linked features,
i.e. the ability to cache images per-datastore and quickly provision a
VM from these caches, either directly or as a linked clone. This is an
experimental feature that must be enabled manually. There are many
things about this feature that may change prior to it being ready for
production.

The patch notes below are broken down into several sections:

* **Goals** -- What is currently supported
* **Non-goals** -- What is not on the table right now
* **Architecture**
    * **Activation** -- How to enable this experimental feature
    * **Placement** --  Request datastore recommendations
    * **Image cache** -- A general-purpose VM image cache
    * **Create VM** -- Create directly from cached disk

The following goals are what is considered in-scope for this
experimental feature at this time. Just because something is not listed,
it does not mean it will not be added before the feature is made
generally available:

* Support all VM images that are OVFs
* Support multiple zones
* Support workload-domain isolation
* Support all datastore types, including host-local and vSAN
* Support for configuring a default fast-deploy mode
* Support picking the fast-deploy mode per VM (direct, linked)
* Support disabling fast-deploy per VM
* Support VM encryption for VMs deployed with fast deploy direct
* Support backup/restore for VMs deployed with fast deploy direct
* Support site replication for VMs deployed with fast deploy direct
* Support datastore maintenance/migration for VMs deployed with fast
  deploy direct

The following is a list of non-goals that are not in scope at this time,
although most of them should be revisited prior to this feature
graduating to production:

* Support VM images that are VM templates (VMTX)

    The architecture behind Fast Deploy makes it trivial to support
    deploying VM images that point to VM templates. While not in scope
    at this time, it is likely this becomes part of the feature prior to
    it graduating to production-ready.

The architecture is broken down into the following sections:

* **Activation** -- How to enable this experimental feature
* **Placement**  -- Request datastore recommendations
* **Image cache** -- A general-purpose VM image cache
* **Create VM**  -- Create directly from cached disk

Enabling the experimental Fast Deploy feature requires setting the
environment variable `FSS_WCP_VMSERVICE_FAST_DEPLOY` to `true` in the VM
Operator deployment. The environment variable `FAST_DEPLOY_MODE` may be
set to one of the following values to configure the default mode for the
fast-deploy feature:

* `direct` -- VMs are deployed using cached disks
* `linked` -- VMs are deployed as a linked clone
* the value is empty -- `direct` mode is used
* the value is anything else -- fast deploy is disabled

It is possible to override the default mode per-VM by setting the
annotation `vmoperator.vmware.com/fast-deploy`. The values of this
annotation follow the same rules described above.

Please note, setting the environment variable `FAST_DEPLOY_MODE` or the
annotation `vmoperator.vmware.com/fast-deploy` has no effect if the
feature is not enabled.

Please refer to PR #823 for information on placement as the logic from
that change has stayed the same in this one.

The way the images/disks are cached has completely changed since PR

* not visible to DevOps users
* a namespace-scoped resource that only exists in the same namespace as
  the VM Operator controller pod
* used to cache the OVF and an image's disks

A `VirtualMachineImageCache` resource is created per unique library item
resource. That means even if there are 20,000 VMI resources spread
across a multitude of namespaces or at the cluster scope, if they all
point to the same underlying library item, then for all those VMI
resources there will be a single `VirtualMachineImageCache` resource in
the VM Operator namespace.

The `VirtualMachineImageCache` controller caches the OVF for the image
in a `ConfigMap` resource in the VM Operator namespace. This completely
obviates the need to maintain a bespoke, in-memory OVF cache.

The `VirtualMachineImageCache` resource caches the image's disks on
specified datastores by setting `spec.locations` with entries that map
to unique datacenter/datastore IDs. The resource's status reveals the
location(s) of the cached disk(s).

For a more in-depth look on how the disks are actually cached, please
refer to PR #823.

If the `VirtualMachineImageCache` object is not ready with the cached
OVF or disks, then the VM will be re-enqueued once the
`VirtualMachineImageCache` _is_ ready. Please note, while placement is
required to know where to cache the disks, additional placement calls
are not issued if a VM is actively awaiting a `VirtualMachineImageCache`
resource. Beyond that, the create VM workflow depends on the fast-deploy
mode:

1. The cached disks are copied into the VM's folder.

2. The ConfigSpec is updated to reference the disks.

  a. Please note, if the VM is encrypted, the disks are not as part of
     the create call. This is because it is not possible to change the
     encryption state of disks when adding them to a VM. Thus the disks
     are encrypted after the VM is created, before it is powered on.

3. The `CreateVM_Task` VMODL1 API is used to create the VM.

1. The `VirtualDisk` devices in the ConfigSpec used to create the VM are
   updated with `VirtualDiskFlatVer2BackingInfo` backings that specify a
   parent backing which refers to the cached, base disk from above.

   The path to each of the VM's disks is constructed based on the index
   of the disk, ex.:
   `[<DATASTORE>] <KUBE_VM_OBJ_UUID>/<KUBE_VM_NAME>-<DISK_INDEX>.vmdk`.

2. The `CreateVM_Task` VMODL1 API is used to create the VM. Because the
   the VM's disks have parent backings, this new VM is effectively a
   linked clone.

.golangci.yml

@@ -82,6 +82,8 @@ linters-settings:
         pkg: github.com/vmware-tanzu/vm-operator/pkg/config
       - alias: pkgctx
         pkg: github.com/vmware-tanzu/vm-operator/pkg/context
+      - alias: pkgerr
+        pkg: github.com/vmware-tanzu/vm-operator/pkg/pkgerr
       - alias: ctxop
         pkg: github.com/vmware-tanzu/vm-operator/pkg/context/operation
       - alias: pkgmgr
@@ -90,6 +92,10 @@ linters-settings:
         pkg: github.com/vmware-tanzu/vm-operator/pkg/util
       - alias: proberctx
         pkg: github.com/vmware-tanzu/vm-operator/pkg/prober/context
+      - alias: dsutil
+        pkg: github.com/vmware-tanzu/vm-operator/pkg/util/vsphere/datastore
+      - alias: clsutil
+        pkg: github.com/vmware-tanzu/vm-operator/pkg/util/vsphere/library
 
   depguard:
     rules:

api/v1alpha3/virtualmachineimage_types.go

@@ -252,6 +252,14 @@ type VirtualMachineImageStatus struct {
 	Type string `json:"type,omitempty"`
 }
 
+func (i VirtualMachineImageStatus) GetConditions() []metav1.Condition {
+	return i.Conditions
+}
+
+func (i *VirtualMachineImageStatus) SetConditions(conditions []metav1.Condition) {
+	i.Conditions = conditions
+}
+
 // +kubebuilder:object:root=true
 // +kubebuilder:resource:scope=Namespaced,shortName=vmi;vmimage
 // +kubebuilder:storageversion
@@ -273,7 +281,7 @@ type VirtualMachineImage struct {
 	Status VirtualMachineImageStatus `json:"status,omitempty"`
 }
 
-func (i *VirtualMachineImage) GetConditions() []metav1.Condition {
+func (i VirtualMachineImage) GetConditions() []metav1.Condition {
 	return i.Status.Conditions
 }
 
@@ -311,7 +319,7 @@ type ClusterVirtualMachineImage struct {
 	Status VirtualMachineImageStatus `json:"status,omitempty"`
 }
 
-func (i *ClusterVirtualMachineImage) GetConditions() []metav1.Condition {
+func (i ClusterVirtualMachineImage) GetConditions() []metav1.Condition {
 	return i.Status.Conditions
 }
 

api/v1alpha3/virtualmachineimagecache_types.go

@@ -0,0 +1,202 @@
+// // © Broadcom. All Rights Reserved.
+// The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
+// SPDX-License-Identifier: Apache-2.0
+
+package v1alpha3
+
+import (
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+const (
+	// VirtualMachineImageCacheConditionProviderReady indicates the underlying
+	// provider is fully synced, including its disks being available locally on
+	// some datastore.
+	VirtualMachineImageCacheConditionProviderReady = "VirtualMachineImageCacheProviderReady"
+
+	// VirtualMachineImageCacheConditionDisksReady indicates the disks are
+	// cached in a given location.
+	VirtualMachineImageCacheConditionDisksReady = "VirtualMachineImageCacheDisksReady"
+
+	// VirtualMachineImageCacheConditionOVFReady indicates the OVF is cached.
+	VirtualMachineImageCacheConditionOVFReady = "VirtualMachineImageCacheOVFReady"
+)
+
+type VirtualMachineImageCacheObjectRef struct {
+	// Kind is a string value representing the REST resource this object
+	// represents.
+	// Servers may infer this from the endpoint the client submits requests to.
+	// Cannot be updated.
+	// In CamelCase.
+	// More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+	Kind string `json:"kind"`
+
+	// +optional
+
+	// Namespace refers to a namespace.
+	// This field will be empty if Kind refers to a cluster-scoped resource.
+	Namespace string `json:"namespace,omitempty"`
+
+	// Name refers to a unique resource.
+	// More info: http://kubernetes.io/docs/user-guide/identifiers#names
+	Name string `json:"name"`
+}
+
+type VirtualMachineImageCacheLocationSpec struct {
+	// DatacenterID describes the ID of the datacenter to which the image should
+	// be cached.
+	DatacenterID string `json:"datacenterID"`
+
+	// DatastoreID describes the ID of the datastore to which the image should
+	// be cached.
+	DatastoreID string `json:"datastoreID"`
+}
+
+// VirtualMachineImageCacheSpec defines the desired state of
+// VirtualMachineImageCache.
+type VirtualMachineImageCacheSpec struct {
+	// ProviderID describes the ID of the provider item to which the image
+	// corresponds.
+	// If the provider is Content Library, the ID refers to a Content Library
+	// item.
+	ProviderID string `json:"providerID"`
+
+	// ProviderVersion describes the version of the provider item to which the
+	// image corresponds.
+	// The provider is Content Library, the version is the content version.
+	ProviderVersion string `json:"providerVersion"`
+
+	// +optional
+	// +listType=map
+	// +listMapKey=datacenterID
+	// +listMapKey=datastoreID
+
+	// Locations describes the locations where the image should be cached.
+	Locations []VirtualMachineImageCacheLocationSpec `json:"locations,omitempty"`
+}
+
+func (s *VirtualMachineImageCacheSpec) SetLocation(dcID, dsID string) {
+	for i := range s.Locations {
+		l := s.Locations[i]
+		if l.DatacenterID == dcID && l.DatastoreID == dsID {
+			return
+		}
+	}
+	s.Locations = append(s.Locations, VirtualMachineImageCacheLocationSpec{
+		DatacenterID: dcID,
+		DatastoreID:  dsID,
+	})
+}
+
+type VirtualMachineImageCacheLocationStatus struct {
+
+	// DatacenterID describes the ID of the datacenter to which the image should
+	// be cached.
+	DatacenterID string `json:"datacenterID"`
+
+	// DatastoreID describes the ID of the datastore to which the image should
+	// be cached.
+	DatastoreID string `json:"datastoreID"`
+
+	// +optional
+
+	// Files describes the paths to the image's cached files on this datastore.
+	Files []string `json:"files,omitempty"`
+
+	// +optional
+
+	// Conditions describes any conditions associated with this cache location.
+	//
+	// Generally this should just include the ReadyType condition.
+	Conditions []metav1.Condition `json:"conditions,omitempty"`
+}
+
+func (i VirtualMachineImageCacheLocationStatus) GetConditions() []metav1.Condition {
+	return i.Conditions
+}
+
+func (i *VirtualMachineImageCacheLocationStatus) SetConditions(conditions []metav1.Condition) {
+	i.Conditions = conditions
+}
+
+type VirtualMachineImageCacheOVFStatus struct {
+
+	// +optional
+
+	// ConfigMapName describes the name of the ConfigMap resource that contains
+	// the image's OVF envelope encoded as YAML. The data is located in the
+	// ConfigMap key "value".
+	ConfigMapName string `json:"configMapName,omitempty"`
+
+	// +optional
+
+	// ProviderVersion describes the observed provider version at which the OVF
+	// is cached.
+	// The provider is Content Library, the version is the content version.
+	ProviderVersion string `json:"providerVersion,omitempty"`
+}
+
+// VirtualMachineImageCacheStatus defines the observed state of
+// VirtualMachineImageCache.
+type VirtualMachineImageCacheStatus struct {
+
+	// +optional
+	// +listType=map
+	// +listMapKey=datacenterID
+	// +listMapKey=datastoreID
+
+	// Locations describe the observed locations where the image is cached.
+	Locations []VirtualMachineImageCacheLocationStatus `json:"locations,omitempty"`
+
+	// +optional
+
+	// OVF describes the observed status of the cached OVF content.
+	OVF *VirtualMachineImageCacheOVFStatus `json:"ovf,omitempty"`
+
+	// +optional
+
+	// Conditions describes any conditions associated with this cached image.
+	//
+	// Generally this should just include the ReadyType condition, which will
+	// only be True if all of the cached locations also have True ReadyType
+	// condition.
+	Conditions []metav1.Condition `json:"conditions,omitempty"`
+}
+
+func (i VirtualMachineImageCache) GetConditions() []metav1.Condition {
+	return i.Status.Conditions
+}
+
+func (i *VirtualMachineImageCache) SetConditions(conditions []metav1.Condition) {
+	i.Status.Conditions = conditions
+}
+
+// +kubebuilder:object:root=true
+// +kubebuilder:resource:scope=Namespaced,shortName=vmic;vmicache;vmimagecache
+// +kubebuilder:storageversion
+// +kubebuilder:subresource:status
+
+// VirtualMachineImageCache is the schema for the
+// virtualmachineimagecaches API.
+type VirtualMachineImageCache struct {
+	metav1.TypeMeta   `json:",inline"`
+	metav1.ObjectMeta `json:"metadata,omitempty"`
+
+	Spec   VirtualMachineImageCacheSpec   `json:"spec,omitempty"`
+	Status VirtualMachineImageCacheStatus `json:"status,omitempty"`
+}
+
+// +kubebuilder:object:root=true
+
+// VirtualMachineImageCacheList contains a list of VirtualMachineImageCache.
+type VirtualMachineImageCacheList struct {
+	metav1.TypeMeta `json:",inline"`
+	metav1.ListMeta `json:"metadata,omitempty"`
+	Items           []VirtualMachineImageCache `json:"items"`
+}
+
+func init() {
+	objectTypes = append(objectTypes,
+		&VirtualMachineImageCache{},
+		&VirtualMachineImageCacheList{})
+}