reconciler.io is an opinionated framework for authoring and testing Kubernetes reconcilers using controller-runtime project. controller-runtime provides infrastructure for creating and operating controllers, but provides little support for the business logic of implementing a reconciler within the controller. The Reconciler interface provided by controller-runtime is the primary hand-off point with reconciler.io.
Within an existing Kubebuilder or controller-runtime project, reconcilers.io may be adopted incrementally without disrupting existing controllers. A common approach for adopting reconciler.io runtime to use the testing support to test existing reconcilers in a project. If there is a use case runtime does not handle well, you may drop down to the controller-runtime APIs directly, or use any other library that is compatible with controller-runtime.
Reconcilers can operate on three different types of objects:
- structured types (e.g.
corev1.Pod) - unstructured types (e.g.
unstructured.Unstructured) - semi-structured duck types (e.g.
PodSpecable,ProvisionedService)
Structured types are often the best choice as they allow easy interaction with the full object and have full client support. The type must be registered with the Scheme. The type must be pre-defined and compiled into the controller.
Unstructured types are useful when the resources are not known at compile time and full access to the resource and client methods is desired. Since the type is not known in advance, it cannot be registered with the scheme. Interacting with the object is difficult as traversing the object requires lots of casts or reflection. The TypeMeta APIVersion and Kind fields must be defined for the client to operate on the object.
Semi-structured duck types offer a middle ground. They are strongly typed, but only cover a subset of the full object. They are intended to facilitate normalized operations across a number of concrete types that share a common subset of their own schema. The concrete objects compatible with this type are not required to be known at compile time. Because duck types are not full objects, client operations for Create and Update are disallowed (Patch is available). Like unstructured objects, the duck type should not be registered in the scheme, and the TypeMeta APIVersion and Kind fields must be defined for the client to operate on the object.
The controller-runtime client is able to work with structured and unstructured objects natively, reconciler.io runtime adds support for duck typed objects via the duck.NewDuckAwareClientWrapper.
A ResourceReconciler (formerly ParentReconciler) is responsible for orchestrating the reconciliation of a single resource. The reconciler delegates the manipulation of other resources to SubReconcilers.
The resource reconciler is responsible for:
- fetching the resource being reconciled
- creating a stash to pass state between sub reconcilers
- passing the resource to each sub reconciler in turn
- initialize conditions on the status by calling status.InitializeConditions() if defined (not available for Unstructured resources)
- normalizing the .status.conditions[].lastTransitionTime for status conditions that are metav1.Condition (the previous timestamp is preserved if the condition is otherwise unchanged) (not available for Unstructured resources)
- reflects the observed generation on the status (not available for Unstructured resources)
- updates the resource status if it was modified
- logging the reconcilers activities
- records events for mutations and errors
The implementor is responsible for:
- defining the set of sub reconcilers
Example:
Resource reconcilers tend to be quite simple, as they delegate their work to sub reconcilers. We'll use an example from projectriff of the Function resource, which uses Kpack to build images from a git repo. In this case the FunctionTargetImageReconciler resolves the target image for the function, and FunctionChildImageReconciler creates a child Kpack Image resource based on the resolve value.
func FunctionReconciler(c reconcilers.Config) *reconcilers.ResourceReconciler[*buildv1alpha1.Function] {
return &reconcilers.ResourceReconciler[*buildv1alpha1.Function]{
Name: "Function",
Reconciler: reconcilers.Sequence[*buildv1alpha1.Function]{
FunctionTargetImageReconciler(c),
FunctionChildImageReconciler(c),
},
Config: c,
}
}Recommended RBAC:
Replace <group> and <resource> with values for the reconciled resource type.
// +kubebuilder:rbac:groups=<group>,resources=<resource>,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=<group>,resources=<resource>/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=core,resources=events,verbs=get;list;watch;create;update;patch;deleteor
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: # any name that is bound to the ServiceAccount used by the client
rules:
- apiGroups: ["<group>"]
resources: ["<resource>"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["<group>"]
resources: ["<resource>/status"]
verbs: ["get", "update", "patch"]
- apiGroups: ["core"]
resources: ["events"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]An AggregateReconciler is responsible for synthesizing a single resource, aggregated from other state. The AggregateReconciler is a fusion of the ResourceReconciler and ChildReconciler. Instead of operating on all resources of a type, it will only operate on a specific resource identified by the type and request (namespace and name). Unlike the child reconciler, the "parent" and "child" resources are the same.
The aggregate reconciler is responsible for:
- fetching the resource being reconciled
- creating a stash to pass state between sub reconcilers
- passing the resource to each sub reconciler in turn
- creates the resource if it does not exist
- updates the resource if it drifts from the desired state
- deletes the resource if no longer desired
- logging the reconcilers activities
- records events for mutations and errors
The implementor is responsible for:
- specifying the type, namespace and name of the aggregate resource
- defining the desired state
- merging the actual resource with the desired state (often as simple as copying the spec and labels)
Example:
Aggregate reconcilers resemble a simplified child reconciler with many of the same methods combined directly into a parent reconciler. The Reconcile method is used to collect reference data and the DesiredResource method defines the desired state. Unlike with a child reconciler, the desired resource may be a direct mutation of the argument.
In the example, we are controlling and existing ValidatingWebhookConfiguration named my-trigger (defined by Request). Based on other state in the cluster, the Reconcile method delegates to DeriveWebhookRules() to stash the rules for the webhook. Those rules are retrieved in the DesiredResource method, augmenting the ValidatingWebhookConfiguration. The MergeBeforeUpdate function is responsible for merging the desired state into the actual resource, when there is a significant change, the resource is updated on the api server.
The resulting ValidatingWebhookConfiguration will have the current desired rules defined by this reconciler, combined with existing state like the location of the webhook server, and other policies.
// AdmissionTriggerReconciler reconciles a ValidatingWebhookConfiguration object to
// dynamically be notified of resource mutations. A less reliable, but potentially more
// efficient than an informer watching each tracked resource.
func AdmissionTriggerReconciler(c reconcilers.Config) *reconcilers.AggregateReconciler[*admissionregistrationv1.ValidatingWebhookConfiguration] {
return &reconcilers.AggregateReconciler[*admissionregistrationv1.ValidatingWebhookConfiguration]{
Name: "AdmissionTrigger",
Request: reconcilers.Request{
NamesspacedName: types.NamesspacedName{
// no namespace since ValidatingWebhookConfiguration is cluster scoped
Name: "my-trigger",
},
},
Reconciler: reconcilers.Sequence[*admissionregistrationv1.ValidatingWebhookConfiguration]{
DeriveWebhookRules(),
},
DesiredResource: func(ctx context.Context, resource *admissionregistrationv1.ValidatingWebhookConfiguration) (*admissionregistrationv1.ValidatingWebhookConfiguration, error) {
// assumes other aspects of the webhook config are part of a preexisting
// install, and that there is a server ready to receive the requests.
rules := RetrieveWebhookRules(ctx)
resource.Webhooks[0].Rules = rules
return resource, nil
},
MergeBeforeUpdate: func(current, desired *admissionregistrationv1.ValidatingWebhookConfiguration) {
current.Webhooks[0].Rules = desired.Webhooks[0].Rules
},
Sanitize: func(resource *admissionregistrationv1.ValidatingWebhookConfiguration) interface{} {
return resource.Webhooks[0].Rules
},
Config: c,
}
}Recommended RBAC:
Replace <group> and <resource> with values for the reconciled resource type.
// +kubebuilder:rbac:groups=<group>,resources=<resource>,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=events,verbs=get;list;watch;create;update;patch;deleteor
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: # any name that is bound to the ServiceAccount used by the client
rules:
- apiGroups: ["<group>"]
resources: ["<resource>"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["core"]
resources: ["events"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]The SubReconciler interface defines the contract between the host and sub reconcilers.
The SyncReconciler is the minimal type-aware sub reconciler. It is used to manage a portion of the resource reconciliation that is custom, or whose behavior is not covered by another sub reconciler type. Common uses include looking up reference data for the reconciliation, or controlling APIs that are not Kubernetes resources.
When a resource is deleted that has pending finalizers, the Finalize method is called instead of the Sync method. If the SyncDuringFinalization field is true, the Sync method will also by called. If creating state that must be manually cleaned up, it is the users responsibility to define and clear finalizers. Using the finalizer helper methods is strongly encouraged with working under a ResourceReconciler.
Example:
While sync reconcilers have the ability to do anything a reconciler can do, it's best to keep them focused on a single goal, letting the resource reconciler structure multiple sub reconcilers together. In this case, we use the reconciled resource and the client to resolve the target image and stash the value on the resource's status. The status is a good place to stash simple values that can be made public. More advanced forms of stashing are also available. Learn more about status and its contract.
func FunctionTargetImageReconciler(c reconcilers.Config) reconcilers.SubReconciler[*buildv1alpha1.Function] {
return &reconcilers.SyncReconciler[*buildv1alpha1.Function]{
Name: "TargetImage",
Sync: func(ctx context.Context, resource *buildv1alpha1.Function) error {
log := logr.FromContextOrDiscard(ctx)
targetImage, err := resolveTargetImage(ctx, c.Client, resource)
if err != nil {
return err
}
resource.Status.MarkImageResolved()
resource.Status.TargetImage = targetImage
return nil
},
}
}The ChildReconciler is a sub reconciler that is responsible for managing a single controlled resource. Within a child reconciler, the reconciled resource is referred to as the parent resource to avoid ambiguity with the child resource. A developer defines their desired state for the child resource (if any), and the reconciler creates/updates/deletes the resource to match the desired state. The child resource is also used to update the parent's status. Mutations and errors are recorded for the parent.
The ChildReconciler is responsible for:
- looking up an existing child
- creating/updating/deleting the child resource based on the desired state
- setting the owner reference on the child resource (when not using a finalizer)
- logging the reconcilers activities
- enqueuing the parent resource for reconciliation when the child is mutated
- recording child mutations and errors for the parent resource
- adapting to child resource changes applied by mutating webhooks
- adding and clearing of a finalizer, if specified
The implementor is responsible for:
- defining the desired resource
- merging the actual resource with the desired state (often as simple as copying the spec and labels)
- updating the parent's status from the child
- defining the status subresource according to the contract
When a finalizer is defined, the parent resource is patched to add the finalizer before creating the child; it is removed after the child is deleted. If the parent resource is pending deletion, the desired child method is not called, and existing children are deleted.
Using a finalizer means that the child resource will not use an owner reference. The OurChild method must be implemented in a way that can uniquely and unambiguously identify the child that this parent resource is responsible for from any other resources of the same kind. The child resource is tracked explicitly to watch for mutations triggering the parent resource to be reconciled.
Warning: It is crucial that each
ChildReconcilerusing a finalizer have a unique and stable finalizer name. Two reconcilers that use the same finalizer, or a reconciler that changed the name of its finalizer, may leak the child resource when the parent is deleted, or the parent resource may never terminate.
Example:
Now it's time to create the child Image resource that will do the work of building our Function. This reconciler looks more more complex than what we have seen so far, each function on the reconciler provides a focused hook into the lifecycle being orchestrated by the ChildReconciler.
func FunctionChildImageReconciler(c reconcilers.Config) reconcilers.SubReconciler[*buildv1alpha1.Function] {
return &reconcilers.ChildReconciler[*buildv1alpha1.Function, *kpackbuildv1alpha1.Image, *kpackbuildv1alpha1.ImageList]{
Name: "ChildImage",
DesiredChild: func(ctx context.Context, parent *buildv1alpha1.Function) (*kpackbuildv1alpha1.Image, error) {
if parent.Spec.Source == nil {
// don't create an Image, and delete any existing Image
return nil, nil
}
child := &kpackbuildv1alpha1.Image{
ObjectMeta: metav1.ObjectMeta{
Labels: reconcilers.MergeMaps(parent.Labels, map[string]string{
buildv1alpha1.FunctionLabelKey: parent.Name,
}),
Annotations: make(map[string]string),
// Name or GenerateName are supported
GenerateName: fmt.Sprintf("%s-function-", parent.Name),
Namespace: parent.Namespace,
},
Spec: kpackbuildv1alpha1.ImageSpec{
Tag: parent.Status.TargetImage, // value set by sync reconciler
// ... abbreviated
},
}
return child, nil
},
MergeBeforeUpdate: func(actual, desired *kpackbuildv1alpha1.Image) {
// mutate actual resource with desired state
actual.Labels = desired.Labels
actual.Spec = desired.Spec
},
ReflectChildStatusOnParent: func(ctx context.Context, parent *buildv1alpha1.Function, child *kpackbuildv1alpha1.Image, err error) {
// child is the value of the freshly created/updated/deleted child
// resource as returned from the api server
// If a fixed desired resource name is used instead of a generated
// name, check if the err is because the resource already exists.
// The ChildReconciler will not claim ownership of another resource.
//
// See https://github.com/projectriff/system/blob/1fcdb7a090565d6750f9284a176eb00a3fe14663/pkg/controllers/core/deployer_reconciler.go#L277-L283
if child == nil {
// image was deleted
parent.Status.LatestImage = parent.Status.TargetImage
parent.Status.MarkBuildNotUsed()
} else {
// image was created/updated/unchanged
parent.Status.KpackImageRef = refs.NewTypedLocalObjectReferenceForObject(child, c.Scheme)
parent.Status.LatestImage = child.Status.LatestImage
parent.Status.PropagateKpackImageStatus(&child.Status)
}
},
Sanitize: func(child *kpackbuildv1alpha1.Image) interface{} {
// log only the resources spec. If the resource contained sensitive
// values (like a Secret) we'd remove them here so they don't end
// up in our logs
return child.Spec
},
}
}Recommended RBAC:
Replace <group> and <resource> with values for the child type.
// +kubebuilder:rbac:groups=<group>,resources=<resource>,verbs=get;list;watch;create;update;patch;deleteor
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: # any name that is bound to the ServiceAccount used by the client
rules:
- apiGroups: ["<group>"]
resources: ["<resource>"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]The ChildSetReconciler is an orchestration of zero to many, dynamically defined ChildReconcilers. Concepts from ChildReconciler apply here unless noted otherwise.
Unlike ChildReconciler where a single desired child is defined, the ChildSetReconciler returns zero to many desired children, each child resource must contain a stable identifier extracted from the child resource by IdentifyChild, which is used to correlate desired children with actual children within the cluster.
Based on the combined set of identifiers for desired and actual children, a ChildReconciler is created for each identifier. Each ChildReconciler is reconciled in order, sorted by the identifier. The result from each ChildReconciler are aggregated and presented at once to be reflected onto the reconciled resource's status within ReflectChildrenStatusOnParent.
As there is some overhead in the dynamic creation of reconcilers. When the number of children is limited and known in advance, it is preferable to statically construct many ChildReconciler.
When a finalizer is defined, the dynamic reconciler is wrapped with WithFinalizer. Using a finalizer means that the child resource will not use an owner reference. The OurChild method must be implemented in a way that can uniquely and unambiguously identify the children that this parent resource is responsible for from any other resources of the same kind. The child resources are tracked explicitly to watch for mutations triggering the parent resource to be reconciled.
Recommended RBAC:
Replace <group> and <resource> with values for the child type.
// +kubebuilder:rbac:groups=<group>,resources=<resource>,verbs=get;list;watch;create;update;patch;deleteor
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: # any name that is bound to the ServiceAccount used by the client
rules:
- apiGroups: ["<group>"]
resources: ["<resource>"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]Higher order reconcilers are SubReconcilers that do not perform work directly, but instead compose other SubReconcilers in new patterns.
A CastResource (formerly CastParent) casts the ResourceReconciler's type by projecting the resource data onto a new struct. Casting the reconciled resource is useful to create cross cutting reconcilers that can operate on common portion of multiple resource kinds, commonly referred to as a duck type.
The CastResource can also be used to bridge a SubReconciler for a specific struct to a generic SubReconciler that can process any object by using client.Object as the CastType generic type. In this case, the resource is passed through without coercion.
Example:
func FunctionReconciler(c reconcilers.Config) *reconcilers.ResourceReconciler[*buildv1alpha1.Function] {
return &reconcilers.ResourceReconciler[*buildv1alpha1.Function]{
Name: "Function",
Reconciler: reconcilers.Sequence[*buildv1alpha1.Function]{
&reconcilers.CastResource[*buildv1alpha1.Function, *duckv1alpha1.ImageRef]{
// Reconciler that now operates on the ImageRef type. This SubReconciler is likely
// shared between multiple ResourceReconcilers that operate on different types,
// otherwise it would be easier to work directly with the Function type directly.
Reconciler: &reconcilers.SyncReconciler[*duckv1alpha1.ImageRef]{
Sync: func(ctx context.Context, resource *duckv1alpha1.ImageRef) error {
// do something with the duckv1alpha1.ImageRef instead of a buildv1alpha1.Function
return nil
},
},
},
FunctionChildImageReconciler(c),
},
Config: c,
}
}A Sequence composes multiple SubReconcilers as a single SubReconciler. Each sub reconciler is called in turn, aggregating the result of each sub reconciler. A reconciler returning an error will interrupt the sequence.
Example:
A Sequence is commonly used in a ResourceReconciler, but may be used anywhere a SubReconciler is accepted.
func FunctionReconciler(c reconcilers.Config) *reconcilers.ResourceReconciler[*buildv1alpha1.Function] {
return &reconcilers.ResourceReconciler[*buildv1alpha1.Function]{
Name: "Function",
Reconciler: reconcilers.Sequence[*buildv1alpha1.Function]{
FunctionTargetImageReconciler(c),
FunctionChildImageReconciler(c),
},
Config: c,
}
}Advice is a sub reconciler for advising the lifecycle of another sub reconciler in an aspect oriented programming (AOP) style. Before is called before the delegated reconciler and After afterward. Around is used between Before and After to have full control over how the delegated reconciler is called, including suppressing the call, modifying the input or result, or calling the reconciler multiple times.
Example:
Advice can be used to control calls to a reconciler at a lower level. In this case the reconciler is called twice aggregating the results while returning immediately on error.
func CallTwice(reconciler reconciler.SubReconciler[*buildv1alpha1.Function]) *reconcilers.SubReconciler[*buildv1alpha1.Function] {
return &reconcilers.Advice[*buildv1alpha1.Function]{
Reconciler: reconciler,
Around: func(ctx context.Context, resource *resources.TestResource, reconciler reconcilers.SubReconciler[*resources.TestResource]) (reconcile.Result, error) {
result := reconcilers.Result{}
for i := 0; i < 2; i++ {
if r, err := reconciler.Reconcile(ctx, resource); true {
result = reconcilers.AggregateResults(result, r)
} else if err != nil {
return result, err
}
}
return result, nil
},
}
}An IfThen branches execution of the current reconcile request based on a condition. The false Else branch is optional and ignored if not defined.
Example:
An IfThen can be used to gate a capability of the reconciler only for a resource that opts-in to the behavior.
func GatedReconciler() *reconcilers.SubReconciler[*buildv1alpha1.Function] {
return &reconcilers.IfThen[*buildv1alpha1.Function]{
If: func(ctx context.Context, resource *buildv1alpha1.Function) bool {
return resource.Labels["capability-gate.example/enabled"] == "true"
}
Then: reconcilers.Sequence[*buildv1alpha1.Function]{
// use the gated feature
},
}
}A While calls the reconciler so long as the condition is true, up to the maximum number of iterations (defaults to 100). The current iteration index can be retrieved with RetrieveIteration.
This reconciler must not be used to wait for external state to change, or for polling as this will block the reconciler queue. It is best to return with the result requesting to be requeued, or to watch the external state for changes that enqueue the reconcile request.
Example:
An While can be used to fan out.
func TenTimesReconciler() *reconcilers.SubReconciler[*buildv1alpha1.Function] {
return &reconcilers.While[*buildv1alpha1.Function]{
Condition: func(ctx context.Context, resource *buildv1alpha1.Function) bool {
return reconcilers.RetrieveIteration(ctx) < 10
}
Reconciler: reconcilers.SyncReconciler[*buildv1alpha1.Function]{
Sync: func(ctx context.Context, resource *buildv1alpha1.Function) error {
// called ten times
return nil
}
},
}
}A ForEach calls the reconciler for each item returned from Items. A cursor marks the item being reconciled. The current cursor can be retrieved with CursorStasher.
Nested iteration is allowed so long as the types being iterated over contain unique names. Otherwise the stash keys will collide. For testing the nested reconciler outside the scope of the loop, use the CursorStasher's Key method to lookup the StashKey, do not expect the StashKey to be stable between releases.
Example:
A ForEach can be used to interact with each volume mount on a pod.
func VolumeMountReconciler() *reconcilers.SubReconciler[*corev1.Pod] {
containerCursorStasher := reconcilers.CursorStasher[corev1.Container]()
volumeMountCursorStasher := reconcilers.CursorStasher[corev1.VolumeMount]()
return &reconcilers.ForEach[*corev1.Pod, corev1.Container]{
Items: func(ctx context.Context, resource *corev1.Pod) ([]corev1.Container, error) {
return resource.Spec.Containers, nil
}
Reconciler: &reconcilers.ForEach[*corev1.Pod, corev1.VolumeMount]{
Items: func(ctx context.Context, resource *corev1.Pod) ([]corev1.VolumeMount, error) {
containerCursor := containerCursorStasher.RetrieveOrDie(ctx)
return containerCursor.Item.VolumeMounts, nil
}
Reconciler: reconcilers.SyncReconciler[*corev1.Pod]{
Sync: func(ctx context.Context, resource *corev1.Pod) error {
containerCursor := containerCursorStasher.RetrieveOrDie(ctx)
volumeMountCursor := volumeMountCursorStasher.RetrieveOrDie(ctx)
// do something
return nil
}
},
},
}
}A TryCatch is used to recover from errors returned by a reconciler. The Catch method is called with the result and error from the Try reconciler, giving it the option to either continue the existing results, or replace them with new results.
The Finally reconciler is always called before returning, but does not alter the existing result and err values unless it itself errors. The Finally reconciler should avoid complex logic and be limited to cleaning up common state from the Try reconciler.
Example:
A TryCatch can be used to handle errors.
func IgnoreErrorsReconciler() *reconcilers.SubReconciler[*buildv1alpha1.Function] {
return &reconcilers.TryCatch[*buildv1alpha1.Function]{
Try: &reconcilers.SyncReconciler[*buildv1alpha1.Function]{
Sync: func(ctx context.Context, resource *buildv1alpha1.Function) error {
return fmt.Errorf("always error")
}
},
Catch: func(ctx context.Context, resource *buildv1alpha1.Function, result reconcile.Result, err error) (reconcile.Result, error) {
// suppress error
return result, nil
},
}
}An OverrideSetup is used to suppress or replace the setup behavior for a reconciler.
Example:
OverrideSetup is useful when the default setup behavior for a reconciler is problematic in a particular context.
func CustomSetupReconciler() *reconcilers.SubReconciler[*buildv1alpha1.Function] {
return &reconcilers.OverrideSetup[*buildv1alpha1.Function]{
Reconciler: &reconciler.SyncReconciler[*buildv1alpha1.Function]{
Setup: func(ctx context.Context, mgr Manager, bldr *Builder) error {
// not called
panic()
}
Sync: func(ctx context.Context, resource *buildv1alpha1.Function) error {
// called normally
return nil
}
},
Setup: func(ctx context.Context, mgr Manager, bldr *Builder) error {
// custom setup behavior, optional
return nil
},
}
}WithConfig overrides the config that nested reconcilers consume. The config can be retrieved from the context via RetrieveConfig. For interactions with the reconciled resource, the config originally used to load that resource should be used, which can be retrieved from the context via RetrieveOriginalConfig.
Example:
WithConfig can be used to change the REST Config backing the clients. This could be to make requests to the same cluster with a user defined service account, or target an entirely different Kubernetes cluster.
func SwapRESTConfig(rc *rest.Config) *reconcilers.SubReconciler[*resources.MyResource] {
return &reconcilers.WithConfig[*resources.MyResource]{
Reconciler: reconcilers.Sequence[*resources.MyResource]{
LookupReferenceDataReconciler(),
DoSomethingChildReconciler(),
},
Config: func(ctx context.Context, c reconciler.Config) (reconciler.Config, error ) {
// the rest config could also be stashed from a lookup in a SyncReconciler based on a dynamic value
cl, err := clusters.New(rc)
if err != nil {
return reconciler.Config{}, err
}
return c.WithCluster(cl), nil
}
}
}WithFinalizer allows external state to be allocated and then cleaned up once the resource is deleted. When the resource is not terminating, the finalizer is set on the reconciled resource before the nested reconciler is called. When the resource is terminating, the finalizer is cleared only after the nested reconciler returns without an error and ReadyToClearFinalizer returns true.
ReadyToClearFinalizer can be used to define custom rules for clearing the finalizer. For example, deletion of a resource can be blocked until all child resources are fully deleted replicating the behavior of an owner reference in parent-child relationships that are not supported by owner references. Client lookups and advanced logic should be avoided as errors cannot be returned. Computed values can be retrieved that were stashed from a previous reconciler, like a SyncReconciler#Finalize hook.
The Finalizers utilities are used to manage the finalizer on the reconciled resource.
Warning: It is crucial that each WithFinalizer have a unique and stable finalizer name. Two reconcilers that use the same finalizer, or a reconciler that changed the name of its finalizer, may leak the external state when the reconciled resource is deleted, or the resource may never terminate.
Example:
WithFinalizer can be used to wrap any other SubReconciler, which can then safely allocate external state while the resource is not terminating, and then cleanup that state once the resource is terminating.
func SyncExternalState() *reconcilers.SubReconciler[*resources.MyResource] {
return &reconcilers.WithFinalizer[*resources.MyResource]{
Finalizer: "unique.finalizer.name"
Reconciler: &reconcilers.SyncReconciler[*resources.MyResource]{
Sync: func(ctx context.Context, resource *resources.MyResource) error {
// allocate external state
return nil
},
Finalize: func(ctx context.Context, resource *resources.MyResource) error {
// cleanup the external state
return nil
},
},
}
}AdmissionWebhookAdapter allows using SubReconciler to process admission webhook requests. The full suite of sub-reconcilers are available, however, behavior that is generally not accepted within a webhook is discouraged. For example, new requests against the API server are discouraged (reading from an informer is ok), mutation requests against the API Server can cause a loop with the webhook processing its own requests.
All requests are allowed by default unless the response.Allowed field is explicitly set, or the reconciler returns an error. The raw admission request and response can be retrieved from the context via the RetrieveAdmissionRequest and RetrieveAdmissionResponse methods, respectively. The Result typically returned by a reconciler is unused.
The request object is unmarshaled from the request object for most operations, and the old object for delete operations. If the webhhook handles multiple resources or versions of the same resource with different shapes, use of an unstructured type is recommended.
If the resource being reconciled is mutated and the response does not already define a patch, a json patch is computed for the mutation and set on the response.
Testing can be done on the reconciler directly with SubReconcilerTests, or through the webhook with AdmissionWebhookTests.
Example
The Service Binding controller uses a mutating webhook to intercept the creation and updating of workload resources. It projects services into the workload based on ServiceBindings that reference that workload, mutating the resource. If the resource is mutated, a patch is automatically created and added to the webhook response. The webhook allows workloads to be bound at admission time.
func AdmissionProjectorWebhook(c reconcilers.Config) *reconcilers.AdmissionWebhookAdapter[*unstructured.Unstructured] {
return &reconcilers.AdmissionWebhookAdapter{
Name: "AdmissionProjectorWebhook",
Reconciler: &reconcilers.SyncReconciler[*unstructured.Unstructured]{
Sync: func(ctx context.Context, workload *unstructured.Unstructured) error {
c := reconcilers.RetrieveConfigOrDie(ctx)
// find matching service bindings
serviceBindings := &servicebindingv1beta1.ServiceBindingList{}
if err := c.List(ctx, serviceBindings, client.InNamespace(workload.Namespace)); err != nil {
return err
}
// check that bindings are for this specific workload
activeServiceBindings := ...
// project active bindings into workload, the workload is mutated by the projector
projector := projector.New(resolver.New(c))
for i := range activeServiceBindings {
sb := activeServiceBindings[i].DeepCopy()
sb.Default()
if err := projector.Project(ctx, sb, workload); err != nil {
return err
}
}
return nil
},
},
Config: c,
}
}The webhook adapter can be registered with the controller manager at a path, in this case /interceptor. There MutatingWebhookConfiguration resource that intercepts
mgr.GetWebhookServer().Register("/interceptor", controllers.AdmissionProjectorWebhook(config).Build())While controller-runtime focuses its testing efforts on integration testing by spinning up a new API Server and etcd, reconciler.io focuses on unit testing reconcilers. The state for each test case is pure, preventing side effects from one test case impacting the next.
The table test pattern is used to declare each test case in a test suite with the resource being reconciled, other given resources in the cluster, and all expected resource mutations (create, update, delete).
The tests make extensive use of given and mutated resources. It is recommended to use a library like dies to reduce boilerplate code and to highlight the delta unique to each test.
There are three test suites: for testing reconcilers, an optimized harness for testing sub reconcilers, and for testing admission webhooks.
Colorized diffs are available in assertion error messages by setting the environment variable COLOR_DIFF=true
ReconcilerTestCase run the full reconciler via the controller runtime Reconciler's Reconcile method. There are two ways to compose a ReconcilerTestCase either as an unordered set using ReconcilerTests, or an order list using ReconcilerTestSuite. When using ReconcilerTests the key for each test case is used as the name for that test case.
Example:
testRequest := ... // request for the resource to reconcile
inMemoryGatewayImagesConfigMap := ... // ConfigMap with images
inMemoryGateway := ... // resource to reconcile
gatewayCreate := ... // expected to be created
scheme := ... // scheme registered with all resource types the reconcile interacts with
rts := rtesting.ReconcilerTests{
"creates gateway": {
Request: testRequest,
GivenObjects: []client.Object{
inMemoryGateway,
inMemoryGatewayImagesConfigMap,
},
ExpectTracks: []client.Object{
rtesting.NewTrackRequest(inMemoryGatewayImagesConfigMap, inMemoryGateway, scheme),
},
ExpectEvents: []rtesting.Event{
rtesting.NewEvent(inMemoryGateway, scheme, corev1.EventTypeNormal, "Created",
`Created Gateway "%s"`, testName),
rtesting.NewEvent(inMemoryGateway, scheme, corev1.EventTypeNormal, "StatusUpdated",
`Updated status`),
},
ExpectCreates: []client.Object{
gatewayCreate,
},
ExpectStatusUpdates: []client.Object{
// example using an https://reconciler.io/dies style die to mutate the resource
inMemoryGateway.
StatusDie(func(d *diestreamingv1alpha1.InMemoryGatewayStatusDie) {
d.ObservedGeneration(1)
d.ConditionsDie(
// the condition will be unknown since the child resource
// was just created and hasn't been reconciled by its
// controller yet
inMemoryGatewayConditionGatewayReady.Unknown(),
inMemoryGatewayConditionReady.Unknown(),
)
}),
},
},
...
}}
rts.Run(t, scheme, func(t *testing.T, rtc *rtesting.ReconcilerTestCase, c reconcilers.Config) reconcile.Reconciler {
return streaming.InMemoryGatewayReconciler(c, testSystemNamespace)
})For more complex reconcilers, the number of moving parts can make it difficult to fully cover all aspects of the reonciler and handle corner cases and sources of error. The SubReconcilerTestCase enables testing a single sub reconciler in isolation from the resource. While very similar to ReconcilerTestCase, these are the differences:
Requestis replaced withResourcesince the resource is not lookedup, but handed to the reconciler.ExpectResourceis the mutated value of the resource after the reconciler runs.GivenStashedValuesis a map of stashed value to seed,ExpectStashedValuesare individually compared with the actual stashed value after the reconciler runs.ExpectStatusUpdatesis not available
There are two ways to compose a SubReconcilerTestCase either as an unordered set using SubReconcilerTests, or an order list using SubReconcilerTestSuite. When using SubReconcilerTests the key for each test case is used as the name for that test case.
Example:
Like with the tracking example, the processor reconciler in projectriff also looks up images from a ConfigMap. The sub reconciler under test is responsible for tracking the ConfigMap, loading and stashing its contents. Sub reconciler tests make it trivial to test this behavior in isolation, including error conditions.
processor := ...
processorImagesConfigMap := ...
rts := rtesting.SubReconcilerTests[*streamingv1alpha1.Processor]{
"missing images configmap": {
Resource: processor,
ExpectTracks: []rtesting.TrackRequest{
rtesting.NewTrackRequest(processorImagesConfigMap, processor, scheme),
},
ShouldErr: true,
},
"stash processor image": {
Resource: processor,
GivenObjects: []client.Object{
processorImagesConfigMap,
},
ExpectTracks: []rtesting.TrackRequest{
rtesting.NewTrackRequest(processorImagesConfigMap, processor, scheme),
},
ExpectStashedValues: map[reconcilers.StashKey]interface{}{
streaming.ProcessorImagesStashKey: processorImagesConfigMap.Create().Data,
},
},
}
rts.Run(t, scheme, func(t *testing.T, rtc *rtesting.SubReconcilerTestCase[*streamingv1alpha1.Processor], c reconcilers.Config) reconcilers.SubReconciler[*streamingv1alpha1.Processor] {
return streaming.ProcessorSyncProcessorImages(c, testSystemNamespace)
})AdmissionWebhookTestCase runs the full webhook handler via the controller runtime webhook handler's Handle method. There are two ways to compose a AdmissionWebhookTestCase either as an unordered set using AdmissionWebhookTests, or an order list using AdmissionWebhookTestSuite. When using AdmissionWebhookTestSuite the key for each test case is used as the name for that test case.
Example
Service bindings project into workloads with a controller and a mutating webhook. The admission request for the workload resource along with a given ServiceBinding resource is projected mutating the resource, which is treated as a patch in the admission response.
workload := dieappsv1.DeploymentBlank.
...
serviceBinding := dieservicebindingv1beta1.ServiceBindingBlank.
...
request := dieadmissionv1.AdmissionRequestBlank.
KindDie(func(d *diemetav1.GroupVersionKindDie) {
d.Group("apps")
d.Version("v1")
d.Kind("Deployment")
}).
ResourceDie(func(d *diemetav1.GroupVersionResourceDie) {
d.Group("apps")
d.Version("v1")
d.Resource("deployments")
}).
UID(requestUID).
Operation(admissionv1.Create).
Namespace(namespace).
Name(name)
response := dieadmissionv1.AdmissionResponseBlank.
Allowed(true)
wts := rtesting.AdmissionWebhookTests{
"project binding": {
GivenObjects: []client.Object{
serviceBinding,
},
Request: &admission.Request{
AdmissionRequest: request.
Object(workload.DieReleaseRawExtension()).
DieRelease(),
},
ExpectedResponse: admission.Response{
AdmissionResponse: response.DieRelease(),
Patches: []jsonpatch.Operation{
{
Operation: "add",
Path: "/spec/template/spec/containers/0/env",
Value: []interface{}{
map[string]interface{}{
"name": "SERVICE_BINDING_ROOT",
"value": "/bindings",
},
},
},
...
},
},
},
}
wts.Run(t, scheme, func(t *testing.T, wtc *rtesting.AdmissionWebhookTestCase, c reconcilers.Config) *admission.Webhook {
return controllers.AdmissionProjectorWebhook(c).Build()
})The ExpectConfig is a testing object that can create a Config with given test state that will observe the reconciler's behavior against the config and can assert that the observed behavior matches the expected behavior. When used with the AdditionalConfigs field of ReconcilerTestCase and SubReconcilerTestCase, the corresponding configs can be obtained with RetrieveAdditionalConfigs. Use of RetrieveAdditionalConfigs should be limited to a reconciler that is dedicated to work with multiple configs like WithConfig; reconcilers nested under WithConfig should interact with the default config.
The Config is a single object that contains the common remote APIs needed by a reconciler. The config object includes:
Clientas the primary interaction with the Kubernetes API Server. Gets and Lists are read from informers when available.APIReaderread-only Kubernetes API Server client that bypasses informers.Recorderrecord Kubernetes events for a resource.Trackertrack relationships between resource, and later lookup resources tracking a specific resource.
Root reconcilers like ResourceReconciler and AdmissionWebhookAdapter accept a Config to use that is then passed to SubReconciler via the context, and retrieved using RetrieveConfigOrDie. The active config may be modified at runtime using WithConfig.
To setup a Config for a test and make assertions that the expected behavior matches the observed behavior, use ExpectConfig.
The stash allows passing arbitrary state between sub reconcilers within the scope of a single reconciler request. Values are stored on the context by StashValue and accessed via RetrieveValue.
A Stasher provides a convenient way to interact with typed values. Create a NewStasher using the type of the value being stashed and a unique stash key. All operations through a stasher, including retrieval are type safe with options for handling missing values on retrieval.
For testing, given stashed values can be defined in a SubReconcilerTests with GivenStashedValues. Newly stashed or mutated values expectations are defined with ExpectStashedValues. An optional, custom function for asserting stashed values can be provided via VerifyStashedValue.
Example:
var exampleStasher = reconcilers.NewStasher[Example]("example")
func StashExampleSubReconciler(c reconcilers.Config) reconcilers.SubReconciler[*examplev1.MyExample] {
return &reconcilers.SyncReconciler[*examplev1.MyExample]{
Name: "StashExample",
Sync: func(ctx context.Context, resource *examplev1.MyExample) error {
value := Example{} // something we want to expose to a sub reconciler later in this chain
exampleStasher.Store(ctx, value)
return nil
},
}
}
func StashExampleSubReconciler(c reconcilers.Config) reconcilers.SubReconciler[*examplev1.MyExample] {
return &reconcilers.SyncReconciler[*examplev1.MyExample]{
Name: "StashExample",
Sync: func(ctx context.Context, resource *examplev1.MyExample) error {
value, err := exampleStasher.RetrieveOrError(ctx)
if err != nil {
return nil, fmt.Errorf("%w for key %q", err, exampleStasher.Key())
}
... // do something with the value
},
}
}The Tracker provides a means for one resource to watch another resource for mutations, triggering the reconciliation of the resource defining the reference.
Resources can either be tracked by name or with a label selector using TrackReference.
It's common to work with a resource that is also tracked. The Config.TrackAndGet method uses the same signature as client.Get, but additionally tracks the resource. Likewise, the Config.TrackAndList method uses the same signature as client.List, but additionally tracks resources matching the query.
In the Setup method, a watch is created that will notify the handler every time a resource of that kind is mutated. The EnqueueTracked helper returns a list of resources that are tracking the given resource, those resources are enqueued for the reconciler.
The tracker will automatically expire a track request if not periodically renewed. By default, the TTL is 2x the resync internal. This ensures all tracked resources will naturally have the tracking relationship refreshed as part of the normal reconciliation resource. There is no need to manually untrack a resource.
The tracked resource and its tracker are managed by reference and do not need concrete resources. Tracking of a resource can start before it exists. Once the tracked resource is created, the tracker resource will be enqueued for reconciliation.
Example:
The stream gateways in projectriff fetch the image references they use to run from a ConfigMap. When the ConfigMap changes, we want to detect and rollout the updated images.
func InMemoryGatewaySyncConfigReconciler(c reconcilers.Config, namespace string) reconcilers.SubReconciler[*streamingv1alpha1.InMemoryGateway] {
return &reconcilers.SyncReconciler[*streamingv1alpha1.InMemoryGateway]{
Name: "SyncConfig",
Sync: func(ctx context.Context, resource *streamingv1alpha1.InMemoryGateway) error {
log := logr.FromContextOrDiscard(ctx)
c := reconciler.RetrieveConfig(ctx)
var config corev1.ConfigMap
key := types.NamespacedName{Namespace: namespace, Name: inmemoryGatewayImages}
// track config for new images, get the configmap
if err := c.TrackAndGet(ctx, key, &config); err != nil {
return err
}
// consume the configmap
resource.Status.GatewayImage = config.Data[gatewayImageKey]
resource.Status.ProvisionerImage = config.Data[provisionerImageKey]
return nil
},
Setup: func(ctx context.Context, mgr reconcilers.Manager, bldr *reconcilers.Builder) error {
// enqueue the tracking resource for reconciliation from changes to
// tracked ConfigMaps. Internally `EnqueueTracked` handels informer
// events to watch for changes of the target resource. When the
// informer emits an event, the tracking resources are looked up
// from the tracker and enqueded for reconciliation.
bldr.Watches(&corev1.ConfigMap{}, reconcilers.EnqueueTracked(ctx))
return nil
},
}
}The apis package provides means for conveniently managing a custom resource's .status.
A resource's status subresource is expected to meet the following contract:
type MyStatus struct {
ObservedGeneration int64 `json:"observedGeneration,omitempty"`
Conditions []metav1.Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type"`
}Example:
Embed api.Status into your resource's status and add more fields:
type MyResourceStatus struct {
apis.Status `json:",inline"`
UsefulMessage string `json:"usefulMessage,omitempty"`
}
type MyResource struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec MyResourceSpec `json:"spec"`
// +optional
Status MyResourceStatus `json:"status"`
}Finalizers allow a reconciler to clean up state for a resource that has been deleted by a client, and not yet fully removed. Terminating resources have .metadata.deletionTimestamp set. Resources with finalizers will stay in this terminating state until all finalizers are cleared from the resource. While using the Kubernetes garbage collector is recommended when possible, finalizer are useful for cases when state exists outside of the same cluster, scope, and namespace of the reconciled resource that needs to be cleaned up when no longer used.
Deleting a resource that uses finalizers requires the controller to be running.
Note: WithFinalizer can be used in lieu of, or in conjunction with, ChildReconciler#Finalizer. The distinction is the scope within the reconciler tree where a finalizer is applied. While a reconciler can define as many finalizers on the resource as it desires, in practice, it's best to minimize the number of finalizers as setting and clearing each finalizer makes a request to the API Server.
A single WithFinalizer will always add a finalizer to the reconciled resource. It can then compose multiple ChildReconcilers, as well as other reconcilers that do not natively support managing finalizers (e.g. SyncReconciler). On the other hand, the ChildReconciler will only set the finalizer when it is required potentially reducing the number of finalizers, but only covers that exact sub-reconciler. It's important the external state that needs to be cleaned up be covered by a finalizer, it does not matter which finalizer is used.
The AddFinalizer and ClearFinalizer functions patch the reconciled resource to update its finalizers. These methods work with CastResource resources and use the same client the ResourceReconciler used to originally load the reconciled resource. They can be called inside SubReconcilers that may use a different client.
When an update is required, only the .metadata.finalizers field is patched. The reconciled resource's .metadata.resourceVersion is used as an optimistic concurrency lock, and is updated with the value returned from the server. Any error from the server will cause the resource reconciliation to err. When testing with SubReconcilerTests, the resource version of the resource defaults to "999", the patch bytes include the resource version and the response increments the reonciled resource's version. For a resource with the default version that patches a finalizer, the expected reconciled resource will have a resource version of "1000".
A minimal test case for a sub reconciler that adds a finalizer may look like:
...
{
Name: "add 'test.finalizer' finalizer",
Resource: resourceDie.DieReleasePtr(),
ExpectEvents: []rtesting.Event{
rtesting.NewEvent(resourceDie, scheme, corev1.EventTypeNormal, "FinalizerPatched",
`Patched finalizer %q`, "test.finalizer"),
},
ExpectResource: resourceDie.
MetadataDie(func(d *diemetav1.ObjectMetaDie) {
d.Finalizers("test.finalizer")
d.ResourceVersion("1000")
}),
ExpectPatches: []rtesting.PatchRef{
{
Group: "testing.reconciler.runtime",
Kind: "TestResource",
Namespace: resourceDie.GetNamespace(),
Name: resourceDie.GetName(),
PatchType: types.MergePatchType,
Patch: []byte(`{"metadata":{"finalizers":["test.finalizer"],"resourceVersion":"999"}}`),
},
},
},
...The ObjectManager is an interface providing a means to manage a single resource by synchronizing the current and desired state. The resource will be created if it does not exist, deleted if no longer desired and updated when semantically different. The same resource manager should be reused to manage multiple resources and must be reused when managing the same resource over time in order to take full effect. This utility is used by the ChildReconciler, ChildSetReconciler and AggregateReconciler.
The interface is designed to allow for multiple synchronization strategies to be used.
The Manage(ctx context.Context, resource, actual, desired client.Object) (client.Object, error) method take three objects and returns another object:
resourceis the reconciled resource, events, tracks and finalizer are against this object. May be an object of any underlaying type.actualthe resource that exists on the API Server. Must be compatible with theType.desiredthe resource that should exist on the API Server after this call. Must be compatible with theType.- the returned object is the value as persisted by the API Server.
Use a provided ObjectManager or define a custom strategy to change specific behavior or employ entirely new approaches to sync state to the API Server.
The UpdatingObjectManager (previously ResourceManager) uses the client.Client#{Create, Update, Delete} methods to synchronize state to the API Server.
Internally, a mutations made to the resource at admission time (like defaults applied by a mutating webhook) are captured and reapplied to the desired state before checking if an update is needed. This reduces requests that are functionally a no-op but create churn on the API Server. The mutation cache is defensive and fails open to make an API request.
If configured, a finalizer can be managed on the resource which will be added before create/udpate and removed after sucessful delete.
If requested, the managed resource will be tracked for the resource.
Reconcilers that capture timestamps can be notoriously difficult to test, as the output will be different for every execution. While we don't have a time machine, reconciler.io runtime provides an alterate API to fetch the current time within a reconciler. rtime.RetrieveTime(context.Context) can be used within a reconciler to get the time.Time when the reconciler request started processing. The value returned is guaranteed to remain stable for the lifespan of the reconcile request. Calls to time.Now will continue to return an up to date timestamp.
Reconciler tests can seed this timestamp by defining the Now field on the test case. The reconciler will be run with the desired time instead of "now". The timestamp set on the test case can also be used in the expectations to pin values that would otherwise float.
Known breaking changes are captured in the release notes, it is strongly recomened to review the release notes before upgrading to a new version of reconciler.io. When possible, breaking changes are first marked as deprecations before full removal in a later release. Patch releases will be issued to fix significant bugs and unintentional breaking changes.
We strive to release reconciler.io runtime against the latest Kubernetes and controller-runtime releases. Upstream breaking changes in either dependency may also force changes in runtime without a deprecation period.
reconciler.io runtime is rapidly evolving. While we strive for API compatability between releases, functionality that is better handled using a different API may be removed. Release version numbers follow semver.
Backwards support may be removed in a future release, users are encouraged to migrate.
- status
InitializeConditions()is deprecated in favor ofInitializeConditions(context.Context). ConditionSet#Manageis deprecated in favor ofConditionSet#ManageWithContext.HaltSubReconcilersis deprecated in favor ofErrHaltSubReconcilers.
The reconciler.io projects follow the Contributor Covenant Code of Conduct. In short, be kind and treat others with respect.
General discussion and questions about the project can occur either on the Kubernetes Slack #reconcilerio channel, or in the project's GitHub discussions. Use the channel you find most comfortable.
The reconciler.io runtime project team welcomes contributions from the community. A contributor license agreement (CLA) is not required. You own full rights to your contribution and agree to license the work to the community under the Apache License v2.0, via a Developer Certificate of Origin (DCO). For more detailed information, refer to CONTRIBUTING.md.
reconciler-runtime was conceived at VMware within projectriff/system and implemented initially by Scott Andrews, Glyn Normington and the riff community at large, drawing inspiration from Kubebuilder and Knative reconcilers.
All commits before 7cc9b34 are copyright VMware and consumed under the Apache License v2.0, unless otherwise marked. After this point all commits are copyright of the respective author and licensed to the community under the Apache License v2.0, via a Developer Certificate of Origin (DCO).
Apache License v2.0: see LICENSE for details.