* Add PVC mutating webhook using StorageProfiles
The webhook mutates the PVC Spec based on the available StorageProfiles,
so for example you can create PVC without accessModes and it will be
auto-completed.
To use this feature, enable the `WebhookPvcRendering` feature gate.
For any PVC you want to use StorageProfile, label it with:
cdi.kubevirt.io/useStorageProfile: "true"
If you want to use volumeMode preferred by CDI according to
StorageProfiles, set it to FromStorageProfile. Otherwise if not
explicitly set to Block, it will be Filesystem by k8s default.
E.g.:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-test
labels:
cdi.kubevirt.io/useStorageProfile: "true"
spec:
storageClassName: rook-ceph-block
volumeMode: FromStorageProfile
resources:
requests:
storage: 1Mi
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Move webhook create/delete to callback
plus some CR fixes and cleanups
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Move webhook CR creation to sit with callbacks
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Update existing webhook if modified
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Eliminate unnecessary CR update
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Bump k8s/OpenShift/ctrl-runtime/lifecycle-sdk & make deps-update
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Operator: adapt for dependency bump
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Controller: adapt watch calls for dependency bump
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Controller: adapt to ctrl-runtime's cache API changes
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Operator: fix unit tests by deleting resources properly in fake client
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Controller: fix unit tests by deleting resources properly in fake client
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Controller: adapt to fake client honoring status subresource
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Fix codegen script & make generate
There are some issues in the new script, so we
will still use the deprecated one.
More context in f4d1a5431b
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Functests: Adapt to NamespacedName now implementing MarshalLog
ns/name -> {"name":"name","namespace":"ns"}
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Functests & API server: address deprecation of wait.PollImmediate
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
---------
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* apiserver support for cdi.kubevirt.io/allowClaimAdoption
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* controller support for cdi.kubevirt.io/allowClaimAdoption on existing PVC
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* if cdi.kubevirt.io/allowClaimAdoption specified on DataVolume do not apply on PVC until DV is succeeded
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add DataVolumeClaimAdoption featuregate and integrate with apiserver and controller
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* functional tests for claim adoption
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Fix func test failure and address some review comments
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* rename pvcRequiresNoWork to pvcRequiresWork
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* allow unbound PVC to be adopted
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
We slap on the clone-without-source index key regardless of source.PVC/source.Snapshot so
it was possible for a DV with PVC source to get queued for the snapshot controller
(and hence the nil ptr).
The test addition demonstrates a real use case where this would happen.
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
This commit ups the cpu request for for all our installed compopents
(cdi-deployment, cdi-apiserver, cdi-uploadproxy, cdi-operator)
for 10m (1% of a core) to 100m (10% of a core).
The main driver of this is BZ: 2216038.
Without this change, it is pretty easy to create a large number of
concurrent clone operations and get token timeout errors.
Upping resource requests and concurrency addresses the issue
in a very direct way.
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove CSI clone
bye bye
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* no more smart clone
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* PVC clone same namespace
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* cross namespace pvc clone
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* various fixes to get some functional tests to work
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* delete smart clone controller again
somehow reappeared after rebase
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* mostly pvc clone functional test fixes
make sure size detect pod only runs on kubevirt content type
clone populator was skipping last round op applying pvc' annotations
various func test fixes
review comments
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* more various test fixes
host clone phase should (implicitly) wait for clone source pod to exit
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove "smart" clone from snapshot
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* DataVolume clone from snapshot uses populator
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* improve clone populator/datavolume coordination on "running" condition
For host clone, not much changes, values still comming from annotations on host clone PVC
For smart/csi clone the DataVolume will be "running" if not in pending or error phase
Will have the same values for terminal "completed" state regardless of clone type
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* unit tests for pvc/snapshot clone controllers
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove skipped test added in https://github.com/kubevirt/containerized-data-importer/pull/2759
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* attempt address AfterSuite and generate-verify failures
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* handle snapshot clone with no target size specified
also add more validation to some snapshot clone tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove Patch calls
Using the controller runtime Patch API with controller runtime cached client seems to be a pretty bad fit
At least given the way the CR API is designed where an old object is compared to new.
I like patch in theory though and will revisit
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Clone populator should plan and execute even if PVC is bound
It was possible to miss "preallocation applied" annotation otherwise
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add long term token to datavolume
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Rename ProgressReporter to StatusReporter
Should have been done back when annotations were addded to "progress"
Also, if pvc is bound do not call phase Reconcile functions only Status
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* move cleanup out of dv deletion
It seemed off to call cleanup in the prepare function
just because we don't call cleanup unless the dv is deleting.
Instead we check in the clenup function itself if it should be
done: in this 2 specific cases in case of deletion and in case
the dv succeeded.
The cleanup will be used in future commit also for population cleanup
which we also want to happen not only on deletion.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Use populator if csi storage class exists
Add new datavolume phase PendingPopulation to
indicate wffc when using populators, this new
phase will be used in kubevirt in order to know
that there is no need for dummy pod to pass wffc phase
and that the population will occur once creating the vm.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Update population targetPVC with pvc prime annotations
The annotations will be used to update dv that uses the
populators.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Adjust UT with new behavior
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* updates after review
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Fix import populator report progress
The import pod should be taken from pvcprime
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Prevent requeue upload dv when failing to find progress report pod
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Remove size inflation in populators
The populators are handling existing PVCs.
The PVC already has a defined requested size,
inflating the PVC' with fsoverhead will only be
on the PVC' spec and will not reflect on the target
PVC, this seems undesired.
Instead if the populators is using by PVC that the
datavolume controller created the inflation will happen
there if needed.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Adjust functional tests to handle dvs using populators
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Fix clone test
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* add shouldUpdateProgress variable to know if need to update progress
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Change update of annotation from denied list to allowed list
Instead if checking if the annotation on pvcPrime is not desired
go over desired list and if the annotation exists add it.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* fix removing annotations from pv when rebinding
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* More fixes and UT
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* a bit more updates and UTs
Signed-off-by: Shelly Kagan <skagan@redhat.com>
---------
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* StorageProfile API for declaring format of resulting cron disk images
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Integrate recommended format in dataimportcron controller
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Take snapclass existence into consideration when populating cloneStrategy and sourceFormat
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
---------
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* touch up zero restoresize snapshot
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* clone populator
only supports PVC source now
snapshot coming soon
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* more unit tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* unit test for clone populator
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* func tests for clone populator
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* move clone populator cleanup function to planner
other review comments
verifier pod should bount readonly
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add readonly flag to test executor pods
synchronize get hash calls
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* increase linter timeout
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* better/explicit readonly support for test pods
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* check pv for driver info before looking up storageclass as it may not exist
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* addressed review comments
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* chooseStrategy shoud generate more events
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Create PVC if possible even if SC is missing
When PVC is created with storageClassName and the SC is not found,
k8s looks for PV with the storageClassName for satisfying this claim.
In this case k8s supports also a blank (“”, not the nil default one)
storageClassName. To support this behavior we added:
-DV controller support for this flow (for both “” and non-existing SC)
-Condition update and event when StorageSpec PVC rendering errors and
PVC is not created (e.g. missing both AccessModes and SC/PV)
-PVC is created even if a satisfying SC/PV doesn't exist if pvc/storage
AccessModes is set (otherwise k8s PVC validation fails). PVC/DV phase
will be Pending until a satisfying SC/PV is found
-PV watch to reconcile DVs waiting for the PV storageClassName
-PV storageClassName indexer, so we can list the relevant PVs
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Create populators package to be used for all populators
This commit introduces the basic reconciler for
populators with common function that can be used
by the different populators.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* unite getcontenttype func across code
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add VolumeImportSource CRD for import populator
This commit adds the VolumeImportSource CRD into CDI.
CRs created from this CRD will be referenced in the dataSourceRef field to populate PVCs with the import populator.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Refactor common populator code to be shared among all populators
This commit introduces and modifies several functions so we can reuse common code between all populators.
Other than having a common reconcile function, a new populatorController interface has been introduced so we are able to call populator-specific methods from the populator-base reconciler.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Create Import Populator
The import populator is a controller that handles the import of data in PVCs without the need of DataVolumes while still taking advantage of the import-controller flow.
This controller creates an additional PVC' with import annotations. After the import process succeeds, the controller rebinds the PV to the original target PVc and deletes the PVC prime.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add functional tests to cover the import populator flow
This commit updates the import tests to cover the new import populator flow.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add unit tests for import populator
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Minor fixes and enhancements in import/common populator code
* Modify indexes and other related code to support namespaced dataSourceRefs. Cross-namespace population is still not supported as it depends on alpha feature gates.
* Add functional test to cover static binding.
* Fix selected node annotation bug in scratch space PVCs
* Fix linter alerts
Signed-off-by: Alvaro Romero <alromero@redhat.com>
---------
Signed-off-by: Shelly Kagan <skagan@redhat.com>
Signed-off-by: Alvaro Romero <alromero@redhat.com>
Co-authored-by: Shelly Kagan <skagan@redhat.com>
* Add support for imagePullSecrets in the CDI CR, to support pulling
images from repositories that require secrets.
The imagePullSecrets is propagated to the following components: cdi-apiserver,
cdi-deployment, and cdi-uploadproxy. The definition of imagePullSecrets in
cdi-operator must be done manually.
Signed-off-by: Gleb Aronsky <gleb.aronsky@windriver.com>
* Modifying code to incorporate review comments.
Signed-off-by: Gleb Aronsky <gleb.aronsky@windriver.com>
---------
Signed-off-by: Gleb Aronsky <gleb.aronsky@windriver.com>
Co-authored-by: Gleb Aronsky <gleb.aronsky@windriver.com>
* Fix hostpath CSI being skipped as "Not HPP"
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Fall back to host assisted if immediate bind requested
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
---------
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
There should be no state shared between sync() and updateStatus().
updateStatus() should stand on it's own, and come to it's own
conclusions based on what it observes. It is okay if it is "late to the
party" and does not observe the latest changes in sync(). It will
eventually converge. This is what kubevirt does.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* function should return dataVolumeSyncResult, take *dataVolumeSyncResult as a parameter
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* checkStaticVolume implemetation for import DataVolume
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* upload support for checkStaticVolume
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* checkStaticVolume for clone datavolumes
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* checkStaticVolume for snapshot clone
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* checkStaticVolume for external populator source
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* tignten up static volume check
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* expand functional test to compare creation timestamps
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* updates from code review mostly add md5 verification to test and refacto common index creation
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* webhook changes, allow clone source DataVolumes (with special annotations)
even if source does not exist or user has no permission
BUT no token is added so this is really just for the static/prepopulate cases
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
DV with sourceRef DataSource pointing to non-existing PVC, is to be
reconciled when the PVC is created and populated and the DataSource
is Ready.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* API for VolumeSnapshot clone source
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Refactor doCrossNamespaceClone to get rid of some source PVC assumptions
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Refactor isCrossNamespaceClone to get rid of some some source PVC assumptions
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Clone from VolumeSnapshot source controller
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Func tests for cloning from volumesnapshot source
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Extend cross ns clone token mechanism for VolumeSnapshot cloning
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Validate source volumesnapshot on create
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
---------
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
We are introducing cloning from snapshot soon as a new controller,
as part of those efforts, it was identified to be beneficial to have this split.
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
