* Default virt storage class
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add alert for multiple default virt storage classes
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Refactor content type funcs to not return strings
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
---------
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
We slap on the clone-without-source index key regardless of source.PVC/source.Snapshot so
it was possible for a DV with PVC source to get queued for the snapshot controller
(and hence the nil ptr).
The test addition demonstrates a real use case where this would happen.
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
This annotation causes CDI transfer pods (importer, uploader, cloner) to be retained after a successful or failed completion.
This makes debugging and testing easier, as users can get the pod state and logs after completion.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Ensure priority class is assigned to pod populating pvc prime
Priority class was not being passed to the pvc prime from the PVC
and thus was not being added to the importer pod populating the
pvc prime. There is a list of allowed annotations that can be passed
and the priority class annotation was not in it. This commit adds
the annotation to the allowed list.
Cleaned up unneed log argument to a function related to passing the
annotations.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Restore the logger as it was logging the pvc name as well
as the log message. Modified the name to make it clearer
this is the case.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Have upload also use the populator, can't do clone
because the pod disappears too quickly without retain
Signed-off-by: Alexander Wels <awels@redhat.com>
* Test for priority class in upload as well as fix typo in cloning test.
Signed-off-by: Alexander Wels <awels@redhat.com>
---------
Signed-off-by: Alexander Wels <awels@redhat.com>
Currently there is a repeating reconcile error due to exponential
backoff: "DataVolume.storage spec is missing accessMode and no
storageClass to choose profile", however the error is not needed as we
already have SC watch for this case, e.g. when the DV is using the
default SC but no default SC is configured yet.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
This commit ups the cpu request for for all our installed compopents
(cdi-deployment, cdi-apiserver, cdi-uploadproxy, cdi-operator)
for 10m (1% of a core) to 100m (10% of a core).
The main driver of this is BZ: 2216038.
Without this change, it is pretty easy to create a large number of
concurrent clone operations and get token timeout errors.
Upping resource requests and concurrency addresses the issue
in a very direct way.
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* dataVolume: Add default instance type labels from source
f229aeb started to pass default instance type labels from
DataImportCrons to DataVolumes and DataVolumes to any associated
destination DataSources or PVCs. As documented in issue #2782 this does
not however pass these labels from the initial source of a DataVolume to
either the DataVolume or the destination DataSources or PVCs
This change corrects this by updating DataVolumes when reconciled,
adding any labels found on PVC or DataSource sources. These labels will
then be passed on to the destination PVC or DataSources by the existing
functionality highlighted above.
Note that if any default instance type labels already exist on the
DataVolume then the process is skipped as it is assumed these are
provided either directly by the user or via a DataImportCron.
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
* refactor: Use DefaultInstanceTypeLabels more often
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
---------
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
* Update VolumeImportSource API to support multi-stage imports
This commit modifies the VolumeImportSource API to support multi-stage imports, adding the following fields:
- Checkpoints, to represent the stages of a multistage import
- TargetClaim, the name of the specific PVC to be imported
- FinalCheckpoint, to indicate that the current Checkpoint is the final one
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Support multi-stage imports in import-populator
This commit updates the import populator to support multi-stage imports. The API and functionality remains the same as with DataVolumes, with the only difference that the used VolumeImportSource will now require a populated "TargetClaim" field that reffers to the specific PVC to be populated.
The DataVolume controller is also updated to allow using the populator flow with VDDK and ImageIO sources.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add unit tests for multistage import support in populators
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add functional tests to test multistage import populator flow
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Fix multi-stage import logic in import-populator and add remaining tests
This commit fixes several bugs in the import-populator logic for multi-stage imports.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
---------
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Allow ImmediateBind annotation when using populators
In case of using PVC with populators if the PVC has this
annotation we prevent from waiting for it to be schedueled
and we proceed with the process.
When using datavolumes with populators in case the dv has the annotation
it will be passed to the PVC. we prevent from being in pendingPopulation
in case the created pvc has the annotaion.
Plus when having honorWaitForFirstConsumer feature gate disabled we will
put on the target PVC the immediateBind annotation.
Now we allow to use populators when having the annotation the the
feature gate disabled.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add functional tests to population using PVCs
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Support immediate binding with clone datavolume
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Pass allowed annotations from target pvc to pvc prime
This annotations are used for the import/upload/clone
pods to define netork configurations.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
---------
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* remove CSI clone
bye bye
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* no more smart clone
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* PVC clone same namespace
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* cross namespace pvc clone
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* various fixes to get some functional tests to work
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* delete smart clone controller again
somehow reappeared after rebase
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* mostly pvc clone functional test fixes
make sure size detect pod only runs on kubevirt content type
clone populator was skipping last round op applying pvc' annotations
various func test fixes
review comments
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* more various test fixes
host clone phase should (implicitly) wait for clone source pod to exit
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove "smart" clone from snapshot
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* DataVolume clone from snapshot uses populator
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* improve clone populator/datavolume coordination on "running" condition
For host clone, not much changes, values still comming from annotations on host clone PVC
For smart/csi clone the DataVolume will be "running" if not in pending or error phase
Will have the same values for terminal "completed" state regardless of clone type
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* unit tests for pvc/snapshot clone controllers
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove skipped test added in https://github.com/kubevirt/containerized-data-importer/pull/2759
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* attempt address AfterSuite and generate-verify failures
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* handle snapshot clone with no target size specified
also add more validation to some snapshot clone tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove Patch calls
Using the controller runtime Patch API with controller runtime cached client seems to be a pretty bad fit
At least given the way the CR API is designed where an old object is compared to new.
I like patch in theory though and will revisit
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Clone populator should plan and execute even if PVC is bound
It was possible to miss "preallocation applied" annotation otherwise
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add long term token to datavolume
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Rename ProgressReporter to StatusReporter
Should have been done back when annotations were addded to "progress"
Also, if pvc is bound do not call phase Reconcile functions only Status
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
We were hitting a panic because we're passing the original DV object to renderPvcSpec (source is nil)
instead of the mutated DV which has sourceRef converted (source not nil)
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* move cleanup out of dv deletion
It seemed off to call cleanup in the prepare function
just because we don't call cleanup unless the dv is deleting.
Instead we check in the clenup function itself if it should be
done: in this 2 specific cases in case of deletion and in case
the dv succeeded.
The cleanup will be used in future commit also for population cleanup
which we also want to happen not only on deletion.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Use populator if csi storage class exists
Add new datavolume phase PendingPopulation to
indicate wffc when using populators, this new
phase will be used in kubevirt in order to know
that there is no need for dummy pod to pass wffc phase
and that the population will occur once creating the vm.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Update population targetPVC with pvc prime annotations
The annotations will be used to update dv that uses the
populators.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Adjust UT with new behavior
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* updates after review
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Fix import populator report progress
The import pod should be taken from pvcprime
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Prevent requeue upload dv when failing to find progress report pod
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Remove size inflation in populators
The populators are handling existing PVCs.
The PVC already has a defined requested size,
inflating the PVC' with fsoverhead will only be
on the PVC' spec and will not reflect on the target
PVC, this seems undesired.
Instead if the populators is using by PVC that the
datavolume controller created the inflation will happen
there if needed.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Adjust functional tests to handle dvs using populators
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Fix clone test
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* add shouldUpdateProgress variable to know if need to update progress
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Change update of annotation from denied list to allowed list
Instead if checking if the annotation on pvcPrime is not desired
go over desired list and if the annotation exists add it.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* fix removing annotations from pv when rebinding
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* More fixes and UT
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* a bit more updates and UTs
Signed-off-by: Shelly Kagan <skagan@redhat.com>
---------
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* touch up zero restoresize snapshot
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* clone populator
only supports PVC source now
snapshot coming soon
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* more unit tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* unit test for clone populator
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* func tests for clone populator
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* move clone populator cleanup function to planner
other review comments
verifier pod should bount readonly
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add readonly flag to test executor pods
synchronize get hash calls
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* increase linter timeout
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* better/explicit readonly support for test pods
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* check pv for driver info before looking up storageclass as it may not exist
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* addressed review comments
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* chooseStrategy shoud generate more events
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Create PVC if possible even if SC is missing
When PVC is created with storageClassName and the SC is not found,
k8s looks for PV with the storageClassName for satisfying this claim.
In this case k8s supports also a blank (“”, not the nil default one)
storageClassName. To support this behavior we added:
-DV controller support for this flow (for both “” and non-existing SC)
-Condition update and event when StorageSpec PVC rendering errors and
PVC is not created (e.g. missing both AccessModes and SC/PV)
-PVC is created even if a satisfying SC/PV doesn't exist if pvc/storage
AccessModes is set (otherwise k8s PVC validation fails). PVC/DV phase
will be Pending until a satisfying SC/PV is found
-PV watch to reconcile DVs waiting for the PV storageClassName
-PV storageClassName indexer, so we can list the relevant PVs
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Create populators package to be used for all populators
This commit introduces the basic reconciler for
populators with common function that can be used
by the different populators.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* unite getcontenttype func across code
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add VolumeImportSource CRD for import populator
This commit adds the VolumeImportSource CRD into CDI.
CRs created from this CRD will be referenced in the dataSourceRef field to populate PVCs with the import populator.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Refactor common populator code to be shared among all populators
This commit introduces and modifies several functions so we can reuse common code between all populators.
Other than having a common reconcile function, a new populatorController interface has been introduced so we are able to call populator-specific methods from the populator-base reconciler.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Create Import Populator
The import populator is a controller that handles the import of data in PVCs without the need of DataVolumes while still taking advantage of the import-controller flow.
This controller creates an additional PVC' with import annotations. After the import process succeeds, the controller rebinds the PV to the original target PVc and deletes the PVC prime.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add functional tests to cover the import populator flow
This commit updates the import tests to cover the new import populator flow.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add unit tests for import populator
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Minor fixes and enhancements in import/common populator code
* Modify indexes and other related code to support namespaced dataSourceRefs. Cross-namespace population is still not supported as it depends on alpha feature gates.
* Add functional test to cover static binding.
* Fix selected node annotation bug in scratch space PVCs
* Fix linter alerts
Signed-off-by: Alvaro Romero <alromero@redhat.com>
---------
Signed-off-by: Shelly Kagan <skagan@redhat.com>
Signed-off-by: Alvaro Romero <alromero@redhat.com>
Co-authored-by: Shelly Kagan <skagan@redhat.com>
* update k8s libs to 1.26.
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove some checks in log messages, they're redundant, and the format has changed
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* use 1.26 lib function `CheckVolumeModeMismatches` and `CheckAccessModes`
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Google Cloud Storage Importer
This is a Google Cloud Storage importer for CDI
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* Fix auto-generated swagger and openapi
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* GCS Importer General Fixes
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* Moving back gcs-secret.txt
Moving file back to imageDir to fix unit testing.
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
---------
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
Co-authored-by: Marcelo Parisi <marcelo@dev-box.corp.feitoza.com.br>
DV status can now be updated only by UpdateStatus() and no more by
Update(), so regular users cannot update status without special
permission. In addition, updating just the status would not update
the generation of the object, so we’ll have less “object has been
modified” errors in the reconcile loops.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
There should be no state shared between sync() and updateStatus().
updateStatus() should stand on it's own, and come to it's own
conclusions based on what it observes. It is okay if it is "late to the
party" and does not observe the latest changes in sync(). It will
eventually converge. This is what kubevirt does.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* function should return dataVolumeSyncResult, take *dataVolumeSyncResult as a parameter
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* checkStaticVolume implemetation for import DataVolume
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* upload support for checkStaticVolume
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* checkStaticVolume for clone datavolumes
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* checkStaticVolume for snapshot clone
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* checkStaticVolume for external populator source
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* tignten up static volume check
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* expand functional test to compare creation timestamps
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* updates from code review mostly add md5 verification to test and refacto common index creation
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* webhook changes, allow clone source DataVolumes (with special annotations)
even if source does not exist or user has no permission
BUT no token is added so this is really just for the static/prepopulate cases
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* API for VolumeSnapshot clone source
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Refactor doCrossNamespaceClone to get rid of some source PVC assumptions
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Refactor isCrossNamespaceClone to get rid of some some source PVC assumptions
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Clone from VolumeSnapshot source controller
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Func tests for cloning from volumesnapshot source
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Extend cross ns clone token mechanism for VolumeSnapshot cloning
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Validate source volumesnapshot on create
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
---------
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
Now that we update status in the end of a loop regardless of err,
it is important that we have the PVC in place for that.
Otherwise, if we fail at cleanup/prepare, syncRes.pvc is nil,
and we get false status updates like the following:
```bash
NAMESPACE NAME PHASE PROGRESS RESTARTS
default cloned-target-dv Succeeded N/A 23s
default cloned-target-dv Pending N/A 23s
default cloned-target-dv Pending N/A 23s
default cloned-target-dv Pending N/A 23s
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
golden-ns golden-snap-source Bound pvc-63ed1368-9f84-44ce-8b4b-7e9e3c4574f4 8Gi RWO rook-ceph-block 5m59s
(Target got deleted)
```
This results in a nasty bug where the target PVC simply disappears.
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
Right now if cleanup finishes succesfully we just keep going
on in the reconcile loop, even though deletion timestamp is set
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
We are introducing cloning from snapshot soon as a new controller,
as part of those efforts, it was identified to be beneficial to have this split.
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
