* Use new 1.23.6 builder
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Bump linter to 1.60.3 for go 1.23 support
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Disable linter failures over G115
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Fix lint issues related to error format formatting
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Address remaining lint failures
len is enough/sprintf not really used
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
---------
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
After several releases with GC disabled by default, we decided to
deprecate it, as unfortunately it violates fundamental principle of
Kubernetes. CR should not be auto-deleted when it completes its role
(Job with TTLSecondsAfterFinished is an exception), and once CR was
created we can assume it is there until explicitly deleted. In addition,
CR should keep idempotency, so the same CR manifest can be applied
multiple times, as long as it is a valid update (e.g. DataVolume
validation webhook does not allow updating the spec).
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
Currently, completing an incomplete StorageProfile does not reconcile
the relevant incomplete DVs waiting fro it, unlike the case of missing
StorageClass.
Even after updating the StorageProfile claimPropertySets to have
relevant AccessMode and VolumeMode, the DV is stuck with incomplete
AccessMode and VolumeMode until its exponential back-off reconcile, or
until it's manually updated (e.g. annotated) and reconciled.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove sourceref validation
The source can be changed and updated, we want the error to be
presented by an event and have the datavolume pending instead
of preventing the creation of the datavolume.
We are already handling the option of the source not existing yet
so lets also handle all the validation in the controller.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Move pvc clone specific functions to pvc-clone-controller
Now that the validation is no longer done in the webhook
no need for that code to be in the controller common util
file. Moved the UT accordingly.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Move function to new Describe node
Will add in future commits other tests to that section
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* pvc-clone-controller: make better event and update dv if validation fails
In order to make it more clear that the validation failed
we should add the reason of validation failure to the event
Added an update watch to the clone controller so if something changes
in the source the clone will reconcile. Hence no need to return an
error, the reconcile will be triggered if someting will change in the
source.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* snap-clone: always validate snapshot source
This commit adds missing validation due to the remove validation
from the dv webhook, but also fixes an existing missing validation
in cases where the clone was created before the source. In such case
the webhook would not validate the source since it didnt exist yet,
and then if the clone happened with populators then we would not
validate the source size at all.
We should validate the snapshot source before continuing with the clone
whether it is populator/legacy clones.
Moved all the validation to one function.
Updated the event to include the reason for the validation failure and
updated dv status accordingly.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Cloner_test: remove clone to small size test
The test checks the rejection of the webhook
which no longer exists then clone to appropriate
size dv which should succeed which is done in
any other clone test.
Im not adding a test to check the reconcile loop
since it is covered in the UT.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* tests/datasource: remove wait for pvc
Now since we added the validation also in the populators
path we are not creating the pvc until the source exists.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
---------
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* make deps-update
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* ReourceRequirements -> VolumeResourceRequirements
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix calls to controller.Watch()
controller-runtime changed the API!
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Fix errors with actual openshift/library-go lib
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* make all works now and everything compiles
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix "make update-codegen" because generate_groups.sh deprecated
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* run "make generate"
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix transfer unittest because of change to controller-runtime
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Fix progress metric registration and parsing
Use default metric registration. We shouldn't use the controller-runtime
registration as we have no controller here and it will not register the
metric correctly.
Fix the metric parsing to match its new name. Otherwise the DV progress
will not be updated until its 100%.
Regression introduced in #3254
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add kubevirt_cdi_import_progress_total metric
Use it in the importer instead of kubevirt_cdi_clone_progress_total and
fix metric parsing accordingly.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Move ProgressFromClaim to host-clone
Nobody else is using it.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add ProgressMetric interface
ProgressReader can now work with either import or clone progress metric.
FIXME: consider removing the direct Add/Get and use only via interface.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Refactor ProgressMetric interface
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Refactor progress parsing
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Refer metric names from the metrics package
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Move gosec into golangci-lint
Remove gosec target and scripts and use the golangci-lint linter
This ensures we stay up-to-date (so long as golangci-lint is up to date
too).
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G601: Disable for-loop variable aliassing warning (not relevant fro Go>=1.22)
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G101: Ignore warning about plain-text credentials
They are false positives
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G102: Don't listen to all interfaces
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G107: Ignore potentially tainted GET requests
They are all in test code
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G109: Avoid integer overflows after parsing strings
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G110: Potential DoS vulnerability via decompression bomb
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G112: Potential Slowloris Attack because ReadHeaderTimeout is not configured
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G114: Use of net/http serve function that has no support for setting timeouts
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G204: Subprocess launched with a potential tainted input or cmd arguments
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G305: File traversal when extracting zip/tar archive
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G306: Expect WriteFile permissions to be 0600 or less
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Bugfix: Misuse of file descriptor flags in file permission bits
os.WriteFile always uses O_WRONLY|O_CREATE|O_TRUNC, the third argument
is for the file's permission bits. This code is misleading, it will
truncate the file and not append to it. For that you'd need
os.Openfile(path, os.O_APPEND, 0600)
I also simplified the unnecessary []byte conversion.
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G401: Use of weak cryptographic primitive
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G402: Insecure TLS
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G404: Use of weak random number generator (math/rand instead of crypto/rand)
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G501: Blocklisted import crypto/md5: weak cryptographic primitive
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Run autoformatters
Unrelated to the PR but this way we keep everything formatted
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Remove references to gosec in the makefile
Gosec has been moved into Golangci-lint
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
---------
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Enable gofmt linter
From the docs:
> Gofmt checks whether code was gofmt-ed. By default this tool runs with
> -s option to check for code simplification.
https://golangci-lint.run/usage/linters/#gofmt
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Run gomft on the project
Ran this command after adding the gofmt linter:
golangci-lint run ./... --fix
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Enable whitespace linter
From the docs:
> Whitespace is a linter that checks for unnecessary newlines at the
> start and end of functions, if, for, etc.
https://golangci-lint.run/usage/linters/#whitespace
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Run whitespace on the project
Ran this command after adding the whitespace linter:
golangci-lint run ./... --fix
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Enable GCI linter
Per the docs:
> Gci controls Go package import order and makes it always deterministic.
https://golangci-lint.run/usage/linters/#gci
NOTE: I noticed that many files separate their imports in a particular
way, so I set the linter to enforce this standard.
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Run GCI on the project
Ran this command after adding the GCI linter:
golangci-lint run ./... --fix
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
---------
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Enable nakedret linter
> Checks that functions with naked returns are not longer than a maximum
> size (can be zero).
https://github.com/alexkohler/nakedret
Relevant from the GO style guide:
> Naked returns are okay if the function is a handful of lines. Once
> it’s a medium sized function, be explicit with your return values.
https://go.dev/wiki/CodeReviewComments#named-result-parameters
I think a naked return is never easier to read than a regular return,
so I set the linter to always complain about it. Disagreements are
welcome.
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Refactor functions with naked returns
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
---------
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Add misspell to list of linters
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Fix spelling errors
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
---------
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Suppress alerts to reduce noise of dependent ones
This is a follow-up to #2998 introducing the following changes to alert
rules:
- CDIDefaultStorageClassDegraded - do not fire when no default SC
(either k8s or virt).
- CDIDataImportCronOutdated - do not fire when no default SC (either
k8s or virt), as DIC import DVs use default SC.
- CDINoDefaultStorageClass - fire not only when there is a pending DV
(and PVC) but also when DV has an empty status (waiting for default
SC etc.)
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove expensive func test
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Refactor updateDataImportCronCondition
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add some verify comments
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add IsDataVolumeUsingDefaultStorageClass helper for readability
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
Have to better track when a PVC was created for a particular DV (populate it) or createed for someone else (adopt it)
Fixes: https://issues.redhat.com/browse/CNV-39618
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Add PVC mutating webhook using StorageProfiles
The webhook mutates the PVC Spec based on the available StorageProfiles,
so for example you can create PVC without accessModes and it will be
auto-completed.
To use this feature, enable the `WebhookPvcRendering` feature gate.
For any PVC you want to use StorageProfile, label it with:
cdi.kubevirt.io/useStorageProfile: "true"
If you want to use volumeMode preferred by CDI according to
StorageProfiles, set it to FromStorageProfile. Otherwise if not
explicitly set to Block, it will be Filesystem by k8s default.
E.g.:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-test
labels:
cdi.kubevirt.io/useStorageProfile: "true"
spec:
storageClassName: rook-ceph-block
volumeMode: FromStorageProfile
resources:
requests:
storage: 1Mi
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Move webhook create/delete to callback
plus some CR fixes and cleanups
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Move webhook CR creation to sit with callbacks
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Update existing webhook if modified
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Eliminate unnecessary CR update
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Bump k8s/OpenShift/ctrl-runtime/lifecycle-sdk & make deps-update
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Operator: adapt for dependency bump
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Controller: adapt watch calls for dependency bump
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Controller: adapt to ctrl-runtime's cache API changes
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Operator: fix unit tests by deleting resources properly in fake client
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Controller: fix unit tests by deleting resources properly in fake client
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Controller: adapt to fake client honoring status subresource
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Fix codegen script & make generate
There are some issues in the new script, so we
will still use the deprecated one.
More context in f4d1a5431b
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Functests: Adapt to NamespacedName now implementing MarshalLog
ns/name -> {"name":"name","namespace":"ns"}
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Functests & API server: address deprecation of wait.PollImmediate
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
---------
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Fix DIC DV creation when last import is not found
The `imports` local variable was not updated correctly, and later
referenced, so another reconcile was required for the DV to be created.
Fixes CNV-33973
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix DV controller Watch comment
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* apiserver support for cdi.kubevirt.io/allowClaimAdoption
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* controller support for cdi.kubevirt.io/allowClaimAdoption on existing PVC
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* if cdi.kubevirt.io/allowClaimAdoption specified on DataVolume do not apply on PVC until DV is succeeded
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add DataVolumeClaimAdoption featuregate and integrate with apiserver and controller
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* functional tests for claim adoption
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Fix func test failure and address some review comments
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* rename pvcRequiresNoWork to pvcRequiresWork
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* allow unbound PVC to be adopted
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Add tests to verify pvc is recreated and repopulated after delete
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Create import/upload source CR in case population pod is not succeeded
In case someone deleted the pvc a new pvc is created and the
status of the dv is still succeded. when using populators
in order for the population to restart and the dv state to change accordingly
we need to retrigger the population, this depends on the existing of the source CR.
That CR should exist as long as the pvc pod state is not succeeded.
When the pvc is recreated the succeded pod state annotation was removed.
Adjusted UT accordingly
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Adjust tests after review
Move cloner test to a parallel test suite.
Simplify import test.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
---------
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add Prometheus alerts and label existing alerts
- CDINoDefaultStorageClass - not having a default (or virt default)
SC is surely not an OpenShift error, as admins may prefer their cluster
users to only use explicit SC names. However, in the CDI context when
DV is created with default SC but default does not exist, we will fire
an error event and the PVC will be Pending for the default SC, so when
there are such Pending PVCs we will fire an alert.
- CDIDefaultStorageClassDegraded - when the default (or virt default)
SC does not support CSI/Snapshot clone (smart clone) or does not have
ReadWriteMany access mode (for live migration).
- CDIStorageProfilesIncomplete - add storageClass and provisioner
labels.
- CDIDataImportCronOutdated - add dataImportCron namespace and name
labels.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Create stub VolumeSnapshotClass for testing
Including the VolumeSnapshot/Class/Content crds for the
CDIDefaultStorageClassDegraded alert func test.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add snapshot manifests for tests
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Deploy snapshot CRDs in the hpp destructive lane
Remove stub snapshot CRDs
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add label explanation to new metric help
Also rename the metric kubevirt_cdi_storageprofile_status to
kubevirt_cdi_storageprofile_info since it always reports value 1,
where the label values provide the details about the storage
class and storage profile.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Revert NoProvisioner check removal
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Nicify StorageProfile metric update
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
When scratch space is needed during imports/uploads, the pod typically fails so it can be restarted with the required space. This is expected behavior that's automatically handled by CDI.
However, since we use an error exit code, this behavior leads to unwanted VM statuses during provisioning in Kubevirt.
This commit addresses this behavior by using a different DV condition, so kubevirt ignores it and the VM can be provisioned as expected.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Default virt storage class
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add alert for multiple default virt storage classes
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Refactor content type funcs to not return strings
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
---------
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
We slap on the clone-without-source index key regardless of source.PVC/source.Snapshot so
it was possible for a DV with PVC source to get queued for the snapshot controller
(and hence the nil ptr).
The test addition demonstrates a real use case where this would happen.
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
This annotation causes CDI transfer pods (importer, uploader, cloner) to be retained after a successful or failed completion.
This makes debugging and testing easier, as users can get the pod state and logs after completion.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Ensure priority class is assigned to pod populating pvc prime
Priority class was not being passed to the pvc prime from the PVC
and thus was not being added to the importer pod populating the
pvc prime. There is a list of allowed annotations that can be passed
and the priority class annotation was not in it. This commit adds
the annotation to the allowed list.
Cleaned up unneed log argument to a function related to passing the
annotations.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Restore the logger as it was logging the pvc name as well
as the log message. Modified the name to make it clearer
this is the case.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Have upload also use the populator, can't do clone
because the pod disappears too quickly without retain
Signed-off-by: Alexander Wels <awels@redhat.com>
* Test for priority class in upload as well as fix typo in cloning test.
Signed-off-by: Alexander Wels <awels@redhat.com>
---------
Signed-off-by: Alexander Wels <awels@redhat.com>
Currently there is a repeating reconcile error due to exponential
backoff: "DataVolume.storage spec is missing accessMode and no
storageClass to choose profile", however the error is not needed as we
already have SC watch for this case, e.g. when the DV is using the
default SC but no default SC is configured yet.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
This commit ups the cpu request for for all our installed compopents
(cdi-deployment, cdi-apiserver, cdi-uploadproxy, cdi-operator)
for 10m (1% of a core) to 100m (10% of a core).
The main driver of this is BZ: 2216038.
Without this change, it is pretty easy to create a large number of
concurrent clone operations and get token timeout errors.
Upping resource requests and concurrency addresses the issue
in a very direct way.
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* dataVolume: Add default instance type labels from source
f229aeb started to pass default instance type labels from
DataImportCrons to DataVolumes and DataVolumes to any associated
destination DataSources or PVCs. As documented in issue #2782 this does
not however pass these labels from the initial source of a DataVolume to
either the DataVolume or the destination DataSources or PVCs
This change corrects this by updating DataVolumes when reconciled,
adding any labels found on PVC or DataSource sources. These labels will
then be passed on to the destination PVC or DataSources by the existing
functionality highlighted above.
Note that if any default instance type labels already exist on the
DataVolume then the process is skipped as it is assumed these are
provided either directly by the user or via a DataImportCron.
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
* refactor: Use DefaultInstanceTypeLabels more often
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
---------
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
* Update VolumeImportSource API to support multi-stage imports
This commit modifies the VolumeImportSource API to support multi-stage imports, adding the following fields:
- Checkpoints, to represent the stages of a multistage import
- TargetClaim, the name of the specific PVC to be imported
- FinalCheckpoint, to indicate that the current Checkpoint is the final one
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Support multi-stage imports in import-populator
This commit updates the import populator to support multi-stage imports. The API and functionality remains the same as with DataVolumes, with the only difference that the used VolumeImportSource will now require a populated "TargetClaim" field that reffers to the specific PVC to be populated.
The DataVolume controller is also updated to allow using the populator flow with VDDK and ImageIO sources.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add unit tests for multistage import support in populators
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add functional tests to test multistage import populator flow
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Fix multi-stage import logic in import-populator and add remaining tests
This commit fixes several bugs in the import-populator logic for multi-stage imports.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
---------
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Allow ImmediateBind annotation when using populators
In case of using PVC with populators if the PVC has this
annotation we prevent from waiting for it to be schedueled
and we proceed with the process.
When using datavolumes with populators in case the dv has the annotation
it will be passed to the PVC. we prevent from being in pendingPopulation
in case the created pvc has the annotaion.
Plus when having honorWaitForFirstConsumer feature gate disabled we will
put on the target PVC the immediateBind annotation.
Now we allow to use populators when having the annotation the the
feature gate disabled.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add functional tests to population using PVCs
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Support immediate binding with clone datavolume
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Pass allowed annotations from target pvc to pvc prime
This annotations are used for the import/upload/clone
pods to define netork configurations.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
---------
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* remove CSI clone
bye bye
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* no more smart clone
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* PVC clone same namespace
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* cross namespace pvc clone
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* various fixes to get some functional tests to work
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* delete smart clone controller again
somehow reappeared after rebase
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* mostly pvc clone functional test fixes
make sure size detect pod only runs on kubevirt content type
clone populator was skipping last round op applying pvc' annotations
various func test fixes
review comments
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* more various test fixes
host clone phase should (implicitly) wait for clone source pod to exit
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove "smart" clone from snapshot
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* DataVolume clone from snapshot uses populator
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* improve clone populator/datavolume coordination on "running" condition
For host clone, not much changes, values still comming from annotations on host clone PVC
For smart/csi clone the DataVolume will be "running" if not in pending or error phase
Will have the same values for terminal "completed" state regardless of clone type
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* unit tests for pvc/snapshot clone controllers
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove skipped test added in https://github.com/kubevirt/containerized-data-importer/pull/2759
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* attempt address AfterSuite and generate-verify failures
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* handle snapshot clone with no target size specified
also add more validation to some snapshot clone tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove Patch calls
Using the controller runtime Patch API with controller runtime cached client seems to be a pretty bad fit
At least given the way the CR API is designed where an old object is compared to new.
I like patch in theory though and will revisit
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Clone populator should plan and execute even if PVC is bound
It was possible to miss "preallocation applied" annotation otherwise
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add long term token to datavolume
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Rename ProgressReporter to StatusReporter
Should have been done back when annotations were addded to "progress"
Also, if pvc is bound do not call phase Reconcile functions only Status
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
We were hitting a panic because we're passing the original DV object to renderPvcSpec (source is nil)
instead of the mutated DV which has sourceRef converted (source not nil)
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* move cleanup out of dv deletion
It seemed off to call cleanup in the prepare function
just because we don't call cleanup unless the dv is deleting.
Instead we check in the clenup function itself if it should be
done: in this 2 specific cases in case of deletion and in case
the dv succeeded.
The cleanup will be used in future commit also for population cleanup
which we also want to happen not only on deletion.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Use populator if csi storage class exists
Add new datavolume phase PendingPopulation to
indicate wffc when using populators, this new
phase will be used in kubevirt in order to know
that there is no need for dummy pod to pass wffc phase
and that the population will occur once creating the vm.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Update population targetPVC with pvc prime annotations
The annotations will be used to update dv that uses the
populators.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Adjust UT with new behavior
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* updates after review
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Fix import populator report progress
The import pod should be taken from pvcprime
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Prevent requeue upload dv when failing to find progress report pod
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Remove size inflation in populators
The populators are handling existing PVCs.
The PVC already has a defined requested size,
inflating the PVC' with fsoverhead will only be
on the PVC' spec and will not reflect on the target
PVC, this seems undesired.
Instead if the populators is using by PVC that the
datavolume controller created the inflation will happen
there if needed.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Adjust functional tests to handle dvs using populators
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Fix clone test
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* add shouldUpdateProgress variable to know if need to update progress
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Change update of annotation from denied list to allowed list
Instead if checking if the annotation on pvcPrime is not desired
go over desired list and if the annotation exists add it.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* fix removing annotations from pv when rebinding
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* More fixes and UT
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* a bit more updates and UTs
Signed-off-by: Shelly Kagan <skagan@redhat.com>
---------
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* touch up zero restoresize snapshot
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* clone populator
only supports PVC source now
snapshot coming soon
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* more unit tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* unit test for clone populator
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* func tests for clone populator
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* move clone populator cleanup function to planner
other review comments
verifier pod should bount readonly
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add readonly flag to test executor pods
synchronize get hash calls
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* increase linter timeout
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* better/explicit readonly support for test pods
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* check pv for driver info before looking up storageclass as it may not exist
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* addressed review comments
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* chooseStrategy shoud generate more events
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Create PVC if possible even if SC is missing
When PVC is created with storageClassName and the SC is not found,
k8s looks for PV with the storageClassName for satisfying this claim.
In this case k8s supports also a blank (“”, not the nil default one)
storageClassName. To support this behavior we added:
-DV controller support for this flow (for both “” and non-existing SC)
-Condition update and event when StorageSpec PVC rendering errors and
PVC is not created (e.g. missing both AccessModes and SC/PV)
-PVC is created even if a satisfying SC/PV doesn't exist if pvc/storage
AccessModes is set (otherwise k8s PVC validation fails). PVC/DV phase
will be Pending until a satisfying SC/PV is found
-PV watch to reconcile DVs waiting for the PV storageClassName
-PV storageClassName indexer, so we can list the relevant PVs
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Create populators package to be used for all populators
This commit introduces the basic reconciler for
populators with common function that can be used
by the different populators.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* unite getcontenttype func across code
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add VolumeImportSource CRD for import populator
This commit adds the VolumeImportSource CRD into CDI.
CRs created from this CRD will be referenced in the dataSourceRef field to populate PVCs with the import populator.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Refactor common populator code to be shared among all populators
This commit introduces and modifies several functions so we can reuse common code between all populators.
Other than having a common reconcile function, a new populatorController interface has been introduced so we are able to call populator-specific methods from the populator-base reconciler.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Create Import Populator
The import populator is a controller that handles the import of data in PVCs without the need of DataVolumes while still taking advantage of the import-controller flow.
This controller creates an additional PVC' with import annotations. After the import process succeeds, the controller rebinds the PV to the original target PVc and deletes the PVC prime.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add functional tests to cover the import populator flow
This commit updates the import tests to cover the new import populator flow.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add unit tests for import populator
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Minor fixes and enhancements in import/common populator code
* Modify indexes and other related code to support namespaced dataSourceRefs. Cross-namespace population is still not supported as it depends on alpha feature gates.
* Add functional test to cover static binding.
* Fix selected node annotation bug in scratch space PVCs
* Fix linter alerts
Signed-off-by: Alvaro Romero <alromero@redhat.com>
---------
Signed-off-by: Shelly Kagan <skagan@redhat.com>
Signed-off-by: Alvaro Romero <alromero@redhat.com>
Co-authored-by: Shelly Kagan <skagan@redhat.com>
* update k8s libs to 1.26.
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove some checks in log messages, they're redundant, and the format has changed
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* use 1.26 lib function `CheckVolumeModeMismatches` and `CheckAccessModes`
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Google Cloud Storage Importer
This is a Google Cloud Storage importer for CDI
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* Fix auto-generated swagger and openapi
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* GCS Importer General Fixes
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* Moving back gcs-secret.txt
Moving file back to imageDir to fix unit testing.
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
---------
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
Co-authored-by: Marcelo Parisi <marcelo@dev-box.corp.feitoza.com.br>
DV status can now be updated only by UpdateStatus() and no more by
Update(), so regular users cannot update status without special
permission. In addition, updating just the status would not update
the generation of the object, so we’ll have less “object has been
modified” errors in the reconcile loops.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
There should be no state shared between sync() and updateStatus().
updateStatus() should stand on it's own, and come to it's own
conclusions based on what it observes. It is okay if it is "late to the
party" and does not observe the latest changes in sync(). It will
eventually converge. This is what kubevirt does.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* function should return dataVolumeSyncResult, take *dataVolumeSyncResult as a parameter
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* checkStaticVolume implemetation for import DataVolume
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* upload support for checkStaticVolume
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* checkStaticVolume for clone datavolumes
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* checkStaticVolume for snapshot clone
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* checkStaticVolume for external populator source
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* tignten up static volume check
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* expand functional test to compare creation timestamps
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* updates from code review mostly add md5 verification to test and refacto common index creation
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* webhook changes, allow clone source DataVolumes (with special annotations)
even if source does not exist or user has no permission
BUT no token is added so this is really just for the static/prepopulate cases
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* API for VolumeSnapshot clone source
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Refactor doCrossNamespaceClone to get rid of some source PVC assumptions
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Refactor isCrossNamespaceClone to get rid of some some source PVC assumptions
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Clone from VolumeSnapshot source controller
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Func tests for cloning from volumesnapshot source
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Extend cross ns clone token mechanism for VolumeSnapshot cloning
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Validate source volumesnapshot on create
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
---------
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
Now that we update status in the end of a loop regardless of err,
it is important that we have the PVC in place for that.
Otherwise, if we fail at cleanup/prepare, syncRes.pvc is nil,
and we get false status updates like the following:
```bash
NAMESPACE NAME PHASE PROGRESS RESTARTS
default cloned-target-dv Succeeded N/A 23s
default cloned-target-dv Pending N/A 23s
default cloned-target-dv Pending N/A 23s
default cloned-target-dv Pending N/A 23s
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
golden-ns golden-snap-source Bound pvc-63ed1368-9f84-44ce-8b4b-7e9e3c4574f4 8Gi RWO rook-ceph-block 5m59s
(Target got deleted)
```
This results in a nasty bug where the target PVC simply disappears.
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
