* Fix PVC' deletion logic in CDI populators
This commit ensures that we only proceed with PVC' deletion after confirming that the target PVC is rebound.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Refactor IsBound so we check PVC status instead of spec
PVC status should be the definitive source to know whether the PVC is bound or not. This commit updates our IsBound function to use status instead of spec.
It also updates unit tests and related code to match the new behavior.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add check for lost target PVC in populators
Signed-off-by: Alvaro Romero <alromero@redhat.com>
---------
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* cleanup: Extract label copying logic into common pkg
Extract the label copying logic from populator-base.go into the common
pkg as CopyAllowedLabels func.
Signed-off-by: Felix Matouschek <fmatouschek@redhat.com>
* fix(import-populator): Make copying of labels more robust
Make copying of labels from a prime PVC to the target PVC more robust,
by moving it before rebinding the PV from prime to target. This way we
can ensure the labels are already present once the PVC becomes ready.
Signed-off-by: Felix Matouschek <fmatouschek@redhat.com>
* cleanup: Do not pass labels from DIC to DS anymore
Do not pass labels from a DataImportCron to a DataSource in the
dataimportcron-controller anymore. In the future this will be
handled by the datasource-controller.
Signed-off-by: Felix Matouschek <fmatouschek@redhat.com>
* feat: Copy labels from source to DataSource
Copy labels from the source of a DataSource to the labels of the
DataSource in the datasource-controller.
Signed-off-by: Felix Matouschek <fmatouschek@redhat.com>
* tests: Add e2e tests for copying labels to DataSources
Add e2e tests that cover all scenarios where labels should be copied
from the source of a DataSource to the DataSource itself.
Signed-off-by: Felix Matouschek <fmatouschek@redhat.com>
* feat(dataimportcron-controller): Copy labels to VolumeSnapshots
When using VolumeSnapshots copy the labels found on the source PVC to
the created or an existing VolumeSnapshot.
Signed-off-by: Felix Matouschek <fmatouschek@redhat.com>
---------
Signed-off-by: Felix Matouschek <fmatouschek@redhat.com>
* Remove sourceref validation
The source can be changed and updated, we want the error to be
presented by an event and have the datavolume pending instead
of preventing the creation of the datavolume.
We are already handling the option of the source not existing yet
so lets also handle all the validation in the controller.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Move pvc clone specific functions to pvc-clone-controller
Now that the validation is no longer done in the webhook
no need for that code to be in the controller common util
file. Moved the UT accordingly.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Move function to new Describe node
Will add in future commits other tests to that section
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* pvc-clone-controller: make better event and update dv if validation fails
In order to make it more clear that the validation failed
we should add the reason of validation failure to the event
Added an update watch to the clone controller so if something changes
in the source the clone will reconcile. Hence no need to return an
error, the reconcile will be triggered if someting will change in the
source.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* snap-clone: always validate snapshot source
This commit adds missing validation due to the remove validation
from the dv webhook, but also fixes an existing missing validation
in cases where the clone was created before the source. In such case
the webhook would not validate the source since it didnt exist yet,
and then if the clone happened with populators then we would not
validate the source size at all.
We should validate the snapshot source before continuing with the clone
whether it is populator/legacy clones.
Moved all the validation to one function.
Updated the event to include the reason for the validation failure and
updated dv status accordingly.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Cloner_test: remove clone to small size test
The test checks the rejection of the webhook
which no longer exists then clone to appropriate
size dv which should succeed which is done in
any other clone test.
Im not adding a test to check the reconcile loop
since it is covered in the UT.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* tests/datasource: remove wait for pvc
Now since we added the validation also in the populators
path we are not creating the pvc until the source exists.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
---------
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* make deps-update
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* ReourceRequirements -> VolumeResourceRequirements
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix calls to controller.Watch()
controller-runtime changed the API!
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Fix errors with actual openshift/library-go lib
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* make all works now and everything compiles
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix "make update-codegen" because generate_groups.sh deprecated
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* run "make generate"
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix transfer unittest because of change to controller-runtime
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Fix progress metric registration and parsing
Use default metric registration. We shouldn't use the controller-runtime
registration as we have no controller here and it will not register the
metric correctly.
Fix the metric parsing to match its new name. Otherwise the DV progress
will not be updated until its 100%.
Regression introduced in #3254
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add kubevirt_cdi_import_progress_total metric
Use it in the importer instead of kubevirt_cdi_clone_progress_total and
fix metric parsing accordingly.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Move ProgressFromClaim to host-clone
Nobody else is using it.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add ProgressMetric interface
ProgressReader can now work with either import or clone progress metric.
FIXME: consider removing the direct Add/Get and use only via interface.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Refactor ProgressMetric interface
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Refactor progress parsing
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Refer metric names from the metrics package
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Move gosec into golangci-lint
Remove gosec target and scripts and use the golangci-lint linter
This ensures we stay up-to-date (so long as golangci-lint is up to date
too).
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G601: Disable for-loop variable aliassing warning (not relevant fro Go>=1.22)
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G101: Ignore warning about plain-text credentials
They are false positives
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G102: Don't listen to all interfaces
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G107: Ignore potentially tainted GET requests
They are all in test code
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G109: Avoid integer overflows after parsing strings
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G110: Potential DoS vulnerability via decompression bomb
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G112: Potential Slowloris Attack because ReadHeaderTimeout is not configured
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G114: Use of net/http serve function that has no support for setting timeouts
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G204: Subprocess launched with a potential tainted input or cmd arguments
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G305: File traversal when extracting zip/tar archive
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G306: Expect WriteFile permissions to be 0600 or less
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Bugfix: Misuse of file descriptor flags in file permission bits
os.WriteFile always uses O_WRONLY|O_CREATE|O_TRUNC, the third argument
is for the file's permission bits. This code is misleading, it will
truncate the file and not append to it. For that you'd need
os.Openfile(path, os.O_APPEND, 0600)
I also simplified the unnecessary []byte conversion.
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G401: Use of weak cryptographic primitive
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G402: Insecure TLS
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G404: Use of weak random number generator (math/rand instead of crypto/rand)
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* G501: Blocklisted import crypto/md5: weak cryptographic primitive
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Run autoformatters
Unrelated to the PR but this way we keep everything formatted
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Remove references to gosec in the makefile
Gosec has been moved into Golangci-lint
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
---------
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Enable revive linter
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Simplify cdi-func-test-proxy
This function had quite a bit of redundant code (caught by the linter).
The workgroup was never Done because all exit paths went through a
log.Fatal.
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Fix 'revive' linter warnings
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Fix tests that asserted on modified error messages
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Run make format
The formatted code has nothing to do with this PR but we may as well
include it.
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Use lower-case variables and use built-in min function in vddk-datasource
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Use contexts in cdi-func-test-proxy
This added quite a bit of boilerplate per call, so I put everything in
a loop.
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
---------
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Use direct io with qemu-img convert if target supports it
For a while now we have been switching between cache=none (direct io) and cache=writeback (page cache)
for qemu-img's writes.
We have settled on cache=writeback for quite some time since https://github.com/kubevirt/containerized-data-importer/pull/1919,
however, this has proven to be problematic;
Our pod's live in a constrained memory environment (default limit 600M).
cgroupsv1 compares utilization of page cache vs the host's dirty_ratio.
This means that on a standard system (30% dirty ratio) pages only get forced to disk at 0.3 * HOST_MEM (basically never),
easily triggering OOM on hosts with lots of free memory.
cgroupsv2 does come to the rescue here:
- It considers dirty_ratio against CGROUP_MEM
- Has a new memory.high knob that throttles instead of OOM killing
Sadly, k8s is yet to capitalize on memory.high since this feature is still alpha:
https://kubernetes.io/blog/2023/05/05/qos-memory-resources/
Leaving us with no way to avoid frequent OOMs.
This commit changes the way we write to bypass page cache if the target supports it,
otherwise, fall back to cache=writeback (use page cache).
There have previously been issues where target did not support O_DIRECT. A quick example is tmpfs (ram-based)
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Capitalize on cache mode=trynone if importer is being OOMKilled
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
---------
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Enable gofmt linter
From the docs:
> Gofmt checks whether code was gofmt-ed. By default this tool runs with
> -s option to check for code simplification.
https://golangci-lint.run/usage/linters/#gofmt
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Run gomft on the project
Ran this command after adding the gofmt linter:
golangci-lint run ./... --fix
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Enable whitespace linter
From the docs:
> Whitespace is a linter that checks for unnecessary newlines at the
> start and end of functions, if, for, etc.
https://golangci-lint.run/usage/linters/#whitespace
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Run whitespace on the project
Ran this command after adding the whitespace linter:
golangci-lint run ./... --fix
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Enable GCI linter
Per the docs:
> Gci controls Go package import order and makes it always deterministic.
https://golangci-lint.run/usage/linters/#gci
NOTE: I noticed that many files separate their imports in a particular
way, so I set the linter to enforce this standard.
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Run GCI on the project
Ran this command after adding the GCI linter:
golangci-lint run ./... --fix
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
---------
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Enable nakedret linter
> Checks that functions with naked returns are not longer than a maximum
> size (can be zero).
https://github.com/alexkohler/nakedret
Relevant from the GO style guide:
> Naked returns are okay if the function is a handful of lines. Once
> it’s a medium sized function, be explicit with your return values.
https://go.dev/wiki/CodeReviewComments#named-result-parameters
I think a naked return is never easier to read than a regular return,
so I set the linter to always complain about it. Disagreements are
welcome.
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Refactor functions with naked returns
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
---------
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Enable errorlint linter
Per the readme:
> go-errorlint is a source code linter for Go software that can be used
to find code that will cause problems with the error wrapping scheme
introduced in Go 1.13.
https://github.com/polyfloyd/go-errorlint
Essentially, it checks that you properly use errors.Is and errors.As
It also forces you to use %w, but that is a bit too much. Using %v is
fine when you don't want to leak the internal error types of the
package. This is why I disabled this setting.
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Replace error comparisons with errors.Is
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Replace error type assertions with errors.As
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Enable errname linter
Per the readme:
> Checks that sentinel errors are prefixed with the Err and error
> types are suffixed with the Error.
https://github.com/Antonboom/errname
Not a big deal, but helps keep the naming consistent.
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Fix errname linter warnings
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
---------
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Enable unconvert linter
This linter's doc describes it as:
The unconvert program analyzes Go packages to identify unnecessary
type conversions; i.e., expressions T(x) where x already has type T.
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Unrestrict the number of linter warnings
It is best to show all warnings at once than to reveal them piece-meal,
particularly in CI where the feedback loop can be a bit slow.
By default, linters may only print the same message three times
(https://golangci-lint.run/usage/configuration/#issues-configuration)
The unconvert linter always prints the same message, so it specially
affected by this setting.
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Remove redundant type conversions
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
---------
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Add misspell to list of linters
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Fix spelling errors
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
---------
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
Joining hostname+port with string concatenation leads to wrong URLs
when the hostame is an IPv6:
HOST PORT Sprintf CORRECT
::1 8000 ::1:8000 [::1]:8000
To avoid this issue, it's best to use net.JoinHostPort. I added a test
that verifies this fix.
This type of issue can be discovered running the following command:
golangci-lint run ./... --enable nosprintfhostport
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Suppress alerts to reduce noise of dependent ones
This is a follow-up to #2998 introducing the following changes to alert
rules:
- CDIDefaultStorageClassDegraded - do not fire when no default SC
(either k8s or virt).
- CDIDataImportCronOutdated - do not fire when no default SC (either
k8s or virt), as DIC import DVs use default SC.
- CDINoDefaultStorageClass - fire not only when there is a pending DV
(and PVC) but also when DV has an empty status (waiting for default
SC etc.)
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove expensive func test
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Refactor updateDataImportCronCondition
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add some verify comments
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add IsDataVolumeUsingDefaultStorageClass helper for readability
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Clone from snapshot: fix volume/access mode inferring for temp host assisted source PVC
Sometimes with snapshot cloning we have to fall back to host assisted.
To do this, we create a temporary restore from the snapshot and initiate a host assisted clone
from that -> target PVC.
The issue this commit fixes is that we set the wrong access/volume modes on this temporary restore PVC
(we set it to the target's).
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Annotate dataimportcron-created snapshot with their source volume mode
This would then get capitalized on by host assisted fallbacks from snapshot cloning
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
---------
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
Have to better track when a PVC was created for a particular DV (populate it) or createed for someone else (adopt it)
Fixes: https://issues.redhat.com/browse/CNV-39618
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Add PVC mutating webhook using StorageProfiles
The webhook mutates the PVC Spec based on the available StorageProfiles,
so for example you can create PVC without accessModes and it will be
auto-completed.
To use this feature, enable the `WebhookPvcRendering` feature gate.
For any PVC you want to use StorageProfile, label it with:
cdi.kubevirt.io/useStorageProfile: "true"
If you want to use volumeMode preferred by CDI according to
StorageProfiles, set it to FromStorageProfile. Otherwise if not
explicitly set to Block, it will be Filesystem by k8s default.
E.g.:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-test
labels:
cdi.kubevirt.io/useStorageProfile: "true"
spec:
storageClassName: rook-ceph-block
volumeMode: FromStorageProfile
resources:
requests:
storage: 1Mi
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Move webhook create/delete to callback
plus some CR fixes and cleanups
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Move webhook CR creation to sit with callbacks
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Update existing webhook if modified
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Eliminate unnecessary CR update
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* apiserver support for cdi.kubevirt.io/allowClaimAdoption
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* controller support for cdi.kubevirt.io/allowClaimAdoption on existing PVC
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* if cdi.kubevirt.io/allowClaimAdoption specified on DataVolume do not apply on PVC until DV is succeeded
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add DataVolumeClaimAdoption featuregate and integrate with apiserver and controller
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* functional tests for claim adoption
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Fix func test failure and address some review comments
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* rename pvcRequiresNoWork to pvcRequiresWork
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* allow unbound PVC to be adopted
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Add Prometheus alerts and label existing alerts
- CDINoDefaultStorageClass - not having a default (or virt default)
SC is surely not an OpenShift error, as admins may prefer their cluster
users to only use explicit SC names. However, in the CDI context when
DV is created with default SC but default does not exist, we will fire
an error event and the PVC will be Pending for the default SC, so when
there are such Pending PVCs we will fire an alert.
- CDIDefaultStorageClassDegraded - when the default (or virt default)
SC does not support CSI/Snapshot clone (smart clone) or does not have
ReadWriteMany access mode (for live migration).
- CDIStorageProfilesIncomplete - add storageClass and provisioner
labels.
- CDIDataImportCronOutdated - add dataImportCron namespace and name
labels.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Create stub VolumeSnapshotClass for testing
Including the VolumeSnapshot/Class/Content crds for the
CDIDefaultStorageClassDegraded alert func test.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add snapshot manifests for tests
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Deploy snapshot CRDs in the hpp destructive lane
Remove stub snapshot CRDs
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add label explanation to new metric help
Also rename the metric kubevirt_cdi_storageprofile_status to
kubevirt_cdi_storageprofile_info since it always reports value 1,
where the label values provide the details about the storage
class and storage profile.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Revert NoProvisioner check removal
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Nicify StorageProfile metric update
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
Some CSI drivers [0] don't allow mounting block ro,
so they simply cannot perform host assisted clones.
Host assisted clones are important as they are a last resort in some cases
like switching over to a new storage provisioner.
This may or may not have some performance implications (parallel cloning from single source)
but we are choosing compatibility over performance in this case.
[0] 76cb78decd/service/node.go (L420-L424)
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Default virt storage class
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add alert for multiple default virt storage classes
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Refactor content type funcs to not return strings
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
---------
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Ensure priority class is assigned to pod populating pvc prime
Priority class was not being passed to the pvc prime from the PVC
and thus was not being added to the importer pod populating the
pvc prime. There is a list of allowed annotations that can be passed
and the priority class annotation was not in it. This commit adds
the annotation to the allowed list.
Cleaned up unneed log argument to a function related to passing the
annotations.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Restore the logger as it was logging the pvc name as well
as the log message. Modified the name to make it clearer
this is the case.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Have upload also use the populator, can't do clone
because the pod disappears too quickly without retain
Signed-off-by: Alexander Wels <awels@redhat.com>
* Test for priority class in upload as well as fix typo in cloning test.
Signed-off-by: Alexander Wels <awels@redhat.com>
---------
Signed-off-by: Alexander Wels <awels@redhat.com>
* dataVolume: Add default instance type labels from source
f229aeb started to pass default instance type labels from
DataImportCrons to DataVolumes and DataVolumes to any associated
destination DataSources or PVCs. As documented in issue #2782 this does
not however pass these labels from the initial source of a DataVolume to
either the DataVolume or the destination DataSources or PVCs
This change corrects this by updating DataVolumes when reconciled,
adding any labels found on PVC or DataSource sources. These labels will
then be passed on to the destination PVC or DataSources by the existing
functionality highlighted above.
Note that if any default instance type labels already exist on the
DataVolume then the process is skipped as it is assumed these are
provided either directly by the user or via a DataImportCron.
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
* refactor: Use DefaultInstanceTypeLabels more often
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
---------
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
* Update VolumeImportSource API to support multi-stage imports
This commit modifies the VolumeImportSource API to support multi-stage imports, adding the following fields:
- Checkpoints, to represent the stages of a multistage import
- TargetClaim, the name of the specific PVC to be imported
- FinalCheckpoint, to indicate that the current Checkpoint is the final one
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Support multi-stage imports in import-populator
This commit updates the import populator to support multi-stage imports. The API and functionality remains the same as with DataVolumes, with the only difference that the used VolumeImportSource will now require a populated "TargetClaim" field that reffers to the specific PVC to be populated.
The DataVolume controller is also updated to allow using the populator flow with VDDK and ImageIO sources.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add unit tests for multistage import support in populators
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add functional tests to test multistage import populator flow
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Fix multi-stage import logic in import-populator and add remaining tests
This commit fixes several bugs in the import-populator logic for multi-stage imports.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
---------
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Allow ImmediateBind annotation when using populators
In case of using PVC with populators if the PVC has this
annotation we prevent from waiting for it to be schedueled
and we proceed with the process.
When using datavolumes with populators in case the dv has the annotation
it will be passed to the PVC. we prevent from being in pendingPopulation
in case the created pvc has the annotaion.
Plus when having honorWaitForFirstConsumer feature gate disabled we will
put on the target PVC the immediateBind annotation.
Now we allow to use populators when having the annotation the the
feature gate disabled.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add functional tests to population using PVCs
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Support immediate binding with clone datavolume
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Pass allowed annotations from target pvc to pvc prime
This annotations are used for the import/upload/clone
pods to define netork configurations.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
---------
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* remove CSI clone
bye bye
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* no more smart clone
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* PVC clone same namespace
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* cross namespace pvc clone
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* various fixes to get some functional tests to work
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* delete smart clone controller again
somehow reappeared after rebase
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* mostly pvc clone functional test fixes
make sure size detect pod only runs on kubevirt content type
clone populator was skipping last round op applying pvc' annotations
various func test fixes
review comments
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* more various test fixes
host clone phase should (implicitly) wait for clone source pod to exit
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove "smart" clone from snapshot
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* DataVolume clone from snapshot uses populator
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* improve clone populator/datavolume coordination on "running" condition
For host clone, not much changes, values still comming from annotations on host clone PVC
For smart/csi clone the DataVolume will be "running" if not in pending or error phase
Will have the same values for terminal "completed" state regardless of clone type
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* unit tests for pvc/snapshot clone controllers
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove skipped test added in https://github.com/kubevirt/containerized-data-importer/pull/2759
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* attempt address AfterSuite and generate-verify failures
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* handle snapshot clone with no target size specified
also add more validation to some snapshot clone tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove Patch calls
Using the controller runtime Patch API with controller runtime cached client seems to be a pretty bad fit
At least given the way the CR API is designed where an old object is compared to new.
I like patch in theory though and will revisit
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Clone populator should plan and execute even if PVC is bound
It was possible to miss "preallocation applied" annotation otherwise
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add long term token to datavolume
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Rename ProgressReporter to StatusReporter
Should have been done back when annotations were addded to "progress"
Also, if pvc is bound do not call phase Reconcile functions only Status
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* dataimportcron: code change: Use better matchers in tests
Signed-off-by: Andrej Krejcir <akrejcir@redhat.com>
* dataimportcron: Pass dynamic credential support label
The label is passed from DataImportCron to DataVolume
and DataSource.
Signed-off-by: Andrej Krejcir <akrejcir@redhat.com>
---------
Signed-off-by: Andrej Krejcir <akrejcir@redhat.com>
* Disable DV GC by default
DataVolume garbage collection is a nice feature, but unfortunately it
violates fundamental principle of Kubernetes. CR should not be
auto-deleted when it completes its role (Job with TTLSecondsAfter-
Finished is an exception), and once CR was created we can assume it is
there until explicitly deleted. In addition, CR should keep idempotency,
so the same CR manifest can be applied multiple times, as long as it is
a valid update (e.g. DataVolume validation webhook does not allow
updating the spec).
When GC is enabled, some systems (e.g GitOps / ArgoCD) may require a
workaround (DV annotation deleteAfterCompletion = "false") to prevent
GC and function correctly.
On the next kubevirt-bot Bump kubevirtci PR (with bump-cdi), it will
fail on all kubevirtci lanes with tests referring DVs, as the tests
IsDataVolumeGC() looks at CDIConfig Spec.DataVolumeTTLSeconds and
assumes default is enabled. This should be fixed there.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix test waiting for PVC deletion with UID
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix clone test assuming DV was GCed
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix DIC controller DV/PVC deletion when snapshot is ready
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* move cleanup out of dv deletion
It seemed off to call cleanup in the prepare function
just because we don't call cleanup unless the dv is deleting.
Instead we check in the clenup function itself if it should be
done: in this 2 specific cases in case of deletion and in case
the dv succeeded.
The cleanup will be used in future commit also for population cleanup
which we also want to happen not only on deletion.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Use populator if csi storage class exists
Add new datavolume phase PendingPopulation to
indicate wffc when using populators, this new
phase will be used in kubevirt in order to know
that there is no need for dummy pod to pass wffc phase
and that the population will occur once creating the vm.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Update population targetPVC with pvc prime annotations
The annotations will be used to update dv that uses the
populators.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Adjust UT with new behavior
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* updates after review
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Fix import populator report progress
The import pod should be taken from pvcprime
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Prevent requeue upload dv when failing to find progress report pod
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Remove size inflation in populators
The populators are handling existing PVCs.
The PVC already has a defined requested size,
inflating the PVC' with fsoverhead will only be
on the PVC' spec and will not reflect on the target
PVC, this seems undesired.
Instead if the populators is using by PVC that the
datavolume controller created the inflation will happen
there if needed.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Adjust functional tests to handle dvs using populators
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Fix clone test
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* add shouldUpdateProgress variable to know if need to update progress
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Change update of annotation from denied list to allowed list
Instead if checking if the annotation on pvcPrime is not desired
go over desired list and if the annotation exists add it.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* fix removing annotations from pv when rebinding
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* More fixes and UT
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* a bit more updates and UTs
Signed-off-by: Shelly Kagan <skagan@redhat.com>
---------
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* StorageProfile API for declaring format of resulting cron disk images
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Integrate recommended format in dataimportcron controller
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Take snapclass existence into consideration when populating cloneStrategy and sourceFormat
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
---------
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add clone from snapshot functionalities to the clone populator
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Update clone populator unit tests to cover clone from snapshot capabilities
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Fix storage class assignation in temp-source claim for host-assisted clone from snapshot
This commit also includes other minor and styling-related fixes
Signed-off-by: Alvaro Romero <alromero@redhat.com>
---------
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* touch up zero restoresize snapshot
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* clone populator
only supports PVC source now
snapshot coming soon
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* more unit tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* unit test for clone populator
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* func tests for clone populator
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* move clone populator cleanup function to planner
other review comments
verifier pod should bount readonly
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add readonly flag to test executor pods
synchronize get hash calls
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* increase linter timeout
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* better/explicit readonly support for test pods
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* check pv for driver info before looking up storageclass as it may not exist
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* addressed review comments
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* chooseStrategy shoud generate more events
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Create PVC if possible even if SC is missing
When PVC is created with storageClassName and the SC is not found,
k8s looks for PV with the storageClassName for satisfying this claim.
In this case k8s supports also a blank (“”, not the nil default one)
storageClassName. To support this behavior we added:
-DV controller support for this flow (for both “” and non-existing SC)
-Condition update and event when StorageSpec PVC rendering errors and
PVC is not created (e.g. missing both AccessModes and SC/PV)
-PVC is created even if a satisfying SC/PV doesn't exist if pvc/storage
AccessModes is set (otherwise k8s PVC validation fails). PVC/DV phase
will be Pending until a satisfying SC/PV is found
-PV watch to reconcile DVs waiting for the PV storageClassName
-PV storageClassName indexer, so we can list the relevant PVs
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Create populators package to be used for all populators
This commit introduces the basic reconciler for
populators with common function that can be used
by the different populators.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* unite getcontenttype func across code
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add VolumeImportSource CRD for import populator
This commit adds the VolumeImportSource CRD into CDI.
CRs created from this CRD will be referenced in the dataSourceRef field to populate PVCs with the import populator.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Refactor common populator code to be shared among all populators
This commit introduces and modifies several functions so we can reuse common code between all populators.
Other than having a common reconcile function, a new populatorController interface has been introduced so we are able to call populator-specific methods from the populator-base reconciler.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Create Import Populator
The import populator is a controller that handles the import of data in PVCs without the need of DataVolumes while still taking advantage of the import-controller flow.
This controller creates an additional PVC' with import annotations. After the import process succeeds, the controller rebinds the PV to the original target PVc and deletes the PVC prime.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add functional tests to cover the import populator flow
This commit updates the import tests to cover the new import populator flow.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add unit tests for import populator
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Minor fixes and enhancements in import/common populator code
* Modify indexes and other related code to support namespaced dataSourceRefs. Cross-namespace population is still not supported as it depends on alpha feature gates.
* Add functional test to cover static binding.
* Fix selected node annotation bug in scratch space PVCs
* Fix linter alerts
Signed-off-by: Alvaro Romero <alromero@redhat.com>
---------
Signed-off-by: Shelly Kagan <skagan@redhat.com>
Signed-off-by: Alvaro Romero <alromero@redhat.com>
Co-authored-by: Shelly Kagan <skagan@redhat.com>
* Google Cloud Storage Importer
This is a Google Cloud Storage importer for CDI
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* Fix auto-generated swagger and openapi
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* GCS Importer General Fixes
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* Moving back gcs-secret.txt
Moving file back to imageDir to fix unit testing.
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
---------
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
Co-authored-by: Marcelo Parisi <marcelo@dev-box.corp.feitoza.com.br>
* Add support for imagePullSecrets in the CDI CR, to support pulling
images from repositories that require secrets.
The imagePullSecrets is propagated to the following components: cdi-apiserver,
cdi-deployment, and cdi-uploadproxy. The definition of imagePullSecrets in
cdi-operator must be done manually.
Signed-off-by: Gleb Aronsky <gleb.aronsky@windriver.com>
* Modifying code to incorporate review comments.
Signed-off-by: Gleb Aronsky <gleb.aronsky@windriver.com>
---------
Signed-off-by: Gleb Aronsky <gleb.aronsky@windriver.com>
Co-authored-by: Gleb Aronsky <gleb.aronsky@windriver.com>
* Fix hostpath CSI being skipped as "Not HPP"
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Fall back to host assisted if immediate bind requested
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
---------
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* function should return dataVolumeSyncResult, take *dataVolumeSyncResult as a parameter
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* checkStaticVolume implemetation for import DataVolume
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* upload support for checkStaticVolume
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* checkStaticVolume for clone datavolumes
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* checkStaticVolume for snapshot clone
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* checkStaticVolume for external populator source
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* tignten up static volume check
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* expand functional test to compare creation timestamps
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* updates from code review mostly add md5 verification to test and refacto common index creation
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* webhook changes, allow clone source DataVolumes (with special annotations)
even if source does not exist or user has no permission
BUT no token is added so this is really just for the static/prepopulate cases
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
