The feature is available since v1.59, and we enable it by default to
allow increasing PVC size to the minimum supported by its provisioner
(#3711), and mainly in order to support:
https://github.com/kubevirt/kubevirt/pull/14637
As a bonus, the related Serial tests are now parallel. Thanks akalenyu:)
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add ExtraArgs field to VDDK CRD.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add tests for VDDK ExtraArgs field.
Add one unit test and rework existing functional test into a table for
both annotation and field entries.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Update ExtraArgs DataVolume documentation.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
---------
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add PVC source support for DataImportCron
A PVC from any namespace can now be the source for a DataImportCron. The
source digest is based on the PVC UID, which is polled by the schedule
similarly to image stream, so when a new PVC is detected it will be
imported.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Cleanups
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add annotation for extra VDDK library arguments.
The VDDK library itself accepts infrequently-used arguments in a
configuration file, and some of these arguments have been tested to show
a significant transfer speedup in some environments. This adds an
annotation that references a ConfigMap holding the contents of this VDDK
configuration file, and mounts it to the importer pod. The first file in
the mounted directory is passed to the VDDK.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add functional test for VDDK args annotation.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add unit test for extra VDDK arguments annotation.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add documentation for extra VDDK arguments.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Simplify new functional test annotation creation.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Look for specific file instead of first file.
Instead of listing the mounted VDDK arguments directory and filtering
out hidden files, just hard-code the expected file name and ConfigMap
key.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Move extra VDDK arguments functional test.
Put this in import_test and assert the values there, instead of in the
VDDK test plugin. The VDDK plugin logs the given values, and then the
test scans the log for what it expects to see.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Clean up lint error.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Move VDDK configuration test back, change test ID.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Avoid using kubectl for scanning nbdkit logs.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Temporary: show whole nbdkit log after failure.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Revert "Temporary: show whole nbdkit log after failure."
This reverts commit 488849f8fd.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Copy extra VDDK args annotation for populators.
Also add a related unit test.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Correct VDDK args config map mount comment.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
---------
Signed-off-by: Matthew Arnold <marnold@redhat.com>
After several releases with GC disabled by default, we decided to
deprecate it, as unfortunately it violates fundamental principle of
Kubernetes. CR should not be auto-deleted when it completes its role
(Job with TTLSecondsAfterFinished is an exception), and once CR was
created we can assume it is there until explicitly deleted. In addition,
CR should keep idempotency, so the same CR manifest can be applied
multiple times, as long as it is a valid update (e.g. DataVolume
validation webhook does not allow updating the spec).
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Update quota.md
The override Defaults didn't work for me, I updated them to what did work for me however i'm not sure how to check them, as the status of the CDIS object still shows the old settings. Adding information that states how to update it would be beneficial.
I also added docs that modifies the CDI directly.
Signed-off-by: cccsss01 <56396984+cccsss01@users.noreply.github.com>
* Update doc/quota.md
Co-authored-by: Michael Henriksen <mhenriks@redhat.com>
Signed-off-by: cccsss01 <56396984+cccsss01@users.noreply.github.com>
---------
Signed-off-by: cccsss01 <56396984+cccsss01@users.noreply.github.com>
Co-authored-by: Michael Henriksen <mhenriks@redhat.com>
* Suppress CDIDefaultStorageClassDegraded on SNO
On single-node OpenShift, even if none of the default/virt default
storage classes supports `ReadWriteMany` (but supports smart clone),
we will not fire the `CDIDefaultStorageClassDegraded` alert.
We added `degraded` label to `kubevirt_cdi_storageprofile_info` to
simplify the alert expression.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Cleanup utest
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Docs: Replace instances of pvc with storage
All instances of DataVolume > Spec > PVC have been replaced with storage,
except for two cases in datavolumes.md:
- Data Volumes>Target Storage/PVC>PVC
https://github.com/kubevirt/containerized-data-importer/blob/338bafe/doc/datavolumes.md#pvc
- Data Volumes>Source>PVC source
https://github.com/kubevirt/containerized-data-importer/blob/338bafe/doc/datavolumes.md#pvc-source
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Docs: Replace `Storage` with `storage` in doc/datavolumes.md
I also reworded it so that the paragraph does not start in lower-case.
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Docs: Remove optional default 'accessModes: ReadWriteOnce' from docs
Removed from storage options, I still kept it for the pvc.
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Docs: Remove resources/requests/storage when cloning a PVC
This argument is optional. Only removed for 'storage', not 'pvc'.
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Replace 'pvc' with 'storage' API in manifests
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Docs: Moved 'Target Storage/PVC' section above 'Source'
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Docs: Rephrase the Storage target section to emphasize it over PVC
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Fix empty field in YAML file
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Rephrase prose around filesystem volume mode
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Fix grammar and spelling
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Write acces mode in upgrade-testing-artifacts manifest
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Update doc/datavolumes.md
Co-authored-by: Alexander Wels <awels@redhat.com>
Signed-off-by: Edu Gómez Escandell <edu1997xyz@gmail.com>
---------
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
Signed-off-by: Edu Gómez Escandell <edu1997xyz@gmail.com>
Co-authored-by: Alexander Wels <awels@redhat.com>
* Fix progress metric registration and parsing
Use default metric registration. We shouldn't use the controller-runtime
registration as we have no controller here and it will not register the
metric correctly.
Fix the metric parsing to match its new name. Otherwise the DV progress
will not be updated until its 100%.
Regression introduced in #3254
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add kubevirt_cdi_import_progress_total metric
Use it in the importer instead of kubevirt_cdi_clone_progress_total and
fix metric parsing accordingly.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Move ProgressFromClaim to host-clone
Nobody else is using it.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add ProgressMetric interface
ProgressReader can now work with either import or clone progress metric.
FIXME: consider removing the direct Add/Get and use only via interface.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Refactor ProgressMetric interface
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Refactor progress parsing
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Refer metric names from the metrics package
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* clone_progress metric refactor
The clone_progress metric is not in the monitoring package. The metric
is with incorrect name, based on the kubevirt and Prometheus metrics
naming conventions. It's not documented and not located under
/pkg/monitoring. After the code refactoring we should not have
Prometheus metrics in other places in the code, other than the
/monitoring/metrics package, and metrics should be registered using
operator-observability package.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* openstack_populator_progress metric refactor
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* clone_progress metric refactor CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* ovirt_progress metric refactor
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Align progress metrics names with linter rules
Also add the metrics to the doc and json generation tools.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove redundant ListMetrics
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Mock NBD functions in VDDK unit tests.
Also add some example tests for GetBlockStatus.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Avoid infinite loop by returning whole block.
Also add a unit test to trigger this code. Without the fix, this spins
"No new block status data" messages forever, as reported in the bug.
With the fix, this continues the data transfer without zero-range or
hole-punch optimizations.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Correctly extend merged block length.
Bring in the previously proposed fix to make sure merged blocks have
correct lengths, avoiding the initial issue.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add two more GetBlockStatus unit tests.
Hopefully this makes it more obvious what GetBlockStatus actually does.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add unit tests for larger block sizes.
The first test intentionally overflows the result block's length field
and causes an infinite loop in GetBlockStatus, as pointed out in the
problem report. This will be fixed in the next commit.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Avoid integer overflow and sort out casts.
Increase the size of BlockStatusData.Length to an int64, to avoid
overflow in GetBlockStatus. Also change BlockStatusData.Offset to an
int64 and remove a handful of unnerving integer conversions. Favor int64
over uint64 to match system libraries, and add a few necessary
conversions mostly isolated to libnbd interfaces.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fix alignment for multi-stage VDDK documentation.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
---------
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Clone from snapshot: fix volume/access mode inferring for temp host assisted source PVC
Sometimes with snapshot cloning we have to fall back to host assisted.
To do this, we create a temporary restore from the snapshot and initiate a host assisted clone
from that -> target PVC.
The issue this commit fixes is that we set the wrong access/volume modes on this temporary restore PVC
(we set it to the target's).
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Annotate dataimportcron-created snapshot with their source volume mode
This would then get capitalized on by host assisted fallbacks from snapshot cloning
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
---------
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
Add docs which explain how PVCs of imported containerdisks can be
decorated with additional labels.
Signed-off-by: Felix Matouschek <fmatouschek@redhat.com>
* Add PVC mutating webhook using StorageProfiles
The webhook mutates the PVC Spec based on the available StorageProfiles,
so for example you can create PVC without accessModes and it will be
auto-completed.
To use this feature, enable the `WebhookPvcRendering` feature gate.
For any PVC you want to use StorageProfile, label it with:
cdi.kubevirt.io/useStorageProfile: "true"
If you want to use volumeMode preferred by CDI according to
StorageProfiles, set it to FromStorageProfile. Otherwise if not
explicitly set to Block, it will be Filesystem by k8s default.
E.g.:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-test
labels:
cdi.kubevirt.io/useStorageProfile: "true"
spec:
storageClassName: rook-ceph-block
volumeMode: FromStorageProfile
resources:
requests:
storage: 1Mi
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Move webhook create/delete to callback
plus some CR fixes and cleanups
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Move webhook CR creation to sit with callbacks
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Update existing webhook if modified
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Eliminate unnecessary CR update
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add Prometheus alerts and label existing alerts
- CDINoDefaultStorageClass - not having a default (or virt default)
SC is surely not an OpenShift error, as admins may prefer their cluster
users to only use explicit SC names. However, in the CDI context when
DV is created with default SC but default does not exist, we will fire
an error event and the PVC will be Pending for the default SC, so when
there are such Pending PVCs we will fire an alert.
- CDIDefaultStorageClassDegraded - when the default (or virt default)
SC does not support CSI/Snapshot clone (smart clone) or does not have
ReadWriteMany access mode (for live migration).
- CDIStorageProfilesIncomplete - add storageClass and provisioner
labels.
- CDIDataImportCronOutdated - add dataImportCron namespace and name
labels.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Create stub VolumeSnapshotClass for testing
Including the VolumeSnapshot/Class/Content crds for the
CDIDefaultStorageClassDegraded alert func test.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add snapshot manifests for tests
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Deploy snapshot CRDs in the hpp destructive lane
Remove stub snapshot CRDs
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add label explanation to new metric help
Also rename the metric kubevirt_cdi_storageprofile_status to
kubevirt_cdi_storageprofile_info since it always reports value 1,
where the label values provide the details about the storage
class and storage profile.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Revert NoProvisioner check removal
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Nicify StorageProfile metric update
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Default virt storage class
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add alert for multiple default virt storage classes
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Refactor content type funcs to not return strings
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
---------
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
Since clone_progress metric used for internal purpose only we should not
document it in metrics.md or test it in the metrics linter. therefore
clone_progress moved to separate list (not used by the doc and linter
generators).
Signed-off-by: Aviv Litman <alitman@alitman-thinkpadp1gen4i.tlv.csb>
Co-authored-by: Aviv Litman <alitman@alitman-thinkpadp1gen4i.tlv.csb>
* Allow modifying log verbosity from cdiConfig
This commit adds a new field in the cdiConfig API to allow specifying a log verbosity level to initialize all loggers.
Changes in this field will mean restarting all CDI components to initialize all loggers.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Update debug documentation to cover log verbosity
Signed-off-by: Alvaro Romero <alromero@redhat.com>
---------
Signed-off-by: Alvaro Romero <alromero@redhat.com>
This fix updated metrics names to meet the metrics naming conventions.
The old metrics names will not be available after this fix.
Signed-off-by: Aviv Litman <alitman@redhat.com>
* Disable DV GC by default
DataVolume garbage collection is a nice feature, but unfortunately it
violates fundamental principle of Kubernetes. CR should not be
auto-deleted when it completes its role (Job with TTLSecondsAfter-
Finished is an exception), and once CR was created we can assume it is
there until explicitly deleted. In addition, CR should keep idempotency,
so the same CR manifest can be applied multiple times, as long as it is
a valid update (e.g. DataVolume validation webhook does not allow
updating the spec).
When GC is enabled, some systems (e.g GitOps / ArgoCD) may require a
workaround (DV annotation deleteAfterCompletion = "false") to prevent
GC and function correctly.
On the next kubevirt-bot Bump kubevirtci PR (with bump-cdi), it will
fail on all kubevirtci lanes with tests referring DVs, as the tests
IsDataVolumeGC() looks at CDIConfig Spec.DataVolumeTTLSeconds and
assumes default is enabled. This should be fixed there.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix test waiting for PVC deletion with UID
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix clone test assuming DV was GCed
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix DIC controller DV/PVC deletion when snapshot is ready
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Google Cloud Storage Importer
This is a Google Cloud Storage importer for CDI
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* Fix auto-generated swagger and openapi
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* GCS Importer General Fixes
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* Moving back gcs-secret.txt
Moving file back to imageDir to fix unit testing.
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
---------
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
Co-authored-by: Marcelo Parisi <marcelo@dev-box.corp.feitoza.com.br>
This commit adds documentation for the external-population support in DataVolumes.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Avoid generating most of the v1alpha1 CRDs, regenerate
We leave the CDI CRD alone as that one is installed via a YAML file.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Avoid references to v1alpha1 APIs in docs
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Remove v1alpha1 specific tests
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Manually adjust code not to reference v1alpha1 APIs
v1alpha1 upload paths are kept as virtctl image upload still uses them
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Factor out scaling deployment to function
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Handle removal of v1alpha1 version from CRDs
If it was ever a storage version, we perform manual storage migration:
we ensure v1beta1 is storage version, get & update every object, and
remove the v1alpha1 storage version.
The CDI CRD v1alpha1 version is kept, so updating from clusters that
had old versions by applying release YAMLs still works.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Test upgrade from a version with v1alpha1 storage version.
Artificially create objects by scaling down cdi-operator & creating
a DV. Make sure it's there even after we restore CDI and that v1beta1
is the only storedVersion.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Regenerate (again, due to rebase)
Signed-off-by: Maya Rashish <mrashish@redhat.com>
Signed-off-by: Maya Rashish <mrashish@redhat.com>
