Commit Graph

113 Commits

Author SHA1 Message Date
Arnon Gilboa
0b3900c60c
Garbage Collect Completed DVs (#2233)
* Garbage Collect Completed DVs

See design at:
https://github.com/kubevirt/community/blob/main/design-proposals/garbage-collect-completed-dvs.md

ToDos:
-DataImportCron and DataSource controllers adaptation and func tests
-Add doc for DataVolume, CDIConfig and DataImportCron changes
-Extend unit tests and functional tests
-KubeVirt adaptation

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Controller minor fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Adapt tests to GC

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add DV mutate unit test for GC

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Improve GC skip per annotation test

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Use DescribeTable for the GC tests

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2022-05-19 00:09:31 +02:00
Shelly Kagan
aa6b9dd403
Fix smart clone request size update (#2276)
* Fix smart clone request size update

In case the pvc got an actual size as the expected
size there wasn't an update of the pvc spec with the
actual user request size, which left the pvc spec with
a smaller size then the user requested(the data size).
This caused a discrepancy when trying to restore such pvc,
which restored a pvc with the small size instead of the user request
size.

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* review fixes

Signed-off-by: Shelly Kagan <skagan@redhat.com>
2022-05-18 17:21:33 +02:00
Alexander Wels
bbee324537
Make CDI clone strategy strings correct types (#2277)
Noticed the consts were strings instead of the right types
this changes the type to the right type and modifies the usage
to not have to cast to the right type all over the place.

Signed-off-by: Alexander Wels <awels@redhat.com>
2022-05-17 14:19:25 +02:00
Maya Rashish
c05a750a9a
Detect storage capabilities for no-provisioner storage classes (#2226)
* Detect storage capabilities for no-provisioner storage classes

Assume there's a persistent volume that we can look up to infer the
correct values for volume mode and access modes.

Limit ourselves to detecting no-provisioner capabilities on LSO to
avoid greatly increasing the number of storage classes we provide
capabilities for. This is similar to our current flow where we
only provide capabilities for known storage classes.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Regenerate bazel stuff for pkg/monitoring's existence

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Add a watcher for no-provisioner PVs

We maintain a map of storage class names and provisioners whenever
storage classes are changed.

If a PV has one of the storage classes with no-provisioner as a
provisioner, reconcile that storage class.

This is because we infer the storage profile based on PVs, and
new ones might have different storage capabilities.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Use a client to do our storage class caching

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Pass a client as an argument, not global.

Suggested by awels, thanks!

Signed-off-by: Maya Rashish <mrashish@redhat.com>
2022-05-12 14:00:02 +02:00
Alexander Wels
b3d08268e2
Start smart clone controller from datavolume controller when needed (#2265)
* periodic sync CSI snapshot CRD check

It was possible for the CSI snapshot CRD check to fail silently and
prevent the smart clone controller from starting during the cdi deployment
pod start up. This would prevent smart clone from working properly.

This adds a periodic sync of 1 minute for checking the CRDs. We also
log failures that are not is not found so we can more easily detect this
situation as humans.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Change location of the start controller call.

Signed-off-by: Alexander Wels <awels@redhat.com>
2022-05-12 04:28:01 +02:00
Bartosz Rybacki
5fe69cb718
Bugfix - Correctly handle populated PVC (#2227)
* Test handling of populated PVC

Populated PVC created from clone operation should not start
any CDI actions. It can only update DV status.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Handle prepopulated pvc with network clone

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Handle prepopulated PVC for Smart and CSI clone

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2022-04-29 14:48:01 +02:00
Bartosz Rybacki
11e9a53533
Extract the clone logic from main reconcile loop (#2241)
* Extract the clone logic for reconcile

This change only extracts the code branch that handles the clone.
The main idea is to divide the reconcile loop into parts doing import,
upload and clone. This makes code easier to read and understand.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Cleanup handling not existing PVC

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2022-04-21 04:13:51 +02:00
akalenyu
71522a1f2d
Switch VolumeSnapshot to v1 (#2235)
* Switch VolumeSnapshot to v1

VolumeSnapshot v1beta is being deprecated:
https://kubernetes.io/blog/2022/04/07/upcoming-changes-in-kubernetes-1-24/#api-removals-deprecations-and-other-changes-for-kubernetes-1-24

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Fix unit tests; change version we look for in IsCsiCrdsDeployed

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
2022-04-12 18:50:19 +02:00
Bartosz Rybacki
2f7987d407
Improve handling prePopulated DV (#2205)
Previously, when a DataVolume was Reconciled then CDI created PVC
regardless of prePopulated annotation.

There is a possibility that DataVolume and PVC pair is being
restored or moved from other cluster. When DataVolume is marked
as prePopulated, the CDI should not try to populate it.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2022-04-07 21:55:44 +02:00
Arnon Gilboa
e9fc2009a5
Reconcile DVs waiting for default SC (#2196)
* Reconcile DVs waiting for default SC

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2022-03-24 12:44:59 +01:00
akalenyu
18c815261d
kubevirtci bump/controller change to overcome AfterSuite flake (#2162)
* Update kubevirtci to overcome AfterSuite flake

Update kubevirtci to get a fix for a flake where PVC cant be removed
because it still holds the `pvc-as-source-protection` finalizer:
https://github.com/kubernetes-csi/external-snapshotter/issues/349

More info in https://github.com/kubevirt/kubevirtci/pull/750.

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Don't create multiple VolumeSnapshots

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
2022-02-23 17:44:00 +01:00
akalenyu
e678f5d0e9
Fix SnapshotForSmartCloneInProgress sometimes not occuring (#2158)
Sometimes we skip SnapshotForSmartCloneInProgress and directly go to
Succeeded.
This happens when the `r.updateDataVolume(dataVolumeCopy)` call fails.

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
2022-02-16 22:42:21 +01:00
Matthew Arnold
e92013d079
Fix interaction between multi-stage import and retainAfterCompletion. (#2146)
* Append checkpoint ID to multi-stage importer pods.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Ignore completed pods for multi-stage imports.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Reset current import pod when checkpoint is done.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Don't prevent pod deletion for scratch space.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Only ignore pod when retainAfterCompletion is set.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Fix data volume unit tests.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Tests for checkpoint suffix and completed pods.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Test for retained pods exiting for scratch space.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add functional test for retaining multistage pods.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Clean up lint error.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Remove scratch handling that is fixed elsewhere.

This is part of shouldDeletePod now.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add unit tests for long PVC/checkpoint names.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Match retainAfterCompletion test to description.

Signed-off-by: Matthew Arnold <marnold@redhat.com>
2022-02-11 23:14:07 +01:00
Bartosz Rybacki
e18fc68718
BugId: 2038679 - Clone with volume mode file system using Storage API fails (#2096)
* Update clone size validation logic

The case with DV using spec.storage API needs
more complex validation that will be added in the
clone controller. The API webhook validation
for that case is removed.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Improve DV phase failure message in tests

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Add test and warning event for clone size

During clone check if actual requested size on source volume is bigger
than target requested size and emit an event to notify user about situation.

Actual size on filesystem is lower that requested, because of possible filesystem overhead. When using storage API the overhead will be applied on target.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Code Review cleanup - Removing debug logs

Removed some garbage left after troubleshooting.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Move fn GetUsableSpace to common utils

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2022-02-02 17:53:10 +01:00
Matthew Arnold
7806e77bdf
Allow optional per-DataVolume VDDK image. (#2102)
* Add optional VDDK initImageURL field.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Pass VDDK image URL through to PVC annotation.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Unit tests for per-DV VDDK image URL.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Functional test for VDDK initImageURL field.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Update documentation for VDDK initImageURL.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Fix lint error.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Check for absence of AwaitingVDDK in unit test.

Signed-off-by: Matthew Arnold <marnold@redhat.com>
2022-01-19 22:49:48 +01:00
akalenyu
483359bf69
Add label on our PVCs to prevent unnecessary alert from going off (#2093)
We want to silence the KubePersistentVolumeFillingUp for all our PVCs that hold virtual machine disks,
since these disks consume the entire PVC by design.

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
2022-01-14 15:35:06 +01:00
Maya Rashish
f2939fcee8
add PVC claimName to datavolume status (#2060)
* Make it possible to find the underlying PVC name using the DV

Right now a lot of things assume that the underlying PVC has the
same name/namespace, let's make it possible to reach over and not
need to have this implicit knowledge in a lot of places.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Install some artifacts on the old version of CDI during upgrade tests

And use this to test that DataVolume.Status.ClaimName is set after
upgrades.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Bump CDI pod update timeout

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Only check if non-testing CDI pods have updated.

We don't update the testing environment, so it looks like some of the
update fails.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Restore lower timeouts

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* As elsewhere, don't use local registry artifacts with external provider

Signed-off-by: Maya Rashish <mrashish@redhat.com>
2022-01-07 20:28:25 +01:00
Shelly Kagan
41df5c240e
Emit event and update dv conditions when pvc fails to create due to quota (#2016)
* Update datavolume conditions when quota exceeded when creating pvc

When creating the pvc from the dv the pvc size
can exceed the allowed quota, in such case so far the only
indication was to look in the logs.
Now added indication in the data volume conditions
(when possible) and emitted event.

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Add functional tests to check the new conditons and event

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* tests cosmetics

-use existing functions
-add missing checks on errors
-remove unused code
-etc..

Signed-off-by: Shelly Kagan <skagan@redhat.com>
2021-11-16 17:29:39 +01:00
Matthew Arnold
703e421a8a
Allow user-specified headers in HTTP data source. (#1994)
* Update HTTP data source API to allow custom headers.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Implement custom HTTP headers API.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Document custom headers in HTTP data source.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Correct secretExtraHeader comment to reference Secret.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add volume mounts for secret headers.

Replaces environment variables for headers from secrets.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Avoid failing when there are no extra headers.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Redact contents of headers that come from secrets.

Also split up getExtraHeaders to reduce Sonar Cloud complexity.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Ensure all HTTP client requests use extra headers.

Missed redirect check and content length retrieval, both of which might
need the extra headers.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add some unit tests for extra HTTP headers.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Do not quote headers in nbdkit curl arguments.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add functional tests for extra HTTP headers.

Avoids new test server by specifiying basic authorization headers to the
existing file host port that requires it.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Use filepath.Walk to read secrets.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Minor documentation update for secrets.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Re-run 'make generate' for verification failure.

Signed-off-by: Matthew Arnold <marnold@redhat.com>
2021-11-12 21:06:57 +01:00
Shelly Kagan
1eda6db87e
Datacontroller cosmetics (#2008)
* move common code to controller util

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* remove unused functions and cosmetics

Signed-off-by: Shelly Kagan <skagan@redhat.com>
2021-11-04 18:05:55 +01:00
Shelly Kagan
e7dd62eb26
Upload archive (#1969)
* Add support for archive upload

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* fix golang errors

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Change storage profile property set to support more then one set

So far CDI supported only 1 claim propery set. We want to be able
to support more then one so in case the user provides to the
DV storage volumeMode without accessMode or vice versa cdi
will be able to fit to it the most appropriate match.
Added to rook ceph block a second default of filesystem
volume mode with RWO access mode, it will support archive
upload which has default of filesystem mode.

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* CR fix - change to one endpoint for the user

upload proxy will identify if the upload is archive
or not by looking at the content type annotation on
the pvc. If the content type is archive it will route
the uplaod to upload server to a new archive upload uri.

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Add storage profile and data volume controllers unit tests

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* CR fixes

* add default volume mode to archive content type
* upload server use data processor for archive upload
* tests for volume mode with archive content type
* tests for archive upload of compressed tar

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Adjust imports acording to new apis dir

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* CR small fixes

Signed-off-by: Shelly Kagan <skagan@redhat.com>
2021-11-03 20:11:47 +01:00
Michael Henriksen
aedaf513ec
Move apis to staging, push to containerized-data-importer-api (#1997)
* move apis to new staging area

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* add script to push to staging

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix lint check and api reference

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* push staging to api repo

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2021-10-28 13:40:24 +02:00
akalenyu
50c93e8b0e
Deploy alerts infra as part of our installation (#1979)
* Deploy alerts infra as part of our installation

Conditionally deploy the infrastructure that is needed to fire alerts for our users
when bad things are happening to CDI.

Testing with `KUBEVIRT_DEPLOY_PROMETHEUS=true`

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Watch and unit test all prometheus related resources

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* add gateway for changing monitoring namespace (rbac purposes)

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* refactor test to check for exact alert name and firing state

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Align pattern of ensuring prometheus resource exists for all

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Remove potential noisy event

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Extract duplicate code to function

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Dont use empty value for prometheus label due to open issue

https://github.com/prometheus-operator/prometheus-operator/issues/4325

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
2021-10-26 21:26:07 +02:00
Bartosz Rybacki
5d2eba8e13
Test the new handling of storage class name (#1958)
Added missing tests for change "Explicitly set the storage class name #1936".
Corrected the behavior when storage class is not provided and not available.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2021-09-29 03:20:17 +02:00
Bartosz Rybacki
a82a0a7a59
Correct the cloneStrategy on StorageProfile (#1951)
Moves the cloneStrategy up one level from claimPropertySet to
StorageProfile.spec and StorageProfile.status.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2021-09-22 18:06:18 +02:00
Arnon Gilboa
addf25b4f9
Support registry import using node docker cache (#1913)
* Support registry import using node docker cache

The new CRI (container runtime interface) importer pod is created with three containers and a shared emptyDir volume:
-Init container: copies static http server binary to empty dir
-Server container: container image container configured to run the http binary and serve up the image file in /data
-Client container: import.sh uses cdi-import to import from server container, and writes "done" file on emptydir
-Server container sees "done" file and exits

Thanks mhenriks for the PoC!

Done:
-added ImportMethod to DataVolumeSourceRegistry (DataVolume.Spec.Source.Registry, DataImportCron.Spec.Source.Registry).
Import method can be "skopeo" (default), or "cri" for container runtime interface based import
-added cdi-containerimage-server & import.sh to the cdi-importer container

ToDo:
-utests and func tests
-doc

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add tests, fix CR comments

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Use deployment docker prefix and tag in func tests

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add OpenShift ImageStreams import support

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add importer pod lookup annotation for image streams

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add pullMethod and imageStream doc

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2021-09-20 22:05:36 +02:00
Bartosz Rybacki
483b9733d5
Explicitly set that storage class name (#1936)
When the storage request is specified using the 'storage' API in
a DataVolume, CDI uses Storage Profiles to look up any missing
values that would be required to create the underlying PVC.
If the user omits the storage class parameter in their DV,
CDI assumes the cluster's default storage class will be used.

Since CDI calculates the storage class that should be used in
this case, it should explicitly set that storage class name
in the PVC it creates.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2021-09-17 13:17:31 +02:00
Michael Henriksen
87a13c2f29
Add long term token to pvcs when host assisted cloning cross namespaces (#1922)
* Add long term token (10 years) to pvcs when host assisted cloning between namespaces

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* clone controller should retry if source in use

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* minor refactor if/else -> switch

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2021-09-17 01:24:00 +02:00
Michael Henriksen
2889d68766
BugId: 1999571 - fix clone into larger capacity nfs volume (#1939)
* BugId: 1999571 - fix clone into larger capacity nfs volume

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix lint issues

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2021-09-15 20:22:35 +02:00
Bartosz Rybacki
a308404b07
Overhead on profile and usable space toghether (#1926)
* Correct the fsOverhead calculation in profile

Calculation needs play well with the actual resize that is done in data-processor

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Properly reverse the calculation for overhead.

Signed-off-by: Alexander Wels <awels@redhat.com>

Co-authored-by: Alexander Wels <awels@redhat.com>
2021-09-07 16:42:03 +02:00
Matthew Arnold
33b43a6767
Avoid trying to get metrics from non-running pod. (#1911)
* Avoid trying to get metrics from non-running pod.

If the importer pod is not running, the attempt to get metrics can hit a
30-second HTTP timeout. For multi-stage imports, this causes the
transition to "Paused" to take much longer than it needs to.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Return nil instead of new error.

Signed-off-by: Matthew Arnold <marnold@redhat.com>
2021-09-03 02:14:38 +02:00
Bartosz Rybacki
8f4be85e0c
Handle for CSI Clone not possible (#1902)
CSI clone is not possible on non-csi storage

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2021-08-25 18:57:20 +02:00
Alexander Wels
b27ff563d1
Always align size of disk image to 1Mi blocks. (#1873)
Signed-off-by: Alexander Wels <awels@redhat.com>
2021-08-05 17:16:43 +02:00
Bartosz Rybacki
c49f4c826a
CSI Volume clone (#1824)
* CSI Volume Clone for same namespace

CSI Volume Cloning is available on the same namespace and also
works with namespace transfer and volume expansion.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Update documentation for CSI Volume Clone

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Cleanup and refactor - extract common code into functions

Remove csi-clone-controller (only set cloneOf annotation)

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Corrects reconcile results

Do not requeue reconciliation loop when not needed.
Mark DV as Failed when the PVC Claim is lost.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Handles PVC recovery from ClaimLost

Make sure that CSI clone continues when target pvc recovers from
ClaimLost to to Bound or Pending.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Code Review improvements

Extracted common code for doCrossNamespaceClone and expandAfterClone, and some updates to comments/cleanups.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2021-08-05 14:01:48 +02:00
akalenyu
2254cf0c1f
Add relationship labels (#1864)
Users don't want 👽 resources in clusters,
and we should also be able to tell if were part of a broader installation.

Note:
- Operator created resources were handled in https://github.com/kubevirt/controller-lifecycle-operator-sdk/pull/18
as these labels will be common to all resources deployed by the HCO.
- Now that the controller is guaranteed to have the labels, we can set env vars
that reference the label values (fieldRef) to spare calling GET on the CR in the controllers.
(thanks mhenriks).

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
2021-07-28 20:05:24 +02:00
Alexander Wels
c1aad16882
Revert "Have the healthz server check for the uploadserver first. (#1661)" (#1865)
This reverts commit 2b476387a5.

Signed-off-by: Alexander Wels <awels@redhat.com>
2021-07-22 15:57:30 +02:00
Arnon Gilboa
f1fb79bba0
Add DataSource as an optional sourceRef for DataVolumes (#1851)
* Add DataSource as an optional sourceRef for DataVolumes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add some sourceRef validation unit and functional tests

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add cloner functional test for SourceRef PVC DataSource

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add test_ids

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Minor CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add DV mutate utest

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2021-07-14 16:56:34 +02:00
Bartosz Rybacki
2f21a36545
Preferred Clone strategy selection (#1846)
* Refactor: simplify by extracting methods

Prepare for new clone logic - extracted smartClone reconcile functions.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Select clone strategy based on storageProfile

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Changes from CR comments.

A series of small fixes, and cleanups.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Documentation update

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2021-07-03 01:53:14 +02:00
Alexander Wels
39f22603a7
Fix smartclone sometimes not triggering due to capicity not being reported. (#1844)
* Fix smartclone sometimes not triggering.
Updated tests to use a real image instead of data that is filled.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Refactor getSnapshotClass into two functions

Signed-off-by: Alexander Wels <awels@redhat.com>

* Use constant instead of magic number for size.

Signed-off-by: Alexander Wels <awels@redhat.com>

* force bind for WFFC storage on tests.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Updated based on comments.
Fixed failing functional test.

Signed-off-by: Alexander Wels <awels@redhat.com>
2021-06-25 19:31:02 +02:00
Michael Henriksen
a45795c562
reserve PV for target before transfer (#1835)
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2021-06-22 17:15:37 +02:00
Michael Henriksen
d92c2f459d
update deps and bazel (#1815)
* update deps and bazel

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix apidocs and unit tests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix generate-verify

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2021-06-08 01:31:59 +02:00
Bartosz Rybacki
bfb1bf76da
Fix bug - ignoring errors (#1810)
An error in the logic that updates DV, it would ignore errors
during the update. Also small typo in tests corrected.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2021-05-27 20:22:56 +02:00
Michael Henriksen
f1f4f26ee6
[BugFix] fix synchronization between smart clone and datavolume controller (#1801)
* fix synchronization between smart clone and datavolume controller

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* pvc transfer controller should be more aggressive to force binding

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2021-05-26 09:37:49 +02:00
Bartosz Rybacki
c0b54768a6
Fix: Compute fs overhead only for fs volumeMode (#1791)
* Fix: Compute fs overhead only for fs volumeMode

Correctly compute fs overhead for an effective VolumeMode. Effective, means one that is
 computed based on value in storage spec and the storageProfile.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Test: Add more tests for fs overhead

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2021-05-20 04:31:33 +02:00
Matthew Arnold
2960a3f6d3
Copy VDDK version to DV annotation. (#1752)
* Add an interface to watch nbdkit logs.

Useful for fishing out various pieces of information. Save VDDK library
version and connected ESX host by appending to the importer pod's
termination message. Turns nbdkit logging up to verbose for VDDK data
sources, so only the last few lines are printed for debugging.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Copy VDDK info from termination message to PVC/DV.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add unit tests for saved VDDK information.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add functional test for VDDK annotations.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Fix unit test, forgot to check for nil pvc.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Don't ignore errors updating PVC with VDDK info.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Watch nbdkit with Scanner instead of ReadString.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Move VDDK info test into existing functional test.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Make nbdkit stop sequence slightly clearer.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Save VDDK info in regular DV reconciler.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Don't save VDDK info when PVC is being deleted.

Also, piggyback off existing PVC update instead of introducing a new
error handling path.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Fix VDDK-info unit tests.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Use scanner for all nbdkit logging.

Also fix up a minor merge mistake.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Try to satisfy complaints from SonarCloud.

Signed-off-by: Matthew Arnold <marnold@redhat.com>
2021-05-08 21:06:18 +02:00
Alexander Wels
144f621395
Namespace transfer fixing nitpicks (#1770)
Signed-off-by: Alexander Wels <awels@redhat.com>
2021-05-07 18:18:21 +02:00
Michael Henriksen
74a2c86608
use namespace transfer for smart clone (#1763)
* use namespace transfer for smart clone

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* updates from test failures

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* add expansion func tests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* add dv phases for expansion and transfer

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* rebase and integrate with storage profiles

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2021-04-30 15:18:43 +02:00
Bartosz Rybacki
f81ab950fb
Use storage profiles when handling DataVolumes (#1753)
* Create new Storage type

A new Storage type similar to the PVC Spec is now available to use
in the DataVolume Spec. This is more permissive than PVC, and together
with StorageProfile this allows CDI to apply additional logic for
missing or optional fields.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Use the StorageProfile

Handle the StorageProfile recommended params when creating the PVC for
a DataVolume. When parameters like volumeMode or accessModes are
not provided, CDI checks the StorageProfile for a given StorageClass
to set the recommended defaults. This enables user to create DataVolume
without the need to provide all the parameters.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Allow multiple accessModes

CDI allows multiple access modes to be specified in the DataVolume.spec.storage and in the StorageProfile. This now works the same way as in PVC specification.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Handle the storage.size field

The storage.size specifies how much space a user wants to have.
When creating image on the fileSystem storage CDI takes into
account the file system overhead and requests PVC big enough to
fit an image and file system metadata.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Test storage profile with DV

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Document Storage Profiles

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Refactor: Render the effective PVC early

The helper 'render PVC' was moved earlier in the control flow, so
it can be used in more places. Removing the need for if/else logic.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Test handling size on import, upload and clone

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Code Review: Refactor resolving of volumeMode

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Fix: render target pvc spec correctly in smart clone controller

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2021-04-29 13:10:24 -05:00
Vishesh Tanksale
2014ddecfd
Adding priority class name for importer/upload pod to data volume object (#1740)
* Adding priority class name for importer/upload pod to data volume object

Signed-off-by: Vishesh Ajay Tanksale <vtanksale@apple.com>

* Addressing review comments

Signed-off-by: Vishesh Ajay Tanksale <vtanksale@apple.com>

* Adding controller logic to assign priority class on importer,cloner and uploader pod

Signed-off-by: Vishesh Ajay Tanksale <vtanksale@apple.com>

* Adding functional test

Signed-off-by: Vishesh Ajay Tanksale <vtanksale@apple.com>

* Addressing review comments

Signed-off-by: Vishesh Ajay Tanksale <vtanksale@apple.com>

* Updating Data Volume doc

Signed-off-by: Vishesh Ajay Tanksale <vtanksale@apple.com>

Co-authored-by: Vishesh Ajay Tanksale <vtanksale@apple.com>
2021-04-28 09:38:42 -05:00
Michael Henriksen
3447bb84c7
Cluster scoped DataVolume/PVC namespace transfer API (#1673)
* Cluster-scoped namespace transfer api and controller

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* unit tests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* ObjectTransfer webhook

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* new functests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* experiment with termination grace period

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* quota test

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2021-02-24 20:45:24 +01:00