Commit Graph

117 Commits

Author SHA1 Message Date
Alexander Wels
a12446110b
Ensure scratch space is large enough to hold disk image (#3384)
* Ensure scratch space is large enough to hold disk image

It is possible the source image is fully allocated
and in that case the scratch space would not be large
enough to hold the disk image. This takes fsOverhead
into account when calculating the scratch space size.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Fix bug when copying the size of the pvc for the scratchspace
This was not copying but re-using the same struct. This caused the
size of the regular PVC to be overwritten by the scratchspace PVC
in certain conditions.

Use storage stanza instead of pvc stanza in the cloner tests.

Signed-off-by: Alexander Wels <awels@redhat.com>

---------

Signed-off-by: Alexander Wels <awels@redhat.com>
2024-09-03 07:38:09 +02:00
Shelly Kagan
8dab1078c7
Remove datavolume clone source validation (#3331)
* Remove sourceref validation

The source can be changed and updated, we want the error to be
presented by an event and have the datavolume pending instead
of preventing the creation of the datavolume.
We are already handling the option of the source not existing yet
so lets also handle all the validation in the controller.

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Move pvc clone specific functions to pvc-clone-controller

Now that the validation is no longer done in the webhook
no need for that code to be in the controller common util
file. Moved the UT accordingly.

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Move function to new Describe node

Will add in future commits other tests to that section

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* pvc-clone-controller: make better event and update dv if validation fails

In order to make it more clear that the validation failed
we should add the reason of validation failure to the event
Added an update watch to the clone controller so if something changes
in the source the clone will reconcile. Hence no need to return an
error, the reconcile will be triggered if someting will change in the
source.

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* snap-clone: always validate snapshot source

This commit adds missing validation due to the remove validation
from the dv webhook, but also fixes an existing missing validation
in cases where the clone was created before the source. In such case
the webhook would not validate the source since it didnt exist yet,
and then if the clone happened with populators then we would not
validate the source size at all.
We should validate the snapshot source before continuing with the clone
whether it is populator/legacy clones.
Moved all the validation to one function.
Updated the event to include the reason for the validation failure and
updated dv status accordingly.

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Cloner_test: remove clone to small size test

The test checks the rejection of the webhook
which no longer exists then clone to appropriate
size dv which should succeed which is done in
any other clone test.
Im not adding a test to check the reconcile loop
since it is covered in the UT.

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* tests/datasource: remove wait for pvc

Now since we added the validation also in the populators
path we are not creating the pvc until the source exists.

Signed-off-by: Shelly Kagan <skagan@redhat.com>

---------

Signed-off-by: Shelly Kagan <skagan@redhat.com>
2024-07-26 06:25:09 +02:00
Michael Henriksen
5195176c16
update to k8s 1.30 libs and controller-runtime 0.18.4 (#3336)
* make deps-update

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* ReourceRequirements -> VolumeResourceRequirements

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix calls to controller.Watch()

controller-runtime changed the API!

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* Fix errors with actual openshift/library-go lib

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* make all works now and everything compiles

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix "make update-codegen" because generate_groups.sh deprecated

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* run "make generate"

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix transfer unittest because of change to controller-runtime

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

---------

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2024-07-14 20:12:50 +02:00
Michael Henriksen
073336b16c
Make upload client/server certs configurable (#3252)
* Add client cert config to CDI resource

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* make client certs configurable

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* Create uploadserver.Config

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* uploadserver should read certs from files

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* make sure to not close doneChan when error occurs

generally tighten up handling of "done" "uploading" and "processing"

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* add deadline support to uploadserver

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* Add deadline support to upload controller

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* clone controller should use configured client cert duration

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* make lint check happy

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* Extend existing func test to validate client certs configurable and will be rotated

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* Use deadline/rotation for clone pods as well

Forgot about the case where a source PVC may be in use.  Bay be a big delay from when target pod is created and source.

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

---------

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2024-06-03 22:39:34 +02:00
Edu Gómez Escandell
0e750262a3
Enable autoformatting linters (#3179)
* Enable gofmt linter

From the docs:

> Gofmt checks whether code was gofmt-ed. By default this tool runs with
> -s option to check for code simplification.

https://golangci-lint.run/usage/linters/#gofmt

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

* Run gomft on the project

Ran this command after adding the gofmt linter:

	golangci-lint run ./... --fix

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

* Enable whitespace linter

From the docs:
> Whitespace is a linter that checks for unnecessary newlines at the
> start and end of functions, if, for, etc.

https://golangci-lint.run/usage/linters/#whitespace

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

* Run whitespace on the project

Ran this command after adding the whitespace linter:

	golangci-lint run ./... --fix

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

* Enable GCI linter

Per the docs:

> Gci controls Go package import order and makes it always deterministic.

https://golangci-lint.run/usage/linters/#gci

NOTE: I noticed that many files separate their imports in a particular
way, so I set the linter to enforce this standard.

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

* Run GCI on the project

Ran this command after adding the GCI linter:

	golangci-lint run ./... --fix

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

---------

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
2024-04-24 13:52:22 +02:00
Nahshon Unna Tsameret
475a70c8cd
Add new DV reason: ImagePullFailed (#3194)
When the image pull fails, the DataVolume Running condition, will have
the Reason field of `ImagePullFailed`, to allow better error handling in
code taht uses it.

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
2024-04-18 19:35:59 +02:00
Felix Matouschek
3239d50b4e
cleanup: Return modified labels added from termination message (#3147)
Instead of changing the labels map that is passed in as parameter to the
setLabelsFromTerminationMessage function, it now returns a modified copy
of the passed in labels map and is renamed to
addLabelsFromTerminationMessage. If the passed in map is nil a new map
will be allocated and initialized.

Signed-off-by: Felix Matouschek <fmatouschek@redhat.com>
2024-03-27 07:33:28 +01:00
Felix Matouschek
ca8085a25c
feat(import): Set PVC labels derived from env vars on containerdisks (#3103)
* feat(cdi-containerimage-server): Add info endpoint

The info endpoint returns a ServerInfo object containing all
environment variables of the server serialized to json. This allows the
extraction of env vars from a containerdisk when using pullMethod node.

Signed-off-by: Felix Matouschek <fmatouschek@redhat.com>

* feat(importer): Add conversion of env vars to label

This adds the conversion of env vars containing KUBEVIRT_IO_ to a label
key/value pair.

Example: TEST_KUBEVIRT_IO_TEST=testvalue becomes test.kubevirt.io/test:
testvalue.

Signed-off-by: Felix Matouschek <fmatouschek@redhat.com>

* feat(importer): Extract labels from registry datasource

This allows the registry-datasource to return a termination message with
labels extracted from the env vars of a source containerdisk when using
pullMethod pod.

Signed-off-by: Felix Matouschek <fmatouschek@redhat.com>

* feat(importer): Extract labels from http datasource

This allows the http-datasource to return a termination message with
labels extracted from the env vars of a source containerdisk when using
pullMethod node.

Signed-off-by: Felix Matouschek <fmatouschek@redhat.com>

* feat(controller): Set PVC labels from importer termination message

With this change the import-controller is able set labels on destination
PVCs returned from the importer in its termination message.

Signed-off-by: Felix Matouschek <fmatouschek@redhat.com>

* tests: Add tests for conversion of containerdisk env vars to PVC labels

This adds tests for the conversion of containerdisk env vars to PVC
labels for both pullMethods pod and node.

Signed-off-by: Felix Matouschek <fmatouschek@redhat.com>

* fix: Fix race in import-populator

By running reconcileTargetPVC of populatorController on every reconcile
cycle, the import-populator controller is able to retry seting labels and
annotations on the target PVC when import-controller modified the target
PVC at the same time.

Signed-off-by: Felix Matouschek <fmatouschek@redhat.com>

---------

Signed-off-by: Felix Matouschek <fmatouschek@redhat.com>
2024-03-21 23:13:16 +01:00
Felix Matouschek
cd7ee9e9b3
feat: Make importer datasource communication explicit (#3101)
Make the communication of datasources in the importer explicit by adding
a GetTerminationMessage method to the DataSourceInterface.
Then use this method to communicate additional information to the import
controller once the importer pod has terminated, instead of writing
additional data to the termination message in the Close method of
datasources.

Signed-off-by: Felix Matouschek <fmatouschek@redhat.com>
2024-03-13 22:09:23 +01:00
alromeros
aff18f09e4
Address error condition when scratch space is needed to avoid VM failures (#2945)
When scratch space is needed during imports/uploads, the pod typically fails so it can be restarted with the required space. This is expected behavior that's automatically handled by CDI.

However, since we use an error exit code, this behavior leads to unwanted VM statuses during provisioning in Kubevirt.

This commit addresses this behavior by using a different DV condition, so kubevirt ignores it and the VM can be provisioned as expected.

Signed-off-by: Alvaro Romero <alromero@redhat.com>
2023-10-31 03:26:24 +01:00
Maya Rashish
5e4cb68044
Update to ginkgo v2 (#2788)
* Run `make deps-update`

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Update to ginkgo v2

Avoid using table extension to avoid compilation errors
Switch to v2 everywhere
Update qe-tools as well (required)

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Fix/avoid deprecation warnings

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Do not use v1 reporter

For unit tests: stop using custom reporter, unnecessary
For functional tests: borrow code from kubevirt to keep reporting

Avoid deprecated warnings by golangci for using deprecated reporter

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Increase ginkgo timeout to 24h (default in ginkgo v1)

this may seem excessive, but we have a lower timeout in Prow, let's save
ourselves the future trouble of bumping timeouts in two places.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* use the ginkgo built-in junit reporter

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Avoid using deprecated --ginkgo.noColor, use --ginkgo.no-color instead

Signed-off-by: Maya Rashish <mrashish@redhat.com>

---------

Signed-off-by: Maya Rashish <mrashish@redhat.com>
2023-07-26 02:35:55 +02:00
Michael Henriksen
f88fab69dc
PVC Clone Populator (#2709)
* touch up zero restoresize snapshot

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* clone populator

only supports PVC source now

snapshot coming soon

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* more unit tests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* unit test for clone populator

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* func tests for clone populator

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* move clone populator cleanup function to planner

other review comments

verifier pod should bount readonly

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* add readonly flag to test executor pods

synchronize get hash calls

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* increase linter timeout

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* better/explicit readonly support for test pods

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* check pv for driver info before looking up storageclass as it may not exist

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* addressed review comments

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* chooseStrategy shoud generate more events

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

---------

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2023-05-24 05:11:52 +02:00
alromeros
c5f767d910
Import populator (#2690)
* Create populators package to be used for all populators

This commit introduces the basic reconciler for
populators with common function that can be used
by the different populators.

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* unite getcontenttype func across code

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Add VolumeImportSource CRD for import populator

This commit adds the VolumeImportSource CRD into CDI.

CRs created from this CRD will be referenced in the dataSourceRef field to populate PVCs with the import populator.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Refactor common populator code to be shared among all populators

This commit introduces and modifies several functions so we can reuse common code between all populators.

Other than having a common reconcile function, a new populatorController interface has been introduced so we are able to call populator-specific methods from the populator-base reconciler.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Create Import Populator

The import populator is a controller that handles the import of data in PVCs without the need of DataVolumes while still taking advantage of the import-controller flow.

This controller creates an additional PVC' with import annotations. After the import process succeeds, the controller rebinds the PV to the original target PVc and deletes the PVC prime.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Add functional tests to cover the import populator flow

This commit updates the import tests to cover the new import populator flow.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Add unit tests for import populator

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Minor fixes and enhancements in import/common populator code

* Modify indexes and other related code to support namespaced dataSourceRefs. Cross-namespace population is still not supported as it depends on alpha feature gates.
* Add functional test to cover static binding.
* Fix selected node annotation bug in scratch space PVCs
* Fix linter alerts

Signed-off-by: Alvaro Romero <alromero@redhat.com>

---------

Signed-off-by: Shelly Kagan <skagan@redhat.com>
Signed-off-by: Alvaro Romero <alromero@redhat.com>
Co-authored-by: Shelly Kagan <skagan@redhat.com>
2023-04-28 00:10:59 +02:00
Nahshon Unna Tsameret
34a98256a2
Start integrate golangci lint into CI (#2675)
* Start adding the golangci-lint to CI

golangci-lint is a collection of many linters. This PR adds
golangci-lint to the CI. For strat, it enables the govet linter, and fix
its single finding.

The PR adds this linter to the `test-lint` Makefile target.

The new .golangci.yml file is the configuration for the linter.

golangci-lint version was set to the latest one - v1.52.2.
It is defined in hack/build/run-linters.sh

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

* golangci-lint: enable gosimple and fix findings

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

* golangci-lint: enable unused and fix findings

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

---------

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
2023-04-11 13:59:33 +01:00
Arnon Gilboa
bfe30a8964
Split and refactor DV controller (#2483)
- Split the huge DV controller into smaller op-specific DV controllers -
  import, clone, upload
- Add common watch-adding function so each controller watches only its
  relevant DVs
- Refactor the common Reconcile() to use interface DataVolumeReconciler
  implemented by each controller
- Move all functions, structs, consts to the relevant controller
- Split the utests per controller

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2022-12-22 01:03:15 +00:00
akalenyu
f4978d920b
Bump volumesnapshot client to v6 (#2513)
* make deps-update on clean repo

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Bump volumesnapshot client to v6

In case we want to utilize https://kubernetes.io/docs/concepts/storage/volume-snapshots/#convert-volume-mode
new API field sourceVolumeMode.

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
2022-12-20 20:07:16 +00:00
Michael Henriksen
46c6aa994a
Support restricted PSA for worker pods (#2410)
* remove root worker pods

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* remove selinux requirement for worker pods

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* run tests in restricted namespace and required changes

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* handle empty tar

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* add PSA label when running functional tests in OpenShift

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* cannot use restricted PSA with istio (for now)

refactor scc management

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix clean script

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2022-09-14 21:16:23 +01:00
alromeros
efe5109ba5
Improve error handling when pod creation fails (#2335)
* Improve the error handling when pod creation fails

When pod creation fails, the error is usually logged without providing additional information to the user. This behavior is especially risky when the user lacks the permits to check the logs, making it unintuitive and almost impossible to find the source of the problem.

This commit improves the error handling of the pod-creation process, so pertinent info about the failure is included in the pod's PVC.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Update functional tests to check for events when pod-creation fails

Since error handling in pod-creation has been improved in our controllers, this commit introduces several changes in the corresponding functional tests to properly cover the new behavior included when pod-creation fails.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Update unit-tests after improving error-handling of pods for proper coverage

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Minor fixes and improvements on error handling for pods

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Modify datavolume-controller to change the running condition of datavolumes when a pod fails

Until this commit, the way of handling pod errors in the datavolume-controller has been to change the affected datavolume's phase to failed, which conflicts with the declarative approach of the controllers.

This commit modifies this behavior so that, when a pod fails, the affected datavolume's running condition is changed to false while the phase remains unchanged.

Signed-off-by: Alvaro Romero <alromero@redhat.com>
2022-07-15 10:28:53 +02:00
alromeros
5516641864
Allow creating clones without source PVC (#2306)
* Modify datavolume admission webhook to enable creating clones without source PVC

This commit modifies the datavolume admission webhook to follow a more descriptive approach, enabling the creation of clones without a source PVC.

This clone will later be handled by the datavolume-controller until the source PVC is created.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Modify the datavolume-controller to improve the handling of clones without source

Since we are allowing the creation of clones without source PVC in the admission webhook, we need to improve the handling of these clones once in the datavolume-controller.

This commit modifies said controller, so we do proper error handling and validation until the source PVC is created.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Add unit tests to check the creation of clones without source PVC

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Include a mechanism to reconcile clones without source once the source PVC is created

This commit introduces a new datavolume-controller watch so, if a clone without source is created, we make sure to reconcile it once a proper PVC is created.

It also updates/includes unit tests for proper coverage.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Include functional tests to cover the creation of clones without source PVC

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Minor refactoring of clone-related code in DataVolume reconciler to improve readability

After enabling the creation of clones without source PVC in the datavolume controller, the clone-related logic outside its reconciler has increased in size and become sparse.

This commit rearranges all this code and condenses it into the clone reconciler, without changing the loop behavior.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Modify datavolume-mutate webhook to reject the creation of clones if the source PVC's namespace doesn't exist

In previous commits, a mechanism to allow the creation of clones without source PVC was added, without ever checking if the source PVC's namespace exists or not.

This behavior could lead to permission conflicts between the user and the source's namespace since the webhook skipped all the related validation.

This commit modifies the datavolume-mutate admission webhook to reject the clone if the source PVC's namespace doesn't exist.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Update unit tests for proper coverage of clone-validation functions

This commit adds and updates several unit tests to improve the coverage of the clone-validation mechanism after several functions were moved to the controller.

It also introduces minor changes on related code in the datavolume-controller and functional tests following PR review.

Signed-off-by: Alvaro Romero <alromero@redhat.com>
2022-06-21 18:05:38 +02:00
alromeros
7eebbfaf1c
Make size optional when cloning using the Storage API (#2222)
* Allow empty DV size when cloning using storage API

When cloning a Data Volume, the size of the target can be potentially obtainable via the source PVC, which discards the need to explicitly specify it.

Considering that, this commit introduces a change in the correspondent validation webhook to allow omitting the resources.request.storage field when cloning a PVC using the storage API.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Modify datavolume-controller to allow obtaining storage size from source PVC when cloning

When cloning a PVC, if the target's size is not specified, said value can be attainable from the source PVC.

This commit introduces a change in datavolume controller so, in case of detecting an empty storage size, said value can be obtained when performing CSI and Smart cloning.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Update unit tests for datavolume-validation after enabling cloning with empty size

This commit updates the unit testing for the datavolume validation webhook, covering the possibility of cloning a PVC without setting any storage size.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Update unit testing for controller-related functions after enabling cloning with empty size

This commit includes unit tests for the volumeSize() function after enabling creating clones with blank size.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Update the datavolume controller to create a size-detection pod when performing host-assisted clone

When performing a host-assisted clone with empty clone size, simply copying the original PVC size could lead to potential overhead miscalculations if the source's VolumeMode is "filesystem".

When that's the case, an inspection pod will be created in the datavolume controller so it extracts the size of the virtual image using qemu-img.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Include an image-size detection tool to allow cloning with empty DV size

This commit introduces a new tool in charge of collecting the virtual image size when cloning with an empty DV size. In some cases where said value is unattainable from the original PVC's spec, the datavolume controller will create a new pod containing this new tool.

The binary will then run the 'qemu-img' command and handle its results appropriately.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Optimize the clone-size lookup process to avoid creating unnecessary size-detection pods

When performing host-assisted clone with an empty DV size, in some cases, a size-detection pod is used to obtain the required capacity.

This commit tries to optimize this process to keep the collected value as a PVC annotation, that is checked in subsequent clones to avoid creating redundant pods.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Minor fixes and improvements on mechanism for cloning with empty storage size
* Add new optional flag on size-detection binary to enable using a different URI scheme
* Improve the pod-creation mechanism so the pod is not created until the source PVC has finished the import
* Modify size-finlation mechanism to account for possible round-downs when importing the source image
* Improve the size inflation mechanism so only PVCs with filesystem as volume mode are considered
* Minor style corrections

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Modify the clone-controller to allow skipping the clone size validation in some cases

Due to filesystem overhead differences, the target's size can sometimes be smaller than the source's one when obtaining said value with the size-detection pod.

This commit introduces minor changes in the clone-controller so we can skip the size validation in those cases.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Minor changes and improvements in size-detection mechanism following PR review
* Added new UT that covers using empty storage API for non-cloning sources
* Added new watch on datavolume-controller that looks for changes in the size-detection pod
* Removed redundant and unnecessary specs on size-detection pod
* Added error handling when reading the pod's termination message
* Moved general-usage functions to 'util.go' file
* Updated 'datavolumes' documentation to reference the possibility of omitting the storage size when cloning
* Minor style corrections

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Add unit tests that cover the size-detection mechanism in the DataVolume controller

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Include functional tests for cloning without specifying storage size

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Improve error handling in the creation/deletion process of the size-detection pod

This commit introduces additional handling in case of error after and during the size-detection pod is created.

It also updates several related unit tests.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Minor fixes to improve fsOverhead calculations when cloning with empty storage size
* Modified the size-detection mechanism so we account for fsOverhead when cloning to filesystem volume mode in all cases
* Clean up the code for reconciling when cloning a PVC that is not ready
* Minor fix in functional test so it works when cloning from block to filesystem volume mode

Signed-off-by: Alvaro Romero <alromero@redhat.com>
2022-05-31 17:03:45 +03:00
Maya Rashish
c05a750a9a
Detect storage capabilities for no-provisioner storage classes (#2226)
* Detect storage capabilities for no-provisioner storage classes

Assume there's a persistent volume that we can look up to infer the
correct values for volume mode and access modes.

Limit ourselves to detecting no-provisioner capabilities on LSO to
avoid greatly increasing the number of storage classes we provide
capabilities for. This is similar to our current flow where we
only provide capabilities for known storage classes.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Regenerate bazel stuff for pkg/monitoring's existence

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Add a watcher for no-provisioner PVs

We maintain a map of storage class names and provisioners whenever
storage classes are changed.

If a PV has one of the storage classes with no-provisioner as a
provisioner, reconcile that storage class.

This is because we infer the storage profile based on PVs, and
new ones might have different storage capabilities.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Use a client to do our storage class caching

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Pass a client as an argument, not global.

Suggested by awels, thanks!

Signed-off-by: Maya Rashish <mrashish@redhat.com>
2022-05-12 14:00:02 +02:00
akalenyu
71522a1f2d
Switch VolumeSnapshot to v1 (#2235)
* Switch VolumeSnapshot to v1

VolumeSnapshot v1beta is being deprecated:
https://kubernetes.io/blog/2022/04/07/upcoming-changes-in-kubernetes-1-24/#api-removals-deprecations-and-other-changes-for-kubernetes-1-24

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Fix unit tests; change version we look for in IsCsiCrdsDeployed

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
2022-04-12 18:50:19 +02:00
Roman Mohr
b9c0684469
Separate sdk api (#2208)
* Introduce controller-runtime-sdk api package

Split controller-runtime-sdk into the base package and
controller-runtime-sdk/api.

Signed-off-by: Roman Mohr <rmohr@redhat.com>

* go mod vendor

Signed-off-by: Roman Mohr <rmohr@redhat.com>

* Update code references

Signed-off-by: Roman Mohr <rmohr@redhat.com>
2022-03-31 21:31:18 +02:00
Alexander Wels
013cb6b62b
Set http(s)_proxy to lower case env variable (#2132)
* Set htpp(s)_proxy to lower case env variable

CURL used by nbdkit doesn't read upper case http(s)_proxy environment
variables, and thus was not using the proxy. Changed the variable to
be lower case.

Added a significant number of tests to test many more variations of
using a proxy. Also added https + auth endpoint to the file-host
container, so we can test https + auth with the proxy.

Added https endpoint to proxy, so we can test an https proxy.

Cleaned up some of the error handling in the import controller for
the proxy, in particular if a trustedCAProxy is defined.

Fixed some of the cluster wide proxy configuration so it works properly
inside an openshift cluster.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Add https proxy support to registry import. Added extra
functional tests to test all registry import combinations

Signed-off-by: Alexander Wels <awels@redhat.com>

* Fixed some tests to work better in Open Shift.

Signed-off-by: Alexander Wels <awels@redhat.com>
2022-02-03 18:09:41 +01:00
Shelly Kagan
e7dd62eb26
Upload archive (#1969)
* Add support for archive upload

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* fix golang errors

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Change storage profile property set to support more then one set

So far CDI supported only 1 claim propery set. We want to be able
to support more then one so in case the user provides to the
DV storage volumeMode without accessMode or vice versa cdi
will be able to fit to it the most appropriate match.
Added to rook ceph block a second default of filesystem
volume mode with RWO access mode, it will support archive
upload which has default of filesystem mode.

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* CR fix - change to one endpoint for the user

upload proxy will identify if the upload is archive
or not by looking at the content type annotation on
the pvc. If the content type is archive it will route
the uplaod to upload server to a new archive upload uri.

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Add storage profile and data volume controllers unit tests

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* CR fixes

* add default volume mode to archive content type
* upload server use data processor for archive upload
* tests for volume mode with archive content type
* tests for archive upload of compressed tar

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Adjust imports acording to new apis dir

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* CR small fixes

Signed-off-by: Shelly Kagan <skagan@redhat.com>
2021-11-03 20:11:47 +01:00
Michael Henriksen
aedaf513ec
Move apis to staging, push to containerized-data-importer-api (#1997)
* move apis to new staging area

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* add script to push to staging

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix lint check and api reference

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* push staging to api repo

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2021-10-28 13:40:24 +02:00
Bartosz Rybacki
a82a0a7a59
Correct the cloneStrategy on StorageProfile (#1951)
Moves the cloneStrategy up one level from claimPropertySet to
StorageProfile.spec and StorageProfile.status.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2021-09-22 18:06:18 +02:00
Bartosz Rybacki
2f21a36545
Preferred Clone strategy selection (#1846)
* Refactor: simplify by extracting methods

Prepare for new clone logic - extracted smartClone reconcile functions.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Select clone strategy based on storageProfile

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Changes from CR comments.

A series of small fixes, and cleanups.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Documentation update

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2021-07-03 01:53:14 +02:00
Maya Rashish
19d109837d
Add a datavolume condition for the image being too large (#1818)
* Strip newlines when writing a termination message.

Otherwise it isn't visible, at least when viewing in the -o yaml view.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Write down the nbdkit output and add it to the error output

With the added output from nbdkit, we can see the reason for the
non-existence of the nbdkit socket.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Don't set that we're waiting if it's CrashLoopBackOff

It's better to have the reason for the crash (terminate message)
than "backing off 5 minutes"

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Simplify all "image too large to fit" terminate reasons.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Move verifyConditions to utils, no functional change

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Switch test for image too large to test condition and not log

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Remove unused branch

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Rename setConditionFromPodWithPrefix to setAnnotationsFromPodWithPrefix

No functional change. Intended to be followed by some refactoring.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Fold restart count logic into the common setAnnotationsFromPodWithPrefix

Changing to >= rather than > to ensure a zero pod restart count is
always used -- the import controller unit tests request this.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Reduce indentation by returning right away.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Fold check for pod being nil into common code.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* move saveVddkAnnotations into util and make it unconditional

Call it setVddkAnnotations for consistency.
Check for not-terminated inside the function, not outside.

Removes check for source being VDDK (to avoid passing more arguments):
it won't match the regex anyway.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Reduce indentation by bailing on failure.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Reorder parameters to mirror the order in the function name

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Use a named variable for first container state

Yields shorter, more legible lines.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Use a constant for the nbdkit log.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Add more information to function description - also logs to file

Signed-off-by: Maya Rashish <mrashish@redhat.com>
2021-06-29 12:47:05 +02:00
Alexander Wels
5fedb458e5
Set some reasonable requests/limits for workloads (#1842)
* Set some reasonable requests/limits for workloads

Signed-off-by: Alexander Wels <awels@redhat.com>

* Fix unit tests

Signed-off-by: Alexander Wels <awels@redhat.com>
2021-06-28 20:34:01 +02:00
Arnon Gilboa
13275ce351
OS image poll and update API (#1808)
* Add CRD for DataSource definition

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add optional sourceRef to DataSource in DataVolumeSpec

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add CRD for DataImportCron definition

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add DataSource and DataImportCron generated files

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Code review fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* More code review fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Code genrated after rebase

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Fix DV source reference in utests

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Remove api validation tests for missing data volume source

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add standard fields to condition structs

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2021-06-14 13:58:42 +02:00
Michael Henriksen
74a2c86608
use namespace transfer for smart clone (#1763)
* use namespace transfer for smart clone

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* updates from test failures

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* add expansion func tests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* add dv phases for expansion and transfer

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* rebase and integrate with storage profiles

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2021-04-30 15:18:43 +02:00
Bartosz Rybacki
f81ab950fb
Use storage profiles when handling DataVolumes (#1753)
* Create new Storage type

A new Storage type similar to the PVC Spec is now available to use
in the DataVolume Spec. This is more permissive than PVC, and together
with StorageProfile this allows CDI to apply additional logic for
missing or optional fields.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Use the StorageProfile

Handle the StorageProfile recommended params when creating the PVC for
a DataVolume. When parameters like volumeMode or accessModes are
not provided, CDI checks the StorageProfile for a given StorageClass
to set the recommended defaults. This enables user to create DataVolume
without the need to provide all the parameters.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Allow multiple accessModes

CDI allows multiple access modes to be specified in the DataVolume.spec.storage and in the StorageProfile. This now works the same way as in PVC specification.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Handle the storage.size field

The storage.size specifies how much space a user wants to have.
When creating image on the fileSystem storage CDI takes into
account the file system overhead and requests PVC big enough to
fit an image and file system metadata.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Test storage profile with DV

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Document Storage Profiles

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Refactor: Render the effective PVC early

The helper 'render PVC' was moved earlier in the control flow, so
it can be used in more places. Removing the need for if/else logic.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Test handling size on import, upload and clone

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Code Review: Refactor resolving of volumeMode

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Fix: render target pvc spec correctly in smart clone controller

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2021-04-29 13:10:24 -05:00
Tomasz Barański
84f51f32f6
Preallocate cloning DV (#1719)
* Refactoring - move PreallocationApplied flag definition to common

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* Preallocate cloning DataVolumes

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
2021-04-27 15:06:30 +02:00
Marcelo Carneiro do Amaral
8d1721db0a
Add support for proxy in the CDI import pod, reconciling the info from OpenShift cluster wide proxy when available (#1507)
* api: Add new ImportProxy structure to CDIConfig in v1beta1 core api

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* common: Add constants related to configure Import proxy in Import pod and controller reconcile

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* config controller: Add import proxy reconcile, monitoring OCP proxy updates

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* config controller: Add unit test for import proxy reconcile, monitoring OCP proxy updates and creating Secret and ConfigMap

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* import controller: Add unit support of ImportProxy in the tests

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* controller util: Add support to retrieve an OpenShift cluster wide proxy object as well as its needed configurations

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* controller util:  Add unit test for the get cluster wide proxy functionality

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* controller: moved isOpenshift function to utils common and export it

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* importer: Clone http transport to keep the default proxy config such as the usage of environment variables and add support for proxy CA

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* operator: Update CDIConfigCRD with import proxy info

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* operator: Update CDIListCRD with import proxy info

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* operator: Add service account rights to CDI for accessing OpenShift proxy obj

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* operator: Add OpenShift proxy obj scheme support

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* test: Add new forwarding proxy for testing things that require a proxy

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* test: Add test to verify the importer pod with proxy

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* doc: Update cdi config doc with the ImportProxy info

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* add support to build the new proxy docker image

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* test: remove unwanted F parameter from test

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* controller: fix error handling

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* test: update the used method to get the pod of the importer pod and proxy

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* fixed comments from the revision

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* controller config: update the methods to use the log object from the reconcile object

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* controller: update function GetImportProxyConfig to return errors and include a new unit test

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* controller: add back a test that was removed by mistake

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* update updateCDIConfigByUpdatingTheClusterWideProxy function

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* test: decrease the import pod log pooling interval and increase the image size to be able to verify the import pod before it is deleted

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* test: update the pvc and proxied requests verification

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* tools: add X-Forwarded-For header in the proxy requests  used for testing

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* test: reset cluster wide proxy with the original values

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* test: fix proxy update to change spec instead of status and other minor updates

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* doc: update import proxy description

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* update generated files

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
2021-02-16 13:43:01 +01:00
Nahshon Unna Tsameret
93627f4fe8
Stop Using Deprecated Packages (#1548)
* sigs.k8s.io/controller-runtime/pkg/runtime/* packages are deprecated, and were moved to new paths.

Trying to upgrade sigs.k8s.io/controller-runtime to version v0.7.0 in HCO created a conflict because in v0.7.0 the deprecated packages were removed and cannot be used.

This PR replaces the deprecated packages with their new paths.

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

* Run `make deps-update`

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

* fix logger init

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

* fix test loggers

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
2020-12-24 07:08:50 +01:00
Tomasz Barański
91a15c57d1
Preallocation support (#1498)
* [WIP] doc: User-facing doc for preallocation support

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* apis: CDI accepts `preallocation` option.

With this commit CDI accepts (but does handle) `preallocation` settings
for DataVolumes and in CDIConfig.

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* core: Implementing preallocation

This commit implements preallocation support for import and upload.

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* test: Functional tests for preallocation support

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* core: Remove "preallocation for StorageClasses" config

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* test: Removed unused function

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* test: Fix rook-ceph test failures

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* Updated dependencies
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* core: Uss PVC annotation to pass preallocation parameters

DataVolume controller now uses a PVC annotation to pass preallocation
configuration to import and update controllers.

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
2020-12-18 16:46:16 -05:00
Michael Henriksen
86c32e87b8
Add CDIConfig to CDI (#1475)
* Add CDIConfigSpec to CDI

Make CDIConfig singleton mirror data in "active" CDI

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix functional tests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2020-11-11 23:09:54 +01:00
Jakub Dzon
7f368900de
Updated controller-lifecycle-operator-sdk dependency (#1389)
Signed-off-by: Jakub Dzon <jdzon@redhat.com>
2020-09-24 14:39:29 +02:00
Alexander Wels
d2da27b41b
Ensure we can specify different CDI CR name. (#1383)
Fix bug where the CR name was hard coded to CDI and it failed in HCO.

Signed-off-by: Alexander Wels <awels@redhat.com>
2020-09-22 00:13:28 +02:00
Alexander Wels
6dce12f090
Move CRDS from apiextensions v1beta1 to v1. (#1307)
* Move CRDS from apiextensions v1beta1 to v1.
Ensure that our code based schema validation matches the types in the api.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Ran go mod tidy and vendor in attempt to see if we could use newer runtime controller, but our go version too old.
Addressed review comments.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Addressed more review comments and fixed k8s-1.18 functional test failing.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Remove categories 'all' from cluster scoped CRDs

Signed-off-by: Alexander Wels <awels@redhat.com>
2020-08-01 01:01:50 +02:00
Bartosz Rybacki
ab48911b9b
WaitForFirstConsumer handling for DataVolumes (#1242)
* Set the WaitForFirstConsumer phase on DataVolume when storage uses the WaitForFirstConsumer binding mode and is not bound yet.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Skip PVC if not bound in import|clone|upload controllers.

This is done so the VM pod(not the cdi pod) will be the first consumer, and the PVC can be scheduled on the same location as the pod.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

fixup! Skip PVC if not bound in import|clone|upload controllers.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Update importer tests to force bind the PCV by scheduling a pod for pvc, when storage class is wffc.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Update datavolume tests to force bind the PCV by scheduling a pod for pvc, when storage class is wffc.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Update upload controller and upload tests to correctly handle force binding the PCV by scheduling a pod for pvc, when storage class is wffc.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Update clone tests to force bind the PCV by scheduling a pod for pvc when the storage class is wffc.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Update cloner multi-node tests to force bind the PCV by scheduling a pod for pvc when storage class is wffc.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Correct after automerge

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Improve/simplify tests

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Fix error in import test.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Update transport_test,operator_test.go

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Update rbac_test.go and leaderelection_test.go

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Improve Datavolume and PVC Checks for WFFC.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Handle wffc only if feature gate is open - import-controller

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* TEST for Handle wffc only if feature gate is open - import-controller - TEST

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Handle wffc only if feature gate is open - upload-controller with test

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* rename and simplify checks

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* cleanup after rebase

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* update tests after rebase

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* update tests after rebase

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* more cleanups

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Document new WFFC behavior

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Document new HonorWaitForFirstConsumer option

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* update docs according to comments

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* extract common function, cleanup - code review fixes

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* add comment for another pr - 1210, so it can have easier merge/rebase

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* typo

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Simplify getStoragebindingMode - code review comments

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Add FeatureGates interface - code review fix

Additionally pass the features gates instead of the particular feature gate value,
and let shouldReconcilePVC decide what to do with the feature gate. That way shouldReconcilePVC
contains all the logic, and the caller does not need to do additional calls to provide parameters.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Update matcher

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2020-07-22 16:23:44 +02:00
Michael Henriksen
9e2c79b1e0
move api groups to v1beta1 (#1232)
* move upload.cdi.kubevirt.io API group to v1beta1

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* move core api to v1beta1

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix os-3.11 cluster sync and add functional tests for alpha api

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* change more occurences of v1alpha1

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* updates after rebase

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2020-07-10 15:47:38 +02:00
Michael Henriksen
bd69c350bb
Wait for PVCs to be unused before initiating clone operations (#1263)
* don't create snapshot or clone pods if pvcs in use

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* cleanup pods during functional tests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* kill mmore pods blocking clone tests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix typos

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2020-06-30 19:08:50 +02:00
Michael Henriksen
d45574678b
alpha to beta snapshot API (#1206)
* move from alpha to beta snapshot API

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix broken clone tests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* don't generate snapshot client

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2020-05-20 15:25:28 +02:00
Alexander Wels
c707e781a9
Add conditions to DataVolume and add condition annotation to PVCs (#1194)
* Add conditions to DataVolume and add condition annotation to PVCs

Signed-off-by: Alexander Wels <awels@redhat.com>

* Added reason codes to conditions
Updated functional tests to use reason codes
Fixed broken functional test.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Added last transition time
Added documentation

Signed-off-by: Alexander Wels <awels@redhat.com>

* Switch to conditions slice from conditions pointer slice.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Add heartbeat time for conditions.
Renamed AnnRunningCondition to AnnLastTerminationMessage
Only update reason/message on heartbeat or state change.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Rename heartBeat to heartbeat
Re-factor datavolume-conditions to properly update heartbeat time.

Signed-off-by: Alexander Wels <awels@redhat.com>

* renamed AnnRunningConditionReason to AnnLastTerminationReason

Signed-off-by: Alexander Wels <awels@redhat.com>

* Revert lastTerminatedMessage/Reason rename
Added separate AnnScratchBoundCondition/Message/Reason for scratch space

Signed-off-by: Alexander Wels <awels@redhat.com>

* Update bound condition to take scratch or source PVC status into consideration.
Update running condition to take source pod status into consideration

Signed-off-by: Alexander Wels <awels@redhat.com>

* Addressed comments in review.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Properly get scratch space pvc phase.

Signed-off-by: Alexander Wels <awels@redhat.com>
2020-05-12 23:39:23 +02:00
Alexander Wels
08ed6f2b39
Clean up client-go left overs after converting to runtime library. (#1163)
* Clean up client-go left overs after converting to runtime library.
Clean unused functions from controller utils.go
Turn util_test.go into gingko test.
Moved functions from util to proper controller if only used by that controller.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Don't export reconciler variables.

Signed-off-by: Alexander Wels <awels@redhat.com>
2020-04-02 22:59:54 +02:00
Alexander Wels
1d48514c2b
Use controller runtime library for upload controller (#1085)
Signed-off-by: Alexander Wels <awels@redhat.com>
2020-03-03 15:30:56 +01:00
Alexander Wels
79b144088f
Copy kubevirt.io annotations to scratch space pvc, but NOT cdi.kubevirt.io as those are what triggers import/upload/clone. (#1109)
Signed-off-by: Alexander Wels <awels@redhat.com>
2020-02-20 16:50:50 +01:00
Alexander Wels
17db2920b1
Update clone controller to use runtime library. (#1075)
Signed-off-by: Alexander Wels <awels@redhat.com>

Co-authored-by: Michael Henriksen <mhenriks@redhat.com>
2020-02-14 23:04:46 +01:00