Commit Graph

137 Commits

Author SHA1 Message Date
Michael Henriksen
aedaf513ec
Move apis to staging, push to containerized-data-importer-api (#1997)
* move apis to new staging area

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* add script to push to staging

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix lint check and api reference

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* push staging to api repo

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2021-10-28 13:40:24 +02:00
akalenyu
50c93e8b0e
Deploy alerts infra as part of our installation (#1979)
* Deploy alerts infra as part of our installation

Conditionally deploy the infrastructure that is needed to fire alerts for our users
when bad things are happening to CDI.

Testing with `KUBEVIRT_DEPLOY_PROMETHEUS=true`

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Watch and unit test all prometheus related resources

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* add gateway for changing monitoring namespace (rbac purposes)

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* refactor test to check for exact alert name and firing state

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Align pattern of ensuring prometheus resource exists for all

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Remove potential noisy event

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Extract duplicate code to function

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Dont use empty value for prometheus label due to open issue

https://github.com/prometheus-operator/prometheus-operator/issues/4325

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
2021-10-26 21:26:07 +02:00
Arnon Gilboa
addf25b4f9
Support registry import using node docker cache (#1913)
* Support registry import using node docker cache

The new CRI (container runtime interface) importer pod is created with three containers and a shared emptyDir volume:
-Init container: copies static http server binary to empty dir
-Server container: container image container configured to run the http binary and serve up the image file in /data
-Client container: import.sh uses cdi-import to import from server container, and writes "done" file on emptydir
-Server container sees "done" file and exits

Thanks mhenriks for the PoC!

Done:
-added ImportMethod to DataVolumeSourceRegistry (DataVolume.Spec.Source.Registry, DataImportCron.Spec.Source.Registry).
Import method can be "skopeo" (default), or "cri" for container runtime interface based import
-added cdi-containerimage-server & import.sh to the cdi-importer container

ToDo:
-utests and func tests
-doc

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add tests, fix CR comments

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Use deployment docker prefix and tag in func tests

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add OpenShift ImageStreams import support

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add importer pod lookup annotation for image streams

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add pullMethod and imageStream doc

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2021-09-20 22:05:36 +02:00
akalenyu
2254cf0c1f
Add relationship labels (#1864)
Users don't want 👽 resources in clusters,
and we should also be able to tell if were part of a broader installation.

Note:
- Operator created resources were handled in https://github.com/kubevirt/controller-lifecycle-operator-sdk/pull/18
as these labels will be common to all resources deployed by the HCO.
- Now that the controller is guaranteed to have the labels, we can set env vars
that reference the label values (fieldRef) to spare calling GET on the CR in the controllers.
(thanks mhenriks).

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
2021-07-28 20:05:24 +02:00
Maya Rashish
19d109837d
Add a datavolume condition for the image being too large (#1818)
* Strip newlines when writing a termination message.

Otherwise it isn't visible, at least when viewing in the -o yaml view.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Write down the nbdkit output and add it to the error output

With the added output from nbdkit, we can see the reason for the
non-existence of the nbdkit socket.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Don't set that we're waiting if it's CrashLoopBackOff

It's better to have the reason for the crash (terminate message)
than "backing off 5 minutes"

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Simplify all "image too large to fit" terminate reasons.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Move verifyConditions to utils, no functional change

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Switch test for image too large to test condition and not log

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Remove unused branch

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Rename setConditionFromPodWithPrefix to setAnnotationsFromPodWithPrefix

No functional change. Intended to be followed by some refactoring.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Fold restart count logic into the common setAnnotationsFromPodWithPrefix

Changing to >= rather than > to ensure a zero pod restart count is
always used -- the import controller unit tests request this.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Reduce indentation by returning right away.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Fold check for pod being nil into common code.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* move saveVddkAnnotations into util and make it unconditional

Call it setVddkAnnotations for consistency.
Check for not-terminated inside the function, not outside.

Removes check for source being VDDK (to avoid passing more arguments):
it won't match the regex anyway.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Reduce indentation by bailing on failure.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Reorder parameters to mirror the order in the function name

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Use a named variable for first container state

Yields shorter, more legible lines.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Use a constant for the nbdkit log.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Add more information to function description - also logs to file

Signed-off-by: Maya Rashish <mrashish@redhat.com>
2021-06-29 12:47:05 +02:00
Arnon Gilboa
c963ebb276
Add DataVolume annotation to retain the transfer pods after completion (#1841)
* Add annotation to retain the transfer pods after completion

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add cloner test

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2021-06-28 17:39:31 +02:00
Matthew Arnold
56cfd15099
Add "AwaitingVDDK" back to condition reason. (#1816)
* Move AwaitingVDDK constant to common.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Copy pending PVC bound condition reason to DV.

Replace the fixed "Pending" string and tweak the unit test that checked
this.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add a functional test for AwaitingVDDK.

Delete v2v-vmware ConfigMap and create a DataVolume, and the bound
condition should have a reason of "AwaitingVDDK".

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Move AwaitingVDDK to its own functional test.

Retain common test code with existing table, but tack on an extra
cleanup step so v2v-vmware ConfigMap can be restored afterward.

Signed-off-by: Matthew Arnold <marnold@redhat.com>
2021-06-22 21:11:40 +02:00
Michael Henriksen
d92c2f459d
update deps and bazel (#1815)
* update deps and bazel

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix apidocs and unit tests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix generate-verify

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2021-06-08 01:31:59 +02:00
Matthew Arnold
2960a3f6d3
Copy VDDK version to DV annotation. (#1752)
* Add an interface to watch nbdkit logs.

Useful for fishing out various pieces of information. Save VDDK library
version and connected ESX host by appending to the importer pod's
termination message. Turns nbdkit logging up to verbose for VDDK data
sources, so only the last few lines are printed for debugging.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Copy VDDK info from termination message to PVC/DV.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add unit tests for saved VDDK information.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add functional test for VDDK annotations.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Fix unit test, forgot to check for nil pvc.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Don't ignore errors updating PVC with VDDK info.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Watch nbdkit with Scanner instead of ReadString.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Move VDDK info test into existing functional test.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Make nbdkit stop sequence slightly clearer.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Save VDDK info in regular DV reconciler.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Don't save VDDK info when PVC is being deleted.

Also, piggyback off existing PVC update instead of introducing a new
error handling path.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Fix VDDK-info unit tests.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Use scanner for all nbdkit logging.

Also fix up a minor merge mistake.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Try to satisfy complaints from SonarCloud.

Signed-off-by: Matthew Arnold <marnold@redhat.com>
2021-05-08 21:06:18 +02:00
Vishesh Tanksale
2014ddecfd
Adding priority class name for importer/upload pod to data volume object (#1740)
* Adding priority class name for importer/upload pod to data volume object

Signed-off-by: Vishesh Ajay Tanksale <vtanksale@apple.com>

* Addressing review comments

Signed-off-by: Vishesh Ajay Tanksale <vtanksale@apple.com>

* Adding controller logic to assign priority class on importer,cloner and uploader pod

Signed-off-by: Vishesh Ajay Tanksale <vtanksale@apple.com>

* Adding functional test

Signed-off-by: Vishesh Ajay Tanksale <vtanksale@apple.com>

* Addressing review comments

Signed-off-by: Vishesh Ajay Tanksale <vtanksale@apple.com>

* Updating Data Volume doc

Signed-off-by: Vishesh Ajay Tanksale <vtanksale@apple.com>

Co-authored-by: Vishesh Ajay Tanksale <vtanksale@apple.com>
2021-04-28 09:38:42 -05:00
Tomasz Barański
84f51f32f6
Preallocate cloning DV (#1719)
* Refactoring - move PreallocationApplied flag definition to common

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* Preallocate cloning DataVolumes

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
2021-04-27 15:06:30 +02:00
Arnon Gilboa
3c192b42b5
Add insecureRegistries to CDIConfig; keep supporting ConfigMap for upgrade (#1754)
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2021-04-22 04:21:05 +02:00
Michael Henriksen
3447bb84c7
Cluster scoped DataVolume/PVC namespace transfer API (#1673)
* Cluster-scoped namespace transfer api and controller

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* unit tests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* ObjectTransfer webhook

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* new functests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* experiment with termination grace period

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* quota test

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2021-02-24 20:45:24 +01:00
Arnon Gilboa
6ff7537caa
Fix default content type validation for clone (#1659)
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2021-02-18 00:01:01 +01:00
Marcelo Carneiro do Amaral
8d1721db0a
Add support for proxy in the CDI import pod, reconciling the info from OpenShift cluster wide proxy when available (#1507)
* api: Add new ImportProxy structure to CDIConfig in v1beta1 core api

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* common: Add constants related to configure Import proxy in Import pod and controller reconcile

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* config controller: Add import proxy reconcile, monitoring OCP proxy updates

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* config controller: Add unit test for import proxy reconcile, monitoring OCP proxy updates and creating Secret and ConfigMap

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* import controller: Add unit support of ImportProxy in the tests

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* controller util: Add support to retrieve an OpenShift cluster wide proxy object as well as its needed configurations

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* controller util:  Add unit test for the get cluster wide proxy functionality

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* controller: moved isOpenshift function to utils common and export it

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* importer: Clone http transport to keep the default proxy config such as the usage of environment variables and add support for proxy CA

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* operator: Update CDIConfigCRD with import proxy info

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* operator: Update CDIListCRD with import proxy info

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* operator: Add service account rights to CDI for accessing OpenShift proxy obj

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* operator: Add OpenShift proxy obj scheme support

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* test: Add new forwarding proxy for testing things that require a proxy

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* test: Add test to verify the importer pod with proxy

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* doc: Update cdi config doc with the ImportProxy info

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* add support to build the new proxy docker image

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* test: remove unwanted F parameter from test

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* controller: fix error handling

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* test: update the used method to get the pod of the importer pod and proxy

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* fixed comments from the revision

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* controller config: update the methods to use the log object from the reconcile object

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* controller: update function GetImportProxyConfig to return errors and include a new unit test

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* controller: add back a test that was removed by mistake

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* update updateCDIConfigByUpdatingTheClusterWideProxy function

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* test: decrease the import pod log pooling interval and increase the image size to be able to verify the import pod before it is deleted

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* test: update the pvc and proxied requests verification

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* tools: add X-Forwarded-For header in the proxy requests  used for testing

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* test: reset cluster wide proxy with the original values

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* test: fix proxy update to change spec instead of status and other minor updates

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* doc: update import proxy description

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>

* update generated files

Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
2021-02-16 13:43:01 +01:00
Tomasz Barański
a29c3d4165
Preallocate even if the size is too small (#1637)
This PR removes "skipped" condition for preallocation. Importer/uploader
will preallocate to the available size. Filesystem overhead needs to be
taken into account.

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
2021-02-10 21:18:56 +01:00
Maya Rashish
9d1b94f47e
Simplify shouldReconcile function arguments. (#1602)
* Simplify shouldReconcile function arguments.

By having the function itself grab things it needs and are easily
obtained.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Adapt unit tests to simpler shouldReconcilePVC

Don't set any feature gates for WFFC being disabled.
When the second argument is true, pass the immediate binding annotation
to the PVC itself.

Signed-off-by: Maya Rashish <mrashish@redhat.com>
2021-02-04 14:38:26 +01:00
Matthew Arnold
cdb3d03ff0
Add error to DV when VDDK configmap is missing. (#1627)
Assists resolution of BZ#1886566. Use existing mechanism to copy certain
PVC annotations into DV status conditions.

Signed-off-by: Matthew Arnold <marnold@redhat.com>
2021-02-02 13:56:51 +01:00
Tomasz Barański
9668eb7d41
core: Preallocate blank block volumes (#1559)
Importer pod needs to be started for blank block volumes and it needs to
handle the case.

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
2021-01-18 19:23:32 +01:00
Bartosz Rybacki
2abf7330c1
Add a DV/PVC annotation "storage.bind.immediate.requested" (#1560)
* Add a DV/PVC annotation "storage.bind.immediate.requested"

When the annotation is applied the CDI will force bind the PVC (by scheduling worker pods), ignoring the logic to handle WaitForFirstConsumer binding mode.

This is useful when uploading "template" images to the cluster on local storage with WaitForFirstConsumer binding. In this case the image has to be available somewhere on the cluster, the actual placement of image on specific node is not important, so the CDI worker node can be used as a first consumer.

For storage with immediate binging it is effectively a NOOP.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Test import/upload/clone with annotation "storage.bind.immediate.requested"

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2021-01-15 08:15:10 +01:00
Tomasz Barański
d31591de94
core: Preallocate blank image disks as well (#1555)
This fixes a bug where blank image disks were not preallocated.

Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1914177

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
2021-01-08 23:20:59 +01:00
Tomasz Barański
9a6ce7eee0
Preallocation check all paths (#1535)
* [WIP] doc: User-facing doc for preallocation support

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* apis: CDI accepts `preallocation` option.

With this commit CDI accepts (but does handle) `preallocation` settings
for DataVolumes and in CDIConfig.

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* core: Implementing preallocation

This commit implements preallocation support for import and upload.

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* test: Functional tests for preallocation support

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* core: Remove "preallocation for StorageClasses" config

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* test: Removed unused function

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* Updated dependencies
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* core: Uss PVC annotation to pass preallocation parameters

DataVolume controller now uses a PVC annotation to pass preallocation
configuration to import and update controllers.

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* core: Implementing preallocation

This commit implements preallocation support for import and upload.

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* core: Make sure all import/upload paths do preallocation

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
2021-01-08 04:48:59 +01:00
Tomasz Barański
91a15c57d1
Preallocation support (#1498)
* [WIP] doc: User-facing doc for preallocation support

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* apis: CDI accepts `preallocation` option.

With this commit CDI accepts (but does handle) `preallocation` settings
for DataVolumes and in CDIConfig.

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* core: Implementing preallocation

This commit implements preallocation support for import and upload.

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* test: Functional tests for preallocation support

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* core: Remove "preallocation for StorageClasses" config

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* test: Removed unused function

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* test: Fix rook-ceph test failures

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* Updated dependencies
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* core: Uss PVC annotation to pass preallocation parameters

DataVolume controller now uses a PVC annotation to pass preallocation
configuration to import and update controllers.

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
2020-12-18 16:46:16 -05:00
Matthew Arnold
e54bc4c2fe
VDDK: incremental copy with changed block tracking (#1517)
* Plumb new checkpoint API through to VDDK importer.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add incremental data copy from VDDK.

Create a new data source implementation similar to vddk-datasource, but
only for blocks of data that changed between two snapshots. Also factor
out common things between the two VDDK data sources.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Check block status for warm and cold imports.

Addresses a bunch of runtime issues, but progress tracking isn't right.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Find snapshots correctly.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Remove separate warm/cold VDDK importers.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Advance through the checkpoint list in the spec.

Move DataVolume to Paused after each checkpoint, and start a new
importer pod for the next available checkpoint. Keep track of which
checkpoints have been copied by adding PVC annotations associating each
checkpoint with the UID of the pod that copied it.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Allow spec updates to drive multi-stage imports.

A multi-stage import can create checkpoints at any time, so CDI needs to
be able to receive updates to the list of checkpoints. Implement this by
allowing spec changes only for fields related to multi-stage imports.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Avoid deleting destination in multi-stage import.

A multi-stage import will have an initial data copy to the destination
file followed by separate copies for individual deltas. The destination
file should not be deleted before starting these delta copies.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Get VDDK data source to pass formatting tests.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Unit tests for multi-stage import admission rules.

Make sure only updates to checkpoint-related fields are accepted.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add warm import unit tests for VDDK data source.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add VDDK warm import functional test.

Put two snapshots in the vCenter simulator inventory, and run them
through a multi-stage import process.  Also clean up some issues
reported by test-lint.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add some documentation about multi-stage imports.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Pass existing multi-stage DataVolume unit tests.

Also remove MD5 sum step used for debugging, since it can take a long time.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Remove tabs from documentation.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Pass failing import-controller unit test.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* More unit tests for multi-stage field updates.

Also factor these tests into a DescribeTable.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add nbdkit retry filter.

Available as of Fedora 33 update.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Give correct file name to nbdkit in more cases.

The backing file in the spec might not always match the backing file in
the snapshot, so try harder to match those files by disk ID. May still
need to allow updates to backingFile, depending on how this gets used.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add more unit tests for datavolume-controller.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Fix linter error from last commit.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add unit tests for some govmomi API calls.

Move original calls into mock interfaces to make this work.

Signed-off-by: Matthew Arnold <marnold@redhat.com>
2020-12-18 15:58:47 +01:00
Arnon Gilboa
ff42ea0597
Pass specific PVC annotations to the transfer pods (#1480)
* Pass specific PVC annotations to the transfer pods

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add annotations importer/uploader/cloner functional tests

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2020-11-30 19:20:38 +01:00
Samuel Lucidi
a453c52ecd
Controller support for Multistage Imports (#1450)
* Add checkpoints to DataVolume CRD and reconciliation

* Add Previous, Current, and FinalCheckpoint to DataVolume CRD
* Use checkpoints to set annotations on the PVC
* If an importer pod succeeds while checkpoint annotations are set,
  then set the DataVolume status to Paused intstead of Succeeded.
* Clear the PVC checkpoint annotations

Signed-off-by: Sam Lucidi <slucidi@redhat.com>

* Add new fields to DataVolume CRD creation

Signed-off-by: Sam Lucidi <slucidi@redhat.com>

* Generate updated code for the DataVolume changes

Signed-off-by: Sam Lucidi <slucidi@redhat.com>

* Add tests for multistage import annotations

Signed-off-by: Sam Lucidi <slucidi@redhat.com>
2020-11-20 12:51:59 +01:00
Bartosz Rybacki
8f49ab2945
Scratch import bug (#1424)
* check if pod used

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Test scratch deletion during import

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Handle scratch missing just after being created in utility function.

There was a possibility that just after creating the scratch it is deleted, and the next get results in error. This error was ignored - not returned from util - resulting in wrong behavior.
Other fix here is to skip pods with status PodSucceeded and PodFailed in getPodsUsingPVCs function - such Pods are not using PVC, as these pods are already done.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Reschedule reconcile for the pvc until pvc import is complete.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Increase number of NFS volumes, and document the test.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Try another quick fix for verifier pod flakiness.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Cleanup defaults

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2020-11-03 16:31:50 +01:00
Maya Rashish
b91887e1b7
Reserve overhead when validating that a Filesystem has enough space (#1319)
* When validating disk space, reserve space for filesystem overhead

The amount of available space in a filesystem is not exactly
the advertise amount. Things like indirect blocks or metadata
may use up some of this space. Reserving it to avoid reaching
full capacity by default.

This value is configurable from the CDIConfig object spec,
both globally and per-storageclass.

The default value is 0.055, or "5.5% of the space is
reserved". This value was chosen because some filesystems
reserve 5% of the space as overhead for the root user and
this space doubles as reservation for the worst case
behaviour for unclear space usage. I've chosen a value
that is slightly higher.

This validation is only necessary because we use sparse
images instead of fallocated ones, which was done to have
reasonable alerts regarding space usage from various
storage providers.

---

Update CDIConfig filesystemOverhead status, validate, and
pass the final value to importer/upload pods.

Only the status values controlled by the config controller
are used, and it's filled out for all available storage
classes in the cluster.

Use this value in Validate calls to ensure that some of the
space is reserved for the filesystem overhead to guard from
accidents.

Caveats:

Doesn't use Default: to define the default of 0.055, instead
it is hard-coded in reconcile. It seems like we can't use a
default value.

Validates the per-storageClass values in reconcile, and
doesn't reject bad values.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Use util GetStorageClassByName

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Test filesystem overhead validation against async upload endpoint

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* wait for NFS PVs to be deleted before continuing

Intended to help with flakes, but didn't make a difference.
Probably still worth doing.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Avoid using the uncached client unnecessarily

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Add error handling for the case where even a default SC is not found

Note that this change isn't expected to make a difference, as we
check if the targetStorageClass is nil later on and have the same
behaviour, but this is probably more correct API usage.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Add testing for the validation of filesystem overhead values

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Fix logical error in waiting for NFS PVs.

Wait for all of them, not just the last one.

Signed-off-by: Maya Rashish <mrashish@redhat.com>
2020-10-01 18:31:32 +02:00
Jakub Dzon
7f368900de
Updated controller-lifecycle-operator-sdk dependency (#1389)
Signed-off-by: Jakub Dzon <jdzon@redhat.com>
2020-09-24 14:39:29 +02:00
Jakub Dzon
5aa47587d3
Introducing operator lifecycle sdk (#1350)
Signed-off-by: Jakub Dzon <jdzon@redhat.com>
2020-09-17 23:25:26 +02:00
Matthew Arnold
dc6bad6473
Fill out VDDK data volume source. (#1320)
* Add support for a VDDK datasource to the controllers

This commit does not implement the VDDK datasource. It simply
adds the DataVolumeSourceVDDK type and adds support to the
datavolume and import controllers for it. The datasource itself
will need to be done in a follow-up.

Signed-off-by: Sam Lucidi <slucidi@redhat.com>

* Fix tests and clean up lint

Signed-off-by: Sam Lucidi <slucidi@redhat.com>

* Add DataVolumeSourceVDDK to the v1alpha1 api group

Signed-off-by: Sam Lucidi <slucidi@redhat.com>

* Install nbdkit.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add govmomi to vendor directory.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Integrate govmomi into build.

Incomplete! This allows 'find' and 'object' imports from govmomi, but
further dependencies may require more bazel tweaks later.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Create a container image just to hold the VDDK.

Mount the VDDK folder to /opt. Also copy the libxcrypt-compat RPM to
/opt for run-time installation as a temporary workaround for my
inability to install it at container creation time. Make VDDK optional
by storing a dummy file in its directory, so an empty VDDK directory
will not break the build.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Fill in VDDK data source.

Add the simplest possible VDDK data source: set up an nbdkit server with
the given VMware parameters, and copy data out using qemu-img.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add vddk to resource validation schema.

Stops "error validating data" messages.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Move libxcrypt-compat install out of VDDK image.

Requiring users to add this to the VDDK image is kind of onerous, so
move it straight into the importer image instead.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Generate BUILD.bazel for the rest of govmomi.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Get VDDK sidecar image from v2v-vmware ConfigMap.

Convention from https://github.com/kubevirt/web-ui-components/pull/534

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Reduce nbdkit startup timeout.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Update govmomi dependencies.

Cleans up 'inconsistent vendoring' errors.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Fix schema verification test.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Change disk.raw to disk.img.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add vCenter simulator for VDDK datasource tests.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Functional test with vcsim and dummy VDDK plugin.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Clean up lint errors.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Check kubevirt-hyperconverged namespace for VDDK.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add VDDK unit tests and API functional test.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Read a raw image for VDDK test plugin.

Have the fake VDDK test plugin for nbdkit read a raw image passed in
/opt/updates/nbdtest.img instead of returning fixed data.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add some documentation about VDDK data sources.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Remove unit test focus and trailing spaces.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Use updated build container image.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add some extra comments about VDDK fields.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Clarify VDDK ConfigMap a little more.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Clarify namespace for vddk-init sidecar.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Remove NodePort from vCenter simulator manifest.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Re-word explanations on VDDK source types.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add VDDK to comment with types needing URL check.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Rebuild types.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Move AnnThumbprint to storage.import.vddk.thumbprint.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Remove run-time installation of libxcrypt-compat.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Remove fixed namespaces for vddk-init reference.

The importer now looks for the v2v-vmware ConfigMap in the current CDI
namespace instead of openshift-cnv/kubevirt-hyperconverged.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Find CDI install namespace from data volume test.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

Co-authored-by: Sam Lucidi <slucidi@redhat.com>
2020-09-11 23:37:22 +02:00
Maya Rashish
68f70bb43a
Sync with HCO definition of node placement values. (#1360)
But omitting +listType=set because it introduces errors.

Signed-off-by: Maya Rashish <mrashish@redhat.com>
2020-09-10 23:19:22 +02:00
Maya Rashish
e3436e0199
Allow specifying nodeSelector, affinity and tolerations for CDI pods (#1346)
* Generate CDI CRD using controller-tools.

This is only done for CDI CRD as it requires the existence of source
code. Other CRDs we create are created by a more bare bones pod.

CDIUninstallStrategy was missing a comment describing it, so add
one. This was spotted manually so there might be more missing.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Allow users to specify which nodes CDI pods will live on.

nodeSelector, affinity and tolerations are possible values.

This is done in the CDI CR (rather than CDIConfig) as we are
interested in having this field be populated by external operators.

Unit tests now require the existence of a CDI CR, so create it.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Add a unit test covering some node placement functions

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Specify that all our pods are linux-only.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Avoid duplicate test, accidental left over.

Pointed out by awels, thanks.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Rename to cdiOperatorDeployment for clarity.

Suggested by awels

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Specify we only run on linux using the CDI CR, no need to embed this
into the code.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Don't dereference workloadPlacement for no reason

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Split off operator test to have its own AfterEach, BeforeEach.

Use even more descriptive function names.

Do all the CDI delete/restore logic in AfterEach, to ensure that
it happens and restores the deployment with the original CR even
if the test fails.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Remove XXX. This is the proper way.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Adapt to latest changes in controller_test.go (renaming import)

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Simplify, not storing intermediate value.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Don't dereference nodeplacement in callers to CreateDeployment

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Remove redundant save & restore. Unit tests do this for us.

Pointed out by awels, thanks.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Split out "find toplevel" to a utility function

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Wait for the CDI CR update to apply before continuing.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Simplify, not storing intermediate value.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Make it clear that the chosen node placement will not be schedulable.

Signed-off-by: Maya Rashish <mrashish@redhat.com>
2020-09-03 22:13:18 +02:00
Michael Henriksen
75f4fd6f2f
update k8s deps to 18.6 and controller runtime to 0.6.2 (#1330)
* update k8s deps to 1.18.6 and controller runtime to 0.6.2

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* remove building code generators from docker image.  This way the k8s ligray version only has to be updated in go.mod

Do more stuff in the bazel container.  Faster and better interop

Fix unit tests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* make format

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* remove unnecessary rsync

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* redo code generator dep management

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* builder uses go modules

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2020-08-07 14:09:52 +02:00
Bartosz Rybacki
ab48911b9b
WaitForFirstConsumer handling for DataVolumes (#1242)
* Set the WaitForFirstConsumer phase on DataVolume when storage uses the WaitForFirstConsumer binding mode and is not bound yet.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Skip PVC if not bound in import|clone|upload controllers.

This is done so the VM pod(not the cdi pod) will be the first consumer, and the PVC can be scheduled on the same location as the pod.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

fixup! Skip PVC if not bound in import|clone|upload controllers.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Update importer tests to force bind the PCV by scheduling a pod for pvc, when storage class is wffc.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Update datavolume tests to force bind the PCV by scheduling a pod for pvc, when storage class is wffc.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Update upload controller and upload tests to correctly handle force binding the PCV by scheduling a pod for pvc, when storage class is wffc.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Update clone tests to force bind the PCV by scheduling a pod for pvc when the storage class is wffc.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Update cloner multi-node tests to force bind the PCV by scheduling a pod for pvc when storage class is wffc.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Correct after automerge

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Improve/simplify tests

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Fix error in import test.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Update transport_test,operator_test.go

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Update rbac_test.go and leaderelection_test.go

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Improve Datavolume and PVC Checks for WFFC.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Handle wffc only if feature gate is open - import-controller

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* TEST for Handle wffc only if feature gate is open - import-controller - TEST

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Handle wffc only if feature gate is open - upload-controller with test

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* rename and simplify checks

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* cleanup after rebase

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* update tests after rebase

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* update tests after rebase

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* more cleanups

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Document new WFFC behavior

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Document new HonorWaitForFirstConsumer option

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* update docs according to comments

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* extract common function, cleanup - code review fixes

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* add comment for another pr - 1210, so it can have easier merge/rebase

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* typo

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Simplify getStoragebindingMode - code review comments

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Add FeatureGates interface - code review fix

Additionally pass the features gates instead of the particular feature gate value,
and let shouldReconcilePVC decide what to do with the feature gate. That way shouldReconcilePVC
contains all the logic, and the caller does not need to do additional calls to provide parameters.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Update matcher

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2020-07-22 16:23:44 +02:00
Michael Henriksen
9e2c79b1e0
move api groups to v1beta1 (#1232)
* move upload.cdi.kubevirt.io API group to v1beta1

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* move core api to v1beta1

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix os-3.11 cluster sync and add functional tests for alpha api

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* change more occurences of v1alpha1

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* updates after rebase

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2020-07-10 15:47:38 +02:00
Alexander Wels
c26da98fdb
Print annotations saved to PVC on update. (#1254)
Signed-off-by: Alexander Wels <awels@redhat.com>
2020-06-30 16:10:50 +02:00
Bartosz Rybacki
ab8b9c025e
Generating label names (#1200)
* Handle labels length correctly

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Handle service name generation correctly

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Remove not needed labels

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Store import pod name in annotation

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Enable long DV name

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Handle name with dot when creating service/label name

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Test long names on import,  upload and clone

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Store upload pod name in annotation

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Store importer scratch pvc name in annotation

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Quick fix for tests (need improvements)

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Cleanup handling scratch name

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Ensure pod/service name conflicts are handled

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Handle client errors when trying to get the import pod

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Style improvements, and other code review fixes.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Store clone source pod name in an annotation

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Correct name initialization and tests

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Do not init name if pod already exists. It is not needed.

The situation of having a pod but not name on annotation can happen after the upgrade, when we have a legacy pvc and pod already existing, but clone operation not finished. But when we already have the pod, then in the code (currently) we do not need the name from annotation.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Cleanup scratch name handling

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Use constant for max dv name in validation

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Simplify clone source pod name initialization

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2020-05-29 19:55:32 +02:00
Alexander Wels
c707e781a9
Add conditions to DataVolume and add condition annotation to PVCs (#1194)
* Add conditions to DataVolume and add condition annotation to PVCs

Signed-off-by: Alexander Wels <awels@redhat.com>

* Added reason codes to conditions
Updated functional tests to use reason codes
Fixed broken functional test.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Added last transition time
Added documentation

Signed-off-by: Alexander Wels <awels@redhat.com>

* Switch to conditions slice from conditions pointer slice.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Add heartbeat time for conditions.
Renamed AnnRunningCondition to AnnLastTerminationMessage
Only update reason/message on heartbeat or state change.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Rename heartBeat to heartbeat
Re-factor datavolume-conditions to properly update heartbeat time.

Signed-off-by: Alexander Wels <awels@redhat.com>

* renamed AnnRunningConditionReason to AnnLastTerminationReason

Signed-off-by: Alexander Wels <awels@redhat.com>

* Revert lastTerminatedMessage/Reason rename
Added separate AnnScratchBoundCondition/Message/Reason for scratch space

Signed-off-by: Alexander Wels <awels@redhat.com>

* Update bound condition to take scratch or source PVC status into consideration.
Update running condition to take source pod status into consideration

Signed-off-by: Alexander Wels <awels@redhat.com>

* Addressed comments in review.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Properly get scratch space pvc phase.

Signed-off-by: Alexander Wels <awels@redhat.com>
2020-05-12 23:39:23 +02:00
Bartosz Rybacki
bfe63cb61c
Added GetPodName function used to correctly generate pod names in con… (#1175)
* Create corect naming functions for resources

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Use name generator for importer resources

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2020-05-07 16:31:19 +02:00
Alexander Wels
08ed6f2b39
Clean up client-go left overs after converting to runtime library. (#1163)
* Clean up client-go left overs after converting to runtime library.
Clean unused functions from controller utils.go
Turn util_test.go into gingko test.
Moved functions from util to proper controller if only used by that controller.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Don't export reconciler variables.

Signed-off-by: Alexander Wels <awels@redhat.com>
2020-04-02 22:59:54 +02:00
Bartosz Rybacki
4605cf1dc2
Add a field to DataVolume to track the number of retries/pod restarts (#1155)
* Add a field to DataVolume to track the number of retries/pod restarts

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Add a field to DataVolume to track the number of retries/pod restarts

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Make RESTARTS non-empty on DataVolume (shows as as 0 'zero')

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Test reporting restarts on DataVolume when importing.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Fix tests

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Code review fixes

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Restart Count status test for upload and clone controller

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2020-03-30 23:17:49 +02:00
Alexander Wels
f2657d5927
Set secondary GID of user on import and upload pods to be 107 (qemu) … (#1148)
* Set secondary GID of user on import and upload pods to be 107 (qemu) for kubevirt content type.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Fix broken unit tests on upload.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Add import and clone test.

Signed-off-by: Alexander Wels <awels@redhat.com>
2020-03-26 14:33:48 +01:00
Piotr Kliczewski
78cee6d166
Imageio - new import data source (#1042)
This PR introduces new import data source to fetch disks from ovirt
and store them in storage available in the cluster. It uses ovirt-sdk
to check whether the this is there, the size of the disk and it obtains
signed ticket which allows to get the disk using ovirt-imageio-proxy.

In order to use it a user needs to create configmap with ovirt api
certificate and secret with ovirt engine user and password. Both are
required and validated by webhook. In order to start import process
we need to post following definition:

---
apiVersion: cdi.kubevirt.io/v1alpha1
kind: DataVolume
metadata:
  name: "imageio-dv"
spec:
  source:
      imageio:
         url: "<engine-api-endpoint>"
         secretRef: "<secret-name-containing-engine-credentials>"
         certConfigMap: "<configmap-name-containing-engine-api-certificate>"
         diskId: "<disk-id-which-we-want-to-import>"
  pvc:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: "64Mi"
---

Signed-off-by: Piotr Kliczewski <piotr.kliczewski@gmail.com>
2020-03-19 19:49:44 +01:00
Alexander Wels
79b144088f
Copy kubevirt.io annotations to scratch space pvc, but NOT cdi.kubevirt.io as those are what triggers import/upload/clone. (#1109)
Signed-off-by: Alexander Wels <awels@redhat.com>
2020-02-20 16:50:50 +01:00
Alexander Wels
17db2920b1
Update clone controller to use runtime library. (#1075)
Signed-off-by: Alexander Wels <awels@redhat.com>

Co-authored-by: Michael Henriksen <mhenriks@redhat.com>
2020-02-14 23:04:46 +01:00
Alexander Wels
5ed55cb22c
Remove code that potentially let DV get into failed state with RestartOnFailure policy. (#1099)
Signed-off-by: Alexander Wels <awels@redhat.com>
2020-02-10 23:29:15 +01:00
Michael Henriksen
99f8af5b86 k8s client upgrade to 1.16 (#1079)
* initial client upgrade to 1.16

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix Route detection in OpenShift

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2020-01-14 13:43:17 +01:00
Woohyung Han
2adcc80f95 Fix to create temporary pod with default resource quota (#1055)
If we create a temporary pod in the namespace which has
resource quota, `must specify quota` error occurs. Fix to
create temporary pod with default resource quota.

Co-authored-by: hyoung-90 <ssarang520@gmail.com>
Co-authored-by: Woohyung Han <woohhan@gmail.com>
Signed-off-by: anencore94 <anencore94@kaist.ac.kr>

Co-authored-by: anencore94 <37469330+anencore94@users.noreply.github.com>
Co-authored-by: hyoung-90 <58063867+hyoung-90@users.noreply.github.com>
2020-01-09 17:43:14 +01:00
Alexander Wels
f090ddc2e6 Update Import controller to use runtime controller library. (#1054)
Refactor import related utilities out of util.go into the controller.

Signed-off-by: Alexander Wels <awels@redhat.com>
2019-12-20 19:28:43 +01:00
Alexander Wels
d4a4ece09b Requeue on failure like sample k8s controller (#1015)
Signed-off-by: Alexander Wels <awels@redhat.com>
2019-11-12 13:21:15 +01:00
Michael Henriksen
66a1c8c996 Unique client cert per clone source pod (#987)
* unique cert per clone source pod

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* upload controller shouldn't cache certs

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix test variable names

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2019-10-09 15:10:55 -04:00
zvikorn
7dd334121c Creating blank disk on a block device (#962)
Signed-off-by: tavni <tavni@redhat.com>
2019-09-18 19:06:02 -04:00
Michael Henriksen
834b85ecbf Network clone (#897)
* network cloning

* fix clone progress
2019-08-01 16:01:25 -04:00
Alexander Wels
1d534d58d5 Record import failure messages
Signed-off-by: Alexander Wels <awels@redhat.com>
2019-05-30 09:22:58 -04:00
Alexander Wels
630a23ef23 Fix a bunch of go score card issues.
Signed-off-by: Alexander Wels <awels@redhat.com>
2019-05-06 16:52:03 -04:00
tavni
d4c7b0be54 Adding support to import disk image into Block PV
Signed-off-by: tavni <tavni@redhat.com>
2019-03-28 11:43:56 +02:00
Alexander Wels
afe77c3514 Allow missing storage class for scratch space, if none exists, create scratch PVC without SC
Signed-off-by: Alexander Wels <awels@redhat.com>
2019-03-27 11:41:15 -04:00
Alexander Wels
d2ca78bec3 Use CDIConfig for scratch storage class.
Signed-off-by: Alexander Wels <awels@redhat.com>
2019-03-21 15:37:07 -04:00
Alexander Wels
1e46004ad2 Scratch space retry framework logic.
Signed-off-by: Alexander Wels <awels@redhat.com>
2019-03-07 14:54:08 -05:00
Y.Horie
c9104059fb move to klog from glog 2019-03-05 13:33:20 +09:00
Michael Henriksen
6f1d130d97 tests and review comments 2019-02-25 20:12:56 -05:00
Michael Henriksen
62f1e77f66 change INSECURE_REGISTRY to INSECURE_TLS because may want to have insecure hosts for http import 2019-02-25 20:12:56 -05:00
Michael Henriksen
3892a7310d add configmap for insecure regestries 2019-02-25 20:12:56 -05:00
j-griffith
30c0e1b814 Delete importer pod on pvc delete request
Related to issue #525, we have a similar situation when using PVCs
without DataVolumes.  The user requests a PVC with CDI Importer
annotations, if the Importer POD is in a fail/retry loop and the user
requests deletion of the PVC the PVC will be stuck in terminating
waiting for the user to forcibly delete the running Import POD.

This PR adds a check for PVC delete in the import controller and
will delete the POD explicitly if it sees a PVC delete request.  This
will also be picked up for Import DataVolumes, so we can remove the
extra checks that were previously added in the DV controller.

fixes #649
fixes #525
2019-02-20 13:43:09 -07:00
Michael Henriksen
717ceaa919 add ability for user to specify certs for import from registry 2019-02-18 09:06:46 -05:00
Michael Henriksen
fbaee779f7 don't set pvc pod phase as error when import pod is starting for the first time 2019-02-11 09:57:04 -05:00
tavni
aebc2b3e87 Add ability to create empty disk images #497
Signed-off-by: tavni <tavni@redhat.com>
2018-11-27 16:15:44 +02:00
Alexander Wels
260d4fa8e0 Automatically resize image to fit the requested PVC.
Combined code from PR#489 and PR#490 by
@gites and
@danielerez
Added some tests and rebased on current master.

Signed-off-by: Alexander Wels <awels@redhat.com>
2018-11-20 15:40:13 -05:00
Alexander Wels
70666e5946 Add source and contentType annotations
Signed-off-by: Alexander Wels <awels@redhat.com>
2018-11-14 12:20:26 -05:00
tavni
bc6a7b8474 change the import controller not to have global parameter #504
Signed-off-by: tavni <tavni@redhat.com>
2018-10-25 20:49:04 +03:00
tavni
c7e40ebffd Add structure to hold all CDI importer environment variables #501
Signed-off-by: tavni <tavni@redhat.com>
2018-10-25 13:21:28 +03:00
tavni
4db7a28eb5 Import/upload pvc 'cdi.kubevirt.io/storage.pod.phase' annotation value is wrong when import fails. Issue #494
Signed-off-by: tavni <tavni@redhat.com>
2018-10-18 17:47:57 +03:00
j-griffith
371e5c2ba0 Set pod restart policy to "OnFailure" and cleanup
This patch does two things:
1. We modify the restar policy from never to "OnFailure" for the
    Importer (we already did this for Upload, and we don't want to do it yet
    for the Clone pods)
2. Upon succesful completion delete our worker pods
    We're not doing this on failed pods right now because in general an
    admin will likely want to inspect the pod to see why it failed.

The first question someone will likely ask is "well what about a node
failure", well in the case of a node failure for CDI we have a bigger
outstanding issue remaining to figure out how to deal with volume
connections.  We can't just simply reschedule the job.
2018-10-09 08:25:22 -06:00
tavni
ce22c48834 CDI controller code dedupe
Signed-off-by: tavni <tavni@redhat.com>
2018-09-17 19:44:09 +03:00
John Griffith
c455577cca Fix lint errors on common package (#449)
This change just updates the common package to pass golint.  Of course
that has some reaching implications into other packages with renaming of
constants.

I've intentionally kept this patch set to the bare minimum for the lint
test, I do think that we should put some effort into our use of
constants (especially those in common.go) in the future.

Ideally we'd declare constants where they're used, and we wouldn't
export them if they don't need to be.  There are some in here that
are only used in one or two packages, and that can be cleaned up pretty
easily however it's likely that there might be plans to use them
elsewhere in the future.
2018-09-10 09:14:09 -04:00
David Vossel
7908038b94 Ensure expectation keys are deleted when object is deleted from cluster (#433)
Signed-off-by: David Vossel <davidvossel@gmail.com>
2018-09-07 08:38:28 -04:00
John Griffith
9468ba94c7 Fix golint errors on pkg directory (#414)
Note we are intentionally skipping test files for now, golint
pkg/controller... will skip them, so I'll skip them to.
2018-09-05 12:17:38 -07:00
John Griffith
69709bf863 Update no annotation PVC import test. (#403) (#408)
- Set verbosity on controller to 3 to enable simpler tests.

Signed-off-by: Alexander Wels <awels@redhat.com>
2018-09-05 11:10:24 -07:00
David Vossel
fdb1b27467 Single Queue and Expectations (#249)
* Pull in expectations code from kubevirt

Signed-off-by: David Vossel <davidvossel@gmail.com>

* add expectations to datavolume controller

Signed-off-by: David Vossel <davidvossel@gmail.com>

* Refactor importer pod to use single queue and expectations

Signed-off-by: David Vossel <davidvossel@gmail.com>

* update utils tests to use single queue

Signed-off-by: David Vossel <davidvossel@gmail.com>

* re-introduce import controller test suite

Signed-off-by: David Vossel <davidvossel@gmail.com>

* make clone controllers pass after utils refactor

Signed-off-by: David Vossel <davidvossel@gmail.com>

* Use log level constants in cdi controllers

Signed-off-by: David Vossel <davidvossel@gmail.com>

* Remove useless shadow variables in expectations pkg

Signed-off-by: David Vossel <davidvossel@gmail.com>

* required changes to get owner references working with openshift 1.10

Signed-off-by: David Vossel <davidvossel@gmail.com>

* update functional tests to work with importer controller refactor

Signed-off-by: David Vossel <davidvossel@gmail.com>

* Rename AnnImportPVC to LabelImportPvc in order to reflect its use

Signed-off-by: David Vossel <davidvossel@gmail.com>

* Add comment about expectations code

Signed-off-by: David Vossel <davidvossel@gmail.com>
2018-07-30 12:09:15 -05:00
Jonathan Cope
11841e6d2d Fixed unreachable log line (#263) 2018-07-16 13:10:18 -07:00
David Vossel
7106bf73dc Conform with top level kubevirt github project (#198)
* Place cdi packages under the top level kubevirt.io project path

Signed-off-by: David Vossel <davidvossel@gmail.com>

* Change all references to cdi.io to cdi.kubevirt.io

Signed-off-by: David Vossel <davidvossel@gmail.com>

* Add go_import_path to travis config

Signed-off-by: David Vossel <davidvossel@gmail.com>

* update test package imports

Signed-off-by: David Vossel <davidvossel@gmail.com>
2018-06-21 15:44:52 -07:00
jeff vance
e8233a3786 change cdi anno from to 2018-06-14 13:50:49 -07:00
Jonathan Cope
941e6e8413
Revert "change controller labels from 'kubevirt.io' to 'cdi.io'" 2018-06-14 12:03:27 -05:00
jeff vance
bc3aec2280 change controller labels to 'cdi.io' 2018-06-12 11:16:26 -07:00
David Vossel
9a9f19dcd6 datavolume controller
Signed-off-by: David Vossel <davidvossel@gmail.com>
2018-06-11 21:58:15 -04:00
David Vossel
3e37aead88 Refactor cdi controller in preparation for multiple controllers
Signed-off-by: David Vossel <davidvossel@gmail.com>
2018-06-11 21:58:15 -04:00