* Support registry import using node docker cache
The new CRI (container runtime interface) importer pod is created with three containers and a shared emptyDir volume:
-Init container: copies static http server binary to empty dir
-Server container: container image container configured to run the http binary and serve up the image file in /data
-Client container: import.sh uses cdi-import to import from server container, and writes "done" file on emptydir
-Server container sees "done" file and exits
Thanks mhenriks for the PoC!
Done:
-added ImportMethod to DataVolumeSourceRegistry (DataVolume.Spec.Source.Registry, DataImportCron.Spec.Source.Registry).
Import method can be "skopeo" (default), or "cri" for container runtime interface based import
-added cdi-containerimage-server & import.sh to the cdi-importer container
ToDo:
-utests and func tests
-doc
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add tests, fix CR comments
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Use deployment docker prefix and tag in func tests
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add OpenShift ImageStreams import support
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add importer pod lookup annotation for image streams
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add pullMethod and imageStream doc
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add qemu-img rebase and commit operations.
Also only fail images with backing files that do not exist, so that
ImageIO snapshots can be downloaded and applied to a base disk image.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add merge phase to data processor.
This keeps qemu-img details out of the ImageIO data source.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Beef up transfer ticket finalization/cancellation.
Snapshots seem to be more prone to getting locked indefinitely than
disks if not correctly finalized or cancelled, so do this more carefully
than before.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Allow downloading snapshots from ImageIO.
Download the first snapshot as a raw whole-disk image, and download
subsequent snapshots as QCOW images to be committed to that base.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Allow multi-stage fields on ImageIO data sources.
Also avoid removing base disk image when cleaning data directory.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add ImageIO multi-stage functional tests.
Pick up fakeovirt update for stub functionality, so inventory responses
can be changed on the fly for individual tests.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Update multi-stage documentation for ImageIO.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Move if-else test block to functions.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Reset ImageIO inventory for a test I missed.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Clean up from some review comments.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Sort out calls to cleanupTransfer.
Failures during the creation of a transfer ticket call the original
cleanupTransfer in a single location, and any exits after the data
source is created call a wrapper function. The wrapper has a lock and a
'done' flag to make sure it is only called once on exit, even when
interrupted from the goroutine that waits for SIGTERM.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
Switch admissionregistration.k8s.io/v1beta, apiregistration.k8s.io/v1beta1 to v1
as they are deprecated and will be removed from k8s-1.22.
apiextensions.k8s.io/v1beta1 was updated to v1 by https://github.com/kubevirt/containerized-data-importer/pull/1307.
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Plumb new checkpoint API through to VDDK importer.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add incremental data copy from VDDK.
Create a new data source implementation similar to vddk-datasource, but
only for blocks of data that changed between two snapshots. Also factor
out common things between the two VDDK data sources.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Check block status for warm and cold imports.
Addresses a bunch of runtime issues, but progress tracking isn't right.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Find snapshots correctly.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove separate warm/cold VDDK importers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Advance through the checkpoint list in the spec.
Move DataVolume to Paused after each checkpoint, and start a new
importer pod for the next available checkpoint. Keep track of which
checkpoints have been copied by adding PVC annotations associating each
checkpoint with the UID of the pod that copied it.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Allow spec updates to drive multi-stage imports.
A multi-stage import can create checkpoints at any time, so CDI needs to
be able to receive updates to the list of checkpoints. Implement this by
allowing spec changes only for fields related to multi-stage imports.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Avoid deleting destination in multi-stage import.
A multi-stage import will have an initial data copy to the destination
file followed by separate copies for individual deltas. The destination
file should not be deleted before starting these delta copies.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Get VDDK data source to pass formatting tests.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Unit tests for multi-stage import admission rules.
Make sure only updates to checkpoint-related fields are accepted.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add warm import unit tests for VDDK data source.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add VDDK warm import functional test.
Put two snapshots in the vCenter simulator inventory, and run them
through a multi-stage import process. Also clean up some issues
reported by test-lint.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add some documentation about multi-stage imports.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Pass existing multi-stage DataVolume unit tests.
Also remove MD5 sum step used for debugging, since it can take a long time.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove tabs from documentation.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Pass failing import-controller unit test.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* More unit tests for multi-stage field updates.
Also factor these tests into a DescribeTable.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add nbdkit retry filter.
Available as of Fedora 33 update.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Give correct file name to nbdkit in more cases.
The backing file in the spec might not always match the backing file in
the snapshot, so try harder to match those files by disk ID. May still
need to allow updates to backingFile, depending on how this gets used.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add more unit tests for datavolume-controller.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fix linter error from last commit.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add unit tests for some govmomi API calls.
Move original calls into mock interfaces to make this work.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* move upload.cdi.kubevirt.io API group to v1beta1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* move core api to v1beta1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix os-3.11 cluster sync and add functional tests for alpha api
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* change more occurences of v1alpha1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* updates after rebase
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Handle labels length correctly
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handle service name generation correctly
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Remove not needed labels
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Store import pod name in annotation
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Enable long DV name
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handle name with dot when creating service/label name
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Test long names on import, upload and clone
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Store upload pod name in annotation
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Store importer scratch pvc name in annotation
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Quick fix for tests (need improvements)
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Cleanup handling scratch name
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Ensure pod/service name conflicts are handled
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handle client errors when trying to get the import pod
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Style improvements, and other code review fixes.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Store clone source pod name in an annotation
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Correct name initialization and tests
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Do not init name if pod already exists. It is not needed.
The situation of having a pod but not name on annotation can happen after the upgrade, when we have a legacy pvc and pod already existing, but clone operation not finished. But when we already have the pod, then in the code (currently) we do not need the name from annotation.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Cleanup scratch name handling
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Use constant for max dv name in validation
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Simplify clone source pod name initialization
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
These are basically no-op DataVolumes for PVCs that already exist.
I see this mostly as an internal thing to facilitate VM restore from snapshot
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* allow user with 'create pod' permission in source namespace to clone PVCs also refactored Kubevirt interface to clone permission check
* make sure DataVolume spec is immutable also don't check perms when clone source namespace = dest namespace
* baseline refactoring of webhook package
* datavolume clone validation webhook
* rename datavolumes/clone-init to datavolumes/source
* add RBAC doc
* updates from review
* make clone permission check exportable function
* force dumb cloning in in functional test
