* Add ExtraArgs field to VDDK CRD.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add tests for VDDK ExtraArgs field.
Add one unit test and rework existing functional test into a table for
both annotation and field entries.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Update ExtraArgs DataVolume documentation.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
---------
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add annotation for extra VDDK library arguments.
The VDDK library itself accepts infrequently-used arguments in a
configuration file, and some of these arguments have been tested to show
a significant transfer speedup in some environments. This adds an
annotation that references a ConfigMap holding the contents of this VDDK
configuration file, and mounts it to the importer pod. The first file in
the mounted directory is passed to the VDDK.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add functional test for VDDK args annotation.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add unit test for extra VDDK arguments annotation.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add documentation for extra VDDK arguments.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Simplify new functional test annotation creation.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Look for specific file instead of first file.
Instead of listing the mounted VDDK arguments directory and filtering
out hidden files, just hard-code the expected file name and ConfigMap
key.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Move extra VDDK arguments functional test.
Put this in import_test and assert the values there, instead of in the
VDDK test plugin. The VDDK plugin logs the given values, and then the
test scans the log for what it expects to see.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Clean up lint error.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Move VDDK configuration test back, change test ID.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Avoid using kubectl for scanning nbdkit logs.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Temporary: show whole nbdkit log after failure.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Revert "Temporary: show whole nbdkit log after failure."
This reverts commit 488849f8fd.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Copy extra VDDK args annotation for populators.
Also add a related unit test.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Correct VDDK args config map mount comment.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
---------
Signed-off-by: Matthew Arnold <marnold@redhat.com>
After several releases with GC disabled by default, we decided to
deprecate it, as unfortunately it violates fundamental principle of
Kubernetes. CR should not be auto-deleted when it completes its role
(Job with TTLSecondsAfterFinished is an exception), and once CR was
created we can assume it is there until explicitly deleted. In addition,
CR should keep idempotency, so the same CR manifest can be applied
multiple times, as long as it is a valid update (e.g. DataVolume
validation webhook does not allow updating the spec).
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Docs: Replace instances of pvc with storage
All instances of DataVolume > Spec > PVC have been replaced with storage,
except for two cases in datavolumes.md:
- Data Volumes>Target Storage/PVC>PVC
https://github.com/kubevirt/containerized-data-importer/blob/338bafe/doc/datavolumes.md#pvc
- Data Volumes>Source>PVC source
https://github.com/kubevirt/containerized-data-importer/blob/338bafe/doc/datavolumes.md#pvc-source
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Docs: Replace `Storage` with `storage` in doc/datavolumes.md
I also reworded it so that the paragraph does not start in lower-case.
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Docs: Remove optional default 'accessModes: ReadWriteOnce' from docs
Removed from storage options, I still kept it for the pvc.
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Docs: Remove resources/requests/storage when cloning a PVC
This argument is optional. Only removed for 'storage', not 'pvc'.
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Replace 'pvc' with 'storage' API in manifests
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Docs: Moved 'Target Storage/PVC' section above 'Source'
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Docs: Rephrase the Storage target section to emphasize it over PVC
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Fix empty field in YAML file
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Rephrase prose around filesystem volume mode
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Fix grammar and spelling
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Write acces mode in upgrade-testing-artifacts manifest
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
* Update doc/datavolumes.md
Co-authored-by: Alexander Wels <awels@redhat.com>
Signed-off-by: Edu Gómez Escandell <edu1997xyz@gmail.com>
---------
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
Signed-off-by: Edu Gómez Escandell <edu1997xyz@gmail.com>
Co-authored-by: Alexander Wels <awels@redhat.com>
* Mock NBD functions in VDDK unit tests.
Also add some example tests for GetBlockStatus.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Avoid infinite loop by returning whole block.
Also add a unit test to trigger this code. Without the fix, this spins
"No new block status data" messages forever, as reported in the bug.
With the fix, this continues the data transfer without zero-range or
hole-punch optimizations.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Correctly extend merged block length.
Bring in the previously proposed fix to make sure merged blocks have
correct lengths, avoiding the initial issue.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add two more GetBlockStatus unit tests.
Hopefully this makes it more obvious what GetBlockStatus actually does.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add unit tests for larger block sizes.
The first test intentionally overflows the result block's length field
and causes an infinite loop in GetBlockStatus, as pointed out in the
problem report. This will be fixed in the next commit.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Avoid integer overflow and sort out casts.
Increase the size of BlockStatusData.Length to an int64, to avoid
overflow in GetBlockStatus. Also change BlockStatusData.Offset to an
int64 and remove a handful of unnerving integer conversions. Favor int64
over uint64 to match system libraries, and add a few necessary
conversions mostly isolated to libnbd interfaces.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fix alignment for multi-stage VDDK documentation.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
---------
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Disable DV GC by default
DataVolume garbage collection is a nice feature, but unfortunately it
violates fundamental principle of Kubernetes. CR should not be
auto-deleted when it completes its role (Job with TTLSecondsAfter-
Finished is an exception), and once CR was created we can assume it is
there until explicitly deleted. In addition, CR should keep idempotency,
so the same CR manifest can be applied multiple times, as long as it is
a valid update (e.g. DataVolume validation webhook does not allow
updating the spec).
When GC is enabled, some systems (e.g GitOps / ArgoCD) may require a
workaround (DV annotation deleteAfterCompletion = "false") to prevent
GC and function correctly.
On the next kubevirt-bot Bump kubevirtci PR (with bump-cdi), it will
fail on all kubevirtci lanes with tests referring DVs, as the tests
IsDataVolumeGC() looks at CDIConfig Spec.DataVolumeTTLSeconds and
assumes default is enabled. This should be fixed there.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix test waiting for PVC deletion with UID
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix clone test assuming DV was GCed
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix DIC controller DV/PVC deletion when snapshot is ready
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Google Cloud Storage Importer
This is a Google Cloud Storage importer for CDI
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* Fix auto-generated swagger and openapi
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* GCS Importer General Fixes
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* Moving back gcs-secret.txt
Moving file back to imageDir to fix unit testing.
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
---------
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
Co-authored-by: Marcelo Parisi <marcelo@dev-box.corp.feitoza.com.br>
On the next kubevirt-bot Bump kubevirtci PR (with bump-cdi), it will
fail on all kubevirtci lanes with tests referring DVs, as the tests
IsDataVolumeGC() looks at CDIConfig Spec.DataVolumeTTLSeconds and
assumes default is disabled. This should be fixed there.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix initImageURL example.
Trying the example as written does not work, there should be no HTTP
prefix.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Correct VDDK search path.
Remove LD_LIBRARY_PATH and point to base vmware-vix-disklib-distrib
directory as recommended in nbdkit documentation. Without this, CDI's
version of nbdkit fails with:
"/opt/vmware-vix-disklib-distrib/lib64/lib64/libvixDiskLib.so.7: cannot
open shared object file: No such file or directory"
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add sqlite-libs to VDDK importer extras.
Without this, VDDK imports cause nbdkit to fail with:
"libsqlite3.so.0: cannot open shared object file: No such file or
directory"
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add cacheextents filter to VDDK imports.
This reduces the time it takes to get VDDK transfers started, in some
cases pretty significantly (~25 seconds down to ~5 seconds in one test).
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fix lint error.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Allow empty DV size when cloning using storage API
When cloning a Data Volume, the size of the target can be potentially obtainable via the source PVC, which discards the need to explicitly specify it.
Considering that, this commit introduces a change in the correspondent validation webhook to allow omitting the resources.request.storage field when cloning a PVC using the storage API.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Modify datavolume-controller to allow obtaining storage size from source PVC when cloning
When cloning a PVC, if the target's size is not specified, said value can be attainable from the source PVC.
This commit introduces a change in datavolume controller so, in case of detecting an empty storage size, said value can be obtained when performing CSI and Smart cloning.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Update unit tests for datavolume-validation after enabling cloning with empty size
This commit updates the unit testing for the datavolume validation webhook, covering the possibility of cloning a PVC without setting any storage size.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Update unit testing for controller-related functions after enabling cloning with empty size
This commit includes unit tests for the volumeSize() function after enabling creating clones with blank size.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Update the datavolume controller to create a size-detection pod when performing host-assisted clone
When performing a host-assisted clone with empty clone size, simply copying the original PVC size could lead to potential overhead miscalculations if the source's VolumeMode is "filesystem".
When that's the case, an inspection pod will be created in the datavolume controller so it extracts the size of the virtual image using qemu-img.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Include an image-size detection tool to allow cloning with empty DV size
This commit introduces a new tool in charge of collecting the virtual image size when cloning with an empty DV size. In some cases where said value is unattainable from the original PVC's spec, the datavolume controller will create a new pod containing this new tool.
The binary will then run the 'qemu-img' command and handle its results appropriately.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Optimize the clone-size lookup process to avoid creating unnecessary size-detection pods
When performing host-assisted clone with an empty DV size, in some cases, a size-detection pod is used to obtain the required capacity.
This commit tries to optimize this process to keep the collected value as a PVC annotation, that is checked in subsequent clones to avoid creating redundant pods.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Minor fixes and improvements on mechanism for cloning with empty storage size
* Add new optional flag on size-detection binary to enable using a different URI scheme
* Improve the pod-creation mechanism so the pod is not created until the source PVC has finished the import
* Modify size-finlation mechanism to account for possible round-downs when importing the source image
* Improve the size inflation mechanism so only PVCs with filesystem as volume mode are considered
* Minor style corrections
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Modify the clone-controller to allow skipping the clone size validation in some cases
Due to filesystem overhead differences, the target's size can sometimes be smaller than the source's one when obtaining said value with the size-detection pod.
This commit introduces minor changes in the clone-controller so we can skip the size validation in those cases.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Minor changes and improvements in size-detection mechanism following PR review
* Added new UT that covers using empty storage API for non-cloning sources
* Added new watch on datavolume-controller that looks for changes in the size-detection pod
* Removed redundant and unnecessary specs on size-detection pod
* Added error handling when reading the pod's termination message
* Moved general-usage functions to 'util.go' file
* Updated 'datavolumes' documentation to reference the possibility of omitting the storage size when cloning
* Minor style corrections
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add unit tests that cover the size-detection mechanism in the DataVolume controller
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Include functional tests for cloning without specifying storage size
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Improve error handling in the creation/deletion process of the size-detection pod
This commit introduces additional handling in case of error after and during the size-detection pod is created.
It also updates several related unit tests.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Minor fixes to improve fsOverhead calculations when cloning with empty storage size
* Modified the size-detection mechanism so we account for fsOverhead when cloning to filesystem volume mode in all cases
* Clean up the code for reconciling when cloning a PVC that is not ready
* Minor fix in functional test so it works when cloning from block to filesystem volume mode
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add optional VDDK initImageURL field.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Pass VDDK image URL through to PVC annotation.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Unit tests for per-DV VDDK image URL.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Functional test for VDDK initImageURL field.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Update documentation for VDDK initImageURL.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fix lint error.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Check for absence of AwaitingVDDK in unit test.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Update HTTP data source API to allow custom headers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Implement custom HTTP headers API.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Document custom headers in HTTP data source.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Correct secretExtraHeader comment to reference Secret.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add volume mounts for secret headers.
Replaces environment variables for headers from secrets.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Avoid failing when there are no extra headers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Redact contents of headers that come from secrets.
Also split up getExtraHeaders to reduce Sonar Cloud complexity.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Ensure all HTTP client requests use extra headers.
Missed redirect check and content length retrieval, both of which might
need the extra headers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add some unit tests for extra HTTP headers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Do not quote headers in nbdkit curl arguments.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add functional tests for extra HTTP headers.
Avoids new test server by specifiying basic authorization headers to the
existing file host port that requires it.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Use filepath.Walk to read secrets.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Minor documentation update for secrets.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Re-run 'make generate' for verification failure.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* VDDK: get changed disk areas from change IDs.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* VDDK: Add unit test for change IDs.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Update documentation for VDDK change IDs.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fix lint errors.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add qemu-img rebase and commit operations.
Also only fail images with backing files that do not exist, so that
ImageIO snapshots can be downloaded and applied to a base disk image.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add merge phase to data processor.
This keeps qemu-img details out of the ImageIO data source.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Beef up transfer ticket finalization/cancellation.
Snapshots seem to be more prone to getting locked indefinitely than
disks if not correctly finalized or cancelled, so do this more carefully
than before.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Allow downloading snapshots from ImageIO.
Download the first snapshot as a raw whole-disk image, and download
subsequent snapshots as QCOW images to be committed to that base.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Allow multi-stage fields on ImageIO data sources.
Also avoid removing base disk image when cleaning data directory.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add ImageIO multi-stage functional tests.
Pick up fakeovirt update for stub functionality, so inventory responses
can be changed on the fly for individual tests.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Update multi-stage documentation for ImageIO.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Move if-else test block to functions.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Reset ImageIO inventory for a test I missed.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Clean up from some review comments.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Sort out calls to cleanupTransfer.
Failures during the creation of a transfer ticket call the original
cleanupTransfer in a single location, and any exits after the data
source is created call a wrapper function. The wrapper has a lock and a
'done' flag to make sure it is only called once on exit, even when
interrupted from the goroutine that waits for SIGTERM.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* CSI Volume Clone for same namespace
CSI Volume Cloning is available on the same namespace and also
works with namespace transfer and volume expansion.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update documentation for CSI Volume Clone
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Cleanup and refactor - extract common code into functions
Remove csi-clone-controller (only set cloneOf annotation)
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Corrects reconcile results
Do not requeue reconciliation loop when not needed.
Mark DV as Failed when the PVC Claim is lost.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handles PVC recovery from ClaimLost
Make sure that CSI clone continues when target pvc recovers from
ClaimLost to to Bound or Pending.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Code Review improvements
Extracted common code for doCrossNamespaceClone and expandAfterClone, and some updates to comments/cleanups.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Create new Storage type
A new Storage type similar to the PVC Spec is now available to use
in the DataVolume Spec. This is more permissive than PVC, and together
with StorageProfile this allows CDI to apply additional logic for
missing or optional fields.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Use the StorageProfile
Handle the StorageProfile recommended params when creating the PVC for
a DataVolume. When parameters like volumeMode or accessModes are
not provided, CDI checks the StorageProfile for a given StorageClass
to set the recommended defaults. This enables user to create DataVolume
without the need to provide all the parameters.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Allow multiple accessModes
CDI allows multiple access modes to be specified in the DataVolume.spec.storage and in the StorageProfile. This now works the same way as in PVC specification.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handle the storage.size field
The storage.size specifies how much space a user wants to have.
When creating image on the fileSystem storage CDI takes into
account the file system overhead and requests PVC big enough to
fit an image and file system metadata.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Test storage profile with DV
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Document Storage Profiles
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Refactor: Render the effective PVC early
The helper 'render PVC' was moved earlier in the control flow, so
it can be used in more places. Removing the need for if/else logic.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Test handling size on import, upload and clone
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Code Review: Refactor resolving of volumeMode
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Fix: render target pvc spec correctly in smart clone controller
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
Describes new annotation for skipping WaitForFirstConsumer handling, and adds WaitForFirstConsumer phase to DV docs.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Plumb new checkpoint API through to VDDK importer.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add incremental data copy from VDDK.
Create a new data source implementation similar to vddk-datasource, but
only for blocks of data that changed between two snapshots. Also factor
out common things between the two VDDK data sources.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Check block status for warm and cold imports.
Addresses a bunch of runtime issues, but progress tracking isn't right.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Find snapshots correctly.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove separate warm/cold VDDK importers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Advance through the checkpoint list in the spec.
Move DataVolume to Paused after each checkpoint, and start a new
importer pod for the next available checkpoint. Keep track of which
checkpoints have been copied by adding PVC annotations associating each
checkpoint with the UID of the pod that copied it.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Allow spec updates to drive multi-stage imports.
A multi-stage import can create checkpoints at any time, so CDI needs to
be able to receive updates to the list of checkpoints. Implement this by
allowing spec changes only for fields related to multi-stage imports.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Avoid deleting destination in multi-stage import.
A multi-stage import will have an initial data copy to the destination
file followed by separate copies for individual deltas. The destination
file should not be deleted before starting these delta copies.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Get VDDK data source to pass formatting tests.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Unit tests for multi-stage import admission rules.
Make sure only updates to checkpoint-related fields are accepted.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add warm import unit tests for VDDK data source.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add VDDK warm import functional test.
Put two snapshots in the vCenter simulator inventory, and run them
through a multi-stage import process. Also clean up some issues
reported by test-lint.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add some documentation about multi-stage imports.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Pass existing multi-stage DataVolume unit tests.
Also remove MD5 sum step used for debugging, since it can take a long time.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove tabs from documentation.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Pass failing import-controller unit test.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* More unit tests for multi-stage field updates.
Also factor these tests into a DescribeTable.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add nbdkit retry filter.
Available as of Fedora 33 update.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Give correct file name to nbdkit in more cases.
The backing file in the spec might not always match the backing file in
the snapshot, so try harder to match those files by disk ID. May still
need to allow updates to backingFile, depending on how this gets used.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add more unit tests for datavolume-controller.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fix linter error from last commit.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add unit tests for some govmomi API calls.
Move original calls into mock interfaces to make this work.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add support for a VDDK datasource to the controllers
This commit does not implement the VDDK datasource. It simply
adds the DataVolumeSourceVDDK type and adds support to the
datavolume and import controllers for it. The datasource itself
will need to be done in a follow-up.
Signed-off-by: Sam Lucidi <slucidi@redhat.com>
* Fix tests and clean up lint
Signed-off-by: Sam Lucidi <slucidi@redhat.com>
* Add DataVolumeSourceVDDK to the v1alpha1 api group
Signed-off-by: Sam Lucidi <slucidi@redhat.com>
* Install nbdkit.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add govmomi to vendor directory.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Integrate govmomi into build.
Incomplete! This allows 'find' and 'object' imports from govmomi, but
further dependencies may require more bazel tweaks later.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Create a container image just to hold the VDDK.
Mount the VDDK folder to /opt. Also copy the libxcrypt-compat RPM to
/opt for run-time installation as a temporary workaround for my
inability to install it at container creation time. Make VDDK optional
by storing a dummy file in its directory, so an empty VDDK directory
will not break the build.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fill in VDDK data source.
Add the simplest possible VDDK data source: set up an nbdkit server with
the given VMware parameters, and copy data out using qemu-img.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add vddk to resource validation schema.
Stops "error validating data" messages.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Move libxcrypt-compat install out of VDDK image.
Requiring users to add this to the VDDK image is kind of onerous, so
move it straight into the importer image instead.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Generate BUILD.bazel for the rest of govmomi.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Get VDDK sidecar image from v2v-vmware ConfigMap.
Convention from https://github.com/kubevirt/web-ui-components/pull/534
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Reduce nbdkit startup timeout.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Update govmomi dependencies.
Cleans up 'inconsistent vendoring' errors.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fix schema verification test.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Change disk.raw to disk.img.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add vCenter simulator for VDDK datasource tests.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Functional test with vcsim and dummy VDDK plugin.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Clean up lint errors.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Check kubevirt-hyperconverged namespace for VDDK.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add VDDK unit tests and API functional test.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Read a raw image for VDDK test plugin.
Have the fake VDDK test plugin for nbdkit read a raw image passed in
/opt/updates/nbdtest.img instead of returning fixed data.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add some documentation about VDDK data sources.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove unit test focus and trailing spaces.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Use updated build container image.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add some extra comments about VDDK fields.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Clarify VDDK ConfigMap a little more.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Clarify namespace for vddk-init sidecar.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove NodePort from vCenter simulator manifest.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Re-word explanations on VDDK source types.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add VDDK to comment with types needing URL check.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Rebuild types.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Move AnnThumbprint to storage.import.vddk.thumbprint.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove run-time installation of libxcrypt-compat.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove fixed namespaces for vddk-init reference.
The importer now looks for the v2v-vmware ConfigMap in the current CDI
namespace instead of openshift-cnv/kubevirt-hyperconverged.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Find CDI install namespace from data volume test.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
Co-authored-by: Sam Lucidi <slucidi@redhat.com>
* move upload.cdi.kubevirt.io API group to v1beta1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* move core api to v1beta1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix os-3.11 cluster sync and add functional tests for alpha api
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* change more occurences of v1alpha1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* updates after rebase
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Add conditions to DataVolume and add condition annotation to PVCs
Signed-off-by: Alexander Wels <awels@redhat.com>
* Added reason codes to conditions
Updated functional tests to use reason codes
Fixed broken functional test.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Added last transition time
Added documentation
Signed-off-by: Alexander Wels <awels@redhat.com>
* Switch to conditions slice from conditions pointer slice.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Add heartbeat time for conditions.
Renamed AnnRunningCondition to AnnLastTerminationMessage
Only update reason/message on heartbeat or state change.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Rename heartBeat to heartbeat
Re-factor datavolume-conditions to properly update heartbeat time.
Signed-off-by: Alexander Wels <awels@redhat.com>
* renamed AnnRunningConditionReason to AnnLastTerminationReason
Signed-off-by: Alexander Wels <awels@redhat.com>
* Revert lastTerminatedMessage/Reason rename
Added separate AnnScratchBoundCondition/Message/Reason for scratch space
Signed-off-by: Alexander Wels <awels@redhat.com>
* Update bound condition to take scratch or source PVC status into consideration.
Update running condition to take source pod status into consideration
Signed-off-by: Alexander Wels <awels@redhat.com>
* Addressed comments in review.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Properly get scratch space pvc phase.
Signed-off-by: Alexander Wels <awels@redhat.com>
Changes and flow:
* datavolume-controller:
- When clone pvc is requested and smart-clone is applicable -> Create a new VolumeSnapshot.
* Introduced smart-clone-controller:
- Listens to VolumeSnapshot changes and create a new PVC based on the snapshot.
- Upon PVC successful creation, deletes the snapshot for cleaning up.
Change-Id: I369fc92e72edb8a2cf584c8cb916795415f9d6e0
Signed-off-by: Daniel Erez <derez@redhat.com>
Co-authored-by: Fred Rolland <frolland@redhat.com>
As a user, I want to specify a registry source to pre-populate my PV with a VM disk image.
* Using skopeo[1] cli to copy image from docker registry.
* Extracting image layers (tar files) into data folder.
* Cleaning up .wh. files and remove corresponding files.
* Supporting docker registries such as: docker hub, quay, etc.
[1] https://github.com/containers/skopeo
Change-Id: Ib4e20bb7be842ebaabeb925202402791827edfd3
Signed-off-by: Daniel Erez <derez@redhat.com>
Co-authored-by: Fred Rolland <frolland@redhat.com>
