This PR removes "skipped" condition for preallocation. Importer/uploader
will preallocate to the available size. Filesystem overhead needs to be
taken into account.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
Instead of (ab)using raw strings, preallocation status in importer and
uploader use now a custom type.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* [WIP] doc: User-facing doc for preallocation support
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* apis: CDI accepts `preallocation` option.
With this commit CDI accepts (but does handle) `preallocation` settings
for DataVolumes and in CDIConfig.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Implementing preallocation
This commit implements preallocation support for import and upload.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Functional tests for preallocation support
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Remove "preallocation for StorageClasses" config
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Removed unused function
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Fix rook-ceph test failures
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Updated dependencies
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Uss PVC annotation to pass preallocation parameters
DataVolume controller now uses a PVC annotation to pass preallocation
configuration to import and update controllers.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Plumb new checkpoint API through to VDDK importer.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add incremental data copy from VDDK.
Create a new data source implementation similar to vddk-datasource, but
only for blocks of data that changed between two snapshots. Also factor
out common things between the two VDDK data sources.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Check block status for warm and cold imports.
Addresses a bunch of runtime issues, but progress tracking isn't right.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Find snapshots correctly.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove separate warm/cold VDDK importers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Advance through the checkpoint list in the spec.
Move DataVolume to Paused after each checkpoint, and start a new
importer pod for the next available checkpoint. Keep track of which
checkpoints have been copied by adding PVC annotations associating each
checkpoint with the UID of the pod that copied it.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Allow spec updates to drive multi-stage imports.
A multi-stage import can create checkpoints at any time, so CDI needs to
be able to receive updates to the list of checkpoints. Implement this by
allowing spec changes only for fields related to multi-stage imports.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Avoid deleting destination in multi-stage import.
A multi-stage import will have an initial data copy to the destination
file followed by separate copies for individual deltas. The destination
file should not be deleted before starting these delta copies.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Get VDDK data source to pass formatting tests.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Unit tests for multi-stage import admission rules.
Make sure only updates to checkpoint-related fields are accepted.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add warm import unit tests for VDDK data source.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add VDDK warm import functional test.
Put two snapshots in the vCenter simulator inventory, and run them
through a multi-stage import process. Also clean up some issues
reported by test-lint.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add some documentation about multi-stage imports.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Pass existing multi-stage DataVolume unit tests.
Also remove MD5 sum step used for debugging, since it can take a long time.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove tabs from documentation.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Pass failing import-controller unit test.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* More unit tests for multi-stage field updates.
Also factor these tests into a DescribeTable.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add nbdkit retry filter.
Available as of Fedora 33 update.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Give correct file name to nbdkit in more cases.
The backing file in the spec might not always match the backing file in
the snapshot, so try harder to match those files by disk ID. May still
need to allow updates to backingFile, depending on how this gets used.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add more unit tests for datavolume-controller.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fix linter error from last commit.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add unit tests for some govmomi API calls.
Move original calls into mock interfaces to make this work.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add libnbd to build.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add libnbd Go bindings to vendor.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add libnbd to importer container image.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Enable cgo builds by changing bazel platform.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Break dependency chain on pkg/importer.
Without this, pretty much every generated binary (including tests.test)
requires libnbd.so.0.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Replace qemu-img with libnbd in VDDK importer.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add progress tracking back to VDDK datasource.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add fadvise before sequential VDDK transfer.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Improve VDDK debug logging and error handling.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Update to libnbd 1.4.1.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Update build container tag.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fix uploadserver/uploadproxy tests.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add a mirror for libnbd RPM, like in #1444.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add libnbd to uploadserver.
This still depends on pkg/importer, so it needs to have libnbd.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Use pointer to data sink, and add final sync.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* When validating disk space, reserve space for filesystem overhead
The amount of available space in a filesystem is not exactly
the advertise amount. Things like indirect blocks or metadata
may use up some of this space. Reserving it to avoid reaching
full capacity by default.
This value is configurable from the CDIConfig object spec,
both globally and per-storageclass.
The default value is 0.055, or "5.5% of the space is
reserved". This value was chosen because some filesystems
reserve 5% of the space as overhead for the root user and
this space doubles as reservation for the worst case
behaviour for unclear space usage. I've chosen a value
that is slightly higher.
This validation is only necessary because we use sparse
images instead of fallocated ones, which was done to have
reasonable alerts regarding space usage from various
storage providers.
---
Update CDIConfig filesystemOverhead status, validate, and
pass the final value to importer/upload pods.
Only the status values controlled by the config controller
are used, and it's filled out for all available storage
classes in the cluster.
Use this value in Validate calls to ensure that some of the
space is reserved for the filesystem overhead to guard from
accidents.
Caveats:
Doesn't use Default: to define the default of 0.055, instead
it is hard-coded in reconcile. It seems like we can't use a
default value.
Validates the per-storageClass values in reconcile, and
doesn't reject bad values.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Use util GetStorageClassByName
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Test filesystem overhead validation against async upload endpoint
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* wait for NFS PVs to be deleted before continuing
Intended to help with flakes, but didn't make a difference.
Probably still worth doing.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Avoid using the uncached client unnecessarily
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add error handling for the case where even a default SC is not found
Note that this change isn't expected to make a difference, as we
check if the targetStorageClass is nil later on and have the same
behaviour, but this is probably more correct API usage.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add testing for the validation of filesystem overhead values
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Fix logical error in waiting for NFS PVs.
Wait for all of them, not just the last one.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add support for a VDDK datasource to the controllers
This commit does not implement the VDDK datasource. It simply
adds the DataVolumeSourceVDDK type and adds support to the
datavolume and import controllers for it. The datasource itself
will need to be done in a follow-up.
Signed-off-by: Sam Lucidi <slucidi@redhat.com>
* Fix tests and clean up lint
Signed-off-by: Sam Lucidi <slucidi@redhat.com>
* Add DataVolumeSourceVDDK to the v1alpha1 api group
Signed-off-by: Sam Lucidi <slucidi@redhat.com>
* Install nbdkit.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add govmomi to vendor directory.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Integrate govmomi into build.
Incomplete! This allows 'find' and 'object' imports from govmomi, but
further dependencies may require more bazel tweaks later.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Create a container image just to hold the VDDK.
Mount the VDDK folder to /opt. Also copy the libxcrypt-compat RPM to
/opt for run-time installation as a temporary workaround for my
inability to install it at container creation time. Make VDDK optional
by storing a dummy file in its directory, so an empty VDDK directory
will not break the build.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fill in VDDK data source.
Add the simplest possible VDDK data source: set up an nbdkit server with
the given VMware parameters, and copy data out using qemu-img.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add vddk to resource validation schema.
Stops "error validating data" messages.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Move libxcrypt-compat install out of VDDK image.
Requiring users to add this to the VDDK image is kind of onerous, so
move it straight into the importer image instead.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Generate BUILD.bazel for the rest of govmomi.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Get VDDK sidecar image from v2v-vmware ConfigMap.
Convention from https://github.com/kubevirt/web-ui-components/pull/534
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Reduce nbdkit startup timeout.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Update govmomi dependencies.
Cleans up 'inconsistent vendoring' errors.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fix schema verification test.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Change disk.raw to disk.img.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add vCenter simulator for VDDK datasource tests.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Functional test with vcsim and dummy VDDK plugin.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Clean up lint errors.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Check kubevirt-hyperconverged namespace for VDDK.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add VDDK unit tests and API functional test.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Read a raw image for VDDK test plugin.
Have the fake VDDK test plugin for nbdkit read a raw image passed in
/opt/updates/nbdtest.img instead of returning fixed data.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add some documentation about VDDK data sources.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove unit test focus and trailing spaces.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Use updated build container image.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add some extra comments about VDDK fields.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Clarify VDDK ConfigMap a little more.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Clarify namespace for vddk-init sidecar.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove NodePort from vCenter simulator manifest.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Re-word explanations on VDDK source types.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add VDDK to comment with types needing URL check.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Rebuild types.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Move AnnThumbprint to storage.import.vddk.thumbprint.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove run-time installation of libxcrypt-compat.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove fixed namespaces for vddk-init reference.
The importer now looks for the v2v-vmware ConfigMap in the current CDI
namespace instead of openshift-cnv/kubevirt-hyperconverged.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Find CDI install namespace from data volume test.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
Co-authored-by: Sam Lucidi <slucidi@redhat.com>
* move upload.cdi.kubevirt.io API group to v1beta1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* move core api to v1beta1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix os-3.11 cluster sync and add functional tests for alpha api
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* change more occurences of v1alpha1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* updates after rebase
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* use dedicated SCC
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* SCC was not getting on initial deploy
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Set secondary GID of user on import and upload pods to be 107 (qemu) for kubevirt content type.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Fix broken unit tests on upload.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Add import and clone test.
Signed-off-by: Alexander Wels <awels@redhat.com>
This PR introduces new import data source to fetch disks from ovirt
and store them in storage available in the cluster. It uses ovirt-sdk
to check whether the this is there, the size of the disk and it obtains
signed ticket which allows to get the disk using ovirt-imageio-proxy.
In order to use it a user needs to create configmap with ovirt api
certificate and secret with ovirt engine user and password. Both are
required and validated by webhook. In order to start import process
we need to post following definition:
---
apiVersion: cdi.kubevirt.io/v1alpha1
kind: DataVolume
metadata:
name: "imageio-dv"
spec:
source:
imageio:
url: "<engine-api-endpoint>"
secretRef: "<secret-name-containing-engine-credentials>"
certConfigMap: "<configmap-name-containing-engine-api-certificate>"
diskId: "<disk-id-which-we-want-to-import>"
pvc:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "64Mi"
---
Signed-off-by: Piotr Kliczewski <piotr.kliczewski@gmail.com>
* baseline refactoring of webhook package
* datavolume clone validation webhook
* rename datavolumes/clone-init to datavolumes/source
* add RBAC doc
* updates from review
* make clone permission check exportable function
* force dumb cloning in in functional test
Changes and flow:
* datavolume-controller:
- When clone pvc is requested and smart-clone is applicable -> Create a new VolumeSnapshot.
* Introduced smart-clone-controller:
- Listens to VolumeSnapshot changes and create a new PVC based on the snapshot.
- Upon PVC successful creation, deletes the snapshot for cleaning up.
Change-Id: I369fc92e72edb8a2cf584c8cb916795415f9d6e0
Signed-off-by: Daniel Erez <derez@redhat.com>
Co-authored-by: Fred Rolland <frolland@redhat.com>
As a user, I want to specify a registry source to pre-populate my PV with a VM disk image.
* Using skopeo[1] cli to copy image from docker registry.
* Extracting image layers (tar files) into data folder.
* Cleaning up .wh. files and remove corresponding files.
* Supporting docker registries such as: docker hub, quay, etc.
[1] https://github.com/containers/skopeo
Change-Id: Ib4e20bb7be842ebaabeb925202402791827edfd3
Signed-off-by: Daniel Erez <derez@redhat.com>
Co-authored-by: Fred Rolland <frolland@redhat.com>
This patch adds the ability to import and extract an archived content
from a tar file directly onto the target PVC.
That is, a tar file is expanded while keeping file ownership,
permissions, and tree structure identical to the source.
Signed-off-by: Idan Shaby <ishaby@redhat.com>
Combined code from PR#489 and PR#490 by
@gites and
@danielerez
Added some tests and rebased on current master.
Signed-off-by: Alexander Wels <awels@redhat.com>
Fixed a couple of typos in common.go file.
```release-note
NONE
```
Change-Id: I88e911ef85879f32dcd8f8685dc7b71495cd50fe
Signed-off-by: Daniel Erez <derez@redhat.com>
This change just updates the common package to pass golint. Of course
that has some reaching implications into other packages with renaming of
constants.
I've intentionally kept this patch set to the bare minimum for the lint
test, I do think that we should put some effort into our use of
constants (especially those in common.go) in the future.
Ideally we'd declare constants where they're used, and we wouldn't
export them if they don't need to be. There are some in here that
are only used in one or two packages, and that can be cleaned up pretty
easily however it's likely that there might be plans to use them
elsewhere in the future.
- Updated controller to use ENTRYPOINT (no cmd) instead of /bin/sh
- Updated Dockerfile to set path to /usr/bin/cloner_startup.sh instead of /tmp
- Updated Dockerfile to use ENTRYPOINT.
Fixed issue #374
Signed-off-by: Alexander Wels <awels@redhat.com>
