This commit adds a test that verifies, that filesystem overhead is taken
into account when creating a blank image (with preallocation).
Also, some test cleanup.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Validate images fit on a filesystem in more cases.
Background:
When the backing store is a filesystem, we store the images
as sparse files. So the file may eventually grow to be bigger
than the available storage. This will cause unfortunate
failures down the line.
Prior to this commit, we validated the size:
- In case the backing store implicitly did it for us (block volumes)
- On async upload
- When resizing (by the operation failing if the image cannot fit
in the available space).
The Resize phase is encountered quite commonly:
Transfer->Convert->Resize
TransferFile->Resize
Adding validation here for the non-resize case covers almost all
the cases.
The only exceptions that aren't validated now are:
- DataVolumeArchive via the HTTP datasource
- VDDK
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* When resizing, take into account filesystem overhead.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add testing for too large upload/import
- Import/sync upload of too large physical size image (raw.xz, qcow2)
- Import/sync upload of too large virtual size image (raw.xz, qcow2)
- Import of a too large raw image file, if filesystem overhead is
taken into account
- Async upload of too large physical size qcow2.
The async upload cases do not mirror the sync upload ones because if
a block device is used as scratch space, it will hit a size limit
before the validation pause, and fail differently.
This scenario is identical to the sync upload case which was added.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Refactor code in a way that requires less comments to explain.
We can just validate that the requested image size will fit in the
available space, and not rely on the fact we typically resize the
images to the full size.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* When calculating usable space, round down to a multiple of 512.
Our validation is indirectly:
image after resize to usable space <= usable space
For this to pass, we need to ensure that qemu-img's rounding
up to 512 doesn't change the size.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Adjust qemu-img to the ones emitted by nbdkit:
- In some cases, we clearly don't rely on the qemu-img error,
so don't check for it.
- In one case, switch it to looking for the nbdkit equivalent
error message.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* [WIP] doc: User-facing doc for preallocation support
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* apis: CDI accepts `preallocation` option.
With this commit CDI accepts (but does handle) `preallocation` settings
for DataVolumes and in CDIConfig.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Implementing preallocation
This commit implements preallocation support for import and upload.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Functional tests for preallocation support
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Remove "preallocation for StorageClasses" config
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Removed unused function
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Updated dependencies
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Uss PVC annotation to pass preallocation parameters
DataVolume controller now uses a PVC annotation to pass preallocation
configuration to import and update controllers.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Implementing preallocation
This commit implements preallocation support for import and upload.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Make sure all import/upload paths do preallocation
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
- D/S the importer pod has verbosity=1 => the preallocation messages are not present in log
- Due to this, we hit an infinite loop in the preallocation tests
(Import completed without preallocation message => pod doesn't exist anymore but sampling continues)
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* [WIP] doc: User-facing doc for preallocation support
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* apis: CDI accepts `preallocation` option.
With this commit CDI accepts (but does handle) `preallocation` settings
for DataVolumes and in CDIConfig.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Implementing preallocation
This commit implements preallocation support for import and upload.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Functional tests for preallocation support
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Remove "preallocation for StorageClasses" config
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Removed unused function
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Fix rook-ceph test failures
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Updated dependencies
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Uss PVC annotation to pass preallocation parameters
DataVolume controller now uses a PVC annotation to pass preallocation
configuration to import and update controllers.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Add CDIConfigSpec to CDI
Make CDIConfig singleton mirror data in "active" CDI
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix functional tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Wait for the CDIConfig Status to be updated, too
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add test IDs and a couple of positive fs overhead test cases
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* When validating disk space, reserve space for filesystem overhead
The amount of available space in a filesystem is not exactly
the advertise amount. Things like indirect blocks or metadata
may use up some of this space. Reserving it to avoid reaching
full capacity by default.
This value is configurable from the CDIConfig object spec,
both globally and per-storageclass.
The default value is 0.055, or "5.5% of the space is
reserved". This value was chosen because some filesystems
reserve 5% of the space as overhead for the root user and
this space doubles as reservation for the worst case
behaviour for unclear space usage. I've chosen a value
that is slightly higher.
This validation is only necessary because we use sparse
images instead of fallocated ones, which was done to have
reasonable alerts regarding space usage from various
storage providers.
---
Update CDIConfig filesystemOverhead status, validate, and
pass the final value to importer/upload pods.
Only the status values controlled by the config controller
are used, and it's filled out for all available storage
classes in the cluster.
Use this value in Validate calls to ensure that some of the
space is reserved for the filesystem overhead to guard from
accidents.
Caveats:
Doesn't use Default: to define the default of 0.055, instead
it is hard-coded in reconcile. It seems like we can't use a
default value.
Validates the per-storageClass values in reconcile, and
doesn't reject bad values.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Use util GetStorageClassByName
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Test filesystem overhead validation against async upload endpoint
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* wait for NFS PVs to be deleted before continuing
Intended to help with flakes, but didn't make a difference.
Probably still worth doing.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Avoid using the uncached client unnecessarily
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add error handling for the case where even a default SC is not found
Note that this change isn't expected to make a difference, as we
check if the targetStorageClass is nil later on and have the same
behaviour, but this is probably more correct API usage.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add testing for the validation of filesystem overhead values
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Fix logical error in waiting for NFS PVs.
Wait for all of them, not just the last one.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* update k8s deps to 1.18.6 and controller runtime to 0.6.2
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove building code generators from docker image. This way the k8s ligray version only has to be updated in go.mod
Do more stuff in the bazel container. Faster and better interop
Fix unit tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* make format
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove unnecessary rsync
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* redo code generator dep management
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* builder uses go modules
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Add support for external openshift cluster, in particular CRC.
make cluster-sync, and functional tests should all work.
Added documentation on how to enable CRC to work.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Updates based on review.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Changes based on review comments:
- removed registry in favor of making people use an external registry.
- added ceph for external provider.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Fix review comments
Signed-off-by: Alexander Wels <awels@redhat.com>
* Update cdi-builder to use go version 1.14.6 (current latest)
Fix functional test framework to work with go >= 1.13. We were not
using the ginkgo ordering correctly, and our framework would call
flag.Parse() too soon, and mess with the go testing flag parsing.
As a result we had to alter some of the variables that we use to build
urls to be functions so that when called the required information is
available.
Fixed several tests not checking for errors.
Fixed dumping code being initialized several times messing with the
order in which failures were reported.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Fixed failing tests.
Signed-off-by: Alexander Wels <awels@redhat.com>
It might happen that `kill 1` is executed so fast that the parent process tries to exit and kills
the child process - `bash` which actually runs the `kill 1` command. In effect our process exits
with code 137. In case of this test it is not a problem, we expect process to kill parent.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Set the WaitForFirstConsumer phase on DataVolume when storage uses the WaitForFirstConsumer binding mode and is not bound yet.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Skip PVC if not bound in import|clone|upload controllers.
This is done so the VM pod(not the cdi pod) will be the first consumer, and the PVC can be scheduled on the same location as the pod.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
fixup! Skip PVC if not bound in import|clone|upload controllers.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update importer tests to force bind the PCV by scheduling a pod for pvc, when storage class is wffc.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update datavolume tests to force bind the PCV by scheduling a pod for pvc, when storage class is wffc.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update upload controller and upload tests to correctly handle force binding the PCV by scheduling a pod for pvc, when storage class is wffc.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update clone tests to force bind the PCV by scheduling a pod for pvc when the storage class is wffc.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update cloner multi-node tests to force bind the PCV by scheduling a pod for pvc when storage class is wffc.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Correct after automerge
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Improve/simplify tests
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Fix error in import test.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update transport_test,operator_test.go
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update rbac_test.go and leaderelection_test.go
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Improve Datavolume and PVC Checks for WFFC.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handle wffc only if feature gate is open - import-controller
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* TEST for Handle wffc only if feature gate is open - import-controller - TEST
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handle wffc only if feature gate is open - upload-controller with test
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* rename and simplify checks
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* cleanup after rebase
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* update tests after rebase
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* update tests after rebase
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* more cleanups
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Document new WFFC behavior
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Document new HonorWaitForFirstConsumer option
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* update docs according to comments
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* extract common function, cleanup - code review fixes
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* add comment for another pr - 1210, so it can have easier merge/rebase
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* typo
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Simplify getStoragebindingMode - code review comments
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Add FeatureGates interface - code review fix
Additionally pass the features gates instead of the particular feature gate value,
and let shouldReconcilePVC decide what to do with the feature gate. That way shouldReconcilePVC
contains all the logic, and the caller does not need to do additional calls to provide parameters.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update matcher
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Remove unused calls to utils.UploadPodName.
This function has no side effects, it just returns a name.
We don't use the return value, so this is a no-op
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Use utils.UploadPodName rather than hard-coding the name.
No functional change expected.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Combine datavolume name tests into a single table.
They seem to vary only by the DataVolume name.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Fix lint failures
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* move upload.cdi.kubevirt.io API group to v1beta1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* move core api to v1beta1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix os-3.11 cluster sync and add functional tests for alpha api
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* change more occurences of v1alpha1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* updates after rebase
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* use go package to properly handle CORS requests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add Origin header to upload requests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* verifier pods hanging around keeping cloning from working
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* GetDiskGroup will wait for pod to actually be deleted if requested
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Handle labels length correctly
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handle service name generation correctly
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Remove not needed labels
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Store import pod name in annotation
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Enable long DV name
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handle name with dot when creating service/label name
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Test long names on import, upload and clone
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Store upload pod name in annotation
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Store importer scratch pvc name in annotation
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Quick fix for tests (need improvements)
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Cleanup handling scratch name
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Ensure pod/service name conflicts are handled
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handle client errors when trying to get the import pod
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Style improvements, and other code review fixes.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Store clone source pod name in an annotation
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Correct name initialization and tests
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Do not init name if pod already exists. It is not needed.
The situation of having a pod but not name on annotation can happen after the upgrade, when we have a legacy pvc and pod already existing, but clone operation not finished. But when we already have the pod, then in the code (currently) we do not need the name from annotation.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Cleanup scratch name handling
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Use constant for max dv name in validation
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Simplify clone source pod name initialization
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Set secondary GID of user on import and upload pods to be 107 (qemu) for kubevirt content type.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Fix broken unit tests on upload.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Add import and clone test.
Signed-off-by: Alexander Wels <awels@redhat.com>
(Leaving in some of the checks introduced in that change)
Having sparse disks works better with external storage being
monitored for low capacity. If we allocate the full size at
start, it shows up as a disk capacity alert immediately.
Another motivation to pre-allocate was that cloning would
sometimes fail for capacity reasons.
We now validate available size before cloning, so we don't
expect that problem to return.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
