* Validate images fit on a filesystem in more cases.
Background:
When the backing store is a filesystem, we store the images
as sparse files. So the file may eventually grow to be bigger
than the available storage. This will cause unfortunate
failures down the line.
Prior to this commit, we validated the size:
- In case the backing store implicitly did it for us (block volumes)
- On async upload
- When resizing (by the operation failing if the image cannot fit
in the available space).
The Resize phase is encountered quite commonly:
Transfer->Convert->Resize
TransferFile->Resize
Adding validation here for the non-resize case covers almost all
the cases.
The only exceptions that aren't validated now are:
- DataVolumeArchive via the HTTP datasource
- VDDK
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* When resizing, take into account filesystem overhead.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add testing for too large upload/import
- Import/sync upload of too large physical size image (raw.xz, qcow2)
- Import/sync upload of too large virtual size image (raw.xz, qcow2)
- Import of a too large raw image file, if filesystem overhead is
taken into account
- Async upload of too large physical size qcow2.
The async upload cases do not mirror the sync upload ones because if
a block device is used as scratch space, it will hit a size limit
before the validation pause, and fail differently.
This scenario is identical to the sync upload case which was added.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Refactor code in a way that requires less comments to explain.
We can just validate that the requested image size will fit in the
available space, and not rely on the fact we typically resize the
images to the full size.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* When calculating usable space, round down to a multiple of 512.
Our validation is indirectly:
image after resize to usable space <= usable space
For this to pass, we need to ensure that qemu-img's rounding
up to 512 doesn't change the size.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Adjust qemu-img to the ones emitted by nbdkit:
- In some cases, we clearly don't rely on the qemu-img error,
so don't check for it.
- In one case, switch it to looking for the nbdkit equivalent
error message.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Updated dependencies
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Replace skopeo with containers API.
This commit removes dependence on skopeo (binary) and uses containers
API. By doing that we're able to opimize the use of storage (scratch)
space, storage I/O and download bandwith.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Fixing rebase - dependencies kerfuffle.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Handling docker-format images as well as OCI.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Fix for missing code-generator module.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Remove regex, image file in registry images are matched by a path
prefix.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Added nginx proxy in front of docker registry for a rate-limited access.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
Modified function that gets the size of a block device/available to return error as well as -1, so we
can distinguish the path not existing from the binary not existing in case the container doesn't have
the required binaries.
Last lane also passed, but due to slow CI timed out before reporting results.
Signed-off-by: Alexander Wels <awels@redhat.com>
(Leaving in some of the checks introduced in that change)
Having sparse disks works better with external storage being
monitored for low capacity. If we allocate the full size at
start, it shows up as a disk capacity alert immediately.
Another motivation to pre-allocate was that cloning would
sometimes fail for capacity reasons.
We now validate available size before cloning, so we don't
expect that problem to return.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* initial client upgrade to 1.16
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix Route detection in OpenShift
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
In a normal Linux container, /sys/class/block/ is inherited from the host, and the result presented by lsblk is the metadata of the host, which does not effectively capture the size of the block device transmitted to the container
Signed-off-by: clarklee92 <clarklee1992@hotmail.com>
Added functional tests to ensure that malicious
qcow2 images are being rejected. I.e. the creation
of import datavolume should fail when converted to raw.
Change-Id: I3372105fdcd28b8e01811d7e6752f04f5dba9e0d
Signed-off-by: Daniel Erez <derez@redhat.com>
Signed-off-by: Fred Rolland <frolland@redhat.com>
Co-authored-by: Fred Rolland <frolland@redhat.com>
Added UnArchiveLocalTar func to util for using in skoepo
(instead of ExtractTar func).
Change-Id: I074400426d3895027a980af79bec0604f97f4c1f
Signed-off-by: Daniel Erez <derez@redhat.com>
Combined code from PR#489 and PR#490 by
@gites and
@danielerez
Added some tests and rebased on current master.
Signed-off-by: Alexander Wels <awels@redhat.com>
- Removed file:// end points from code base.
- Updated unit tests to use httptest server instead of file://
- Refactored data stream unit tests to use ginko/gomega
