* Remove older nbdkit
Signed-off-by: Alexander Wels <awels@redhat.com>
* When converting always use scratch space importing instead
of ndbkit. Once we are able to get nbdkit 1.35.8 or newer
we can revert this change since that will include improvements
to the downloading speed.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Disable metrics test for import because straight import doesn't
return total, and this means the metrics are disabled.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Fix broken functional tests
Signed-off-by: Alexander Wels <awels@redhat.com>
* Address review comments
Signed-off-by: Alexander Wels <awels@redhat.com>
* Additional review comments.
Fixed functional test that was not doing the right
thing while running the test.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Always set preallocation on block devices when directly
writing to the device
Signed-off-by: Alexander Wels <awels@redhat.com>
---------
Signed-off-by: Alexander Wels <awels@redhat.com>
* Run `make deps-update`
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Update to ginkgo v2
Avoid using table extension to avoid compilation errors
Switch to v2 everywhere
Update qe-tools as well (required)
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Fix/avoid deprecation warnings
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Do not use v1 reporter
For unit tests: stop using custom reporter, unnecessary
For functional tests: borrow code from kubevirt to keep reporting
Avoid deprecated warnings by golangci for using deprecated reporter
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Increase ginkgo timeout to 24h (default in ginkgo v1)
this may seem excessive, but we have a lower timeout in Prow, let's save
ourselves the future trouble of bumping timeouts in two places.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* use the ginkgo built-in junit reporter
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Avoid using deprecated --ginkgo.noColor, use --ginkgo.no-color instead
Signed-off-by: Maya Rashish <mrashish@redhat.com>
---------
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Start adding the golangci-lint to CI
golangci-lint is a collection of many linters. This PR adds
golangci-lint to the CI. For strat, it enables the govet linter, and fix
its single finding.
The PR adds this linter to the `test-lint` Makefile target.
The new .golangci.yml file is the configuration for the linter.
golangci-lint version was set to the latest one - v1.52.2.
It is defined in hack/build/run-linters.sh
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
* golangci-lint: enable gosimple and fix findings
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
* golangci-lint: enable unused and fix findings
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
---------
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
The io/ioutil package has been deprecated as of Go 1.16 [1]. This commit
replaces the existing io/ioutil functions with their new definitions in
io and os packages.
[1]: https://golang.org/doc/go1.16#ioutil
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* remove root worker pods
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove selinux requirement for worker pods
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* run tests in restricted namespace and required changes
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* handle empty tar
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add PSA label when running functional tests in OpenShift
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* cannot use restricted PSA with istio (for now)
refactor scc management
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix clean script
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Don't use nbdkit for xz or gz
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix detection of VDI and VHD
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* update docs
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Update HTTP data source API to allow custom headers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Implement custom HTTP headers API.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Document custom headers in HTTP data source.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Correct secretExtraHeader comment to reference Secret.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add volume mounts for secret headers.
Replaces environment variables for headers from secrets.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Avoid failing when there are no extra headers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Redact contents of headers that come from secrets.
Also split up getExtraHeaders to reduce Sonar Cloud complexity.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Ensure all HTTP client requests use extra headers.
Missed redirect check and content length retrieval, both of which might
need the extra headers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add some unit tests for extra HTTP headers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Do not quote headers in nbdkit curl arguments.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add functional tests for extra HTTP headers.
Avoids new test server by specifiying basic authorization headers to the
existing file host port that requires it.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Use filepath.Walk to read secrets.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Minor documentation update for secrets.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Re-run 'make generate' for verification failure.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* move apis to new staging area
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add script to push to staging
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix lint check and api reference
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* push staging to api repo
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Use nbdkit for imports using scratch space, too
Avoids QEMU curl bugs like:
https://bugs.launchpad.net/qemu/+bug/1916501
As well as being simpler.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Remove test for a bug in QEMU that doesn't exist in nbdkit.
Nbdkit just handles this correctly, looking like a failure.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Expect the URL to be the nbdkit socket
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* refactor: unify nbdkit usage
Nbdkit is used in the http and vddk importer and the code is completely
separated. This PR unifies the usages of nbdkit in a single file. For
the http importer, nbdkit was used together with qemu-img. With this
change nbdkit is started a separate process, and qemu-img read from the
local nbdkit socket.
Signed-off-by: Alice Frosi <afrosi@redhat.com>
* test: adjust the tests
Removed nbdkit together with qemu-img. Hence, we need to adjust the
tests to reflect the new behavior.
Signed-off-by: Alice Frosi <afrosi@redhat.com>
* Add nbdkit with qemu-img
Nbdkit can be used to stream directly the content of the image together
with qemu-img. It supports different plugins and filters.
Signed-off-by: Alice Frosi <afrosi@redhat.com>
* Use nbdkit to the http importer
Add nbdkit to stream directly the image content avoiding the scratch
space for the http importer.
Signed-off-by: Alice Frosi <afrosi@redhat.com>
* Adjust functional test cases with nbdkit
With the introduction of nbdkit, certain error messages do not match anymore,
and needed to be adjusted to the new behavior.
The scratch space is not needed anymore when the source is http for the majority of the cases.
Certain test cases checked the use of the scratch space, and those are not valid anymore.
Signed-off-by: Alice Frosi <afrosi@redhat.com>
This phase can mean three things:
HTTP, imageio, s3, upload: Go directly to Convert
VDDK: unreachable code, points to arbitrary other phase.
registry: do minor processing after transfer.
Only registry makes actual use of this phase.
Point all current users directly to Convert, and fold the
work done in the registry Process phase into the previous
phase.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Combine tests in bad_webserver.go into a table
No functional change intended.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Use a QCOW2 image for testing badserver.
This has the benefit of going through qemu-img for conversion,
which can introduce more bugs.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Don't pass URLs to qemu-img if Accept-Ranges header isn't bytes.
Adapt unit tests so they don't also test this feature.
Add a handler to bad-webserver to do a very plain HTTP response
using a downloaded file from cdi-file-host. This one doesn't
come with builtin support for range requests.
Use this handler to test that CDI can still import images needing
conversion.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Invert polarity and test for "none".
It's highly unlikely that a value other than bytes becomes a
valid choice, but let's not rule that out.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* move upload.cdi.kubevirt.io API group to v1beta1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* move core api to v1beta1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix os-3.11 cluster sync and add functional tests for alpha api
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* change more occurences of v1alpha1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* updates after rebase
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Tolerate failing to find a Content-Length via HEAD requests.
Some servers don't accept HEAD requests, and Content-Length is
an optional field anyway.
In such cases, qemu-img won't be able to do a streaming
conversion of the URL, so download using our Go download code
to scratch space and convert the resulting file.
Use the field "brokenForQemuImg" to note this scenario.
This fixes creating a DataVolume using one of our examples,
manifests/example/import-kubevirt-datavolume.yaml
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Don't bother with basic auth, this test isn't about that
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Test for reasonable behaviour if HEAD requests are forbidden
Make sure that we mark it broken for qemu-img, but still succeed
in continuing.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Use %v to print err
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Fix linter failure
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* initial client upgrade to 1.16
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix Route detection in OpenShift
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
