* Don't use nbdkit for xz or gz
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix detection of VDI and VHD
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* update docs
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Update HTTP data source API to allow custom headers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Implement custom HTTP headers API.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Document custom headers in HTTP data source.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Correct secretExtraHeader comment to reference Secret.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add volume mounts for secret headers.
Replaces environment variables for headers from secrets.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Avoid failing when there are no extra headers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Redact contents of headers that come from secrets.
Also split up getExtraHeaders to reduce Sonar Cloud complexity.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Ensure all HTTP client requests use extra headers.
Missed redirect check and content length retrieval, both of which might
need the extra headers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add some unit tests for extra HTTP headers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Do not quote headers in nbdkit curl arguments.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add functional tests for extra HTTP headers.
Avoids new test server by specifiying basic authorization headers to the
existing file host port that requires it.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Use filepath.Walk to read secrets.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Minor documentation update for secrets.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Re-run 'make generate' for verification failure.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* move apis to new staging area
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add script to push to staging
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix lint check and api reference
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* push staging to api repo
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Use nbdkit for imports using scratch space, too
Avoids QEMU curl bugs like:
https://bugs.launchpad.net/qemu/+bug/1916501
As well as being simpler.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Remove test for a bug in QEMU that doesn't exist in nbdkit.
Nbdkit just handles this correctly, looking like a failure.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Expect the URL to be the nbdkit socket
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* refactor: unify nbdkit usage
Nbdkit is used in the http and vddk importer and the code is completely
separated. This PR unifies the usages of nbdkit in a single file. For
the http importer, nbdkit was used together with qemu-img. With this
change nbdkit is started a separate process, and qemu-img read from the
local nbdkit socket.
Signed-off-by: Alice Frosi <afrosi@redhat.com>
* test: adjust the tests
Removed nbdkit together with qemu-img. Hence, we need to adjust the
tests to reflect the new behavior.
Signed-off-by: Alice Frosi <afrosi@redhat.com>
* Add nbdkit with qemu-img
Nbdkit can be used to stream directly the content of the image together
with qemu-img. It supports different plugins and filters.
Signed-off-by: Alice Frosi <afrosi@redhat.com>
* Use nbdkit to the http importer
Add nbdkit to stream directly the image content avoiding the scratch
space for the http importer.
Signed-off-by: Alice Frosi <afrosi@redhat.com>
* Adjust functional test cases with nbdkit
With the introduction of nbdkit, certain error messages do not match anymore,
and needed to be adjusted to the new behavior.
The scratch space is not needed anymore when the source is http for the majority of the cases.
Certain test cases checked the use of the scratch space, and those are not valid anymore.
Signed-off-by: Alice Frosi <afrosi@redhat.com>
This phase can mean three things:
HTTP, imageio, s3, upload: Go directly to Convert
VDDK: unreachable code, points to arbitrary other phase.
registry: do minor processing after transfer.
Only registry makes actual use of this phase.
Point all current users directly to Convert, and fold the
work done in the registry Process phase into the previous
phase.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Combine tests in bad_webserver.go into a table
No functional change intended.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Use a QCOW2 image for testing badserver.
This has the benefit of going through qemu-img for conversion,
which can introduce more bugs.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Don't pass URLs to qemu-img if Accept-Ranges header isn't bytes.
Adapt unit tests so they don't also test this feature.
Add a handler to bad-webserver to do a very plain HTTP response
using a downloaded file from cdi-file-host. This one doesn't
come with builtin support for range requests.
Use this handler to test that CDI can still import images needing
conversion.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Invert polarity and test for "none".
It's highly unlikely that a value other than bytes becomes a
valid choice, but let's not rule that out.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* move upload.cdi.kubevirt.io API group to v1beta1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* move core api to v1beta1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix os-3.11 cluster sync and add functional tests for alpha api
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* change more occurences of v1alpha1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* updates after rebase
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Tolerate failing to find a Content-Length via HEAD requests.
Some servers don't accept HEAD requests, and Content-Length is
an optional field anyway.
In such cases, qemu-img won't be able to do a streaming
conversion of the URL, so download using our Go download code
to scratch space and convert the resulting file.
Use the field "brokenForQemuImg" to note this scenario.
This fixes creating a DataVolume using one of our examples,
manifests/example/import-kubevirt-datavolume.yaml
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Don't bother with basic auth, this test isn't about that
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Test for reasonable behaviour if HEAD requests are forbidden
Make sure that we mark it broken for qemu-img, but still succeed
in continuing.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Use %v to print err
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Fix linter failure
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* initial client upgrade to 1.16
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix Route detection in OpenShift
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
