* Add PVC mutating webhook using StorageProfiles
The webhook mutates the PVC Spec based on the available StorageProfiles,
so for example you can create PVC without accessModes and it will be
auto-completed.
To use this feature, enable the `WebhookPvcRendering` feature gate.
For any PVC you want to use StorageProfile, label it with:
cdi.kubevirt.io/useStorageProfile: "true"
If you want to use volumeMode preferred by CDI according to
StorageProfiles, set it to FromStorageProfile. Otherwise if not
explicitly set to Block, it will be Filesystem by k8s default.
E.g.:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-test
labels:
cdi.kubevirt.io/useStorageProfile: "true"
spec:
storageClassName: rook-ceph-block
volumeMode: FromStorageProfile
resources:
requests:
storage: 1Mi
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Move webhook create/delete to callback
plus some CR fixes and cleanups
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Move webhook CR creation to sit with callbacks
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Update existing webhook if modified
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Eliminate unnecessary CR update
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
---------
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
When scratch space is needed during imports/uploads, the pod typically fails so it can be restarted with the required space. This is expected behavior that's automatically handled by CDI.
However, since we use an error exit code, this behavior leads to unwanted VM statuses during provisioning in Kubevirt.
This commit addresses this behavior by using a different DV condition, so kubevirt ignores it and the VM can be provisioned as expected.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
When importing via node container runtime cache, we always have the image handy locally.
This manifests itself in the form of a bug where we loop over
```bash
E0813 13:32:38.443088 1 data-processor.go:251] scratch space required and none found
E0813 13:32:38.443102 1 importer.go:181] scratch space required and none found
```
On registry node pull imports where images are not raw
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Google Cloud Storage Importer
This is a Google Cloud Storage importer for CDI
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* Fix auto-generated swagger and openapi
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* GCS Importer General Fixes
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* Moving back gcs-secret.txt
Moving file back to imageDir to fix unit testing.
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
---------
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
Co-authored-by: Marcelo Parisi <marcelo@dev-box.corp.feitoza.com.br>
* API for VolumeSnapshot clone source
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Refactor doCrossNamespaceClone to get rid of some source PVC assumptions
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Refactor isCrossNamespaceClone to get rid of some some source PVC assumptions
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Clone from VolumeSnapshot source controller
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Func tests for cloning from volumesnapshot source
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Extend cross ns clone token mechanism for VolumeSnapshot cloning
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Validate source volumesnapshot on create
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
---------
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add cron-job sa to scc
Signed-off-by: Alexander Wels <awels@redhat.com>
* Make sure user is added on upgrade
Signed-off-by: Alexander Wels <awels@redhat.com>
Signed-off-by: Alexander Wels <awels@redhat.com>
* Add TLS Security Profile API
TLSSecurityProfile is used by operators to apply cluster-wide TLS security settings to operands.
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Update apiserver & uploadproxy server TLS config on CDIConfig TLS knob change
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Propagate TLS config to uploadserver as well
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add functests for apiserver and upload that ensure value is respected
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Delete erroneous DV on DIC desired digest update
When a DataImportCron import DV is condition Running=False with
Reason=Error it indicates this DIC might get stuck with this DV forever,
so no new import DVs will be created even if the source sha256 is
updated. With this change, when digest is updated, before creating the
new DV, we simply delete the erroneous DV if necessary.
Also includes some DataImportCron tests improvements and cleanup.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix flaky tests
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* More CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Set htpp(s)_proxy to lower case env variable
CURL used by nbdkit doesn't read upper case http(s)_proxy environment
variables, and thus was not using the proxy. Changed the variable to
be lower case.
Added a significant number of tests to test many more variations of
using a proxy. Also added https + auth endpoint to the file-host
container, so we can test https + auth with the proxy.
Added https endpoint to proxy, so we can test an https proxy.
Cleaned up some of the error handling in the import controller for
the proxy, in particular if a trustedCAProxy is defined.
Fixed some of the cluster wide proxy configuration so it works properly
inside an openshift cluster.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Add https proxy support to registry import. Added extra
functional tests to test all registry import combinations
Signed-off-by: Alexander Wels <awels@redhat.com>
* Fixed some tests to work better in Open Shift.
Signed-off-by: Alexander Wels <awels@redhat.com>
We want to silence the KubePersistentVolumeFillingUp for all our PVCs that hold virtual machine disks,
since these disks consume the entire PVC by design.
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add DataImportCron status conditions
The `DataImportCron` controller updates the status conditions in a
controlled `DataImportCron` and its managed `DataSource`.
DataImportCron:
- UpToDate - indicates if the the most recent import is successful and
`DataSource` is up-to-date. Updated to False whenever the source
digest (latest sha256) is updated.
- Progressing - indicates whether the cron is currently in the process
of importing. Updated to True if there is a current import and its
`DataVolume` is `ImportInProgress`, otherwise False.
DataSource:
- Ready - indicates that the corresponding pvc exists and is populated.
Update according to `DataImportCron.Status.LastImportedPVC`
`DataVolume`'s `DataVolumeReady` condition, if the `DataVolume`
exists. Otherwise False. Unlike `DataImportCron` `UpToDate`
condition, this one does not care about newer source digest.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron RetentionPolicy and remove OwnerReferences
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* More CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add tests for retention policies and datasource/datavolume recreation if deleted
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add status condition tests
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* SetRecommendedLabels for all created CRs
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add degraded alert
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add unusual restart count metric
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add actual firing alerts (degraded/restartcount)
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Test newly added metrics
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Review: Rename metric to match conventions, func to check if test is eligible to run metric tests
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Get rid of similar funcs, reconcile more generally
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Update HTTP data source API to allow custom headers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Implement custom HTTP headers API.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Document custom headers in HTTP data source.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Correct secretExtraHeader comment to reference Secret.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add volume mounts for secret headers.
Replaces environment variables for headers from secrets.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Avoid failing when there are no extra headers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Redact contents of headers that come from secrets.
Also split up getExtraHeaders to reduce Sonar Cloud complexity.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Ensure all HTTP client requests use extra headers.
Missed redirect check and content length retrieval, both of which might
need the extra headers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add some unit tests for extra HTTP headers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Do not quote headers in nbdkit curl arguments.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add functional tests for extra HTTP headers.
Avoids new test server by specifiying basic authorization headers to the
existing file host port that requires it.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Use filepath.Walk to read secrets.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Minor documentation update for secrets.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Re-run 'make generate' for verification failure.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add support for archive upload
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* fix golang errors
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Change storage profile property set to support more then one set
So far CDI supported only 1 claim propery set. We want to be able
to support more then one so in case the user provides to the
DV storage volumeMode without accessMode or vice versa cdi
will be able to fit to it the most appropriate match.
Added to rook ceph block a second default of filesystem
volume mode with RWO access mode, it will support archive
upload which has default of filesystem mode.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* CR fix - change to one endpoint for the user
upload proxy will identify if the upload is archive
or not by looking at the content type annotation on
the pvc. If the content type is archive it will route
the uplaod to upload server to a new archive upload uri.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add storage profile and data volume controllers unit tests
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* CR fixes
* add default volume mode to archive content type
* upload server use data processor for archive upload
* tests for volume mode with archive content type
* tests for archive upload of compressed tar
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Adjust imports acording to new apis dir
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* CR small fixes
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Deploy alerts infra as part of our installation
Conditionally deploy the infrastructure that is needed to fire alerts for our users
when bad things are happening to CDI.
Testing with `KUBEVIRT_DEPLOY_PROMETHEUS=true`
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Watch and unit test all prometheus related resources
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* add gateway for changing monitoring namespace (rbac purposes)
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* refactor test to check for exact alert name and firing state
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Align pattern of ensuring prometheus resource exists for all
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Remove potential noisy event
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Extract duplicate code to function
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Dont use empty value for prometheus label due to open issue
https://github.com/prometheus-operator/prometheus-operator/issues/4325
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Support registry import using node docker cache
The new CRI (container runtime interface) importer pod is created with three containers and a shared emptyDir volume:
-Init container: copies static http server binary to empty dir
-Server container: container image container configured to run the http binary and serve up the image file in /data
-Client container: import.sh uses cdi-import to import from server container, and writes "done" file on emptydir
-Server container sees "done" file and exits
Thanks mhenriks for the PoC!
Done:
-added ImportMethod to DataVolumeSourceRegistry (DataVolume.Spec.Source.Registry, DataImportCron.Spec.Source.Registry).
Import method can be "skopeo" (default), or "cri" for container runtime interface based import
-added cdi-containerimage-server & import.sh to the cdi-importer container
ToDo:
-utests and func tests
-doc
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add tests, fix CR comments
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Use deployment docker prefix and tag in func tests
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add OpenShift ImageStreams import support
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add importer pod lookup annotation for image streams
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add pullMethod and imageStream doc
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add long term token (10 years) to pvcs when host assisted cloning between namespaces
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* clone controller should retry if source in use
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* minor refactor if/else -> switch
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
Users don't want 👽 resources in clusters,
and we should also be able to tell if were part of a broader installation.
Note:
- Operator created resources were handled in https://github.com/kubevirt/controller-lifecycle-operator-sdk/pull/18
as these labels will be common to all resources deployed by the HCO.
- Now that the controller is guaranteed to have the labels, we can set env vars
that reference the label values (fieldRef) to spare calling GET on the CR in the controllers.
(thanks mhenriks).
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Strip newlines when writing a termination message.
Otherwise it isn't visible, at least when viewing in the -o yaml view.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Write down the nbdkit output and add it to the error output
With the added output from nbdkit, we can see the reason for the
non-existence of the nbdkit socket.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Don't set that we're waiting if it's CrashLoopBackOff
It's better to have the reason for the crash (terminate message)
than "backing off 5 minutes"
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Simplify all "image too large to fit" terminate reasons.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Move verifyConditions to utils, no functional change
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Switch test for image too large to test condition and not log
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Remove unused branch
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Rename setConditionFromPodWithPrefix to setAnnotationsFromPodWithPrefix
No functional change. Intended to be followed by some refactoring.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Fold restart count logic into the common setAnnotationsFromPodWithPrefix
Changing to >= rather than > to ensure a zero pod restart count is
always used -- the import controller unit tests request this.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Reduce indentation by returning right away.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Fold check for pod being nil into common code.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* move saveVddkAnnotations into util and make it unconditional
Call it setVddkAnnotations for consistency.
Check for not-terminated inside the function, not outside.
Removes check for source being VDDK (to avoid passing more arguments):
it won't match the regex anyway.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Reduce indentation by bailing on failure.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Reorder parameters to mirror the order in the function name
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Use a named variable for first container state
Yields shorter, more legible lines.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Use a constant for the nbdkit log.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add more information to function description - also logs to file
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Move AwaitingVDDK constant to common.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Copy pending PVC bound condition reason to DV.
Replace the fixed "Pending" string and tweak the unit test that checked
this.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add a functional test for AwaitingVDDK.
Delete v2v-vmware ConfigMap and create a DataVolume, and the bound
condition should have a reason of "AwaitingVDDK".
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Move AwaitingVDDK to its own functional test.
Retain common test code with existing table, but tack on an extra
cleanup step so v2v-vmware ConfigMap can be restored afterward.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Refactoring - move PreallocationApplied flag definition to common
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Preallocate cloning DataVolumes
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Have the healthz server check for the uploadserver first.
Uploadserver and healthz run in separate goroutine, so we need
to synchronize them. This seems to be the easiest way.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Make healthz unit test check for failure.
It doesn't appear to be possible to use httptest to test two servers,
and we now tie the health report to the upload server.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* api: Add new ImportProxy structure to CDIConfig in v1beta1 core api
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* common: Add constants related to configure Import proxy in Import pod and controller reconcile
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* config controller: Add import proxy reconcile, monitoring OCP proxy updates
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* config controller: Add unit test for import proxy reconcile, monitoring OCP proxy updates and creating Secret and ConfigMap
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* import controller: Add unit support of ImportProxy in the tests
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* controller util: Add support to retrieve an OpenShift cluster wide proxy object as well as its needed configurations
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* controller util: Add unit test for the get cluster wide proxy functionality
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* controller: moved isOpenshift function to utils common and export it
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* importer: Clone http transport to keep the default proxy config such as the usage of environment variables and add support for proxy CA
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* operator: Update CDIConfigCRD with import proxy info
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* operator: Update CDIListCRD with import proxy info
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* operator: Add service account rights to CDI for accessing OpenShift proxy obj
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* operator: Add OpenShift proxy obj scheme support
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* test: Add new forwarding proxy for testing things that require a proxy
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* test: Add test to verify the importer pod with proxy
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* doc: Update cdi config doc with the ImportProxy info
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* add support to build the new proxy docker image
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* test: remove unwanted F parameter from test
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* controller: fix error handling
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* test: update the used method to get the pod of the importer pod and proxy
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* fixed comments from the revision
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* controller config: update the methods to use the log object from the reconcile object
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* controller: update function GetImportProxyConfig to return errors and include a new unit test
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* controller: add back a test that was removed by mistake
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* update updateCDIConfigByUpdatingTheClusterWideProxy function
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* test: decrease the import pod log pooling interval and increase the image size to be able to verify the import pod before it is deleted
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* test: update the pvc and proxied requests verification
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* tools: add X-Forwarded-For header in the proxy requests used for testing
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* test: reset cluster wide proxy with the original values
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* test: fix proxy update to change spec instead of status and other minor updates
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* doc: update import proxy description
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* update generated files
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
This PR removes "skipped" condition for preallocation. Importer/uploader
will preallocate to the available size. Filesystem overhead needs to be
taken into account.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
Instead of (ab)using raw strings, preallocation status in importer and
uploader use now a custom type.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* [WIP] doc: User-facing doc for preallocation support
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* apis: CDI accepts `preallocation` option.
With this commit CDI accepts (but does handle) `preallocation` settings
for DataVolumes and in CDIConfig.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Implementing preallocation
This commit implements preallocation support for import and upload.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Functional tests for preallocation support
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Remove "preallocation for StorageClasses" config
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Removed unused function
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Fix rook-ceph test failures
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Updated dependencies
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Uss PVC annotation to pass preallocation parameters
DataVolume controller now uses a PVC annotation to pass preallocation
configuration to import and update controllers.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Plumb new checkpoint API through to VDDK importer.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add incremental data copy from VDDK.
Create a new data source implementation similar to vddk-datasource, but
only for blocks of data that changed between two snapshots. Also factor
out common things between the two VDDK data sources.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Check block status for warm and cold imports.
Addresses a bunch of runtime issues, but progress tracking isn't right.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Find snapshots correctly.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove separate warm/cold VDDK importers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Advance through the checkpoint list in the spec.
Move DataVolume to Paused after each checkpoint, and start a new
importer pod for the next available checkpoint. Keep track of which
checkpoints have been copied by adding PVC annotations associating each
checkpoint with the UID of the pod that copied it.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Allow spec updates to drive multi-stage imports.
A multi-stage import can create checkpoints at any time, so CDI needs to
be able to receive updates to the list of checkpoints. Implement this by
allowing spec changes only for fields related to multi-stage imports.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Avoid deleting destination in multi-stage import.
A multi-stage import will have an initial data copy to the destination
file followed by separate copies for individual deltas. The destination
file should not be deleted before starting these delta copies.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Get VDDK data source to pass formatting tests.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Unit tests for multi-stage import admission rules.
Make sure only updates to checkpoint-related fields are accepted.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add warm import unit tests for VDDK data source.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add VDDK warm import functional test.
Put two snapshots in the vCenter simulator inventory, and run them
through a multi-stage import process. Also clean up some issues
reported by test-lint.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add some documentation about multi-stage imports.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Pass existing multi-stage DataVolume unit tests.
Also remove MD5 sum step used for debugging, since it can take a long time.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove tabs from documentation.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Pass failing import-controller unit test.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* More unit tests for multi-stage field updates.
Also factor these tests into a DescribeTable.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add nbdkit retry filter.
Available as of Fedora 33 update.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Give correct file name to nbdkit in more cases.
The backing file in the spec might not always match the backing file in
the snapshot, so try harder to match those files by disk ID. May still
need to allow updates to backingFile, depending on how this gets used.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add more unit tests for datavolume-controller.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fix linter error from last commit.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add unit tests for some govmomi API calls.
Move original calls into mock interfaces to make this work.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add libnbd to build.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add libnbd Go bindings to vendor.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add libnbd to importer container image.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Enable cgo builds by changing bazel platform.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Break dependency chain on pkg/importer.
Without this, pretty much every generated binary (including tests.test)
requires libnbd.so.0.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Replace qemu-img with libnbd in VDDK importer.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add progress tracking back to VDDK datasource.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add fadvise before sequential VDDK transfer.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Improve VDDK debug logging and error handling.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Update to libnbd 1.4.1.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Update build container tag.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fix uploadserver/uploadproxy tests.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add a mirror for libnbd RPM, like in #1444.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add libnbd to uploadserver.
This still depends on pkg/importer, so it needs to have libnbd.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Use pointer to data sink, and add final sync.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* When validating disk space, reserve space for filesystem overhead
The amount of available space in a filesystem is not exactly
the advertise amount. Things like indirect blocks or metadata
may use up some of this space. Reserving it to avoid reaching
full capacity by default.
This value is configurable from the CDIConfig object spec,
both globally and per-storageclass.
The default value is 0.055, or "5.5% of the space is
reserved". This value was chosen because some filesystems
reserve 5% of the space as overhead for the root user and
this space doubles as reservation for the worst case
behaviour for unclear space usage. I've chosen a value
that is slightly higher.
This validation is only necessary because we use sparse
images instead of fallocated ones, which was done to have
reasonable alerts regarding space usage from various
storage providers.
---
Update CDIConfig filesystemOverhead status, validate, and
pass the final value to importer/upload pods.
Only the status values controlled by the config controller
are used, and it's filled out for all available storage
classes in the cluster.
Use this value in Validate calls to ensure that some of the
space is reserved for the filesystem overhead to guard from
accidents.
Caveats:
Doesn't use Default: to define the default of 0.055, instead
it is hard-coded in reconcile. It seems like we can't use a
default value.
Validates the per-storageClass values in reconcile, and
doesn't reject bad values.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Use util GetStorageClassByName
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Test filesystem overhead validation against async upload endpoint
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* wait for NFS PVs to be deleted before continuing
Intended to help with flakes, but didn't make a difference.
Probably still worth doing.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Avoid using the uncached client unnecessarily
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add error handling for the case where even a default SC is not found
Note that this change isn't expected to make a difference, as we
check if the targetStorageClass is nil later on and have the same
behaviour, but this is probably more correct API usage.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add testing for the validation of filesystem overhead values
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Fix logical error in waiting for NFS PVs.
Wait for all of them, not just the last one.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add support for a VDDK datasource to the controllers
This commit does not implement the VDDK datasource. It simply
adds the DataVolumeSourceVDDK type and adds support to the
datavolume and import controllers for it. The datasource itself
will need to be done in a follow-up.
Signed-off-by: Sam Lucidi <slucidi@redhat.com>
* Fix tests and clean up lint
Signed-off-by: Sam Lucidi <slucidi@redhat.com>
* Add DataVolumeSourceVDDK to the v1alpha1 api group
Signed-off-by: Sam Lucidi <slucidi@redhat.com>
* Install nbdkit.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add govmomi to vendor directory.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Integrate govmomi into build.
Incomplete! This allows 'find' and 'object' imports from govmomi, but
further dependencies may require more bazel tweaks later.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Create a container image just to hold the VDDK.
Mount the VDDK folder to /opt. Also copy the libxcrypt-compat RPM to
/opt for run-time installation as a temporary workaround for my
inability to install it at container creation time. Make VDDK optional
by storing a dummy file in its directory, so an empty VDDK directory
will not break the build.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fill in VDDK data source.
Add the simplest possible VDDK data source: set up an nbdkit server with
the given VMware parameters, and copy data out using qemu-img.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add vddk to resource validation schema.
Stops "error validating data" messages.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Move libxcrypt-compat install out of VDDK image.
Requiring users to add this to the VDDK image is kind of onerous, so
move it straight into the importer image instead.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Generate BUILD.bazel for the rest of govmomi.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Get VDDK sidecar image from v2v-vmware ConfigMap.
Convention from https://github.com/kubevirt/web-ui-components/pull/534
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Reduce nbdkit startup timeout.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Update govmomi dependencies.
Cleans up 'inconsistent vendoring' errors.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fix schema verification test.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Change disk.raw to disk.img.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add vCenter simulator for VDDK datasource tests.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Functional test with vcsim and dummy VDDK plugin.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Clean up lint errors.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Check kubevirt-hyperconverged namespace for VDDK.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add VDDK unit tests and API functional test.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Read a raw image for VDDK test plugin.
Have the fake VDDK test plugin for nbdkit read a raw image passed in
/opt/updates/nbdtest.img instead of returning fixed data.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add some documentation about VDDK data sources.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove unit test focus and trailing spaces.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Use updated build container image.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add some extra comments about VDDK fields.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Clarify VDDK ConfigMap a little more.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Clarify namespace for vddk-init sidecar.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove NodePort from vCenter simulator manifest.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Re-word explanations on VDDK source types.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add VDDK to comment with types needing URL check.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Rebuild types.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Move AnnThumbprint to storage.import.vddk.thumbprint.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove run-time installation of libxcrypt-compat.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove fixed namespaces for vddk-init reference.
The importer now looks for the v2v-vmware ConfigMap in the current CDI
namespace instead of openshift-cnv/kubevirt-hyperconverged.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Find CDI install namespace from data volume test.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
Co-authored-by: Sam Lucidi <slucidi@redhat.com>
* move upload.cdi.kubevirt.io API group to v1beta1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* move core api to v1beta1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix os-3.11 cluster sync and add functional tests for alpha api
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* change more occurences of v1alpha1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* updates after rebase
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* use dedicated SCC
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* SCC was not getting on initial deploy
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Set secondary GID of user on import and upload pods to be 107 (qemu) for kubevirt content type.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Fix broken unit tests on upload.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Add import and clone test.
Signed-off-by: Alexander Wels <awels@redhat.com>
This PR introduces new import data source to fetch disks from ovirt
and store them in storage available in the cluster. It uses ovirt-sdk
to check whether the this is there, the size of the disk and it obtains
signed ticket which allows to get the disk using ovirt-imageio-proxy.
In order to use it a user needs to create configmap with ovirt api
certificate and secret with ovirt engine user and password. Both are
required and validated by webhook. In order to start import process
we need to post following definition:
---
apiVersion: cdi.kubevirt.io/v1alpha1
kind: DataVolume
metadata:
name: "imageio-dv"
spec:
source:
imageio:
url: "<engine-api-endpoint>"
secretRef: "<secret-name-containing-engine-credentials>"
certConfigMap: "<configmap-name-containing-engine-api-certificate>"
diskId: "<disk-id-which-we-want-to-import>"
pvc:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "64Mi"
---
Signed-off-by: Piotr Kliczewski <piotr.kliczewski@gmail.com>
