* touch up zero restoresize snapshot
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* clone populator
only supports PVC source now
snapshot coming soon
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* more unit tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* unit test for clone populator
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* func tests for clone populator
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* move clone populator cleanup function to planner
other review comments
verifier pod should bount readonly
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add readonly flag to test executor pods
synchronize get hash calls
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* increase linter timeout
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* better/explicit readonly support for test pods
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* check pv for driver info before looking up storageclass as it may not exist
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* addressed review comments
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* chooseStrategy shoud generate more events
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Create populators package to be used for all populators
This commit introduces the basic reconciler for
populators with common function that can be used
by the different populators.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* unite getcontenttype func across code
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add VolumeImportSource CRD for import populator
This commit adds the VolumeImportSource CRD into CDI.
CRs created from this CRD will be referenced in the dataSourceRef field to populate PVCs with the import populator.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Refactor common populator code to be shared among all populators
This commit introduces and modifies several functions so we can reuse common code between all populators.
Other than having a common reconcile function, a new populatorController interface has been introduced so we are able to call populator-specific methods from the populator-base reconciler.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Create Import Populator
The import populator is a controller that handles the import of data in PVCs without the need of DataVolumes while still taking advantage of the import-controller flow.
This controller creates an additional PVC' with import annotations. After the import process succeeds, the controller rebinds the PV to the original target PVc and deletes the PVC prime.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add functional tests to cover the import populator flow
This commit updates the import tests to cover the new import populator flow.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add unit tests for import populator
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Minor fixes and enhancements in import/common populator code
* Modify indexes and other related code to support namespaced dataSourceRefs. Cross-namespace population is still not supported as it depends on alpha feature gates.
* Add functional test to cover static binding.
* Fix selected node annotation bug in scratch space PVCs
* Fix linter alerts
Signed-off-by: Alvaro Romero <alromero@redhat.com>
---------
Signed-off-by: Shelly Kagan <skagan@redhat.com>
Signed-off-by: Alvaro Romero <alromero@redhat.com>
Co-authored-by: Shelly Kagan <skagan@redhat.com>
* Start adding the golangci-lint to CI
golangci-lint is a collection of many linters. This PR adds
golangci-lint to the CI. For strat, it enables the govet linter, and fix
its single finding.
The PR adds this linter to the `test-lint` Makefile target.
The new .golangci.yml file is the configuration for the linter.
golangci-lint version was set to the latest one - v1.52.2.
It is defined in hack/build/run-linters.sh
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
* golangci-lint: enable gosimple and fix findings
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
* golangci-lint: enable unused and fix findings
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
---------
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
* Google Cloud Storage Importer
This is a Google Cloud Storage importer for CDI
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* Fix auto-generated swagger and openapi
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* GCS Importer General Fixes
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
* Moving back gcs-secret.txt
Moving file back to imageDir to fix unit testing.
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
---------
Signed-off-by: Marcelo Parisi <marcelo@feitoza.com.br>
Co-authored-by: Marcelo Parisi <marcelo@dev-box.corp.feitoza.com.br>
* Add support for imagePullSecrets in the CDI CR, to support pulling
images from repositories that require secrets.
The imagePullSecrets is propagated to the following components: cdi-apiserver,
cdi-deployment, and cdi-uploadproxy. The definition of imagePullSecrets in
cdi-operator must be done manually.
Signed-off-by: Gleb Aronsky <gleb.aronsky@windriver.com>
* Modifying code to incorporate review comments.
Signed-off-by: Gleb Aronsky <gleb.aronsky@windriver.com>
---------
Signed-off-by: Gleb Aronsky <gleb.aronsky@windriver.com>
Co-authored-by: Gleb Aronsky <gleb.aronsky@windriver.com>
* Fix hostpath CSI being skipped as "Not HPP"
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Fall back to host assisted if immediate bind requested
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
---------
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add support for volume populators in CDI
This commit enables the use of volume populators in CDI, so datavolume-owned PVCs can be populated using custom logic.
Volume populators are CRDs used to populate volumes externally, independently of CDI. These CRDs can now be specified using the new DataSourceRef API field in the DataVolume spec.
When a DataVolume is created with a populated DataSourceRef field, the datavolume-controller creates the corresponding PVC accordingly but skips all the population-related steps. Once the PVC is bound, the DV phase changes to succeeded.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Modify CDI test infrastructure to support testing of external populators
This commit introduces several changes to CDI ci to support the testing of DataVolumes with external populators:
* A sample volume populator is now deployed in the test infrastructure, in a similar way as bad-webserver or test-proxy. This populator will be used in functional tests from now on.
* A new test file with external population tests has been introduced in the tests directory
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Update dependencies to include lib-volume-populator library
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add functional tests for proper coverage of external population of DataVolumes
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Minor fixes on external-population logic for DataVolumes:
* Added comments for exported structs
* Removed non-inclusive language
* Improved error messages in webhooks
* Fixed logic on datavolume-controller
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Improve DataVolume external-population logic when using the old 'DataSource' API
This commit introduces several changes into the datavolume external-population controller to improve its behavior when using the DataSource field.
It also introduces minor fixes on the generic populator logic.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add unit tests for external-population controller and DV admission
Signed-off-by: Alvaro Romero <alromero@redhat.com>
Signed-off-by: Alvaro Romero <alromero@redhat.com>
- Split the huge DV controller into smaller op-specific DV controllers -
import, clone, upload
- Add common watch-adding function so each controller watches only its
relevant DVs
- Refactor the common Reconcile() to use interface DataVolumeReconciler
implemented by each controller
- Move all functions, structs, consts to the relevant controller
- Split the utests per controller
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* remove root worker pods
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove selinux requirement for worker pods
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* run tests in restricted namespace and required changes
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* handle empty tar
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add PSA label when running functional tests in OpenShift
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* cannot use restricted PSA with istio (for now)
refactor scc management
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix clean script
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Improve the error handling when pod creation fails
When pod creation fails, the error is usually logged without providing additional information to the user. This behavior is especially risky when the user lacks the permits to check the logs, making it unintuitive and almost impossible to find the source of the problem.
This commit improves the error handling of the pod-creation process, so pertinent info about the failure is included in the pod's PVC.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Update functional tests to check for events when pod-creation fails
Since error handling in pod-creation has been improved in our controllers, this commit introduces several changes in the corresponding functional tests to properly cover the new behavior included when pod-creation fails.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Update unit-tests after improving error-handling of pods for proper coverage
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Minor fixes and improvements on error handling for pods
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Modify datavolume-controller to change the running condition of datavolumes when a pod fails
Until this commit, the way of handling pod errors in the datavolume-controller has been to change the affected datavolume's phase to failed, which conflicts with the declarative approach of the controllers.
This commit modifies this behavior so that, when a pod fails, the affected datavolume's running condition is changed to false while the phase remains unchanged.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add TLS Security Profile API
TLSSecurityProfile is used by operators to apply cluster-wide TLS security settings to operands.
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Update apiserver & uploadproxy server TLS config on CDIConfig TLS knob change
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Propagate TLS config to uploadserver as well
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add functests for apiserver and upload that ensure value is respected
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Comply with restricted security context in kubernetes
Ensure CDI pods comply with the restricted security context as much as
possible (have to be root for nbdkit and block devices). Also cannot set
SeccompProfile since SCC won't allow us to set it.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Changed path /var/local/all_certs to stay in /var
Signed-off-by: Alexander Wels <awels@redhat.com>
* Allow empty DV size when cloning using storage API
When cloning a Data Volume, the size of the target can be potentially obtainable via the source PVC, which discards the need to explicitly specify it.
Considering that, this commit introduces a change in the correspondent validation webhook to allow omitting the resources.request.storage field when cloning a PVC using the storage API.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Modify datavolume-controller to allow obtaining storage size from source PVC when cloning
When cloning a PVC, if the target's size is not specified, said value can be attainable from the source PVC.
This commit introduces a change in datavolume controller so, in case of detecting an empty storage size, said value can be obtained when performing CSI and Smart cloning.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Update unit tests for datavolume-validation after enabling cloning with empty size
This commit updates the unit testing for the datavolume validation webhook, covering the possibility of cloning a PVC without setting any storage size.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Update unit testing for controller-related functions after enabling cloning with empty size
This commit includes unit tests for the volumeSize() function after enabling creating clones with blank size.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Update the datavolume controller to create a size-detection pod when performing host-assisted clone
When performing a host-assisted clone with empty clone size, simply copying the original PVC size could lead to potential overhead miscalculations if the source's VolumeMode is "filesystem".
When that's the case, an inspection pod will be created in the datavolume controller so it extracts the size of the virtual image using qemu-img.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Include an image-size detection tool to allow cloning with empty DV size
This commit introduces a new tool in charge of collecting the virtual image size when cloning with an empty DV size. In some cases where said value is unattainable from the original PVC's spec, the datavolume controller will create a new pod containing this new tool.
The binary will then run the 'qemu-img' command and handle its results appropriately.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Optimize the clone-size lookup process to avoid creating unnecessary size-detection pods
When performing host-assisted clone with an empty DV size, in some cases, a size-detection pod is used to obtain the required capacity.
This commit tries to optimize this process to keep the collected value as a PVC annotation, that is checked in subsequent clones to avoid creating redundant pods.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Minor fixes and improvements on mechanism for cloning with empty storage size
* Add new optional flag on size-detection binary to enable using a different URI scheme
* Improve the pod-creation mechanism so the pod is not created until the source PVC has finished the import
* Modify size-finlation mechanism to account for possible round-downs when importing the source image
* Improve the size inflation mechanism so only PVCs with filesystem as volume mode are considered
* Minor style corrections
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Modify the clone-controller to allow skipping the clone size validation in some cases
Due to filesystem overhead differences, the target's size can sometimes be smaller than the source's one when obtaining said value with the size-detection pod.
This commit introduces minor changes in the clone-controller so we can skip the size validation in those cases.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Minor changes and improvements in size-detection mechanism following PR review
* Added new UT that covers using empty storage API for non-cloning sources
* Added new watch on datavolume-controller that looks for changes in the size-detection pod
* Removed redundant and unnecessary specs on size-detection pod
* Added error handling when reading the pod's termination message
* Moved general-usage functions to 'util.go' file
* Updated 'datavolumes' documentation to reference the possibility of omitting the storage size when cloning
* Minor style corrections
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add unit tests that cover the size-detection mechanism in the DataVolume controller
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Include functional tests for cloning without specifying storage size
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Improve error handling in the creation/deletion process of the size-detection pod
This commit introduces additional handling in case of error after and during the size-detection pod is created.
It also updates several related unit tests.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Minor fixes to improve fsOverhead calculations when cloning with empty storage size
* Modified the size-detection mechanism so we account for fsOverhead when cloning to filesystem volume mode in all cases
* Clean up the code for reconciling when cloning a PVC that is not ready
* Minor fix in functional test so it works when cloning from block to filesystem volume mode
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Introduce controller-runtime-sdk api package
Split controller-runtime-sdk into the base package and
controller-runtime-sdk/api.
Signed-off-by: Roman Mohr <rmohr@redhat.com>
* go mod vendor
Signed-off-by: Roman Mohr <rmohr@redhat.com>
* Update code references
Signed-off-by: Roman Mohr <rmohr@redhat.com>
* Create imageio container during CDI build.
Instead of using a really old imageio, use bazel to build a new
imageio based on 2.5.0. Update the tests to use the new image
and paths in that new image. This requires a new repo in quay for
us to push the image to.
Also changed the approach of resolving the warm import potential
dead lock (scratch PVC from previous import pod terminating, while
the new pod is trying to create itself). Instead of trying to avoid
in all scenarios, detect the state, and delete the pod so the dead
lock can be resolved.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Populate test images
Signed-off-by: Alexander Wels <awels@redhat.com>
* Enable disabled test, and fix race condition where the import
controller thought it was done, but we were still on the final
import of a warm migration.
Updated the way we create the ticket on the fake imageio
Signed-off-by: Alexander Wels <awels@redhat.com>
* Append checkpoint ID to multi-stage importer pods.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Ignore completed pods for multi-stage imports.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Reset current import pod when checkpoint is done.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Don't prevent pod deletion for scratch space.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Only ignore pod when retainAfterCompletion is set.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fix data volume unit tests.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Tests for checkpoint suffix and completed pods.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Test for retained pods exiting for scratch space.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add functional test for retaining multistage pods.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Clean up lint error.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove scratch handling that is fixed elsewhere.
This is part of shouldDeletePod now.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add unit tests for long PVC/checkpoint names.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Match retainAfterCompletion test to description.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fix failing imageio test
Always enable scratch space for imageio imports.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Only always enable scratch space with warm migration imports.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Set htpp(s)_proxy to lower case env variable
CURL used by nbdkit doesn't read upper case http(s)_proxy environment
variables, and thus was not using the proxy. Changed the variable to
be lower case.
Added a significant number of tests to test many more variations of
using a proxy. Also added https + auth endpoint to the file-host
container, so we can test https + auth with the proxy.
Added https endpoint to proxy, so we can test an https proxy.
Cleaned up some of the error handling in the import controller for
the proxy, in particular if a trustedCAProxy is defined.
Fixed some of the cluster wide proxy configuration so it works properly
inside an openshift cluster.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Add https proxy support to registry import. Added extra
functional tests to test all registry import combinations
Signed-off-by: Alexander Wels <awels@redhat.com>
* Fixed some tests to work better in Open Shift.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Add optional VDDK initImageURL field.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Pass VDDK image URL through to PVC annotation.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Unit tests for per-DV VDDK image URL.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Functional test for VDDK initImageURL field.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Update documentation for VDDK initImageURL.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fix lint error.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Check for absence of AwaitingVDDK in unit test.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Update datavolume conditions when quota exceeded when creating pvc
When creating the pvc from the dv the pvc size
can exceed the allowed quota, in such case so far the only
indication was to look in the logs.
Now added indication in the data volume conditions
(when possible) and emitted event.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add functional tests to check the new conditons and event
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* tests cosmetics
-use existing functions
-add missing checks on errors
-remove unused code
-etc..
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Update HTTP data source API to allow custom headers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Implement custom HTTP headers API.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Document custom headers in HTTP data source.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Correct secretExtraHeader comment to reference Secret.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add volume mounts for secret headers.
Replaces environment variables for headers from secrets.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Avoid failing when there are no extra headers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Redact contents of headers that come from secrets.
Also split up getExtraHeaders to reduce Sonar Cloud complexity.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Ensure all HTTP client requests use extra headers.
Missed redirect check and content length retrieval, both of which might
need the extra headers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add some unit tests for extra HTTP headers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Do not quote headers in nbdkit curl arguments.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add functional tests for extra HTTP headers.
Avoids new test server by specifiying basic authorization headers to the
existing file host port that requires it.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Use filepath.Walk to read secrets.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Minor documentation update for secrets.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Re-run 'make generate' for verification failure.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add DataImportCron controller
-The new controller polls for updates to a registry source container
image, based on a given schedule. When updates to a container image are
detected, the controller imports the content into a new uniquely named
PVC in a golden image namespace.
-For each DataImportCron, the controller manages a corresponding
DataSource to always point to the latest most up-to-date golden
image PVC.
-DataImportCron takes ownership of an existing DataSource (with
controller: false), allowing an admin to opt-in to using auto
delivery/updates later on.
-The controller has PVC garbage collector removing old PVCs.
ToDo:
-status conditions updates
-verify full image streams support
-utests and func tests
-fixmes and commented out code
-doc
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix CR comments and fixmes
- isolate imagestream and registry specific code
- fix namespace of CronJob, and its job and pod to CDI namespace
- manage CronJob-DataImportCron ownership relationship with a finalizer,
handle DataImportCron deletion (CronJob etc.)
- remove CronJob and job pod for ImageStreams, use RequeueAfter and
cronexpr instead
- add k8s app cdi-source-update-poller executed by CronJob to poll source
image digest via skopeo inspect for url registry source, and annotate
the DataImportCron when the image was updated and pending for import based
on the cron schedule
- add cdi-source-update-poller and skopeo binary to the cdi-importer container
- complete dataimportcron-validate and its tests
- reconcile - use context.Context instead of context.TODO
- remove uncached client
- doc
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix ImageStreams watch
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron DV template instead of source
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix CR comments
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Split updateSucceeded func
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Improve cdi-source-update-poller cmd logs
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove ImageStream reconcile
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove ImageStream watch
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove unnecessary AnnSourceUpdatePending
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* More CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Idempotentify initCron
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Recreate DV in case is't not found
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron spec.importsToKeep and status.currentImports
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron controller functional test
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add insecure TLS support
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove finalizers in cluster clean script
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Bound each import to its sha256 digest instead of latest
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron controller utests
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Tests CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Minor tests CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* move apis to new staging area
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add script to push to staging
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix lint check and api reference
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* push staging to api repo
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Deploy alerts infra as part of our installation
Conditionally deploy the infrastructure that is needed to fire alerts for our users
when bad things are happening to CDI.
Testing with `KUBEVIRT_DEPLOY_PROMETHEUS=true`
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Watch and unit test all prometheus related resources
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* add gateway for changing monitoring namespace (rbac purposes)
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* refactor test to check for exact alert name and firing state
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Align pattern of ensuring prometheus resource exists for all
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Remove potential noisy event
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Extract duplicate code to function
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Dont use empty value for prometheus label due to open issue
https://github.com/prometheus-operator/prometheus-operator/issues/4325
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Support registry import using node docker cache
The new CRI (container runtime interface) importer pod is created with three containers and a shared emptyDir volume:
-Init container: copies static http server binary to empty dir
-Server container: container image container configured to run the http binary and serve up the image file in /data
-Client container: import.sh uses cdi-import to import from server container, and writes "done" file on emptydir
-Server container sees "done" file and exits
Thanks mhenriks for the PoC!
Done:
-added ImportMethod to DataVolumeSourceRegistry (DataVolume.Spec.Source.Registry, DataImportCron.Spec.Source.Registry).
Import method can be "skopeo" (default), or "cri" for container runtime interface based import
-added cdi-containerimage-server & import.sh to the cdi-importer container
ToDo:
-utests and func tests
-doc
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add tests, fix CR comments
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Use deployment docker prefix and tag in func tests
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add OpenShift ImageStreams import support
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add importer pod lookup annotation for image streams
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add pullMethod and imageStream doc
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
Users don't want 👽 resources in clusters,
and we should also be able to tell if were part of a broader installation.
Note:
- Operator created resources were handled in https://github.com/kubevirt/controller-lifecycle-operator-sdk/pull/18
as these labels will be common to all resources deployed by the HCO.
- Now that the controller is guaranteed to have the labels, we can set env vars
that reference the label values (fieldRef) to spare calling GET on the CR in the controllers.
(thanks mhenriks).
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Strip newlines when writing a termination message.
Otherwise it isn't visible, at least when viewing in the -o yaml view.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Write down the nbdkit output and add it to the error output
With the added output from nbdkit, we can see the reason for the
non-existence of the nbdkit socket.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Don't set that we're waiting if it's CrashLoopBackOff
It's better to have the reason for the crash (terminate message)
than "backing off 5 minutes"
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Simplify all "image too large to fit" terminate reasons.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Move verifyConditions to utils, no functional change
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Switch test for image too large to test condition and not log
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Remove unused branch
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Rename setConditionFromPodWithPrefix to setAnnotationsFromPodWithPrefix
No functional change. Intended to be followed by some refactoring.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Fold restart count logic into the common setAnnotationsFromPodWithPrefix
Changing to >= rather than > to ensure a zero pod restart count is
always used -- the import controller unit tests request this.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Reduce indentation by returning right away.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Fold check for pod being nil into common code.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* move saveVddkAnnotations into util and make it unconditional
Call it setVddkAnnotations for consistency.
Check for not-terminated inside the function, not outside.
Removes check for source being VDDK (to avoid passing more arguments):
it won't match the regex anyway.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Reduce indentation by bailing on failure.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Reorder parameters to mirror the order in the function name
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Use a named variable for first container state
Yields shorter, more legible lines.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Use a constant for the nbdkit log.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add more information to function description - also logs to file
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Move AwaitingVDDK constant to common.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Copy pending PVC bound condition reason to DV.
Replace the fixed "Pending" string and tweak the unit test that checked
this.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add a functional test for AwaitingVDDK.
Delete v2v-vmware ConfigMap and create a DataVolume, and the bound
condition should have a reason of "AwaitingVDDK".
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Move AwaitingVDDK to its own functional test.
Retain common test code with existing table, but tack on an extra
cleanup step so v2v-vmware ConfigMap can be restored afterward.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* update deps and bazel
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix apidocs and unit tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix generate-verify
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Add an interface to watch nbdkit logs.
Useful for fishing out various pieces of information. Save VDDK library
version and connected ESX host by appending to the importer pod's
termination message. Turns nbdkit logging up to verbose for VDDK data
sources, so only the last few lines are printed for debugging.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Copy VDDK info from termination message to PVC/DV.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add unit tests for saved VDDK information.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add functional test for VDDK annotations.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fix unit test, forgot to check for nil pvc.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Don't ignore errors updating PVC with VDDK info.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Watch nbdkit with Scanner instead of ReadString.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Move VDDK info test into existing functional test.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Make nbdkit stop sequence slightly clearer.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Save VDDK info in regular DV reconciler.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Don't save VDDK info when PVC is being deleted.
Also, piggyback off existing PVC update instead of introducing a new
error handling path.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fix VDDK-info unit tests.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Use scanner for all nbdkit logging.
Also fix up a minor merge mistake.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Try to satisfy complaints from SonarCloud.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Refactoring - move PreallocationApplied flag definition to common
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Preallocate cloning DataVolumes
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Cluster-scoped namespace transfer api and controller
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* unit tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* ObjectTransfer webhook
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* new functests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* experiment with termination grace period
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* quota test
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* api: Add new ImportProxy structure to CDIConfig in v1beta1 core api
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* common: Add constants related to configure Import proxy in Import pod and controller reconcile
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* config controller: Add import proxy reconcile, monitoring OCP proxy updates
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* config controller: Add unit test for import proxy reconcile, monitoring OCP proxy updates and creating Secret and ConfigMap
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* import controller: Add unit support of ImportProxy in the tests
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* controller util: Add support to retrieve an OpenShift cluster wide proxy object as well as its needed configurations
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* controller util: Add unit test for the get cluster wide proxy functionality
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* controller: moved isOpenshift function to utils common and export it
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* importer: Clone http transport to keep the default proxy config such as the usage of environment variables and add support for proxy CA
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* operator: Update CDIConfigCRD with import proxy info
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* operator: Update CDIListCRD with import proxy info
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* operator: Add service account rights to CDI for accessing OpenShift proxy obj
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* operator: Add OpenShift proxy obj scheme support
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* test: Add new forwarding proxy for testing things that require a proxy
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* test: Add test to verify the importer pod with proxy
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* doc: Update cdi config doc with the ImportProxy info
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* add support to build the new proxy docker image
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* test: remove unwanted F parameter from test
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* controller: fix error handling
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* test: update the used method to get the pod of the importer pod and proxy
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* fixed comments from the revision
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* controller config: update the methods to use the log object from the reconcile object
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* controller: update function GetImportProxyConfig to return errors and include a new unit test
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* controller: add back a test that was removed by mistake
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* update updateCDIConfigByUpdatingTheClusterWideProxy function
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* test: decrease the import pod log pooling interval and increase the image size to be able to verify the import pod before it is deleted
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* test: update the pvc and proxied requests verification
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* tools: add X-Forwarded-For header in the proxy requests used for testing
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* test: reset cluster wide proxy with the original values
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* test: fix proxy update to change spec instead of status and other minor updates
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* doc: update import proxy description
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
* update generated files
Signed-off-by: Marcelo Amaral <marcelo.amaral1@ibm.com>
This PR removes "skipped" condition for preallocation. Importer/uploader
will preallocate to the available size. Filesystem overhead needs to be
taken into account.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Simplify shouldReconcile function arguments.
By having the function itself grab things it needs and are easily
obtained.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Adapt unit tests to simpler shouldReconcilePVC
Don't set any feature gates for WFFC being disabled.
When the second argument is true, pass the immediate binding annotation
to the PVC itself.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
Assists resolution of BZ#1886566. Use existing mechanism to copy certain
PVC annotations into DV status conditions.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add a DV/PVC annotation "storage.bind.immediate.requested"
When the annotation is applied the CDI will force bind the PVC (by scheduling worker pods), ignoring the logic to handle WaitForFirstConsumer binding mode.
This is useful when uploading "template" images to the cluster on local storage with WaitForFirstConsumer binding. In this case the image has to be available somewhere on the cluster, the actual placement of image on specific node is not important, so the CDI worker node can be used as a first consumer.
For storage with immediate binging it is effectively a NOOP.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Test import/upload/clone with annotation "storage.bind.immediate.requested"
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* [WIP] doc: User-facing doc for preallocation support
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* apis: CDI accepts `preallocation` option.
With this commit CDI accepts (but does handle) `preallocation` settings
for DataVolumes and in CDIConfig.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Implementing preallocation
This commit implements preallocation support for import and upload.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Functional tests for preallocation support
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Remove "preallocation for StorageClasses" config
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Removed unused function
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Updated dependencies
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Uss PVC annotation to pass preallocation parameters
DataVolume controller now uses a PVC annotation to pass preallocation
configuration to import and update controllers.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Implementing preallocation
This commit implements preallocation support for import and upload.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Make sure all import/upload paths do preallocation
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* [WIP] doc: User-facing doc for preallocation support
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* apis: CDI accepts `preallocation` option.
With this commit CDI accepts (but does handle) `preallocation` settings
for DataVolumes and in CDIConfig.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Implementing preallocation
This commit implements preallocation support for import and upload.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Functional tests for preallocation support
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Remove "preallocation for StorageClasses" config
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Removed unused function
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Fix rook-ceph test failures
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Updated dependencies
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Uss PVC annotation to pass preallocation parameters
DataVolume controller now uses a PVC annotation to pass preallocation
configuration to import and update controllers.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Plumb new checkpoint API through to VDDK importer.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add incremental data copy from VDDK.
Create a new data source implementation similar to vddk-datasource, but
only for blocks of data that changed between two snapshots. Also factor
out common things between the two VDDK data sources.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Check block status for warm and cold imports.
Addresses a bunch of runtime issues, but progress tracking isn't right.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Find snapshots correctly.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove separate warm/cold VDDK importers.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Advance through the checkpoint list in the spec.
Move DataVolume to Paused after each checkpoint, and start a new
importer pod for the next available checkpoint. Keep track of which
checkpoints have been copied by adding PVC annotations associating each
checkpoint with the UID of the pod that copied it.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Allow spec updates to drive multi-stage imports.
A multi-stage import can create checkpoints at any time, so CDI needs to
be able to receive updates to the list of checkpoints. Implement this by
allowing spec changes only for fields related to multi-stage imports.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Avoid deleting destination in multi-stage import.
A multi-stage import will have an initial data copy to the destination
file followed by separate copies for individual deltas. The destination
file should not be deleted before starting these delta copies.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Get VDDK data source to pass formatting tests.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Unit tests for multi-stage import admission rules.
Make sure only updates to checkpoint-related fields are accepted.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add warm import unit tests for VDDK data source.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add VDDK warm import functional test.
Put two snapshots in the vCenter simulator inventory, and run them
through a multi-stage import process. Also clean up some issues
reported by test-lint.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add some documentation about multi-stage imports.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Pass existing multi-stage DataVolume unit tests.
Also remove MD5 sum step used for debugging, since it can take a long time.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Remove tabs from documentation.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Pass failing import-controller unit test.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* More unit tests for multi-stage field updates.
Also factor these tests into a DescribeTable.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add nbdkit retry filter.
Available as of Fedora 33 update.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Give correct file name to nbdkit in more cases.
The backing file in the spec might not always match the backing file in
the snapshot, so try harder to match those files by disk ID. May still
need to allow updates to backingFile, depending on how this gets used.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add more unit tests for datavolume-controller.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Fix linter error from last commit.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
* Add unit tests for some govmomi API calls.
Move original calls into mock interfaces to make this work.
Signed-off-by: Matthew Arnold <marnold@redhat.com>
