* touch up zero restoresize snapshot
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* clone populator
only supports PVC source now
snapshot coming soon
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* more unit tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* unit test for clone populator
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* func tests for clone populator
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* move clone populator cleanup function to planner
other review comments
verifier pod should bount readonly
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add readonly flag to test executor pods
synchronize get hash calls
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* increase linter timeout
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* better/explicit readonly support for test pods
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* check pv for driver info before looking up storageclass as it may not exist
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* addressed review comments
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* chooseStrategy shoud generate more events
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
---------
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Add webhooks for volumeimportsource and volumeuploadsource CRs
This commit implements a new webhook to validate the creation and update of volumeimportsources and volumeuploadsources.
It also adds proper testing for the new webhook.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add functional tests for the populator webhook
Signed-off-by: Alvaro Romero <alromero@redhat.com>
---------
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Create CRD for volumeuploadsource populator
This CRD will be used in the DataSourceRef on PVCs
to trigger population that upload to the volume.
This will be performed by the upload populator
that will be added in future commits.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Create upload populator controller
The upload populator controller can be used
standalone without the need of datavolume.
It reconciles pvc with upload dataSourceRef
and uses populators API to populated the pvc
with an upload command.
The controller creates pvc' with upload
annotation. After the upload completes it
rebinds the pv to the original target pvc and
deletes pvc prime.
Eventually we get a bound PVC which is already
populated.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Adjust upload-proxy to handle upload population
In case of pvc with datasourceref to upload population
we should create the url to the upload server with the
pvc' name.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add tests for upload population
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add unit tests for upload populator
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add preallocation to volumeuploadsource crd
Also some other small fixes
Signed-off-by: Shelly Kagan <skagan@redhat.com>
---------
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Create populators package to be used for all populators
This commit introduces the basic reconciler for
populators with common function that can be used
by the different populators.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* unite getcontenttype func across code
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add VolumeImportSource CRD for import populator
This commit adds the VolumeImportSource CRD into CDI.
CRs created from this CRD will be referenced in the dataSourceRef field to populate PVCs with the import populator.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Refactor common populator code to be shared among all populators
This commit introduces and modifies several functions so we can reuse common code between all populators.
Other than having a common reconcile function, a new populatorController interface has been introduced so we are able to call populator-specific methods from the populator-base reconciler.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Create Import Populator
The import populator is a controller that handles the import of data in PVCs without the need of DataVolumes while still taking advantage of the import-controller flow.
This controller creates an additional PVC' with import annotations. After the import process succeeds, the controller rebinds the PV to the original target PVc and deletes the PVC prime.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add functional tests to cover the import populator flow
This commit updates the import tests to cover the new import populator flow.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add unit tests for import populator
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Minor fixes and enhancements in import/common populator code
* Modify indexes and other related code to support namespaced dataSourceRefs. Cross-namespace population is still not supported as it depends on alpha feature gates.
* Add functional test to cover static binding.
* Fix selected node annotation bug in scratch space PVCs
* Fix linter alerts
Signed-off-by: Alvaro Romero <alromero@redhat.com>
---------
Signed-off-by: Shelly Kagan <skagan@redhat.com>
Signed-off-by: Alvaro Romero <alromero@redhat.com>
Co-authored-by: Shelly Kagan <skagan@redhat.com>
* Add support for imagePullSecrets in the CDI CR, to support pulling
images from repositories that require secrets.
The imagePullSecrets is propagated to the following components: cdi-apiserver,
cdi-deployment, and cdi-uploadproxy. The definition of imagePullSecrets in
cdi-operator must be done manually.
Signed-off-by: Gleb Aronsky <gleb.aronsky@windriver.com>
* Modifying code to incorporate review comments.
Signed-off-by: Gleb Aronsky <gleb.aronsky@windriver.com>
---------
Signed-off-by: Gleb Aronsky <gleb.aronsky@windriver.com>
Co-authored-by: Gleb Aronsky <gleb.aronsky@windriver.com>
* Avoid generating most of the v1alpha1 CRDs, regenerate
We leave the CDI CRD alone as that one is installed via a YAML file.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Avoid references to v1alpha1 APIs in docs
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Remove v1alpha1 specific tests
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Manually adjust code not to reference v1alpha1 APIs
v1alpha1 upload paths are kept as virtctl image upload still uses them
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Factor out scaling deployment to function
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Handle removal of v1alpha1 version from CRDs
If it was ever a storage version, we perform manual storage migration:
we ensure v1beta1 is storage version, get & update every object, and
remove the v1alpha1 storage version.
The CDI CRD v1alpha1 version is kept, so updating from clusters that
had old versions by applying release YAMLs still works.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Test upgrade from a version with v1alpha1 storage version.
Artificially create objects by scaling down cdi-operator & creating
a DV. Make sure it's there even after we restore CDI and that v1beta1
is the only storedVersion.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Regenerate (again, due to rebase)
Signed-off-by: Maya Rashish <mrashish@redhat.com>
Signed-off-by: Maya Rashish <mrashish@redhat.com>
We create our SCC with `priority: 10` (not sure why), which means
that it might get picked up by random pods whose service account is privileged enough to access all SCCs:
```bash
$ oc get pods -n openshift-cluster-storage-operator cluster-storage-operator-5648cb555d-zh76b -o yaml| grep scc
openshift.io/scc: containerized-data-importer
```
Happens because it can physically do that
```bash
$ oc get clusterrolebinding cluster-storage-operator-role -o yaml | grep cluster-admin
name: cluster-admin
```
A nice readout about this in context of the openshift oauth pods:
https://access.redhat.com/solutions/4727461
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
We only need the cluster access pkg/operator/resources/cluster/cronjob.go
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Introduce controller-runtime-sdk api package
Split controller-runtime-sdk into the base package and
controller-runtime-sdk/api.
Signed-off-by: Roman Mohr <rmohr@redhat.com>
* go mod vendor
Signed-off-by: Roman Mohr <rmohr@redhat.com>
* Update code references
Signed-off-by: Roman Mohr <rmohr@redhat.com>
* Update operator-lifecycle-sdk to get fix for labels on upgrade
Update dep to get https://github.com/kubevirt/controller-lifecycle-operator-sdk/pull/19
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Reconcile labels also for CDIConfig
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Reconcile labels on storageprofile
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Reconcile remaining operator resources for updated labels
BZ#2017478
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add degraded alert
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add unusual restart count metric
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add actual firing alerts (degraded/restartcount)
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Test newly added metrics
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Review: Rename metric to match conventions, func to check if test is eligible to run metric tests
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Get rid of similar funcs, reconcile more generally
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add DataImportCron controller
-The new controller polls for updates to a registry source container
image, based on a given schedule. When updates to a container image are
detected, the controller imports the content into a new uniquely named
PVC in a golden image namespace.
-For each DataImportCron, the controller manages a corresponding
DataSource to always point to the latest most up-to-date golden
image PVC.
-DataImportCron takes ownership of an existing DataSource (with
controller: false), allowing an admin to opt-in to using auto
delivery/updates later on.
-The controller has PVC garbage collector removing old PVCs.
ToDo:
-status conditions updates
-verify full image streams support
-utests and func tests
-fixmes and commented out code
-doc
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix CR comments and fixmes
- isolate imagestream and registry specific code
- fix namespace of CronJob, and its job and pod to CDI namespace
- manage CronJob-DataImportCron ownership relationship with a finalizer,
handle DataImportCron deletion (CronJob etc.)
- remove CronJob and job pod for ImageStreams, use RequeueAfter and
cronexpr instead
- add k8s app cdi-source-update-poller executed by CronJob to poll source
image digest via skopeo inspect for url registry source, and annotate
the DataImportCron when the image was updated and pending for import based
on the cron schedule
- add cdi-source-update-poller and skopeo binary to the cdi-importer container
- complete dataimportcron-validate and its tests
- reconcile - use context.Context instead of context.TODO
- remove uncached client
- doc
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix ImageStreams watch
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron DV template instead of source
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix CR comments
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Split updateSucceeded func
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Improve cdi-source-update-poller cmd logs
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove ImageStream reconcile
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove ImageStream watch
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove unnecessary AnnSourceUpdatePending
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* More CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Idempotentify initCron
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Recreate DV in case is't not found
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron spec.importsToKeep and status.currentImports
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron controller functional test
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add insecure TLS support
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove finalizers in cluster clean script
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Bound each import to its sha256 digest instead of latest
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron controller utests
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Tests CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Minor tests CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* move apis to new staging area
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add script to push to staging
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix lint check and api reference
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* push staging to api repo
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Deploy alerts infra as part of our installation
Conditionally deploy the infrastructure that is needed to fire alerts for our users
when bad things are happening to CDI.
Testing with `KUBEVIRT_DEPLOY_PROMETHEUS=true`
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Watch and unit test all prometheus related resources
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* add gateway for changing monitoring namespace (rbac purposes)
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* refactor test to check for exact alert name and firing state
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Align pattern of ensuring prometheus resource exists for all
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Remove potential noisy event
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Extract duplicate code to function
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Dont use empty value for prometheus label due to open issue
https://github.com/prometheus-operator/prometheus-operator/issues/4325
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
Users don't want 👽 resources in clusters,
and we should also be able to tell if were part of a broader installation.
Note:
- Operator created resources were handled in https://github.com/kubevirt/controller-lifecycle-operator-sdk/pull/18
as these labels will be common to all resources deployed by the HCO.
- Now that the controller is guaranteed to have the labels, we can set env vars
that reference the label values (fieldRef) to spare calling GET on the CR in the controllers.
(thanks mhenriks).
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* update deps and bazel
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix apidocs and unit tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix generate-verify
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
Switch admissionregistration.k8s.io/v1beta, apiregistration.k8s.io/v1beta1 to v1
as they are deprecated and will be removed from k8s-1.22.
apiextensions.k8s.io/v1beta1 was updated to v1 by https://github.com/kubevirt/containerized-data-importer/pull/1307.
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Cluster-scoped namespace transfer api and controller
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* unit tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* ObjectTransfer webhook
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* new functests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* experiment with termination grace period
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* quota test
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Formally designate CDI as owner of CDIConfig by adding annotation cdi.kubevirt.io/configAuthority
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* More robust upgrade handling. No error if beta api not installed yet.
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Generate CDI CRD using controller-tools.
This is only done for CDI CRD as it requires the existence of source
code. Other CRDs we create are created by a more bare bones pod.
CDIUninstallStrategy was missing a comment describing it, so add
one. This was spotted manually so there might be more missing.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Allow users to specify which nodes CDI pods will live on.
nodeSelector, affinity and tolerations are possible values.
This is done in the CDI CR (rather than CDIConfig) as we are
interested in having this field be populated by external operators.
Unit tests now require the existence of a CDI CR, so create it.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add a unit test covering some node placement functions
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Specify that all our pods are linux-only.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Avoid duplicate test, accidental left over.
Pointed out by awels, thanks.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Rename to cdiOperatorDeployment for clarity.
Suggested by awels
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Specify we only run on linux using the CDI CR, no need to embed this
into the code.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Don't dereference workloadPlacement for no reason
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Split off operator test to have its own AfterEach, BeforeEach.
Use even more descriptive function names.
Do all the CDI delete/restore logic in AfterEach, to ensure that
it happens and restores the deployment with the original CR even
if the test fails.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Remove XXX. This is the proper way.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Adapt to latest changes in controller_test.go (renaming import)
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Simplify, not storing intermediate value.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Don't dereference nodeplacement in callers to CreateDeployment
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Remove redundant save & restore. Unit tests do this for us.
Pointed out by awels, thanks.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Split out "find toplevel" to a utility function
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Wait for the CDI CR update to apply before continuing.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Simplify, not storing intermediate value.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Make it clear that the chosen node placement will not be schedulable.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add events to operator condition changes
Add events to operator create/delete/update of managed resources.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Updated unit tests based on comments
Signed-off-by: Alexander Wels <awels@redhat.com>
* rebase on betav1
Signed-off-by: Alexander Wels <awels@redhat.com>
* Removed start events to reduce event generation spam
Signed-off-by: Alexander Wels <awels@redhat.com>
* Move CRDS from apiextensions v1beta1 to v1.
Ensure that our code based schema validation matches the types in the api.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Ran go mod tidy and vendor in attempt to see if we could use newer runtime controller, but our go version too old.
Addressed review comments.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Addressed more review comments and fixed k8s-1.18 functional test failing.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Remove categories 'all' from cluster scoped CRDs
Signed-off-by: Alexander Wels <awels@redhat.com>
* move upload.cdi.kubevirt.io API group to v1beta1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* move core api to v1beta1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix os-3.11 cluster sync and add functional tests for alpha api
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* change more occurences of v1alpha1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* updates after rebase
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
Fix#1212
Make sure that the `Status.ObservedVersion` fiels on upgrade, even if it was not set in the previous version.
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
* use dedicated SCC
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* SCC was not getting on initial deploy
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* initial client upgrade to 1.16
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix Route detection in OpenShift
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove DOCKER_REPO from operator
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* make generate and update CDI schema
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
