* periodic sync CSI snapshot CRD check
It was possible for the CSI snapshot CRD check to fail silently and
prevent the smart clone controller from starting during the cdi deployment
pod start up. This would prevent smart clone from working properly.
This adds a periodic sync of 1 minute for checking the CRDs. We also
log failures that are not is not found so we can more easily detect this
situation as humans.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Change location of the start controller call.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Add alert for DataImportCron failing
DataImportCrons now have conditions (particularly UpToDate) that tell us if
things are going as planned. We can utilize those to alert whenever were not UpToDate for a while.
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Address CR review; don't List, increment when needed via corresponding instance
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Address review & bugfix: don't update metric if err occurs
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* upToDateCondition => prevUpToDateCondition so it's clear we're deciding if we should inc/dec based on that
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Don't store state in controller; change metric type to GaugeVec (bool metric per DIC)
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add DataImportCron status conditions
The `DataImportCron` controller updates the status conditions in a
controlled `DataImportCron` and its managed `DataSource`.
DataImportCron:
- UpToDate - indicates if the the most recent import is successful and
`DataSource` is up-to-date. Updated to False whenever the source
digest (latest sha256) is updated.
- Progressing - indicates whether the cron is currently in the process
of importing. Updated to True if there is a current import and its
`DataVolume` is `ImportInProgress`, otherwise False.
DataSource:
- Ready - indicates that the corresponding pvc exists and is populated.
Update according to `DataImportCron.Status.LastImportedPVC`
`DataVolume`'s `DataVolumeReady` condition, if the `DataVolume`
exists. Otherwise False. Unlike `DataImportCron` `UpToDate`
condition, this one does not care about newer source digest.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron RetentionPolicy and remove OwnerReferences
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* More CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add tests for retention policies and datasource/datavolume recreation if deleted
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add status condition tests
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* SetRecommendedLabels for all created CRs
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add alert for incomplete storage profiles
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Run metric tests on both openshift and k8s
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add functional test for storageprofile metrics
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Delete profile as a follow up to storage class getting deleted
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Address review, alter tests to cover List metric approach
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Address review; individually loop over metric decrement, shorten reconcile.Result{}
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Address review; deletion timestamp not possible when err/teardown in AfterEach
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add DataImportCron controller
-The new controller polls for updates to a registry source container
image, based on a given schedule. When updates to a container image are
detected, the controller imports the content into a new uniquely named
PVC in a golden image namespace.
-For each DataImportCron, the controller manages a corresponding
DataSource to always point to the latest most up-to-date golden
image PVC.
-DataImportCron takes ownership of an existing DataSource (with
controller: false), allowing an admin to opt-in to using auto
delivery/updates later on.
-The controller has PVC garbage collector removing old PVCs.
ToDo:
-status conditions updates
-verify full image streams support
-utests and func tests
-fixmes and commented out code
-doc
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix CR comments and fixmes
- isolate imagestream and registry specific code
- fix namespace of CronJob, and its job and pod to CDI namespace
- manage CronJob-DataImportCron ownership relationship with a finalizer,
handle DataImportCron deletion (CronJob etc.)
- remove CronJob and job pod for ImageStreams, use RequeueAfter and
cronexpr instead
- add k8s app cdi-source-update-poller executed by CronJob to poll source
image digest via skopeo inspect for url registry source, and annotate
the DataImportCron when the image was updated and pending for import based
on the cron schedule
- add cdi-source-update-poller and skopeo binary to the cdi-importer container
- complete dataimportcron-validate and its tests
- reconcile - use context.Context instead of context.TODO
- remove uncached client
- doc
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix ImageStreams watch
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron DV template instead of source
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix CR comments
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Split updateSucceeded func
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Improve cdi-source-update-poller cmd logs
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove ImageStream reconcile
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove ImageStream watch
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove unnecessary AnnSourceUpdatePending
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* More CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Idempotentify initCron
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Recreate DV in case is't not found
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron spec.importsToKeep and status.currentImports
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron controller functional test
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add insecure TLS support
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove finalizers in cluster clean script
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Bound each import to its sha256 digest instead of latest
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron controller utests
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Tests CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Minor tests CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Use a more descriptive name for variable.
(Part of more inclusive language initiative)
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Avoid references to "master" branch when "main" is default
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Rename the functional test "master" flag to "kubeURL".
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add a script to check for new additions of non-inclusive language
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* cdi-api-reference default branch is master, add an exception for it.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add long term token (10 years) to pvcs when host assisted cloning between namespaces
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* clone controller should retry if source in use
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* minor refactor if/else -> switch
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
Users don't want 👽 resources in clusters,
and we should also be able to tell if were part of a broader installation.
Note:
- Operator created resources were handled in https://github.com/kubevirt/controller-lifecycle-operator-sdk/pull/18
as these labels will be common to all resources deployed by the HCO.
- Now that the controller is guaranteed to have the labels, we can set env vars
that reference the label values (fieldRef) to spare calling GET on the CR in the controllers.
(thanks mhenriks).
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* update deps and bazel
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix apidocs and unit tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix generate-verify
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* use namespace transfer for smart clone
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* updates from test failures
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add expansion func tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add dv phases for expansion and transfer
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* rebase and integrate with storage profiles
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Fix cluster scope for StorageProfile
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Automatically provide StorageProfiles based on well known providers
A new controller scans Storage Classes and creates a new StorageProfile for each. It provides recommended parameters for storage classes with well known providers.
Add StorageProfiles CRD to "make generate-verify" target
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Refactor - extracted getStorageProfile to make code readable
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Cluster-scoped namespace transfer api and controller
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* unit tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* ObjectTransfer webhook
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* new functests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* experiment with termination grace period
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* quota test
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* sigs.k8s.io/controller-runtime/pkg/runtime/* packages are deprecated, and were moved to new paths.
Trying to upgrade sigs.k8s.io/controller-runtime to version v0.7.0 in HCO created a conflict because in v0.7.0 the deprecated packages were removed and cannot be used.
This PR replaces the deprecated packages with their new paths.
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
* Run `make deps-update`
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
* fix logger init
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
* fix test loggers
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
* Move CRDS from apiextensions v1beta1 to v1.
Ensure that our code based schema validation matches the types in the api.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Ran go mod tidy and vendor in attempt to see if we could use newer runtime controller, but our go version too old.
Addressed review comments.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Addressed more review comments and fixed k8s-1.18 functional test failing.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Remove categories 'all' from cluster scoped CRDs
Signed-off-by: Alexander Wels <awels@redhat.com>
* Extracted reconcileUploadProxyUrl
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Add FeatureGates to CDIConfig CR
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* POC of FeatureGates
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Remove FeatureGates from spec in CDIConfig, add correct handling of communication errors
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Removed feature gate from DataVolume.
This will be added when actually the first featureGate will be used to toggle a feature.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update to v1beta after rebase
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Create a featuregates package
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Rename first featureGate to HonorWaitForFirstConsumerEnabled
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* return error from feature gates api
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* move from alpha to beta snapshot API
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix broken clone tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* don't generate snapshot client
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Clean up client-go left overs after converting to runtime library.
Clean unused functions from controller utils.go
Turn util_test.go into gingko test.
Moved functions from util to proper controller if only used by that controller.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Don't export reconciler variables.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Update datavolume controller to use runtime controller library
Signed-off-by: Alexander Wels <awels@redhat.com>
* Update smart clone controller to use controller runtime library
Signed-off-by: Alexander Wels <awels@redhat.com>
* Mark smart cloned cloned PVCs are k8s.io/CloneOf
Signed-off-by: Alexander Wels <awels@redhat.com>
* Removed unused informer factories.
Fixed naming of controller from import to smart clone.
Ignore not found error on delete.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Only delete snapshot if pvc is bound
Signed-off-by: Alexander Wels <awels@redhat.com>
* Update datavolume controller to use runtime controller library
Signed-off-by: Alexander Wels <awels@redhat.com>
* Fix connection refused check. It would cause errors to be returned constantly.
Signed-off-by: Alexander Wels <awels@redhat.com>
If we create a temporary pod in the namespace which has
resource quota, `must specify quota` error occurs. Fix to
create temporary pod with default resource quota.
Co-authored-by: hyoung-90 <ssarang520@gmail.com>
Co-authored-by: Woohyung Han <woohhan@gmail.com>
Signed-off-by: anencore94 <anencore94@kaist.ac.kr>
Co-authored-by: anencore94 <37469330+anencore94@users.noreply.github.com>
Co-authored-by: hyoung-90 <58063867+hyoung-90@users.noreply.github.com>
* baseline refactoring of webhook package
* datavolume clone validation webhook
* rename datavolumes/clone-init to datavolumes/source
* add RBAC doc
* updates from review
* make clone permission check exportable function
* force dumb cloning in in functional test
Changes and flow:
* datavolume-controller:
- When clone pvc is requested and smart-clone is applicable -> Create a new VolumeSnapshot.
* Introduced smart-clone-controller:
- Listens to VolumeSnapshot changes and create a new PVC based on the snapshot.
- Upon PVC successful creation, deletes the snapshot for cleaning up.
Change-Id: I369fc92e72edb8a2cf584c8cb916795415f9d6e0
Signed-off-by: Daniel Erez <derez@redhat.com>
Co-authored-by: Fred Rolland <frolland@redhat.com>
Looks like before some of imports used glog package that set all flags
under init() method, but after dependencies code stopped to run glog
init() method on container creation, so we need to set flags from klog
and parse flags in the right order.
This finishes up the last of the golint implementation, with the
addition of the cmd, tests and tools directories we are now running
golint tests on all of the current go source files in the project.
This change adds all the little fixes (mostly just commenting and
naming) and also enables the new diretories in the lint test that we
gate on.
