* Add DataImportCron controller
-The new controller polls for updates to a registry source container
image, based on a given schedule. When updates to a container image are
detected, the controller imports the content into a new uniquely named
PVC in a golden image namespace.
-For each DataImportCron, the controller manages a corresponding
DataSource to always point to the latest most up-to-date golden
image PVC.
-DataImportCron takes ownership of an existing DataSource (with
controller: false), allowing an admin to opt-in to using auto
delivery/updates later on.
-The controller has PVC garbage collector removing old PVCs.
ToDo:
-status conditions updates
-verify full image streams support
-utests and func tests
-fixmes and commented out code
-doc
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix CR comments and fixmes
- isolate imagestream and registry specific code
- fix namespace of CronJob, and its job and pod to CDI namespace
- manage CronJob-DataImportCron ownership relationship with a finalizer,
handle DataImportCron deletion (CronJob etc.)
- remove CronJob and job pod for ImageStreams, use RequeueAfter and
cronexpr instead
- add k8s app cdi-source-update-poller executed by CronJob to poll source
image digest via skopeo inspect for url registry source, and annotate
the DataImportCron when the image was updated and pending for import based
on the cron schedule
- add cdi-source-update-poller and skopeo binary to the cdi-importer container
- complete dataimportcron-validate and its tests
- reconcile - use context.Context instead of context.TODO
- remove uncached client
- doc
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix ImageStreams watch
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron DV template instead of source
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix CR comments
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Split updateSucceeded func
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Improve cdi-source-update-poller cmd logs
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove ImageStream reconcile
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove ImageStream watch
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove unnecessary AnnSourceUpdatePending
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* More CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Idempotentify initCron
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Recreate DV in case is't not found
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron spec.importsToKeep and status.currentImports
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron controller functional test
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add insecure TLS support
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove finalizers in cluster clean script
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Bound each import to its sha256 digest instead of latest
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron controller utests
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Tests CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Minor tests CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Use a more descriptive name for variable.
(Part of more inclusive language initiative)
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Avoid references to "master" branch when "main" is default
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Rename the functional test "master" flag to "kubeURL".
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add a script to check for new additions of non-inclusive language
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* cdi-api-reference default branch is master, add an exception for it.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add long term token (10 years) to pvcs when host assisted cloning between namespaces
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* clone controller should retry if source in use
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* minor refactor if/else -> switch
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
Users don't want 👽 resources in clusters,
and we should also be able to tell if were part of a broader installation.
Note:
- Operator created resources were handled in https://github.com/kubevirt/controller-lifecycle-operator-sdk/pull/18
as these labels will be common to all resources deployed by the HCO.
- Now that the controller is guaranteed to have the labels, we can set env vars
that reference the label values (fieldRef) to spare calling GET on the CR in the controllers.
(thanks mhenriks).
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* update deps and bazel
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix apidocs and unit tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix generate-verify
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* use namespace transfer for smart clone
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* updates from test failures
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add expansion func tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add dv phases for expansion and transfer
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* rebase and integrate with storage profiles
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Fix cluster scope for StorageProfile
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Automatically provide StorageProfiles based on well known providers
A new controller scans Storage Classes and creates a new StorageProfile for each. It provides recommended parameters for storage classes with well known providers.
Add StorageProfiles CRD to "make generate-verify" target
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Refactor - extracted getStorageProfile to make code readable
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Cluster-scoped namespace transfer api and controller
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* unit tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* ObjectTransfer webhook
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* new functests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* experiment with termination grace period
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* quota test
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* sigs.k8s.io/controller-runtime/pkg/runtime/* packages are deprecated, and were moved to new paths.
Trying to upgrade sigs.k8s.io/controller-runtime to version v0.7.0 in HCO created a conflict because in v0.7.0 the deprecated packages were removed and cannot be used.
This PR replaces the deprecated packages with their new paths.
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
* Run `make deps-update`
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
* fix logger init
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
* fix test loggers
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
* Move CRDS from apiextensions v1beta1 to v1.
Ensure that our code based schema validation matches the types in the api.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Ran go mod tidy and vendor in attempt to see if we could use newer runtime controller, but our go version too old.
Addressed review comments.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Addressed more review comments and fixed k8s-1.18 functional test failing.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Remove categories 'all' from cluster scoped CRDs
Signed-off-by: Alexander Wels <awels@redhat.com>
* Extracted reconcileUploadProxyUrl
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Add FeatureGates to CDIConfig CR
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* POC of FeatureGates
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Remove FeatureGates from spec in CDIConfig, add correct handling of communication errors
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Removed feature gate from DataVolume.
This will be added when actually the first featureGate will be used to toggle a feature.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update to v1beta after rebase
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Create a featuregates package
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Rename first featureGate to HonorWaitForFirstConsumerEnabled
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* return error from feature gates api
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* move from alpha to beta snapshot API
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix broken clone tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* don't generate snapshot client
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Clean up client-go left overs after converting to runtime library.
Clean unused functions from controller utils.go
Turn util_test.go into gingko test.
Moved functions from util to proper controller if only used by that controller.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Don't export reconciler variables.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Update datavolume controller to use runtime controller library
Signed-off-by: Alexander Wels <awels@redhat.com>
* Update smart clone controller to use controller runtime library
Signed-off-by: Alexander Wels <awels@redhat.com>
* Mark smart cloned cloned PVCs are k8s.io/CloneOf
Signed-off-by: Alexander Wels <awels@redhat.com>
* Removed unused informer factories.
Fixed naming of controller from import to smart clone.
Ignore not found error on delete.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Only delete snapshot if pvc is bound
Signed-off-by: Alexander Wels <awels@redhat.com>
* Update datavolume controller to use runtime controller library
Signed-off-by: Alexander Wels <awels@redhat.com>
* Fix connection refused check. It would cause errors to be returned constantly.
Signed-off-by: Alexander Wels <awels@redhat.com>
If we create a temporary pod in the namespace which has
resource quota, `must specify quota` error occurs. Fix to
create temporary pod with default resource quota.
Co-authored-by: hyoung-90 <ssarang520@gmail.com>
Co-authored-by: Woohyung Han <woohhan@gmail.com>
Signed-off-by: anencore94 <anencore94@kaist.ac.kr>
Co-authored-by: anencore94 <37469330+anencore94@users.noreply.github.com>
Co-authored-by: hyoung-90 <58063867+hyoung-90@users.noreply.github.com>
* baseline refactoring of webhook package
* datavolume clone validation webhook
* rename datavolumes/clone-init to datavolumes/source
* add RBAC doc
* updates from review
* make clone permission check exportable function
* force dumb cloning in in functional test
Changes and flow:
* datavolume-controller:
- When clone pvc is requested and smart-clone is applicable -> Create a new VolumeSnapshot.
* Introduced smart-clone-controller:
- Listens to VolumeSnapshot changes and create a new PVC based on the snapshot.
- Upon PVC successful creation, deletes the snapshot for cleaning up.
Change-Id: I369fc92e72edb8a2cf584c8cb916795415f9d6e0
Signed-off-by: Daniel Erez <derez@redhat.com>
Co-authored-by: Fred Rolland <frolland@redhat.com>
Looks like before some of imports used glog package that set all flags
under init() method, but after dependencies code stopped to run glog
init() method on container creation, so we need to set flags from klog
and parse flags in the right order.
This finishes up the last of the golint implementation, with the
addition of the cmd, tests and tools directories we are now running
golint tests on all of the current go source files in the project.
This change adds all the little fixes (mostly just commenting and
naming) and also enables the new diretories in the lint test that we
gate on.
This change just updates the common package to pass golint. Of course
that has some reaching implications into other packages with renaming of
constants.
I've intentionally kept this patch set to the bare minimum for the lint
test, I do think that we should put some effort into our use of
constants (especially those in common.go) in the future.
Ideally we'd declare constants where they're used, and we wouldn't
export them if they don't need to be. There are some in here that
are only used in one or two packages, and that can be cleaned up pretty
easily however it's likely that there might be plans to use them
elsewhere in the future.
[WIP] CDI: expectations and single queue - cloning refactoring
Adding 'expectPodCreate' call before creating target pod
minor change
Adding 'expectPodCreate' call before creating target pod
updating according feedback
Adding cloner unit test
code updates
Signed-off-by: tavni <tavni@redhat.com>
more updates
Signed-off-by: tavni <tavni@redhat.com>
minor change for util_test.go
Signed-off-by: tavni <tavni@redhat.com>
updating unit tests
Signed-off-by: tavni <tavni@redhat.com>
finished unit test for cloning refactoring
Signed-off-by: tavni <tavni@redhat.com>
rebase
Signed-off-by: tavni <tavni@redhat.com>
updating unit test after rebase
Signed-off-by: tavni <tavni@redhat.com>
gofmt errors
Signed-off-by: tavni <tavni@redhat.com>
* Pull in expectations code from kubevirt
Signed-off-by: David Vossel <davidvossel@gmail.com>
* add expectations to datavolume controller
Signed-off-by: David Vossel <davidvossel@gmail.com>
* Refactor importer pod to use single queue and expectations
Signed-off-by: David Vossel <davidvossel@gmail.com>
* update utils tests to use single queue
Signed-off-by: David Vossel <davidvossel@gmail.com>
* re-introduce import controller test suite
Signed-off-by: David Vossel <davidvossel@gmail.com>
* make clone controllers pass after utils refactor
Signed-off-by: David Vossel <davidvossel@gmail.com>
* Use log level constants in cdi controllers
Signed-off-by: David Vossel <davidvossel@gmail.com>
* Remove useless shadow variables in expectations pkg
Signed-off-by: David Vossel <davidvossel@gmail.com>
* required changes to get owner references working with openshift 1.10
Signed-off-by: David Vossel <davidvossel@gmail.com>
* update functional tests to work with importer controller refactor
Signed-off-by: David Vossel <davidvossel@gmail.com>
* Rename AnnImportPVC to LabelImportPvc in order to reflect its use
Signed-off-by: David Vossel <davidvossel@gmail.com>
* Add comment about expectations code
Signed-off-by: David Vossel <davidvossel@gmail.com>
Adds go templated controller manifest
Adds make targets for template generation
Removes hard coded version values
Enables template generation and publishing in CI
