* check if pod used
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Test scratch deletion during import
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handle scratch missing just after being created in utility function.
There was a possibility that just after creating the scratch it is deleted, and the next get results in error. This error was ignored - not returned from util - resulting in wrong behavior.
Other fix here is to skip pods with status PodSucceeded and PodFailed in getPodsUsingPVCs function - such Pods are not using PVC, as these pods are already done.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Reschedule reconcile for the pvc until pvc import is complete.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Increase number of NFS volumes, and document the test.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Try another quick fix for verifier pod flakiness.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Cleanup defaults
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* When validating disk space, reserve space for filesystem overhead
The amount of available space in a filesystem is not exactly
the advertise amount. Things like indirect blocks or metadata
may use up some of this space. Reserving it to avoid reaching
full capacity by default.
This value is configurable from the CDIConfig object spec,
both globally and per-storageclass.
The default value is 0.055, or "5.5% of the space is
reserved". This value was chosen because some filesystems
reserve 5% of the space as overhead for the root user and
this space doubles as reservation for the worst case
behaviour for unclear space usage. I've chosen a value
that is slightly higher.
This validation is only necessary because we use sparse
images instead of fallocated ones, which was done to have
reasonable alerts regarding space usage from various
storage providers.
---
Update CDIConfig filesystemOverhead status, validate, and
pass the final value to importer/upload pods.
Only the status values controlled by the config controller
are used, and it's filled out for all available storage
classes in the cluster.
Use this value in Validate calls to ensure that some of the
space is reserved for the filesystem overhead to guard from
accidents.
Caveats:
Doesn't use Default: to define the default of 0.055, instead
it is hard-coded in reconcile. It seems like we can't use a
default value.
Validates the per-storageClass values in reconcile, and
doesn't reject bad values.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Use util GetStorageClassByName
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Test filesystem overhead validation against async upload endpoint
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* wait for NFS PVs to be deleted before continuing
Intended to help with flakes, but didn't make a difference.
Probably still worth doing.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Avoid using the uncached client unnecessarily
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add error handling for the case where even a default SC is not found
Note that this change isn't expected to make a difference, as we
check if the targetStorageClass is nil later on and have the same
behaviour, but this is probably more correct API usage.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add testing for the validation of filesystem overhead values
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Fix logical error in waiting for NFS PVs.
Wait for all of them, not just the last one.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add library function to determine if a PVC has been populated fully.
The logic is as following:
If PVC has no ownerRef, then we assume something else fully populated it and
will return true
If PVC has an ownerRef and its a DataVolume, then look up the DataVolume
If DV.status.Phase == succeeded, return true, return false otherwise.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Renamed functions to better indicate its purpose.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Generate CDI CRD using controller-tools.
This is only done for CDI CRD as it requires the existence of source
code. Other CRDs we create are created by a more bare bones pod.
CDIUninstallStrategy was missing a comment describing it, so add
one. This was spotted manually so there might be more missing.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Allow users to specify which nodes CDI pods will live on.
nodeSelector, affinity and tolerations are possible values.
This is done in the CDI CR (rather than CDIConfig) as we are
interested in having this field be populated by external operators.
Unit tests now require the existence of a CDI CR, so create it.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add a unit test covering some node placement functions
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Specify that all our pods are linux-only.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Avoid duplicate test, accidental left over.
Pointed out by awels, thanks.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Rename to cdiOperatorDeployment for clarity.
Suggested by awels
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Specify we only run on linux using the CDI CR, no need to embed this
into the code.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Don't dereference workloadPlacement for no reason
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Split off operator test to have its own AfterEach, BeforeEach.
Use even more descriptive function names.
Do all the CDI delete/restore logic in AfterEach, to ensure that
it happens and restores the deployment with the original CR even
if the test fails.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Remove XXX. This is the proper way.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Adapt to latest changes in controller_test.go (renaming import)
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Simplify, not storing intermediate value.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Don't dereference nodeplacement in callers to CreateDeployment
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Remove redundant save & restore. Unit tests do this for us.
Pointed out by awels, thanks.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Split out "find toplevel" to a utility function
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Wait for the CDI CR update to apply before continuing.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Simplify, not storing intermediate value.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Make it clear that the chosen node placement will not be schedulable.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* update k8s deps to 1.18.6 and controller runtime to 0.6.2
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove building code generators from docker image. This way the k8s ligray version only has to be updated in go.mod
Do more stuff in the bazel container. Faster and better interop
Fix unit tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* make format
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove unnecessary rsync
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* redo code generator dep management
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* builder uses go modules
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Move CRDS from apiextensions v1beta1 to v1.
Ensure that our code based schema validation matches the types in the api.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Ran go mod tidy and vendor in attempt to see if we could use newer runtime controller, but our go version too old.
Addressed review comments.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Addressed more review comments and fixed k8s-1.18 functional test failing.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Remove categories 'all' from cluster scoped CRDs
Signed-off-by: Alexander Wels <awels@redhat.com>
* Set the WaitForFirstConsumer phase on DataVolume when storage uses the WaitForFirstConsumer binding mode and is not bound yet.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Skip PVC if not bound in import|clone|upload controllers.
This is done so the VM pod(not the cdi pod) will be the first consumer, and the PVC can be scheduled on the same location as the pod.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
fixup! Skip PVC if not bound in import|clone|upload controllers.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update importer tests to force bind the PCV by scheduling a pod for pvc, when storage class is wffc.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update datavolume tests to force bind the PCV by scheduling a pod for pvc, when storage class is wffc.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update upload controller and upload tests to correctly handle force binding the PCV by scheduling a pod for pvc, when storage class is wffc.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update clone tests to force bind the PCV by scheduling a pod for pvc when the storage class is wffc.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update cloner multi-node tests to force bind the PCV by scheduling a pod for pvc when storage class is wffc.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Correct after automerge
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Improve/simplify tests
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Fix error in import test.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update transport_test,operator_test.go
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update rbac_test.go and leaderelection_test.go
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Improve Datavolume and PVC Checks for WFFC.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handle wffc only if feature gate is open - import-controller
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* TEST for Handle wffc only if feature gate is open - import-controller - TEST
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handle wffc only if feature gate is open - upload-controller with test
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* rename and simplify checks
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* cleanup after rebase
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* update tests after rebase
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* update tests after rebase
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* more cleanups
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Document new WFFC behavior
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Document new HonorWaitForFirstConsumer option
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* update docs according to comments
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* extract common function, cleanup - code review fixes
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* add comment for another pr - 1210, so it can have easier merge/rebase
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* typo
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Simplify getStoragebindingMode - code review comments
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Add FeatureGates interface - code review fix
Additionally pass the features gates instead of the particular feature gate value,
and let shouldReconcilePVC decide what to do with the feature gate. That way shouldReconcilePVC
contains all the logic, and the caller does not need to do additional calls to provide parameters.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update matcher
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* move upload.cdi.kubevirt.io API group to v1beta1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* move core api to v1beta1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix os-3.11 cluster sync and add functional tests for alpha api
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* change more occurences of v1alpha1
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* updates after rebase
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* don't create snapshot or clone pods if pvcs in use
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* cleanup pods during functional tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* kill mmore pods blocking clone tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix typos
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Handle labels length correctly
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handle service name generation correctly
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Remove not needed labels
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Store import pod name in annotation
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Enable long DV name
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handle name with dot when creating service/label name
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Test long names on import, upload and clone
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Store upload pod name in annotation
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Store importer scratch pvc name in annotation
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Quick fix for tests (need improvements)
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Cleanup handling scratch name
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Ensure pod/service name conflicts are handled
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handle client errors when trying to get the import pod
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Style improvements, and other code review fixes.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Store clone source pod name in an annotation
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Correct name initialization and tests
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Do not init name if pod already exists. It is not needed.
The situation of having a pod but not name on annotation can happen after the upgrade, when we have a legacy pvc and pod already existing, but clone operation not finished. But when we already have the pod, then in the code (currently) we do not need the name from annotation.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Cleanup scratch name handling
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Use constant for max dv name in validation
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Simplify clone source pod name initialization
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* move from alpha to beta snapshot API
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix broken clone tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* don't generate snapshot client
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Add conditions to DataVolume and add condition annotation to PVCs
Signed-off-by: Alexander Wels <awels@redhat.com>
* Added reason codes to conditions
Updated functional tests to use reason codes
Fixed broken functional test.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Added last transition time
Added documentation
Signed-off-by: Alexander Wels <awels@redhat.com>
* Switch to conditions slice from conditions pointer slice.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Add heartbeat time for conditions.
Renamed AnnRunningCondition to AnnLastTerminationMessage
Only update reason/message on heartbeat or state change.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Rename heartBeat to heartbeat
Re-factor datavolume-conditions to properly update heartbeat time.
Signed-off-by: Alexander Wels <awels@redhat.com>
* renamed AnnRunningConditionReason to AnnLastTerminationReason
Signed-off-by: Alexander Wels <awels@redhat.com>
* Revert lastTerminatedMessage/Reason rename
Added separate AnnScratchBoundCondition/Message/Reason for scratch space
Signed-off-by: Alexander Wels <awels@redhat.com>
* Update bound condition to take scratch or source PVC status into consideration.
Update running condition to take source pod status into consideration
Signed-off-by: Alexander Wels <awels@redhat.com>
* Addressed comments in review.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Properly get scratch space pvc phase.
Signed-off-by: Alexander Wels <awels@redhat.com>
These are basically no-op DataVolumes for PVCs that already exist.
I see this mostly as an internal thing to facilitate VM restore from snapshot
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Clean up client-go left overs after converting to runtime library.
Clean unused functions from controller utils.go
Turn util_test.go into gingko test.
Moved functions from util to proper controller if only used by that controller.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Don't export reconciler variables.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Add a field to DataVolume to track the number of retries/pod restarts
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Add a field to DataVolume to track the number of retries/pod restarts
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Make RESTARTS non-empty on DataVolume (shows as as 0 'zero')
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Test reporting restarts on DataVolume when importing.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Fix tests
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Code review fixes
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Restart Count status test for upload and clone controller
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
This PR introduces new import data source to fetch disks from ovirt
and store them in storage available in the cluster. It uses ovirt-sdk
to check whether the this is there, the size of the disk and it obtains
signed ticket which allows to get the disk using ovirt-imageio-proxy.
In order to use it a user needs to create configmap with ovirt api
certificate and secret with ovirt engine user and password. Both are
required and validated by webhook. In order to start import process
we need to post following definition:
---
apiVersion: cdi.kubevirt.io/v1alpha1
kind: DataVolume
metadata:
name: "imageio-dv"
spec:
source:
imageio:
url: "<engine-api-endpoint>"
secretRef: "<secret-name-containing-engine-credentials>"
certConfigMap: "<configmap-name-containing-engine-api-certificate>"
diskId: "<disk-id-which-we-want-to-import>"
pvc:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "64Mi"
---
Signed-off-by: Piotr Kliczewski <piotr.kliczewski@gmail.com>
* initial client upgrade to 1.16
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix Route detection in OpenShift
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
If we create a temporary pod in the namespace which has
resource quota, `must specify quota` error occurs. Fix to
create temporary pod with default resource quota.
Co-authored-by: hyoung-90 <ssarang520@gmail.com>
Co-authored-by: Woohyung Han <woohhan@gmail.com>
Signed-off-by: anencore94 <anencore94@kaist.ac.kr>
Co-authored-by: anencore94 <37469330+anencore94@users.noreply.github.com>
Co-authored-by: hyoung-90 <58063867+hyoung-90@users.noreply.github.com>
Use the term that appears in YAML files, volumeMode, and print
which one it is (may be an implicit value picked from the default)
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* unique cert per clone source pod
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* upload controller shouldn't cache certs
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix test variable names
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* baseline refactoring of webhook package
* datavolume clone validation webhook
* rename datavolumes/clone-init to datavolumes/source
* add RBAC doc
* updates from review
* make clone permission check exportable function
* force dumb cloning in in functional test
Changes and flow:
* datavolume-controller:
- When clone pvc is requested and smart-clone is applicable -> Create a new VolumeSnapshot.
* Introduced smart-clone-controller:
- Listens to VolumeSnapshot changes and create a new PVC based on the snapshot.
- Upon PVC successful creation, deletes the snapshot for cleaning up.
Change-Id: I369fc92e72edb8a2cf584c8cb916795415f9d6e0
Signed-off-by: Daniel Erez <derez@redhat.com>
Co-authored-by: Fred Rolland <frolland@redhat.com>
