* Create CRD for volumeuploadsource populator
This CRD will be used in the DataSourceRef on PVCs
to trigger population that upload to the volume.
This will be performed by the upload populator
that will be added in future commits.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Create upload populator controller
The upload populator controller can be used
standalone without the need of datavolume.
It reconciles pvc with upload dataSourceRef
and uses populators API to populated the pvc
with an upload command.
The controller creates pvc' with upload
annotation. After the upload completes it
rebinds the pv to the original target pvc and
deletes pvc prime.
Eventually we get a bound PVC which is already
populated.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Adjust upload-proxy to handle upload population
In case of pvc with datasourceref to upload population
we should create the url to the upload server with the
pvc' name.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add tests for upload population
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add unit tests for upload populator
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add preallocation to volumeuploadsource crd
Also some other small fixes
Signed-off-by: Shelly Kagan <skagan@redhat.com>
---------
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Start adding the golangci-lint to CI
golangci-lint is a collection of many linters. This PR adds
golangci-lint to the CI. For strat, it enables the govet linter, and fix
its single finding.
The PR adds this linter to the `test-lint` Makefile target.
The new .golangci.yml file is the configuration for the linter.
golangci-lint version was set to the latest one - v1.52.2.
It is defined in hack/build/run-linters.sh
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
* golangci-lint: enable gosimple and fix findings
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
* golangci-lint: enable unused and fix findings
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
---------
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
* Add support for imagePullSecrets in the CDI CR, to support pulling
images from repositories that require secrets.
The imagePullSecrets is propagated to the following components: cdi-apiserver,
cdi-deployment, and cdi-uploadproxy. The definition of imagePullSecrets in
cdi-operator must be done manually.
Signed-off-by: Gleb Aronsky <gleb.aronsky@windriver.com>
* Modifying code to incorporate review comments.
Signed-off-by: Gleb Aronsky <gleb.aronsky@windriver.com>
---------
Signed-off-by: Gleb Aronsky <gleb.aronsky@windriver.com>
Co-authored-by: Gleb Aronsky <gleb.aronsky@windriver.com>
* Fix hostpath CSI being skipped as "Not HPP"
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Fall back to host assisted if immediate bind requested
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
---------
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
- Split the huge DV controller into smaller op-specific DV controllers -
import, clone, upload
- Add common watch-adding function so each controller watches only its
relevant DVs
- Refactor the common Reconcile() to use interface DataVolumeReconciler
implemented by each controller
- Move all functions, structs, consts to the relevant controller
- Split the utests per controller
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* remove root worker pods
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove selinux requirement for worker pods
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* run tests in restricted namespace and required changes
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* handle empty tar
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add PSA label when running functional tests in OpenShift
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* cannot use restricted PSA with istio (for now)
refactor scc management
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix clean script
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Modify upload-controller to keep the annotations of a PVC if a pod fails in another controller
When a pod fails, some annotations (running condition, pod phase...) are updated in the affected PVC to let other controllers know.
However, due to a lack of synchronization between some controllers, a race condition could happen where, if a pod succeeds just after a pod fails in a different controller, the annotations set in the error handling would just be updated inapropiately.
This commit modifies the upload-controller to check for clone failures before updating annotations.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add unit tests to updateUploadAnnotations
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Update functional tests in cloner_test to check for conditions and annotations after a pod fails
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Improve error handling in source-clone pod to avoid overwritting the DV running condition
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Improve the error handling when pod creation fails
When pod creation fails, the error is usually logged without providing additional information to the user. This behavior is especially risky when the user lacks the permits to check the logs, making it unintuitive and almost impossible to find the source of the problem.
This commit improves the error handling of the pod-creation process, so pertinent info about the failure is included in the pod's PVC.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Update functional tests to check for events when pod-creation fails
Since error handling in pod-creation has been improved in our controllers, this commit introduces several changes in the corresponding functional tests to properly cover the new behavior included when pod-creation fails.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Update unit-tests after improving error-handling of pods for proper coverage
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Minor fixes and improvements on error handling for pods
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Modify datavolume-controller to change the running condition of datavolumes when a pod fails
Until this commit, the way of handling pod errors in the datavolume-controller has been to change the affected datavolume's phase to failed, which conflicts with the declarative approach of the controllers.
This commit modifies this behavior so that, when a pod fails, the affected datavolume's running condition is changed to false while the phase remains unchanged.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add TLS Security Profile API
TLSSecurityProfile is used by operators to apply cluster-wide TLS security settings to operands.
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Update apiserver & uploadproxy server TLS config on CDIConfig TLS knob change
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Propagate TLS config to uploadserver as well
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add functests for apiserver and upload that ensure value is respected
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Comply with restricted security context in kubernetes
Ensure CDI pods comply with the restricted security context as much as
possible (have to be root for nbdkit and block devices). Also cannot set
SeccompProfile since SCC won't allow us to set it.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Changed path /var/local/all_certs to stay in /var
Signed-off-by: Alexander Wels <awels@redhat.com>
* Test: Clone fs to block fails on size validation
When requesting size `X` with filesystem volume mode and storage api the size
is increased for the fs overhead. When trying to clone to block using
the same size `X` the clone fails because the target is smaller than source.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Improve size validation for clone
Skip size validation for filesystem in webhook and include filesystem
overhead when doing the validation in controller.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Correct size validation for smart clone
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Correct unit test with fs overhead
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Restore CDI Config after each clone test
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Review cleanup
Removing redundant conversions and not useful comments
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Allow empty DV size when cloning using storage API
When cloning a Data Volume, the size of the target can be potentially obtainable via the source PVC, which discards the need to explicitly specify it.
Considering that, this commit introduces a change in the correspondent validation webhook to allow omitting the resources.request.storage field when cloning a PVC using the storage API.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Modify datavolume-controller to allow obtaining storage size from source PVC when cloning
When cloning a PVC, if the target's size is not specified, said value can be attainable from the source PVC.
This commit introduces a change in datavolume controller so, in case of detecting an empty storage size, said value can be obtained when performing CSI and Smart cloning.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Update unit tests for datavolume-validation after enabling cloning with empty size
This commit updates the unit testing for the datavolume validation webhook, covering the possibility of cloning a PVC without setting any storage size.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Update unit testing for controller-related functions after enabling cloning with empty size
This commit includes unit tests for the volumeSize() function after enabling creating clones with blank size.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Update the datavolume controller to create a size-detection pod when performing host-assisted clone
When performing a host-assisted clone with empty clone size, simply copying the original PVC size could lead to potential overhead miscalculations if the source's VolumeMode is "filesystem".
When that's the case, an inspection pod will be created in the datavolume controller so it extracts the size of the virtual image using qemu-img.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Include an image-size detection tool to allow cloning with empty DV size
This commit introduces a new tool in charge of collecting the virtual image size when cloning with an empty DV size. In some cases where said value is unattainable from the original PVC's spec, the datavolume controller will create a new pod containing this new tool.
The binary will then run the 'qemu-img' command and handle its results appropriately.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Optimize the clone-size lookup process to avoid creating unnecessary size-detection pods
When performing host-assisted clone with an empty DV size, in some cases, a size-detection pod is used to obtain the required capacity.
This commit tries to optimize this process to keep the collected value as a PVC annotation, that is checked in subsequent clones to avoid creating redundant pods.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Minor fixes and improvements on mechanism for cloning with empty storage size
* Add new optional flag on size-detection binary to enable using a different URI scheme
* Improve the pod-creation mechanism so the pod is not created until the source PVC has finished the import
* Modify size-finlation mechanism to account for possible round-downs when importing the source image
* Improve the size inflation mechanism so only PVCs with filesystem as volume mode are considered
* Minor style corrections
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Modify the clone-controller to allow skipping the clone size validation in some cases
Due to filesystem overhead differences, the target's size can sometimes be smaller than the source's one when obtaining said value with the size-detection pod.
This commit introduces minor changes in the clone-controller so we can skip the size validation in those cases.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Minor changes and improvements in size-detection mechanism following PR review
* Added new UT that covers using empty storage API for non-cloning sources
* Added new watch on datavolume-controller that looks for changes in the size-detection pod
* Removed redundant and unnecessary specs on size-detection pod
* Added error handling when reading the pod's termination message
* Moved general-usage functions to 'util.go' file
* Updated 'datavolumes' documentation to reference the possibility of omitting the storage size when cloning
* Minor style corrections
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Add unit tests that cover the size-detection mechanism in the DataVolume controller
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Include functional tests for cloning without specifying storage size
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Improve error handling in the creation/deletion process of the size-detection pod
This commit introduces additional handling in case of error after and during the size-detection pod is created.
It also updates several related unit tests.
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Minor fixes to improve fsOverhead calculations when cloning with empty storage size
* Modified the size-detection mechanism so we account for fsOverhead when cloning to filesystem volume mode in all cases
* Clean up the code for reconciling when cloning a PVC that is not ready
* Minor fix in functional test so it works when cloning from block to filesystem volume mode
Signed-off-by: Alvaro Romero <alromero@redhat.com>
* Introduce controller-runtime-sdk api package
Split controller-runtime-sdk into the base package and
controller-runtime-sdk/api.
Signed-off-by: Roman Mohr <rmohr@redhat.com>
* go mod vendor
Signed-off-by: Roman Mohr <rmohr@redhat.com>
* Update code references
Signed-off-by: Roman Mohr <rmohr@redhat.com>
* Update clone size validation logic
The case with DV using spec.storage API needs
more complex validation that will be added in the
clone controller. The API webhook validation
for that case is removed.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Improve DV phase failure message in tests
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Add test and warning event for clone size
During clone check if actual requested size on source volume is bigger
than target requested size and emit an event to notify user about situation.
Actual size on filesystem is lower that requested, because of possible filesystem overhead. When using storage API the overhead will be applied on target.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Code Review cleanup - Removing debug logs
Removed some garbage left after troubleshooting.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Move fn GetUsableSpace to common utils
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* uploadserver key/cert in secret
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix func test to accept "secret get"
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Add degraded alert
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add unusual restart count metric
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Add actual firing alerts (degraded/restartcount)
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Test newly added metrics
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Review: Rename metric to match conventions, func to check if test is eligible to run metric tests
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Get rid of similar funcs, reconcile more generally
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Update datavolume conditions when quota exceeded when creating pvc
When creating the pvc from the dv the pvc size
can exceed the allowed quota, in such case so far the only
indication was to look in the logs.
Now added indication in the data volume conditions
(when possible) and emitted event.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add functional tests to check the new conditons and event
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* tests cosmetics
-use existing functions
-add missing checks on errors
-remove unused code
-etc..
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add support for archive upload
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* fix golang errors
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Change storage profile property set to support more then one set
So far CDI supported only 1 claim propery set. We want to be able
to support more then one so in case the user provides to the
DV storage volumeMode without accessMode or vice versa cdi
will be able to fit to it the most appropriate match.
Added to rook ceph block a second default of filesystem
volume mode with RWO access mode, it will support archive
upload which has default of filesystem mode.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* CR fix - change to one endpoint for the user
upload proxy will identify if the upload is archive
or not by looking at the content type annotation on
the pvc. If the content type is archive it will route
the uplaod to upload server to a new archive upload uri.
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Add storage profile and data volume controllers unit tests
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* CR fixes
* add default volume mode to archive content type
* upload server use data processor for archive upload
* tests for volume mode with archive content type
* tests for archive upload of compressed tar
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* Adjust imports acording to new apis dir
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* CR small fixes
Signed-off-by: Shelly Kagan <skagan@redhat.com>
* move apis to new staging area
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* add script to push to staging
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix lint check and api reference
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* push staging to api repo
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
Users don't want 👽 resources in clusters,
and we should also be able to tell if were part of a broader installation.
Note:
- Operator created resources were handled in https://github.com/kubevirt/controller-lifecycle-operator-sdk/pull/18
as these labels will be common to all resources deployed by the HCO.
- Now that the controller is guaranteed to have the labels, we can set env vars
that reference the label values (fieldRef) to spare calling GET on the CR in the controllers.
(thanks mhenriks).
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Strip newlines when writing a termination message.
Otherwise it isn't visible, at least when viewing in the -o yaml view.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Write down the nbdkit output and add it to the error output
With the added output from nbdkit, we can see the reason for the
non-existence of the nbdkit socket.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Don't set that we're waiting if it's CrashLoopBackOff
It's better to have the reason for the crash (terminate message)
than "backing off 5 minutes"
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Simplify all "image too large to fit" terminate reasons.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Move verifyConditions to utils, no functional change
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Switch test for image too large to test condition and not log
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Remove unused branch
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Rename setConditionFromPodWithPrefix to setAnnotationsFromPodWithPrefix
No functional change. Intended to be followed by some refactoring.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Fold restart count logic into the common setAnnotationsFromPodWithPrefix
Changing to >= rather than > to ensure a zero pod restart count is
always used -- the import controller unit tests request this.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Reduce indentation by returning right away.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Fold check for pod being nil into common code.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* move saveVddkAnnotations into util and make it unconditional
Call it setVddkAnnotations for consistency.
Check for not-terminated inside the function, not outside.
Removes check for source being VDDK (to avoid passing more arguments):
it won't match the regex anyway.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Reduce indentation by bailing on failure.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Reorder parameters to mirror the order in the function name
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Use a named variable for first container state
Yields shorter, more legible lines.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Use a constant for the nbdkit log.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add more information to function description - also logs to file
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* update deps and bazel
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix apidocs and unit tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix generate-verify
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Refactoring - move PreallocationApplied flag definition to common
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Preallocate cloning DataVolumes
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Cluster-scoped namespace transfer api and controller
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* unit tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* ObjectTransfer webhook
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* new functests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* experiment with termination grace period
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* quota test
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
This PR removes "skipped" condition for preallocation. Importer/uploader
will preallocate to the available size. Filesystem overhead needs to be
taken into account.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Simplify shouldReconcile function arguments.
By having the function itself grab things it needs and are easily
obtained.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Adapt unit tests to simpler shouldReconcilePVC
Don't set any feature gates for WFFC being disabled.
When the second argument is true, pass the immediate binding annotation
to the PVC itself.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* validate source and target pvcs has the same content type - for all clones
* if source and target volume modes are different, validate content type is kubevirt, and fallback to host-assisted cloning
* unit & func tests
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add a DV/PVC annotation "storage.bind.immediate.requested"
When the annotation is applied the CDI will force bind the PVC (by scheduling worker pods), ignoring the logic to handle WaitForFirstConsumer binding mode.
This is useful when uploading "template" images to the cluster on local storage with WaitForFirstConsumer binding. In this case the image has to be available somewhere on the cluster, the actual placement of image on specific node is not important, so the CDI worker node can be used as a first consumer.
For storage with immediate binging it is effectively a NOOP.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Test import/upload/clone with annotation "storage.bind.immediate.requested"
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* [WIP] doc: User-facing doc for preallocation support
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* apis: CDI accepts `preallocation` option.
With this commit CDI accepts (but does handle) `preallocation` settings
for DataVolumes and in CDIConfig.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Implementing preallocation
This commit implements preallocation support for import and upload.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Functional tests for preallocation support
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Remove "preallocation for StorageClasses" config
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Removed unused function
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Updated dependencies
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Uss PVC annotation to pass preallocation parameters
DataVolume controller now uses a PVC annotation to pass preallocation
configuration to import and update controllers.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Implementing preallocation
This commit implements preallocation support for import and upload.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Make sure all import/upload paths do preallocation
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* [WIP] doc: User-facing doc for preallocation support
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* apis: CDI accepts `preallocation` option.
With this commit CDI accepts (but does handle) `preallocation` settings
for DataVolumes and in CDIConfig.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Implementing preallocation
This commit implements preallocation support for import and upload.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Functional tests for preallocation support
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Remove "preallocation for StorageClasses" config
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Removed unused function
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* test: Fix rook-ceph test failures
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* Updated dependencies
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* core: Uss PVC annotation to pass preallocation parameters
DataVolume controller now uses a PVC annotation to pass preallocation
configuration to import and update controllers.
Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
* When validating disk space, reserve space for filesystem overhead
The amount of available space in a filesystem is not exactly
the advertise amount. Things like indirect blocks or metadata
may use up some of this space. Reserving it to avoid reaching
full capacity by default.
This value is configurable from the CDIConfig object spec,
both globally and per-storageclass.
The default value is 0.055, or "5.5% of the space is
reserved". This value was chosen because some filesystems
reserve 5% of the space as overhead for the root user and
this space doubles as reservation for the worst case
behaviour for unclear space usage. I've chosen a value
that is slightly higher.
This validation is only necessary because we use sparse
images instead of fallocated ones, which was done to have
reasonable alerts regarding space usage from various
storage providers.
---
Update CDIConfig filesystemOverhead status, validate, and
pass the final value to importer/upload pods.
Only the status values controlled by the config controller
are used, and it's filled out for all available storage
classes in the cluster.
Use this value in Validate calls to ensure that some of the
space is reserved for the filesystem overhead to guard from
accidents.
Caveats:
Doesn't use Default: to define the default of 0.055, instead
it is hard-coded in reconcile. It seems like we can't use a
default value.
Validates the per-storageClass values in reconcile, and
doesn't reject bad values.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Use util GetStorageClassByName
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Test filesystem overhead validation against async upload endpoint
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* wait for NFS PVs to be deleted before continuing
Intended to help with flakes, but didn't make a difference.
Probably still worth doing.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Avoid using the uncached client unnecessarily
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add error handling for the case where even a default SC is not found
Note that this change isn't expected to make a difference, as we
check if the targetStorageClass is nil later on and have the same
behaviour, but this is probably more correct API usage.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add testing for the validation of filesystem overhead values
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Fix logical error in waiting for NFS PVs.
Wait for all of them, not just the last one.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Generate CDI CRD using controller-tools.
This is only done for CDI CRD as it requires the existence of source
code. Other CRDs we create are created by a more bare bones pod.
CDIUninstallStrategy was missing a comment describing it, so add
one. This was spotted manually so there might be more missing.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Allow users to specify which nodes CDI pods will live on.
nodeSelector, affinity and tolerations are possible values.
This is done in the CDI CR (rather than CDIConfig) as we are
interested in having this field be populated by external operators.
Unit tests now require the existence of a CDI CR, so create it.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Add a unit test covering some node placement functions
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Specify that all our pods are linux-only.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Avoid duplicate test, accidental left over.
Pointed out by awels, thanks.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Rename to cdiOperatorDeployment for clarity.
Suggested by awels
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Specify we only run on linux using the CDI CR, no need to embed this
into the code.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Don't dereference workloadPlacement for no reason
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Split off operator test to have its own AfterEach, BeforeEach.
Use even more descriptive function names.
Do all the CDI delete/restore logic in AfterEach, to ensure that
it happens and restores the deployment with the original CR even
if the test fails.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Remove XXX. This is the proper way.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Adapt to latest changes in controller_test.go (renaming import)
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Simplify, not storing intermediate value.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Don't dereference nodeplacement in callers to CreateDeployment
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Remove redundant save & restore. Unit tests do this for us.
Pointed out by awels, thanks.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Split out "find toplevel" to a utility function
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Wait for the CDI CR update to apply before continuing.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Simplify, not storing intermediate value.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* Make it clear that the chosen node placement will not be schedulable.
Signed-off-by: Maya Rashish <mrashish@redhat.com>
* update k8s deps to 1.18.6 and controller runtime to 0.6.2
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove building code generators from docker image. This way the k8s ligray version only has to be updated in go.mod
Do more stuff in the bazel container. Faster and better interop
Fix unit tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* make format
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* remove unnecessary rsync
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* redo code generator dep management
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* builder uses go modules
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Set the WaitForFirstConsumer phase on DataVolume when storage uses the WaitForFirstConsumer binding mode and is not bound yet.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Skip PVC if not bound in import|clone|upload controllers.
This is done so the VM pod(not the cdi pod) will be the first consumer, and the PVC can be scheduled on the same location as the pod.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
fixup! Skip PVC if not bound in import|clone|upload controllers.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update importer tests to force bind the PCV by scheduling a pod for pvc, when storage class is wffc.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update datavolume tests to force bind the PCV by scheduling a pod for pvc, when storage class is wffc.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update upload controller and upload tests to correctly handle force binding the PCV by scheduling a pod for pvc, when storage class is wffc.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update clone tests to force bind the PCV by scheduling a pod for pvc when the storage class is wffc.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update cloner multi-node tests to force bind the PCV by scheduling a pod for pvc when storage class is wffc.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Correct after automerge
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Improve/simplify tests
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Fix error in import test.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update transport_test,operator_test.go
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update rbac_test.go and leaderelection_test.go
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Improve Datavolume and PVC Checks for WFFC.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handle wffc only if feature gate is open - import-controller
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* TEST for Handle wffc only if feature gate is open - import-controller - TEST
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handle wffc only if feature gate is open - upload-controller with test
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* rename and simplify checks
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* cleanup after rebase
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* update tests after rebase
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* update tests after rebase
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* more cleanups
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Document new WFFC behavior
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Document new HonorWaitForFirstConsumer option
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* update docs according to comments
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* extract common function, cleanup - code review fixes
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* add comment for another pr - 1210, so it can have easier merge/rebase
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* typo
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Simplify getStoragebindingMode - code review comments
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Add FeatureGates interface - code review fix
Additionally pass the features gates instead of the particular feature gate value,
and let shouldReconcilePVC decide what to do with the feature gate. That way shouldReconcilePVC
contains all the logic, and the caller does not need to do additional calls to provide parameters.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Update matcher
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* don't create snapshot or clone pods if pvcs in use
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* cleanup pods during functional tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* kill mmore pods blocking clone tests
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* fix typos
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
* Handle labels length correctly
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handle service name generation correctly
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Remove not needed labels
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Store import pod name in annotation
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Enable long DV name
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handle name with dot when creating service/label name
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Test long names on import, upload and clone
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Store upload pod name in annotation
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Store importer scratch pvc name in annotation
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Quick fix for tests (need improvements)
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Cleanup handling scratch name
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Ensure pod/service name conflicts are handled
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Handle client errors when trying to get the import pod
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Style improvements, and other code review fixes.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Store clone source pod name in an annotation
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Correct name initialization and tests
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Do not init name if pod already exists. It is not needed.
The situation of having a pod but not name on annotation can happen after the upgrade, when we have a legacy pvc and pod already existing, but clone operation not finished. But when we already have the pod, then in the code (currently) we do not need the name from annotation.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Cleanup scratch name handling
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Use constant for max dv name in validation
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Simplify clone source pod name initialization
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Add conditions to DataVolume and add condition annotation to PVCs
Signed-off-by: Alexander Wels <awels@redhat.com>
* Added reason codes to conditions
Updated functional tests to use reason codes
Fixed broken functional test.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Added last transition time
Added documentation
Signed-off-by: Alexander Wels <awels@redhat.com>
* Switch to conditions slice from conditions pointer slice.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Add heartbeat time for conditions.
Renamed AnnRunningCondition to AnnLastTerminationMessage
Only update reason/message on heartbeat or state change.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Rename heartBeat to heartbeat
Re-factor datavolume-conditions to properly update heartbeat time.
Signed-off-by: Alexander Wels <awels@redhat.com>
* renamed AnnRunningConditionReason to AnnLastTerminationReason
Signed-off-by: Alexander Wels <awels@redhat.com>
* Revert lastTerminatedMessage/Reason rename
Added separate AnnScratchBoundCondition/Message/Reason for scratch space
Signed-off-by: Alexander Wels <awels@redhat.com>
* Update bound condition to take scratch or source PVC status into consideration.
Update running condition to take source pod status into consideration
Signed-off-by: Alexander Wels <awels@redhat.com>
* Addressed comments in review.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Properly get scratch space pvc phase.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Clean up client-go left overs after converting to runtime library.
Clean unused functions from controller utils.go
Turn util_test.go into gingko test.
Moved functions from util to proper controller if only used by that controller.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Don't export reconciler variables.
Signed-off-by: Alexander Wels <awels@redhat.com>
* Add a field to DataVolume to track the number of retries/pod restarts
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Add a field to DataVolume to track the number of retries/pod restarts
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Make RESTARTS non-empty on DataVolume (shows as as 0 'zero')
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Test reporting restarts on DataVolume when importing.
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Fix tests
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Code review fixes
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
* Restart Count status test for upload and clone controller
Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
