Commit Graph

64 Commits

Author SHA1 Message Date
Michael Henriksen
073336b16c
Make upload client/server certs configurable (#3252)
* Add client cert config to CDI resource

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* make client certs configurable

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* Create uploadserver.Config

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* uploadserver should read certs from files

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* make sure to not close doneChan when error occurs

generally tighten up handling of "done" "uploading" and "processing"

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* add deadline support to uploadserver

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* Add deadline support to upload controller

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* clone controller should use configured client cert duration

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* make lint check happy

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* Extend existing func test to validate client certs configurable and will be rotated

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* Use deadline/rotation for clone pods as well

Forgot about the case where a source PVC may be in use.  Bay be a big delay from when target pod is created and source.

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

---------

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2024-06-03 22:39:34 +02:00
Edu Gómez Escandell
0e750262a3
Enable autoformatting linters (#3179)
* Enable gofmt linter

From the docs:

> Gofmt checks whether code was gofmt-ed. By default this tool runs with
> -s option to check for code simplification.

https://golangci-lint.run/usage/linters/#gofmt

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

* Run gomft on the project

Ran this command after adding the gofmt linter:

	golangci-lint run ./... --fix

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

* Enable whitespace linter

From the docs:
> Whitespace is a linter that checks for unnecessary newlines at the
> start and end of functions, if, for, etc.

https://golangci-lint.run/usage/linters/#whitespace

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

* Run whitespace on the project

Ran this command after adding the whitespace linter:

	golangci-lint run ./... --fix

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

* Enable GCI linter

Per the docs:

> Gci controls Go package import order and makes it always deterministic.

https://golangci-lint.run/usage/linters/#gci

NOTE: I noticed that many files separate their imports in a particular
way, so I set the linter to enforce this standard.

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

* Run GCI on the project

Ran this command after adding the GCI linter:

	golangci-lint run ./... --fix

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

---------

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
2024-04-24 13:52:22 +02:00
Alex Kalenyuk
31d12e426e
update k8s & related libraries to 1.28 (#3078)
* Bump k8s/OpenShift/ctrl-runtime/lifecycle-sdk & make deps-update

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Operator: adapt for dependency bump

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Controller: adapt watch calls for dependency bump

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Controller: adapt to ctrl-runtime's cache API changes

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Operator: fix unit tests by deleting resources properly in fake client

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Controller: fix unit tests by deleting resources properly in fake client

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Controller: adapt to fake client honoring status subresource

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Fix codegen script & make generate

There are some issues in the new script, so we
will still use the deprecated one.
More context in f4d1a5431b

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Functests: Adapt to NamespacedName now implementing MarshalLog

ns/name -> {"name":"name","namespace":"ns"}

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Functests & API server: address deprecation of wait.PollImmediate

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

---------

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
2024-01-23 17:52:05 +01:00
Ben Coxford
3a57e0ebf2
Add support for linkerd injection annotations on datavolumes (#2954)
* add support for linkerd annotations on datavolumes

Signed-off-by: bcoxford <ben.coxford@ncr.com>

* fix default value linkerd injection annotation

Signed-off-by: bcoxford <ben.coxford@ncr.com>

* set default value for linkerd injection to disabled

Signed-off-by: bcoxford <ben.coxford@ncr.com>

* resolve linting on mesh annotation hinting

Signed-off-by: bcoxford <ben.coxford@ncr.com>

* fix linting on AnnPodSidecarInjectionLinkerdDefault hints

Signed-off-by: bcoxford <ben.coxford@ncr.com>

* add note to sidecar unit tests

Signed-off-by: bcoxford <ben.coxford@ncr.com>

---------

Signed-off-by: bcoxford <ben.coxford@ncr.com>
2023-11-16 19:23:21 +01:00
akalenyu
7c1d7db62c
Mount block PVC rw in clone source pods (#2944)
Some CSI drivers [0] don't allow mounting block ro,
so they simply cannot perform host assisted clones.
Host assisted clones are important as they are a last resort in some cases
like switching over to a new storage provisioner.

This may or may not have some performance implications (parallel cloning from single source)
but we are choosing compatibility over performance in this case.

[0] 76cb78decd/service/node.go (L420-L424)

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
2023-10-31 23:06:20 +01:00
Maya Rashish
5e4cb68044
Update to ginkgo v2 (#2788)
* Run `make deps-update`

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Update to ginkgo v2

Avoid using table extension to avoid compilation errors
Switch to v2 everywhere
Update qe-tools as well (required)

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Fix/avoid deprecation warnings

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Do not use v1 reporter

For unit tests: stop using custom reporter, unnecessary
For functional tests: borrow code from kubevirt to keep reporting

Avoid deprecated warnings by golangci for using deprecated reporter

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Increase ginkgo timeout to 24h (default in ginkgo v1)

this may seem excessive, but we have a lower timeout in Prow, let's save
ourselves the future trouble of bumping timeouts in two places.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* use the ginkgo built-in junit reporter

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Avoid using deprecated --ginkgo.noColor, use --ginkgo.no-color instead

Signed-off-by: Maya Rashish <mrashish@redhat.com>

---------

Signed-off-by: Maya Rashish <mrashish@redhat.com>
2023-07-26 02:35:55 +02:00
Michael Henriksen
4ce9272c2c
DataVolume Controller uses VolumeCloneSource Populator (#2750)
* remove CSI clone

bye bye

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* no more smart clone

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* PVC clone same namespace

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* cross namespace pvc clone

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* various fixes to get some functional tests to work

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* delete smart clone controller again

somehow reappeared after rebase

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* mostly pvc clone functional test fixes

make sure size detect pod only runs on kubevirt content type

clone populator was skipping last round op applying pvc' annotations

various func test fixes

review comments

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* more various test fixes

host clone phase should (implicitly) wait for clone source pod to exit

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* remove "smart" clone from snapshot

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* DataVolume clone from snapshot uses populator

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* improve clone populator/datavolume coordination on "running" condition

For host clone, not much changes, values still comming from annotations on host clone PVC

For smart/csi clone the DataVolume will be "running" if not in pending or error phase

Will have the same values for terminal "completed" state regardless of clone type

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* unit tests for pvc/snapshot clone controllers

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* remove skipped test added in https://github.com/kubevirt/containerized-data-importer/pull/2759

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* attempt address AfterSuite and generate-verify failures

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* handle snapshot clone with no target size specified

also add more validation to some snapshot clone tests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* remove Patch calls

Using the controller runtime Patch API with controller runtime cached client seems to be a pretty bad fit

At least given the way the CR API is designed where an old object is compared to new.

I like patch in theory though and will revisit

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* Clone populator should plan and execute even if PVC is bound

It was possible to miss "preallocation applied" annotation otherwise

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* add long term token to datavolume

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* Rename ProgressReporter to StatusReporter

Should have been done back when annotations were addded to "progress"

Also, if pvc is bound do not call phase Reconcile functions only Status

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

---------

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2023-06-30 00:14:54 +02:00
Michael Henriksen
f88fab69dc
PVC Clone Populator (#2709)
* touch up zero restoresize snapshot

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* clone populator

only supports PVC source now

snapshot coming soon

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* more unit tests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* unit test for clone populator

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* func tests for clone populator

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* move clone populator cleanup function to planner

other review comments

verifier pod should bount readonly

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* add readonly flag to test executor pods

synchronize get hash calls

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* increase linter timeout

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* better/explicit readonly support for test pods

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* check pv for driver info before looking up storageclass as it may not exist

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* addressed review comments

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* chooseStrategy shoud generate more events

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

---------

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2023-05-24 05:11:52 +02:00
Nahshon Unna Tsameret
4d14028d5b
enable ginkgolinter and fix findings (#2703)
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
2023-05-04 13:07:36 +02:00
Nahshon Unna Tsameret
e6d2286dfb
golangci-lint: Enable errcheck (#2696)
* golangci-lint: enable errcheck and fix findings

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

* golangci=lint: exit if find something

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

---------

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
2023-04-25 20:40:16 +02:00
Nahshon Unna Tsameret
48166bdc8d
Enable additional linters to golangci-lint configuration (#2691)
* enable the typecheck linter in golangci-lint

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

* enable the staticcheck linter and fix findings

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

* golangci-lint: enable ineffassign and fix finding

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

---------

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
2023-04-17 13:30:28 +01:00
Nahshon Unna Tsameret
34a98256a2
Start integrate golangci lint into CI (#2675)
* Start adding the golangci-lint to CI

golangci-lint is a collection of many linters. This PR adds
golangci-lint to the CI. For strat, it enables the govet linter, and fix
its single finding.

The PR adds this linter to the `test-lint` Makefile target.

The new .golangci.yml file is the configuration for the linter.

golangci-lint version was set to the latest one - v1.52.2.
It is defined in hack/build/run-linters.sh

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

* golangci-lint: enable gosimple and fix findings

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

* golangci-lint: enable unused and fix findings

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

---------

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
2023-04-11 13:59:33 +01:00
Michael Henriksen
0f4758df05
mount block pvc readonly (#2536)
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2023-01-12 04:05:03 +00:00
Arnon Gilboa
bfe30a8964
Split and refactor DV controller (#2483)
- Split the huge DV controller into smaller op-specific DV controllers -
  import, clone, upload
- Add common watch-adding function so each controller watches only its
  relevant DVs
- Refactor the common Reconcile() to use interface DataVolumeReconciler
  implemented by each controller
- Move all functions, structs, consts to the relevant controller
- Split the utests per controller

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2022-12-22 01:03:15 +00:00
Michael Henriksen
46c6aa994a
Support restricted PSA for worker pods (#2410)
* remove root worker pods

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* remove selinux requirement for worker pods

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* run tests in restricted namespace and required changes

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* handle empty tar

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* add PSA label when running functional tests in OpenShift

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* cannot use restricted PSA with istio (for now)

refactor scc management

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix clean script

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2022-09-14 21:16:23 +01:00
Bartosz Rybacki
b58e3924d1
Clone fs to block fails on size validation (#2299)
* Test: Clone fs to block fails on size validation

When requesting size `X` with filesystem volume mode and storage api the size
is increased for the fs overhead. When trying to clone to block using
the same size `X` the clone fails because the target is smaller than source.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Improve size validation for clone

Skip size validation for filesystem in webhook and include filesystem
overhead when doing the validation in controller.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Correct size validation for smart clone

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Correct unit test with fs overhead

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Restore CDI Config after each clone test

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Review cleanup

Removing redundant conversions and not useful comments

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2022-06-27 15:02:54 +02:00
Michael Henriksen
ec690d5f26
one source pod certs in secrets (#2064)
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2021-12-22 20:48:11 +01:00
Michael Henriksen
aedaf513ec
Move apis to staging, push to containerized-data-importer-api (#1997)
* move apis to new staging area

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* add script to push to staging

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix lint check and api reference

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* push staging to api repo

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2021-10-28 13:40:24 +02:00
akalenyu
50c93e8b0e
Deploy alerts infra as part of our installation (#1979)
* Deploy alerts infra as part of our installation

Conditionally deploy the infrastructure that is needed to fire alerts for our users
when bad things are happening to CDI.

Testing with `KUBEVIRT_DEPLOY_PROMETHEUS=true`

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Watch and unit test all prometheus related resources

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* add gateway for changing monitoring namespace (rbac purposes)

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* refactor test to check for exact alert name and firing state

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Align pattern of ensuring prometheus resource exists for all

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Remove potential noisy event

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Extract duplicate code to function

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Dont use empty value for prometheus label due to open issue

https://github.com/prometheus-operator/prometheus-operator/issues/4325

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
2021-10-26 21:26:07 +02:00
Michael Henriksen
87a13c2f29
Add long term token to pvcs when host assisted cloning cross namespaces (#1922)
* Add long term token (10 years) to pvcs when host assisted cloning between namespaces

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* clone controller should retry if source in use

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* minor refactor if/else -> switch

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2021-09-17 01:24:00 +02:00
akalenyu
2254cf0c1f
Add relationship labels (#1864)
Users don't want 👽 resources in clusters,
and we should also be able to tell if were part of a broader installation.

Note:
- Operator created resources were handled in https://github.com/kubevirt/controller-lifecycle-operator-sdk/pull/18
as these labels will be common to all resources deployed by the HCO.
- Now that the controller is guaranteed to have the labels, we can set env vars
that reference the label values (fieldRef) to spare calling GET on the CR in the controllers.
(thanks mhenriks).

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
2021-07-28 20:05:24 +02:00
Alexander Wels
5fedb458e5
Set some reasonable requests/limits for workloads (#1842)
* Set some reasonable requests/limits for workloads

Signed-off-by: Alexander Wels <awels@redhat.com>

* Fix unit tests

Signed-off-by: Alexander Wels <awels@redhat.com>
2021-06-28 20:34:01 +02:00
Michael Henriksen
d92c2f459d
update deps and bazel (#1815)
* update deps and bazel

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix apidocs and unit tests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix generate-verify

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2021-06-08 01:31:59 +02:00
Michael Henriksen
74a2c86608
use namespace transfer for smart clone (#1763)
* use namespace transfer for smart clone

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* updates from test failures

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* add expansion func tests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* add dv phases for expansion and transfer

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* rebase and integrate with storage profiles

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2021-04-30 15:18:43 +02:00
Tomasz Barański
bcacb20251
Hold off AnnCloneOf until the target PVC is correctly updated. (#1727)
* [WIP] Hold off AnnCloneOff until source pod is gone

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>

* Update AnnCloneOf only when the source pod finished.

Signed-off-by: Tomasz Baranski <tbaransk@redhat.com>
2021-04-26 16:56:57 +02:00
Arnon Gilboa
033af2183a
Allow passing default annotation value to transfer pods (#1674)
when the annotation is not set explicitly in the DV/PVC

e.g. disable Istio sidecar injection by default

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2021-02-24 13:35:46 +01:00
Arnon Gilboa
f7703a6bd7
Support cloning from Filesystem to Block and vice-versa (#1597)
* validate source and target pvcs has the same content type - for all clones
* if source and target volume modes are different, validate content type is kubevirt, and fallback to host-assisted cloning
* unit & func tests

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2021-02-03 01:06:52 +01:00
Nahshon Unna Tsameret
93627f4fe8
Stop Using Deprecated Packages (#1548)
* sigs.k8s.io/controller-runtime/pkg/runtime/* packages are deprecated, and were moved to new paths.

Trying to upgrade sigs.k8s.io/controller-runtime to version v0.7.0 in HCO created a conflict because in v0.7.0 the deprecated packages were removed and cannot be used.

This PR replaces the deprecated packages with their new paths.

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

* Run `make deps-update`

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

* fix logger init

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

* fix test loggers

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
2020-12-24 07:08:50 +01:00
Michael Henriksen
4db135f57a
Read-only clone source pods (#1524)
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2020-12-11 20:08:44 +01:00
Arnon Gilboa
ff42ea0597
Pass specific PVC annotations to the transfer pods (#1480)
* Pass specific PVC annotations to the transfer pods

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add annotations importer/uploader/cloner functional tests

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2020-11-30 19:20:38 +01:00
Matthew Arnold
0329289618
VDDK: replace qemu-img with libnbd (#1448)
* Add libnbd to build.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add libnbd Go bindings to vendor.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add libnbd to importer container image.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Enable cgo builds by changing bazel platform.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Break dependency chain on pkg/importer.

Without this, pretty much every generated binary (including tests.test)
requires libnbd.so.0.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Replace qemu-img with libnbd in VDDK importer.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add progress tracking back to VDDK datasource.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add fadvise before sequential VDDK transfer.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Improve VDDK debug logging and error handling.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Update to libnbd 1.4.1.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Update build container tag.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Fix uploadserver/uploadproxy tests.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add a mirror for libnbd RPM, like in #1444.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Add libnbd to uploadserver.

This still depends on pkg/importer, so it needs to have libnbd.

Signed-off-by: Matthew Arnold <marnold@redhat.com>

* Use pointer to data sink, and add final sync.

Signed-off-by: Matthew Arnold <marnold@redhat.com>
2020-11-02 20:09:49 +01:00
Michael Henriksen
3f924adaaa
add finalizer to target PVC before creating clone source pod (#1429)
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2020-10-15 21:49:39 +02:00
Michael Henriksen
7dbc07ab0f
Attempt to schedula clone sourc/target pods on same node (#1426)
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2020-10-13 00:45:38 +02:00
Alexander Wels
a2bdf470a1
Add library function to determine if a PVC has been populated fully. (#1400)
* Add library function to determine if a PVC has been populated fully.

The logic is as following:
If PVC has no ownerRef, then we assume something else fully populated it and
will return true
If PVC has an ownerRef and its a DataVolume, then look up the DataVolume
If DV.status.Phase == succeeded, return true, return false otherwise.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Renamed functions to better indicate its purpose.

Signed-off-by: Alexander Wels <awels@redhat.com>
2020-09-26 02:15:30 +02:00
Maya Rashish
e3436e0199
Allow specifying nodeSelector, affinity and tolerations for CDI pods (#1346)
* Generate CDI CRD using controller-tools.

This is only done for CDI CRD as it requires the existence of source
code. Other CRDs we create are created by a more bare bones pod.

CDIUninstallStrategy was missing a comment describing it, so add
one. This was spotted manually so there might be more missing.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Allow users to specify which nodes CDI pods will live on.

nodeSelector, affinity and tolerations are possible values.

This is done in the CDI CR (rather than CDIConfig) as we are
interested in having this field be populated by external operators.

Unit tests now require the existence of a CDI CR, so create it.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Add a unit test covering some node placement functions

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Specify that all our pods are linux-only.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Avoid duplicate test, accidental left over.

Pointed out by awels, thanks.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Rename to cdiOperatorDeployment for clarity.

Suggested by awels

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Specify we only run on linux using the CDI CR, no need to embed this
into the code.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Don't dereference workloadPlacement for no reason

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Split off operator test to have its own AfterEach, BeforeEach.

Use even more descriptive function names.

Do all the CDI delete/restore logic in AfterEach, to ensure that
it happens and restores the deployment with the original CR even
if the test fails.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Remove XXX. This is the proper way.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Adapt to latest changes in controller_test.go (renaming import)

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Simplify, not storing intermediate value.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Don't dereference nodeplacement in callers to CreateDeployment

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Remove redundant save & restore. Unit tests do this for us.

Pointed out by awels, thanks.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Split out "find toplevel" to a utility function

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Wait for the CDI CR update to apply before continuing.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Simplify, not storing intermediate value.

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Make it clear that the chosen node placement will not be schedulable.

Signed-off-by: Maya Rashish <mrashish@redhat.com>
2020-09-03 22:13:18 +02:00
Michael Henriksen
75f4fd6f2f
update k8s deps to 18.6 and controller runtime to 0.6.2 (#1330)
* update k8s deps to 1.18.6 and controller runtime to 0.6.2

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* remove building code generators from docker image.  This way the k8s ligray version only has to be updated in go.mod

Do more stuff in the bazel container.  Faster and better interop

Fix unit tests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* make format

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* remove unnecessary rsync

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* redo code generator dep management

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* builder uses go modules

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2020-08-07 14:09:52 +02:00
Michael Henriksen
9e2c79b1e0
move api groups to v1beta1 (#1232)
* move upload.cdi.kubevirt.io API group to v1beta1

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* move core api to v1beta1

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix os-3.11 cluster sync and add functional tests for alpha api

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* change more occurences of v1alpha1

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* updates after rebase

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2020-07-10 15:47:38 +02:00
Michael Henriksen
bd69c350bb
Wait for PVCs to be unused before initiating clone operations (#1263)
* don't create snapshot or clone pods if pvcs in use

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* cleanup pods during functional tests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* kill mmore pods blocking clone tests

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix typos

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2020-06-30 19:08:50 +02:00
Bartosz Rybacki
ab8b9c025e
Generating label names (#1200)
* Handle labels length correctly

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Handle service name generation correctly

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Remove not needed labels

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Store import pod name in annotation

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Enable long DV name

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Handle name with dot when creating service/label name

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Test long names on import,  upload and clone

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Store upload pod name in annotation

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Store importer scratch pvc name in annotation

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Quick fix for tests (need improvements)

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Cleanup handling scratch name

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Ensure pod/service name conflicts are handled

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Handle client errors when trying to get the import pod

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Style improvements, and other code review fixes.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Store clone source pod name in an annotation

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Correct name initialization and tests

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Do not init name if pod already exists. It is not needed.

The situation of having a pod but not name on annotation can happen after the upgrade, when we have a legacy pvc and pod already existing, but clone operation not finished. But when we already have the pod, then in the code (currently) we do not need the name from annotation.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Cleanup scratch name handling

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Use constant for max dv name in validation

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Simplify clone source pod name initialization

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2020-05-29 19:55:32 +02:00
Alexander Wels
c707e781a9
Add conditions to DataVolume and add condition annotation to PVCs (#1194)
* Add conditions to DataVolume and add condition annotation to PVCs

Signed-off-by: Alexander Wels <awels@redhat.com>

* Added reason codes to conditions
Updated functional tests to use reason codes
Fixed broken functional test.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Added last transition time
Added documentation

Signed-off-by: Alexander Wels <awels@redhat.com>

* Switch to conditions slice from conditions pointer slice.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Add heartbeat time for conditions.
Renamed AnnRunningCondition to AnnLastTerminationMessage
Only update reason/message on heartbeat or state change.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Rename heartBeat to heartbeat
Re-factor datavolume-conditions to properly update heartbeat time.

Signed-off-by: Alexander Wels <awels@redhat.com>

* renamed AnnRunningConditionReason to AnnLastTerminationReason

Signed-off-by: Alexander Wels <awels@redhat.com>

* Revert lastTerminatedMessage/Reason rename
Added separate AnnScratchBoundCondition/Message/Reason for scratch space

Signed-off-by: Alexander Wels <awels@redhat.com>

* Update bound condition to take scratch or source PVC status into consideration.
Update running condition to take source pod status into consideration

Signed-off-by: Alexander Wels <awels@redhat.com>

* Addressed comments in review.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Properly get scratch space pvc phase.

Signed-off-by: Alexander Wels <awels@redhat.com>
2020-05-12 23:39:23 +02:00
Alexander Wels
08ed6f2b39
Clean up client-go left overs after converting to runtime library. (#1163)
* Clean up client-go left overs after converting to runtime library.
Clean unused functions from controller utils.go
Turn util_test.go into gingko test.
Moved functions from util to proper controller if only used by that controller.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Don't export reconciler variables.

Signed-off-by: Alexander Wels <awels@redhat.com>
2020-04-02 22:59:54 +02:00
Bartosz Rybacki
4605cf1dc2
Add a field to DataVolume to track the number of retries/pod restarts (#1155)
* Add a field to DataVolume to track the number of retries/pod restarts

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Add a field to DataVolume to track the number of retries/pod restarts

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Make RESTARTS non-empty on DataVolume (shows as as 0 'zero')

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Test reporting restarts on DataVolume when importing.

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Fix tests

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Code review fixes

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>

* Restart Count status test for upload and clone controller

Signed-off-by: Bartosz Rybacki <brybacki@redhat.com>
2020-03-30 23:17:49 +02:00
Alexander Wels
44b46774d2
Runtime lib smartclone controller (#1134)
* Update datavolume controller to use runtime controller library

Signed-off-by: Alexander Wels <awels@redhat.com>

* Update smart clone controller to use controller runtime library

Signed-off-by: Alexander Wels <awels@redhat.com>

* Mark smart cloned cloned PVCs are k8s.io/CloneOf

Signed-off-by: Alexander Wels <awels@redhat.com>

* Removed unused informer factories.
Fixed naming of controller from import to smart clone.
Ignore not found error on delete.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Only delete snapshot if pvc is bound

Signed-off-by: Alexander Wels <awels@redhat.com>
2020-03-20 17:11:44 +01:00
Alexander Wels
17db2920b1
Update clone controller to use runtime library. (#1075)
Signed-off-by: Alexander Wels <awels@redhat.com>

Co-authored-by: Michael Henriksen <mhenriks@redhat.com>
2020-02-14 23:04:46 +01:00
Michael Henriksen
bd4c4c950b
cert rotation (#1091)
* initial cert rotation controller

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix typo

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2020-02-03 23:36:58 +01:00
Woohyung Han
2adcc80f95 Fix to create temporary pod with default resource quota (#1055)
If we create a temporary pod in the namespace which has
resource quota, `must specify quota` error occurs. Fix to
create temporary pod with default resource quota.

Co-authored-by: hyoung-90 <ssarang520@gmail.com>
Co-authored-by: Woohyung Han <woohhan@gmail.com>
Signed-off-by: anencore94 <anencore94@kaist.ac.kr>

Co-authored-by: anencore94 <37469330+anencore94@users.noreply.github.com>
Co-authored-by: hyoung-90 <58063867+hyoung-90@users.noreply.github.com>
2020-01-09 17:43:14 +01:00
Michael Henriksen
66a1c8c996 Unique client cert per clone source pod (#987)
* unique cert per clone source pod

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* upload controller shouldn't cache certs

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* fix test variable names

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2019-10-09 15:10:55 -04:00
Michael Henriksen
834b85ecbf Network clone (#897)
* network cloning

* fix clone progress
2019-08-01 16:01:25 -04:00
Michael Henriksen
ec52c85a25 Validating webhook and token authorization for PVC cloning (#869)
* baseline refactoring of webhook package

* datavolume clone validation webhook

* rename datavolumes/clone-init to datavolumes/source

* add RBAC doc

* updates from review

* make clone permission check exportable function

* force dumb cloning in in functional test
2019-07-09 14:02:31 -04:00
Alexander Wels
309a46e72d Verify source and target PVC are correct before proceeding with clone.
Signed-off-by: Alexander Wels <awels@redhat.com>
2019-04-18 10:17:34 -04:00