Commit Graph

39 Commits

Author SHA1 Message Date
Arnon Gilboa
ee431e8b14
[release-v1.57] Suppress alerts to reduce noise of dependent ones (#3185)
* [release-v1.57] Suppress alerts to reduce noise of dependent ones

Manual backport of #3129

This is a follow-up to #2998 introducing the following changes to alert
rules:
- CDIDefaultStorageClassDegraded - do not fire when no default SC
  (either k8s or virt).
- CDIDataImportCronOutdated - do not fire when no default SC (either
  k8s or virt), as DIC import DVs use default SC.
- CDINoDefaultStorageClass - fire not only when there is a pending DV
  (and PVC) but also when DV has an empty status (waiting for default
  SC etc.)

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* [release-v1.57] Get no-provisioner storage capabilities from PVs

Manual backport of #3128

It's time to get the storage capabilities for all no-provisioner storage
classes based on the existing PVs, and not only for those labeled with
`local.storage.openshift.io/owner-name`, as done so far.
It will also silence the unnecessary CDIStorageProfilesIncomplete alert
for local SC in the CI.

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

---------

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2024-04-12 00:59:57 +02:00
Alex Kalenyuk
93304e739c
[release-v1.57] Fix inferring for temp host assisted source PVC in snapshot clones (#3182)
* Clone from snapshot: fix volume/access mode inferring for temp host assisted source PVC

Sometimes with snapshot cloning we have to fall back to host assisted.
To do this, we create a temporary restore from the snapshot and initiate a host assisted clone
from that -> target PVC.
The issue this commit fixes is that we set the wrong access/volume modes on this temporary restore PVC
(we set it to the target's).

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Annotate dataimportcron-created snapshot with their source volume mode

This would then get capitalized on by host assisted fallbacks from snapshot cloning

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

---------

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
2024-04-11 14:58:01 +02:00
Arnon Gilboa
9f38eeef1c
[release-v1.57] Watch DIC-orphan cronjobs and cleanup them (#3121)
Manual backport of #3106

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2024-03-06 14:52:42 +01:00
Arnon Gilboa
3c53106662
[release-v1.57] Fix DIC DV creation when last import is not found (#3082)
Manual backport of #3072

The `imports` local variable was not updated correctly, and later
referenced, so another reconcile was required for the DV to be created.

Fixes CNV-37498

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2024-02-05 17:08:58 +01:00
Arnon Gilboa
5616e9187a
[release-v1.57] Delete old version DVs with DIC GC (#2911)
Manual backport of #2749

Signed-off-by: Ido Aharon <iaharon@redhat.com>
Co-authored-by: Ido Aharon <iaharon@redhat.com>
2023-10-04 14:49:25 +02:00
Lee Yarwood
5958ea648d
dataVolume: Add default instance type labels from source (#2787) (#2883)
* dataVolume: Add default instance type labels from source

f229aeb started to pass default instance type labels from
DataImportCrons to DataVolumes and DataVolumes to any associated
destination DataSources or PVCs. As documented in issue #2782 this does
not however pass these labels from the initial source of a DataVolume to
either the DataVolume or the destination DataSources or PVCs

This change corrects this by updating DataVolumes when reconciled,
adding any labels found on PVC or DataSource sources. These labels will
then be passed on to the destination PVC or DataSources by the existing
functionality highlighted above.

Note that if any default instance type labels already exist on the
DataVolume then the process is skipped as it is assumed these are
provided either directly by the user or via a DataImportCron.

Signed-off-by: Lee Yarwood <lyarwood@redhat.com>

* refactor: Use DefaultInstanceTypeLabels more often

Signed-off-by: Lee Yarwood <lyarwood@redhat.com>

---------

Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
(cherry picked from commit 33358569bb)
2023-09-04 21:01:27 +02:00
kubevirt-bot
9f9486076b
[release-v1.57] Avoid creating snapshot of old storage class DataImportCron PVCs (#2885)
* Play nice with storage class changes; don't attempt to create snapshot from old sc

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Make the DataImportCron format more visible via printed column on get

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

---------

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
Co-authored-by: Alex Kalenyuk <akalenyu@redhat.com>
2023-09-01 14:15:29 +02:00
kubevirt-bot
f506f695fe
Fix DIC returned reconcile.Result for ImageStream (#2826)
regression introduced in #2700

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
Co-authored-by: Arnon Gilboa <agilboa@redhat.com>
2023-07-30 10:20:05 +02:00
akalenyu
50efcee3c2
[release-v1.57] Backport main commits to 1.57 release branch v2 (#2785)
* dataimportcron: Pass dynamic credential support label (#2760)

* dataimportcron: code change: Use better matchers in tests

Signed-off-by: Andrej Krejcir <akrejcir@redhat.com>

* dataimportcron: Pass dynamic credential support label

The label is passed from DataImportCron to DataVolume
and DataSource.

Signed-off-by: Andrej Krejcir <akrejcir@redhat.com>

---------

Signed-off-by: Andrej Krejcir <akrejcir@redhat.com>

* Add DataImportCron snapshot sources docs (#2747)

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* add akalenyu as approver, some others as reviewers (#2766)

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* Run `make rpm-deps` (#2741)

* Run make rpm-deps

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Avoid overlayfs error message by using vfs driver

Signed-off-by: Maya Rashish <mrashish@redhat.com>

---------

Signed-off-by: Maya Rashish <mrashish@redhat.com>

* Fix Destructive test lane failure - missing pod following recreate of CDI (#2744)

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* [WIP] Handle nil ptr in dataimportcron controller (#2769)

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Revert some gomega error checking that produce confusing output (#2772)

One of these tests flakes, but the error is hard to debug because
gomega will yell about
`Unexpected non-nil/non-zero argument at index 0`
Instead of showing the error.

Apparently this is intended:
https://github.com/onsi/gomega/pull/480/files#diff-e696deff1a5be83ad03053b772926cb325cede3b33222fa76c2bb1fcf2efd809R186-R190

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Run bazelisk run //robots/cmd/uploader:uploader -- -workspace /home/prow/go/src/github.com/kubevirt/project-infra/../containerized-data-importer/WORKSPACE -dry-run=false (#2770)

Signed-off-by: kubevirt-bot <kubevirtbot@redhat.com>

* [CI] Add metrics name linter (#2774)

Signed-off-by: Aviv Litman <alitman@redhat.com>

---------

Signed-off-by: Andrej Krejcir <akrejcir@redhat.com>
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
Signed-off-by: Maya Rashish <mrashish@redhat.com>
Signed-off-by: kubevirt-bot <kubevirtbot@redhat.com>
Signed-off-by: Aviv Litman <alitman@redhat.com>
Co-authored-by: Andrej Krejcir <akrejcir@gmail.com>
Co-authored-by: Michael Henriksen <mhenriks@redhat.com>
Co-authored-by: Maya Rashish <mrashish@redhat.com>
Co-authored-by: kubevirt-bot <kubevirtbot@redhat.com>
Co-authored-by: Aviv Litman <64130977+avlitman@users.noreply.github.com>
2023-07-06 16:05:38 +02:00
Alexander Wels
4b2c171ecc
Backport main commits to 1.57 release branch (#2764)
* Enable empty schedule in DataImportCron (#2711)

Allow disabling DataImportCron schedule and support external trigger

Signed-off-by: Ido Aharon <iaharon@redhat.com>

* expand upon #2721 (#2731)

Need to replace requeue bool with requeue duration

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* Add clone from snapshot functionalities to clone-populator (#2724)

* Add clone from snapshot functionalities to the clone populator

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Update clone populator unit tests to cover clone from snapshot capabilities

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Fix storage class assignation in temp-source claim for host-assisted clone from snapshot

This commit also includes other minor and styling-related fixes

Signed-off-by: Alvaro Romero <alromero@redhat.com>

---------

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Prepare CDI testing for the upcoming non-CSI lane (#2730)

* Update functional tests to skip incompatible default storage classes

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Enable the use of non-csi HPP in testing lanes

This commit modifies several scripts to allow the usage of classic HPP as the default SC in tests.

This allows us to test our non-populator flow with a non-csi provisioner.

Signed-off-by: Alvaro Romero <alromero@redhat.com>

---------

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Allow snapshots as format for DataImportCron created sources (#2700)

* StorageProfile API for declaring format of resulting cron disk images

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Integrate recommended format in dataimportcron controller

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Take snapclass existence into consideration when populating cloneStrategy and sourceFormat

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

---------

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* Remove leader election test (#2745)

Now that we are using the standard k8s leases from
the controller runtime library, there is no need to
test our implementation as it is no longer in use.
This will save some testing time and random failures.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Integration of Data volume using CDI populators (#2722)

* move cleanup out of dv deletion

It seemed off to call cleanup in the prepare function
just because we don't call cleanup unless the dv is deleting.
Instead we check in the clenup function itself if it should be
done: in this 2 specific cases in case of deletion and in case
the dv succeeded.
The cleanup will be used in future commit also for population cleanup
which we also want to happen not only on deletion.

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Use populator if csi storage class exists

Add new datavolume phase PendingPopulation to
indicate wffc when using populators, this new
phase will be used in kubevirt in order to know
that there is no need for dummy pod to pass wffc phase
and that the population will occur once creating the vm.

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Update population targetPVC with pvc prime annotations

The annotations will be used to update dv that uses the
populators.

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Adjust UT with new behavior

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* updates after review

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Fix import populator report progress

The import pod should be taken from pvcprime

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Prevent requeue upload dv when failing to find progress report pod

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Remove size inflation in populators

The populators are handling existing PVCs.
The PVC already has a defined requested size,
inflating the PVC' with fsoverhead will only be
on the PVC' spec and will not reflect on the target
PVC, this seems undesired.
Instead if the populators is using by PVC that the
datavolume controller created the inflation will happen
there if needed.

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Adjust functional tests to handle dvs using populators

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Fix clone test

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* add shouldUpdateProgress variable to know if need to update progress

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Change update of annotation from denied list to allowed list

Instead if checking if the annotation on pvcPrime is not desired
go over desired list and if the annotation exists add it.

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* fix removing annotations from pv when rebinding

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* More fixes and UT

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* a bit more updates and UTs

Signed-off-by: Shelly Kagan <skagan@redhat.com>

---------

Signed-off-by: Shelly Kagan <skagan@redhat.com>

* Run bazelisk run //robots/cmd/uploader:uploader -- -workspace /home/prow/go/src/github.com/kubevirt/project-infra/../containerized-data-importer/WORKSPACE -dry-run=false (#2751)

Signed-off-by: kubevirt-bot <kubevirtbot@redhat.com>

* Allow dynamic linked build for non bazel build (#2753)

The current script always passes the static ldflag to the
compiler which will result in a static binary. We would like
to be able to build dynamic libraries instead.

cdi-containerimage-server has to be static because we
are copying it into the context of a container disk container
which is most likely based on a scratch container and has no
libraries for us to use.

Signed-off-by: Alexander Wels <awels@redhat.com>

* Disable DV GC by default (#2754)

* Disable DV GC by default

DataVolume garbage collection is a nice feature, but unfortunately it
violates fundamental principle of Kubernetes. CR should not be
auto-deleted when it completes its role (Job with TTLSecondsAfter-
Finished is an exception), and once CR was created we can assume it is
there until explicitly deleted. In addition, CR should keep idempotency,
so the same CR manifest can be applied multiple times, as long as it is
a valid update (e.g. DataVolume validation webhook does not allow
updating the spec).

When GC is enabled, some systems (e.g GitOps / ArgoCD) may require a
workaround (DV annotation deleteAfterCompletion = "false") to prevent
GC and function correctly.

On the next kubevirt-bot Bump kubevirtci PR (with bump-cdi), it will
fail on all kubevirtci lanes with tests referring DVs, as the tests
IsDataVolumeGC() looks at CDIConfig Spec.DataVolumeTTLSeconds and
assumes default is enabled. This should be fixed there.

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Fix test waiting for PVC deletion with UID

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Fix clone test assuming DV was GCed

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Fix DIC controller DV/PVC deletion when snapshot is ready

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

---------

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

---------

Signed-off-by: Ido Aharon <iaharon@redhat.com>
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
Signed-off-by: Alvaro Romero <alromero@redhat.com>
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
Signed-off-by: Alexander Wels <awels@redhat.com>
Signed-off-by: Shelly Kagan <skagan@redhat.com>
Signed-off-by: kubevirt-bot <kubevirtbot@redhat.com>
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
Co-authored-by: Ido Aharon <iaharon@redhat.com>
Co-authored-by: Michael Henriksen <mhenriks@redhat.com>
Co-authored-by: alromeros <alromero@redhat.com>
Co-authored-by: akalenyu <akalenyu@redhat.com>
Co-authored-by: Shelly Kagan <skagan@redhat.com>
Co-authored-by: kubevirt-bot <kubevirtbot@redhat.com>
Co-authored-by: Arnon Gilboa <agilboa@redhat.com>
2023-06-22 00:59:17 +02:00
Arnon Gilboa
813632f604
Propagate CDI workload NodePlacement to DIC CronJob and initial Job (#2712)
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2023-05-18 21:52:42 +02:00
Nahshon Unna Tsameret
4d14028d5b
enable ginkgolinter and fix findings (#2703)
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
2023-05-04 13:07:36 +02:00
Michael Henriksen
8cb11f53d9
update k8s libs to 1.26. (#2687)
* update k8s libs to 1.26.

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* remove some checks in log messages, they're redundant, and the format has changed

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

* use 1.26 lib function `CheckVolumeModeMismatches` and `CheckAccessModes`

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>

---------

Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2023-04-18 19:30:40 +01:00
Nahshon Unna Tsameret
48166bdc8d
Enable additional linters to golangci-lint configuration (#2691)
* enable the typecheck linter in golangci-lint

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

* enable the staticcheck linter and fix findings

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

* golangci-lint: enable ineffassign and fix finding

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

---------

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
2023-04-17 13:30:28 +01:00
Arnon Gilboa
fc7f855ec9
Annotate DIC-created DV for immediate binding (#2650)
If the storage class binding mode is WaitForFirstConsumer, and the
annotation was not explicitly added to the DIC DV template, the created DV
will get stuck in WFFC phase.

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2023-03-22 03:25:02 +00:00
akalenyu
f58f83feeb
Add sourceRef support for VolumeSnapshot sources (#2582)
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
2023-03-03 16:53:55 +00:00
Arnon Gilboa
fa691809bc
Update succeeded DIC PVC labels also when no DV GC (#2539)
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2023-01-12 00:21:10 +00:00
Arnon Gilboa
3bf0223927
Set ttl for completed cron jobs (#2538)
Keeping the last completed or failed job and pod for a while is needed
for both functional tests and debugging. Since the ttl was not set, the
jobs were not automatically deleted.

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2023-01-12 00:21:03 +00:00
Lee Yarwood
f229aeb5ff
dataimportcron: Pass KubeVirt instance type labels to DataVolume and DataSource (#2534)
* dataimportcron: Pass KubeVirt instance type labels to DataVolume and DataSource

Following on from 4fbcb2d509 a requirement
has arisen to expose the default instance type metadata previously
exposed as annotations also as labels to allow callers such as the UI to
have simple server side filtering of these resources.

The unreleased feature implementation in KubeVirt has now
switched to labels and so CDI should now do the same and pass through
the appropriate labels to the underlying resources.

Signed-off-by: Lee Yarwood <lyarwood@redhat.com>

* instancetype: Pass instance type labels from DataVolume to PVC

Unlike annotations not all labels are copied from a given DataVolume to
a PVC during an import. This change corrects this for instance type
labels ensuring they are passed down to the underlying PVC.

The associated constants are also moved into pkg/controller/common/util
to allow access from the DataImportCron and DataVolume controllers.

Signed-off-by: Lee Yarwood <lyarwood@redhat.com>

Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
2023-01-10 17:54:53 +00:00
Lee Yarwood
4fbcb2d509
dataimportcron: Pass KubeVirt instance type annotations to DV and DS (#2490)
A recent design proposal within the KubeVirt community introduced the
idea of inferring the details of default instance type and preferences
from a given volume associated with a VirtualMachine [1]. The idea being
to further reduce the number of choices a user has to make to get a
bootable VirtualMachine to a single choice of a PVC.

This change aims to support this effort by allowing operators to
annotate the underlying DataVolumes, DataSources and PVCs at import time
through CDI by first annotating the initial DataImportCrons.

This is useful to users of CDI such as the KubeVirt SSP operator that
currently defines a number of DataImportCrons to pull in various boot
sources required by the KubeVirt common-templates project.

Both the DataVolume and DataSource associated with the DataImportCron
are annotated to allow KubeVirt to potentially avoid a deeper lookup of
the associated PVC when attempting to infer these defaults.

[1] https://github.com/kubevirt/community/blob/main/design-proposals/default-instancetypes-from-volumes.md

Signed-off-by: Lee Yarwood <lyarwood@redhat.com>

Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
2023-01-04 15:00:47 +00:00
Arnon Gilboa
bfe30a8964
Split and refactor DV controller (#2483)
- Split the huge DV controller into smaller op-specific DV controllers -
  import, clone, upload
- Add common watch-adding function so each controller watches only its
  relevant DVs
- Refactor the common Reconcile() to use interface DataVolumeReconciler
  implemented by each controller
- Move all functions, structs, consts to the relevant controller
- Split the utests per controller

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2022-12-22 01:03:15 +00:00
Arnon Gilboa
eef03584fb
Add DataImportCron CronJobs Proxy support (#2455)
* Add DataImportCron CronJobs Proxy support

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add cronjob env utest

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2022-11-03 19:43:23 +00:00
Arnon Gilboa
a222359dc6
Adapt DataImportCron and DataSource to DV GC (#2441)
* Adapt DataImportCron and DataSource to DV GC

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add DIC test for GC

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2022-10-20 20:55:47 +01:00
Arnon Gilboa
ea86148af0
Update DataImportCron CronJob when needed (#2316)
* Update DIC CronJob container image if needed

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2022-06-11 01:38:19 +03:00
Arnon Gilboa
ed4c5bf2e5
Reconcile DIC only if DataSource is not managed by another DIC (#2295)
* Reconcile DIC only if DataSource is not managed by another DIC

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2022-06-02 20:23:25 +03:00
Arnon Gilboa
50467a68be
Get rid of unnecessary DIC reconcile updates (#2294)
* Get rid of unnecessary DIC reconcile updates

Also fixed status.lastExecutionTimestamp to be the last polling time
as intended in the design, and not the last reconcile update time.

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Pass DIC last execution time by annotation

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2022-05-27 15:19:38 +03:00
akalenyu
df56b93920
Add unit test to make sure we don't clean up dataimportcrons when not required (#2202)
Saw a flaking test where events were saying that importer can't schedule
because cron PVCs are being deleted:
```bash
            "involvedObject": {
                "kind": "Pod",
                "namespace": "cdi-e2e-tests-importer-x9wfd",
                "name": "importer-datasource-test-90e064fca2f4",
                "uid": "be238296-7adc-41dd-b465-fac45a6a13ab",
                "apiVersion": "v1",
                "resourceVersion": "1609529"
            },
            "reason": "FailedScheduling",
            "message": "0/6 nodes are available: 6 persistentvolumeclaim \"datasource-test-90e064fca2f4\" is being deleted.",
```
But it looks like we are never going to clean up accidentally,
let's test it and keep looking for the failing test.
Pretty cool that we'll have a way to intercept the calls in unit test with controller-runtime.

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
2022-04-13 21:28:53 +02:00
Arnon Gilboa
d4a850468b
Delete erroneous DataVolume on DataImportCron desired digest update (#2169)
* Delete erroneous DV on DIC desired digest update

When a DataImportCron import DV is condition Running=False with
Reason=Error it indicates this DIC might get stuck with this DV forever,
so no new import DVs will be created even if the source sha256 is
updated. With this change, when digest is updated, before creating the
new DV, we simply delete the erroneous DV if necessary.

Also includes some DataImportCron tests improvements and cleanup.

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Fix flaky tests

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* More CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2022-03-01 23:04:12 +01:00
Arnon Gilboa
0ee4a61987
Get rid of DataImportCron finalizer (#2144)
* Get rid of DataImportCron finalizer

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Remove CRDs deletion in operator deletion

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Cleanups

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2022-02-12 05:56:08 +01:00
akalenyu
eb9fafe4ab
Replace batch/v1beta with v1 (#2131)
As CronJob v1beta is being deprecated in 1.25, we'd like to replace usages of it with v1:
https://kubernetes.io/docs/reference/using-api/deprecation-guide/#cronjob-v125

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
2022-02-01 14:22:04 +01:00
Arnon Gilboa
472c38e657
Fix CRDs deletion in operator deletion (#2129)
* Fix CRDs deletion in operator deletion

Also check DataImportCron CRD has no DeletionTimestamp before adding a
finalizer

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2022-01-28 16:13:09 +01:00
Arnon Gilboa
81fa12ed38
Add DataImportCron ImageStream tag support (#2117)
* Add DataImportCron ImageStream tag support

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Fix storageClass possible nil deref

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Change DataSource indexer key separator

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add DataImportCron ImageStream tag utest

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2022-01-25 13:05:05 +01:00
Arnon Gilboa
97288e3c5c
Cleanup DataImportCron jobs on deletion (#2088)
* Cleanup DataImportCron jobs on deletion

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Func test fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2022-01-12 21:10:05 +01:00
Arnon Gilboa
39803bc33b
Remove tag from DataImportCron registry source URL (#2086)
* Remove tag from DataImportCron registry source URL

There are cases when DataImportCron registry source URL contains tag
referring a specific version. We use this tagged URL when polling for
source updates, but on import we would like to remove the tag as we use
a specific sha256. E.g. for docker://quay.io/containerdisks/centos:8.4
the :8.4 tag will be removed.

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Use Docker reference parsing

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2022-01-11 17:58:50 +01:00
Michael Henriksen
d56e0cca05
23 libs (#2077)
Signed-off-by: Michael Henriksen <mhenriks@redhat.com>
2022-01-07 16:56:25 +01:00
Arnon Gilboa
d77abc3fa9
Add DataSource controller to update the Ready condition (#2085)
even when there is no DataImportCron associated

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2022-01-06 21:06:23 +01:00
Arnon Gilboa
abb0c7f508
Use existing DataVolume on DataImportCron source digest match (#2075)
* Use existing DV on DataImportCron source digest match

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add short names for DataImportCron and DataSource

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2022-01-03 17:27:13 +01:00
Arnon Gilboa
fe018f1dc5
Add DataImportCron status conditions (#2045)
* Add DataImportCron status conditions

The `DataImportCron` controller updates the status conditions in a
controlled `DataImportCron` and its managed `DataSource`.

DataImportCron:
- UpToDate - indicates if the the most recent import is successful and
    `DataSource` is up-to-date. Updated to False whenever the source
     digest (latest sha256) is updated.
- Progressing - indicates whether the cron is currently in the process
    of importing. Updated to True if there is a current import and its
    `DataVolume` is `ImportInProgress`, otherwise False.

DataSource:
- Ready - indicates that the corresponding pvc exists and is populated.
    Update according to `DataImportCron.Status.LastImportedPVC`
    `DataVolume`'s `DataVolumeReady` condition, if the `DataVolume`
    exists. Otherwise False. Unlike `DataImportCron` `UpToDate`
    condition, this one does not care about newer source digest.

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add DataImportCron RetentionPolicy and remove OwnerReferences

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* More CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add tests for retention policies and datasource/datavolume recreation if deleted

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add status condition tests

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* SetRecommendedLabels for all created CRs

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2021-12-16 02:21:01 +01:00
Arnon Gilboa
7087b57cd2
Add DataImportCron controller (#1949)
* Add DataImportCron controller

-The new controller polls for updates to a registry source container
image, based on a given schedule. When updates to a container image are
detected, the controller imports the content into a new uniquely named
PVC in a golden image namespace.
-For each DataImportCron, the controller manages a corresponding
DataSource to always point to the latest most up-to-date golden
image PVC.
-DataImportCron takes ownership of an existing DataSource (with
controller: false), allowing an admin to opt-in to using auto
delivery/updates later on.
-The controller has PVC garbage collector removing old PVCs.

ToDo:
-status conditions updates
-verify full image streams support
-utests and func tests
-fixmes and commented out code
-doc

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Fix CR comments and fixmes

- isolate imagestream and registry specific code
- fix namespace of CronJob, and its job and pod to CDI namespace
- manage CronJob-DataImportCron ownership relationship with a finalizer,
  handle DataImportCron deletion (CronJob etc.)
- remove CronJob and job pod for ImageStreams, use RequeueAfter and
  cronexpr instead
- add k8s app cdi-source-update-poller executed by CronJob to poll source
  image digest via skopeo inspect for url registry source, and annotate
  the DataImportCron when the image was updated and pending for import based
  on the cron schedule
- add cdi-source-update-poller and skopeo binary to the cdi-importer container
- complete dataimportcron-validate and its tests
- reconcile - use context.Context instead of context.TODO
- remove uncached client
- doc

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Fix ImageStreams watch

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add DataImportCron DV template instead of source

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Fix CR comments

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Split updateSucceeded func

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Improve cdi-source-update-poller cmd logs

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Remove ImageStream reconcile

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Remove ImageStream watch

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Remove unnecessary AnnSourceUpdatePending

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* More CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Idempotentify initCron

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Recreate DV in case is't not found

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add DataImportCron spec.importsToKeep and status.currentImports

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add DataImportCron controller functional test

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add insecure TLS support

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Remove finalizers in cluster clean script

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Bound each import to its sha256 digest instead of latest

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Add DataImportCron controller utests

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Tests CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Minor tests CR fixes

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
2021-11-11 20:09:48 +01:00