* Get rid of unnecessary DIC reconcile updates
Also fixed status.lastExecutionTimestamp to be the last polling time
as intended in the design, and not the last reconcile update time.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Pass DIC last execution time by annotation
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
Saw a flaking test where events were saying that importer can't schedule
because cron PVCs are being deleted:
```bash
"involvedObject": {
"kind": "Pod",
"namespace": "cdi-e2e-tests-importer-x9wfd",
"name": "importer-datasource-test-90e064fca2f4",
"uid": "be238296-7adc-41dd-b465-fac45a6a13ab",
"apiVersion": "v1",
"resourceVersion": "1609529"
},
"reason": "FailedScheduling",
"message": "0/6 nodes are available: 6 persistentvolumeclaim \"datasource-test-90e064fca2f4\" is being deleted.",
```
But it looks like we are never going to clean up accidentally,
let's test it and keep looking for the failing test.
Pretty cool that we'll have a way to intercept the calls in unit test with controller-runtime.
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
* Delete erroneous DV on DIC desired digest update
When a DataImportCron import DV is condition Running=False with
Reason=Error it indicates this DIC might get stuck with this DV forever,
so no new import DVs will be created even if the source sha256 is
updated. With this change, when digest is updated, before creating the
new DV, we simply delete the erroneous DV if necessary.
Also includes some DataImportCron tests improvements and cleanup.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix flaky tests
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* More CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix CRDs deletion in operator deletion
Also check DataImportCron CRD has no DeletionTimestamp before adding a
finalizer
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove tag from DataImportCron registry source URL
There are cases when DataImportCron registry source URL contains tag
referring a specific version. We use this tagged URL when polling for
source updates, but on import we would like to remove the tag as we use
a specific sha256. E.g. for docker://quay.io/containerdisks/centos:8.4
the :8.4 tag will be removed.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Use Docker reference parsing
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Use existing DV on DataImportCron source digest match
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add short names for DataImportCron and DataSource
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron status conditions
The `DataImportCron` controller updates the status conditions in a
controlled `DataImportCron` and its managed `DataSource`.
DataImportCron:
- UpToDate - indicates if the the most recent import is successful and
`DataSource` is up-to-date. Updated to False whenever the source
digest (latest sha256) is updated.
- Progressing - indicates whether the cron is currently in the process
of importing. Updated to True if there is a current import and its
`DataVolume` is `ImportInProgress`, otherwise False.
DataSource:
- Ready - indicates that the corresponding pvc exists and is populated.
Update according to `DataImportCron.Status.LastImportedPVC`
`DataVolume`'s `DataVolumeReady` condition, if the `DataVolume`
exists. Otherwise False. Unlike `DataImportCron` `UpToDate`
condition, this one does not care about newer source digest.
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron RetentionPolicy and remove OwnerReferences
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* More CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add tests for retention policies and datasource/datavolume recreation if deleted
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add status condition tests
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* SetRecommendedLabels for all created CRs
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron controller
-The new controller polls for updates to a registry source container
image, based on a given schedule. When updates to a container image are
detected, the controller imports the content into a new uniquely named
PVC in a golden image namespace.
-For each DataImportCron, the controller manages a corresponding
DataSource to always point to the latest most up-to-date golden
image PVC.
-DataImportCron takes ownership of an existing DataSource (with
controller: false), allowing an admin to opt-in to using auto
delivery/updates later on.
-The controller has PVC garbage collector removing old PVCs.
ToDo:
-status conditions updates
-verify full image streams support
-utests and func tests
-fixmes and commented out code
-doc
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix CR comments and fixmes
- isolate imagestream and registry specific code
- fix namespace of CronJob, and its job and pod to CDI namespace
- manage CronJob-DataImportCron ownership relationship with a finalizer,
handle DataImportCron deletion (CronJob etc.)
- remove CronJob and job pod for ImageStreams, use RequeueAfter and
cronexpr instead
- add k8s app cdi-source-update-poller executed by CronJob to poll source
image digest via skopeo inspect for url registry source, and annotate
the DataImportCron when the image was updated and pending for import based
on the cron schedule
- add cdi-source-update-poller and skopeo binary to the cdi-importer container
- complete dataimportcron-validate and its tests
- reconcile - use context.Context instead of context.TODO
- remove uncached client
- doc
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix ImageStreams watch
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron DV template instead of source
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Fix CR comments
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Split updateSucceeded func
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Improve cdi-source-update-poller cmd logs
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove ImageStream reconcile
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove ImageStream watch
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove unnecessary AnnSourceUpdatePending
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* More CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Idempotentify initCron
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Recreate DV in case is't not found
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron spec.importsToKeep and status.currentImports
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron controller functional test
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add insecure TLS support
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Remove finalizers in cluster clean script
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Bound each import to its sha256 digest instead of latest
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Add DataImportCron controller utests
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Tests CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
* Minor tests CR fixes
Signed-off-by: Arnon Gilboa <agilboa@redhat.com>
