Commit Graph

205 Commits

Author SHA1 Message Date
Michael Henriksen
ec52c85a25 Validating webhook and token authorization for PVC cloning (#869)
* baseline refactoring of webhook package

* datavolume clone validation webhook

* rename datavolumes/clone-init to datavolumes/source

* add RBAC doc

* updates from review

* make clone permission check exportable function

* force dumb cloning in in functional test
2019-07-09 14:02:31 -04:00
Michael Henriksen
019c843586 make clone pods use selinux type spc_t instead of privileged (#875)
* make clone pods use selinux type spc_t instead of privileged

* fix block mode related tests
2019-07-08 13:58:42 -04:00
Alexander Wels
b538ab3190 Set running user to the one defined in the docker file.
Signed-off-by: Alexander Wels <awels@redhat.com>
2019-06-12 16:55:37 -04:00
Michael Henriksen
d22e5c0c6f run upload/import pods as nonroot 2019-06-05 17:28:15 -04:00
Michael Henriksen
4c7566f1b7 RunAsUser causing issue on OpenShift so let's get rid of it 2019-06-05 16:16:23 -04:00
Michael Henriksen
bc240a91d9 dont't run as root 2019-06-05 16:16:22 -04:00
Irit goihman
be67c1c772 update IsOpenShift function with ocp4 changes
Signed-off-by: Irit goihman <igoihman@redhat.com>
2019-06-03 22:39:08 +03:00
Michael Henriksen
5740cee5fd fix host:port not allowed in insecure registry configmap 2019-05-29 17:05:55 -04:00
Daniel Erez
5b3109a324 smart-clone
Changes and flow:

* datavolume-controller:
- When clone pvc is requested and smart-clone is applicable -> Create a new VolumeSnapshot.

* Introduced smart-clone-controller:
- Listens to VolumeSnapshot changes and create a new PVC based on the snapshot.
- Upon PVC successful creation, deletes the snapshot for cleaning up.

Change-Id: I369fc92e72edb8a2cf584c8cb916795415f9d6e0
Signed-off-by: Daniel Erez <derez@redhat.com>
Co-authored-by: Fred Rolland <frolland@redhat.com>
2019-05-20 11:04:10 +03:00
Artyom Lukianov
40325745d0 Copy triple package from client-go repository
This package droped from client-go 1.13.4, so until
we will have find some good alternatives, we can use local copy.
2019-05-16 09:38:37 +03:00
Alexander Wels
309a46e72d Verify source and target PVC are correct before proceeding with clone.
Signed-off-by: Alexander Wels <awels@redhat.com>
2019-04-18 10:17:34 -04:00
tavni
e9b6b4f501 Adding support to upload disk image to a Raw Block PV
Signed-off-by: tavni <tavni@redhat.com>
2019-04-17 10:27:24 +03:00
tavni
e6665945bc Adding support to clone disk image from one Block PV to another
Signed-off-by: tavni <tavni@redhat.com>
2019-04-15 15:33:52 +03:00
tavni
e0d4ec62b0 Fix for isue #1690415 - CDI appends 'source pod' to a label and may fail on error that label must not be longer than 63 characters
Signed-off-by: tavni <tavni@redhat.com>
2019-04-03 14:11:42 +03:00
tavni
d4c7b0be54 Adding support to import disk image into Block PV
Signed-off-by: tavni <tavni@redhat.com>
2019-03-28 11:43:56 +02:00
Alexander Wels
afe77c3514 Allow missing storage class for scratch space, if none exists, create scratch PVC without SC
Signed-off-by: Alexander Wels <awels@redhat.com>
2019-03-27 11:41:15 -04:00
Alexander Wels
d2ca78bec3 Use CDIConfig for scratch storage class.
Signed-off-by: Alexander Wels <awels@redhat.com>
2019-03-21 15:37:07 -04:00
Alexander Wels
d1b7842a90 Resize on upload.
Signed-off-by: Alexander Wels <awels@redhat.com>
2019-03-08 21:51:05 -05:00
Alexander Wels
1e46004ad2 Scratch space retry framework logic.
Signed-off-by: Alexander Wels <awels@redhat.com>
2019-03-07 14:54:08 -05:00
Y.Horie
09819e8158
Merge branch 'master' into issue-687 2019-03-05 23:05:05 +09:00
Y.Horie
c9104059fb move to klog from glog 2019-03-05 13:33:20 +09:00
Michael Henriksen
49a740d7ba make CDIConfig client cluster scoped and fix occasional crash 2019-03-04 18:54:58 -05:00
Michael Henriksen
ec2736d2c4 ready checks for all upload componets and uploadproxy sync with api server 2019-03-01 10:20:57 -05:00
Irit goihman
c199129e1f add 'cdi.kubevirt.io' label to cdiconfig and remove config on make cluster-clean
Signed-off-by: Irit goihman <igoihman@redhat.com>
2019-02-28 13:19:04 +02:00
Irit goihman
ca34bc5d50 config-controller: keep CDIConfig object when creating a new config
In case that CDIConfig already exists, create call will return the
object with empty fields. Calling update will result with an error
updating empty resource.

Signed-off-by: Irit goihman <igoihman@redhat.com>
2019-02-28 11:44:35 +02:00
Irit goihman
8089d795b9 CDI Config is now owned by the CDI object
Signed-off-by: Irit goihman <igoihman@redhat.com>
2019-02-28 10:29:36 +02:00
Michael Henriksen
6f1d130d97 tests and review comments 2019-02-25 20:12:56 -05:00
Michael Henriksen
62f1e77f66 change INSECURE_REGISTRY to INSECURE_TLS because may want to have insecure hosts for http import 2019-02-25 20:12:56 -05:00
Michael Henriksen
3892a7310d add configmap for insecure regestries 2019-02-25 20:12:56 -05:00
Irit Goihman
1691fe1b39 create cdi config with config controller
Signed-off-by: Irit goihman <igoihman@redhat.com>
2019-02-24 10:23:35 +02:00
Michael Henriksen
717ceaa919 add ability for user to specify certs for import from registry 2019-02-18 09:06:46 -05:00
Michael Henriksen
27d6a606a8 cleanup resources made by controller when uninstalling and flush out status 2019-01-16 20:22:14 -05:00
Michael Henriksen
b135463747 operator scaffolding 2019-01-08 16:03:21 -05:00
tavni
c245fe2ae6 Adding DataVolume support for unpacking imported archive
Signed-off-by: tavni <tavni@redhat.com>
2019-01-08 15:10:02 +02:00
tavni
aebc2b3e87 Add ability to create empty disk images #497
Signed-off-by: tavni <tavni@redhat.com>
2018-11-27 16:15:44 +02:00
Alexander Wels
260d4fa8e0 Automatically resize image to fit the requested PVC.
Combined code from PR#489 and PR#490 by
@gites and
@danielerez
Added some tests and rebased on current master.

Signed-off-by: Alexander Wels <awels@redhat.com>
2018-11-20 15:40:13 -05:00
Alexander Wels
70666e5946 Add source and contentType annotations
Signed-off-by: Alexander Wels <awels@redhat.com>
2018-11-14 12:20:26 -05:00
Alexander Wels
c485f39428 Properly label all CDI components.
Signed-off-by: Alexander Wels <awels@redhat.com>
2018-11-01 15:30:56 -04:00
Alexander Wels
40072b631a Report cloning progress in logs and prometheus.
Signed-off-by: Alexander Wels <awels@redhat.com>
2018-11-01 11:09:03 -04:00
Alexander Wels
f8dc6a8b59 Report import progress with prometheus
- Added code report progress with prometheus

Signed-off-by: Alexander Wels <awels@redhat.com>
2018-11-01 08:21:20 -04:00
Alexander Wels
a5a82aa970 Report import progress with prometheus
- Added code report progress with prometheus

Signed-off-by: Alexander Wels <awels@redhat.com>
2018-10-31 11:26:38 -04:00
tavni
c7e40ebffd Add structure to hold all CDI importer environment variables #501
Signed-off-by: tavni <tavni@redhat.com>
2018-10-25 13:21:28 +03:00
j-griffith
371e5c2ba0 Set pod restart policy to "OnFailure" and cleanup
This patch does two things:
1. We modify the restar policy from never to "OnFailure" for the
    Importer (we already did this for Upload, and we don't want to do it yet
    for the Clone pods)
2. Upon succesful completion delete our worker pods
    We're not doing this on failed pods right now because in general an
    admin will likely want to inspect the pod to see why it failed.

The first question someone will likely ask is "well what about a node
failure", well in the case of a node failure for CDI we have a bigger
outstanding issue remaining to figure out how to deal with volume
connections.  We can't just simply reschedule the job.
2018-10-09 08:25:22 -06:00
Michael Henriksen
452e30bca7 add verification to functional test and change to shared pod phase annotation 2018-09-19 14:37:19 -04:00
Michael Henriksen
5eeee34e83 refactorings from rebase 2018-09-19 12:30:46 -04:00
Michael Henriksen
b806786da1 refactoring and unit tests 2018-09-19 12:30:46 -04:00
Michael Henriksen
762cf6860d api server create explicit key for signing 2018-09-19 12:30:46 -04:00
Michael Henriksen
6ae444d661 have upload controller create cert for upload proxy and configure upload proxy to get key/cert from secret 2018-09-19 12:30:46 -04:00
Michael Henriksen
3d5927d30a incorporate some review comments 2018-09-19 12:30:46 -04:00
Michael Henriksen
28d349f660 minor refactoring and fix lint failure 2018-09-19 12:30:45 -04:00
Michael Henriksen
8910df8cee real tls cert validation 2018-09-19 12:30:45 -04:00
Michael Henriksen
1cbabdc8cf refactor key/cert storage to just use secrets 2018-09-19 12:30:45 -04:00
Michael Henriksen
59e0af3485 tls for upload server 2018-09-19 12:30:45 -04:00
Michael Henriksen
cd81879806 initial implementation of upload controller and upload server 2018-09-19 12:30:45 -04:00
tavni
ce22c48834 CDI controller code dedupe
Signed-off-by: tavni <tavni@redhat.com>
2018-09-17 19:44:09 +03:00
John Griffith
c455577cca Fix lint errors on common package (#449)
This change just updates the common package to pass golint.  Of course
that has some reaching implications into other packages with renaming of
constants.

I've intentionally kept this patch set to the bare minimum for the lint
test, I do think that we should put some effort into our use of
constants (especially those in common.go) in the future.

Ideally we'd declare constants where they're used, and we wouldn't
export them if they don't need to be.  There are some in here that
are only used in one or two packages, and that can be cleaned up pretty
easily however it's likely that there might be plans to use them
elsewhere in the future.
2018-09-10 09:14:09 -04:00
David Vossel
7908038b94 Ensure expectation keys are deleted when object is deleted from cluster (#433)
Signed-off-by: David Vossel <davidvossel@gmail.com>
2018-09-07 08:38:28 -04:00
Jeff Vance
7f01fbf6c5 pr 'make format' (#442) 2018-09-07 07:21:52 -04:00
Alexander Wels
95ac761c63 Fix the standard PVC e2e test failing.
Signed-off-by: Alexander Wels <awels@redhat.com>
2018-09-06 09:47:50 -04:00
John Griffith
9468ba94c7 Fix golint errors on pkg directory (#414)
Note we are intentionally skipping test files for now, golint
pkg/controller... will skip them, so I'll skip them to.
2018-09-05 12:17:38 -07:00
John Griffith
69709bf863 Update no annotation PVC import test. (#403) (#408)
- Set verbosity on controller to 3 to enable simpler tests.

Signed-off-by: Alexander Wels <awels@redhat.com>
2018-09-05 11:10:24 -07:00
danielerez
9f72eeec40 common: fixed key constants capitalization (#412)
Fixed capitalization to correalte with other constants in the file.
2018-09-05 09:32:43 -07:00
Alexander Wels
5c43a1373e
Update no annotation PVC import test. (#403)
- Set verbosity on controller to 3 to enable simpler tests.

Signed-off-by: Alexander Wels <awels@redhat.com>
2018-09-05 08:09:30 -04:00
John Griffith
e742a6cf63 Fix typo for "uniqe" (#397)
We have "CloneUniqeId", but we want "CloneUniqueID" (or something like
it)
2018-08-31 07:22:37 -04:00
zvikorn
d394d2de78 [WIP] CDI: expectations and single queue - cloning refactoring (#325)
[WIP] CDI: expectations and single queue - cloning refactoring

Adding 'expectPodCreate' call before creating target pod

minor change

Adding 'expectPodCreate' call before creating target pod

updating according feedback

Adding cloner unit test

code updates

Signed-off-by: tavni <tavni@redhat.com>

more updates

Signed-off-by: tavni <tavni@redhat.com>

minor change for util_test.go

Signed-off-by: tavni <tavni@redhat.com>

updating unit tests

Signed-off-by: tavni <tavni@redhat.com>

finished unit test for cloning refactoring

Signed-off-by: tavni <tavni@redhat.com>

rebase

Signed-off-by: tavni <tavni@redhat.com>

updating unit test after rebase

Signed-off-by: tavni <tavni@redhat.com>

gofmt errors

Signed-off-by: tavni <tavni@redhat.com>
2018-08-29 10:11:05 -07:00
Alexander Wels
363572626d cloner image uses ENTRYPOINT instead of hard coded path (#375)
- Updated controller to use ENTRYPOINT (no cmd) instead of /bin/sh
- Updated Dockerfile to set path to /usr/bin/cloner_startup.sh instead of /tmp
- Updated Dockerfile to use ENTRYPOINT.

Fixed issue #374

Signed-off-by: Alexander Wels <awels@redhat.com>
2018-08-27 18:35:13 -07:00
John Griffith
8ca745b700 Run gofmt on our directories (#370)
Just do a simple `gofmt -s -w` on pkg, tests and controller directories
to fix the misc space/tab mixes and some other ez formatting issues.
2018-08-23 15:31:14 -07:00
zvikorn
723e22259d Having the cloning label in the target pod, make the pod affinity fails. (#280)
The target pod looks for a pod with a specific label (specified in the pod affinity) that matches the source pod label.
In my case the target pod included this label as well, so we can see that the target pod found matching pod, but it is the WRONG pod. It's itself!!
The target was running without finding the source pod first.
If we remove this label from the target pod, it will find the source pod and then will be scheduled on the same node.
If it does not find the source pod (because the scheduler tried to schedule it before the source pod), it will be in 'Pending' state until the source pod is scheduled, and then will be running on the same node.
https://github.com/kubevirt/containerized-data-importer/issues/279
2018-07-31 09:35:45 -07:00
David Vossel
fdb1b27467 Single Queue and Expectations (#249)
* Pull in expectations code from kubevirt

Signed-off-by: David Vossel <davidvossel@gmail.com>

* add expectations to datavolume controller

Signed-off-by: David Vossel <davidvossel@gmail.com>

* Refactor importer pod to use single queue and expectations

Signed-off-by: David Vossel <davidvossel@gmail.com>

* update utils tests to use single queue

Signed-off-by: David Vossel <davidvossel@gmail.com>

* re-introduce import controller test suite

Signed-off-by: David Vossel <davidvossel@gmail.com>

* make clone controllers pass after utils refactor

Signed-off-by: David Vossel <davidvossel@gmail.com>

* Use log level constants in cdi controllers

Signed-off-by: David Vossel <davidvossel@gmail.com>

* Remove useless shadow variables in expectations pkg

Signed-off-by: David Vossel <davidvossel@gmail.com>

* required changes to get owner references working with openshift 1.10

Signed-off-by: David Vossel <davidvossel@gmail.com>

* update functional tests to work with importer controller refactor

Signed-off-by: David Vossel <davidvossel@gmail.com>

* Rename AnnImportPVC to LabelImportPvc in order to reflect its use

Signed-off-by: David Vossel <davidvossel@gmail.com>

* Add comment about expectations code

Signed-off-by: David Vossel <davidvossel@gmail.com>
2018-07-30 12:09:15 -05:00
tavni
b804b7bd7d Fixing PodAffinity by changing TopologyKey value and adding 'namespaces'
field
2018-07-18 14:59:51 +03:00
Jonathan Cope
aa82c9d577 Refactor random string gen to util.go (#258)
* Refactor random string gen to util.go

* unit test

* gofmt
2018-07-13 11:41:41 -07:00
David Vossel
97d87e39ef Import Controller Fixes (#238)
* Add owner references to import pod

Signed-off-by: David Vossel <davidvossel@gmail.com>

* Unit test to verify import pod spec and the new owner references

Signed-off-by: David Vossel <davidvossel@gmail.com>
2018-07-09 09:30:57 -07:00
Jon Cope
bcf269413d gofmt'd cmd, pkg, and test dirs (no change in cmd) 2018-07-06 14:45:10 -05:00
zvikorn
80b3180370 Host-assisted cloning integrated to CDI (#219)
* [WIP] Host-assisted cloning integrated to CDI

* changing back import from kubevirt.io to github.com/kubevirt

* [WIP] - changing code according feedback

* Applied comments

* changed PullPolicy

* applied comments

* minor fix for logging
2018-07-06 14:38:18 -05:00
David Vossel
7106bf73dc Conform with top level kubevirt github project (#198)
* Place cdi packages under the top level kubevirt.io project path

Signed-off-by: David Vossel <davidvossel@gmail.com>

* Change all references to cdi.io to cdi.kubevirt.io

Signed-off-by: David Vossel <davidvossel@gmail.com>

* Add go_import_path to travis config

Signed-off-by: David Vossel <davidvossel@gmail.com>

* update test package imports

Signed-off-by: David Vossel <davidvossel@gmail.com>
2018-06-21 15:44:52 -07:00
David Vossel
3e37aead88 Refactor cdi controller in preparation for multiple controllers
Signed-off-by: David Vossel <davidvossel@gmail.com>
2018-06-11 21:58:15 -04:00
Scott Creeley
ce83b0a57a refactor of util funcs 2018-05-22 14:00:18 -04:00
Jeff Vance
fbf4347a4f
Update util.go 2018-05-21 09:52:37 -07:00
jeff vance
fd13faa1aa change Patch to Update 2018-05-21 09:22:04 -07:00
Jon Cope
6cc5a1c4e5 Replaced all error returns with Dave Cheney's errors package
gofmt

removed unused import, fixed datastream interfae

Missed a fmt -> errors conversion

Stray fmt.Errorf replacement

vendor errors package

Wrapped controller main() scrope errors for uniformity, fixed double wrapping of errors
2018-05-15 15:12:53 -05:00
jeff vance
a835a15f26 add log verbosity 2018-05-14 15:35:36 -07:00
Jon Cope
2e918908a6 Add hyphen after pod name 2018-05-09 15:22:41 -05:00
jeff vance
2758e01a8d handle pvc.Get err 2018-05-08 17:37:40 -07:00
Jon Cope
6b863a2478 Add pod informer to controller
Start pod informer

also shutodwn podQueue

Added getting the PVC from pod's volume list using a global const for vol name

Generalized set-annotation func, added pod to pvc status writes

Use Filtered IndexInformer

Filter Pods by Label, examine all cluster PVCs

Resource dequeuing funcs return true unless key is malformed or processing succeeds

Pod correctly annotates PVC w/ status

Fix error message

Aligned log line with PVC func
2018-05-03 15:16:08 -05:00
Jon Cope
5dea5526ed Default pull policies to IfNotPreset unless env var is set
Fix testing, change env var & constant name
2018-04-30 10:43:25 -05:00
Scott Creeley
ed139bd58a fix possible race condition by checking annotation 2018-04-27 14:13:56 -04:00
Scott Creeley
9e671dfb57 add cdi label to pvc 2018-04-27 12:53:07 -04:00
Jon Cope
785424cd8f Refactor Controller to internally setup event handlers
Make CDI selector label a global constant

Parameterize selectorlabel

Handled nil selector case

Handel NewController error in testing, use constant importer tag

Changed informer factory back to un-filtered, fixed nil ptr error

removed labels, to be handled in a separate pr

To accomodate testing, NewController takes a pre-created Informer, not a factory

Add global constant for default resync

fix makefile bug

fix bug in deployent manifest
2018-04-26 15:34:35 -05:00
Jon Cope
d9dc1c92c8 Update makefile to push releases to kubevirt repo; Modify controller bin to accept non-default importer images
rename varibales to change 'tag' to 'image' references
2018-04-16 21:11:56 -05:00
jeff vance
cfeae65502 refactor unit tests 2018-04-06 13:08:08 -07:00
jeff vance
f9d6bf4f08 multiple namespaces 2018-04-02 15:55:51 -07:00
Jon Cope
a87c85b417 generate importer pod name, remote linux specific 2018-03-19 13:45:45 -05:00
Shiyang Wang
d06b6da42f add controller test 2018-03-13 17:49:54 +08:00
jeff vance
fcb2791de5 use Patch to update pvc anno 2018-03-05 20:59:30 -08:00
jeff vance
417df8890d replace status anno with importPodName, remove UpdateFunc and DeleteFunc 2018-03-01 23:57:28 -08:00
jeff vance
38d50cf051 support IMPORTER_TAG env var, include pvc name in importer pod name 2018-03-01 16:54:08 -08:00
jeff vance
d0906c4ff1 add common pkg and fix some err msgs to be more consistent 2018-02-27 13:11:42 -08:00
jeff vance
163fd1202c no explicit pvc requeue, use only secret name, and retry setting pvc status anno if stale 2018-02-27 09:41:58 -08:00
jeff vance
ae1d92ffd5 secret added to pod 2018-02-25 22:49:58 -08:00
jeff vance
e9f1fdd3e2 generate podName, create pod 2018-02-25 00:13:04 -08:00
jeff vance
d3587b4cf6 begin pod creation code, be retry is secret is missing 2018-02-24 00:24:04 -08:00
jeff vance
dcf163e0e8 look for pvc status annotation 2018-02-23 12:37:31 -08:00
jeff vance
4e9662b7af secret err handling 2018-02-22 18:24:18 -08:00
jeff vance
13c1c1209c secret and ep progress 2018-02-21 18:41:51 -08:00
jeff vance
ea293130ec controller action skeleton 2018-02-21 15:17:35 -08:00