github/intel-device-plugins-for-kubernetes
7
0
Fork 0
mirror of https://github.com/intel/intel-device-plugins-for-kubernetes.git synced 2025-06-03 03:59:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

Implement IAA plugin, operator, demo

Browse Source 
Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
Signed-off-by: Oleg Zhurakivskyy <oleg.zhurakivskyy@intel.com>
This commit is contained in:
Ed Bartosh 2020-11-26 14:56:04 +02:00 committed by Oleg Zhurakivskyy
parent 3a910f7252
commit 6b27cf1f7c
29 changed files with 1854 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.github/workflows/ci.yaml vendored
View File

@ -108,6 +108,7 @@ jobs:
- intel-sgx-plugin
- intel-sgx-initcontainer
- intel-dsa-plugin
- intel-iaa-plugin
- intel-idxd-config-initcontainer
- intel-dlb-plugin
@ -115,6 +116,7 @@ jobs:
- crypto-perf
- dsa-accel-config-demo
- intel-opencl-icd
- iaa-qpl-demo
- opae-nlb-demo
- openssl-qat-engine
- sgx-sdk-demo

8
README.md
View File

@ -21,6 +21,7 @@ Table of Contents
* [SGX device plugin](#sgx-device-plugin)
* [DSA device plugin](#dsa-device-plugin)
* [DLB device plugin](#dlb-device-plugin)
* [IAA device plugin](#iaa-device-plugin)
* [Device Plugins Operator](#device-plugins-operator)
* [Demos](#demos)
* [Workload Authors](#workload-authors)
@ -173,11 +174,15 @@ The [DSA device plugin](cmd/dsa_plugin/README.md) supports acceleration using th
The [DLB device plugin](cmd/dlb_plugin/README.md) supports Intel Dynamic Load Balancer accelerator(DLB).
### IAA device plugin
The [IAA device plugin](cmd/iaa_plugin/README.md) supports acceleration using the Intel Analytics accelerator(IAA).
## Device Plugins Operator
To simplify the deployment of the device plugins, a unified device plugins operator is implemented.
Currently the operator has support for the QAT, GPU, FPGA, SGX, DSA and DLB device plugins. Each
Currently the operator has support for the DSA, DLB, FPGA, GPU, IAA, QAT, SGX device plugins. Each
device plugin has its own custom resource definition (CRD) and the corresponding controller that
watches CRUD operations to those custom resources.
@ -212,6 +217,7 @@ The summary of resources available via plugins in this repository is given in a
| `dsa.intel.com` | `wq-user-[shared or dedicated]` | [dsa-accel-config-demo-pod.yaml](demo/dsa-accel-config-demo-pod.yaml) |
| `fpga.intel.com` | custom, see [mappings](cmd/fpga_admissionwebhook/README.md#mappings)| [intelfpga-job.yaml](demo/intelfpga-job.yaml) |
| `gpu.intel.com` | `i915` | [intelgpu-job.yaml](demo/intelgpu-job.yaml) |
| `iaa.intel.com` | `wq-user-[shared or dedicated]` | [iaa-qpl-demo-pod.yaml](demo/iaa-qpl-demo-pod.yaml) |
| `qat.intel.com` | `generic` | [crypto-perf-dpdk-pod-requesting-qat.yaml](deployments/qat_dpdk_app/base/crypto-perf-dpdk-pod-requesting-qat.yaml) |
| `sgx.intel.com` | `epc` | [intelsgx-job.yaml](deployments/sgx_enclave_apps/base/intelsgx-job.yaml) |
| `vpu.intel.com` | `hddl` | [intelvpu-job.yaml](demo/intelvpu-job.yaml) |

58
build/docker/intel-iaa-plugin.Dockerfile Normal file
View File

@ -0,0 +1,58 @@
# Copyright 2022 Intel Corporation. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# GOLANG_BASE can be used to make the build reproducible by choosing an
# image by its hash:
# GOLANG_BASE=golang@sha256:9d64369fd3c633df71d7465d67d43f63bb31192193e671742fa1c26ebc3a6210
#
# This is used on release branches before tagging a stable version.
# The main branch defaults to using the latest Golang base image.
ARG GOLANG_BASE=golang:1.17-bullseye
# FINAL_BASE can be used to configure the base image of the final image.
#
# This is used in two ways:
# 1) make <image-name> BUILDER=<docker|buildah>
# 2) docker build ... -f <image-name>.Dockerfile
#
# The project default is 1) which sets FINAL_BASE=gcr.io/distroless/static
# (see build-image.sh).
# 2) and the default FINAL_BASE is primarily used to build Redhat Certified Openshift Operator container images that must be UBI based.
# The RedHat build tool does not allow additional image build parameters.
ARG FINAL_BASE=registry.access.redhat.com/ubi8-micro
FROM ${GOLANG_BASE} as builder
ARG DIR=/intel-device-plugins-for-kubernetes
ARG GO111MODULE=on
ARG BUILDFLAGS="-ldflags=-w -s"
WORKDIR $DIR
COPY . .
RUN cd cmd/iaa_plugin; GO111MODULE=${GO111MODULE} CGO_ENABLED=0 go install "${BUILDFLAGS}"; cd -
RUN install -D /go/bin/iaa_plugin /install_root/usr/local/bin/intel_iaa_device_plugin \
&& install -D ${DIR}/LICENSE /install_root/licenses/intel-device-plugins-for-kubernetes/LICENSE \
&& GO111MODULE=on go install github.com/google/go-licenses@v1.0.0 && go-licenses save "./cmd/iaa_plugin" --save_path /install_root/licenses/go-licenses
FROM ${FINAL_BASE}
LABEL name='intel-iaa-plugin'
LABEL vendor='Intel®'
LABEL version='devel'
LABEL release='1'
LABEL summary='Intel® IAA device plugin for Kubernetes'
LABEL description='The IAA device plugin supports acceleration using the Intel Analytics accelerator(IAA)'
COPY --from=builder /install_root /
ENTRYPOINT ["/usr/local/bin/intel_iaa_device_plugin"]

166
cmd/iaa_plugin/README.md Normal file
View File

@ -0,0 +1,166 @@
# Intel IAA device plugin for Kubernetes
Table of Contents
* [Introduction](#introduction)
* [Installation](#installation)
* [Deploy with pre-built container image](#deploy-with-pre-built-container-image)
* [Getting the source code](#getting-the-source-code)
* [Verify node kubelet config](#verify-node-kubelet-config)
* [Deploying as a DaemonSet](#deploying-as-a-daemonset)
* [Build the plugin image](#build-the-plugin-image)
* [Deploy plugin DaemonSet](#deploy-plugin-daemonset)
* [Deploy by hand](#deploy-by-hand)
* [Build the plugin](#build-the-plugin)
* [Run the plugin as administrator](#run-the-plugin-as-administrator)
* [Verify plugin registration](#verify-plugin-registration)
* [Testing the plugin](#testing-the-plugin)
## Introduction
The IAA device plugin for Kubernetes supports acceleration using the Intel Analytics accelerator(IAA).
The IAA plugin discovers IAA work queues and presents them as a node resources.
The IAA plugin and operator optionally support provisioning of IAA devices and workqueues with the help of [accel-config](https://github.com/intel/idxd-config) utility through initcontainer.
## Installation
The following sections detail how to obtain, build, deploy and test the IAA device plugin.
### Getting the source code
```bash
$ git clone https://github.com/intel/intel-device-plugins-for-kubernetes
```
### Deploying as a DaemonSet
To deploy the IAA plugin as a daemonset, you first need to build a container image for the
plugin and ensure that is visible to your nodes.
#### Build the plugin image
The following will use `docker` to build a local container image called
`intel/intel-iaa-plugin` with the tag `devel`.
```bash
$ cd ${INTEL_DEVICE_PLUGINS_SRC}
$ make intel-iaa-plugin
...
Successfully tagged intel/intel-iaa-plugin:devel
```
#### Deploy plugin DaemonSet
You can then use the [example DaemonSet YAML](/deployments/iaa_plugin/base/intel-iaa-plugin.yaml)
file provided to deploy the plugin. The default kustomization that deploys the YAML as is:
```bash
$ kubectl apply -k deployments/iaa_plugin
daemonset.apps/intel-iaa-plugin created
```
### Deploy with initcontainer
There's a sample [idxd initcontainer](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/build/docker/intel-idxd-initcontainer.Dockerfile) included that provisions IAA devices and workqueues (1 engine / 1 group / 1 wq (user/dedicated)), to deploy:
```bash
$ kubectl apply -k deployments/iaa_plugin/overlays/iaa_initcontainer/
```
The provisioning [script](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/demo/idxd-init.sh) and [template](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/demo/iaa.conf) are available for customization.
### Deploy with initcontainer and provisioning config in the ConfigMap
The provisioning config can be optionally stored in the ProvisioningConfig configMap which is then passed to initcontainer through the volume mount.
There's also a possibility for a node specific congfiguration through passing a nodename via NODE_NAME into initcontainer's environment and passing a node specific profile via configMap volume mount.
To create a custom provisioning config:
```bash
$ kubectl create configmap --namespace=inteldeviceplugins-system intel-iaa-config --from-file=demo/iaa.conf --from-file=demo/iaa-node1.conf --dry-run=client -o yaml > iaa-config.yaml
```
### Deploy by hand
For development purposes, it is sometimes convenient to deploy the plugin 'by hand' on a node.
In this case, you do not need to build the complete container image, and can build just the plugin.
#### Build the plugin
First we build the plugin:
```bash
$ make iaa_plugin
```
#### Run the plugin as administrator
Now we can run the plugin directly on the node:
```bash
$ sudo -E ./cmd/iaa_plugin/iaa_plugin
device-plugin registered
```
### Verify plugin registration
You can verify the plugin has been registered with the expected nodes by searching for the relevant
resource allocation status on the nodes:
```bash
$ kubectl get nodes -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{range $k,$v:=.status.allocatable}}{{" "}}{{$k}}{{": "}}{{$v}}{{"\n"}}{{end}}{{end}}' | grep '^\([^ ]\)\|\( iaa\)'
master
iaa.intel.com/wq-user-dedicated: 2
iaa.intel.com/wq-user-shared: 10
node1
iaa.intel.com/wq-user-dedicated: 4
iaa.intel.com/wq-user-shared: 30
```
### Testing the plugin
We can test the plugin is working by deploying the provided example iaa-qpl-demo test image.
1. Build a Docker image with an accel-config tests:
```bash
$ make iaa-qpl-demo
...
Successfully tagged iaa-qpl-demo:devel
```
1. Create a pod running unit tests off the local Docker image:
```bash
$ kubectl apply -f ./demo/iaa-qpl-demo-pod.yaml
pod/iaa-qpl-demo created
```
1. Wait until pod is completed:
```bash
$ kubectl get pods |grep iaa-qpl-demo
iaa-qpl-demo 0/1 Completed 0 31m
If the pod did not successfully launch, possibly because it could not obtain the IAA
resource, it will be stuck in the `Pending` status:
```bash
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
iaa-qpl-demo 0/1 Pending 0 7s
```
This can be verified by checking the Events of the pod:
```bash
$ kubectl describe pod iaa-qpl-demo | grep -A3 Events:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m26s default-scheduler 0/1 nodes are available: 1 Insufficient iaa.intel.com/wq-user-dedicated, 1 Insufficient iaa.intel.com/wq-user-shared.
```

57
cmd/iaa_plugin/iaa_plugin.go Normal file
View File

@ -0,0 +1,57 @@
// Copyright 2020 Intel Corporation. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"flag"
"os"
dpapi "github.com/intel/intel-device-plugins-for-kubernetes/pkg/deviceplugin"
"github.com/intel/intel-device-plugins-for-kubernetes/pkg/idxd"
"k8s.io/klog/v2"
)
const (
// Device plugin settings.
namespace = "iaa.intel.com"
// SysFS directory.
sysfsDir = "/sys/bus/iax/devices"
// Device directories.
devDir = "/dev/iax"
// Glob pattern for the state sysfs entry.
statePattern = "/sys/bus/iax/devices/iax*/wq*/state"
)
func main() {
var sharedDevNum int
flag.IntVar(&sharedDevNum, "shared-dev-num", 1, "number of containers sharing the same work queue")
flag.Parse()
if sharedDevNum < 1 {
klog.Warning("The number of containers sharing the same work queue must be greater than zero")
os.Exit(1)
}
plugin := idxd.NewDevicePlugin(sysfsDir, statePattern, devDir, sharedDevNum)
if plugin == nil {
klog.Fatal("Cannot create device plugin, please check above error messages.")
}
manager := dpapi.NewManager(namespace, plugin)
manager.Run()
}

6
cmd/operator/main.go
View File

@ -34,6 +34,7 @@ import (
"github.com/intel/intel-device-plugins-for-kubernetes/pkg/controllers/dsa"
"github.com/intel/intel-device-plugins-for-kubernetes/pkg/controllers/fpga"
"github.com/intel/intel-device-plugins-for-kubernetes/pkg/controllers/gpu"
"github.com/intel/intel-device-plugins-for-kubernetes/pkg/controllers/iaa"
"github.com/intel/intel-device-plugins-for-kubernetes/pkg/controllers/qat"
"github.com/intel/intel-device-plugins-for-kubernetes/pkg/controllers/sgx"
"github.com/intel/intel-device-plugins-for-kubernetes/pkg/fpgacontroller"
@ -60,7 +61,7 @@ type devicePluginControllerAndWebhook map[string](func(ctrl.Manager, string, boo
type flagList []string
var supportedDevices = flagList{"dsa", "dlb", "fpga", "gpu", "qat", "sgx"}
var supportedDevices = flagList{"dsa", "dlb", "fpga", "gpu", "iaa", "qat", "sgx"}
var devices flagList
func (flag *flagList) String() string {
@ -118,8 +119,9 @@ func main() {
setupControllerAndWebhook := devicePluginControllerAndWebhook{
"dlb": dlb.SetupReconciler,
"dsa": dsa.SetupReconciler,
"gpu": gpu.SetupReconciler,
"fpga": fpga.SetupReconciler,
"gpu": gpu.SetupReconciler,
"iaa": iaa.SetupReconciler,
"qat": qat.SetupReconciler,
"sgx": sgx.SetupReconciler,
}

87
demo/iaa-node1.conf Normal file
View File

@ -0,0 +1,87 @@
[
{
"dev": "iaxX",
"groups": [
{
"dev": "groupX.0",
"grouped_workqueues": [
{
"dev": "wqX.0",
"type": "user",
"mode": "dedicated",
"size": 16,
"group_id": 0,
"priority": 10,
"name": "iaxX0"
}
],
"grouped_engines": [
{
"dev": "engineX.0",
"group_id": 0
}
]
},
{
"dev": "groupX.1",
"grouped_workqueues": [
{
"dev": "wqX.1",
"type": "user",
"mode": "dedicated",
"size": 16,
"group_id": 1,
"priority": 10,
"name": "iaxX1"
}
],
"grouped_engines": [
{
"dev": "engineX.1",
"group_id": 1
}
]
},
{
"dev": "groupX.2",
"grouped_workqueues": [
{
"dev": "wqX.2",
"type": "user",
"mode": "dedicated",
"size": 16,
"group_id": 2,
"priority": 10,
"name": "iaxX2"
}
],
"grouped_engines": [
{
"dev": "engineX.2",
"group_id": 2
}
]
},
{
"dev": "groupX.3",
"grouped_workqueues": [
{
"dev": "wqX.3",
"type": "user",
"mode": "dedicated",
"size": 16,
"group_id": 3,
"priority": 10,
"name": "iaxX3"
}
],
"grouped_engines": [
{
"dev": "engineX.3",
"group_id": 3
}
]
}
]
}
]

16
demo/iaa-qpl-demo-pod.yaml Normal file
View File

@ -0,0 +1,16 @@
apiVersion: v1
kind: Pod
metadata:
name: iaa-qpl-demo
spec:
containers:
- name: iaa-qpl-demo
image: iaa-qpl-demo:devel
imagePullPolicy: IfNotPresent
resources:
limits:
iaa.intel.com/wq-user-dedicated: 1
iaa.intel.com/wq-user-shared: 1
cpu: 1
restartPolicy: Never

37
demo/iaa-qpl-demo/Dockerfile Normal file
View File

@ -0,0 +1,37 @@
# Copyright 2021-2022 Intel Corporation. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
FROM fedora:32 AS builder
RUN dnf install -y accel-config accel-config-devel \
git cmake make g++ nasm clang libuuid-devel
RUN git clone --recursive --depth 1 --branch v0.1.20 \
https://github.com/intel/qpl.git qpl.git
RUN cd qpl.git && \
mkdir build && \
cd build && \
cmake .. && \
cmake --build . --target install
FROM fedora:32
RUN dnf install -y accel-config accel-config-devel python
COPY --from=builder /usr/local/bin/tests /usr/bin/iaa-tests
COPY --from=builder /usr/local/bin/test_frontend /usr/bin/iaa-test_frontend
COPY --from=builder /usr/local/bin/init_tests /usr/local/bin/iaa-init_tests
ENTRYPOINT /usr/bin/iaa-test_frontend && python init_tests.py --test_path=/usr/local/bin/iaa-init_tests

87
demo/iaa.conf Normal file
View File

@ -0,0 +1,87 @@
[
{
"dev": "iaxX",
"groups": [
{
"dev": "groupX.0",
"grouped_workqueues": [
{
"dev": "wqX.0",
"type": "user",
"mode": "dedicated",
"size": 16,
"group_id": 0,
"priority": 10,
"name": "iaxX0"
}
],
"grouped_engines": [
{
"dev": "engineX.0",
"group_id": 0
}
]
},
{
"dev": "groupX.1",
"grouped_workqueues": [
{
"dev": "wqX.1",
"type": "user",
"mode": "dedicated",
"size": 16,
"group_id": 1,
"priority": 10,
"name": "iaxX1"
}
],
"grouped_engines": [
{
"dev": "engineX.1",
"group_id": 1
}
]
},
{
"dev": "groupX.2",
"grouped_workqueues": [
{
"dev": "wqX.2",
"type": "user",
"mode": "dedicated",
"size": 16,
"group_id": 2,
"priority": 10,
"name": "iaxX2"
}
],
"grouped_engines": [
{
"dev": "engineX.2",
"group_id": 2
}
]
},
{
"dev": "groupX.3",
"grouped_workqueues": [
{
"dev": "wqX.3",
"type": "user",
"mode": "dedicated",
"size": 16,
"group_id": 3,
"priority": 10,
"name": "iaxX3"
}
],
"grouped_engines": [
{
"dev": "engineX.3",
"group_id": 3
}
]
}
]
}
]

2
demo/idxd-init.sh
View File

@ -2,7 +2,7 @@
set -euo pipefail
DEV="${VARIABLE:-dsa}"
DEV="${IDXD_DEVICE_TYPE:-dsa}"
NODE_NAME="${NODE_NAME:-}"
function cmd() {

7
deployments/daemonsets.go
View File

@ -50,6 +50,13 @@ func GPUPluginDaemonSet() *apps.DaemonSet {
return getDaemonset(contentGPU).DeepCopy()
}
//go:embed iaa_plugin/base/*plugin.yaml
var contentIAA []byte
func IAAPluginDaemonSet() *apps.DaemonSet {
return getDaemonset(contentIAA).DeepCopy()
}
//go:embed qat_plugin/base/*qat-plugin.yaml
var contentQAT []byte

53
deployments/iaa_plugin/base/intel-iaa-plugin.yaml Normal file
View File

@ -0,0 +1,53 @@
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: intel-iaa-plugin
labels:
app: intel-iaa-plugin
spec:
selector:
matchLabels:
app: intel-iaa-plugin
template:
metadata:
labels:
app: intel-iaa-plugin
spec:
containers:
- name: intel-iaa-plugin
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
image: intel/intel-iaa-plugin:devel
imagePullPolicy: IfNotPresent
securityContext:
readOnlyRootFilesystem: true
volumeMounts:
- name: devfs
mountPath: /dev/iax
readOnly: true
- name: chardevs
mountPath: /dev/char
readOnly: true
- name: sysfs
mountPath: /sys/bus/iax
readOnly: true
- name: kubeletsockets
mountPath: /var/lib/kubelet/device-plugins
volumes:
- name: devfs
hostPath:
path: /dev/iax
- name: chardevs
hostPath:
path: /dev/char
- name: sysfs
hostPath:
path: /sys/bus/iax
- name: kubeletsockets
hostPath:
path: /var/lib/kubelet/device-plugins
nodeSelector:
kubernetes.io/arch: amd64

182
deployments/iaa_plugin/overlays/iaa_initcontainer/iaa-config.yaml Normal file
View File

@ -0,0 +1,182 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: intel-iaa-config
namespace: inteldeviceplugins-system
data:
iaa.conf: |
[
{
"dev": "iaxX",
"groups": [
{
"dev": "groupX.0",
"grouped_workqueues": [
{
"dev": "wqX.0",
"type": "user",
"mode": "dedicated",
"size": 16,
"group_id": 0,
"priority": 10,
"name": "iaxX0"
}
],
"grouped_engines": [
{
"dev": "engineX.0",
"group_id": 0
}
]
},
{
"dev": "groupX.1",
"grouped_workqueues": [
{
"dev": "wqX.1",
"type": "user",
"mode": "dedicated",
"size": 16,
"group_id": 1,
"priority": 10,
"name": "iaxX1"
}
],
"grouped_engines": [
{
"dev": "engineX.1",
"group_id": 1
}
]
},
{
"dev": "groupX.2",
"grouped_workqueues": [
{
"dev": "wqX.2",
"type": "user",
"mode": "dedicated",
"size": 16,
"group_id": 2,
"priority": 10,
"name": "iaxX2"
}
],
"grouped_engines": [
{
"dev": "engineX.2",
"group_id": 2
}
]
},
{
"dev": "groupX.3",
"grouped_workqueues": [
{
"dev": "wqX.3",
"type": "user",
"mode": "dedicated",
"size": 16,
"group_id": 3,
"priority": 10,
"name": "iaxX3"
}
],
"grouped_engines": [
{
"dev": "engineX.3",
"group_id": 3
}
]
}
]
}
]
iaa-node1.conf: |
[
{
"dev": "iaxX",
"groups": [
{
"dev": "groupX.0",
"grouped_workqueues": [
{
"dev": "wqX.0",
"type": "user",
"mode": "dedicated",
"size": 16,
"group_id": 0,
"priority": 10,
"name": "iaxX0"
}
],
"grouped_engines": [
{
"dev": "engineX.0",
"group_id": 0
}
]
},
{
"dev": "groupX.1",
"grouped_workqueues": [
{
"dev": "wqX.1",
"type": "user",
"mode": "dedicated",
"size": 16,
"group_id": 1,
"priority": 10,
"name": "iaxX1"
}
],
"grouped_engines": [
{
"dev": "engineX.1",
"group_id": 1
}
]
},
{
"dev": "groupX.2",
"grouped_workqueues": [
{
"dev": "wqX.2",
"type": "user",
"mode": "dedicated",
"size": 16,
"group_id": 2,
"priority": 10,
"name": "iaxX2"
}
],
"grouped_engines": [
{
"dev": "engineX.2",
"group_id": 2
}
]
},
{
"dev": "groupX.3",
"grouped_workqueues": [
{
"dev": "wqX.3",
"type": "user",
"mode": "dedicated",
"size": 16,
"group_id": 3,
"priority": 10,
"name": "iaxX3"
}
],
"grouped_engines": [
{
"dev": "engineX.3",
"group_id": 3
}
]
}
]
}
]

31
deployments/iaa_plugin/overlays/iaa_initcontainer/iaa_initcontainer.yaml Normal file
View File

@ -0,0 +1,31 @@
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: intel-iaa-plugin
spec:
template:
spec:
initContainers:
- name: intel-iaa-initcontainer
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: IDXD_DEVICE_TYPE
value: "iaa"
image: intel/intel-idxd-config-initcontainer:devel
securityContext:
privileged: true
volumeMounts:
- mountPath: /sys/devices
name: sys-devices
- mountPath: /idxd-init/conf
name: intel-iaa-config-volume
volumes:
- name: sys-devices
hostPath:
path: /sys/devices
- name: intel-iaa-config-volume
configMap:
name: intel-iaa-config

4
deployments/iaa_plugin/overlays/iaa_initcontainer/kustomization.yaml Normal file
View File

@ -0,0 +1,4 @@
bases:
- ../../base
patchesStrategicMerge:
- iaa_initcontainer.yaml

152
deployments/operator/crd/bases/deviceplugin.intel.com_iaadeviceplugins.yaml Normal file
View File

@ -0,0 +1,152 @@
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.8.0
creationTimestamp: null
name: iaadeviceplugins.deviceplugin.intel.com
spec:
group: deviceplugin.intel.com
names:
kind: IaaDevicePlugin
listKind: IaaDevicePluginList
plural: iaadeviceplugins
singular: iaadeviceplugin
scope: Cluster
versions:
- additionalPrinterColumns:
- jsonPath: .status.desiredNumberScheduled
name: Desired
type: integer
- jsonPath: .status.numberReady
name: Ready
type: integer
- jsonPath: .spec.nodeSelector
name: Node Selector
type: string
- jsonPath: .metadata.creationTimestamp
name: Age
type: date
name: v1
schema:
openAPIV3Schema:
description: IaaDevicePlugin is the Schema for the iaadeviceplugins API. It
represents the IAA device plugin responsible for advertising Intel IAA hardware
resources to the kubelet.
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: IaaDevicePluginSpec defines the desired state of IaaDevicePlugin.
properties:
image:
description: Image is a container image with IAA device plugin executable.
type: string
initImage:
description: InitImage is an initcontainer image to configure and
enable IAA devices and workqueues with accel-config utility
type: string
logLevel:
description: LogLevel sets the plugin's log level.
minimum: 0
type: integer
nodeSelector:
additionalProperties:
type: string
description: NodeSelector provides a simple way to constrain device
plugin pods to nodes with particular labels.
type: object
provisioningConfig:
description: ProvisioningConfig is a ConfigMap used to pass the IAA
configuration into idxd initcontainer.
type: string
sharedDevNum:
description: SharedDevNum is a number of containers that can share
the same IAA device.
minimum: 1
type: integer
type: object
status:
description: IaaDevicePluginStatus defines the observed state of IaaDevicePlugin.
properties:
controlledDaemonSet:
description: ControlledDaemoSet references the DaemonSet controlled
by the operator.
properties:
apiVersion:
description: API version of the referent.
type: string
fieldPath:
description: 'If referring to a piece of an object instead of
an entire object, this string should contain a valid JSON/Go
field access statement, such as desiredState.manifest.containers[2].
For example, if the object reference is to a container within
a pod, this would take on a value like: "spec.containers{name}"
(where "name" refers to the name of the container that triggered
the event) or if no container name is specified "spec.containers[2]"
(container with index 2 in this pod). This syntax is chosen
only to have some well-defined way of referencing a part of
an object. TODO: this design is not final and this field is
subject to change in the future.'
type: string
kind:
description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
name:
description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
type: string
namespace:
description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
type: string
resourceVersion:
description: 'Specific resourceVersion to which this reference
is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
type: string
uid:
description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
type: string
type: object
desiredNumberScheduled:
description: The total number of nodes that should be running the
device plugin pod (including nodes correctly running the device
plugin pod).
format: int32
type: integer
nodeNames:
description: The list of Node names where the device plugin pods are
running.
items:
type: string
type: array
numberReady:
description: The number of nodes that should be running the device
plugin pod and have one or more of the device plugin pod running
and ready.
format: int32
type: integer
required:
- desiredNumberScheduled
- numberReady
type: object
type: object
served: true
storage: true
subresources:
status: {}
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []

1
deployments/operator/crd/kustomization.yaml
View File

@ -7,6 +7,7 @@ resources:
- bases/deviceplugin.intel.com_fpgadeviceplugins.yaml
- bases/deviceplugin.intel.com_sgxdeviceplugins.yaml
- bases/deviceplugin.intel.com_dsadeviceplugins.yaml
- bases/deviceplugin.intel.com_iaadeviceplugins.yaml
- bases/deviceplugin.intel.com_dlbdeviceplugins.yaml
- bases/fpga.intel.com_acceleratorfunctions.yaml
- bases/fpga.intel.com_fpgaregions.yaml

7
deployments/operator/manifests/bases/intel-device-plugins-operator.clusterserviceversion.yaml
View File

@ -73,10 +73,15 @@ spec:
kind: SgxDevicePlugin
name: sgxdeviceplugins.deviceplugin.intel.com
version: v1
description: IaaDevicePlugin is the Schema for the iaadeviceplugins API. It represents the IAA device plugin responsible for advertising Intel IAA hardware resources to the kubelet.
displayName: Intel IAA Device Plugin
kind: IaaDevicePlugin
name: iaadeviceplugins.deviceplugin.intel.com
version: v1
description: |
[Intel Device Plugins for Kubernetes](https://github.com/intel/intel-device-plugins-for-kubernetes) is a collection of
[device plugins](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) advertising Intel specific hardware resources
to the kubelet. Currently the operator has basic support for the QAT, GPU, FPGA, SGX and DSA device plugins: it validates container image references and extends
to the kubelet. Currently the operator has basic support for the QAT, GPU, FPGA, SGX, DSA, IAA device plugins: it validates container image references and extends
reported statuses.
displayName: Intel Device Plugins Operator
icon:

20
deployments/operator/rbac/role.yaml
View File

@ -154,6 +154,26 @@ rules:
- get
- patch
- update
- apiGroups:
- deviceplugin.intel.com
resources:
- iaadeviceplugins
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- deviceplugin.intel.com
resources:
- iaadeviceplugins/status
verbs:
- get
- patch
- update
- apiGroups:
- deviceplugin.intel.com
resources:

12
deployments/operator/samples/deviceplugin_v1_iaadeviceplugin.yaml Normal file
View File

@ -0,0 +1,12 @@
apiVersion: deviceplugin.intel.com/v1
kind: IaaDevicePlugin
metadata:
name: iaadeviceplugin-sample
spec:
image: intel/intel-iaa-plugin:0.23.0
sharedDevNum: 10
logLevel: 4
# TODO: nodeSelector is supported and can be used once it becomes
# available through the node feature discovery
# nodeSelector:
# intel.feature.node.kubernetes.io/dsa: 'true'

1
deployments/operator/samples/kustomization.yaml
View File

@ -2,6 +2,7 @@ resources:
- deviceplugin_v1_dsadeviceplugin.yaml
- deviceplugin_v1_fpgadeviceplugin.yaml
- deviceplugin_v1_gpudeviceplugin.yaml
- deviceplugin_v1_iaadeviceplugin.yaml
- deviceplugin_v1_qatdeviceplugin.yaml
- deviceplugin_v1_sgxdeviceplugin.yaml
- deviceplugin_v1_dlbdeviceplugin.yaml

40
deployments/operator/webhook/manifests.yaml
View File

@ -85,6 +85,26 @@ webhooks:
resources:
- gpudeviceplugins
sideEffects: None
- admissionReviewVersions:
- v1
clientConfig:
service:
name: webhook-service
namespace: system
path: /mutate-deviceplugin-intel-com-v1-iaadeviceplugin
failurePolicy: Fail
name: miaadeviceplugin.kb.io
rules:
- apiGroups:
- deviceplugin.intel.com
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- iaadeviceplugins
sideEffects: None
- admissionReviewVersions:
- v1
clientConfig:
@ -253,6 +273,26 @@ webhooks:
resources:
- gpudeviceplugins
sideEffects: None
- admissionReviewVersions:
- v1
clientConfig:
service:
name: webhook-service
namespace: system
path: /validate-deviceplugin-intel-com-v1-iaadeviceplugin
failurePolicy: Fail
name: viaadeviceplugin.kb.io
rules:
- apiGroups:
- deviceplugin.intel.com
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- iaadeviceplugins
sideEffects: None
- admissionReviewVersions:
- v1
clientConfig:

102
pkg/apis/deviceplugin/v1/iaadeviceplugin_types.go Normal file
View File

@ -0,0 +1,102 @@
// Copyright 2021 Intel Corporation. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package v1
import (
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// NOTE: json tags are required. Any new fields you add must have json tags for the fields to be serialized.
// IaaDevicePluginSpec defines the desired state of IaaDevicePlugin.
type IaaDevicePluginSpec struct {
// Important: Run "make generate" to regenerate code after modifying this file
// NodeSelector provides a simple way to constrain device plugin pods to nodes with particular labels.
NodeSelector map[string]string `json:"nodeSelector,omitempty"`
// Image is a container image with IAA device plugin executable.
Image string `json:"image,omitempty"`
// InitImage is an initcontainer image to configure and enable IAA devices and workqueues with accel-config utility
InitImage string `json:"initImage,omitempty"`
// ProvisioningConfig is a ConfigMap used to pass the IAA configuration into idxd initcontainer.
ProvisioningConfig string `json:"provisioningConfig,omitempty"`
// SharedDevNum is a number of containers that can share the same IAA device.
// +kubebuilder:validation:Minimum=1
SharedDevNum int `json:"sharedDevNum,omitempty"`
// LogLevel sets the plugin's log level.
// +kubebuilder:validation:Minimum=0
LogLevel int `json:"logLevel,omitempty"`
}
// IaaDevicePluginStatus defines the observed state of IaaDevicePlugin.
type IaaDevicePluginStatus struct {
// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
// Important: Run "make generate" to regenerate code after modifying this file
// ControlledDaemoSet references the DaemonSet controlled by the operator.
// +optional
ControlledDaemonSet v1.ObjectReference `json:"controlledDaemonSet,omitempty"`
// The list of Node names where the device plugin pods are running.
// +optional
NodeNames []string `json:"nodeNames,omitempty"`
// The total number of nodes that should be running the device plugin
// pod (including nodes correctly running the device plugin pod).
DesiredNumberScheduled int32 `json:"desiredNumberScheduled"`
// The number of nodes that should be running the device plugin pod and have one
// or more of the device plugin pod running and ready.
NumberReady int32 `json:"numberReady"`
}
// +kubebuilder:object:root=true
// +kubebuilder:resource:path=iaadeviceplugins,scope=Cluster
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Desired",type=integer,JSONPath=`.status.desiredNumberScheduled`
// +kubebuilder:printcolumn:name="Ready",type=integer,JSONPath=`.status.numberReady`
// +kubebuilder:printcolumn:name="Node Selector",type=string,JSONPath=`.spec.nodeSelector`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
// +operator-sdk:csv:customresourcedefinitions:displayName="Intel IAA Device Plugin"
// IaaDevicePlugin is the Schema for the iaadeviceplugins API. It represents
// the IAA device plugin responsible for advertising Intel IAA hardware resources to
// the kubelet.
type IaaDevicePlugin struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec IaaDevicePluginSpec `json:"spec,omitempty"`
Status IaaDevicePluginStatus `json:"status,omitempty"`
}
// +kubebuilder:object:root=true
// IaaDevicePluginList contains a list of IaaDevicePlugin.
type IaaDevicePluginList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items []IaaDevicePlugin `json:"items"`
}
func init() {
SchemeBuilder.Register(&IaaDevicePlugin{}, &IaaDevicePluginList{})
}

101
pkg/apis/deviceplugin/v1/iaadeviceplugin_webhook.go Normal file
View File

@ -0,0 +1,101 @@
// Copyright 2021 Intel Corporation. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package v1
import (
"github.com/pkg/errors"
"k8s.io/apimachinery/pkg/runtime"
ctrl "sigs.k8s.io/controller-runtime"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"sigs.k8s.io/controller-runtime/pkg/webhook"
"github.com/intel/intel-device-plugins-for-kubernetes/pkg/controllers"
)
const (
iaaPluginKind = "IaaDevicePlugin"
)
var (
// iaadevicepluginlog is for logging in this package.
iaadevicepluginlog = logf.Log.WithName("iaadeviceplugin-resource")
iaaMinVersion = controllers.ImageMinVersion
)
// SetupWebhookWithManager sets up a webhook for IaaDevicePlugin custom resources.
func (r *IaaDevicePlugin) SetupWebhookWithManager(mgr ctrl.Manager) error {
return ctrl.NewWebhookManagedBy(mgr).
For(r).
Complete()
}
// +kubebuilder:webhook:path=/mutate-deviceplugin-intel-com-v1-iaadeviceplugin,mutating=true,failurePolicy=fail,groups=deviceplugin.intel.com,resources=iaadeviceplugins,verbs=create;update,versions=v1,name=miaadeviceplugin.kb.io,sideEffects=None,admissionReviewVersions=v1
var _ webhook.Defaulter = &IaaDevicePlugin{}
// Default implements webhook.Defaulter so a webhook will be registered for the type.
func (r *IaaDevicePlugin) Default() {
iaadevicepluginlog.Info("default", "name", r.Name)
if len(r.Spec.Image) == 0 {
r.Spec.Image = "intel/intel-iaa-plugin:" + iaaMinVersion.String()
}
}
// +kubebuilder:webhook:verbs=create;update,path=/validate-deviceplugin-intel-com-v1-iaadeviceplugin,mutating=false,failurePolicy=fail,groups=deviceplugin.intel.com,resources=iaadeviceplugins,versions=v1,name=viaadeviceplugin.kb.io,sideEffects=None,admissionReviewVersions=v1
var _ webhook.Validator = &IaaDevicePlugin{}
// ValidateCreate implements webhook.Validator so a webhook will be registered for the type.
func (r *IaaDevicePlugin) ValidateCreate() error {
iaadevicepluginlog.Info("validate create", "name", r.Name)
if controllers.GetDevicePluginCount(iaaPluginKind) > 0 {
return errors.Errorf("an instance of %q already exists in the cluster", iaaPluginKind)
}
return r.validatePlugin()
}
// ValidateUpdate implements webhook.Validator so a webhook will be registered for the type.
func (r *IaaDevicePlugin) ValidateUpdate(old runtime.Object) error {
iaadevicepluginlog.Info("validate update", "name", r.Name)
return r.validatePlugin()
}
// ValidateDelete implements webhook.Validator so a webhook will be registered for the type.
func (r *IaaDevicePlugin) ValidateDelete() error {
iaadevicepluginlog.Info("validate delete", "name", r.Name)
return nil
}
func (r *IaaDevicePlugin) validatePlugin() error {
if err := validatePluginImage(r.Spec.Image, "intel-iaa-plugin", iaaMinVersion); err != nil {
return err
}
if len(r.Spec.ProvisioningConfig) > 0 && len(r.Spec.InitImage) == 0 {
return errors.Errorf("ProvisioningConfig is set with no InitImage")
}
if len(r.Spec.InitImage) > 0 {
return validatePluginImage(r.Spec.InitImage, "intel-idxd-config-initcontainer", iaaMinVersion)
}
return nil
}

102
pkg/apis/deviceplugin/v1/zz_generated.deepcopy.go
View File

@ -431,6 +431,108 @@ func (in *GpuDevicePluginStatus) DeepCopy() *GpuDevicePluginStatus {
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *IaaDevicePlugin) DeepCopyInto(out *IaaDevicePlugin) {
*out = *in
out.TypeMeta = in.TypeMeta
in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
in.Spec.DeepCopyInto(&out.Spec)
in.Status.DeepCopyInto(&out.Status)
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new IaaDevicePlugin.
func (in *IaaDevicePlugin) DeepCopy() *IaaDevicePlugin {
if in == nil {
return nil
}
out := new(IaaDevicePlugin)
in.DeepCopyInto(out)
return out
}
// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *IaaDevicePlugin) DeepCopyObject() runtime.Object {
if c := in.DeepCopy(); c != nil {
return c
}
return nil
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *IaaDevicePluginList) DeepCopyInto(out *IaaDevicePluginList) {
*out = *in
out.TypeMeta = in.TypeMeta
in.ListMeta.DeepCopyInto(&out.ListMeta)
if in.Items != nil {
in, out := &in.Items, &out.Items
*out = make([]IaaDevicePlugin, len(*in))
for i := range *in {
(*in)[i].DeepCopyInto(&(*out)[i])
}
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new IaaDevicePluginList.
func (in *IaaDevicePluginList) DeepCopy() *IaaDevicePluginList {
if in == nil {
return nil
}
out := new(IaaDevicePluginList)
in.DeepCopyInto(out)
return out
}
// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *IaaDevicePluginList) DeepCopyObject() runtime.Object {
if c := in.DeepCopy(); c != nil {
return c
}
return nil
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *IaaDevicePluginSpec) DeepCopyInto(out *IaaDevicePluginSpec) {
*out = *in
if in.NodeSelector != nil {
in, out := &in.NodeSelector, &out.NodeSelector
*out = make(map[string]string, len(*in))
for key, val := range *in {
(*out)[key] = val
}
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new IaaDevicePluginSpec.
func (in *IaaDevicePluginSpec) DeepCopy() *IaaDevicePluginSpec {
if in == nil {
return nil
}
out := new(IaaDevicePluginSpec)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *IaaDevicePluginStatus) DeepCopyInto(out *IaaDevicePluginStatus) {
*out = *in
out.ControlledDaemonSet = in.ControlledDaemonSet
if in.NodeNames != nil {
in, out := &in.NodeNames, &out.NodeNames
*out = make([]string, len(*in))
copy(*out, *in)
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new IaaDevicePluginStatus.
func (in *IaaDevicePluginStatus) DeepCopy() *IaaDevicePluginStatus {
if in == nil {
return nil
}
out := new(IaaDevicePluginStatus)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *QatDevicePlugin) DeepCopyInto(out *QatDevicePlugin) {
*out = *in

408
pkg/controllers/iaa/controller.go Normal file
View File

@ -0,0 +1,408 @@
// Copyright 2021 Intel Corporation. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package iaa contains IAA specific reconciliation logic.
package iaa
import (
"context"
"reflect"
"strconv"
"strings"
apps "k8s.io/api/apps/v1"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/client-go/tools/reference"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
devicepluginv1 "github.com/intel/intel-device-plugins-for-kubernetes/pkg/apis/deviceplugin/v1"
"github.com/intel/intel-device-plugins-for-kubernetes/pkg/controllers"
"github.com/pkg/errors"
)
const (
ownerKey = ".metadata.controller.iaa"
appLabel = "intel-iaa-plugin"
inicontainerName = "intel-iaa-initcontainer"
)
// +kubebuilder:rbac:groups=deviceplugin.intel.com,resources=iaadeviceplugins,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=deviceplugin.intel.com,resources=iaadeviceplugins/status,verbs=get;update;patch
// SetupReconciler creates a new reconciler for IaaDevicePlugin objects.
func SetupReconciler(mgr ctrl.Manager, namespace string, withWebhook bool) error {
c := &controller{scheme: mgr.GetScheme(), ns: namespace}
if err := controllers.SetupWithManager(mgr, c, devicepluginv1.GroupVersion.String(), "IaaDevicePlugin", ownerKey); err != nil {
return err
}
if withWebhook {
return (&devicepluginv1.IaaDevicePlugin{}).SetupWebhookWithManager(mgr)
}
return nil
}
type controller struct {
controllers.DefaultServiceAccountFactory
scheme *runtime.Scheme
ns string
}
func (c *controller) CreateEmptyObject() client.Object {
return &devicepluginv1.IaaDevicePlugin{}
}
func (c *controller) Upgrade(ctx context.Context, obj client.Object) bool {
dp := obj.(*devicepluginv1.IaaDevicePlugin)
return controllers.UpgradeImages(&dp.Spec.Image, &dp.Spec.InitImage)
}
func (c *controller) GetTotalObjectCount(ctx context.Context, clnt client.Client) (int, error) {
var list devicepluginv1.IaaDevicePluginList
if err := clnt.List(ctx, &list); err != nil {
return 0, err
}
return len(list.Items), nil
}
func removeInitContainer(ds *apps.DaemonSet, dp *devicepluginv1.IaaDevicePlugin) {
newInitContainers := []v1.Container{}
for _, container := range ds.Spec.Template.Spec.InitContainers {
if container.Name == inicontainerName {
continue
}
newInitContainers = append(newInitContainers, container)
}
ds.Spec.Template.Spec.InitContainers = newInitContainers
newVolumes := []v1.Volume{}
for _, volume := range ds.Spec.Template.Spec.Volumes {
if volume.Name == "intel-iaa-config-volume" || volume.Name == "sys-devices" {
continue
}
newVolumes = append(newVolumes, volume)
}
ds.Spec.Template.Spec.Volumes = newVolumes
}
func addInitContainer(ds *apps.DaemonSet, dp *devicepluginv1.IaaDevicePlugin) {
yes := true
ds.Spec.Template.Spec.InitContainers = append(ds.Spec.Template.Spec.InitContainers, v1.Container{
Image: dp.Spec.InitImage,
ImagePullPolicy: "IfNotPresent",
Name: inicontainerName,
Env: []v1.EnvVar{
{
Name: "NODE_NAME",
ValueFrom: &v1.EnvVarSource{
FieldRef: &v1.ObjectFieldSelector{
FieldPath: "spec.nodeName",
},
},
},
{
Name: "IDXD_DEVICE_TYPE",
Value: "iaa",
},
},
SecurityContext: &v1.SecurityContext{
Privileged: &yes,
},
VolumeMounts: []v1.VolumeMount{
{
Name: "sys-devices",
MountPath: "/sys/devices",
},
},
})
ds.Spec.Template.Spec.Volumes = append(ds.Spec.Template.Spec.Volumes, v1.Volume{
Name: "sys-devices",
VolumeSource: v1.VolumeSource{
HostPath: &v1.HostPathVolumeSource{
Path: "/sys/devices",
},
},
})
if dp.Spec.ProvisioningConfig != "" {
ds.Spec.Template.Spec.Volumes = append(ds.Spec.Template.Spec.Volumes, v1.Volume{
Name: "intel-iaa-config-volume",
VolumeSource: v1.VolumeSource{
ConfigMap: &v1.ConfigMapVolumeSource{
LocalObjectReference: v1.LocalObjectReference{Name: dp.Spec.ProvisioningConfig}},
},
})
for i, initcontainer := range ds.Spec.Template.Spec.InitContainers {
if initcontainer.Name == inicontainerName {
ds.Spec.Template.Spec.InitContainers[i].VolumeMounts = append(ds.Spec.Template.Spec.InitContainers[i].VolumeMounts, v1.VolumeMount{
Name: "intel-iaa-config-volume",
MountPath: "/idxd-init/conf",
})
}
}
}
}
func (c *controller) NewDaemonSet(rawObj client.Object) *apps.DaemonSet {
devicePlugin := rawObj.(*devicepluginv1.IaaDevicePlugin)
var nodeSelector map[string]string
dpNodeSelectorSize := len(devicePlugin.Spec.NodeSelector)
if dpNodeSelectorSize > 0 {
nodeSelector = make(map[string]string, dpNodeSelectorSize+1)
for k, v := range devicePlugin.Spec.NodeSelector {
nodeSelector[k] = v
}
nodeSelector["kubernetes.io/arch"] = "amd64"
} else {
nodeSelector = map[string]string{"kubernetes.io/arch": "amd64"}
}
yes := true
daemonSet := apps.DaemonSet{
ObjectMeta: metav1.ObjectMeta{
//Namespace: devicePlugin.Namespace,
Namespace: metav1.NamespaceSystem,
GenerateName: devicePlugin.Name + "-",
Labels: map[string]string{
"app": appLabel,
},
},
Spec: apps.DaemonSetSpec{
Selector: &metav1.LabelSelector{
MatchLabels: map[string]string{
"app": appLabel,
},
},
Template: v1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: map[string]string{
"app": appLabel,
},
},
Spec: v1.PodSpec{
Containers: []v1.Container{
{
Name: appLabel,
Env: []v1.EnvVar{
{
Name: "NODE_NAME",
ValueFrom: &v1.EnvVarSource{
FieldRef: &v1.ObjectFieldSelector{
FieldPath: "spec.nodeName",
},
},
},
},
Args: getPodArgs(devicePlugin),
Image: devicePlugin.Spec.Image,
ImagePullPolicy: "IfNotPresent",
SecurityContext: &v1.SecurityContext{
ReadOnlyRootFilesystem: &yes,
},
VolumeMounts: []v1.VolumeMount{
{
Name: "devfs",
MountPath: "/dev/iax",
ReadOnly: true,
},
{
Name: "chardevs",
MountPath: "/dev/char",
ReadOnly: true,
},
{
Name: "sysfs",
MountPath: "/sys/bus",
ReadOnly: true,
},
{
Name: "kubeletsockets",
MountPath: "/var/lib/kubelet/device-plugins",
},
},
},
},
NodeSelector: nodeSelector,
Volumes: []v1.Volume{
{
Name: "devfs",
VolumeSource: v1.VolumeSource{
HostPath: &v1.HostPathVolumeSource{
Path: "/dev/iax",
},
},
},
{
Name: "chardevs",
VolumeSource: v1.VolumeSource{
HostPath: &v1.HostPathVolumeSource{
Path: "/dev/char",
},
},
},
{
Name: "sysfs",
VolumeSource: v1.VolumeSource{
HostPath: &v1.HostPathVolumeSource{
Path: "/sys/bus",
},
},
},
{
Name: "kubeletsockets",
VolumeSource: v1.VolumeSource{
HostPath: &v1.HostPathVolumeSource{
Path: "/var/lib/kubelet/device-plugins",
},
},
},
},
},
},
},
}
if devicePlugin.Spec.InitImage != "" {
addInitContainer(&daemonSet, devicePlugin)
}
return &daemonSet
}
func provisioningUpdate(ds *apps.DaemonSet, dp *devicepluginv1.IaaDevicePlugin) bool {
update := false
found := false
for _, container := range ds.Spec.Template.Spec.InitContainers {
if container.Name == "intel-iaa-initcontainer" && container.Image != dp.Spec.InitImage {
found = true
update = true
break
}
}
for _, volume := range ds.Spec.Template.Spec.Volumes {
if volume.Name == "intel-iaa-config-volume" && volume.ConfigMap.Name != dp.Spec.ProvisioningConfig {
update = true
break
}
}
if !found && dp.Spec.InitImage != "" {
update = true
}
return update
}
func (c *controller) UpdateDaemonSet(rawObj client.Object, ds *apps.DaemonSet) (updated bool) {
dp := rawObj.(*devicepluginv1.IaaDevicePlugin)
if ds.Spec.Template.Spec.Containers[0].Image != dp.Spec.Image {
ds.Spec.Template.Spec.Containers[0].Image = dp.Spec.Image
updated = true
}
if provisioningUpdate(ds, dp) {
removeInitContainer(ds, dp)
if dp.Spec.InitImage != "" {
addInitContainer(ds, dp)
}
updated = true
}
if dp.Spec.NodeSelector == nil {
dp.Spec.NodeSelector = map[string]string{"kubernetes.io/arch": "amd64"}
} else {
dp.Spec.NodeSelector["kubernetes.io/arch"] = "amd64"
}
if !reflect.DeepEqual(ds.Spec.Template.Spec.NodeSelector, dp.Spec.NodeSelector) {
ds.Spec.Template.Spec.NodeSelector = dp.Spec.NodeSelector
updated = true
}
newargs := getPodArgs(dp)
if strings.Join(ds.Spec.Template.Spec.Containers[0].Args, " ") != strings.Join(newargs, " ") {
ds.Spec.Template.Spec.Containers[0].Args = newargs
updated = true
}
return updated
}
func (c *controller) UpdateStatus(rawObj client.Object, ds *apps.DaemonSet, nodeNames []string) (updated bool, err error) {
dp := rawObj.(*devicepluginv1.IaaDevicePlugin)
dsRef, err := reference.GetReference(c.scheme, ds)
if err != nil {
return false, errors.Wrap(err, "unable to make reference to controlled daemon set")
}
if dp.Status.ControlledDaemonSet.UID != dsRef.UID {
dp.Status.ControlledDaemonSet = *dsRef
updated = true
}
if dp.Status.DesiredNumberScheduled != ds.Status.DesiredNumberScheduled {
dp.Status.DesiredNumberScheduled = ds.Status.DesiredNumberScheduled
updated = true
}
if dp.Status.NumberReady != ds.Status.NumberReady {
dp.Status.NumberReady = ds.Status.NumberReady
updated = true
}
if strings.Join(dp.Status.NodeNames, ",") != strings.Join(nodeNames, ",") {
dp.Status.NodeNames = nodeNames
updated = true
}
return updated, nil
}
func getPodArgs(gdp *devicepluginv1.IaaDevicePlugin) []string {
args := make([]string, 0, 4)
args = append(args, "-v", strconv.Itoa(gdp.Spec.LogLevel))
if gdp.Spec.SharedDevNum > 0 {
args = append(args, "-shared-dev-num", strconv.Itoa(gdp.Spec.SharedDevNum))
} else {
args = append(args, "-shared-dev-num", "1")
}
return args
}

97
test/envtest/iaadeviceplugin_controller_test.go Normal file
View File

@ -0,0 +1,97 @@
// Copyright 2021-2022 Intel Corporation. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package envtest
import (
"context"
"time"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
devicepluginv1 "github.com/intel/intel-device-plugins-for-kubernetes/pkg/apis/deviceplugin/v1"
)
var _ = Describe("IaaDevicePlugin Controller", func() {
const timeout = time.Second * 30
const interval = time.Second * 1
Context("Basic CRUD operations", func() {
It("should handle IaaDevicePlugin objects correctly", func() {
spec := devicepluginv1.IaaDevicePluginSpec{
Image: "testimage",
}
key := types.NamespacedName{
Name: "iaadeviceplugin-test",
Namespace: ns,
}
toCreate := &devicepluginv1.IaaDevicePlugin{
ObjectMeta: metav1.ObjectMeta{
Name: key.Name,
Namespace: ns,
},
Spec: spec,
}
By("creating IaaDevicePlugin successfully")
Expect(k8sClient.Create(context.Background(), toCreate)).Should(Succeed())
time.Sleep(time.Second * 5)
fetched := &devicepluginv1.IaaDevicePlugin{}
Eventually(func() bool {
_ = k8sClient.Get(context.Background(), key, fetched)
return len(fetched.Status.ControlledDaemonSet.UID) > 0
}, timeout, interval).Should(BeTrue())
By("updating image name successfully")
updatedImage := "updated-iaa-testimage"
fetched.Spec.Image = updatedImage
Expect(k8sClient.Update(context.Background(), fetched)).Should(Succeed())
fetchedUpdated := &devicepluginv1.IaaDevicePlugin{}
Eventually(func() string {
_ = k8sClient.Get(context.Background(), key, fetchedUpdated)
return fetchedUpdated.Spec.Image
}, timeout, interval).Should(Equal(updatedImage))
By("deleting IaaDevicePlugin successfully")
Eventually(func() error {
f := &devicepluginv1.IaaDevicePlugin{}
_ = k8sClient.Get(context.Background(), key, f)
return k8sClient.Delete(context.Background(), f)
}, timeout, interval).Should(Succeed())
Eventually(func() error {
f := &devicepluginv1.IaaDevicePlugin{}
return k8sClient.Get(context.Background(), key, f)
}, timeout, interval).ShouldNot(Succeed())
})
})
It("upgrades", func() {
dp := &devicepluginv1.IaaDevicePlugin{}
var image string
testUpgrade("iaa", dp, &image, nil)
Expect(dp.Spec.Image == image).To(BeTrue())
})
})

13
test/envtest/suite_test.go
View File

@ -43,6 +43,7 @@ import (
dsactr "github.com/intel/intel-device-plugins-for-kubernetes/pkg/controllers/dsa"
fpgactr "github.com/intel/intel-device-plugins-for-kubernetes/pkg/controllers/fpga"
gpuctr "github.com/intel/intel-device-plugins-for-kubernetes/pkg/controllers/gpu"
iaactr "github.com/intel/intel-device-plugins-for-kubernetes/pkg/controllers/iaa"
qatctr "github.com/intel/intel-device-plugins-for-kubernetes/pkg/controllers/qat"
sgxctr "github.com/intel/intel-device-plugins-for-kubernetes/pkg/controllers/sgx"
)
@ -117,6 +118,8 @@ func up() {
Expect(gpuctr.SetupReconciler(k8sManager, ns, !withWebhook)).To(BeNil())
Expect(iaactr.SetupReconciler(k8sManager, ns, !withWebhook)).To(BeNil())
Expect(qatctr.SetupReconciler(k8sManager, ns, !withWebhook)).To(BeNil())
Expect(sgxctr.SetupReconciler(k8sManager, ns, !withWebhook)).To(BeNil())
@ -209,6 +212,13 @@ func makeDevicePlugin(name, image, initimage string) client.Object {
InitImage: initimage,
},
}
case "iaa":
obj = &devicepluginv1.IaaDevicePlugin{
Spec: devicepluginv1.IaaDevicePluginSpec{
Image: image,
InitImage: initimage,
},
}
case "qat":
obj = &devicepluginv1.QatDevicePlugin{
Spec: devicepluginv1.QatDevicePluginSpec{
@ -244,6 +254,9 @@ func makeDaemonSet(name, image, initimage string) *apps.DaemonSet {
ds = deployments.GPUPluginDaemonSet()
case "fpga":
ds = deployments.FPGAPluginDaemonSet()
case "iaa":
ds = deployments.IAAPluginDaemonSet()
initcontainerName = "intel-idxd-config-initcontainer"
case "qat":
ds = deployments.QATPluginDaemonSet()
case "sgx":