fpga: make AFU resource name 63 char long
webhook: drop mode from README
webhook: extend mappings description
webhook: tighten CRD definitions
webhook: drop mapping to non-existing afuId
explicitly state mappings names can be in any format
use consistent terminology across fpga webhook and plugin
DPDK uses /sys/class/uio/uioX/device/[control|resource*] and we
had special mounts for the individual uioX paths. However, it turned
out this wasn't working as expected: host's /sys/class/uio/uioX/device/
was mounted to container's /sys/class/uio and DPDK failed to find
uioX/device/[control|resource*] files. Moreover, workloads requesting
more than one QAT resource, still saw only one path.
While cri-o/containerd give sysfs read-only mounts, DPDK needs
device/config RW. Therefore, we need to mount host /sys/class/uio/uioX
to container /sys/class/uio/uioX for each requested device.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Changed code a little bit to improve test coverage:
* call Scan in test code
* call Scan without hddl socket
* call Scan with 0 SharedDevNum
* move SharedDevNum in newDevicePlugin
* use Ticker instead of Sleep
Signed-off-by: Alek Du <alek.du@intel.com>
Move all the fpga components to using klog for logging
and debug. This includes replacing our homebrew 'fatal()'
with klog.Error().
Modify the deployment files to move from `-debug` to
`-v`, and set their default level to '1' (Info), rather
than full debug mode ('4').
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Add NewDevicePlugin() tests to improve test coverage. This also
contributes to "input validation" (part of #321) that wasn't done
properly before.
Fixes: #325
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Move from fmt to klog for logging and debug.
Also add an extra info level message noting when we find
new devices.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Move the framework, and the qat driver, to use `klog`
for logging and debug.
This has a some noticeable effects:
1) Our default log output gains a bunch of annotation:
From:
QAT device plugin started in 'dpdk' mode
To:
I0312 11:51:02.057728 6053 qat_plugin.go:64] QAT device plugin started in 'dpdk' mode
(there is now a command line option to drop those annotations if
necessary).
2) We gain a bunch of command line parameters from klog for controlling log
levels and output. We go from 5 arguments to 17:
---
Usage of ./cmd/qat_plugin/qat_plugin:
-add_dir_header
If true, adds the file directory to the header
-alsologtostderr
log to standard error as well as files
-debug
enable debug output
-dpdk-driver string
DPDK Device driver for configuring the QAT device (default "vfio-pci")
-kernel-vf-drivers string
Comma separated VF Device Driver of the QuickAssist Devices in the system. Devices supported: DH895xCC,C62x,C3xxx and D15xx (default "dh895xccvf,c6xxvf,c3xxxvf,d15xxvf")
-log_backtrace_at value
when logging hits line file:N, emit a stack trace
-log_dir string
If non-empty, write log files in this directory
-log_file string
If non-empty, use this log file
-log_file_max_size uint
Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
-logtostderr
log to standard error instead of files (default true)
-max-num-devices int
maximum number of QAT devices to be provided to the QuickAssist device plugin (default 32)
-mode string
plugin mode which can be either dpdk (default) or kernel (default "dpdk")
-skip_headers
If true, avoid header prefixes in the log messages
-skip_log_headers
If true, avoid headers when opening log files
-stderrthreshold value
logs at or above this threshold go to stderr (default 2)
-v value
number for the log level verbosity
-vmodule value
comma-separated list of pattern=N settings for file-filtered logging
---
3) Our `-debug` flag is now replaced by the `klog` `-v n` flag.
*NOTE:* This is potentially a minor breaking change. Applying
this debug overlay to any previous (pre-klog edit) images will
cause the container to fail to launch, as it will not recognise
the new `-v` arguments.
We also update the kustomize deployment to move from using
DEBUG env vars to adding a VERBOSITY var that controls both
the log verbosity and now the debug mode enabling.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Kerneldrv checks for available devices based on adf_ctl output.
We only accepted two cases: PFs if IOMMU is off and VFs if IOMMU
is on.
The right check is to only skip PFs if IOMMU is on and allow other
cases. This fixes two scenarios: when run in VMs, we accept VFs
regardless of (v)IOMMU presence.
Moreover, do not hard code domain '0000:' because it is not the
case always.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
This commit drops fpga_plugin dependency to k8s.io/kubernetes which
was used to get GetHostname(). After this change, the plugin node
name can be set using new -node-name parameter. The default value for
is read from NODE_NAME environment variable.
If the node annotation override check fails, we continue with the default
mode parameter and do not exist like we did previously.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
go get'ing does not work due to our k8s.io/kubernetes dependency
so guide users to use git clone to get the code.
Fixes: #290
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Not touching "orchestration programmed". Fixing only instances where
this refers directly to the mode recognized by the webhook-deploy.sh
script.
Signed-off-by: Antti Kervinen <antti.kervinen@intel.com>
crypto-perf instructions were outdated and hand implicit
assumptions about the environment. More specifically:
Clear Linux builds DPDK libraries as shared so for the
compress and crypto test applications to run, the memory and
QAT PMD libraries must be explicitly preloaded using '-d' parameter.
Also, the test-crypto1 and test-compress1 deployments expect the
cluster is configured with CPU Manager's static policy.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Just follow the standard format to fix the vpu plugin readme.
Also added the ubuntu OpenVINO demo job long logs.
Signed-off-by: Alek Du <alek.du@intel.com>
Update the CRI-O webhook README, adding notes about what it is and
does, and that it is normally installed as part of the device
plugin daemonset.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Expand and re-arrange the README. Add some details about what the
plugin and other FPGA components provide.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
If we fail to scan for GPU devices (note, that is potentially
different from not finding any devices during a scan), then
warn on it, and go around the poll loop again. Do not treat
it as a fatal error or we might end up in a re-launch death
deploy loop...
Of course, getting a warning in your logs every 5s could also
be annoying, but is somewhat 'less fatal'.
Fixes: #260Fixes: #230
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Update the QAT README. Add some descriptions. Add information about
the command line and config options.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Add draw.io and their generated PNG files for both
orchestrated and preprogrammed FPGA modes. These will
then be used in the documentation.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
The fpga_tool had no README. Add a basic one.
Desired as we should at least reference the tool from the
fpga_plugins document.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Re-arrange the section order a little (such as putting the use
of the DaemonSet before the sudo hand-deploy), and add a lot more
detail of what to expect, and how to check if the pod has launched
correctly.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Fill out the introduction to the GPU README to give some details around
what the plugin supports and how.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
The default deployment gives rather wide host mounts.
Limited sysfs mount only to the subdirectory the plugin
needs.
Mounted sysfs and dev mounts read-only.
Added notes that FPGA plugin can be run as non-root user.
The default deployment gives rather wide host mounts. We can limit
the mounts only to the subdirectories the plugin needs and mount
them read-only.
Also, add notes that both QAT and GPU plugins can be run as non-root
user.
Fixes: #228
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
In the pods generated automatically by Deployment/ReplicaSets
fields name and namespace might be missing.
We can use information about namespace from request itself.
Initcontainer is now built in main build process, no need to download
anythin special.
Added note about checking OCI hooks configuration parameter in CRI-O
Fixes: #192
- user readable output for fpgainfo/fmeinfo/portinfo commands
- new commands: list, list-fme, list-port
- new -q flag to suppres headers, progress and too verbose messages
- install command will now fail if destination file already exist
- new --force flag: allows overwrite files in install command
- removed development and debug output
Extended fpga plugin to support both in-tree(DFL) and
out-of-tree (OPAE) kernel drivers.
- fpga_crihook: move JSON parsing to separate functions
- decreased cyclomatic complexity of the CRI hook main() function
- increased readability
- increased test coverage
Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
- Migrate to OPAE 1.3.2
- Build all the tools from the source
- ignore files in workspace
- minimal fpga_tool utility to check gbs/aocx file parsing and flashing
- implemented kernel IOCTL based flashing of bitstreams
- add PCI and sysfs functions
We plan to use crypto-perf for simple QAT testing. This commit adds
kustomization to make the deployment easier. The original .yaml is
also moved to deployments/ with some changes.
For instance, it turns out also vfio-pci mode with DPDK needs CAP_SYS_ADMIN
(See PR: #187 which states that only igb_uio would need it).
kustomize is available part of kubectl since kubernetes v1.14.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
-mode kerneldrv comes with no documentation. This patch adds few
notes about it and instructions how to get it build if a user chooses
to have it enabled.
Closes: #197
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
When IOMMU is on in the system, the physical function (PF)
devices cannot be used. This prevented using kerneldrv as it
was only written to work with PFs.
However, QAT bare metal functions can also be used when IOMMU
is enabled. In this case, they must be used via virtual functions
(VF).
This commit makes it possible to use kerneldrv when IOMMU is
on. The added side benefit is we can now slice the same QAT HW
for both "dpdk" and "kernel" usages simultaneously.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
In adf_ctl output, qat_devX is a sequence number that includes both
PF and VF devices:
qat_dev0 - type: c6xx, inst_id: 0, node_id: 1, bsf: 84:00.0, #accel: 5 #engines: 10 state: up
qat_dev1 - type: c6xx, inst_id: 1, node_id: 1, bsf: 85:00.0, #accel: 5 #engines: 10 state: up
qat_dev2 - type: c6xx, inst_id: 2, node_id: 1, bsf: 86:00.0, #accel: 5 #engines: 10 state: up
qat_dev3 - type: c6xxvf, inst_id: 0, node_id: 1, bsf: 84:01.0, #accel: 1 #engines: 1 state: up
qat_dev4 - type: c6xxvf, inst_id: 1, node_id: 1, bsf: 84:01.1, #accel: 1 #engines: 1 state: up
...
X cannot be used as the config file identified because it does not match
the real id of the device. inst_id gives this so move to use that to find
the right config file.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Let a user know the plugin can't find any Intel GPU if that's
the case. It might be cumbersome to realize that the plugin runs
on a host which doesn't have any Intel GPUs.
Also make the code less nested for better readability.
This commit adds the possibility to qat2_plugin use pci,
devices with communication chipset 8925 to 8955.
Signed-off-by: Rivera Gonzalez, Julio C <julio.c.rivera.gonzalez@intel.com>
CRDs for AF or Region mappings are scoped to namespaces. So an
admitted pod has to be mutated with CRDs existing in the same
namespace as the pod's.
Closes#167
Not all QAT chips (e.g, 37c9) are available in pci.ids which makes
"grep QAT" to not show them.
Scan all known VF PCI ids in a loop to ensure all configured devices
are shown.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
For easier deployments, fetch plugin command line arguments from ConfigMap.
When using ConfigMaps, qat_plugin.yaml needs no changes and can always
be used as is.
qat_plugin_default_configmap.yaml uses built-in defaults.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Different versions of OPAE tool fpgaconf use different command line
options to specify socket number. Latest version uses -S, which CRI-O
hook used so far.
However, fpgaconf in initcontainer comes from older OPAE version, which
uses -s option for the same purpose. This caused fpgaconf and CRI-O hook
to fail with "Invalid cmdline options".
Used 'fpgaconf-wrapper -s' in the CRI-O hook to fix this.
Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
intel-fpga-initcontainer installs OPAE and OpenCL tools
into /opt/intel/fpga-sw directory.
Used fpgaconf-wrapper, aocl-wrapper and packager tools by
CRI hook to program OPAE and OpenCL bitstreams.
Added support for OpenCL bitstreams: the hook tries to find
either OPAE or OpenCL bitstream suitable for requested region
and AFU and programs it using OPAE or OpenCL tools.
Fixes#71
- To ensure that the device plugin continues to run in case a certain QAT device family is not detected on the node
- Updated the extended resource name to qat/generic in demo/crypto-perf-dpdk-pod-requesting-qat.yaml file
- Added explicit check for QAT device ID to avoid counting other DPDK bound devices as QAT devices
The DRM driver of Intel i915 GPUs allows sharing one device
between many containers.
Make it possible to use the same device from different containers.
The exact number of containers sharing the same device can be limited
with the new option -shared-dev-num set to 1 by default.
closes#53
When the device's firmware crashes the OPAE driver reports all properties
of the device as a stream of binary ones. This effectively makes
interface and afu IDs look like "ffffffffffffffffffffffffffffffff".
Mark such devices as Unhealthy.
closes#77
Currently we have hardcoded mapping from human readable names of
AFs and FPGA regions like arria10-nlb0 to the resource names
produced by the FPGA device plugin. This is not sustainable
long term solution.
Implement CRD based mappings so that a new mapping can be added or
removed dynamically by cluster admins with CRD resources.
Every device plugin is supposed to implement PluginInterfaceServer
interface to be exposed as a gRPC service. But this functionality is
common for all our device plugins and can be hidden in a Manager
which manages all gRPC servers dynamically.
The only mandatory functionality that needs to be provided by a device
plugin and which differentiate one plugin from another is the code
scanning the host for devices present on it.
Refactor the internal deviceplugin package to accept only
one mandatory method implementation from device plugins - Scan().
In addition to that a device plugin can optionally implement a
PostAllocate() method which mutates responses returned by
PluginInterfaceServer.Allocate() method.
Also to narrow the gap between these device plugins and the
kubevirt's collection the naming scheme for resources has been changed.
Now device plugins provide a namespace for the device types they
operate with. E.g. for resources in format "color.example.com/<color>"
the namespace would be "color.example.com". So, the resource name
"intel.com/fpga-region-fffffff" becomes "fpga.intel.com/region-fffffff".
DRM control node is deprecated and removed by latest kernel.
This will skip possible drm control node found on host.
v2: Fix lint error
v3: Fix regex string
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
Check if container annotation com.intel.fpga.mode is set to
"intel.com/fpga-region". This annotation is set by device plugin.
So, the check should help to filter out unwanted workflow that
device plugin is not aware of.
When kubelet notifies the plugin about its restart by removing
the plugin's socket we do reconnect to kubelet, but we don't
send the current list of monitored devices to kubelet. As result
kubelet is not aware of discovered devices if it restarts.
Always send the current list of monitored devices to kubelet
upon ListAndWatch() request.
The hook gets FPGA_REGION and FPGA_BITSTREAM environment variables
defined in a pod spec, finds bitstream file, verifies it and programs
FPGA device with it using fpga-configure tool from OPAE.
When running in the region mode we don't need to know AFU IDs
thus don't read them while in the mode.
It's important not to try to read them because in the region mode
AFUs are supposed to be reprogrammed in the fly anyway and the
afu_id files may become busy.
Device descovery can get EBUSY error when AFU is being programmed.
It causes plugin to crash with error:
Device scan failed: read /sys/class/fpga/intel-fpga-dev.0/intel-fpga-port.0/afu_id:
device or resource busy
This error should be ignored as this is valid use case.
This is harmless as afu will be rediscovered on the next run, when
reprogramming is done.
Moved code that goes through sysfs to the separate function
getSysFsInfo to decrease cyclomatic complexity of the scanFPGAs
function.
This is required to get the next commit through our CI check.
Plugin sets container annotation com.intel.fpga.mode to
intel.com/fpga-region in region mode.
This should allow to configure CRI-O to run reprogramming hooks
only when annotation is set.
In `orchestrated` mode the webhook parses requested resources and to translates
them to a container's ENV variables to let CRI hooks to program the FPGA with
requested bitstreams before starting the container.
In the `af` mode the plugin announces AFUs and tells kubelet
to pass only AFU ports to containers.
In the `region` mode the plugin announces region interfaces and tells
kubelet to pass only AFU ports to containers.
In the `regiondevel` mode the plugin announces region interfaces and
tells kubelet to pass AFU ports and FME devices to containers, so the
conteainers have full access to the regions.
This refactoring brings in device Cache running in its own
thread and scanning FPGA devices once every 5 secs. Then no timers
are used inside ListAndWatch() method of device managers and
no need to run scanning periodically in every device manager's
thread.
Cache generates update events and the plugin creates, updates or
deletes device managers on the fly upon receiving the events.
Introducing new modes of operations is a matter of adding a single
function converting and filtering the content of Cache.
This is done to fix https://goreportcard.com warnnigs:
gofmt 33%
Gofmt formats Go programs. We run gofmt -s on your code, where -s is for the "simplify" command
intel-device-plugins-for-kubernetes/cmd/fpga_plugin/fpga_plugin_test.go
Line 1: warning: file is not gofmted with -s (gofmt)
intel-device-plugins-for-kubernetes/internal/deviceplugin/deviceplugin_test.go
Line 1: warning: file is not gofmted with -s (gofmt)
intel-device-plugins-for-kubernetes/cmd/gpu_plugin/gpu_plugin_test.go
Line 1: warning: file is not gofmted with -s (gofmt)
intel-device-plugins-for-kubernetes/cmd/fpga_plugin/fpga_plugin.go
Line 1: warning: file is not gofmted with -s (gofmt)
Fixed the following golint warnings:
./cmd/fpga_plugin/fpga_plugin.go:71:2: struct field fpgaId should be fpgaID
./cmd/fpga_plugin/fpga_plugin.go:78:44: func parameter fpgaId should be fpgaID
./cmd/fpga_plugin/fpga_plugin.go:104:8: var interfaceId should be interfaceID
./cmd/fpga_plugin/fpga_plugin.go:120:7: var interfaceIdFile should be interfaceIDFile
./cmd/fpga_plugin/fpga_plugin.go:156:8: range var fpgaId should be fpgaID
./cmd/fpga_plugin/fpga_plugin.go:254:6: range var fpgaId should be fpgaID
./cmd/fpga_plugin/fpga_plugin.go:254:14: should omit 2nd value from range; this loop is equivalent to `for fpgaId := range ...`
./internal/deviceplugin/deviceplugin.go:30:6: exported type DeviceInfo should have comment or be unexported
./internal/deviceplugin/deviceplugin.go:35:6: exported type Server should have comment or be unexported
./internal/deviceplugin/deviceplugin.go:39:1: exported method Server.Serve should have comment or be unexported
./internal/deviceplugin/deviceplugin.go:43:1: exported method Server.Stop should have comment or be unexported
Added mode ("af" or "region") prefix to the resource name to
distingush between announced functions and regions, e.g.
intel.com/fpga-af-f7df405cbd7acf7222f144b0b93acd18
intel.com/fpga-region-ce48969398f05f33946d560708be108a
Split into 3 parts:
- main part with high level description of the repository
- Build and test Intel GPU Device Plugin for Kubernetes
- Build and test Intel FPGA Device Plugin for Kubernetes
Added Intel logo to the main README.md
Fixes#2
By default the fpga plugin announce regions' interface IDs. With
added `-mode af` switch the plugins announces IDs of accelerator
functions instead of regions.
