The code was stripping out "0000:" (bus) and then adding
it back in several places.
That's not necessary so this change simplifies QAT VF addr
handling by operating using full BDF IDs.
Moveover, simplify function calls: use getDpdkDevice() once
for each VF device.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
DPDK uses /sys/class/uio/uioX/device/[control|resource*] and we
had special mounts for the individual uioX paths. However, it turned
out this wasn't working as expected: host's /sys/class/uio/uioX/device/
was mounted to container's /sys/class/uio and DPDK failed to find
uioX/device/[control|resource*] files. Moreover, workloads requesting
more than one QAT resource, still saw only one path.
While cri-o/containerd give sysfs read-only mounts, DPDK needs
device/config RW. Therefore, we need to mount host /sys/class/uio/uioX
to container /sys/class/uio/uioX for each requested device.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Add NewDevicePlugin() tests to improve test coverage. This also
contributes to "input validation" (part of #321) that wasn't done
properly before.
Fixes: #325
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Move the framework, and the qat driver, to use `klog`
for logging and debug.
This has a some noticeable effects:
1) Our default log output gains a bunch of annotation:
From:
QAT device plugin started in 'dpdk' mode
To:
I0312 11:51:02.057728 6053 qat_plugin.go:64] QAT device plugin started in 'dpdk' mode
(there is now a command line option to drop those annotations if
necessary).
2) We gain a bunch of command line parameters from klog for controlling log
levels and output. We go from 5 arguments to 17:
---
Usage of ./cmd/qat_plugin/qat_plugin:
-add_dir_header
If true, adds the file directory to the header
-alsologtostderr
log to standard error as well as files
-debug
enable debug output
-dpdk-driver string
DPDK Device driver for configuring the QAT device (default "vfio-pci")
-kernel-vf-drivers string
Comma separated VF Device Driver of the QuickAssist Devices in the system. Devices supported: DH895xCC,C62x,C3xxx and D15xx (default "dh895xccvf,c6xxvf,c3xxxvf,d15xxvf")
-log_backtrace_at value
when logging hits line file:N, emit a stack trace
-log_dir string
If non-empty, write log files in this directory
-log_file string
If non-empty, use this log file
-log_file_max_size uint
Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
-logtostderr
log to standard error instead of files (default true)
-max-num-devices int
maximum number of QAT devices to be provided to the QuickAssist device plugin (default 32)
-mode string
plugin mode which can be either dpdk (default) or kernel (default "dpdk")
-skip_headers
If true, avoid header prefixes in the log messages
-skip_log_headers
If true, avoid headers when opening log files
-stderrthreshold value
logs at or above this threshold go to stderr (default 2)
-v value
number for the log level verbosity
-vmodule value
comma-separated list of pattern=N settings for file-filtered logging
---
3) Our `-debug` flag is now replaced by the `klog` `-v n` flag.
*NOTE:* This is potentially a minor breaking change. Applying
this debug overlay to any previous (pre-klog edit) images will
cause the container to fail to launch, as it will not recognise
the new `-v` arguments.
We also update the kustomize deployment to move from using
DEBUG env vars to adding a VERBOSITY var that controls both
the log verbosity and now the debug mode enabling.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Kerneldrv checks for available devices based on adf_ctl output.
We only accepted two cases: PFs if IOMMU is off and VFs if IOMMU
is on.
The right check is to only skip PFs if IOMMU is on and allow other
cases. This fixes two scenarios: when run in VMs, we accept VFs
regardless of (v)IOMMU presence.
Moreover, do not hard code domain '0000:' because it is not the
case always.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
go get'ing does not work due to our k8s.io/kubernetes dependency
so guide users to use git clone to get the code.
Fixes: #290
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
crypto-perf instructions were outdated and hand implicit
assumptions about the environment. More specifically:
Clear Linux builds DPDK libraries as shared so for the
compress and crypto test applications to run, the memory and
QAT PMD libraries must be explicitly preloaded using '-d' parameter.
Also, the test-crypto1 and test-compress1 deployments expect the
cluster is configured with CPU Manager's static policy.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Update the QAT README. Add some descriptions. Add information about
the command line and config options.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
The default deployment gives rather wide host mounts. We can limit
the mounts only to the subdirectories the plugin needs and mount
them read-only.
Also, add notes that both QAT and GPU plugins can be run as non-root
user.
Fixes: #228
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
We plan to use crypto-perf for simple QAT testing. This commit adds
kustomization to make the deployment easier. The original .yaml is
also moved to deployments/ with some changes.
For instance, it turns out also vfio-pci mode with DPDK needs CAP_SYS_ADMIN
(See PR: #187 which states that only igb_uio would need it).
kustomize is available part of kubectl since kubernetes v1.14.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
-mode kerneldrv comes with no documentation. This patch adds few
notes about it and instructions how to get it build if a user chooses
to have it enabled.
Closes: #197
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
When IOMMU is on in the system, the physical function (PF)
devices cannot be used. This prevented using kerneldrv as it
was only written to work with PFs.
However, QAT bare metal functions can also be used when IOMMU
is enabled. In this case, they must be used via virtual functions
(VF).
This commit makes it possible to use kerneldrv when IOMMU is
on. The added side benefit is we can now slice the same QAT HW
for both "dpdk" and "kernel" usages simultaneously.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
In adf_ctl output, qat_devX is a sequence number that includes both
PF and VF devices:
qat_dev0 - type: c6xx, inst_id: 0, node_id: 1, bsf: 84:00.0, #accel: 5 #engines: 10 state: up
qat_dev1 - type: c6xx, inst_id: 1, node_id: 1, bsf: 85:00.0, #accel: 5 #engines: 10 state: up
qat_dev2 - type: c6xx, inst_id: 2, node_id: 1, bsf: 86:00.0, #accel: 5 #engines: 10 state: up
qat_dev3 - type: c6xxvf, inst_id: 0, node_id: 1, bsf: 84:01.0, #accel: 1 #engines: 1 state: up
qat_dev4 - type: c6xxvf, inst_id: 1, node_id: 1, bsf: 84:01.1, #accel: 1 #engines: 1 state: up
...
X cannot be used as the config file identified because it does not match
the real id of the device. inst_id gives this so move to use that to find
the right config file.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Not all QAT chips (e.g, 37c9) are available in pci.ids which makes
"grep QAT" to not show them.
Scan all known VF PCI ids in a loop to ensure all configured devices
are shown.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
For easier deployments, fetch plugin command line arguments from ConfigMap.
When using ConfigMaps, qat_plugin.yaml needs no changes and can always
be used as is.
qat_plugin_default_configmap.yaml uses built-in defaults.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
- To ensure that the device plugin continues to run in case a certain QAT device family is not detected on the node
- Updated the extended resource name to qat/generic in demo/crypto-perf-dpdk-pod-requesting-qat.yaml file
- Added explicit check for QAT device ID to avoid counting other DPDK bound devices as QAT devices
