setupDeviceIDs() is obsoleted and the preferred approach is driver_override
already implemented in qat-init.sh initcontainer.
The new_id mechanism was added way before we had the initcontainer support in place.
Furthermore, at least for vfio-pci we don't need it at all if the driver uses
ids=8086:<qat VF dev IDs>.
Drop write attemps to new_id in favor of the initcontainer functionality.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
QAT device plugin has some initialization functions that require
special SecurityContext parameters (e.g., setting Apparmor policies
on some OSes).
It's better to move all of the initialization to the privileged
init-container that is already taking care some parts of it.
With this change, we default to vfio-pci "DpdkDrv".
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Plugin used to consider only the value "-1" but there are some
cases when files show "\n" or "\n\x00". This makes plugin to have
wrong status of the device. So, trim the value after \n so only
numerical value can be read.
Signed-off-by: Hyeongju Johannes Lee <hyeongju.lee@intel.com>
currently, the QAT plugin warns when it finds a PCI ID that is
not an enabled QAT device. This is too verbose so lower the
log priority to "Info".
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Linux 6.0 adds sysfs-driver-qat entries to read device capabilities:
42e66b1cc3/Documentation/ABI/testing/sysfs-driver-qat
Implement the logic for reading from sysfs and prefer that over debugfs.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
QAT_401xx is a derivative of 4xxx. Add support for that device
by including the device IDs (both PF and VF).
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
If the kernel has CONFIG_VFIO_NOIOMMU enabled and the node admin
has explicitly set enable_unsafe_noiommu_mode VFIO parameter,
VFIO taints the kernel and writes "vfio-noiommu" to the IOMMU
group name. If these conditions are true, the /dev/vfio/ devices
are prefixed with "noiommu-".
This use-case is documented for DPDK so we don't want to break
it (as it was before because we added DeviceMounts to
/dev/vfio/<iommugroup> files that did not exist).
See DPDK documentation for further information and warnings.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
The new_id based driver binding is failing on kernels 5.11+ when the
QAT VF is not bound to any driver: attempts to write to new_id with
the same device ID repeatedly error with "file exists".
Move the new_id initialization to the beginning of the startup and
write the enabled device IDs only once.
This commit also fixes an issue where VF devices where not correctly detected
in virtual machines where the VF was not bound any driver.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
After a closer review, it was noticed that some of the QAT dpdkdrv
unit tests need updating:
- "Broken igb_uio DPDKdriver..." is actually testing unknown device ID
and we already have tests for it -> drop.
- "igb_uio DPDKdriver with one kernel bound device (not QAT device)" is
testing something impossible: an unknown VF devID is originated from a
QAT PF -> drop.
- creating files for unbind/new_id etc. is unnecessary because
os.WriteFile() creates them during the tests -> drop these lines to
simplify unit tests maintenance.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Add govet-fieldalignment to .golangci.yml
Fix errors that come from adding govet-fieldalignment
- by reordering the fields of structs
- by putting nolint:govet annotations
Signed-off-by: Hyeongju Johannes Lee <hyeongju.lee@intel.com>
<device>/driver symlink does not exist if the device is not bound
to any driver. bindDevice() failed when writing to <device>/driver/unbind
errored but IsNotExist() error is acceptable in case there's no driver
to unbind.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Go 1.16 release notes announced the deprecation of io/ioutil [1]. It's easy
for us to move to use what is was recommended so just do it.
[1] https://golang.org/doc/go1.16#ioutil
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
It looks that for a long time now we have accepted a setup where a valid QAT
device ID is accepted as a QAT device resource even though the device is
not "enabled" via kernelVfDrivers parameter.
Fix device ID validation to skip valid QAT devices that are not
explicitly specified in kernelVfDrivers.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
The updated dp.scan() changes the way how VF devices are detected. The
main reason for the change is to take into account cases where the QAT VF
driver is not present in the system at all but only the PF driver is
loaded (and the SR-IOV devices are are enabled).
The rework also takes into account bare metal and VM deployments and
adds a test case for checking the virtualized environment.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
The plugin now detects/accepts 4xxx and c4xxx devices too
and defaults to those drivers that are part of Linux mainline.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
We have both "path" and "path/filepath" but the latter provides
everything needed so move it completely.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
The code was stripping out "0000:" (bus) and then adding
it back in several places.
That's not necessary so this change simplifies QAT VF addr
handling by operating using full BDF IDs.
Moveover, simplify function calls: use getDpdkDevice() once
for each VF device.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
DPDK uses /sys/class/uio/uioX/device/[control|resource*] and we
had special mounts for the individual uioX paths. However, it turned
out this wasn't working as expected: host's /sys/class/uio/uioX/device/
was mounted to container's /sys/class/uio and DPDK failed to find
uioX/device/[control|resource*] files. Moreover, workloads requesting
more than one QAT resource, still saw only one path.
While cri-o/containerd give sysfs read-only mounts, DPDK needs
device/config RW. Therefore, we need to mount host /sys/class/uio/uioX
to container /sys/class/uio/uioX for each requested device.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Add NewDevicePlugin() tests to improve test coverage. This also
contributes to "input validation" (part of #321) that wasn't done
properly before.
Fixes: #325
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Move the framework, and the qat driver, to use `klog`
for logging and debug.
This has a some noticeable effects:
1) Our default log output gains a bunch of annotation:
From:
QAT device plugin started in 'dpdk' mode
To:
I0312 11:51:02.057728 6053 qat_plugin.go:64] QAT device plugin started in 'dpdk' mode
(there is now a command line option to drop those annotations if
necessary).
2) We gain a bunch of command line parameters from klog for controlling log
levels and output. We go from 5 arguments to 17:
---
Usage of ./cmd/qat_plugin/qat_plugin:
-add_dir_header
If true, adds the file directory to the header
-alsologtostderr
log to standard error as well as files
-debug
enable debug output
-dpdk-driver string
DPDK Device driver for configuring the QAT device (default "vfio-pci")
-kernel-vf-drivers string
Comma separated VF Device Driver of the QuickAssist Devices in the system. Devices supported: DH895xCC,C62x,C3xxx and D15xx (default "dh895xccvf,c6xxvf,c3xxxvf,d15xxvf")
-log_backtrace_at value
when logging hits line file:N, emit a stack trace
-log_dir string
If non-empty, write log files in this directory
-log_file string
If non-empty, use this log file
-log_file_max_size uint
Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
-logtostderr
log to standard error instead of files (default true)
-max-num-devices int
maximum number of QAT devices to be provided to the QuickAssist device plugin (default 32)
-mode string
plugin mode which can be either dpdk (default) or kernel (default "dpdk")
-skip_headers
If true, avoid header prefixes in the log messages
-skip_log_headers
If true, avoid headers when opening log files
-stderrthreshold value
logs at or above this threshold go to stderr (default 2)
-v value
number for the log level verbosity
-vmodule value
comma-separated list of pattern=N settings for file-filtered logging
---
3) Our `-debug` flag is now replaced by the `klog` `-v n` flag.
*NOTE:* This is potentially a minor breaking change. Applying
this debug overlay to any previous (pre-klog edit) images will
cause the container to fail to launch, as it will not recognise
the new `-v` arguments.
We also update the kustomize deployment to move from using
DEBUG env vars to adding a VERBOSITY var that controls both
the log verbosity and now the debug mode enabling.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
