VPU: Add Intel Movidius MyriadX VPU plugin support

This patch is to support Intel VCAC-A card (with MyriadX 2485 VPUs), for other
later on VPUs, we will reuse this plugin and add support.

VCAC-A board info is at:
https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/media-analytics-vcac-a-accelerator-card-by-celestica-datasheet.pdf

Also add openvino HDDL VPU demo for Intel VCAC-A card.

Signed-off-by: Alek Du <alek.du@intel.com>

.travis.yml

@@ -26,7 +26,7 @@ before_install:
   ## install buildah build deps
   - cdir=$(pwd)
   - sudo apt-get update
-  - sudo apt-get -y install e2fslibs-dev libfuse-dev libgpgme11-dev libdevmapper-dev libglib2.0-dev libprotobuf-dev
+  - sudo apt-get -y install e2fslibs-dev libfuse-dev libgpgme11-dev libdevmapper-dev libglib2.0-dev libprotobuf-dev libusb-1.0-0-dev
   # build buildah
   - mkdir -p $GOPATH/src/github.com/containers
   - cd $GOPATH/src/github.com/containers

Jenkinsfile

@@ -52,7 +52,7 @@ pipeline {
             stage("buildah") {
               steps {
                 sh "sudo apt-get update"
-                sh "sudo apt-get -y install e2fslibs-dev libfuse-dev libgpgme11-dev libdevmapper-dev libglib2.0-dev libprotobuf-dev"
+                sh "sudo apt-get -y install e2fslibs-dev libfuse-dev libgpgme11-dev libdevmapper-dev libglib2.0-dev libprotobuf-dev libusb-1.0-0-dev"
                 sh "mkdir -p ${GOPATH}/src/github.com/containers"
                 dir(path: "${GOPATH}/src/github.com/containers") {
                   sh "git clone --single-branch --depth 1 -b $BUILDAH_VERSION https://github.com/containers/buildah"

README.md

@@ -13,6 +13,7 @@
         * [admission controller](#admission-controller)
         * [CRI-O prestart hook](#cri-o-prestart-hook)
     * [QAT device plugin](#qat-device-plugin)
+    * [VPU device plugin](#vpu-device-plugin)
 * [Demos](#demos)
 * [Developers](#developers)
 * [Supported Kubernetes versions](#supported-kubernetes-versions)
@@ -101,6 +102,18 @@ Details for integrating the QAT device plugin into [Kata Containers](https://kat
 can be found in the
 [Kata Containers documentation repository](https://github.com/kata-containers/documentation/blob/master/use-cases/using-Intel-QAT-and-kata.md).
 
+### VPU device plugin
+
+The [VPU device plugin](cmd/vpu_plugin/README.md) supports Intel VCAC-A card
+(https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/media-analytics-vcac-a-accelerator-card-by-celestica-datasheet.pdf)
+the card has:
+- 1 Intel Core i3-7100U processor
+- 12 MyriadX VPUs
+- 8GB DDR4 memory
+
+The demo subdirectory includes details of a OpenVINO deployment and use of the VPU plugin.
+Sources can be found in [openvino-demo](demo/ubuntu-demo-openvino)
+
 ## Demos
 
 The [demo subdirectory](demo/readme.md) contains a number of demonstrations for a variety of the

build/docker/intel-vpu-plugin.Dockerfile

@@ -0,0 +1,37 @@
+# CLEAR_LINUX_BASE and CLEAR_LINUX_VERSION can be used to make the build
+# reproducible by choosing an image by its hash and installing an OS version
+# with --version=:
+# CLEAR_LINUX_BASE=clearlinux@sha256:b8e5d3b2576eb6d868f8d52e401f678c873264d349e469637f98ee2adf7b33d4
+# CLEAR_LINUX_VERSION="--version=29970"
+#
+# This is used on release branches before tagging a stable version.
+# The master branch defaults to using the latest Clear Linux.
+ARG CLEAR_LINUX_BASE=clearlinux/golang:latest
+FROM ${CLEAR_LINUX_BASE} as builder
+ARG CLEAR_LINUX_VERSION=
+
+RUN swupd update --no-boot-update
+RUN swupd bundle-add devpkg-libusb
+RUN mkdir /install_root \
+    && swupd os-install \
+    ${CLEAR_LINUX_VERSION} \
+    --path /install_root \
+    --statedir /swupd-state \
+    --bundles=os-core \
+    --no-boot-update \
+    && rm -rf /install_root/var/lib/swupd/*
+
+ARG DIR=/intel-device-plugins-for-kubernetes
+WORKDIR $DIR
+COPY . .
+RUN cd cmd/vpu_plugin; go install
+RUN chmod a+x /go/bin/vpu_plugin \
+    && install -D /go/bin/vpu_plugin /install_root/usr/local/bin/intel_vpu_device_plugin \
+    && install -D ${DIR}/LICENSE /install_root/usr/local/share/package-licenses/intel-device-plugins-for-kubernetes/LICENSE \
+    && scripts/copy-modules-licenses.sh ./cmd/vpu_plugin /install_root/usr/local/share/package-licenses/ \
+    && install -D /usr/share/package-licenses/libusb/COPYING -t /install_root/usr/local/share/package-licenses/libusb \
+    && install -D /lib64/libusb-1.0.so.0 /install_root/lib64
+
+FROM scratch as final
+COPY --from=builder /install_root /
+ENTRYPOINT ["/usr/local/bin/intel_vpu_device_plugin"]

cmd/vpu_plugin/README.md

@@ -0,0 +1,75 @@
+# Build and test Intel VPU device plugin for Kubernetes
+
+### Get source code:
+```
+$ mkdir -p $GOPATH/src/github.com/intel/
+$ cd $GOPATH/src/github.com/intel/
+$ git clone https://github.com/intel/intel-device-plugins-for-kubernetes.git
+```
+Note: to get VCAC-A card running hddl, please refer to:
+https://github.com/OpenVisualCloud/Dockerfiles/blob/master/VCAC-A/script/setup_hddl.sh
+
+### Verify kubelet socket exists in /var/lib/kubelet/device-plugins/ directory:
+```
+$ ls /var/lib/kubelet/device-plugins/kubelet.sock
+/var/lib/kubelet/device-plugins/kubelet.sock
+```
+
+### Deploy VPU device plugin as host process for development purposes
+
+#### Build VPU device plugin:
+```
+$ cd $GOPATH/src/github.com/intel/intel-device-plugins-for-kubernetes
+$ make vpu_plugin
+```
+
+#### Run VPU device plugin as administrator:
+```
+$ sudo $GOPATH/src/github.com/intel/intel-device-plugins-for-kubernetes/cmd/vpu_plugin/vpu_plugin
+device-plugin start server at: /var/lib/kubelet/device-plugins/vpu.intel.com-hddl.sock
+device-plugin registered
+```
+
+### Deploy VPU device plugin as a DaemonSet:
+
+#### Build plugin image
+```
+$ make intel-vpu-plugin
+```
+
+#### Create plugin DaemonSet
+```
+$ kubectl create -f ./deployments/vpu_plugin/vpu_plugin.yaml
+daemonset.apps/intel-vpu-plugin created
+```
+**Note**: It is also possible to run the VPU device plugin using a non-root user. To do this,
+the nodes' DAC rules must be configured to allow USB device descriptor detection, device plugin
+socket creation and kubelet registration. Furthermore, the deployments `securityContext` must
+be configured with appropriate `runAsUser/runAsGroup`.
+
+### Verify VPU device plugin is registered on master:
+```
+$ kubectl describe node <node name> | grep vpu.intel.com
+ vpu.intel.com/hddl:  1
+ vpu.intel.com/hddl:  1
+```
+
+### Test VPU device plugin:
+
+1. Build a Docker image with an example openvino to VPU:
+   ```
+   $ cd demo
+   $ ./build-image.sh ubuntu-demo-openvino
+   ```
+
+      This command produces a Docker image named `ubuntu-demo-openvino`.
+
+2. Create a pod running unit tests off the local Docker image:
+   ```
+   $ kubectl apply -f demo/intelvpu-job.yaml
+   ```
+
+3. Review the pod's logs:
+   ```
+   $ kubectl logs intelvpu-demo-job-xxxx
+   ```

cmd/vpu_plugin/vpu_plugin.go

@@ -0,0 +1,185 @@
+// Copyright 2020 Intel Corporation. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+	"flag"
+	"fmt"
+	"github.com/google/gousb"
+	"os"
+	"time"
+
+	"github.com/intel/intel-device-plugins-for-kubernetes/pkg/debug"
+	dpapi "github.com/intel/intel-device-plugins-for-kubernetes/pkg/deviceplugin"
+	pluginapi "k8s.io/kubelet/pkg/apis/deviceplugin/v1beta1"
+)
+
+const (
+	// Movidius MyriadX Vendor ID
+	vendorID = 0x03e7
+	// Device plugin settings.
+	namespace  = "vpu.intel.com"
+	deviceType = "hddl"
+
+	hddlSockPath     = "/var/tmp/hddl_service.sock"
+	hddlServicePath1 = "/var/tmp/hddl_service_ready.mutex"
+	hddlServicePath2 = "/var/tmp/hddl_service_alive.mutex"
+	ionDevNode       = "/dev/ion"
+)
+
+var (
+	isdebug = flag.Int("debug", 0, "debug level (0..1)")
+	// Movidius MyriadX Product IDs
+	productIDs = []int{0x2485, 0xf63b}
+)
+
+type gousbContext interface {
+	OpenDevices(opener func(desc *gousb.DeviceDesc) bool) ([]*gousb.Device, error)
+}
+
+type devicePlugin struct {
+	usbContext   gousbContext
+	vendorID     int
+	productIDs   []int
+	sharedDevNum int
+}
+
+func newDevicePlugin(usbContext gousbContext, vendorID int, productIDs []int, sharedDevNum int) *devicePlugin {
+	return &devicePlugin{
+		usbContext:   usbContext,
+		vendorID:     vendorID,
+		productIDs:   productIDs,
+		sharedDevNum: sharedDevNum,
+	}
+}
+
+func (dp *devicePlugin) Scan(notifier dpapi.Notifier) error {
+	for {
+		devTree, err := dp.scan()
+		if err != nil {
+			return err
+		}
+
+		notifier.Notify(devTree)
+
+		time.Sleep(5 * time.Second)
+	}
+}
+
+func fileExists(filename string) bool {
+	info, err := os.Stat(filename)
+	if err == nil && info != nil {
+		return !info.IsDir()
+	}
+	// regard all other case as abnormal
+	return false
+}
+
+func (dp *devicePlugin) scan() (dpapi.DeviceTree, error) {
+	var nUsb int
+	devTree := dpapi.NewDeviceTree()
+
+	// first check if HDDL sock is there
+	if !fileExists(hddlSockPath) {
+		return devTree, nil
+	}
+
+	devs, err := dp.usbContext.OpenDevices(func(desc *gousb.DeviceDesc) bool {
+		thisVendor := desc.Vendor
+		thisProduct := desc.Product
+		for _, v := range dp.productIDs {
+			debug.Printf("checking %04x,%04x vs %s,%s", dp.vendorID, v, thisVendor.String(), thisProduct.String())
+			if (gousb.ID(dp.vendorID) == thisVendor) && (gousb.ID(v) == thisProduct) {
+				nUsb++
+			}
+		}
+		return false
+	})
+	defer func() {
+		for _, d := range devs {
+			d.Close()
+		}
+	}()
+
+	if err != nil {
+		debug.Printf("list usb device %s", err)
+	}
+
+	if nUsb > 0 {
+		for i := 0; i < nUsb*dp.sharedDevNum; i++ {
+			devID := fmt.Sprintf("hddl_service-%d", i)
+			// HDDL use a unix socket as service provider to manage /dev/myriad[n]
+			// Here we only expose an ION device to be allocated for HDDL client in containers
+			nodes := []pluginapi.DeviceSpec{
+				{
+					HostPath:      ionDevNode,
+					ContainerPath: ionDevNode,
+					Permissions:   "rw",
+				},
+			}
+
+			mounts := []pluginapi.Mount{
+				{
+					HostPath:      hddlSockPath,
+					ContainerPath: hddlSockPath,
+				},
+				{
+					HostPath:      hddlServicePath1,
+					ContainerPath: hddlServicePath1,
+				},
+				{
+					HostPath:      hddlServicePath2,
+					ContainerPath: hddlServicePath2,
+				},
+			}
+			devTree.AddDevice(deviceType, devID, dpapi.DeviceInfo{
+				State:  pluginapi.Healthy,
+				Nodes:  nodes,
+				Mounts: mounts,
+			})
+		}
+	}
+
+	return devTree, nil
+}
+
+func main() {
+	var sharedDevNum int
+
+	flag.IntVar(&sharedDevNum, "shared-dev-num", 1, "number of containers sharing the same VPU device")
+
+	flag.Parse()
+
+	if *isdebug > 0 {
+		debug.Activate()
+		debug.Printf("isdebug is on")
+	}
+
+	if sharedDevNum < 1 {
+		fmt.Println("The number of containers sharing the same VPU must greater than zero")
+		os.Exit(1)
+	}
+
+	fmt.Println("VPU device plugin started")
+
+	// add lsusb here
+	ctx := gousb.NewContext()
+	defer ctx.Close()
+	ctx.Debug(*isdebug)
+
+	plugin := newDevicePlugin(ctx, vendorID, productIDs, sharedDevNum)
+	manager := dpapi.NewManager(namespace, plugin)
+	manager.Run()
+}

cmd/vpu_plugin/vpu_plugin_test.go

@@ -0,0 +1,74 @@
+// Copyright 2019 Intel Corporation. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+	"github.com/google/gousb"
+	"github.com/intel/intel-device-plugins-for-kubernetes/pkg/debug"
+	"os"
+	"testing"
+)
+
+func init() {
+	debug.Activate()
+}
+
+type testCase struct {
+	vendorID   int
+	productIDs []int
+	sharedNum  int
+}
+
+//try to inject gousb compatible fake device info
+func (t *testCase) OpenDevices(opener func(desc *gousb.DeviceDesc) bool) ([]*gousb.Device, error) {
+	var ret []*gousb.Device
+	for _, p := range t.productIDs {
+		desc := &gousb.DeviceDesc{
+			Vendor:  gousb.ID(t.vendorID),
+			Product: gousb.ID(p),
+		}
+		if opener(desc) {
+			// only fake desc is enough
+			ret = append(ret, &gousb.Device{Desc: desc})
+		}
+	}
+	return ret, nil
+}
+
+func TestScan(t *testing.T) {
+	f, err := os.Create("/var/tmp/hddl_service.sock")
+	defer f.Close()
+	if err != nil {
+		t.Error("create fake hddl file failed")
+	}
+	//inject our fake gousbContext, just borrow vendorID and productIDs from main
+	tc := &testCase{
+		vendorID: vendorID,
+	}
+	//inject some productIDs that not match our target too
+	tc.productIDs = append(productIDs, 0xdead, 0xbeef)
+	testPlugin := newDevicePlugin(tc, vendorID, productIDs, 10)
+
+	if testPlugin == nil {
+		t.Error("vpu plugin test failed")
+	}
+
+	tree, err := testPlugin.scan()
+	if err != nil {
+		t.Error("vpu plugin test failed")
+	} else {
+		debug.Printf("tree len is %d", len(tree[deviceType]))
+	}
+}

demo/intelvpu-job.yaml

@@ -0,0 +1,22 @@
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: intelvpu-demo-job
+  labels:
+    jobgroup: intelvpu-demo
+spec:
+  template:
+    metadata:
+      labels:
+        jobgroup: intelvpu-demo
+    spec:
+      restartPolicy: Never
+      containers:
+        -
+          name: intelvpu-demo-job-1
+          image: ubuntu-demo-openvino:devel
+          imagePullPolicy: IfNotPresent
+          command: [ "/do_classification.sh" ]
+          resources:
+            limits:
+              vpu.intel.com/hddl: 1

demo/ubuntu-demo-openvino/Dockerfile

@@ -0,0 +1,42 @@
+FROM ubuntu:18.04 as builder
+ARG DOWNLOAD_LINK=http://registrationcenter-download.intel.com/akdlm/irc_nas/15792/l_openvino_toolkit_p_2019.2.275.tgz
+ARG INSTALL_DIR=/opt/intel/openvino
+ARG TEMP_DIR=/tmp/openvino_installer
+ADD $DOWNLOAD_LINK $TEMP_DIR/openvino.tgz
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    cpio \
+    sudo \
+    python3-pip \
+    python3-setuptools \
+    libboost-filesystem1.65 \
+    libboost-thread1.65 \
+    lsb-release
+RUN cd $TEMP_DIR && \
+    tar xf openvino.tgz && \
+    cd l_openvino_toolkit* && \
+    sed -i 's/decline/accept/g' silent.cfg && \
+    ./install.sh -s silent.cfg && \
+    rm -rf $TEMP_DIR
+RUN $INSTALL_DIR/install_dependencies/install_openvino_dependencies.sh
+# build Inference Engine samples
+RUN mkdir $INSTALL_DIR/deployment_tools/inference_engine/samples/build && cd $INSTALL_DIR/deployment_tools/inference_engine/samples/build && \
+    /bin/bash -c "source $INSTALL_DIR/bin/setupvars.sh && cmake .. && make -j1"
+RUN pip3 install networkx==2.3
+RUN cd $INSTALL_DIR/deployment_tools/demo && \
+    /bin/bash -c "source $INSTALL_DIR/bin/setupvars.sh && ./demo_squeezenet_download_convert_run.sh"
+RUN  cp /opt/intel/openvino/deployment_tools/demo/car.png /root && \
+     cp /opt/intel/openvino_2019.2.275/deployment_tools/inference_engine/lib/intel64/plugins.xml /root/inference_engine_samples_build/intel64/Release/lib/ && \
+     cp /opt/intel/openvino_2019.2.275/deployment_tools/inference_engine/lib/intel64/libHDDLPlugin.so /root/inference_engine_samples_build/intel64/Release/lib/ && \
+     cp /opt/intel/openvino_2019.2.275/deployment_tools/inference_engine/external/hddl/lib/libhddlapi.so /root/inference_engine_samples_build/intel64/Release/lib/ && \
+     cp /opt/intel/openvino_2019.2.275/deployment_tools/inference_engine/external/hddl/lib/libion.so.0 /root/inference_engine_samples_build/intel64/Release/lib/ && \
+     cp -r /opt/intel/openvino_2019.2.275/deployment_tools/inference_engine/external/hddl /root && \
+     ldd /root/inference_engine_samples_build/intel64/Release/classification_sample_async | grep opt | awk '{print $3}' | xargs -Iaaa cp aaa /root/inference_engine_samples_build/intel64/Release/lib/
+
+FROM ubuntu:18.04
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    libboost-filesystem1.65 \
+    libboost-thread1.65 && \
+    apt-get clean && rm -rf /var/lib/apt/lists/*
+
+COPY do_classification.sh /
+COPY --from=builder /root/ /root/

demo/ubuntu-demo-openvino/do_classification.sh

@@ -0,0 +1,5 @@
+#!/bin/bash -xe
+
+export HDDL_INSTALL_DIR=/root/hddl
+export LD_LIBRARY_PATH=/root/inference_engine_samples_build/intel64/Release/lib/
+/root/inference_engine_samples_build/intel64/Release/classification_sample_async -m /root/openvino_models/ir/FP16/classification/squeezenet/1.1/caffe/squeezenet1.1.xml -i /root/car.png -d HDDL

deployments/vpu_plugin/vpu_plugin.yaml

@@ -0,0 +1,54 @@
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: intel-vpu-plugin
+  namespace: kube-system
+  labels:
+    app: intel-vpu-plugin
+spec:
+  selector:
+    matchLabels:
+      app: intel-vpu-plugin
+  template:
+    metadata:
+      labels:
+        app: intel-vpu-plugin
+    spec:
+      containers:
+      - name: intel-vpu-plugin
+        env:
+          - name: NODE_NAME
+            valueFrom:
+              fieldRef:
+                fieldPath: spec.nodeName
+        image: intel-vpu-plugin:devel
+        imagePullPolicy: IfNotPresent
+        securityContext:
+          readOnlyRootFilesystem: true
+        volumeMounts:
+        - name: devfs
+          mountPath: /dev/bus/usb
+        - name: sysfs1
+          mountPath: /sys/bus/usb
+        - name: sysfs2
+          mountPath: /sys/devices
+        - name: tmpfs
+          mountPath: /var/tmp
+        - name: kubeletsockets
+          mountPath: /var/lib/kubelet/device-plugins
+      volumes:
+      - name: devfs
+        hostPath:
+          path: /dev/bus/usb
+      - name: sysfs1
+        hostPath:
+          path: /sys/bus/usb
+      - name: sysfs2
+        hostPath:
+          path: /sys/devices
+      - name: tmpfs
+        hostPath:
+          path: /var/tmp
+      - name: kubeletsockets
+        hostPath:
+          path: /var/lib/kubelet/device-plugins

go.mod

@@ -12,9 +12,11 @@ require (
 	github.com/d2g/dhcp4 v0.0.0-20170904100407-a1d1b6c41b1c // indirect
 	github.com/d2g/dhcp4client v0.0.0-20170829104524-6e570ed0a266 // indirect
 	github.com/fsnotify/fsnotify v1.4.7
+	github.com/fzipp/gocyclo v0.0.0-20150627053110-6acd4345c835 // indirect
 	github.com/go-ini/ini v1.46.0
 	github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
 	github.com/google/certificate-transparency-go v1.0.21 // indirect
+	github.com/google/gousb v0.0.0-20190812193832-18f4c1d8a750
 	github.com/heketi/rest v0.0.0-20180404230133-aa6a65207413 // indirect
 	github.com/heketi/utils v0.0.0-20170317161834-435bc5bdfa64 // indirect
 	github.com/jteeuwen/go-bindata v0.0.0-20151023091102-a0ff2567cfb7 // indirect
@@ -27,7 +29,9 @@ require (
 	github.com/sigma/go-inotify v0.0.0-20181102212354-c87b6cf5033d // indirect
 	github.com/vmware/photon-controller-go-sdk v0.0.0-20170310013346-4a435daef6cc // indirect
 	github.com/xanzy/go-cloudstack v0.0.0-20160728180336-1e2cbf647e57 // indirect
+	golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f // indirect
 	golang.org/x/sys v0.0.0-20190826190057-c7b8b68b1456
+	golang.org/x/tools v0.0.0-20200124021010-5c352bb417e0 // indirect
 	google.golang.org/grpc v1.23.1
 	gopkg.in/ini.v1 v1.46.0 // indirect
 	k8s.io/api v0.17.0

go.sum

@@ -142,6 +142,7 @@ github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5Kwzbycv
 github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc=
 github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
+github.com/fzipp/gocyclo v0.0.0-20150627053110-6acd4345c835/go.mod h1:BjL/N0+C+j9uNX+1xcNuM9vdSIcXCZrQZUYbXOFbgN8=
 github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
 github.com/ghodss/yaml v0.0.0-20180820084758-c7ce16629ff4/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
 github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
@@ -288,6 +289,8 @@ github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf h1:+RRA9JqSOZFfKrOeq
 github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf/go.mod h1:HP5RmnzzSNb993RKQDq4+1A4ia9nllfqcQFTQJedwGI=
 github.com/google/gofuzz v1.0.0 h1:A8PeW59pxE9IoFRqBp37U+mSNaQoZ46F1f0f863XSXw=
 github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
+github.com/google/gousb v0.0.0-20190812193832-18f4c1d8a750 h1:DVKHLo3yE4psTjD9aM2pY7EHoicaQbgmaxxvvHC6ZSM=
+github.com/google/gousb v0.0.0-20190812193832-18f4c1d8a750/go.mod h1:Tl4HdAs1ThE3gECkNwz+1MWicX6FXddhJEw7L8jRDiI=
 github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
 github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
 github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
@@ -618,6 +621,7 @@ golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8U
 golang.org/x/crypto v0.0.0-20190617133340-57b3e21c3d56/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586 h1:7KByu05hhLed2MO29w7p1XfZvZ13m8mub3shuVftRs0=
 golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190125153040-c74c464bbbf2/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190312203227-4b39c73a6495/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -628,8 +632,10 @@ golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvx
 golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
 golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
 golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
+golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs=
 golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE=
 golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc=
+golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
 golang.org/x/net v0.0.0-20170114055629-f2499483f923/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20170915142106-8351a756f30f/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -743,7 +749,10 @@ golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgw
 golang.org/x/tools v0.0.0-20190909030654-5b82db07426d/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20190920225731-5eefd052ad72 h1:bw9doJza/SFBEweII/rHQh338oozWyiFsBRHtrflcws=
 golang.org/x/tools v0.0.0-20190920225731-5eefd052ad72/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20200124021010-5c352bb417e0/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 gonum.org/v1/gonum v0.0.0-20190331200053-3d26580ed485 h1:OB/uP/Puiu5vS5QMRPrXCDWUPb+kt8f1KW8oQzFejQw=
 gonum.org/v1/gonum v0.0.0-20190331200053-3d26580ed485/go.mod h1:2ltnJ7xHfj0zHS40VVPYEAAMTa3ZGguvHGBSJeRWqE0=
 gonum.org/v1/netlib v0.0.0-20190313105609-8cb42192e0e0/go.mod h1:wa6Ws7BG/ESfp6dHfk7C6KdzKA7wR7u/rKwOGE66zvw=