From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f68.google.com (mail-wm0-f68.google.com [74.125.82.68]) by dpdk.org (Postfix) with ESMTP id CE91B2BA2 for ; Fri, 10 Jun 2016 09:16:38 +0200 (CEST) Received: by mail-wm0-f68.google.com with SMTP id m124so15667002wme.3 for ; Fri, 10 Jun 2016 00:16:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=aSwh1aDTqYwUZba/rAYshKabf46mQlB0cpmLIHwar90=; b=OHK1s9JsNVuX7+lqrcxcr8EGrhaMAqT7hvnDPtphjUZIPlnHSgKdYjVPeEOzbrviOZ iXzpBmTk5Hn0iQqsqd4xOn0kF7JIx901xxa9b0f8UjZDXHOHVyumwwNO4a2slF0zNLBa mVF7B6D4IkDZ+gHWD7GA1mJoU/defVvd3lfOgL7y2f4b1GcAzhuYl0Mbep56VXBNC+tb eq5Szuwzbck5JXyoKASWsxg/IaRse17RiSMRdWnc3+owAkWFj6DY31YZZJH5xgg8BnRV LNNCYd+O+mni6QUSe8Wl8fq7hlascmHZPYVRNq5sSStUBEHLePmpccApcsjs5f4FYYtD x5jQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=aSwh1aDTqYwUZba/rAYshKabf46mQlB0cpmLIHwar90=; b=HTsiy+jMyl7SpkwXjPFLKgnNn0Vmr8/RExT5pKjrgWKea8XZJ+MfLsxh2/qLhsnWWl gQpxmTdZBAWQEAiWHCYY+kfMlrIfQyn8ZgexO68JQOuiLQadtVWh3BqTxRbemyCf5Yl2 bc3JoEnEjUJxnrOs+G79F7gVVK0GaWp1PkZzZU5AKer3yFsfmV0y+mcKTAqt4wTw2pT1 /5+zdO9rt7/opa4F/aQcXgrcxwRAIhXq2SXZcwrqfWOLA3ZmN1BsG5/s0wdcv73fkkOX os2EeFkqTZ6TUjPKUtRfO2UYFKgZ0S3GRyNzrxOROnWFguB2uIrcV7ufA6bNyG1WnONh 07pQ== X-Gm-Message-State: ALyK8tKTnHVMApCzbqus5DM9amqFEugNZxaedK/pDG9SMitwv9EKC5O/MUeEXrpVyaSz3R81Bdpt9m3qO36EjA== X-Received: by 10.194.153.65 with SMTP id ve1mr624295wjb.106.1465542998434; Fri, 10 Jun 2016 00:16:38 -0700 (PDT) MIME-Version: 1.0 Received: by 10.194.14.1 with HTTP; Fri, 10 Jun 2016 00:16:37 -0700 (PDT) From: Chinmaya Dwibedy Date: Fri, 10 Jun 2016 12:46:37 +0530 Message-ID: To: users@dpdk.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] Issue with Physical Function passthrough on VM Guest (Fedora 20) using qatmux.l.2.6.0-60 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Jun 2016 07:16:39 -0000 Hi, I have assigned Intel QAT device to VMs as a full device (PF - physical function, PCI passthrough). Below is the output of lspci in host (Cent OS 7.2). [root@localhost nova(keystone_admin)]# lspci -nn | grep 0435 83:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [ 8086:0435] 88:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [ 8086:0435] [root@localhost nova(keystone_admin)]# [root@localhost ~]# lspci -vnd 8086:0435 83:00.0 0b40: 8086:0435 Subsystem: 8086:35c5 Physical Slot: 0-1 Flags: bus master, fast devsel, latency 0, IRQ 35 Memory at 387fffd00000 (64-bit, prefetchable) [size=3D512K] Memory at c8200000 (64-bit, non-prefetchable) [size=3D256K] Memory at c8240000 (64-bit, non-prefetchable) [size=3D256K] Capabilities: [b0] MSI: Enable- Count=3D1/1 Maskable+ 64bit+ Capabilities: [60] MSI-X: Enable- Count=3D33 Masked- Capabilities: [6c] Power Management version 3 Capabilities: [74] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [138] Alternative Routing-ID Interpretation (ARI) Capabilities: [140] Single Root I/O Virtualization (SR-IOV) Kernel driver in use: vfio-pci 88:00.0 0b40: 8086:0435 Subsystem: 8086:35c5 Physical Slot: 0-2 Flags: fast devsel, IRQ 39 Memory at 387fffe00000 (64-bit, prefetchable) [size=3D512K] Memory at c8000000 (64-bit, non-prefetchable) [size=3D256K] Memory at c8040000 (64-bit, non-prefetchable) [size=3D256K] Capabilities: [b0] MSI: Enable- Count=3D1/1 Maskable+ 64bit+ Capabilities: [60] MSI-X: Enable- Count=3D33 Masked- Capabilities: [6c] Power Management version 3 Capabilities: [74] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [138] Alternative Routing-ID Interpretation (ARI) Capabilities: [140] Single Root I/O Virtualization (SR-IOV) Kernel driver in use: vfio-pci [root@localhost ~]# In VM, the Intel QAT device shows up in lspci as "Intel Corporation Coleto Creek PCIe Endpoint". [root@test1 QAT]# lspci -nn | grep 0435 00:05.0 Co-processor [0b40]: Intel Corporation Coleto Creek PCIe Endpoint [8086:0435] [root@test1 QAT]# Afterward I installed Intel=C2=AE QuickAssist Technology Software i.e., qatmux.l.2.6.0-60.tgz ( downloaded at https://01.org/packet-processing/intel%C2%AE-quickassist-technology-drivers= -and-patches) on the Guest VM . Here are the details of VM. 1) Fedora release 20 (Heisenbug). 2) Linux Kernel version: 3.12.9-301.fc20.x86_64. 3) BOOT_IMAGE=3D/boot/vmlinuz-3.12.9-301.fc20.x86_64 root=3DUUID=3Dab47cbc9-68ee-403c-96a5-184e68238e65 ro console=3DttyS0,11520= 0n8 intel_iommu=3Doff *Notes: * 1) We are not using SR-IOV and are instead passing through a PF for acceleration services on guest only. Hence we have not installed the Intel=C2=AE QuickAssist Technology Software package on the host. 2) Found an Intel QAT driver with this kernel . Upon doing #lsmod | grep qa, showed icp_qa_al_vf. Thus did rmmod icp_qa_al_vf and service qat_service shutdown. This was done before building Intel QuickAssist Technology Software package. For passthrough of PF, opted the below Install Acceleration option a) export ICP_SRIOV=3D0 b) export ICP_WITHOUT_IOMMU=3D1 c) ./installer.sh build_accel QAT1.6 d) ./installer.sh install QAT1.6 guest e) insmod /root/QAT /QAT1.6/build/icp_qa_al.ko When I issue the below command to start the acceleration software, it fails to start the driver. [root@test1 ~]# service qat_service start Processing file: /etc/dh895xcc_qa_dev0.conf ADF_CONFIG_CTL err: adf_set_bank_core_affinity: Invalid core affinity settings for accelerator 0 bank 8. Max cpu number is 3. Defaulting to 0 ADF_CONFIG_CTL err: adf_set_bank_core_affinity: Invalid core affinity settings for accelerator 0 bank 9. Max cpu number is 3. Defaulting to 0 Parity err reporting is disabled. Error: failed to start device 0 There is 1 acceleration device(s) in the system: icp_dev0 - type=3Ddh895xcc, inst_id=3D0, node_id=3D0, bdf=3D00:05:0, #acc= el=3D6, #engines=3D12, state=3Ddown [root@test1 ~]# Here are the kernel logs [root@test1 ~]# dmesg -c [ 3104.979666] Reading config file. [ 3104.980250] Starting acceleration device icp_dev0. [ 3104.980731] Resetting device icp_dev0 [ 3105.126920] qat_1_6_adf 0000:00:05.0: irq 45 for MSI/MSI-X [ 3105.126939] qat_1_6_adf 0000:00:05.0: irq 46 for MSI/MSI-X [ 3105.126956] qat_1_6_adf 0000:00:05.0: irq 47 for MSI/MSI-X [ 3105.126973] qat_1_6_adf 0000:00:05.0: irq 48 for MSI/MSI-X [ 3105.126989] qat_1_6_adf 0000:00:05.0: irq 49 for MSI/MSI-X [ 3105.127029] qat_1_6_adf 0000:00:05.0: irq 50 for MSI/MSI-X [ 3105.127048] qat_1_6_adf 0000:00:05.0: irq 51 for MSI/MSI-X [ 3105.127065] qat_1_6_adf 0000:00:05.0: irq 52 for MSI/MSI-X [ 3105.127082] qat_1_6_adf 0000:00:05.0: irq 53 for MSI/MSI-X [ 3105.127098] qat_1_6_adf 0000:00:05.0: irq 54 for MSI/MSI-X [ 3105.127115] qat_1_6_adf 0000:00:05.0: irq 55 for MSI/MSI-X [ 3105.127131] qat_1_6_adf 0000:00:05.0: irq 56 for MSI/MSI-X [ 3105.127148] qat_1_6_adf 0000:00:05.0: irq 57 for MSI/MSI-X [ 3105.127164] qat_1_6_adf 0000:00:05.0: irq 58 for MSI/MSI-X [ 3105.127181] qat_1_6_adf 0000:00:05.0: irq 59 for MSI/MSI-X [ 3105.127198] qat_1_6_adf 0000:00:05.0: irq 60 for MSI/MSI-X [ 3105.127215] qat_1_6_adf 0000:00:05.0: irq 61 for MSI/MSI-X [ 3105.127231] qat_1_6_adf 0000:00:05.0: irq 62 for MSI/MSI-X [ 3105.127248] qat_1_6_adf 0000:00:05.0: irq 63 for MSI/MSI-X [ 3105.127265] qat_1_6_adf 0000:00:05.0: irq 64 for MSI/MSI-X [ 3105.127282] qat_1_6_adf 0000:00:05.0: irq 65 for MSI/MSI-X [ 3105.127298] qat_1_6_adf 0000:00:05.0: irq 66 for MSI/MSI-X [ 3105.127315] qat_1_6_adf 0000:00:05.0: irq 67 for MSI/MSI-X [ 3105.127331] qat_1_6_adf 0000:00:05.0: irq 68 for MSI/MSI-X [ 3105.127348] qat_1_6_adf 0000:00:05.0: irq 69 for MSI/MSI-X [ 3105.127364] qat_1_6_adf 0000:00:05.0: irq 70 for MSI/MSI-X [ 3105.127380] qat_1_6_adf 0000:00:05.0: irq 71 for MSI/MSI-X [ 3105.127397] qat_1_6_adf 0000:00:05.0: irq 72 for MSI/MSI-X [ 3105.127413] qat_1_6_adf 0000:00:05.0: irq 73 for MSI/MSI-X [ 3105.127429] qat_1_6_adf 0000:00:05.0: irq 74 for MSI/MSI-X [ 3105.127446] qat_1_6_adf 0000:00:05.0: irq 75 for MSI/MSI-X [ 3105.127462] qat_1_6_adf 0000:00:05.0: irq 76 for MSI/MSI-X [ 3105.127487] qat_1_6_adf 0000:00:05.0: irq 77 for MSI/MSI-X [ 3105.157973] icp_qa_al err: adf_subsystemInit: Invalid SRIOV_Enabled value, driver is built with SRIOV. [ 3105.158891] icp_qa_al err: adf_do_init: adf_subsystemInit error, stopping and shutting down [ 3106.162962] icp_qa_al err: adf_init_all: failed to initialize accelerato= r [ 3106.163670] icp_qa_al err: adf_init_devices: failed init accel 0 [ 3106.164256] icp_qa_al err: adf_ctl_ioctl_dev_start: failed call to adf_init_devices [root@test1 ~]# Can anyone please let me know what might be the issue and its remedy ? please feel free to correct me if I am wrong. If it is not the right forum , please let me know the correct one. Thank you in advance for time and support. Regards, Chinmaya