From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f54.google.com (mail-wm0-f54.google.com [74.125.82.54]) by dpdk.org (Postfix) with ESMTP id 8DDF2C44C for ; Thu, 23 Jun 2016 13:48:50 +0200 (CEST) Received: by mail-wm0-f54.google.com with SMTP id f126so123567332wma.1 for ; Thu, 23 Jun 2016 04:48:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=pbrvrcfzktMpBTEuQ8FrSzbO6731JV4DYKdo6CXd6A0=; b=c9z/un2IxQirC0xLdf/EdLPH/E5KhIK3SkjA+3QKPQuD9S/UZ2AXabvPlmp7Ya2L7W LW/5VwBeK9H7Ef7O9paS5G3pDoqg1ulNScT2ofGVSkHKpfRlZXbs67lVQOX4EOoVZj44 qGj4/Wymh0p85l3+Cmq70NGisPWx1g9q5n5jCBwt2njSw7ZCye3TyfwXJV+gwM92A281 Lc61A+eBT94LzfFalikfakjyk+PlUxitFssfma/4FxxFPduZuVNvS8FWadx6INc8f+/E yKMloBFCtqh1GJDxf4LOn0721vwRlW63q6Po+UGJT4lWui5jyqR21oHREj0IwqpUuNNT imow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=pbrvrcfzktMpBTEuQ8FrSzbO6731JV4DYKdo6CXd6A0=; b=kxkrXXZZN7haRXQfEv/3ZmKfg2K7MS0g9SmSU+6wMsbHVshSWxuj4bEiQ5xMvtxUqt u2J1xXwlLlrRZeDFePs7ykab2Je/h9g8NDIQbmru0vVMtUZ+dzHwkcURdN+Q41Qov4RZ 92J3uozpOjVUdxAXXI7g2PdNxylpQaqOulPWGPGx0c1Lm99eHo+PT7kV845xIVulCaqB bg62zAxFr52NQtH/9s2Bg3LVF2cx7FEohVyB3jhMrcHlCvwvRSGWAIpuF3JyEBR/Wohz aeaZGqs/FCpCaIrJU1rgySwJvlY9Qk4XGSiQMEttTcJlCZMTL2EjrOVgL/Xx8BxHn8Ux XwqA== X-Gm-Message-State: ALyK8tJADNjOIy7bc8cHasiwb1l7lWl4GYWXQm5cesEW5coNcwF/7O1HxNXdvWXhHGxhsUYx9iZ+6MTBdTuYRQ== X-Received: by 10.194.56.134 with SMTP id a6mr20362103wjq.106.1466682530015; Thu, 23 Jun 2016 04:48:50 -0700 (PDT) MIME-Version: 1.0 Received: by 10.194.14.1 with HTTP; Thu, 23 Jun 2016 04:48:49 -0700 (PDT) In-Reply-To: References: <6e0329b1-d67c-3b23-55d2-dc88b78f6f2a@intel.com> From: Chinmaya Dwibedy Date: Thu, 23 Jun 2016 17:18:49 +0530 Message-ID: To: Sergio Gonzalez Monroy Cc: users@dpdk.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode) X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Jun 2016 11:48:50 -0000 Hi All, On host, I installed dpdk-2.2.0 and then run 'cryptodev_qat_autotest=E2=80=99. I found this test case to be failed in dequeue burst API ().Upon looking into the below system logs (#dmesg), it seems, when this function is called, the QAT hardware fails to write the crypto packet to the DMA buffer. When I passed intel_iommu=3Doff command in boot line, it works. I think, when we have no IOMMU, "physical" address space is accessed directly by hardware, so code works. The issue has been observed if the system runs with the kernel parameter =E2=80=9Cintel_iommu=3Don=E2=80=9D. [root@stag48 ~]# dmesg [ 794.746134] dmar: DRHD: handling fault status reg 2 [ 794.746141] dmar: DMAR:[DMA Read] Request device [83:01.0] fault addr 3398c000 DMAR:[fault reason 02] Present bit in context entry is clear [ 794.746158] icp_qa_al err: adf_dh895xcc_adf_isr_handleUncoInterrupt: Uncorrectable Push/Pull Misc Error memory status: No errors occurred - Transaction Id 0x0 - Error type reserve= d Bus Operation Type Push - Id 0x80000 [ 794.746167] Reset needed for device: icp_dev0 [ 794.746167] Auto Reset disabled. Please reset device0 [root@stag48 ~]# *Note: *The QAT VF (*83:01.0*) is bound to the DPDK UIO driver. 83:01.0 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Functio= n Subsystem: Intel Corporation Device 0000 Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=3Dfast >TAbort- SERR- 0002) [27086.464040] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_sta= te [27086.564610] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_sta= te [27086.574052] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_sta= te [27089.786623] kvm: zapping shadow pages for mmio generation wraparound [27106.320042] vfio-pci 0000:88:04.6: irq 136 for MSI/MSI-X [28474.784373] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_sta= te [28475.040313] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_sta= te [28493.271950] vfio-pci 0000:88:04.6: irq 136 for MSI/MSI-X Regards, Chinmaya On Wed, Jun 22, 2016 at 1:39 PM, Chinmaya Dwibedy wrote: > Hi Sergio, > > > Thank your for clarification. I am using dpdk-2.2.0. Here goes our > configuration on VM. Can you please have a look into the same and let me > know if anything is wrong. Will try to run the autotest, ipsec-segw on ho= st > as per your suggestion. > > > [root@vpn-s ~]# uname -r > > 3.12.9-301.fc20.x86_64 > > [root@vpn-s ~]# > > > [root@vpn-s ~]# cat /etc/redhat-release > > Fedora release 20 (Heisenbug) > > [root@vpn-s ~]# > > > > > [root@vpn-s ~]# cat /proc/cmdline > > BOOT_IMAGE=3D/boot/vmlinuz-3.12.9-301.fc20.x86_64 > root=3DUUID=3Dab47cbc9-68ee-403c-96a5-184e68238e65 ro console=3DttyS0,115= 200n8 > iommu=3Dpt intel_iommu=3Don > > [root@vpn-s ~]# > > > [root@vpn-s QAT]# grep -i "iommu.*enabled" /var/log/messages > > Jun 22 07:31:17 vpn-s kernel: [ 0.000000] Intel-IOMMU: enabled > > > [root@vpn-s QAT]# > > [root@vpn-s dpdk-2.2.0]# lspci -nn | grep 0443 > > 00:06.0 Co-processor [0b40]: Intel Corporation Device [8086:0443] > > [root@vpn-s dpdk-2.2.0]# > > > Installed Intel qat driver installation on VM (downloaded from > https://01.org/packet-processing/intel%C2%AE-quickassist-technology-drive= rs-and-patches) > and followed the > > https://01.org/sites/default/files/page/330750-005_qat_gsg.pdf guide. > Note, used the following installation options i.e., ./installer.sh instal= l > QAT1.6 guest. After this , it detects QAT VF. > > > [root@vpn-s QAT]# service qat_service status > > There is 1 acceleration device(s) in the system: > > icp_dev0 - type=3Ddh895xcc, inst_id=3D0, node_id=3D0, bdf=3D00:06:0, #a= ccel=3D1, > #engines=3D1, state=3Dup > > [root@vpn-s QAT]# > > > [root@vpn-s QAT]# lsmod | grep qa > > icp_qa_al_vf 813442 1 > > [root@vpn-s QAT]# > > > > The below are DPDK configuration options in > config/defconfig_x86_64-native-linuxapp-gcc. > > > include "common_linuxapp" > > > > CONFIG_RTE_MACHINE=3D"native" > > CONFIG_RTE_ARCH=3D"x86_64" > > CONFIG_RTE_ARCH_X86_64=3Dy > > CONFIG_RTE_ARCH_64=3Dy > > CONFIG_RTE_TOOLCHAIN=3D"gcc" > > CONFIG_RTE_TOOLCHAIN_GCC=3Dy > > CONFIG_RTE_LIBRTE_PMD_AESNI_MB=3Dy > > CONFIG_RTE_LIBRTE_CRYPTODEV_DEBUG=3Dy > > CONFIG_RTE_LIBRTE_VIRTIO_PMD=3Dy > > CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=3Dy > > CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=3Dy > > CONFIG_RTE_LIBRTE_PMD_QAT=3Dy > > CONFIG_RTE_QAT_PMD_MAX_NB_SESSIONS=3D128 > > CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_INIT=3Dy > > CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_TX=3Dy > > CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_RX=3Dy > > CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_DRIVER=3Dy > > > > The below is my setup.sh > > > export RTE_SDK=3D/opt/dpdk-2.2.0 > > export RTE_TARGET=3Dx86_64-native-linuxapp-gcc > > export RTE_ARCH=3Dx86_64 > > export AESNI_MULTI_BUFFER_LIB_PATH=3D/opt/code > > make config T=3Dx86_64-native-linuxapp-gcc > > sed -ri 's,(PMD_PCAP=3D).*,\1y,' build/.config > > make install T=3Dx86_64-native-linuxapp-gcc > > mkdir -p /mnt/huge > > mount -t hugetlbfs nodev /mnt/huge > > echo 2048 > > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages > > sudo modprobe uio_pci_generic > > sudo modprobe uio > > rmmod igb_uio > > insmod > ./x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_ui= o.ko > > echo -n "0000:00:06.0" > /sys/bus/pci/devices/0000\:00\:06.0/driver/unbin= d > > echo "8086 0443" > /sys/bus/pci/drivers/igb_uio/new_id > > lspci -vvd:443 > > ./tools/dpdk_nic_bind.py --bind=3Duio_pci_generic eth1 > > ./tools/dpdk_nic_bind.py --bind=3Duio_pci_generic 00:03.0 > > ./tools/dpdk_nic_bind.py -s > > > Also modified the cryptodevs_init() ( examples/ipsec-secgw/ipsec-secgw.c) > to rte_cryptodev_start(cdev_id); after rte_cryptodev_queue_pair_setup(). > The purpose of calling this to start the transmit and the receive units > of the QAT device. It returns success. But still getting the same issue > i.e. rte_cryptodev_dequeue_burst () does not return crypto packet. Note > that, when I call rte_cryptodev_queue_pair_setup () with 3rd argument as > NULL (in which case default configuration will be used) , it crashes. Is > it a known issue? > > > > > > Regards, > > Chinmaya > > On Tue, Jun 21, 2016 at 9:10 PM, Sergio Gonzalez Monroy < > sergio.gonzalez.monroy@intel.com> wrote: > >> On 21/06/2016 08:26, Chinmaya Dwibedy wrote: >> >>> >>> Hi Sergio, >>> >>> >>> As suggested by you, run =E2=80=98app/test' application and then run >>> 'cryptodev_qat_autotest' . Found the same issue i.e., It halts in >>> rte_cryptodev_dequeue_burst () (while loop) in process_crypto_request (= ) >>> function (app/test/test_cryptodev.c). >>> >>> Debugged the qat_crypto_pkt_rx_burst() (in >>> drivers/crypto/qat/qat_crypto.c) and found that , the value of resp_msg= is >>> 0x7F7F7F7F (i.e., ADF_RING_EMPTY_SIG) . Thus it is not getting crypto >>> packet (i.e., rte_mbuf) from the RX queue. But I find that, in >>> qat_crypto_pkt_tx_burst(), the qat_alg_write_mbuf_entry() is being cal= led. >>> The qat_alg_write_mbuf_entry () prints qat_req using rte_hexdump () an= d >>> returns zero. >>> >>> >>> One question, the base_addr + tail of tx_q (in >>> qat_crypto_pkt_tx_burst()) should be same as base_addr + head (in >>> qat_crypto_pkt_rx_burst()). Right ? Please feel free to correct me if I= am >>> wrong. >>> >>> >>> Also can you please provide some pointers for further debugging? Would >>> it be configuration issue? Note that, I am using VF in VM. >>> >>> >> In theory you should be able to run ipsec-secgw sample app with QAT VF >> as long as the setup is right. >> That the autotest fails would likely point to some possible issues with >> the setup. >> >> Have you tried to run the autotest in the host? >> >> Sergio >> >> >>> Thank you once again for your prompt response and great support. >>> >>> >>> Regards, >>> >>> Chinmaya >>> >>> >>> On Mon, Jun 20, 2016 at 3:04 PM, Sergio Gonzalez Monroy < >>> sergio.gonzalez.monroy@intel.com >> sergio.gonzalez.monroy@intel.com>> wrote: >>> >>> On 20/06/2016 10:08, Chinmaya Dwibedy wrote: >>> >>> >>> Hi Sergio, >>> >>> >>> Agreed. We might not dequeue the same amount of crypto ops we >>> just previously enqueued, it's asynchronous. But in this case, >>> I have sent just one UDP packet. So there will be one crypto >>> ops. Right? Also I put a sleep (50) after the >>> rte_crypto_enqueue_burst() function in ipsec_processing() >>> (ipsec.c) , so as to allow more time ( for QAT device) for >>> processing. Still getting the same result i.e., the >>> rte_crypto_dequeue_burst () functionreturns zero. >>> >>> >>> In case of S/W crypto device (i.e., AESNI), the VM gets >>> inbound UDP packets on Port 1/eth1, encapsulates (after >>> consulting its SPD) in an IPsec ESP packet and sends to its >>> peer through Port 0/eth0 interface. >>> >>> >>> Yes, the security policy, security association and Routing >>> entries/configurations are exactly same. Please feel free to >>> let me know if you need additional information. >>> >>> >>> Could you try to run 'app/test' application then run >>> 'cryptodev_qat_autotest' ? That is a functional test for cryptodev >>> QAT PMD. >>> >>> Sergio >>> >>> >>> >> >