From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f44.google.com (mail-wm0-f44.google.com [74.125.82.44]) by dpdk.org (Postfix) with ESMTP id 30B5DC51A for ; Fri, 24 Jun 2016 11:34:54 +0200 (CEST) Received: by mail-wm0-f44.google.com with SMTP id f126so15267137wma.1 for ; Fri, 24 Jun 2016 02:34:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=bIxl/dmuQV1DxahLZD33PLNp0epLIgdkjP+jNN+ledw=; b=dkHLUQ2yiATgXAh1s8JAkvJ0KcdMF7AvIiyoNO2ZxyG0D/myNYmkLP04UonZWLroQe lpxOsJ7namy1vxGF/A9Tt+6o5zZKJv90qnjTRq3TaWc5gww6BuHBFbgWmfBOXxT+9JZ1 cgrFoDsTzBgqoxr0qOL4Fhyb0fR/85wvVhMF1z3mU5LIThzn7CaLecwuYdzIGORatimV Qi+XvgI9IwX9Ax28UcrWsle8I29lOhY1WlQgeBRXYkwGa/TUVOrpXEPRO/bc9acr/K5v X1PXYV0Vhyx98XCjpB3zboE7Sj7BX+9SG7VmK5e/OPcDmARmiiZ4XYIjqQh68ihCIZk7 vY8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=bIxl/dmuQV1DxahLZD33PLNp0epLIgdkjP+jNN+ledw=; b=c0ChpOA4F4PrzqFj3CyhZD5ka33AnEsTMZdZB8MWgHrBTbMAoikVXvoqaYBUwUd+IA UQSiBDUSG2MHI/SpxgregH18wMSg3MixNAa+9Us1RqUD1TXaU49CPzlBy4161fEHTvrF vaaLFkxAB3a7w5TombTb7qahUd11sTXrqvBPSY2yE8tSXBOLzcAu7jFms8ApE/z/atUE 43RxEhUwKl2bxcC9GKkGaX/aaUpIRGJ0yOtItnFZi6kPUOFhicPgzg8nIslANqKNWFe9 ns9uyKMUu9xeKWSd84EUAtVnvLN1UeBCBTkSf3zVtScXywaytiBZf69FGYAHlsuD+gax ipJg== X-Gm-Message-State: ALyK8tLYZqemzNblckhCL45QuDuDxHa3AiVhiBW4QapG6Ipn46A32/CjiQN8vUniIiA1vTFwi3/oXCF82nDbAA== X-Received: by 10.194.248.230 with SMTP id yp6mr2808964wjc.144.1466760892775; Fri, 24 Jun 2016 02:34:52 -0700 (PDT) MIME-Version: 1.0 Received: by 10.194.14.1 with HTTP; Fri, 24 Jun 2016 02:34:52 -0700 (PDT) In-Reply-To: References: <6e0329b1-d67c-3b23-55d2-dc88b78f6f2a@intel.com> From: Chinmaya Dwibedy Date: Fri, 24 Jun 2016 15:04:52 +0530 Message-ID: To: Sergio Gonzalez Monroy Cc: users@dpdk.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode) X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Jun 2016 09:34:54 -0000 Hi Sergio/All, Upon further investigation found that, on Guest VM if I remove =E2=80=9Cintel_iommu=3Don=E2=80=9D boot line parameters and run app/test an= d then crypto_qat_autotest tool, it shows all the test cases are passed. Thereafter when I run ipsec-segw application, I got the below error in dmesg on host. Is it an issue with crypto device configuration and queue pair setup? Please suggest. [root@stag48 ~(keystone_admin)]# dmesg -c [ 1264.266965] dmar: DRHD: handling fault status reg 2 [ 1264.266972] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr ff53f000f000 DMAR:[fault reason 06] PTE Read access is not set [ 1264.266978] icp_qa_al err: adf_dh895xcc_adf_isr_LogAE_Int: AE #5 Error Uncorrectable Error: Yes Control Store Error: No Control Store information: Address 0x2A4C - syndrome 0x0 Register ECC Error No - Parity Error: Yes Register information: address 0x2 - type Transfer [ 1264.266980] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr ff53f000e000 DMAR:[fault reason 06] PTE Read access is not set [ 1264.266986] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr ff53f000e000 DMAR:[fault reason 06] PTE Read access is not set [ 1264.266990] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr ff53f000f000 DMAR:[fault reason 06] PTE Read access is not set [ 1264.266993] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr ff53f000e000 DMAR:[fault reason 06] PTE Read access is not set [ 1264.266997] icp_qa_al err: adf_dh895xcc_adf_isr_LogAE_Int: AE #7 Error Uncorrectable Error: Yes Control Store Error: No Control Store information: Address 0x3FFD - syndrome 0x0 Register ECC Error No - Parity Error: Yes Register information: address 0x22 - type Transfer [ 1264.266999] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr ff53f000e000 DMAR:[fault reason 06] PTE Read access is not set [ 1264.267003] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr ff53f000e000 DMAR:[fault reason 06] PTE Read access is not set [ 1264.267007] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr ff53f000f000 DMAR:[fault reason 06] PTE Read access is not set [ 1264.267014] icp_qa_al err: adf_dh895xcc_adf_isr_LogAE_Int: AE #8 Error Uncorrectable Error: Yes Control Store Error: No Control Store information: Address 0x2102 - syndrome 0x0 Register ECC Error No - Parity Error: Yes Register information: address 0x2 - type Transfer [ 1264.267020] icp_qa_al err: adf_dh895xcc_adf_isr_LogAE_Int: AE #9 Error Uncorrectable Error: Yes Control Store Error: No Control Store information: Address 0x0 - syndrome 0x0 Register ECC Error No - Parity Error: Yes Register information: address 0x22 - type Transfer [ 1264.267026] icp_qa_al err: adf_dh895xcc_adf_isr_LogAE_Int: AE #b Error Uncorrectable Error: Yes Control Store Error: No Control Store information: Address 0x3FFF - syndrome 0x0 Register ECC Error No - Parity Error: Yes Register information: address 0x2 - type Transfer [ 1264.267032] icp_qa_al err: adf_dh895xcc_adf_isr_handleUncoInterrupt: Uncorrectable Push/Pull Misc Error memory status: No errors occurred - Transaction Id 0x0 - Error type reserve= d Bus Operation Type Push - Id 0x3859C020 [ 1264.267038] Reset needed for device: icp_dev1 [ 1264.267039] Auto Reset disabled. Please reset device1 [root@stag48 ~(keystone_admin)]# Regards, Chinmaya On Thu, Jun 23, 2016 at 5:18 PM, Chinmaya Dwibedy wrote: > Hi All, > > On host, I installed dpdk-2.2.0 and then run 'cryptodev_qat_autotest=E2= =80=99. I found this test case to be failed in dequeue burst API ().Upon lo= oking into the below system logs (#dmesg), it seems, when this function is = called, the QAT > > hardware fails to write the crypto packet to the DMA buffer. When I pass= ed intel_iommu=3Doff command in boot line, it works. I think, when we have = no IOMMU, "physical" address space is accessed directly by hardware, so cod= e works. > > The issue has been observed if the system runs with the kernel parameter = =E2=80=9Cintel_iommu=3Don=E2=80=9D. > > > > [root@stag48 ~]# dmesg > > [ 794.746134] dmar: DRHD: handling fault status reg 2 > > [ 794.746141] dmar: DMAR:[DMA Read] Request device [83:01.0] fault addr > 3398c000 > > DMAR:[fault reason 02] Present bit in context entry is clear > > [ 794.746158] icp_qa_al err: adf_dh895xcc_adf_isr_handleUncoInterrupt: > Uncorrectable Push/Pull Misc Error > > memory status: No errors occurred - Transaction Id 0x0 - Error type > reserved > > Bus Operation Type Push - Id 0x80000 > > [ 794.746167] Reset needed for device: icp_dev0 > > [ 794.746167] Auto Reset disabled. Please reset device0 > > [root@stag48 ~]# > > > > *Note: *The QAT VF (*83:01.0*) is bound to the DPDK UIO driver. > > > > 83:01.0 Co-processor: Intel Corporation DH895XCC Series QAT Virtual > Function > > Subsystem: Intel Corporation Device 0000 > > Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- > ParErr- Stepping- SERR- FastB2B- DisINTx- > > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=3Dfast >TAbort- > SERR- > Latency: 0 > > Region 0: [virtual] Memory at c8280000 (64-bit, non-prefetchable) > [size=3D4K] > > Region 2: [virtual] Memory at c82a0000 (64-bit, non-prefetchable) > [size=3D4K] > > Capabilities: [90] MSI: Enable- Count=3D1/1 Maskable+ 64bit+ > > Address: 0000000000000000 Data: 0000 > > Masking: 00000000 Pending: 00000000 > > Kernel driver in use: igb_uio > > > > > > Now testing the same on Guest and finding the same issue (i.e., failing o= n dequeue burst API ()). Upon looking into error message (in dmesg) on host= /hypervisor it says i.e., [28475.040313] > > vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state. The PCI d= evice 88:04.6 is the QAT VF, being assigned to Guest. I am trying to figure= out what causes =E2=80=9Cbuffer not found=E2=80=9D issue . > > Please suggest if anyone has an idea on it. > > > > Here goes the output of dmesg > > > > [27086.464014] vfio-pci 0000:88:04.6: enabling device (0000 -> 0002) > > [27086.464040] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_s= tate > > [27086.564610] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_s= tate > > [27086.574052] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_s= tate > > [27089.786623] kvm: zapping shadow pages for mmio generation wraparound > > [27106.320042] vfio-pci 0000:88:04.6: irq 136 for MSI/MSI-X > > [28474.784373] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_s= tate > > [28475.040313] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_s= tate > > [28493.271950] vfio-pci 0000:88:04.6: irq 136 for MSI/MSI-X > > > > > > Regards, > > Chinmaya > > > > > > > > On Wed, Jun 22, 2016 at 1:39 PM, Chinmaya Dwibedy > wrote: > >> Hi Sergio, >> >> >> Thank your for clarification. I am using dpdk-2.2.0. Here goes our >> configuration on VM. Can you please have a look into the same and let me >> know if anything is wrong. Will try to run the autotest, ipsec-segw on h= ost >> as per your suggestion. >> >> >> [root@vpn-s ~]# uname -r >> >> 3.12.9-301.fc20.x86_64 >> >> [root@vpn-s ~]# >> >> >> [root@vpn-s ~]# cat /etc/redhat-release >> >> Fedora release 20 (Heisenbug) >> >> [root@vpn-s ~]# >> >> >> >> >> [root@vpn-s ~]# cat /proc/cmdline >> >> BOOT_IMAGE=3D/boot/vmlinuz-3.12.9-301.fc20.x86_64 >> root=3DUUID=3Dab47cbc9-68ee-403c-96a5-184e68238e65 ro console=3DttyS0,11= 5200n8 >> iommu=3Dpt intel_iommu=3Don >> >> [root@vpn-s ~]# >> >> >> [root@vpn-s QAT]# grep -i "iommu.*enabled" /var/log/messages >> >> Jun 22 07:31:17 vpn-s kernel: [ 0.000000] Intel-IOMMU: enabled >> >> >> [root@vpn-s QAT]# >> >> [root@vpn-s dpdk-2.2.0]# lspci -nn | grep 0443 >> >> 00:06.0 Co-processor [0b40]: Intel Corporation Device [8086:0443] >> >> [root@vpn-s dpdk-2.2.0]# >> >> >> Installed Intel qat driver installation on VM (downloaded from >> https://01.org/packet-processing/intel%C2%AE-quickassist-technology-driv= ers-and-patches) >> and followed the >> >> https://01.org/sites/default/files/page/330750-005_qat_gsg.pdf guide. >> Note, used the following installation options i.e., ./installer.sh insta= ll >> QAT1.6 guest. After this , it detects QAT VF. >> >> >> [root@vpn-s QAT]# service qat_service status >> >> There is 1 acceleration device(s) in the system: >> >> icp_dev0 - type=3Ddh895xcc, inst_id=3D0, node_id=3D0, bdf=3D00:06:0, #= accel=3D1, >> #engines=3D1, state=3Dup >> >> [root@vpn-s QAT]# >> >> >> [root@vpn-s QAT]# lsmod | grep qa >> >> icp_qa_al_vf 813442 1 >> >> [root@vpn-s QAT]# >> >> >> >> The below are DPDK configuration options in >> config/defconfig_x86_64-native-linuxapp-gcc. >> >> >> include "common_linuxapp" >> >> >> >> CONFIG_RTE_MACHINE=3D"native" >> >> CONFIG_RTE_ARCH=3D"x86_64" >> >> CONFIG_RTE_ARCH_X86_64=3Dy >> >> CONFIG_RTE_ARCH_64=3Dy >> >> CONFIG_RTE_TOOLCHAIN=3D"gcc" >> >> CONFIG_RTE_TOOLCHAIN_GCC=3Dy >> >> CONFIG_RTE_LIBRTE_PMD_AESNI_MB=3Dy >> >> CONFIG_RTE_LIBRTE_CRYPTODEV_DEBUG=3Dy >> >> CONFIG_RTE_LIBRTE_VIRTIO_PMD=3Dy >> >> CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=3Dy >> >> CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=3Dy >> >> CONFIG_RTE_LIBRTE_PMD_QAT=3Dy >> >> CONFIG_RTE_QAT_PMD_MAX_NB_SESSIONS=3D128 >> >> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_INIT=3Dy >> >> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_TX=3Dy >> >> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_RX=3Dy >> >> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_DRIVER=3Dy >> >> >> >> The below is my setup.sh >> >> >> export RTE_SDK=3D/opt/dpdk-2.2.0 >> >> export RTE_TARGET=3Dx86_64-native-linuxapp-gcc >> >> export RTE_ARCH=3Dx86_64 >> >> export AESNI_MULTI_BUFFER_LIB_PATH=3D/opt/code >> >> make config T=3Dx86_64-native-linuxapp-gcc >> >> sed -ri 's,(PMD_PCAP=3D).*,\1y,' build/.config >> >> make install T=3Dx86_64-native-linuxapp-gcc >> >> mkdir -p /mnt/huge >> >> mount -t hugetlbfs nodev /mnt/huge >> >> echo 2048 > >> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages >> >> sudo modprobe uio_pci_generic >> >> sudo modprobe uio >> >> rmmod igb_uio >> >> insmod >> ./x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_u= io.ko >> >> echo -n "0000:00:06.0" > /sys/bus/pci/devices/0000\:00\:06.0/driver/unbi= nd >> >> echo "8086 0443" > /sys/bus/pci/drivers/igb_uio/new_id >> >> lspci -vvd:443 >> >> ./tools/dpdk_nic_bind.py --bind=3Duio_pci_generic eth1 >> >> ./tools/dpdk_nic_bind.py --bind=3Duio_pci_generic 00:03.0 >> >> ./tools/dpdk_nic_bind.py -s >> >> >> Also modified the cryptodevs_init() ( examples/ipsec-secgw/ipsec-secgw.c= ) >> to rte_cryptodev_start(cdev_id); after rte_cryptodev_queue_pair_setup(). >> The purpose of calling this to start the transmit and the receive units >> of the QAT device. It returns success. But still getting the same issue >> i.e. rte_cryptodev_dequeue_burst () does not return crypto packet. Note >> that, when I call rte_cryptodev_queue_pair_setup () with 3rd argument as >> NULL (in which case default configuration will be used) , it crashes. Is >> it a known issue? >> >> >> >> >> >> Regards, >> >> Chinmaya >> >> On Tue, Jun 21, 2016 at 9:10 PM, Sergio Gonzalez Monroy < >> sergio.gonzalez.monroy@intel.com> wrote: >> >>> On 21/06/2016 08:26, Chinmaya Dwibedy wrote: >>> >>>> >>>> Hi Sergio, >>>> >>>> >>>> As suggested by you, run =E2=80=98app/test' application and then run >>>> 'cryptodev_qat_autotest' . Found the same issue i.e., It halts in >>>> rte_cryptodev_dequeue_burst () (while loop) in process_crypto_request = () >>>> function (app/test/test_cryptodev.c). >>>> >>>> Debugged the qat_crypto_pkt_rx_burst() (in >>>> drivers/crypto/qat/qat_crypto.c) and found that , the value of resp_ms= g is >>>> 0x7F7F7F7F (i.e., ADF_RING_EMPTY_SIG) . Thus it is not getting crypto >>>> packet (i.e., rte_mbuf) from the RX queue. But I find that, in >>>> qat_crypto_pkt_tx_burst(), the qat_alg_write_mbuf_entry() is being ca= lled. >>>> The qat_alg_write_mbuf_entry () prints qat_req using rte_hexdump () a= nd >>>> returns zero. >>>> >>>> >>>> One question, the base_addr + tail of tx_q (in >>>> qat_crypto_pkt_tx_burst()) should be same as base_addr + head (in >>>> qat_crypto_pkt_rx_burst()). Right ? Please feel free to correct me if = I am >>>> wrong. >>>> >>>> >>>> Also can you please provide some pointers for further debugging? Would >>>> it be configuration issue? Note that, I am using VF in VM. >>>> >>>> >>> In theory you should be able to run ipsec-secgw sample app with QAT VF >>> as long as the setup is right. >>> That the autotest fails would likely point to some possible issues with >>> the setup. >>> >>> Have you tried to run the autotest in the host? >>> >>> Sergio >>> >>> >>>> Thank you once again for your prompt response and great support. >>>> >>>> >>>> Regards, >>>> >>>> Chinmaya >>>> >>>> >>>> On Mon, Jun 20, 2016 at 3:04 PM, Sergio Gonzalez Monroy < >>>> sergio.gonzalez.monroy@intel.com >>> sergio.gonzalez.monroy@intel.com>> wrote: >>>> >>>> On 20/06/2016 10:08, Chinmaya Dwibedy wrote: >>>> >>>> >>>> Hi Sergio, >>>> >>>> >>>> Agreed. We might not dequeue the same amount of crypto ops we >>>> just previously enqueued, it's asynchronous. But in this case, >>>> I have sent just one UDP packet. So there will be one crypto >>>> ops. Right? Also I put a sleep (50) after the >>>> rte_crypto_enqueue_burst() function in ipsec_processing() >>>> (ipsec.c) , so as to allow more time ( for QAT device) for >>>> processing. Still getting the same result i.e., the >>>> rte_crypto_dequeue_burst () functionreturns zero. >>>> >>>> >>>> In case of S/W crypto device (i.e., AESNI), the VM gets >>>> inbound UDP packets on Port 1/eth1, encapsulates (after >>>> consulting its SPD) in an IPsec ESP packet and sends to its >>>> peer through Port 0/eth0 interface. >>>> >>>> >>>> Yes, the security policy, security association and Routing >>>> entries/configurations are exactly same. Please feel free to >>>> let me know if you need additional information. >>>> >>>> >>>> Could you try to run 'app/test' application then run >>>> 'cryptodev_qat_autotest' ? That is a functional test for cryptodev >>>> QAT PMD. >>>> >>>> Sergio >>>> >>>> >>>> >>> >> >