From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf0-f43.google.com (mail-lf0-f43.google.com [209.85.215.43]) by dpdk.org (Postfix) with ESMTP id 8F25E5F3D for ; Wed, 21 Mar 2018 14:44:19 +0100 (CET) Received: by mail-lf0-f43.google.com with SMTP id x205-v6so7939871lfa.0 for ; Wed, 21 Mar 2018 06:44:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=jJDRNCtscoOrHOghtwSsQwae2QyCfbf6nXUM6QVFdf0=; b=f+rN2CiHBdfu0Le5+dvaS5xTO8LunEGXzWQCdE0jorbFOAvoDAFYQdgsnZN5xViu3z 2r/2Nsq/HCR2bvLdDiiDBIGCF0qmxaIje2uOvdY2XRNu3yMigCmcc9S7Fhj5ijO9DGrv mfYJepBsyz9Md/FV4q0sC3UYlFz837SthcfEsc5mmBpyHz9HiCm1OdZ8vC0V8Py/vRUC +FT1kuwBELX5wfIGhthW0BwdJtmguL5AjsrhvEZ0Up/5yTSeaCcZ+OLsh15KFnEw/Ttb lyOH7uvhxInI4fj0nGkRHdYmNiWp20LyFodX6fIeP7ymU7u5zC4V7otqlY6Fp7sA8SXG rANA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=jJDRNCtscoOrHOghtwSsQwae2QyCfbf6nXUM6QVFdf0=; b=TKv0oGTTrhDA0rPUbXQoXYvsJOC9ztTf1avYLjK4hdJP7Lbxv0k6W2GuLCJoaH3rTo 1XlHfXB04bvCJAazEj5c3E6+jnBfPf5z2g7tDJO6I5qf2n7uVo87OzlYpPtj7gsGvTAp v2rMnGI4BIA1MccIeuimsWCsCVmOgGMEGOKTQg/mTZjeoeE5TTGgxyiPeb0a+yfHJzjq 9qogT+YMApMYj7wp5urU693buWekt8ceqk4eTNnk1a8l42FX4nbhxL0CdMC9XHqLEpTp NPiY0XIusSTtYnIhsrKBu2oyByncp0VHoZTcOzr4k9lOiTjp+PFFW7rPS93dv4URq8FM 7sBg== X-Gm-Message-State: AElRT7FqzktyDpABGwiqwpOJKSsKLf5dJtSaaUY5pBh7vQK3/H083vsN QkI4nZtABoSdao2vQDHvqy3SAgW3KSA/+WhVRVo= X-Google-Smtp-Source: AG47ELta8HGJ7wHB+pANHki2F+hwnehPOdFBJclQ8gSDvAbPG6hjiAoaXlmLx+OpBF+wa407wKPMEKCgSmeV5p99PHA= X-Received: by 2002:a19:d945:: with SMTP id q66-v6mr10253209lfg.144.1521639858838; Wed, 21 Mar 2018 06:44:18 -0700 (PDT) MIME-Version: 1.0 Received: by 10.46.134.205 with HTTP; Wed, 21 Mar 2018 06:43:38 -0700 (PDT) From: "RAJESH KUMAR S.R" Date: Wed, 21 Mar 2018 19:13:38 +0530 Message-ID: To: "Rosen, Rami" , users@dpdk.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] [DPDK-PDUMP] Segmentation fault while Restarting PDUMP X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Mar 2018 13:44:19 -0000 Hi Rami, Got some temperary fix. My understanding of the issue is as follows. When the pdump is called for the first time, rte_eth_devices attach to new ports. rte_eth_dev_find_free_port() is called that returns 2 for rx_port and 3 for tx_port. Corresponding, tx_queues and rx_queues and other data are stored in rte_eth_device.data[2] & rte_eth_devices[3].data, obtained from rte_eth_dev_shared_data->data[2] and rte_eth_dev_shared_data->data[3] During the program exit, these data are not getting cleaned up. rte_eth_dev_shared_data->data[2].name and rte_eth_dev_shared_data->data[3].name have the old values (" net_pcap_rx_0","net_pcap_tx_0") rte_eth_dev_find_free_port() iterates over all ports and selects the first port that has no name. When pdump is restarted, rte_eth_devices are getting attached to new ports(rx_port at 4, tx_port at 5). So, rte_eth_dev_find_free_port() returns 4 for rx and 5 for tx instead of 2 and 3 which are free. In configure_vdev in pdump/main.c, the check (port_id > nb_ports) fails and returns. While debugging using gdb, I set ports as rx=3D 2 and tx=3D 3 when it was getting set to rx=3D4 and tx=3D5, it was working fine So, I reverted this DPDK Commit 8ee892a2385c50427c03db5cef1789babceb5999 ethdev: fix port id allocation for (i =3D 0; i < RTE_MAX_ETHPORTS; i++) \{ - if (rte_eth_devices\[i\].state =3D=3D RTE_ETH_DEV_UNUSED) + /* Using shared name field to find a free port. */ + if (rte_eth_dev_data\[i\].name[0] =3D=3D '\0') { + RTE_ASSERT(rte_eth_devices\[i\].state =3D=3D + RTE_ETH_DEV_UNUSED); return i; + \} \} return RTE_MAX_ETHPORTS; \} After reverting, it was working fine. I'm not sure whether this is the right way to do. Regards, Rajesh kumar S R On Tue, Mar 20, 2018 at 2:55 PM, RAJESH KUMAR S.R wrote: > Hi Rami, > > > I don't have a proper understanding of the whole flow. I'm not sure what = I > have missed. Need some help > Got some info while debugging. > > > There is a structure "rte_eth_dev_shared_data". The "data" field in this > structure has some old data from previous run. > > It is getting assigned in rte_eth_dev_shared_data() --> > > mz =3D rte_memzone_lookup(MZ_RTE_ETH_DEV_DATA); > .... > rte_eth_dev_shared_data =3D mz->addr; > > > a. > > Function: rte_eth_dev_find_free_port > backtrace: (create_mp_ring_vdev -->rte_eth_dev_attach((devargs=3D0x7ffff= fffe6e0 > "net_pcap_rx_0,tx_pcap=3D/tmp/rx.pcap", port_id=3D0x7fffffffe6ca)) > --> rte_eal_dev_attach -->rte_eal_hotplug_add --> vdev_plug---> > vdev_probe_all_drivers --> pmd_pcap_probe --> eth_from_pcaps --> > eth_from_pcaps_common -->pmd_init_internals --> rte_eth_vdev_allocate--> > rte_eth_dev_allocate --> rte_eth_dev_find_free_port) > > > During the first run, > > Port_ids that are allocated from rte_eth_dev_find_free_port function are > 2(rx) and 3(tx). > > The data values in rte_eth_dev_shared_data are > > rte_eth_dev_shared_data->data[2] > {name =3D "net_pcap_rx_0", .....} > > p rte_eth_dev_shared_data->data[3] > {name =3D "net_pcap_tx_0", ....} > > The packet capture is working fine. > > b. > > During the second run, > In function, rte_eth_dev_find_free_port > > ".... > if (rte_eth_dev_shared_data->data[i].name[0] =3D=3D '\0') { > RTE_ASSERT(rte_eth_devices[i].state =3D=3D > RTE_ETH_DEV_UNUSED); > return i; > } > ..." > > > p rte_eth_dev_shared_data->data[2] > $3 =3D {name =3D "net_pcap_rx_0", .....} > > p rte_eth_dev_shared_data->data[3] > $4 =3D {name =3D "net_pcap_tx_0", ....} > > > The previous data exists here. > The port_ids that is returned by rte_eth_dev_find_free_port are 4(rx) & > 5(tx). > > c. > pdump/main.c -> In configure_vdev() , for port_id=3D4 > const uint8_t nb_ports =3D rte_eth_dev_count(); > (=3D 3) > This check is failing > if (port_id > nb_ports) > return -1; > > > In configure_vdev() , for port_id=3D5 > const uint8_t nb_ports =3D rte_eth_dev_count(); > (=3D 4) > This check is failing > if (port_id > nb_ports) > return -1; > > rte_eth_dev_configure, rte_eth_dev_start are not getting called. > So, In the rte_eth_tx_burst (dump_packets -->pdump_rxtx-->rte_eth_tx_burs= t) > , some data in rte_eth_devices[port_id] seems to be NULL. > > > I'll update with more info > > > Regards, > Rajesh Kumar S R > > On Mon, Mar 19, 2018 at 4:54 PM, RAJESH KUMAR S.R > wrote: > >> Hi Rami, >> Thanks for the response. >> >> I used "echo 0 > /proc/sys/kernel/randomize_va_space " >> for disabling ASLR, but the error still exists. >> I checked with testpmd as primary process and the same error occurs when >> I restart the pdump tool. >> >> I'm just trying to debug the segmentation fault in rte_eth_tx_burst >> function for now. >> >> >> >> Thanks & Regards, >> Rajesh kumar S R >> >> On Mon, Mar 19, 2018 at 12:39 AM, Rosen, Rami >> wrote: >> >>> Hi Rajesh, >>> >>> >>> >>> First notice the warning you get: >>> >>> >EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in >>> the kernel. >>> >>> >EAL: This may cause issues with mapping memory into secondary >>> processes >>> >>> >>> >>> So first, I would suggest that you will disable ASLR and try again, to >>> verify it is not the cause. IIRC, in the past I had some issue with >>> dpdk-pdump when ASLR was not enabled (but IIRC the issue itself was >>> different =E2=80=93 probably not getting packets written to the pcap fi= le or >>> something else). >>> >>> Disabling ASLR is done easily: >>> >>> Probably you have : cat /proc/sys/kernel/randomize_va_space >>> >>> gives =E2=80=9C2=E2=80=9D >>> >>> To disable it run >>> >>> echo 0 > /proc/sys/kernel/randomize_va_space >>> >>> >>> >>> And also I would suggest to try it with testpmd. I actually ran it toda= y >>> with >>> >>> DPDK 18.02 with testpmd, and it worked for me. I sent L2 traffic with >>> scapy, and captured only RX packets with rx-dev; any other L2 packet >>> generator should be OK. >>> >>> >>> >>> Regards, >>> >>> Rami Rosen >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> *From:* RAJESH KUMAR S.R [mailto:rajuuu1992@gmail.com] >>> *Sent:* Sunday, March 18, 2018 19:52 >>> *To:* Rosen, Rami ; users@dpdk.org >>> *Subject:* Re: [dpdk-users] [DPDK-PDUMP] Issue: RING: Cannot reserve >>> memory >>> >>> >>> >>> >>> >>> Hi Rami, >>> >>> Thanks for the response. >>> >>> DPDK version I'm using is 18.02.0. >>> >>> The primary process I'm using is custom program which uses dpdk. >>> >>> I have initialized pdump in the primary process using >>> "rte_pdump_init(NULL)" similar to that in testpmd example. >>> >>> I'm using the pdump tool without making any changes. >>> >>> >>> >>> I enabled the following flags >>> 1. CONFIG_RTE_LIBRTE_PDUMP=3Dy >>> 2. CONFIG_RTE_LIBRTE_PMD_PCAP=3Dy >>> >>> I added the libpcap library >>> >>> The problem I'm facing is bit different from what I mentioned earlier. >>> >>> The first time when I run, I'm able to see the packets getting saved in >>> the output pcap file, rx.pcap and tx.pcap. >>> >>> The second time when I run, I'm getting a segmentation fault as I have >>> pasted the output in the previous message. >>> >>> I'm using gdb for debugging, the functions rte_eal_init(), >>> create_mp_ring_vdev(); >>> enable_pdump(); are working fine. >>> >>> >>> >>> The segmentation fault is occuring in rte_eth_tx_burst >>> function(dump_packets ->pdump_rxtx-->rte_eth_tx_burst) >>> >>> The rings tx_ring_0 and rx_ring_0 are not getting freed. I checked that >>> with rte_memzone_dump(). So, next time when I start the pdump tool I w= as >>> getting " memzone already exists . RING: Cannot reserve >>> memory" error. >>> >>> So, issue is in rte_eth_tx_burst function, where the >>> "dev->data->tx_queues[queue_id]" is NULL. >>> >>> >>> " >>> EAL: Detected lcore 0 as core 0 on socket 0 >>> EAL: Detected lcore 1 as core 0 on socket 0 >>> EAL: Support maximum 128 logical core(s) by configuration. >>> EAL: Detected 2 lcore(s) >>> EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or >>> directory) >>> EAL: VFIO PCI modules not loaded >>> EAL: Multi-process socket /var/run/.rte_unix_1817_14c6748457c5 >>> [New Thread 0x7ffff67d4700 (LWP 1821)] >>> EAL: Probing VFIO support... >>> EAL: Module /sys/module/vfio not found! error 2 (No such file or >>> directory) >>> EAL: VFIO modules not loaded, skipping VFIO support... >>> EAL: Module /sys/module/vfio not found! error 2 (No such file or >>> directory) >>> EAL: Setting up physically contiguous memory... >>> EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in >>> the kernel. >>> EAL: This may cause issues with mapping memory into secondary >>> processes >>> EAL: Analysing 300 files >>> EAL: Mapped segment 0 of size 0x400000 >>> EAL: Mapped segment 1 of size 0x200000 >>> EAL: Mapped segment 2 of size 0x200000 >>> EAL: Mapped segment 3 of size 0x200000 >>> EAL: Mapped segment 4 of size 0x24c00000 >>> EAL: Mapped segment 5 of size 0x200000 >>> EAL: TSC frequency is ~2904192 KHz >>> EAL: WARNING: cpu flags constant_tsc=3Dyes nonstop_tsc=3Dno -> using >>> unreliable clock cycles ! >>> EAL: Master lcore 0 is ready (tid=3Df7fec8c0;cpuset=3D[0]) >>> [New Thread 0x7ffff5fd3700 (LWP 1822)] >>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 >>> EAL: Invalid NUMA socket, default to 0 >>> EAL: probe driver: 1af4:1000 net_virtio >>> EAL: PCI Port IO found start=3D0xc060 >>> EAL: PCI device 0000:00:04.0 on NUMA socket -1 >>> EAL: Invalid NUMA socket, default to 0 >>> EAL: probe driver: 1af4:1000 net_virtio >>> EAL: PCI Port IO found start=3D0xc080 >>> EAL: PCI device 0000:00:05.0 on NUMA socket -1 >>> EAL: Invalid NUMA socket, default to 0 >>> EAL: probe driver: 1af4:1000 net_virtio >>> EAL: Requested device 0000:00:05.0 cannot be used >>> PMD: Initializing pmd_pcap for net_pcap_rx_0 >>> PMD: Creating pcap-backed ethdev on numa socket -1 >>> PMD: Initializing pmd_pcap for net_pcap_tx_0 >>> PMD: Creating pcap-backed ethdev on numa socket -1 >>> >>> Thread 1 "dpdk-pdump" received signal SIGSEGV, Segmentation fault. >>> 0x000000000043ee07 in rte_eth_tx_burst (port_id=3D4, queue_id=3D0, >>> tx_pkts=3D0x7fffffffe8c0, nb_pkts=3D1) >>> " >>> >>> I don't know why tx_queues is NULL. >>> >>> I'm currently checking where that is getting populated. >>> >>> >>> >>> Thanks & Regards, >>> >>> Rajesh kumar S R >>> >>> >>> >>> >>> >>> On Sat, Mar 17, 2018 at 9:28 PM, Rosen, Rami >>> wrote: >>> >>> Hi Rajesh, >>> >>> Can you please provide more details: >>> >>> - Which DPDK version are you using? >>> >>> - dpdk-pdump is a secondary process and must be launched along with >>> any primary process. Which is the primary process you are running ? >>> is it something that you developed on your own/took something from DPDK >>> examples (with/without extending it), etc ? >>> >>> Regards, >>> Rami Rosen >>> >>> >>> -----Original Message----- >>> From: users [mailto:users-bounces@dpdk.org] On Behalf Of RAJESH KUMAR >>> S.R >>> Sent: Monday, March 12, 2018 12:25 >>> To: users@dpdk.org >>> Subject: [dpdk-users] [DPDK-PDUMP] Issue: RING: Cannot reserve memory >>> >>> Hi, >>> >>> I am new to dpdk. >>> I'm trying to use dpdk-pdump tool. I'm able to capture packets on dpdk >>> ports. >>> >>> But, I'm facing the following issue >>> I'm getting a error "RING:Cannot reserve memory" while trying to restar= t >>> the pdump tool or running 2 instances of pdump. >>> I have also used the rte_eal_cleanup while exiting pdump main function. >>> >>> >>> Output: >>> * sudo /opt/pep/active/bin/dpdk-pdump -- --pdump >>> 'port=3D0,queue=3D*,rx-dev=3D/tmp/rx.pcap,tx-dev=3D/tmp/tx.pcap'* >>> >>> [sudo] password for admin: >>> EAL: Detected 2 lcore(s) >>> EAL: Multi-process socket /var/run/.rte_unix_1061_ce297dfe759 >>> EAL: Probing VFIO support... >>> EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in >>> the kernel. >>> EAL: This may cause issues with mapping memory into secondary >>> processes >>> EAL: WARNING: cpu flags constant_tsc=3Dyes nonstop_tsc=3Dno -> using >>> unreliable clock cycles ! >>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 >>> EAL: Invalid NUMA socket, default to 0 >>> EAL: probe driver: 1af4:1000 net_virtio >>> EAL: PCI device 0000:00:04.0 on NUMA socket -1 >>> EAL: Invalid NUMA socket, default to 0 >>> EAL: probe driver: 1af4:1000 net_virtio >>> EAL: PCI device 0000:00:05.0 on NUMA socket -1 >>> EAL: Invalid NUMA socket, default to 0 >>> EAL: probe driver: 1af4:1000 net_virtio >>> EAL: Requested device 0000:00:05.0 cannot be used >>> PMD: Initializing pmd_pcap for net_pcap_rx_0 >>> PMD: Creating pcap-backed ethdev on numa socket -1 Port 2 MAC: 00 00 00 >>> 01 02 03 >>> PMD: Initializing pmd_pcap for net_pcap_tx_0 >>> PMD: Creating pcap-backed ethdev on numa socket -1 Port 3 MAC: 00 00 00 >>> 01 02 03 ^C >>> >>> Signal 2 received, preparing to exit... >>> ##### PDUMP DEBUG STATS ##### >>> -packets dequeued: 4 >>> -packets transmitted to vdev: 4 >>> -packets freed: 0 >>> >>> >>> Restarting........ >>> >>> *> sudo /opt/pep/active/bin/dpdk-pdump -- --pdump >>> 'port=3D0,queue=3D*,rx-dev=3D/tmp/rx.pcap,tx-dev=3D/tmp/tx.pcap'* >>> EAL: Detected 2 lcore(s) >>> EAL: Multi-process socket /var/run/.rte_unix_1073_cea9cc51241 >>> EAL: Probing VFIO support... >>> EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in >>> the kernel. >>> EAL: This may cause issues with mapping memory into secondary >>> processes >>> EAL: WARNING: cpu flags constant_tsc=3Dyes nonstop_tsc=3Dno -> using >>> unreliable clock cycles ! >>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 >>> EAL: Invalid NUMA socket, default to 0 >>> EAL: probe driver: 1af4:1000 net_virtio >>> EAL: PCI device 0000:00:04.0 on NUMA socket -1 >>> EAL: Invalid NUMA socket, default to 0 >>> EAL: probe driver: 1af4:1000 net_virtio >>> EAL: PCI device 0000:00:05.0 on NUMA socket -1 >>> EAL: Invalid NUMA socket, default to 0 >>> EAL: probe driver: 1af4:1000 net_virtio >>> EAL: Requested device 0000:00:05.0 cannot be used >>> PMD: Initializing pmd_pcap for net_pcap_rx_0 >>> PMD: Creating pcap-backed ethdev on numa socket -1 >>> PMD: Initializing pmd_pcap for net_pcap_tx_0 >>> PMD: Creating pcap-backed ethdev on numa socket -1 Segmentation fault >>> >>> *> sudo /opt/pep/active/bin/dpdk-pdump -- --pdump >>> 'port=3D0,queue=3D*,rx-dev=3D/tmp/rx.pcap,tx-dev=3D/tmp/tx.pcap'* >>> >>> EAL: Detected 2 lcore(s) >>> EAL: Multi-process socket /var/run/.rte_unix_1087_cec3ab006b1 >>> EAL: Probing VFIO support... >>> EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in >>> the kernel. >>> EAL: This may cause issues with mapping memory into secondary >>> processes >>> EAL: WARNING: cpu flags constant_tsc=3Dyes nonstop_tsc=3Dno -> using >>> unreliable clock cycles ! >>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 >>> EAL: Invalid NUMA socket, default to 0 >>> EAL: probe driver: 1af4:1000 net_virtio >>> EAL: PCI device 0000:00:04.0 on NUMA socket -1 >>> EAL: Invalid NUMA socket, default to 0 >>> EAL: probe driver: 1af4:1000 net_virtio >>> EAL: PCI device 0000:00:05.0 on NUMA socket -1 >>> EAL: Invalid NUMA socket, default to 0 >>> EAL: probe driver: 1af4:1000 net_virtio >>> EAL: Requested device 0000:00:05.0 cannot be used >>> RING: Cannot reserve memory >>> EAL: Error - exiting with code: 1 >>> Cause: File exists:create_mp_ring_vdev:634 >>> >>> >>> >>> Hugepage info: >>> cat /proc/meminfo | grep Huge >>> AnonHugePages: 6144 kB >>> HugePages_Total: 300 >>> HugePages_Free: 0 >>> HugePages_Rsvd: 0 >>> HugePages_Surp: 0 >>> Hugepagesize: 2048 kB >>> >>> >>> Can you please help me in finding the issue. >>> >>> >>> >>> Thanks, >>> Rajesh kumar S R >>> >>> >>> >> >> >