* [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? @ 2014-03-04 15:23 Srinivasan J 2014-03-05 14:28 ` Prashant Upadhyaya 2014-03-05 14:55 ` [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? David Marchand 0 siblings, 2 replies; 21+ messages in thread From: Srinivasan J @ 2014-03-04 15:23 UTC (permalink / raw) To: dev I want to try dpdk with vmxnet3 in Vmware Esxi 5.1. I see that the latest dpdk 1.6.0r1 includes a vmxnet3 pmd. The vmxnet3-usermap-1.1.tar.gz as well includes a vmxnet3 pmd driver. I'm confused as to which vmxnet3 pmd driver to use along with which vmxnet3 kernel driver (vmxnet3 native kernel driver or vmxnet3-usermap kernel driver). I also want to try RSS with vmxnet3 and dpdk. As per Intel DPDK Programmer's Guide (January 2014) RSS is supported with vmxnet3 since dpdk version 1.6.0. Thanks, Srini ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? 2014-03-04 15:23 [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? Srinivasan J @ 2014-03-05 14:28 ` Prashant Upadhyaya 2014-03-05 15:10 ` David Marchand 2014-03-05 14:55 ` [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? David Marchand 1 sibling, 1 reply; 21+ messages in thread From: Prashant Upadhyaya @ 2014-03-05 14:28 UTC (permalink / raw) To: Srinivasan J, dev Hi, I am also keen to know the answer to the question posted by Srini. The real question is -- is dpdk1.6.0r1 self-sufficient so that I don't need any extensions etc., or do I still need something from outside like the usermap kernel driver etc. Secondly, if I turn on all the debug options for the vmxnet3 pmd in the config file, 1.6.0r1 compilation runs into a problem and reports a function which is defined but not used. I am trying to bring up DPDK inside Fedora18 Guest on ESXi -- when I used DPDK1.6.0r1 (without debug options turned on for vmxnet3 pmd) the igb_uio could take over the vmxnet3 NIC but I encountered a core dump in the dev init function for the vmxnet3 driver -- anybody encountered a similar issue ? Regards -Prashant -----Original Message----- From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Srinivasan J Sent: Tuesday, March 04, 2014 8:53 PM To: dev@dpdk.org Subject: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? I want to try dpdk with vmxnet3 in Vmware Esxi 5.1. I see that the latest dpdk 1.6.0r1 includes a vmxnet3 pmd. The vmxnet3-usermap-1.1.tar.gz as well includes a vmxnet3 pmd driver. I'm confused as to which vmxnet3 pmd driver to use along with which vmxnet3 kernel driver (vmxnet3 native kernel driver or vmxnet3-usermap kernel driver). I also want to try RSS with vmxnet3 and dpdk. As per Intel DPDK Programmer's Guide (January 2014) RSS is supported with vmxnet3 since dpdk version 1.6.0. Thanks, Srini =============================================================================== Please refer to http://www.aricent.com/legal/email_disclaimer.html for important disclosures regarding this electronic communication. =============================================================================== ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? 2014-03-05 14:28 ` Prashant Upadhyaya @ 2014-03-05 15:10 ` David Marchand 2014-03-05 15:30 ` Prashant Upadhyaya 2014-03-06 6:50 ` Prashant Upadhyaya 0 siblings, 2 replies; 21+ messages in thread From: David Marchand @ 2014-03-05 15:10 UTC (permalink / raw) To: Prashant Upadhyaya; +Cc: dev Hello Prashant, On Wed, Mar 5, 2014 at 3:28 PM, Prashant Upadhyaya < prashant.upadhyaya@aricent.com> wrote: > Hi, > > I am also keen to know the answer to the question posted by Srini. > The real question is -- is dpdk1.6.0r1 self-sufficient so that I don't > need any extensions etc., or do I still need something from outside like > the usermap kernel driver etc. > > Secondly, if I turn on all the debug options for the vmxnet3 pmd in the > config file, 1.6.0r1 compilation runs into a problem and reports a function > which is defined but not used. > Can you send your build error ? (maybe in a separate thread ?) > I am trying to bring up DPDK inside Fedora18 Guest on ESXi -- when I used > DPDK1.6.0r1 (without debug options turned on for vmxnet3 pmd) the igb_uio > could take over the vmxnet3 NIC but I encountered a core dump in the dev > init function for the vmxnet3 driver -- anybody encountered a similar issue > ? > I encountered these problems as well. - igb_uio module does not check if you disable vmxnet3-uio pmds, it will always try to take over vmxnet3 devices. I have a patch waiting in my working dir to cleanly disable vmxnet3-uio pmd. - If you don't bind vmxnet3 devices to uio, but forget to enable vmxnet3-usermap pmd (by specifiying -d librte_pmd_vmxnet3.so), then internal vmxnet3-uio pmd will try to initialise and crash. I did not look any deeper into this, the easiest way is to disable vmxnet3-uio pmd + apply the patch I will send in a few minutes, as a first workaround. Regards, -- David Marchand ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? 2014-03-05 15:10 ` David Marchand @ 2014-03-05 15:30 ` Prashant Upadhyaya 2014-03-06 6:50 ` Prashant Upadhyaya 1 sibling, 0 replies; 21+ messages in thread From: Prashant Upadhyaya @ 2014-03-05 15:30 UTC (permalink / raw) To: David Marchand; +Cc: dev Hi David, The compilation error with debug flags on was that the functions -- vmxnet3_txq_dump, vmxnet3_rxq_dump - are defined but not used. Not a serious error, I will try to get rid of the compiler flag which generates this. However, I must reiterate, I _did_ bind my vmxnet3 device with igb_uio (and I did not use any .so, because I was intending to use the builtin vmxnet3 driver of dpdk 1.6.0r1), the bind succeeded, but then when I started the application, the dev init for vmxnet3 gave a core dump. Your patch and solution seems to be suggesting the reverse, i.e. when I don't bind with igb_uio but try to use the native driver. So please do try the above combination as well. Regards -Prashant From: David Marchand [mailto:david.marchand@6wind.com] Sent: Wednesday, March 05, 2014 8:41 PM To: Prashant Upadhyaya Cc: Srinivasan J; dev@dpdk.org Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? Hello Prashant, On Wed, Mar 5, 2014 at 3:28 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com<mailto:prashant.upadhyaya@aricent.com>> wrote: Hi, I am also keen to know the answer to the question posted by Srini. The real question is -- is dpdk1.6.0r1 self-sufficient so that I don't need any extensions etc., or do I still need something from outside like the usermap kernel driver etc. Secondly, if I turn on all the debug options for the vmxnet3 pmd in the config file, 1.6.0r1 compilation runs into a problem and reports a function which is defined but not used. Can you send your build error ? (maybe in a separate thread ?) I am trying to bring up DPDK inside Fedora18 Guest on ESXi -- when I used DPDK1.6.0r1 (without debug options turned on for vmxnet3 pmd) the igb_uio could take over the vmxnet3 NIC but I encountered a core dump in the dev init function for the vmxnet3 driver -- anybody encountered a similar issue ? I encountered these problems as well. - igb_uio module does not check if you disable vmxnet3-uio pmds, it will always try to take over vmxnet3 devices. I have a patch waiting in my working dir to cleanly disable vmxnet3-uio pmd. - If you don't bind vmxnet3 devices to uio, but forget to enable vmxnet3-usermap pmd (by specifiying -d librte_pmd_vmxnet3.so), then internal vmxnet3-uio pmd will try to initialise and crash. I did not look any deeper into this, the easiest way is to disable vmxnet3-uio pmd + apply the patch I will send in a few minutes, as a first workaround. Regards, -- David Marchand =============================================================================== Please refer to http://www.aricent.com/legal/email_disclaimer.html for important disclosures regarding this electronic communication. =============================================================================== ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? 2014-03-05 15:10 ` David Marchand 2014-03-05 15:30 ` Prashant Upadhyaya @ 2014-03-06 6:50 ` Prashant Upadhyaya 2014-03-06 7:18 ` Prashant Upadhyaya 1 sibling, 1 reply; 21+ messages in thread From: Prashant Upadhyaya @ 2014-03-06 6:50 UTC (permalink / raw) To: David Marchand; +Cc: dev Hi, Some further update on the crash I am facing. I am using DPDK1.6.0r1 and take over the vmxnet3 with igb_uio and then start the application. (so no external ko or vmxnet usermap etc.) During the port initializations, the crash is happening in the following function - eth_vmxnet3_dev_init and the crash is happening at the following line - /* Check h/w version compatibility with driver. */ ver = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_VRRS); Any hints regarding what could be wrong ? Regards -Prashant From: Prashant Upadhyaya Sent: Wednesday, March 05, 2014 9:01 PM To: 'David Marchand' Cc: Srinivasan J; dev@dpdk.org Subject: RE: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? Hi David, The compilation error with debug flags on was that the functions -- vmxnet3_txq_dump, vmxnet3_rxq_dump - are defined but not used. Not a serious error, I will try to get rid of the compiler flag which generates this. However, I must reiterate, I _did_ bind my vmxnet3 device with igb_uio (and I did not use any .so, because I was intending to use the builtin vmxnet3 driver of dpdk 1.6.0r1), the bind succeeded, but then when I started the application, the dev init for vmxnet3 gave a core dump. Your patch and solution seems to be suggesting the reverse, i.e. when I don't bind with igb_uio but try to use the native driver. So please do try the above combination as well. Regards -Prashant From: David Marchand [mailto:david.marchand@6wind.com] Sent: Wednesday, March 05, 2014 8:41 PM To: Prashant Upadhyaya Cc: Srinivasan J; dev@dpdk.org<mailto:dev@dpdk.org> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? Hello Prashant, On Wed, Mar 5, 2014 at 3:28 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com<mailto:prashant.upadhyaya@aricent.com>> wrote: Hi, I am also keen to know the answer to the question posted by Srini. The real question is -- is dpdk1.6.0r1 self-sufficient so that I don't need any extensions etc., or do I still need something from outside like the usermap kernel driver etc. Secondly, if I turn on all the debug options for the vmxnet3 pmd in the config file, 1.6.0r1 compilation runs into a problem and reports a function which is defined but not used. Can you send your build error ? (maybe in a separate thread ?) I am trying to bring up DPDK inside Fedora18 Guest on ESXi -- when I used DPDK1.6.0r1 (without debug options turned on for vmxnet3 pmd) the igb_uio could take over the vmxnet3 NIC but I encountered a core dump in the dev init function for the vmxnet3 driver -- anybody encountered a similar issue ? I encountered these problems as well. - igb_uio module does not check if you disable vmxnet3-uio pmds, it will always try to take over vmxnet3 devices. I have a patch waiting in my working dir to cleanly disable vmxnet3-uio pmd. - If you don't bind vmxnet3 devices to uio, but forget to enable vmxnet3-usermap pmd (by specifiying -d librte_pmd_vmxnet3.so), then internal vmxnet3-uio pmd will try to initialise and crash. I did not look any deeper into this, the easiest way is to disable vmxnet3-uio pmd + apply the patch I will send in a few minutes, as a first workaround. Regards, -- David Marchand =============================================================================== Please refer to http://www.aricent.com/legal/email_disclaimer.html for important disclosures regarding this electronic communication. =============================================================================== ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? 2014-03-06 6:50 ` Prashant Upadhyaya @ 2014-03-06 7:18 ` Prashant Upadhyaya 2014-03-07 3:38 ` Srinivasan J 0 siblings, 1 reply; 21+ messages in thread From: Prashant Upadhyaya @ 2014-03-06 7:18 UTC (permalink / raw) To: David Marchand; +Cc: dev And if it is any help, here is the startup log -- EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 1 on socket 0 EAL: Detected lcore 2 as core 2 on socket 0 EAL: Detected lcore 3 as core 3 on socket 0 EAL: Detected lcore 4 as core 4 on socket 0 EAL: Detected lcore 5 as core 5 on socket 0 EAL: Detected lcore 6 as core 6 on socket 0 EAL: Detected lcore 7 as core 7 on socket 0 EAL: Skip lcore 8 (not detected) EAL: Skip lcore 9 (not detected) EAL: Skip lcore 10 (not detected) EAL: Skip lcore 11 (not detected) EAL: Skip lcore 12 (not detected) EAL: Skip lcore 13 (not detected) EAL: Skip lcore 14 (not detected) EAL: Skip lcore 15 (not detected) EAL: Skip lcore 16 (not detected) EAL: Skip lcore 17 (not detected) EAL: Skip lcore 18 (not detected) EAL: Skip lcore 19 (not detected) EAL: Skip lcore 20 (not detected) EAL: Skip lcore 21 (not detected) EAL: Skip lcore 22 (not detected) EAL: Skip lcore 23 (not detected) EAL: Skip lcore 24 (not detected) EAL: Skip lcore 25 (not detected) EAL: Skip lcore 26 (not detected) EAL: Skip lcore 27 (not detected) EAL: Skip lcore 28 (not detected) EAL: Skip lcore 29 (not detected) EAL: Skip lcore 30 (not detected) EAL: Skip lcore 31 (not detected) EAL: Skip lcore 32 (not detected) EAL: Skip lcore 33 (not detected) EAL: Skip lcore 34 (not detected) EAL: Skip lcore 35 (not detected) EAL: Skip lcore 36 (not detected) EAL: Skip lcore 37 (not detected) EAL: Skip lcore 38 (not detected) EAL: Skip lcore 39 (not detected) EAL: Skip lcore 40 (not detected) EAL: Skip lcore 41 (not detected) EAL: Skip lcore 42 (not detected) EAL: Skip lcore 43 (not detected) EAL: Skip lcore 44 (not detected) EAL: Skip lcore 45 (not detected) EAL: Skip lcore 46 (not detected) EAL: Skip lcore 47 (not detected) EAL: Skip lcore 48 (not detected) EAL: Skip lcore 49 (not detected) EAL: Skip lcore 50 (not detected) EAL: Skip lcore 51 (not detected) EAL: Skip lcore 52 (not detected) EAL: Skip lcore 53 (not detected) EAL: Skip lcore 54 (not detected) EAL: Skip lcore 55 (not detected) EAL: Skip lcore 56 (not detected) EAL: Skip lcore 57 (not detected) EAL: Skip lcore 58 (not detected) EAL: Skip lcore 59 (not detected) EAL: Skip lcore 60 (not detected) EAL: Skip lcore 61 (not detected) EAL: Skip lcore 62 (not detected) EAL: Skip lcore 63 (not detected) EAL: Setting up memory... EAL: Ask a virtual area of 0x80000000 bytes EAL: Virtual area found at 0x7f848ae00000 (size = 0x80000000) EAL: Requesting 1024 pages of size 2MB from socket 0 EAL: TSC frequency is ~2000000 KHz EAL: Master core 0 is ready (tid=b3f3f00) EAL: Core 1 is ready (tid=8a1f2700) EAL: Core 2 is ready (tid=899f1700) EAL: Core 3 is ready (tid=891f0700) EAL: Core 4 is ready (tid=889ef700) EAL: Core 5 is ready (tid=7bfff700) EAL: Core 6 is ready (tid=7b7fe700) EAL: Core 7 is ready (tid=7affd700) Pool initialized Global Variables initialized PMD: rte_vmxnet3_pmd_init(): >> EAL: PCI device 0000:0b:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd PMD: eth_vmxnet3_dev_init(): >> Segmentation fault (core dumped) -----Original Message----- From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Prashant Upadhyaya Sent: Thursday, March 06, 2014 12:20 PM To: David Marchand Cc: dev@dpdk.org Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? Hi, Some further update on the crash I am facing. I am using DPDK1.6.0r1 and take over the vmxnet3 with igb_uio and then start the application. (so no external ko or vmxnet usermap etc.) During the port initializations, the crash is happening in the following function - eth_vmxnet3_dev_init and the crash is happening at the following line - /* Check h/w version compatibility with driver. */ ver = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_VRRS); Any hints regarding what could be wrong ? Regards -Prashant From: Prashant Upadhyaya Sent: Wednesday, March 05, 2014 9:01 PM To: 'David Marchand' Cc: Srinivasan J; dev@dpdk.org Subject: RE: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? Hi David, The compilation error with debug flags on was that the functions -- vmxnet3_txq_dump, vmxnet3_rxq_dump - are defined but not used. Not a serious error, I will try to get rid of the compiler flag which generates this. However, I must reiterate, I _did_ bind my vmxnet3 device with igb_uio (and I did not use any .so, because I was intending to use the builtin vmxnet3 driver of dpdk 1.6.0r1), the bind succeeded, but then when I started the application, the dev init for vmxnet3 gave a core dump. Your patch and solution seems to be suggesting the reverse, i.e. when I don't bind with igb_uio but try to use the native driver. So please do try the above combination as well. Regards -Prashant From: David Marchand [mailto:david.marchand@6wind.com] Sent: Wednesday, March 05, 2014 8:41 PM To: Prashant Upadhyaya Cc: Srinivasan J; dev@dpdk.org<mailto:dev@dpdk.org> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? Hello Prashant, On Wed, Mar 5, 2014 at 3:28 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com<mailto:prashant.upadhyaya@aricent.com>> wrote: Hi, I am also keen to know the answer to the question posted by Srini. The real question is -- is dpdk1.6.0r1 self-sufficient so that I don't need any extensions etc., or do I still need something from outside like the usermap kernel driver etc. Secondly, if I turn on all the debug options for the vmxnet3 pmd in the config file, 1.6.0r1 compilation runs into a problem and reports a function which is defined but not used. Can you send your build error ? (maybe in a separate thread ?) I am trying to bring up DPDK inside Fedora18 Guest on ESXi -- when I used DPDK1.6.0r1 (without debug options turned on for vmxnet3 pmd) the igb_uio could take over the vmxnet3 NIC but I encountered a core dump in the dev init function for the vmxnet3 driver -- anybody encountered a similar issue ? I encountered these problems as well. - igb_uio module does not check if you disable vmxnet3-uio pmds, it will always try to take over vmxnet3 devices. I have a patch waiting in my working dir to cleanly disable vmxnet3-uio pmd. - If you don't bind vmxnet3 devices to uio, but forget to enable vmxnet3-usermap pmd (by specifiying -d librte_pmd_vmxnet3.so), then internal vmxnet3-uio pmd will try to initialise and crash. I did not look any deeper into this, the easiest way is to disable vmxnet3-uio pmd + apply the patch I will send in a few minutes, as a first workaround. Regards, -- David Marchand =============================================================================== Please refer to http://www.aricent.com/legal/email_disclaimer.html for important disclosures regarding this electronic communication. =============================================================================== =============================================================================== Please refer to http://www.aricent.com/legal/email_disclaimer.html for important disclosures regarding this electronic communication. =============================================================================== ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? 2014-03-06 7:18 ` Prashant Upadhyaya @ 2014-03-07 3:38 ` Srinivasan J 2014-03-07 5:13 ` Prashant Upadhyaya 0 siblings, 1 reply; 21+ messages in thread From: Srinivasan J @ 2014-03-07 3:38 UTC (permalink / raw) To: Prashant Upadhyaya; +Cc: dev Prashant, Are you still hitting a crash? Do you have hugetlbfs enabled? if not try adding enabling hugetlbfs. I was able to get vxnet3 up using the vmxnet3 pmd alone in DPDK 1.60r1. I did not use the vmxnet3-usermap kernel driver. I am using Esxi 5.5 Eg: echo 1024 > /proc/sys/vm/nr_hugepages; mkdir /huge; mount -t hugetlbfs nodev /huge; On Thu, Mar 6, 2014 at 12:48 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: > And if it is any help, here is the startup log -- > > EAL: Detected lcore 0 as core 0 on socket 0 > EAL: Detected lcore 1 as core 1 on socket 0 > EAL: Detected lcore 2 as core 2 on socket 0 > EAL: Detected lcore 3 as core 3 on socket 0 > EAL: Detected lcore 4 as core 4 on socket 0 > EAL: Detected lcore 5 as core 5 on socket 0 > EAL: Detected lcore 6 as core 6 on socket 0 > EAL: Detected lcore 7 as core 7 on socket 0 > EAL: Skip lcore 8 (not detected) > EAL: Skip lcore 9 (not detected) > EAL: Skip lcore 10 (not detected) > EAL: Skip lcore 11 (not detected) > EAL: Skip lcore 12 (not detected) > EAL: Skip lcore 13 (not detected) > EAL: Skip lcore 14 (not detected) > EAL: Skip lcore 15 (not detected) > EAL: Skip lcore 16 (not detected) > EAL: Skip lcore 17 (not detected) > EAL: Skip lcore 18 (not detected) > EAL: Skip lcore 19 (not detected) > EAL: Skip lcore 20 (not detected) > EAL: Skip lcore 21 (not detected) > EAL: Skip lcore 22 (not detected) > EAL: Skip lcore 23 (not detected) > EAL: Skip lcore 24 (not detected) > EAL: Skip lcore 25 (not detected) > EAL: Skip lcore 26 (not detected) > EAL: Skip lcore 27 (not detected) > EAL: Skip lcore 28 (not detected) > EAL: Skip lcore 29 (not detected) > EAL: Skip lcore 30 (not detected) > EAL: Skip lcore 31 (not detected) > EAL: Skip lcore 32 (not detected) > EAL: Skip lcore 33 (not detected) > EAL: Skip lcore 34 (not detected) > EAL: Skip lcore 35 (not detected) > EAL: Skip lcore 36 (not detected) > EAL: Skip lcore 37 (not detected) > EAL: Skip lcore 38 (not detected) > EAL: Skip lcore 39 (not detected) > EAL: Skip lcore 40 (not detected) > EAL: Skip lcore 41 (not detected) > EAL: Skip lcore 42 (not detected) > EAL: Skip lcore 43 (not detected) > EAL: Skip lcore 44 (not detected) > EAL: Skip lcore 45 (not detected) > EAL: Skip lcore 46 (not detected) > EAL: Skip lcore 47 (not detected) > EAL: Skip lcore 48 (not detected) > EAL: Skip lcore 49 (not detected) > EAL: Skip lcore 50 (not detected) > EAL: Skip lcore 51 (not detected) > EAL: Skip lcore 52 (not detected) > EAL: Skip lcore 53 (not detected) > EAL: Skip lcore 54 (not detected) > EAL: Skip lcore 55 (not detected) > EAL: Skip lcore 56 (not detected) > EAL: Skip lcore 57 (not detected) > EAL: Skip lcore 58 (not detected) > EAL: Skip lcore 59 (not detected) > EAL: Skip lcore 60 (not detected) > EAL: Skip lcore 61 (not detected) > EAL: Skip lcore 62 (not detected) > EAL: Skip lcore 63 (not detected) > EAL: Setting up memory... > EAL: Ask a virtual area of 0x80000000 bytes > EAL: Virtual area found at 0x7f848ae00000 (size = 0x80000000) > EAL: Requesting 1024 pages of size 2MB from socket 0 > EAL: TSC frequency is ~2000000 KHz > EAL: Master core 0 is ready (tid=b3f3f00) > EAL: Core 1 is ready (tid=8a1f2700) > EAL: Core 2 is ready (tid=899f1700) > EAL: Core 3 is ready (tid=891f0700) > EAL: Core 4 is ready (tid=889ef700) > EAL: Core 5 is ready (tid=7bfff700) > EAL: Core 6 is ready (tid=7b7fe700) > EAL: Core 7 is ready (tid=7affd700) > Pool initialized > Global Variables initialized > PMD: rte_vmxnet3_pmd_init(): >> > EAL: PCI device 0000:0b:00.0 on NUMA socket -1 > EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd > PMD: eth_vmxnet3_dev_init(): >> > Segmentation fault (core dumped) > > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Prashant Upadhyaya > Sent: Thursday, March 06, 2014 12:20 PM > To: David Marchand > Cc: dev@dpdk.org > Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? > > Hi, > > Some further update on the crash I am facing. > I am using DPDK1.6.0r1 and take over the vmxnet3 with igb_uio and then start the application. (so no external ko or vmxnet usermap etc.) During the port initializations, the crash is happening in the following function - > > eth_vmxnet3_dev_init > > and the crash is happening at the following line - > /* Check h/w version compatibility with driver. */ > ver = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_VRRS); > > Any hints regarding what could be wrong ? > > Regards > -Prashant > > > From: Prashant Upadhyaya > Sent: Wednesday, March 05, 2014 9:01 PM > To: 'David Marchand' > Cc: Srinivasan J; dev@dpdk.org > Subject: RE: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? > > Hi David, > > The compilation error with debug flags on was that the functions -- vmxnet3_txq_dump, vmxnet3_rxq_dump - are defined but not used. > Not a serious error, I will try to get rid of the compiler flag which generates this. > > However, I must reiterate, I _did_ bind my vmxnet3 device with igb_uio (and I did not use any .so, because I was intending to use the builtin vmxnet3 driver of dpdk 1.6.0r1), the bind succeeded, but then when I started the application, the dev init for vmxnet3 gave a core dump. > Your patch and solution seems to be suggesting the reverse, i.e. when I don't bind with igb_uio but try to use the native driver. So please do try the above combination as well. > > Regards > -Prashant > > > From: David Marchand [mailto:david.marchand@6wind.com] > Sent: Wednesday, March 05, 2014 8:41 PM > To: Prashant Upadhyaya > Cc: Srinivasan J; dev@dpdk.org<mailto:dev@dpdk.org> > Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? > > Hello Prashant, > > On Wed, Mar 5, 2014 at 3:28 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com<mailto:prashant.upadhyaya@aricent.com>> wrote: > Hi, > > I am also keen to know the answer to the question posted by Srini. > The real question is -- is dpdk1.6.0r1 self-sufficient so that I don't need any extensions etc., or do I still need something from outside like the usermap kernel driver etc. > > Secondly, if I turn on all the debug options for the vmxnet3 pmd in the config file, 1.6.0r1 compilation runs into a problem and reports a function which is defined but not used. > > Can you send your build error ? (maybe in a separate thread ?) > > > I am trying to bring up DPDK inside Fedora18 Guest on ESXi -- when I used DPDK1.6.0r1 (without debug options turned on for vmxnet3 pmd) the igb_uio could take over the vmxnet3 NIC but I encountered a core dump in the dev init function for the vmxnet3 driver -- anybody encountered a similar issue ? > > I encountered these problems as well. > > - igb_uio module does not check if you disable vmxnet3-uio pmds, it will always try to take over vmxnet3 devices. > I have a patch waiting in my working dir to cleanly disable vmxnet3-uio pmd. > > - If you don't bind vmxnet3 devices to uio, but forget to enable vmxnet3-usermap pmd (by specifiying -d librte_pmd_vmxnet3.so), then internal vmxnet3-uio pmd will try to initialise and crash. > I did not look any deeper into this, the easiest way is to disable vmxnet3-uio pmd + apply the patch I will send in a few minutes, as a first workaround. > > > Regards, > -- > David Marchand > > > > > =============================================================================== > Please refer to http://www.aricent.com/legal/email_disclaimer.html > for important disclosures regarding this electronic communication. > =============================================================================== > > > > > =============================================================================== > Please refer to http://www.aricent.com/legal/email_disclaimer.html > for important disclosures regarding this electronic communication. > =============================================================================== ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? 2014-03-07 3:38 ` Srinivasan J @ 2014-03-07 5:13 ` Prashant Upadhyaya 2014-03-08 19:08 ` Srinivasan J 0 siblings, 1 reply; 21+ messages in thread From: Prashant Upadhyaya @ 2014-03-07 5:13 UTC (permalink / raw) To: Srinivasan J; +Cc: dev Hi Srini, Yes, I am still stuck, I am using ESXi5.1 though and Fedora18 in Guest. Please let me know what is the guest OS distro you are using in the VM. I'll try with that once. Regards -Prashant -----Original Message----- From: Srinivasan J [mailto:srinidpdk@gmail.com] Sent: Friday, March 07, 2014 9:08 AM To: Prashant Upadhyaya Cc: David Marchand; dev@dpdk.org Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? Prashant, Are you still hitting a crash? Do you have hugetlbfs enabled? if not try adding enabling hugetlbfs. I was able to get vxnet3 up using the vmxnet3 pmd alone in DPDK 1.60r1. I did not use the vmxnet3-usermap kernel driver. I am using Esxi 5.5 Eg: echo 1024 > /proc/sys/vm/nr_hugepages; mkdir /huge; mount -t hugetlbfs nodev /huge; On Thu, Mar 6, 2014 at 12:48 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: > And if it is any help, here is the startup log -- > > EAL: Detected lcore 0 as core 0 on socket 0 > EAL: Detected lcore 1 as core 1 on socket 0 > EAL: Detected lcore 2 as core 2 on socket 0 > EAL: Detected lcore 3 as core 3 on socket 0 > EAL: Detected lcore 4 as core 4 on socket 0 > EAL: Detected lcore 5 as core 5 on socket 0 > EAL: Detected lcore 6 as core 6 on socket 0 > EAL: Detected lcore 7 as core 7 on socket 0 > EAL: Skip lcore 8 (not detected) > EAL: Skip lcore 9 (not detected) > EAL: Skip lcore 10 (not detected) > EAL: Skip lcore 11 (not detected) > EAL: Skip lcore 12 (not detected) > EAL: Skip lcore 13 (not detected) > EAL: Skip lcore 14 (not detected) > EAL: Skip lcore 15 (not detected) > EAL: Skip lcore 16 (not detected) > EAL: Skip lcore 17 (not detected) > EAL: Skip lcore 18 (not detected) > EAL: Skip lcore 19 (not detected) > EAL: Skip lcore 20 (not detected) > EAL: Skip lcore 21 (not detected) > EAL: Skip lcore 22 (not detected) > EAL: Skip lcore 23 (not detected) > EAL: Skip lcore 24 (not detected) > EAL: Skip lcore 25 (not detected) > EAL: Skip lcore 26 (not detected) > EAL: Skip lcore 27 (not detected) > EAL: Skip lcore 28 (not detected) > EAL: Skip lcore 29 (not detected) > EAL: Skip lcore 30 (not detected) > EAL: Skip lcore 31 (not detected) > EAL: Skip lcore 32 (not detected) > EAL: Skip lcore 33 (not detected) > EAL: Skip lcore 34 (not detected) > EAL: Skip lcore 35 (not detected) > EAL: Skip lcore 36 (not detected) > EAL: Skip lcore 37 (not detected) > EAL: Skip lcore 38 (not detected) > EAL: Skip lcore 39 (not detected) > EAL: Skip lcore 40 (not detected) > EAL: Skip lcore 41 (not detected) > EAL: Skip lcore 42 (not detected) > EAL: Skip lcore 43 (not detected) > EAL: Skip lcore 44 (not detected) > EAL: Skip lcore 45 (not detected) > EAL: Skip lcore 46 (not detected) > EAL: Skip lcore 47 (not detected) > EAL: Skip lcore 48 (not detected) > EAL: Skip lcore 49 (not detected) > EAL: Skip lcore 50 (not detected) > EAL: Skip lcore 51 (not detected) > EAL: Skip lcore 52 (not detected) > EAL: Skip lcore 53 (not detected) > EAL: Skip lcore 54 (not detected) > EAL: Skip lcore 55 (not detected) > EAL: Skip lcore 56 (not detected) > EAL: Skip lcore 57 (not detected) > EAL: Skip lcore 58 (not detected) > EAL: Skip lcore 59 (not detected) > EAL: Skip lcore 60 (not detected) > EAL: Skip lcore 61 (not detected) > EAL: Skip lcore 62 (not detected) > EAL: Skip lcore 63 (not detected) > EAL: Setting up memory... > EAL: Ask a virtual area of 0x80000000 bytes > EAL: Virtual area found at 0x7f848ae00000 (size = 0x80000000) > EAL: Requesting 1024 pages of size 2MB from socket 0 > EAL: TSC frequency is ~2000000 KHz > EAL: Master core 0 is ready (tid=b3f3f00) > EAL: Core 1 is ready (tid=8a1f2700) > EAL: Core 2 is ready (tid=899f1700) > EAL: Core 3 is ready (tid=891f0700) > EAL: Core 4 is ready (tid=889ef700) > EAL: Core 5 is ready (tid=7bfff700) > EAL: Core 6 is ready (tid=7b7fe700) > EAL: Core 7 is ready (tid=7affd700) > Pool initialized > Global Variables initialized > PMD: rte_vmxnet3_pmd_init(): >> > EAL: PCI device 0000:0b:00.0 on NUMA socket -1 > EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd > PMD: eth_vmxnet3_dev_init(): >> > Segmentation fault (core dumped) > > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Prashant > Upadhyaya > Sent: Thursday, March 06, 2014 12:20 PM > To: David Marchand > Cc: dev@dpdk.org > Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? > > Hi, > > Some further update on the crash I am facing. > I am using DPDK1.6.0r1 and take over the vmxnet3 with igb_uio and then > start the application. (so no external ko or vmxnet usermap etc.) > During the port initializations, the crash is happening in the > following function - > > eth_vmxnet3_dev_init > > and the crash is happening at the following line - > /* Check h/w version compatibility with driver. */ > ver = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_VRRS); > > Any hints regarding what could be wrong ? > > Regards > -Prashant > > > From: Prashant Upadhyaya > Sent: Wednesday, March 05, 2014 9:01 PM > To: 'David Marchand' > Cc: Srinivasan J; dev@dpdk.org > Subject: RE: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? > > Hi David, > > The compilation error with debug flags on was that the functions -- vmxnet3_txq_dump, vmxnet3_rxq_dump - are defined but not used. > Not a serious error, I will try to get rid of the compiler flag which generates this. > > However, I must reiterate, I _did_ bind my vmxnet3 device with igb_uio (and I did not use any .so, because I was intending to use the builtin vmxnet3 driver of dpdk 1.6.0r1), the bind succeeded, but then when I started the application, the dev init for vmxnet3 gave a core dump. > Your patch and solution seems to be suggesting the reverse, i.e. when I don't bind with igb_uio but try to use the native driver. So please do try the above combination as well. > > Regards > -Prashant > > > From: David Marchand [mailto:david.marchand@6wind.com] > Sent: Wednesday, March 05, 2014 8:41 PM > To: Prashant Upadhyaya > Cc: Srinivasan J; dev@dpdk.org<mailto:dev@dpdk.org> > Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? > > Hello Prashant, > > On Wed, Mar 5, 2014 at 3:28 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com<mailto:prashant.upadhyaya@aricent.com>> wrote: > Hi, > > I am also keen to know the answer to the question posted by Srini. > The real question is -- is dpdk1.6.0r1 self-sufficient so that I don't need any extensions etc., or do I still need something from outside like the usermap kernel driver etc. > > Secondly, if I turn on all the debug options for the vmxnet3 pmd in the config file, 1.6.0r1 compilation runs into a problem and reports a function which is defined but not used. > > Can you send your build error ? (maybe in a separate thread ?) > > > I am trying to bring up DPDK inside Fedora18 Guest on ESXi -- when I used DPDK1.6.0r1 (without debug options turned on for vmxnet3 pmd) the igb_uio could take over the vmxnet3 NIC but I encountered a core dump in the dev init function for the vmxnet3 driver -- anybody encountered a similar issue ? > > I encountered these problems as well. > > - igb_uio module does not check if you disable vmxnet3-uio pmds, it will always try to take over vmxnet3 devices. > I have a patch waiting in my working dir to cleanly disable vmxnet3-uio pmd. > > - If you don't bind vmxnet3 devices to uio, but forget to enable vmxnet3-usermap pmd (by specifiying -d librte_pmd_vmxnet3.so), then internal vmxnet3-uio pmd will try to initialise and crash. > I did not look any deeper into this, the easiest way is to disable vmxnet3-uio pmd + apply the patch I will send in a few minutes, as a first workaround. > > > Regards, > -- > David Marchand > > > > > ====================================================================== > ========= Please refer to > http://www.aricent.com/legal/email_disclaimer.html > for important disclosures regarding this electronic communication. > ====================================================================== > ========= > > > > > ====================================================================== > ========= Please refer to > http://www.aricent.com/legal/email_disclaimer.html > for important disclosures regarding this electronic communication. > ====================================================================== > ========= =============================================================================== Please refer to http://www.aricent.com/legal/email_disclaimer.html for important disclosures regarding this electronic communication. =============================================================================== ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? 2014-03-07 5:13 ` Prashant Upadhyaya @ 2014-03-08 19:08 ` Srinivasan J 2014-03-10 8:00 ` Prashant Upadhyaya 0 siblings, 1 reply; 21+ messages in thread From: Srinivasan J @ 2014-03-08 19:08 UTC (permalink / raw) To: Prashant Upadhyaya; +Cc: dev Prashant, I was also able to hit the issue your hitting using Esxi 5.1.0 evaluation and Fedora 20 X86_64 guest. I was able to fix the issue by setting CONFIG_RTE_EAL_UNBIND_PORTS=y option in defconfig_x86_64-default-linuxapp-gcc configuration file. Issue seen ======== EAL: PCI device 0000:03:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd EAL: Device is blacklisted, not initializing EAL: PCI device 0000:0b:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd Program received signal SIGSEGV, Segmentation fault. eth_vmxnet3_dev_init (eth_drv=<optimized out>, eth_dev=0x754480 <rte_eth_devices>) at /root/source/dpdk-1.6.0r1/lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c:218 218 ver = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_VRRS); Missing separate debuginfos, use: debuginfo-install glibc-2.18-11.fc20.x86_64 (gdb) p hw $1 = (struct vmxnet3_hw *) 0x7fffd8fc1040 (gdb) p *hw $2 = {hw_addr0 = 0x0, hw_addr1 = 0x0, back = 0x0, device_id = 1968, vendor_id = 5549, subsystem_device_id = 0, subsystem_vendor_id = 0, adapter_stopped = 0, perm_addr = "\000\000\000\000\000", num_tx_queues = 1 '\001', num_rx_queues = 1 '\001', bufs_per_pkt = 1 '\001', cur_mtu = 0, tqd_start = 0x0, rqd_start = 0x0, shared = 0x0, sharedPA = 0, queueDescPA = 0, queue_desc_len = 0, rss_conf = 0x0, rss_confPA = 0, mf_table = 0x0} (gdb) #define VMXNET3_PCI_BAR1_REG_ADDR(hw, reg) \ ((volatile uint32_t *)((char *)(hw)->hw_addr1 + (reg))) #define VMXNET3_READ_BAR1_REG(hw, reg) \ vmxnet3_read_addr(VMXNET3_PCI_BAR1_REG_ADDR((hw), (reg))) lib/librte_pmd_vmxnet3/vmxnet3_ethdev.h Issue not seen after enabling CONFIG_RTE_EAL_UNBIND_PORTS=y ================================================== [root@localhost build]# ./l2fwd -c 0xf -b 0000:03:00.0 -n 1 -- -p 0x6 EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 1 on socket 0 EAL: Detected lcore 2 as core 2 on socket 0 EAL: Detected lcore 3 as core 3 on socket 0 EAL: Skip lcore 4 (not detected) EAL: Skip lcore 5 (not detected) EAL: Skip lcore 6 (not detected) EAL: Skip lcore 7 (not detected) EAL: Skip lcore 8 (not detected) EAL: Skip lcore 9 (not detected) EAL: Skip lcore 10 (not detected) EAL: Skip lcore 11 (not detected) EAL: Skip lcore 12 (not detected) EAL: Skip lcore 13 (not detected) EAL: Skip lcore 14 (not detected) EAL: Skip lcore 15 (not detected) EAL: Skip lcore 16 (not detected) EAL: Skip lcore 17 (not detected) EAL: Skip lcore 18 (not detected) EAL: Skip lcore 19 (not detected) EAL: Skip lcore 20 (not detected) EAL: Skip lcore 21 (not detected) EAL: Skip lcore 22 (not detected) EAL: Skip lcore 23 (not detected) EAL: Skip lcore 24 (not detected) EAL: Skip lcore 25 (not detected) EAL: Skip lcore 26 (not detected) EAL: Skip lcore 27 (not detected) EAL: Skip lcore 28 (not detected) EAL: Skip lcore 29 (not detected) EAL: Skip lcore 30 (not detected) EAL: Skip lcore 31 (not detected) EAL: Skip lcore 32 (not detected) EAL: Skip lcore 33 (not detected) EAL: Skip lcore 34 (not detected) EAL: Skip lcore 35 (not detected) EAL: Skip lcore 36 (not detected) EAL: Skip lcore 37 (not detected) EAL: Skip lcore 38 (not detected) EAL: Skip lcore 39 (not detected) EAL: Skip lcore 40 (not detected) EAL: Skip lcore 41 (not detected) EAL: Skip lcore 42 (not detected) EAL: Skip lcore 43 (not detected) EAL: Skip lcore 44 (not detected) EAL: Skip lcore 45 (not detected) EAL: Skip lcore 46 (not detected) EAL: Skip lcore 47 (not detected) EAL: Skip lcore 48 (not detected) EAL: Skip lcore 49 (not detected) EAL: Skip lcore 50 (not detected) EAL: Skip lcore 51 (not detected) EAL: Skip lcore 52 (not detected) EAL: Skip lcore 53 (not detected) EAL: Skip lcore 54 (not detected) EAL: Skip lcore 55 (not detected) EAL: Skip lcore 56 (not detected) EAL: Skip lcore 57 (not detected) EAL: Skip lcore 58 (not detected) EAL: Skip lcore 59 (not detected) EAL: Skip lcore 60 (not detected) EAL: Skip lcore 61 (not detected) EAL: Skip lcore 62 (not detected) EAL: Skip lcore 63 (not detected) EAL: Setting up memory... EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7f3a76a00000 (size = 0x200000) EAL: Ask a virtual area of 0x7c00000 bytes EAL: Virtual area found at 0x7f3a6ec00000 (size = 0x7c00000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7f3a6e800000 (size = 0x200000) EAL: Requesting 64 pages of size 2MB from socket 0 EAL: TSC frequency is ~2200000 KHz EAL: Master core 0 is ready (tid=76cfe880) EAL: Core 3 is ready (tid=6cffc700) EAL: Core 1 is ready (tid=6dffe700) EAL: Core 2 is ready (tid=6d7fd700) EAL: PCI device 0000:03:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd EAL: Device is blacklisted, not initializing EAL: PCI device 0000:0b:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd EAL: PCI memory mapped at 0x7f3a76d10000 EAL: PCI memory mapped at 0x7f3a76d0f000 EAL: PCI memory mapped at 0x7f3a76d0d000 EAL: PCI device 0000:13:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd EAL: PCI memory mapped at 0x7f3a76d0c000 EAL: PCI memory mapped at 0x7f3a76d0b000 EAL: PCI memory mapped at 0x7f3a76d09000 Notice: odd number of ports in portmask. Lcore 0: RX port 1 Skipping disabled port 0 Initializing port 1... done: Port 1, MAC address: 00:0C:29:F1:9D:A4 Checking link statusdone Port 1 Link Up - speed 10000 Mbps - full-duplex L2FWD: lcore 1 has nothing to do L2FWD: lcore 2 has nothing to do L2FWD: lcore 3 has nothing to do L2FWD: entering main loop on lcore 0 L2FWD: -- lcoreid=0 portid=1 Port statistics ==================================== Statistics for port 1 ------------------------------ Packets sent: 0 Packets received: 0 Packets dropped: 0 Statistics for port 2 ------------------------------ Packets sent: 0 Packets received: 0 Packets dropped: 0 Aggregate statistics =============================== Total packets sent: 0 Total packets received: 0 Total packets dropped: 0 ==================================================== On Fri, Mar 7, 2014 at 10:43 AM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: > Hi Srini, > > Yes, I am still stuck, I am using ESXi5.1 though and Fedora18 in Guest. > Please let me know what is the guest OS distro you are using in the VM. I'll try with that once. > > Regards > -Prashant > > > -----Original Message----- > From: Srinivasan J [mailto:srinidpdk@gmail.com] > Sent: Friday, March 07, 2014 9:08 AM > To: Prashant Upadhyaya > Cc: David Marchand; dev@dpdk.org > Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? > > Prashant, > Are you still hitting a crash? Do you have hugetlbfs enabled? if not try adding enabling hugetlbfs. I was able to get > vxnet3 up using the vmxnet3 pmd alone in DPDK 1.60r1. I did not use the vmxnet3-usermap kernel driver. I am using Esxi 5.5 > > Eg: echo 1024 > /proc/sys/vm/nr_hugepages; mkdir /huge; mount -t hugetlbfs nodev /huge; > > On Thu, Mar 6, 2014 at 12:48 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: >> And if it is any help, here is the startup log -- >> >> EAL: Detected lcore 0 as core 0 on socket 0 >> EAL: Detected lcore 1 as core 1 on socket 0 >> EAL: Detected lcore 2 as core 2 on socket 0 >> EAL: Detected lcore 3 as core 3 on socket 0 >> EAL: Detected lcore 4 as core 4 on socket 0 >> EAL: Detected lcore 5 as core 5 on socket 0 >> EAL: Detected lcore 6 as core 6 on socket 0 >> EAL: Detected lcore 7 as core 7 on socket 0 >> EAL: Skip lcore 8 (not detected) >> EAL: Skip lcore 9 (not detected) >> EAL: Skip lcore 10 (not detected) >> EAL: Skip lcore 11 (not detected) >> EAL: Skip lcore 12 (not detected) >> EAL: Skip lcore 13 (not detected) >> EAL: Skip lcore 14 (not detected) >> EAL: Skip lcore 15 (not detected) >> EAL: Skip lcore 16 (not detected) >> EAL: Skip lcore 17 (not detected) >> EAL: Skip lcore 18 (not detected) >> EAL: Skip lcore 19 (not detected) >> EAL: Skip lcore 20 (not detected) >> EAL: Skip lcore 21 (not detected) >> EAL: Skip lcore 22 (not detected) >> EAL: Skip lcore 23 (not detected) >> EAL: Skip lcore 24 (not detected) >> EAL: Skip lcore 25 (not detected) >> EAL: Skip lcore 26 (not detected) >> EAL: Skip lcore 27 (not detected) >> EAL: Skip lcore 28 (not detected) >> EAL: Skip lcore 29 (not detected) >> EAL: Skip lcore 30 (not detected) >> EAL: Skip lcore 31 (not detected) >> EAL: Skip lcore 32 (not detected) >> EAL: Skip lcore 33 (not detected) >> EAL: Skip lcore 34 (not detected) >> EAL: Skip lcore 35 (not detected) >> EAL: Skip lcore 36 (not detected) >> EAL: Skip lcore 37 (not detected) >> EAL: Skip lcore 38 (not detected) >> EAL: Skip lcore 39 (not detected) >> EAL: Skip lcore 40 (not detected) >> EAL: Skip lcore 41 (not detected) >> EAL: Skip lcore 42 (not detected) >> EAL: Skip lcore 43 (not detected) >> EAL: Skip lcore 44 (not detected) >> EAL: Skip lcore 45 (not detected) >> EAL: Skip lcore 46 (not detected) >> EAL: Skip lcore 47 (not detected) >> EAL: Skip lcore 48 (not detected) >> EAL: Skip lcore 49 (not detected) >> EAL: Skip lcore 50 (not detected) >> EAL: Skip lcore 51 (not detected) >> EAL: Skip lcore 52 (not detected) >> EAL: Skip lcore 53 (not detected) >> EAL: Skip lcore 54 (not detected) >> EAL: Skip lcore 55 (not detected) >> EAL: Skip lcore 56 (not detected) >> EAL: Skip lcore 57 (not detected) >> EAL: Skip lcore 58 (not detected) >> EAL: Skip lcore 59 (not detected) >> EAL: Skip lcore 60 (not detected) >> EAL: Skip lcore 61 (not detected) >> EAL: Skip lcore 62 (not detected) >> EAL: Skip lcore 63 (not detected) >> EAL: Setting up memory... >> EAL: Ask a virtual area of 0x80000000 bytes >> EAL: Virtual area found at 0x7f848ae00000 (size = 0x80000000) >> EAL: Requesting 1024 pages of size 2MB from socket 0 >> EAL: TSC frequency is ~2000000 KHz >> EAL: Master core 0 is ready (tid=b3f3f00) >> EAL: Core 1 is ready (tid=8a1f2700) >> EAL: Core 2 is ready (tid=899f1700) >> EAL: Core 3 is ready (tid=891f0700) >> EAL: Core 4 is ready (tid=889ef700) >> EAL: Core 5 is ready (tid=7bfff700) >> EAL: Core 6 is ready (tid=7b7fe700) >> EAL: Core 7 is ready (tid=7affd700) >> Pool initialized >> Global Variables initialized >> PMD: rte_vmxnet3_pmd_init(): >> >> EAL: PCI device 0000:0b:00.0 on NUMA socket -1 >> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd >> PMD: eth_vmxnet3_dev_init(): >> >> Segmentation fault (core dumped) >> >> -----Original Message----- >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Prashant >> Upadhyaya >> Sent: Thursday, March 06, 2014 12:20 PM >> To: David Marchand >> Cc: dev@dpdk.org >> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >> >> Hi, >> >> Some further update on the crash I am facing. >> I am using DPDK1.6.0r1 and take over the vmxnet3 with igb_uio and then >> start the application. (so no external ko or vmxnet usermap etc.) >> During the port initializations, the crash is happening in the >> following function - >> >> eth_vmxnet3_dev_init >> >> and the crash is happening at the following line - >> /* Check h/w version compatibility with driver. */ >> ver = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_VRRS); >> >> Any hints regarding what could be wrong ? >> >> Regards >> -Prashant >> >> >> From: Prashant Upadhyaya >> Sent: Wednesday, March 05, 2014 9:01 PM >> To: 'David Marchand' >> Cc: Srinivasan J; dev@dpdk.org >> Subject: RE: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >> >> Hi David, >> >> The compilation error with debug flags on was that the functions -- vmxnet3_txq_dump, vmxnet3_rxq_dump - are defined but not used. >> Not a serious error, I will try to get rid of the compiler flag which generates this. >> >> However, I must reiterate, I _did_ bind my vmxnet3 device with igb_uio (and I did not use any .so, because I was intending to use the builtin vmxnet3 driver of dpdk 1.6.0r1), the bind succeeded, but then when I started the application, the dev init for vmxnet3 gave a core dump. >> Your patch and solution seems to be suggesting the reverse, i.e. when I don't bind with igb_uio but try to use the native driver. So please do try the above combination as well. >> >> Regards >> -Prashant >> >> >> From: David Marchand [mailto:david.marchand@6wind.com] >> Sent: Wednesday, March 05, 2014 8:41 PM >> To: Prashant Upadhyaya >> Cc: Srinivasan J; dev@dpdk.org<mailto:dev@dpdk.org> >> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >> >> Hello Prashant, >> >> On Wed, Mar 5, 2014 at 3:28 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com<mailto:prashant.upadhyaya@aricent.com>> wrote: >> Hi, >> >> I am also keen to know the answer to the question posted by Srini. >> The real question is -- is dpdk1.6.0r1 self-sufficient so that I don't need any extensions etc., or do I still need something from outside like the usermap kernel driver etc. >> >> Secondly, if I turn on all the debug options for the vmxnet3 pmd in the config file, 1.6.0r1 compilation runs into a problem and reports a function which is defined but not used. >> >> Can you send your build error ? (maybe in a separate thread ?) >> >> >> I am trying to bring up DPDK inside Fedora18 Guest on ESXi -- when I used DPDK1.6.0r1 (without debug options turned on for vmxnet3 pmd) the igb_uio could take over the vmxnet3 NIC but I encountered a core dump in the dev init function for the vmxnet3 driver -- anybody encountered a similar issue ? >> >> I encountered these problems as well. >> >> - igb_uio module does not check if you disable vmxnet3-uio pmds, it will always try to take over vmxnet3 devices. >> I have a patch waiting in my working dir to cleanly disable vmxnet3-uio pmd. >> >> - If you don't bind vmxnet3 devices to uio, but forget to enable vmxnet3-usermap pmd (by specifiying -d librte_pmd_vmxnet3.so), then internal vmxnet3-uio pmd will try to initialise and crash. >> I did not look any deeper into this, the easiest way is to disable vmxnet3-uio pmd + apply the patch I will send in a few minutes, as a first workaround. >> >> >> Regards, >> -- >> David Marchand >> >> >> >> >> ====================================================================== >> ========= Please refer to >> http://www.aricent.com/legal/email_disclaimer.html >> for important disclosures regarding this electronic communication. >> ====================================================================== >> ========= >> >> >> >> >> ====================================================================== >> ========= Please refer to >> http://www.aricent.com/legal/email_disclaimer.html >> for important disclosures regarding this electronic communication. >> ====================================================================== >> ========= > > > > > =============================================================================== > Please refer to http://www.aricent.com/legal/email_disclaimer.html > for important disclosures regarding this electronic communication. > =============================================================================== ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? 2014-03-08 19:08 ` Srinivasan J @ 2014-03-10 8:00 ` Prashant Upadhyaya 2014-03-10 10:54 ` Prashant Upadhyaya ` (2 more replies) 0 siblings, 3 replies; 21+ messages in thread From: Prashant Upadhyaya @ 2014-03-10 8:00 UTC (permalink / raw) To: Srinivasan J; +Cc: dev Hi Srini, Thanks, I could also make it work, thanks to your cue ! Now then, this multi-segment not being supported in vmxnet3 driver is a big party-pooper for me. Unfortunately in my usecase, I do indeed make heavy use of multisegment buffers for sending out the data, so my usecase has failed and I will have to fix that. Also, can you please adivse how much is the max data rates you have been able to achieve with one vmxnet3 10G port. Thanks a lot for the advice once again. Regards -Prashant -----Original Message----- From: Srinivasan J [mailto:srinidpdk@gmail.com] Sent: Sunday, March 09, 2014 12:38 AM To: Prashant Upadhyaya Cc: David Marchand; dev@dpdk.org Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? Prashant, I was also able to hit the issue your hitting using Esxi 5.1.0 evaluation and Fedora 20 X86_64 guest. I was able to fix the issue by setting CONFIG_RTE_EAL_UNBIND_PORTS=y option in defconfig_x86_64-default-linuxapp-gcc configuration file. Issue seen ======== EAL: PCI device 0000:03:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd EAL: Device is blacklisted, not initializing EAL: PCI device 0000:0b:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd Program received signal SIGSEGV, Segmentation fault. eth_vmxnet3_dev_init (eth_drv=<optimized out>, eth_dev=0x754480 <rte_eth_devices>) at /root/source/dpdk-1.6.0r1/lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c:218 218 ver = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_VRRS); Missing separate debuginfos, use: debuginfo-install glibc-2.18-11.fc20.x86_64 (gdb) p hw $1 = (struct vmxnet3_hw *) 0x7fffd8fc1040 (gdb) p *hw $2 = {hw_addr0 = 0x0, hw_addr1 = 0x0, back = 0x0, device_id = 1968, vendor_id = 5549, subsystem_device_id = 0, subsystem_vendor_id = 0, adapter_stopped = 0, perm_addr = "\000\000\000\000\000", num_tx_queues = 1 '\001', num_rx_queues = 1 '\001', bufs_per_pkt = 1 '\001', cur_mtu = 0, tqd_start = 0x0, rqd_start = 0x0, shared = 0x0, sharedPA = 0, queueDescPA = 0, queue_desc_len = 0, rss_conf = 0x0, rss_confPA = 0, mf_table = 0x0} (gdb) #define VMXNET3_PCI_BAR1_REG_ADDR(hw, reg) \ ((volatile uint32_t *)((char *)(hw)->hw_addr1 + (reg))) #define VMXNET3_READ_BAR1_REG(hw, reg) \ vmxnet3_read_addr(VMXNET3_PCI_BAR1_REG_ADDR((hw), (reg))) lib/librte_pmd_vmxnet3/vmxnet3_ethdev.h Issue not seen after enabling CONFIG_RTE_EAL_UNBIND_PORTS=y ================================================== [root@localhost build]# ./l2fwd -c 0xf -b 0000:03:00.0 -n 1 -- -p 0x6 EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 1 on socket 0 EAL: Detected lcore 2 as core 2 on socket 0 EAL: Detected lcore 3 as core 3 on socket 0 EAL: Skip lcore 4 (not detected) EAL: Skip lcore 5 (not detected) EAL: Skip lcore 6 (not detected) EAL: Skip lcore 7 (not detected) EAL: Skip lcore 8 (not detected) EAL: Skip lcore 9 (not detected) EAL: Skip lcore 10 (not detected) EAL: Skip lcore 11 (not detected) EAL: Skip lcore 12 (not detected) EAL: Skip lcore 13 (not detected) EAL: Skip lcore 14 (not detected) EAL: Skip lcore 15 (not detected) EAL: Skip lcore 16 (not detected) EAL: Skip lcore 17 (not detected) EAL: Skip lcore 18 (not detected) EAL: Skip lcore 19 (not detected) EAL: Skip lcore 20 (not detected) EAL: Skip lcore 21 (not detected) EAL: Skip lcore 22 (not detected) EAL: Skip lcore 23 (not detected) EAL: Skip lcore 24 (not detected) EAL: Skip lcore 25 (not detected) EAL: Skip lcore 26 (not detected) EAL: Skip lcore 27 (not detected) EAL: Skip lcore 28 (not detected) EAL: Skip lcore 29 (not detected) EAL: Skip lcore 30 (not detected) EAL: Skip lcore 31 (not detected) EAL: Skip lcore 32 (not detected) EAL: Skip lcore 33 (not detected) EAL: Skip lcore 34 (not detected) EAL: Skip lcore 35 (not detected) EAL: Skip lcore 36 (not detected) EAL: Skip lcore 37 (not detected) EAL: Skip lcore 38 (not detected) EAL: Skip lcore 39 (not detected) EAL: Skip lcore 40 (not detected) EAL: Skip lcore 41 (not detected) EAL: Skip lcore 42 (not detected) EAL: Skip lcore 43 (not detected) EAL: Skip lcore 44 (not detected) EAL: Skip lcore 45 (not detected) EAL: Skip lcore 46 (not detected) EAL: Skip lcore 47 (not detected) EAL: Skip lcore 48 (not detected) EAL: Skip lcore 49 (not detected) EAL: Skip lcore 50 (not detected) EAL: Skip lcore 51 (not detected) EAL: Skip lcore 52 (not detected) EAL: Skip lcore 53 (not detected) EAL: Skip lcore 54 (not detected) EAL: Skip lcore 55 (not detected) EAL: Skip lcore 56 (not detected) EAL: Skip lcore 57 (not detected) EAL: Skip lcore 58 (not detected) EAL: Skip lcore 59 (not detected) EAL: Skip lcore 60 (not detected) EAL: Skip lcore 61 (not detected) EAL: Skip lcore 62 (not detected) EAL: Skip lcore 63 (not detected) EAL: Setting up memory... EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7f3a76a00000 (size = 0x200000) EAL: Ask a virtual area of 0x7c00000 bytes EAL: Virtual area found at 0x7f3a6ec00000 (size = 0x7c00000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7f3a6e800000 (size = 0x200000) EAL: Requesting 64 pages of size 2MB from socket 0 EAL: TSC frequency is ~2200000 KHz EAL: Master core 0 is ready (tid=76cfe880) EAL: Core 3 is ready (tid=6cffc700) EAL: Core 1 is ready (tid=6dffe700) EAL: Core 2 is ready (tid=6d7fd700) EAL: PCI device 0000:03:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd EAL: Device is blacklisted, not initializing EAL: PCI device 0000:0b:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd EAL: PCI memory mapped at 0x7f3a76d10000 EAL: PCI memory mapped at 0x7f3a76d0f000 EAL: PCI memory mapped at 0x7f3a76d0d000 EAL: PCI device 0000:13:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd EAL: PCI memory mapped at 0x7f3a76d0c000 EAL: PCI memory mapped at 0x7f3a76d0b000 EAL: PCI memory mapped at 0x7f3a76d09000 Notice: odd number of ports in portmask. Lcore 0: RX port 1 Skipping disabled port 0 Initializing port 1... done: Port 1, MAC address: 00:0C:29:F1:9D:A4 Checking link statusdone Port 1 Link Up - speed 10000 Mbps - full-duplex L2FWD: lcore 1 has nothing to do L2FWD: lcore 2 has nothing to do L2FWD: lcore 3 has nothing to do L2FWD: entering main loop on lcore 0 L2FWD: -- lcoreid=0 portid=1 Port statistics ==================================== Statistics for port 1 ------------------------------ Packets sent: 0 Packets received: 0 Packets dropped: 0 Statistics for port 2 ------------------------------ Packets sent: 0 Packets received: 0 Packets dropped: 0 Aggregate statistics =============================== Total packets sent: 0 Total packets received: 0 Total packets dropped: 0 ==================================================== On Fri, Mar 7, 2014 at 10:43 AM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: > Hi Srini, > > Yes, I am still stuck, I am using ESXi5.1 though and Fedora18 in Guest. > Please let me know what is the guest OS distro you are using in the VM. I'll try with that once. > > Regards > -Prashant > > > -----Original Message----- > From: Srinivasan J [mailto:srinidpdk@gmail.com] > Sent: Friday, March 07, 2014 9:08 AM > To: Prashant Upadhyaya > Cc: David Marchand; dev@dpdk.org > Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? > > Prashant, > Are you still hitting a crash? Do you have hugetlbfs > enabled? if not try adding enabling hugetlbfs. I was able to get > vxnet3 up using the vmxnet3 pmd alone in DPDK 1.60r1. I did not use > the vmxnet3-usermap kernel driver. I am using Esxi 5.5 > > Eg: echo 1024 > /proc/sys/vm/nr_hugepages; mkdir /huge; mount -t > hugetlbfs nodev /huge; > > On Thu, Mar 6, 2014 at 12:48 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: >> And if it is any help, here is the startup log -- >> >> EAL: Detected lcore 0 as core 0 on socket 0 >> EAL: Detected lcore 1 as core 1 on socket 0 >> EAL: Detected lcore 2 as core 2 on socket 0 >> EAL: Detected lcore 3 as core 3 on socket 0 >> EAL: Detected lcore 4 as core 4 on socket 0 >> EAL: Detected lcore 5 as core 5 on socket 0 >> EAL: Detected lcore 6 as core 6 on socket 0 >> EAL: Detected lcore 7 as core 7 on socket 0 >> EAL: Skip lcore 8 (not detected) >> EAL: Skip lcore 9 (not detected) >> EAL: Skip lcore 10 (not detected) >> EAL: Skip lcore 11 (not detected) >> EAL: Skip lcore 12 (not detected) >> EAL: Skip lcore 13 (not detected) >> EAL: Skip lcore 14 (not detected) >> EAL: Skip lcore 15 (not detected) >> EAL: Skip lcore 16 (not detected) >> EAL: Skip lcore 17 (not detected) >> EAL: Skip lcore 18 (not detected) >> EAL: Skip lcore 19 (not detected) >> EAL: Skip lcore 20 (not detected) >> EAL: Skip lcore 21 (not detected) >> EAL: Skip lcore 22 (not detected) >> EAL: Skip lcore 23 (not detected) >> EAL: Skip lcore 24 (not detected) >> EAL: Skip lcore 25 (not detected) >> EAL: Skip lcore 26 (not detected) >> EAL: Skip lcore 27 (not detected) >> EAL: Skip lcore 28 (not detected) >> EAL: Skip lcore 29 (not detected) >> EAL: Skip lcore 30 (not detected) >> EAL: Skip lcore 31 (not detected) >> EAL: Skip lcore 32 (not detected) >> EAL: Skip lcore 33 (not detected) >> EAL: Skip lcore 34 (not detected) >> EAL: Skip lcore 35 (not detected) >> EAL: Skip lcore 36 (not detected) >> EAL: Skip lcore 37 (not detected) >> EAL: Skip lcore 38 (not detected) >> EAL: Skip lcore 39 (not detected) >> EAL: Skip lcore 40 (not detected) >> EAL: Skip lcore 41 (not detected) >> EAL: Skip lcore 42 (not detected) >> EAL: Skip lcore 43 (not detected) >> EAL: Skip lcore 44 (not detected) >> EAL: Skip lcore 45 (not detected) >> EAL: Skip lcore 46 (not detected) >> EAL: Skip lcore 47 (not detected) >> EAL: Skip lcore 48 (not detected) >> EAL: Skip lcore 49 (not detected) >> EAL: Skip lcore 50 (not detected) >> EAL: Skip lcore 51 (not detected) >> EAL: Skip lcore 52 (not detected) >> EAL: Skip lcore 53 (not detected) >> EAL: Skip lcore 54 (not detected) >> EAL: Skip lcore 55 (not detected) >> EAL: Skip lcore 56 (not detected) >> EAL: Skip lcore 57 (not detected) >> EAL: Skip lcore 58 (not detected) >> EAL: Skip lcore 59 (not detected) >> EAL: Skip lcore 60 (not detected) >> EAL: Skip lcore 61 (not detected) >> EAL: Skip lcore 62 (not detected) >> EAL: Skip lcore 63 (not detected) >> EAL: Setting up memory... >> EAL: Ask a virtual area of 0x80000000 bytes >> EAL: Virtual area found at 0x7f848ae00000 (size = 0x80000000) >> EAL: Requesting 1024 pages of size 2MB from socket 0 >> EAL: TSC frequency is ~2000000 KHz >> EAL: Master core 0 is ready (tid=b3f3f00) >> EAL: Core 1 is ready (tid=8a1f2700) >> EAL: Core 2 is ready (tid=899f1700) >> EAL: Core 3 is ready (tid=891f0700) >> EAL: Core 4 is ready (tid=889ef700) >> EAL: Core 5 is ready (tid=7bfff700) >> EAL: Core 6 is ready (tid=7b7fe700) >> EAL: Core 7 is ready (tid=7affd700) >> Pool initialized >> Global Variables initialized >> PMD: rte_vmxnet3_pmd_init(): >> >> EAL: PCI device 0000:0b:00.0 on NUMA socket -1 >> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd >> PMD: eth_vmxnet3_dev_init(): >> >> Segmentation fault (core dumped) >> >> -----Original Message----- >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Prashant >> Upadhyaya >> Sent: Thursday, March 06, 2014 12:20 PM >> To: David Marchand >> Cc: dev@dpdk.org >> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >> >> Hi, >> >> Some further update on the crash I am facing. >> I am using DPDK1.6.0r1 and take over the vmxnet3 with igb_uio and >> then start the application. (so no external ko or vmxnet usermap >> etc.) During the port initializations, the crash is happening in the >> following function - >> >> eth_vmxnet3_dev_init >> >> and the crash is happening at the following line - >> /* Check h/w version compatibility with driver. */ >> ver = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_VRRS); >> >> Any hints regarding what could be wrong ? >> >> Regards >> -Prashant >> >> >> From: Prashant Upadhyaya >> Sent: Wednesday, March 05, 2014 9:01 PM >> To: 'David Marchand' >> Cc: Srinivasan J; dev@dpdk.org >> Subject: RE: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >> >> Hi David, >> >> The compilation error with debug flags on was that the functions -- vmxnet3_txq_dump, vmxnet3_rxq_dump - are defined but not used. >> Not a serious error, I will try to get rid of the compiler flag which generates this. >> >> However, I must reiterate, I _did_ bind my vmxnet3 device with igb_uio (and I did not use any .so, because I was intending to use the builtin vmxnet3 driver of dpdk 1.6.0r1), the bind succeeded, but then when I started the application, the dev init for vmxnet3 gave a core dump. >> Your patch and solution seems to be suggesting the reverse, i.e. when I don't bind with igb_uio but try to use the native driver. So please do try the above combination as well. >> >> Regards >> -Prashant >> >> >> From: David Marchand [mailto:david.marchand@6wind.com] >> Sent: Wednesday, March 05, 2014 8:41 PM >> To: Prashant Upadhyaya >> Cc: Srinivasan J; dev@dpdk.org<mailto:dev@dpdk.org> >> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >> >> Hello Prashant, >> >> On Wed, Mar 5, 2014 at 3:28 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com<mailto:prashant.upadhyaya@aricent.com>> wrote: >> Hi, >> >> I am also keen to know the answer to the question posted by Srini. >> The real question is -- is dpdk1.6.0r1 self-sufficient so that I don't need any extensions etc., or do I still need something from outside like the usermap kernel driver etc. >> >> Secondly, if I turn on all the debug options for the vmxnet3 pmd in the config file, 1.6.0r1 compilation runs into a problem and reports a function which is defined but not used. >> >> Can you send your build error ? (maybe in a separate thread ?) >> >> >> I am trying to bring up DPDK inside Fedora18 Guest on ESXi -- when I used DPDK1.6.0r1 (without debug options turned on for vmxnet3 pmd) the igb_uio could take over the vmxnet3 NIC but I encountered a core dump in the dev init function for the vmxnet3 driver -- anybody encountered a similar issue ? >> >> I encountered these problems as well. >> >> - igb_uio module does not check if you disable vmxnet3-uio pmds, it will always try to take over vmxnet3 devices. >> I have a patch waiting in my working dir to cleanly disable vmxnet3-uio pmd. >> >> - If you don't bind vmxnet3 devices to uio, but forget to enable vmxnet3-usermap pmd (by specifiying -d librte_pmd_vmxnet3.so), then internal vmxnet3-uio pmd will try to initialise and crash. >> I did not look any deeper into this, the easiest way is to disable vmxnet3-uio pmd + apply the patch I will send in a few minutes, as a first workaround. >> >> >> Regards, >> -- >> David Marchand >> >> >> >> >> ===================================================================== >> = >> ========= Please refer to >> http://www.aricent.com/legal/email_disclaimer.html >> for important disclosures regarding this electronic communication. >> ===================================================================== >> = >> ========= >> >> >> >> >> ===================================================================== >> = >> ========= Please refer to >> http://www.aricent.com/legal/email_disclaimer.html >> for important disclosures regarding this electronic communication. >> ===================================================================== >> = >> ========= > > > > > ====================================================================== > ========= Please refer to > http://www.aricent.com/legal/email_disclaimer.html > for important disclosures regarding this electronic communication. > ====================================================================== > ========= =============================================================================== Please refer to http://www.aricent.com/legal/email_disclaimer.html for important disclosures regarding this electronic communication. =============================================================================== ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? 2014-03-10 8:00 ` Prashant Upadhyaya @ 2014-03-10 10:54 ` Prashant Upadhyaya 2014-03-10 15:50 ` Stephen Hemminger 2014-03-11 5:33 ` [dpdk-dev] On vmxnet-pmd crash in DPDK 1.6.0r1 Daniel Kan 2 siblings, 0 replies; 21+ messages in thread From: Prashant Upadhyaya @ 2014-03-10 10:54 UTC (permalink / raw) To: Srinivasan J; +Cc: dev Hi, Regarding performance with vmxnet3 driver, the programmer's guide says thus -- " Currently, the driver provides basic support for using the device in an Intel(r) DPDK application running on a guest OS. Optimization is needed on the backend, that is, the VMware* ESXi vmkernel switch, to achieve optimal performance end-to-end. " Can someone advise what are the techniques of 'backend optimization on the vmkernel switch' I ran some initial tests without any attempt for above optimization and the data rates I am achieving with vmxnet3 are not very encouraging. Regards -Prashant -----Original Message----- From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Prashant Upadhyaya Sent: Monday, March 10, 2014 1:31 PM To: Srinivasan J Cc: dev@dpdk.org Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? Hi Srini, Thanks, I could also make it work, thanks to your cue ! Now then, this multi-segment not being supported in vmxnet3 driver is a big party-pooper for me. Unfortunately in my usecase, I do indeed make heavy use of multisegment buffers for sending out the data, so my usecase has failed and I will have to fix that. Also, can you please adivse how much is the max data rates you have been able to achieve with one vmxnet3 10G port. Thanks a lot for the advice once again. Regards -Prashant -----Original Message----- From: Srinivasan J [mailto:srinidpdk@gmail.com] Sent: Sunday, March 09, 2014 12:38 AM To: Prashant Upadhyaya Cc: David Marchand; dev@dpdk.org Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? Prashant, I was also able to hit the issue your hitting using Esxi 5.1.0 evaluation and Fedora 20 X86_64 guest. I was able to fix the issue by setting CONFIG_RTE_EAL_UNBIND_PORTS=y option in defconfig_x86_64-default-linuxapp-gcc configuration file. Issue seen ======== EAL: PCI device 0000:03:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd EAL: Device is blacklisted, not initializing EAL: PCI device 0000:0b:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd Program received signal SIGSEGV, Segmentation fault. eth_vmxnet3_dev_init (eth_drv=<optimized out>, eth_dev=0x754480 <rte_eth_devices>) at /root/source/dpdk-1.6.0r1/lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c:218 218 ver = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_VRRS); Missing separate debuginfos, use: debuginfo-install glibc-2.18-11.fc20.x86_64 (gdb) p hw $1 = (struct vmxnet3_hw *) 0x7fffd8fc1040 (gdb) p *hw $2 = {hw_addr0 = 0x0, hw_addr1 = 0x0, back = 0x0, device_id = 1968, vendor_id = 5549, subsystem_device_id = 0, subsystem_vendor_id = 0, adapter_stopped = 0, perm_addr = "\000\000\000\000\000", num_tx_queues = 1 '\001', num_rx_queues = 1 '\001', bufs_per_pkt = 1 '\001', cur_mtu = 0, tqd_start = 0x0, rqd_start = 0x0, shared = 0x0, sharedPA = 0, queueDescPA = 0, queue_desc_len = 0, rss_conf = 0x0, rss_confPA = 0, mf_table = 0x0} (gdb) #define VMXNET3_PCI_BAR1_REG_ADDR(hw, reg) \ ((volatile uint32_t *)((char *)(hw)->hw_addr1 + (reg))) #define VMXNET3_READ_BAR1_REG(hw, reg) \ vmxnet3_read_addr(VMXNET3_PCI_BAR1_REG_ADDR((hw), (reg))) lib/librte_pmd_vmxnet3/vmxnet3_ethdev.h Issue not seen after enabling CONFIG_RTE_EAL_UNBIND_PORTS=y ================================================== [root@localhost build]# ./l2fwd -c 0xf -b 0000:03:00.0 -n 1 -- -p 0x6 EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 1 on socket 0 EAL: Detected lcore 2 as core 2 on socket 0 EAL: Detected lcore 3 as core 3 on socket 0 EAL: Skip lcore 4 (not detected) EAL: Skip lcore 5 (not detected) EAL: Skip lcore 6 (not detected) EAL: Skip lcore 7 (not detected) EAL: Skip lcore 8 (not detected) EAL: Skip lcore 9 (not detected) EAL: Skip lcore 10 (not detected) EAL: Skip lcore 11 (not detected) EAL: Skip lcore 12 (not detected) EAL: Skip lcore 13 (not detected) EAL: Skip lcore 14 (not detected) EAL: Skip lcore 15 (not detected) EAL: Skip lcore 16 (not detected) EAL: Skip lcore 17 (not detected) EAL: Skip lcore 18 (not detected) EAL: Skip lcore 19 (not detected) EAL: Skip lcore 20 (not detected) EAL: Skip lcore 21 (not detected) EAL: Skip lcore 22 (not detected) EAL: Skip lcore 23 (not detected) EAL: Skip lcore 24 (not detected) EAL: Skip lcore 25 (not detected) EAL: Skip lcore 26 (not detected) EAL: Skip lcore 27 (not detected) EAL: Skip lcore 28 (not detected) EAL: Skip lcore 29 (not detected) EAL: Skip lcore 30 (not detected) EAL: Skip lcore 31 (not detected) EAL: Skip lcore 32 (not detected) EAL: Skip lcore 33 (not detected) EAL: Skip lcore 34 (not detected) EAL: Skip lcore 35 (not detected) EAL: Skip lcore 36 (not detected) EAL: Skip lcore 37 (not detected) EAL: Skip lcore 38 (not detected) EAL: Skip lcore 39 (not detected) EAL: Skip lcore 40 (not detected) EAL: Skip lcore 41 (not detected) EAL: Skip lcore 42 (not detected) EAL: Skip lcore 43 (not detected) EAL: Skip lcore 44 (not detected) EAL: Skip lcore 45 (not detected) EAL: Skip lcore 46 (not detected) EAL: Skip lcore 47 (not detected) EAL: Skip lcore 48 (not detected) EAL: Skip lcore 49 (not detected) EAL: Skip lcore 50 (not detected) EAL: Skip lcore 51 (not detected) EAL: Skip lcore 52 (not detected) EAL: Skip lcore 53 (not detected) EAL: Skip lcore 54 (not detected) EAL: Skip lcore 55 (not detected) EAL: Skip lcore 56 (not detected) EAL: Skip lcore 57 (not detected) EAL: Skip lcore 58 (not detected) EAL: Skip lcore 59 (not detected) EAL: Skip lcore 60 (not detected) EAL: Skip lcore 61 (not detected) EAL: Skip lcore 62 (not detected) EAL: Skip lcore 63 (not detected) EAL: Setting up memory... EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7f3a76a00000 (size = 0x200000) EAL: Ask a virtual area of 0x7c00000 bytes EAL: Virtual area found at 0x7f3a6ec00000 (size = 0x7c00000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7f3a6e800000 (size = 0x200000) EAL: Requesting 64 pages of size 2MB from socket 0 EAL: TSC frequency is ~2200000 KHz EAL: Master core 0 is ready (tid=76cfe880) EAL: Core 3 is ready (tid=6cffc700) EAL: Core 1 is ready (tid=6dffe700) EAL: Core 2 is ready (tid=6d7fd700) EAL: PCI device 0000:03:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd EAL: Device is blacklisted, not initializing EAL: PCI device 0000:0b:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd EAL: PCI memory mapped at 0x7f3a76d10000 EAL: PCI memory mapped at 0x7f3a76d0f000 EAL: PCI memory mapped at 0x7f3a76d0d000 EAL: PCI device 0000:13:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd EAL: PCI memory mapped at 0x7f3a76d0c000 EAL: PCI memory mapped at 0x7f3a76d0b000 EAL: PCI memory mapped at 0x7f3a76d09000 Notice: odd number of ports in portmask. Lcore 0: RX port 1 Skipping disabled port 0 Initializing port 1... done: Port 1, MAC address: 00:0C:29:F1:9D:A4 Checking link statusdone Port 1 Link Up - speed 10000 Mbps - full-duplex L2FWD: lcore 1 has nothing to do L2FWD: lcore 2 has nothing to do L2FWD: lcore 3 has nothing to do L2FWD: entering main loop on lcore 0 L2FWD: -- lcoreid=0 portid=1 Port statistics ==================================== Statistics for port 1 ------------------------------ Packets sent: 0 Packets received: 0 Packets dropped: 0 Statistics for port 2 ------------------------------ Packets sent: 0 Packets received: 0 Packets dropped: 0 Aggregate statistics =============================== Total packets sent: 0 Total packets received: 0 Total packets dropped: 0 ==================================================== On Fri, Mar 7, 2014 at 10:43 AM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: > Hi Srini, > > Yes, I am still stuck, I am using ESXi5.1 though and Fedora18 in Guest. > Please let me know what is the guest OS distro you are using in the VM. I'll try with that once. > > Regards > -Prashant > > > -----Original Message----- > From: Srinivasan J [mailto:srinidpdk@gmail.com] > Sent: Friday, March 07, 2014 9:08 AM > To: Prashant Upadhyaya > Cc: David Marchand; dev@dpdk.org > Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? > > Prashant, > Are you still hitting a crash? Do you have hugetlbfs > enabled? if not try adding enabling hugetlbfs. I was able to get > vxnet3 up using the vmxnet3 pmd alone in DPDK 1.60r1. I did not use > the vmxnet3-usermap kernel driver. I am using Esxi 5.5 > > Eg: echo 1024 > /proc/sys/vm/nr_hugepages; mkdir /huge; mount -t > hugetlbfs nodev /huge; > > On Thu, Mar 6, 2014 at 12:48 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: >> And if it is any help, here is the startup log -- >> >> EAL: Detected lcore 0 as core 0 on socket 0 >> EAL: Detected lcore 1 as core 1 on socket 0 >> EAL: Detected lcore 2 as core 2 on socket 0 >> EAL: Detected lcore 3 as core 3 on socket 0 >> EAL: Detected lcore 4 as core 4 on socket 0 >> EAL: Detected lcore 5 as core 5 on socket 0 >> EAL: Detected lcore 6 as core 6 on socket 0 >> EAL: Detected lcore 7 as core 7 on socket 0 >> EAL: Skip lcore 8 (not detected) >> EAL: Skip lcore 9 (not detected) >> EAL: Skip lcore 10 (not detected) >> EAL: Skip lcore 11 (not detected) >> EAL: Skip lcore 12 (not detected) >> EAL: Skip lcore 13 (not detected) >> EAL: Skip lcore 14 (not detected) >> EAL: Skip lcore 15 (not detected) >> EAL: Skip lcore 16 (not detected) >> EAL: Skip lcore 17 (not detected) >> EAL: Skip lcore 18 (not detected) >> EAL: Skip lcore 19 (not detected) >> EAL: Skip lcore 20 (not detected) >> EAL: Skip lcore 21 (not detected) >> EAL: Skip lcore 22 (not detected) >> EAL: Skip lcore 23 (not detected) >> EAL: Skip lcore 24 (not detected) >> EAL: Skip lcore 25 (not detected) >> EAL: Skip lcore 26 (not detected) >> EAL: Skip lcore 27 (not detected) >> EAL: Skip lcore 28 (not detected) >> EAL: Skip lcore 29 (not detected) >> EAL: Skip lcore 30 (not detected) >> EAL: Skip lcore 31 (not detected) >> EAL: Skip lcore 32 (not detected) >> EAL: Skip lcore 33 (not detected) >> EAL: Skip lcore 34 (not detected) >> EAL: Skip lcore 35 (not detected) >> EAL: Skip lcore 36 (not detected) >> EAL: Skip lcore 37 (not detected) >> EAL: Skip lcore 38 (not detected) >> EAL: Skip lcore 39 (not detected) >> EAL: Skip lcore 40 (not detected) >> EAL: Skip lcore 41 (not detected) >> EAL: Skip lcore 42 (not detected) >> EAL: Skip lcore 43 (not detected) >> EAL: Skip lcore 44 (not detected) >> EAL: Skip lcore 45 (not detected) >> EAL: Skip lcore 46 (not detected) >> EAL: Skip lcore 47 (not detected) >> EAL: Skip lcore 48 (not detected) >> EAL: Skip lcore 49 (not detected) >> EAL: Skip lcore 50 (not detected) >> EAL: Skip lcore 51 (not detected) >> EAL: Skip lcore 52 (not detected) >> EAL: Skip lcore 53 (not detected) >> EAL: Skip lcore 54 (not detected) >> EAL: Skip lcore 55 (not detected) >> EAL: Skip lcore 56 (not detected) >> EAL: Skip lcore 57 (not detected) >> EAL: Skip lcore 58 (not detected) >> EAL: Skip lcore 59 (not detected) >> EAL: Skip lcore 60 (not detected) >> EAL: Skip lcore 61 (not detected) >> EAL: Skip lcore 62 (not detected) >> EAL: Skip lcore 63 (not detected) >> EAL: Setting up memory... >> EAL: Ask a virtual area of 0x80000000 bytes >> EAL: Virtual area found at 0x7f848ae00000 (size = 0x80000000) >> EAL: Requesting 1024 pages of size 2MB from socket 0 >> EAL: TSC frequency is ~2000000 KHz >> EAL: Master core 0 is ready (tid=b3f3f00) >> EAL: Core 1 is ready (tid=8a1f2700) >> EAL: Core 2 is ready (tid=899f1700) >> EAL: Core 3 is ready (tid=891f0700) >> EAL: Core 4 is ready (tid=889ef700) >> EAL: Core 5 is ready (tid=7bfff700) >> EAL: Core 6 is ready (tid=7b7fe700) >> EAL: Core 7 is ready (tid=7affd700) >> Pool initialized >> Global Variables initialized >> PMD: rte_vmxnet3_pmd_init(): >> >> EAL: PCI device 0000:0b:00.0 on NUMA socket -1 >> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd >> PMD: eth_vmxnet3_dev_init(): >> >> Segmentation fault (core dumped) >> >> -----Original Message----- >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Prashant >> Upadhyaya >> Sent: Thursday, March 06, 2014 12:20 PM >> To: David Marchand >> Cc: dev@dpdk.org >> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >> >> Hi, >> >> Some further update on the crash I am facing. >> I am using DPDK1.6.0r1 and take over the vmxnet3 with igb_uio and >> then start the application. (so no external ko or vmxnet usermap >> etc.) During the port initializations, the crash is happening in the >> following function - >> >> eth_vmxnet3_dev_init >> >> and the crash is happening at the following line - >> /* Check h/w version compatibility with driver. */ >> ver = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_VRRS); >> >> Any hints regarding what could be wrong ? >> >> Regards >> -Prashant >> >> >> From: Prashant Upadhyaya >> Sent: Wednesday, March 05, 2014 9:01 PM >> To: 'David Marchand' >> Cc: Srinivasan J; dev@dpdk.org >> Subject: RE: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >> >> Hi David, >> >> The compilation error with debug flags on was that the functions -- vmxnet3_txq_dump, vmxnet3_rxq_dump - are defined but not used. >> Not a serious error, I will try to get rid of the compiler flag which generates this. >> >> However, I must reiterate, I _did_ bind my vmxnet3 device with igb_uio (and I did not use any .so, because I was intending to use the builtin vmxnet3 driver of dpdk 1.6.0r1), the bind succeeded, but then when I started the application, the dev init for vmxnet3 gave a core dump. >> Your patch and solution seems to be suggesting the reverse, i.e. when I don't bind with igb_uio but try to use the native driver. So please do try the above combination as well. >> >> Regards >> -Prashant >> >> >> From: David Marchand [mailto:david.marchand@6wind.com] >> Sent: Wednesday, March 05, 2014 8:41 PM >> To: Prashant Upadhyaya >> Cc: Srinivasan J; dev@dpdk.org<mailto:dev@dpdk.org> >> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >> >> Hello Prashant, >> >> On Wed, Mar 5, 2014 at 3:28 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com<mailto:prashant.upadhyaya@aricent.com>> wrote: >> Hi, >> >> I am also keen to know the answer to the question posted by Srini. >> The real question is -- is dpdk1.6.0r1 self-sufficient so that I don't need any extensions etc., or do I still need something from outside like the usermap kernel driver etc. >> >> Secondly, if I turn on all the debug options for the vmxnet3 pmd in the config file, 1.6.0r1 compilation runs into a problem and reports a function which is defined but not used. >> >> Can you send your build error ? (maybe in a separate thread ?) >> >> >> I am trying to bring up DPDK inside Fedora18 Guest on ESXi -- when I used DPDK1.6.0r1 (without debug options turned on for vmxnet3 pmd) the igb_uio could take over the vmxnet3 NIC but I encountered a core dump in the dev init function for the vmxnet3 driver -- anybody encountered a similar issue ? >> >> I encountered these problems as well. >> >> - igb_uio module does not check if you disable vmxnet3-uio pmds, it will always try to take over vmxnet3 devices. >> I have a patch waiting in my working dir to cleanly disable vmxnet3-uio pmd. >> >> - If you don't bind vmxnet3 devices to uio, but forget to enable vmxnet3-usermap pmd (by specifiying -d librte_pmd_vmxnet3.so), then internal vmxnet3-uio pmd will try to initialise and crash. >> I did not look any deeper into this, the easiest way is to disable vmxnet3-uio pmd + apply the patch I will send in a few minutes, as a first workaround. >> >> >> Regards, >> -- >> David Marchand >> >> >> >> >> ===================================================================== >> = >> ========= Please refer to >> http://www.aricent.com/legal/email_disclaimer.html >> for important disclosures regarding this electronic communication. >> ===================================================================== >> = >> ========= >> >> >> >> >> ===================================================================== >> = >> ========= Please refer to >> http://www.aricent.com/legal/email_disclaimer.html >> for important disclosures regarding this electronic communication. >> ===================================================================== >> = >> ========= > > > > > ====================================================================== > ========= Please refer to > http://www.aricent.com/legal/email_disclaimer.html > for important disclosures regarding this electronic communication. > ====================================================================== > ========= =============================================================================== Please refer to http://www.aricent.com/legal/email_disclaimer.html for important disclosures regarding this electronic communication. =============================================================================== =============================================================================== Please refer to http://www.aricent.com/legal/email_disclaimer.html for important disclosures regarding this electronic communication. =============================================================================== ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? 2014-03-10 8:00 ` Prashant Upadhyaya 2014-03-10 10:54 ` Prashant Upadhyaya @ 2014-03-10 15:50 ` Stephen Hemminger 2014-03-11 5:27 ` Prashant Upadhyaya 2014-03-11 5:33 ` [dpdk-dev] On vmxnet-pmd crash in DPDK 1.6.0r1 Daniel Kan 2 siblings, 1 reply; 21+ messages in thread From: Stephen Hemminger @ 2014-03-10 15:50 UTC (permalink / raw) To: Prashant Upadhyaya; +Cc: dev On Mon, 10 Mar 2014 13:30:48 +0530 Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: > Hi Srini, > > Thanks, I could also make it work, thanks to your cue ! > > Now then, this multi-segment not being supported in vmxnet3 driver is a big party-pooper for me. Unfortunately in my usecase, I do indeed make heavy use of multisegment buffers for sending out the data, so my usecase has failed and I will have to fix that. > > Also, can you please adivse how much is the max data rates you have been able to achieve with one vmxnet3 10G port. > > Thanks a lot for the advice once again. > > Regards > -Prashant I am integrating our driver with the 1.6.1 DPDK driver. We support multi-segment, if you want I will backport that feature first. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? 2014-03-10 15:50 ` Stephen Hemminger @ 2014-03-11 5:27 ` Prashant Upadhyaya 2014-03-11 8:52 ` Prashant Upadhyaya 2014-03-21 8:11 ` Prashant Upadhyaya 0 siblings, 2 replies; 21+ messages in thread From: Prashant Upadhyaya @ 2014-03-11 5:27 UTC (permalink / raw) To: Stephen Hemminger; +Cc: dev Hi Stephen, This is great news ! I can wait for a formal release of DPDK with your driver. Please let me know when is the release expected. I will happily migrate to that. Regards -Prashant -----Original Message----- From: Stephen Hemminger [mailto:stephen@networkplumber.org] Sent: Monday, March 10, 2014 9:21 PM To: Prashant Upadhyaya Cc: Srinivasan J; dev@dpdk.org Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? On Mon, 10 Mar 2014 13:30:48 +0530 Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: > Hi Srini, > > Thanks, I could also make it work, thanks to your cue ! > > Now then, this multi-segment not being supported in vmxnet3 driver is a big party-pooper for me. Unfortunately in my usecase, I do indeed make heavy use of multisegment buffers for sending out the data, so my usecase has failed and I will have to fix that. > > Also, can you please adivse how much is the max data rates you have been able to achieve with one vmxnet3 10G port. > > Thanks a lot for the advice once again. > > Regards > -Prashant I am integrating our driver with the 1.6.1 DPDK driver. We support multi-segment, if you want I will backport that feature first. =============================================================================== Please refer to http://www.aricent.com/legal/email_disclaimer.html for important disclosures regarding this electronic communication. =============================================================================== ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? 2014-03-11 5:27 ` Prashant Upadhyaya @ 2014-03-11 8:52 ` Prashant Upadhyaya 2014-03-21 8:11 ` Prashant Upadhyaya 1 sibling, 0 replies; 21+ messages in thread From: Prashant Upadhyaya @ 2014-03-11 8:52 UTC (permalink / raw) To: Stephen Hemminger; +Cc: dev Hi Stephen, Can you please advise on your experience of the kind of data rates you have been able to achieve with vmxnet3. Also did you have to do any special optimizations at the vmnic of ESXi for the above, kindly let me know. Regards -Prashant -----Original Message----- From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Prashant Upadhyaya Sent: Tuesday, March 11, 2014 10:57 AM To: Stephen Hemminger Cc: dev@dpdk.org Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? Hi Stephen, This is great news ! I can wait for a formal release of DPDK with your driver. Please let me know when is the release expected. I will happily migrate to that. Regards -Prashant -----Original Message----- From: Stephen Hemminger [mailto:stephen@networkplumber.org] Sent: Monday, March 10, 2014 9:21 PM To: Prashant Upadhyaya Cc: Srinivasan J; dev@dpdk.org Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? On Mon, 10 Mar 2014 13:30:48 +0530 Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: > Hi Srini, > > Thanks, I could also make it work, thanks to your cue ! > > Now then, this multi-segment not being supported in vmxnet3 driver is a big party-pooper for me. Unfortunately in my usecase, I do indeed make heavy use of multisegment buffers for sending out the data, so my usecase has failed and I will have to fix that. > > Also, can you please adivse how much is the max data rates you have been able to achieve with one vmxnet3 10G port. > > Thanks a lot for the advice once again. > > Regards > -Prashant I am integrating our driver with the 1.6.1 DPDK driver. We support multi-segment, if you want I will backport that feature first. =============================================================================== Please refer to http://www.aricent.com/legal/email_disclaimer.html for important disclosures regarding this electronic communication. =============================================================================== =============================================================================== Please refer to http://www.aricent.com/legal/email_disclaimer.html for important disclosures regarding this electronic communication. =============================================================================== ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? 2014-03-11 5:27 ` Prashant Upadhyaya 2014-03-11 8:52 ` Prashant Upadhyaya @ 2014-03-21 8:11 ` Prashant Upadhyaya 2014-05-06 5:48 ` Prashant Upadhyaya 1 sibling, 1 reply; 21+ messages in thread From: Prashant Upadhyaya @ 2014-03-21 8:11 UTC (permalink / raw) To: Stephen Hemminger; +Cc: dev Hi Stephen, I believe the 1.6.0r2 is baking. It would be great if you could enhance the vmxnet3 driver on the above with multi-segment support. Any serious usecase ends up using multi-segment, so would be great if r2 can capture it. Regards -Prashant -----Original Message----- From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Prashant Upadhyaya Sent: Tuesday, March 11, 2014 10:57 AM To: Stephen Hemminger Cc: dev@dpdk.org Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? Hi Stephen, This is great news ! I can wait for a formal release of DPDK with your driver. Please let me know when is the release expected. I will happily migrate to that. Regards -Prashant -----Original Message----- From: Stephen Hemminger [mailto:stephen@networkplumber.org] Sent: Monday, March 10, 2014 9:21 PM To: Prashant Upadhyaya Cc: Srinivasan J; dev@dpdk.org Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? On Mon, 10 Mar 2014 13:30:48 +0530 Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: > Hi Srini, > > Thanks, I could also make it work, thanks to your cue ! > > Now then, this multi-segment not being supported in vmxnet3 driver is a big party-pooper for me. Unfortunately in my usecase, I do indeed make heavy use of multisegment buffers for sending out the data, so my usecase has failed and I will have to fix that. > > Also, can you please adivse how much is the max data rates you have been able to achieve with one vmxnet3 10G port. > > Thanks a lot for the advice once again. > > Regards > -Prashant I am integrating our driver with the 1.6.1 DPDK driver. We support multi-segment, if you want I will backport that feature first. =============================================================================== Please refer to http://www.aricent.com/legal/email_disclaimer.html for important disclosures regarding this electronic communication. =============================================================================== =============================================================================== Please refer to http://www.aricent.com/legal/email_disclaimer.html for important disclosures regarding this electronic communication. =============================================================================== ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? 2014-03-21 8:11 ` Prashant Upadhyaya @ 2014-05-06 5:48 ` Prashant Upadhyaya 0 siblings, 0 replies; 21+ messages in thread From: Prashant Upadhyaya @ 2014-05-06 5:48 UTC (permalink / raw) To: Stephen Hemminger; +Cc: dev Hi Stephen, Kindly let me know if the multi-segment support for vmxnet3 pmd is already in, in a formal release of DPDK. Or which formal release you are targeting this for. Regards -Prashant -----Original Message----- From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Prashant Upadhyaya Sent: Friday, March 21, 2014 1:42 PM To: Stephen Hemminger Cc: dev@dpdk.org Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? Hi Stephen, I believe the 1.6.0r2 is baking. It would be great if you could enhance the vmxnet3 driver on the above with multi-segment support. Any serious usecase ends up using multi-segment, so would be great if r2 can capture it. Regards -Prashant -----Original Message----- From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Prashant Upadhyaya Sent: Tuesday, March 11, 2014 10:57 AM To: Stephen Hemminger Cc: dev@dpdk.org Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? Hi Stephen, This is great news ! I can wait for a formal release of DPDK with your driver. Please let me know when is the release expected. I will happily migrate to that. Regards -Prashant -----Original Message----- From: Stephen Hemminger [mailto:stephen@networkplumber.org] Sent: Monday, March 10, 2014 9:21 PM To: Prashant Upadhyaya Cc: Srinivasan J; dev@dpdk.org Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? On Mon, 10 Mar 2014 13:30:48 +0530 Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: > Hi Srini, > > Thanks, I could also make it work, thanks to your cue ! > > Now then, this multi-segment not being supported in vmxnet3 driver is a big party-pooper for me. Unfortunately in my usecase, I do indeed make heavy use of multisegment buffers for sending out the data, so my usecase has failed and I will have to fix that. > > Also, can you please adivse how much is the max data rates you have been able to achieve with one vmxnet3 10G port. > > Thanks a lot for the advice once again. > > Regards > -Prashant I am integrating our driver with the 1.6.1 DPDK driver. We support multi-segment, if you want I will backport that feature first. =============================================================================== Please refer to http://www.aricent.com/legal/email_disclaimer.html for important disclosures regarding this electronic communication. =============================================================================== =============================================================================== Please refer to http://www.aricent.com/legal/email_disclaimer.html for important disclosures regarding this electronic communication. =============================================================================== "DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus." ^ permalink raw reply [flat|nested] 21+ messages in thread
* [dpdk-dev] On vmxnet-pmd crash in DPDK 1.6.0r1 2014-03-10 8:00 ` Prashant Upadhyaya 2014-03-10 10:54 ` Prashant Upadhyaya 2014-03-10 15:50 ` Stephen Hemminger @ 2014-03-11 5:33 ` Daniel Kan 2014-03-11 6:20 ` Daniel Kan 2 siblings, 1 reply; 21+ messages in thread From: Daniel Kan @ 2014-03-11 5:33 UTC (permalink / raw) To: dev I also got the segfault. Setting CONFIG_RTE_EAL_UNBIND_PORTS=y is not ideal because it would also unbind my VMNET3 management interface. Does anyone know why the crash is happening? According to the stacktrace, hw_addrX is 0x0 during eth_vmxnet3_dev_init. I’m running on ubuntu 12.04LTS on ESXi 5.5 with Intel I350 NIC as the physical adapter for VMNET3. Program received signal SIGSEGV, Segmentation fault. 0x00000000004fe0eb in vmxnet3_read_addr (addr=0x0) at /home/dkan/nyansa/3rd-party/dpdk-1.6.0r1/lib/librte_pmd_vmxnet3/vmxnet3_ethdev.h:139 (gdb) up #1 0x00000000004fe331 in eth_vmxnet3_dev_init (eth_drv=0x799440 <rte_vmxnet3_pmd>, eth_dev=0x7d5280 <rte_eth_devices>) at /home/dkan/nyansa/3rd-party/dpdk-1.6.0r1/lib/librte_pmd_v\ mxnet3/vmxnet3_ethdev.c:218 (gdb) print *hw $12 = {hw_addr0 = 0x0, hw_addr1 = 0x0, back = 0x0, device_id = 1968, vendor_id = 5549, subsystem_device_id = 0, subsystem_vendor_id = 0, adapter_stopped = 0, perm_addr = "\000\000\\ 000\000\000", num_tx_queues = 1 '\001', num_rx_queues = 1 '\001', bufs_per_pkt = 1 '\001', cur_mtu = 1500, tqd_start = 0x0, rqd_start = 0x0, shared = 0x0, sharedPA = 0, queueDescPA\ = 0, queue_desc_len = 0, rss_conf = 0x0, rss_confPA = 0, mf_table = 0x0} Thanks in advance. Dan On Mar 10, 2014, at 1:00 AM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: > Hi Srini, > > Thanks, I could also make it work, thanks to your cue ! > > Now then, this multi-segment not being supported in vmxnet3 driver is a big party-pooper for me. Unfortunately in my usecase, I do indeed make heavy use of multisegment buffers for sending out the data, so my usecase has failed and I will have to fix that. > > Also, can you please adivse how much is the max data rates you have been able to achieve with one vmxnet3 10G port. > > Thanks a lot for the advice once again. > > Regards > -Prashant > > > -----Original Message----- > From: Srinivasan J [mailto:srinidpdk@gmail.com] > Sent: Sunday, March 09, 2014 12:38 AM > To: Prashant Upadhyaya > Cc: David Marchand; dev@dpdk.org > Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? > > Prashant, > I was also able to hit the issue your hitting using Esxi > 5.1.0 evaluation and Fedora 20 X86_64 guest. I was able to fix the issue by setting CONFIG_RTE_EAL_UNBIND_PORTS=y option in defconfig_x86_64-default-linuxapp-gcc configuration file. > > Issue seen > ======== > > EAL: PCI device 0000:03:00.0 on NUMA socket -1 > EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd > EAL: Device is blacklisted, not initializing > EAL: PCI device 0000:0b:00.0 on NUMA socket -1 > EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd > > Program received signal SIGSEGV, Segmentation fault. > eth_vmxnet3_dev_init (eth_drv=<optimized out>, eth_dev=0x754480 > <rte_eth_devices>) at > /root/source/dpdk-1.6.0r1/lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c:218 > 218 ver = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_VRRS); > Missing separate debuginfos, use: debuginfo-install glibc-2.18-11.fc20.x86_64 > (gdb) p hw > $1 = (struct vmxnet3_hw *) 0x7fffd8fc1040 > (gdb) p *hw > $2 = {hw_addr0 = 0x0, hw_addr1 = 0x0, back = 0x0, device_id = 1968, vendor_id = 5549, subsystem_device_id = 0, subsystem_vendor_id = 0, > adapter_stopped = 0, perm_addr = "\000\000\000\000\000", num_tx_queues = 1 '\001', num_rx_queues = 1 '\001', bufs_per_pkt = 1 '\001', cur_mtu = 0, > tqd_start = 0x0, rqd_start = 0x0, shared = 0x0, sharedPA = 0, queueDescPA = 0, queue_desc_len = 0, rss_conf = 0x0, rss_confPA = 0, mf_table = 0x0} > (gdb) > > #define VMXNET3_PCI_BAR1_REG_ADDR(hw, reg) \ > ((volatile uint32_t *)((char *)(hw)->hw_addr1 + (reg))) #define VMXNET3_READ_BAR1_REG(hw, reg) \ > vmxnet3_read_addr(VMXNET3_PCI_BAR1_REG_ADDR((hw), (reg))) lib/librte_pmd_vmxnet3/vmxnet3_ethdev.h > > Issue not seen after enabling CONFIG_RTE_EAL_UNBIND_PORTS=y ================================================== > > [root@localhost build]# ./l2fwd -c 0xf -b 0000:03:00.0 -n 1 -- -p 0x6 > EAL: Detected lcore 0 as core 0 on socket 0 > EAL: Detected lcore 1 as core 1 on socket 0 > EAL: Detected lcore 2 as core 2 on socket 0 > EAL: Detected lcore 3 as core 3 on socket 0 > EAL: Skip lcore 4 (not detected) > EAL: Skip lcore 5 (not detected) > EAL: Skip lcore 6 (not detected) > EAL: Skip lcore 7 (not detected) > EAL: Skip lcore 8 (not detected) > EAL: Skip lcore 9 (not detected) > EAL: Skip lcore 10 (not detected) > EAL: Skip lcore 11 (not detected) > EAL: Skip lcore 12 (not detected) > EAL: Skip lcore 13 (not detected) > EAL: Skip lcore 14 (not detected) > EAL: Skip lcore 15 (not detected) > EAL: Skip lcore 16 (not detected) > EAL: Skip lcore 17 (not detected) > EAL: Skip lcore 18 (not detected) > EAL: Skip lcore 19 (not detected) > EAL: Skip lcore 20 (not detected) > EAL: Skip lcore 21 (not detected) > EAL: Skip lcore 22 (not detected) > EAL: Skip lcore 23 (not detected) > EAL: Skip lcore 24 (not detected) > EAL: Skip lcore 25 (not detected) > EAL: Skip lcore 26 (not detected) > EAL: Skip lcore 27 (not detected) > EAL: Skip lcore 28 (not detected) > EAL: Skip lcore 29 (not detected) > EAL: Skip lcore 30 (not detected) > EAL: Skip lcore 31 (not detected) > EAL: Skip lcore 32 (not detected) > EAL: Skip lcore 33 (not detected) > EAL: Skip lcore 34 (not detected) > EAL: Skip lcore 35 (not detected) > EAL: Skip lcore 36 (not detected) > EAL: Skip lcore 37 (not detected) > EAL: Skip lcore 38 (not detected) > EAL: Skip lcore 39 (not detected) > EAL: Skip lcore 40 (not detected) > EAL: Skip lcore 41 (not detected) > EAL: Skip lcore 42 (not detected) > EAL: Skip lcore 43 (not detected) > EAL: Skip lcore 44 (not detected) > EAL: Skip lcore 45 (not detected) > EAL: Skip lcore 46 (not detected) > EAL: Skip lcore 47 (not detected) > EAL: Skip lcore 48 (not detected) > EAL: Skip lcore 49 (not detected) > EAL: Skip lcore 50 (not detected) > EAL: Skip lcore 51 (not detected) > EAL: Skip lcore 52 (not detected) > EAL: Skip lcore 53 (not detected) > EAL: Skip lcore 54 (not detected) > EAL: Skip lcore 55 (not detected) > EAL: Skip lcore 56 (not detected) > EAL: Skip lcore 57 (not detected) > EAL: Skip lcore 58 (not detected) > EAL: Skip lcore 59 (not detected) > EAL: Skip lcore 60 (not detected) > EAL: Skip lcore 61 (not detected) > EAL: Skip lcore 62 (not detected) > EAL: Skip lcore 63 (not detected) > EAL: Setting up memory... > EAL: Ask a virtual area of 0x200000 bytes > EAL: Virtual area found at 0x7f3a76a00000 (size = 0x200000) > EAL: Ask a virtual area of 0x7c00000 bytes > EAL: Virtual area found at 0x7f3a6ec00000 (size = 0x7c00000) > EAL: Ask a virtual area of 0x200000 bytes > EAL: Virtual area found at 0x7f3a6e800000 (size = 0x200000) > EAL: Requesting 64 pages of size 2MB from socket 0 > EAL: TSC frequency is ~2200000 KHz > EAL: Master core 0 is ready (tid=76cfe880) > EAL: Core 3 is ready (tid=6cffc700) > EAL: Core 1 is ready (tid=6dffe700) > EAL: Core 2 is ready (tid=6d7fd700) > EAL: PCI device 0000:03:00.0 on NUMA socket -1 > EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd > EAL: Device is blacklisted, not initializing > EAL: PCI device 0000:0b:00.0 on NUMA socket -1 > EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd > EAL: PCI memory mapped at 0x7f3a76d10000 > EAL: PCI memory mapped at 0x7f3a76d0f000 > EAL: PCI memory mapped at 0x7f3a76d0d000 > EAL: PCI device 0000:13:00.0 on NUMA socket -1 > EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd > EAL: PCI memory mapped at 0x7f3a76d0c000 > EAL: PCI memory mapped at 0x7f3a76d0b000 > EAL: PCI memory mapped at 0x7f3a76d09000 > Notice: odd number of ports in portmask. > Lcore 0: RX port 1 > Skipping disabled port 0 > Initializing port 1... done: > Port 1, MAC address: 00:0C:29:F1:9D:A4 > > > Checking link statusdone > Port 1 Link Up - speed 10000 Mbps - full-duplex > L2FWD: lcore 1 has nothing to do > L2FWD: lcore 2 has nothing to do > L2FWD: lcore 3 has nothing to do > L2FWD: entering main loop on lcore 0 > L2FWD: -- lcoreid=0 portid=1 > > Port statistics ==================================== > Statistics for port 1 ------------------------------ > Packets sent: 0 > Packets received: 0 > Packets dropped: 0 > Statistics for port 2 ------------------------------ > Packets sent: 0 > Packets received: 0 > Packets dropped: 0 > Aggregate statistics =============================== > Total packets sent: 0 > Total packets received: 0 > Total packets dropped: 0 > ==================================================== > > > > > On Fri, Mar 7, 2014 at 10:43 AM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: >> Hi Srini, >> >> Yes, I am still stuck, I am using ESXi5.1 though and Fedora18 in Guest. >> Please let me know what is the guest OS distro you are using in the VM. I'll try with that once. >> >> Regards >> -Prashant >> >> >> -----Original Message----- >> From: Srinivasan J [mailto:srinidpdk@gmail.com] >> Sent: Friday, March 07, 2014 9:08 AM >> To: Prashant Upadhyaya >> Cc: David Marchand; dev@dpdk.org >> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >> >> Prashant, >> Are you still hitting a crash? Do you have hugetlbfs >> enabled? if not try adding enabling hugetlbfs. I was able to get >> vxnet3 up using the vmxnet3 pmd alone in DPDK 1.60r1. I did not use >> the vmxnet3-usermap kernel driver. I am using Esxi 5.5 >> >> Eg: echo 1024 > /proc/sys/vm/nr_hugepages; mkdir /huge; mount -t >> hugetlbfs nodev /huge; >> >> On Thu, Mar 6, 2014 at 12:48 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: >>> And if it is any help, here is the startup log -- >>> >>> EAL: Detected lcore 0 as core 0 on socket 0 >>> EAL: Detected lcore 1 as core 1 on socket 0 >>> EAL: Detected lcore 2 as core 2 on socket 0 >>> EAL: Detected lcore 3 as core 3 on socket 0 >>> EAL: Detected lcore 4 as core 4 on socket 0 >>> EAL: Detected lcore 5 as core 5 on socket 0 >>> EAL: Detected lcore 6 as core 6 on socket 0 >>> EAL: Detected lcore 7 as core 7 on socket 0 >>> EAL: Skip lcore 8 (not detected) >>> EAL: Skip lcore 9 (not detected) >>> EAL: Skip lcore 10 (not detected) >>> EAL: Skip lcore 11 (not detected) >>> EAL: Skip lcore 12 (not detected) >>> EAL: Skip lcore 13 (not detected) >>> EAL: Skip lcore 14 (not detected) >>> EAL: Skip lcore 15 (not detected) >>> EAL: Skip lcore 16 (not detected) >>> EAL: Skip lcore 17 (not detected) >>> EAL: Skip lcore 18 (not detected) >>> EAL: Skip lcore 19 (not detected) >>> EAL: Skip lcore 20 (not detected) >>> EAL: Skip lcore 21 (not detected) >>> EAL: Skip lcore 22 (not detected) >>> EAL: Skip lcore 23 (not detected) >>> EAL: Skip lcore 24 (not detected) >>> EAL: Skip lcore 25 (not detected) >>> EAL: Skip lcore 26 (not detected) >>> EAL: Skip lcore 27 (not detected) >>> EAL: Skip lcore 28 (not detected) >>> EAL: Skip lcore 29 (not detected) >>> EAL: Skip lcore 30 (not detected) >>> EAL: Skip lcore 31 (not detected) >>> EAL: Skip lcore 32 (not detected) >>> EAL: Skip lcore 33 (not detected) >>> EAL: Skip lcore 34 (not detected) >>> EAL: Skip lcore 35 (not detected) >>> EAL: Skip lcore 36 (not detected) >>> EAL: Skip lcore 37 (not detected) >>> EAL: Skip lcore 38 (not detected) >>> EAL: Skip lcore 39 (not detected) >>> EAL: Skip lcore 40 (not detected) >>> EAL: Skip lcore 41 (not detected) >>> EAL: Skip lcore 42 (not detected) >>> EAL: Skip lcore 43 (not detected) >>> EAL: Skip lcore 44 (not detected) >>> EAL: Skip lcore 45 (not detected) >>> EAL: Skip lcore 46 (not detected) >>> EAL: Skip lcore 47 (not detected) >>> EAL: Skip lcore 48 (not detected) >>> EAL: Skip lcore 49 (not detected) >>> EAL: Skip lcore 50 (not detected) >>> EAL: Skip lcore 51 (not detected) >>> EAL: Skip lcore 52 (not detected) >>> EAL: Skip lcore 53 (not detected) >>> EAL: Skip lcore 54 (not detected) >>> EAL: Skip lcore 55 (not detected) >>> EAL: Skip lcore 56 (not detected) >>> EAL: Skip lcore 57 (not detected) >>> EAL: Skip lcore 58 (not detected) >>> EAL: Skip lcore 59 (not detected) >>> EAL: Skip lcore 60 (not detected) >>> EAL: Skip lcore 61 (not detected) >>> EAL: Skip lcore 62 (not detected) >>> EAL: Skip lcore 63 (not detected) >>> EAL: Setting up memory... >>> EAL: Ask a virtual area of 0x80000000 bytes >>> EAL: Virtual area found at 0x7f848ae00000 (size = 0x80000000) >>> EAL: Requesting 1024 pages of size 2MB from socket 0 >>> EAL: TSC frequency is ~2000000 KHz >>> EAL: Master core 0 is ready (tid=b3f3f00) >>> EAL: Core 1 is ready (tid=8a1f2700) >>> EAL: Core 2 is ready (tid=899f1700) >>> EAL: Core 3 is ready (tid=891f0700) >>> EAL: Core 4 is ready (tid=889ef700) >>> EAL: Core 5 is ready (tid=7bfff700) >>> EAL: Core 6 is ready (tid=7b7fe700) >>> EAL: Core 7 is ready (tid=7affd700) >>> Pool initialized >>> Global Variables initialized >>> PMD: rte_vmxnet3_pmd_init(): >> >>> EAL: PCI device 0000:0b:00.0 on NUMA socket -1 >>> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd >>> PMD: eth_vmxnet3_dev_init(): >> >>> Segmentation fault (core dumped) >>> >>> -----Original Message----- >>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Prashant >>> Upadhyaya >>> Sent: Thursday, March 06, 2014 12:20 PM >>> To: David Marchand >>> Cc: dev@dpdk.org >>> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >>> >>> Hi, >>> >>> Some further update on the crash I am facing. >>> I am using DPDK1.6.0r1 and take over the vmxnet3 with igb_uio and >>> then start the application. (so no external ko or vmxnet usermap >>> etc.) During the port initializations, the crash is happening in the >>> following function - >>> >>> eth_vmxnet3_dev_init >>> >>> and the crash is happening at the following line - >>> /* Check h/w version compatibility with driver. */ >>> ver = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_VRRS); >>> >>> Any hints regarding what could be wrong ? >>> >>> Regards >>> -Prashant >>> >>> >>> From: Prashant Upadhyaya >>> Sent: Wednesday, March 05, 2014 9:01 PM >>> To: 'David Marchand' >>> Cc: Srinivasan J; dev@dpdk.org >>> Subject: RE: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >>> >>> Hi David, >>> >>> The compilation error with debug flags on was that the functions -- vmxnet3_txq_dump, vmxnet3_rxq_dump - are defined but not used. >>> Not a serious error, I will try to get rid of the compiler flag which generates this. >>> >>> However, I must reiterate, I _did_ bind my vmxnet3 device with igb_uio (and I did not use any .so, because I was intending to use the builtin vmxnet3 driver of dpdk 1.6.0r1), the bind succeeded, but then when I started the application, the dev init for vmxnet3 gave a core dump. >>> Your patch and solution seems to be suggesting the reverse, i.e. when I don't bind with igb_uio but try to use the native driver. So please do try the above combination as well. >>> >>> Regards >>> -Prashant >>> >>> >>> From: David Marchand [mailto:david.marchand@6wind.com] >>> Sent: Wednesday, March 05, 2014 8:41 PM >>> To: Prashant Upadhyaya >>> Cc: Srinivasan J; dev@dpdk.org<mailto:dev@dpdk.org> >>> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >>> >>> Hello Prashant, >>> >>> On Wed, Mar 5, 2014 at 3:28 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com<mailto:prashant.upadhyaya@aricent.com>> wrote: >>> Hi, >>> >>> I am also keen to know the answer to the question posted by Srini. >>> The real question is -- is dpdk1.6.0r1 self-sufficient so that I don't need any extensions etc., or do I still need something from outside like the usermap kernel driver etc. >>> >>> Secondly, if I turn on all the debug options for the vmxnet3 pmd in the config file, 1.6.0r1 compilation runs into a problem and reports a function which is defined but not used. >>> >>> Can you send your build error ? (maybe in a separate thread ?) >>> >>> >>> I am trying to bring up DPDK inside Fedora18 Guest on ESXi -- when I used DPDK1.6.0r1 (without debug options turned on for vmxnet3 pmd) the igb_uio could take over the vmxnet3 NIC but I encountered a core dump in the dev init function for the vmxnet3 driver -- anybody encountered a similar issue ? >>> >>> I encountered these problems as well. >>> >>> - igb_uio module does not check if you disable vmxnet3-uio pmds, it will always try to take over vmxnet3 devices. >>> I have a patch waiting in my working dir to cleanly disable vmxnet3-uio pmd. >>> >>> - If you don't bind vmxnet3 devices to uio, but forget to enable vmxnet3-usermap pmd (by specifiying -d librte_pmd_vmxnet3.so), then internal vmxnet3-uio pmd will try to initialise and crash. >>> I did not look any deeper into this, the easiest way is to disable vmxnet3-uio pmd + apply the patch I will send in a few minutes, as a first workaround. >>> >>> >>> Regards, >>> -- >>> David Marchand >>> >>> >>> >>> >>> ===================================================================== >>> = >>> ========= Please refer to >>> http://www.aricent.com/legal/email_disclaimer.html >>> for important disclosures regarding this electronic communication. >>> ===================================================================== >>> = >>> ========= >>> >>> >>> >>> >>> ===================================================================== >>> = >>> ========= Please refer to >>> http://www.aricent.com/legal/email_disclaimer.html >>> for important disclosures regarding this electronic communication. >>> ===================================================================== >>> = >>> ========= >> >> >> >> >> ====================================================================== >> ========= Please refer to >> http://www.aricent.com/legal/email_disclaimer.html >> for important disclosures regarding this electronic communication. >> ====================================================================== >> ========= > > > > > =============================================================================== > Please refer to http://www.aricent.com/legal/email_disclaimer.html > for important disclosures regarding this electronic communication. > =============================================================================== ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [dpdk-dev] On vmxnet-pmd crash in DPDK 1.6.0r1 2014-03-11 5:33 ` [dpdk-dev] On vmxnet-pmd crash in DPDK 1.6.0r1 Daniel Kan @ 2014-03-11 6:20 ` Daniel Kan 2014-03-11 6:29 ` Daniel Kan 0 siblings, 1 reply; 21+ messages in thread From: Daniel Kan @ 2014-03-11 6:20 UTC (permalink / raw) To: dev Upon further trace, I know what caused it. The uio hardware resources were never memory mapped when RTE_EAL_UNBIND_PORTS is not enabled. Specifically, pci_dev->mem_resource[] is not mapped. This explains why setting CONFIG_RTE_EAL_UNBIND_PORTS=y fixes the problem. lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c 266 static struct eth_driver rte_vmxnet3_pmd = { 267 { 268 .name = "rte_vmxnet3_pmd", 269 .id_table = pci_id_vmxnet3_map, 270 #ifdef RTE_EAL_UNBIND_PORTS 271 .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO, 272 #endif 273 }, 274 .eth_dev_init = eth_vmxnet3_dev_init, 275 .dev_private_size = sizeof(struct vmxnet3_adapter), 276 }; Note drv_flags will be 0. lib/librte_eal/linuxapp/eal/eal_pci.c 1039 #ifdef RTE_EAL_UNBIND_PORTS 1040 if (dr->drv_flags & RTE_PCI_DRV_NEED_IGB_UIO) { 1041 /* unbind driver and load uio resources for Intel NICs */ 1042 if (pci_switch_module(dr, dev, 1, IGB_UIO_NAME) < 0) ... 1050 #else 1051 if (dr->drv_flags & RTE_PCI_DRV_NEED_IGB_UIO) 1052 /* just map resources for Intel NICs */ 1053 if (pci_uio_map_resource(dev) < 0) 1054 return -1; 1055 #endif If RTE_EAL_UNBIND_PORTS is defined, pci_switch_module will call pci_uio_map_resource. I then looked at the bsdapp’s version, it simply has: lib/librte_eal/bsdapp/eal/eal_pci.c 479 /* just map the NIC resources */ 480 if (pci_uio_map_resource(dev) < 0) 481 return -1; I don’t know the history behind why .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO only for RTE_EAL_UNBIND_PORTS. Can we just ensure pci_uio_map_resource is called in linux just like bsd version? One way is to ensure drv_flags is always set to RTE_PCI_DRV_NEED_IGB_UIO, but I don’t know if this fix will break other parts. +++ vmxnet3_ethdev.c 2014-03-10 23:18:02.087742434 -0700 @@ -267,9 +267,7 @@ { .name = "rte_vmxnet3_pmd", .id_table = pci_id_vmxnet3_map, -#ifdef RTE_EAL_UNBIND_PORTS .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO, -#endif }, .eth_dev_init = eth_vmxnet3_dev_init, .dev_private_size = sizeof(struct vmxnet3_adapter), Thanks. Dan On Mar 10, 2014, at 10:33 PM, Daniel Kan <dan@nyansa.com> wrote: > I also got the segfault. Setting CONFIG_RTE_EAL_UNBIND_PORTS=y is not ideal because it would also unbind my VMNET3 management interface. > > Does anyone know why the crash is happening? According to the stacktrace, hw_addrX is 0x0 during eth_vmxnet3_dev_init. > > I’m running on ubuntu 12.04LTS on ESXi 5.5 with Intel I350 NIC as the physical adapter for VMNET3. > > Program received signal SIGSEGV, Segmentation fault. > 0x00000000004fe0eb in vmxnet3_read_addr (addr=0x0) at /home/dkan/nyansa/3rd-party/dpdk-1.6.0r1/lib/librte_pmd_vmxnet3/vmxnet3_ethdev.h:139 > (gdb) up > #1 0x00000000004fe331 in eth_vmxnet3_dev_init (eth_drv=0x799440 <rte_vmxnet3_pmd>, eth_dev=0x7d5280 <rte_eth_devices>) at /home/dkan/nyansa/3rd-party/dpdk-1.6.0r1/lib/librte_pmd_v\ > mxnet3/vmxnet3_ethdev.c:218 > (gdb) print *hw > $12 = {hw_addr0 = 0x0, hw_addr1 = 0x0, back = 0x0, device_id = 1968, vendor_id = 5549, subsystem_device_id = 0, subsystem_vendor_id = 0, adapter_stopped = 0, perm_addr = "\000\000\\ > 000\000\000", num_tx_queues = 1 '\001', num_rx_queues = 1 '\001', bufs_per_pkt = 1 '\001', cur_mtu = 1500, tqd_start = 0x0, rqd_start = 0x0, shared = 0x0, sharedPA = 0, queueDescPA\ > = 0, queue_desc_len = 0, rss_conf = 0x0, rss_confPA = 0, mf_table = 0x0} > > Thanks in advance. > > Dan > > On Mar 10, 2014, at 1:00 AM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: > >> Hi Srini, >> >> Thanks, I could also make it work, thanks to your cue ! >> >> Now then, this multi-segment not being supported in vmxnet3 driver is a big party-pooper for me. Unfortunately in my usecase, I do indeed make heavy use of multisegment buffers for sending out the data, so my usecase has failed and I will have to fix that. >> >> Also, can you please adivse how much is the max data rates you have been able to achieve with one vmxnet3 10G port. >> >> Thanks a lot for the advice once again. >> >> Regards >> -Prashant >> >> >> -----Original Message----- >> From: Srinivasan J [mailto:srinidpdk@gmail.com] >> Sent: Sunday, March 09, 2014 12:38 AM >> To: Prashant Upadhyaya >> Cc: David Marchand; dev@dpdk.org >> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >> >> Prashant, >> I was also able to hit the issue your hitting using Esxi >> 5.1.0 evaluation and Fedora 20 X86_64 guest. I was able to fix the issue by setting CONFIG_RTE_EAL_UNBIND_PORTS=y option in defconfig_x86_64-default-linuxapp-gcc configuration file. >> >> Issue seen >> ======== >> >> EAL: PCI device 0000:03:00.0 on NUMA socket -1 >> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd >> EAL: Device is blacklisted, not initializing >> EAL: PCI device 0000:0b:00.0 on NUMA socket -1 >> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd >> >> Program received signal SIGSEGV, Segmentation fault. >> eth_vmxnet3_dev_init (eth_drv=<optimized out>, eth_dev=0x754480 >> <rte_eth_devices>) at >> /root/source/dpdk-1.6.0r1/lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c:218 >> 218 ver = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_VRRS); >> Missing separate debuginfos, use: debuginfo-install glibc-2.18-11.fc20.x86_64 >> (gdb) p hw >> $1 = (struct vmxnet3_hw *) 0x7fffd8fc1040 >> (gdb) p *hw >> $2 = {hw_addr0 = 0x0, hw_addr1 = 0x0, back = 0x0, device_id = 1968, vendor_id = 5549, subsystem_device_id = 0, subsystem_vendor_id = 0, >> adapter_stopped = 0, perm_addr = "\000\000\000\000\000", num_tx_queues = 1 '\001', num_rx_queues = 1 '\001', bufs_per_pkt = 1 '\001', cur_mtu = 0, >> tqd_start = 0x0, rqd_start = 0x0, shared = 0x0, sharedPA = 0, queueDescPA = 0, queue_desc_len = 0, rss_conf = 0x0, rss_confPA = 0, mf_table = 0x0} >> (gdb) >> >> #define VMXNET3_PCI_BAR1_REG_ADDR(hw, reg) \ >> ((volatile uint32_t *)((char *)(hw)->hw_addr1 + (reg))) #define VMXNET3_READ_BAR1_REG(hw, reg) \ >> vmxnet3_read_addr(VMXNET3_PCI_BAR1_REG_ADDR((hw), (reg))) lib/librte_pmd_vmxnet3/vmxnet3_ethdev.h >> >> Issue not seen after enabling CONFIG_RTE_EAL_UNBIND_PORTS=y ================================================== >> >> [root@localhost build]# ./l2fwd -c 0xf -b 0000:03:00.0 -n 1 -- -p 0x6 >> EAL: Detected lcore 0 as core 0 on socket 0 >> EAL: Detected lcore 1 as core 1 on socket 0 >> EAL: Detected lcore 2 as core 2 on socket 0 >> EAL: Detected lcore 3 as core 3 on socket 0 >> EAL: Skip lcore 4 (not detected) >> EAL: Skip lcore 5 (not detected) >> EAL: Skip lcore 6 (not detected) >> EAL: Skip lcore 7 (not detected) >> EAL: Skip lcore 8 (not detected) >> EAL: Skip lcore 9 (not detected) >> EAL: Skip lcore 10 (not detected) >> EAL: Skip lcore 11 (not detected) >> EAL: Skip lcore 12 (not detected) >> EAL: Skip lcore 13 (not detected) >> EAL: Skip lcore 14 (not detected) >> EAL: Skip lcore 15 (not detected) >> EAL: Skip lcore 16 (not detected) >> EAL: Skip lcore 17 (not detected) >> EAL: Skip lcore 18 (not detected) >> EAL: Skip lcore 19 (not detected) >> EAL: Skip lcore 20 (not detected) >> EAL: Skip lcore 21 (not detected) >> EAL: Skip lcore 22 (not detected) >> EAL: Skip lcore 23 (not detected) >> EAL: Skip lcore 24 (not detected) >> EAL: Skip lcore 25 (not detected) >> EAL: Skip lcore 26 (not detected) >> EAL: Skip lcore 27 (not detected) >> EAL: Skip lcore 28 (not detected) >> EAL: Skip lcore 29 (not detected) >> EAL: Skip lcore 30 (not detected) >> EAL: Skip lcore 31 (not detected) >> EAL: Skip lcore 32 (not detected) >> EAL: Skip lcore 33 (not detected) >> EAL: Skip lcore 34 (not detected) >> EAL: Skip lcore 35 (not detected) >> EAL: Skip lcore 36 (not detected) >> EAL: Skip lcore 37 (not detected) >> EAL: Skip lcore 38 (not detected) >> EAL: Skip lcore 39 (not detected) >> EAL: Skip lcore 40 (not detected) >> EAL: Skip lcore 41 (not detected) >> EAL: Skip lcore 42 (not detected) >> EAL: Skip lcore 43 (not detected) >> EAL: Skip lcore 44 (not detected) >> EAL: Skip lcore 45 (not detected) >> EAL: Skip lcore 46 (not detected) >> EAL: Skip lcore 47 (not detected) >> EAL: Skip lcore 48 (not detected) >> EAL: Skip lcore 49 (not detected) >> EAL: Skip lcore 50 (not detected) >> EAL: Skip lcore 51 (not detected) >> EAL: Skip lcore 52 (not detected) >> EAL: Skip lcore 53 (not detected) >> EAL: Skip lcore 54 (not detected) >> EAL: Skip lcore 55 (not detected) >> EAL: Skip lcore 56 (not detected) >> EAL: Skip lcore 57 (not detected) >> EAL: Skip lcore 58 (not detected) >> EAL: Skip lcore 59 (not detected) >> EAL: Skip lcore 60 (not detected) >> EAL: Skip lcore 61 (not detected) >> EAL: Skip lcore 62 (not detected) >> EAL: Skip lcore 63 (not detected) >> EAL: Setting up memory... >> EAL: Ask a virtual area of 0x200000 bytes >> EAL: Virtual area found at 0x7f3a76a00000 (size = 0x200000) >> EAL: Ask a virtual area of 0x7c00000 bytes >> EAL: Virtual area found at 0x7f3a6ec00000 (size = 0x7c00000) >> EAL: Ask a virtual area of 0x200000 bytes >> EAL: Virtual area found at 0x7f3a6e800000 (size = 0x200000) >> EAL: Requesting 64 pages of size 2MB from socket 0 >> EAL: TSC frequency is ~2200000 KHz >> EAL: Master core 0 is ready (tid=76cfe880) >> EAL: Core 3 is ready (tid=6cffc700) >> EAL: Core 1 is ready (tid=6dffe700) >> EAL: Core 2 is ready (tid=6d7fd700) >> EAL: PCI device 0000:03:00.0 on NUMA socket -1 >> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd >> EAL: Device is blacklisted, not initializing >> EAL: PCI device 0000:0b:00.0 on NUMA socket -1 >> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd >> EAL: PCI memory mapped at 0x7f3a76d10000 >> EAL: PCI memory mapped at 0x7f3a76d0f000 >> EAL: PCI memory mapped at 0x7f3a76d0d000 >> EAL: PCI device 0000:13:00.0 on NUMA socket -1 >> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd >> EAL: PCI memory mapped at 0x7f3a76d0c000 >> EAL: PCI memory mapped at 0x7f3a76d0b000 >> EAL: PCI memory mapped at 0x7f3a76d09000 >> Notice: odd number of ports in portmask. >> Lcore 0: RX port 1 >> Skipping disabled port 0 >> Initializing port 1... done: >> Port 1, MAC address: 00:0C:29:F1:9D:A4 >> >> >> Checking link statusdone >> Port 1 Link Up - speed 10000 Mbps - full-duplex >> L2FWD: lcore 1 has nothing to do >> L2FWD: lcore 2 has nothing to do >> L2FWD: lcore 3 has nothing to do >> L2FWD: entering main loop on lcore 0 >> L2FWD: -- lcoreid=0 portid=1 >> >> Port statistics ==================================== >> Statistics for port 1 ------------------------------ >> Packets sent: 0 >> Packets received: 0 >> Packets dropped: 0 >> Statistics for port 2 ------------------------------ >> Packets sent: 0 >> Packets received: 0 >> Packets dropped: 0 >> Aggregate statistics =============================== >> Total packets sent: 0 >> Total packets received: 0 >> Total packets dropped: 0 >> ==================================================== >> >> >> >> >> On Fri, Mar 7, 2014 at 10:43 AM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: >>> Hi Srini, >>> >>> Yes, I am still stuck, I am using ESXi5.1 though and Fedora18 in Guest. >>> Please let me know what is the guest OS distro you are using in the VM. I'll try with that once. >>> >>> Regards >>> -Prashant >>> >>> >>> -----Original Message----- >>> From: Srinivasan J [mailto:srinidpdk@gmail.com] >>> Sent: Friday, March 07, 2014 9:08 AM >>> To: Prashant Upadhyaya >>> Cc: David Marchand; dev@dpdk.org >>> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >>> >>> Prashant, >>> Are you still hitting a crash? Do you have hugetlbfs >>> enabled? if not try adding enabling hugetlbfs. I was able to get >>> vxnet3 up using the vmxnet3 pmd alone in DPDK 1.60r1. I did not use >>> the vmxnet3-usermap kernel driver. I am using Esxi 5.5 >>> >>> Eg: echo 1024 > /proc/sys/vm/nr_hugepages; mkdir /huge; mount -t >>> hugetlbfs nodev /huge; >>> >>> On Thu, Mar 6, 2014 at 12:48 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: >>>> And if it is any help, here is the startup log -- >>>> >>>> EAL: Detected lcore 0 as core 0 on socket 0 >>>> EAL: Detected lcore 1 as core 1 on socket 0 >>>> EAL: Detected lcore 2 as core 2 on socket 0 >>>> EAL: Detected lcore 3 as core 3 on socket 0 >>>> EAL: Detected lcore 4 as core 4 on socket 0 >>>> EAL: Detected lcore 5 as core 5 on socket 0 >>>> EAL: Detected lcore 6 as core 6 on socket 0 >>>> EAL: Detected lcore 7 as core 7 on socket 0 >>>> EAL: Skip lcore 8 (not detected) >>>> EAL: Skip lcore 9 (not detected) >>>> EAL: Skip lcore 10 (not detected) >>>> EAL: Skip lcore 11 (not detected) >>>> EAL: Skip lcore 12 (not detected) >>>> EAL: Skip lcore 13 (not detected) >>>> EAL: Skip lcore 14 (not detected) >>>> EAL: Skip lcore 15 (not detected) >>>> EAL: Skip lcore 16 (not detected) >>>> EAL: Skip lcore 17 (not detected) >>>> EAL: Skip lcore 18 (not detected) >>>> EAL: Skip lcore 19 (not detected) >>>> EAL: Skip lcore 20 (not detected) >>>> EAL: Skip lcore 21 (not detected) >>>> EAL: Skip lcore 22 (not detected) >>>> EAL: Skip lcore 23 (not detected) >>>> EAL: Skip lcore 24 (not detected) >>>> EAL: Skip lcore 25 (not detected) >>>> EAL: Skip lcore 26 (not detected) >>>> EAL: Skip lcore 27 (not detected) >>>> EAL: Skip lcore 28 (not detected) >>>> EAL: Skip lcore 29 (not detected) >>>> EAL: Skip lcore 30 (not detected) >>>> EAL: Skip lcore 31 (not detected) >>>> EAL: Skip lcore 32 (not detected) >>>> EAL: Skip lcore 33 (not detected) >>>> EAL: Skip lcore 34 (not detected) >>>> EAL: Skip lcore 35 (not detected) >>>> EAL: Skip lcore 36 (not detected) >>>> EAL: Skip lcore 37 (not detected) >>>> EAL: Skip lcore 38 (not detected) >>>> EAL: Skip lcore 39 (not detected) >>>> EAL: Skip lcore 40 (not detected) >>>> EAL: Skip lcore 41 (not detected) >>>> EAL: Skip lcore 42 (not detected) >>>> EAL: Skip lcore 43 (not detected) >>>> EAL: Skip lcore 44 (not detected) >>>> EAL: Skip lcore 45 (not detected) >>>> EAL: Skip lcore 46 (not detected) >>>> EAL: Skip lcore 47 (not detected) >>>> EAL: Skip lcore 48 (not detected) >>>> EAL: Skip lcore 49 (not detected) >>>> EAL: Skip lcore 50 (not detected) >>>> EAL: Skip lcore 51 (not detected) >>>> EAL: Skip lcore 52 (not detected) >>>> EAL: Skip lcore 53 (not detected) >>>> EAL: Skip lcore 54 (not detected) >>>> EAL: Skip lcore 55 (not detected) >>>> EAL: Skip lcore 56 (not detected) >>>> EAL: Skip lcore 57 (not detected) >>>> EAL: Skip lcore 58 (not detected) >>>> EAL: Skip lcore 59 (not detected) >>>> EAL: Skip lcore 60 (not detected) >>>> EAL: Skip lcore 61 (not detected) >>>> EAL: Skip lcore 62 (not detected) >>>> EAL: Skip lcore 63 (not detected) >>>> EAL: Setting up memory... >>>> EAL: Ask a virtual area of 0x80000000 bytes >>>> EAL: Virtual area found at 0x7f848ae00000 (size = 0x80000000) >>>> EAL: Requesting 1024 pages of size 2MB from socket 0 >>>> EAL: TSC frequency is ~2000000 KHz >>>> EAL: Master core 0 is ready (tid=b3f3f00) >>>> EAL: Core 1 is ready (tid=8a1f2700) >>>> EAL: Core 2 is ready (tid=899f1700) >>>> EAL: Core 3 is ready (tid=891f0700) >>>> EAL: Core 4 is ready (tid=889ef700) >>>> EAL: Core 5 is ready (tid=7bfff700) >>>> EAL: Core 6 is ready (tid=7b7fe700) >>>> EAL: Core 7 is ready (tid=7affd700) >>>> Pool initialized >>>> Global Variables initialized >>>> PMD: rte_vmxnet3_pmd_init(): >> >>>> EAL: PCI device 0000:0b:00.0 on NUMA socket -1 >>>> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd >>>> PMD: eth_vmxnet3_dev_init(): >> >>>> Segmentation fault (core dumped) >>>> >>>> -----Original Message----- >>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Prashant >>>> Upadhyaya >>>> Sent: Thursday, March 06, 2014 12:20 PM >>>> To: David Marchand >>>> Cc: dev@dpdk.org >>>> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >>>> >>>> Hi, >>>> >>>> Some further update on the crash I am facing. >>>> I am using DPDK1.6.0r1 and take over the vmxnet3 with igb_uio and >>>> then start the application. (so no external ko or vmxnet usermap >>>> etc.) During the port initializations, the crash is happening in the >>>> following function - >>>> >>>> eth_vmxnet3_dev_init >>>> >>>> and the crash is happening at the following line - >>>> /* Check h/w version compatibility with driver. */ >>>> ver = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_VRRS); >>>> >>>> Any hints regarding what could be wrong ? >>>> >>>> Regards >>>> -Prashant >>>> >>>> >>>> From: Prashant Upadhyaya >>>> Sent: Wednesday, March 05, 2014 9:01 PM >>>> To: 'David Marchand' >>>> Cc: Srinivasan J; dev@dpdk.org >>>> Subject: RE: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >>>> >>>> Hi David, >>>> >>>> The compilation error with debug flags on was that the functions -- vmxnet3_txq_dump, vmxnet3_rxq_dump - are defined but not used. >>>> Not a serious error, I will try to get rid of the compiler flag which generates this. >>>> >>>> However, I must reiterate, I _did_ bind my vmxnet3 device with igb_uio (and I did not use any .so, because I was intending to use the builtin vmxnet3 driver of dpdk 1.6.0r1), the bind succeeded, but then when I started the application, the dev init for vmxnet3 gave a core dump. >>>> Your patch and solution seems to be suggesting the reverse, i.e. when I don't bind with igb_uio but try to use the native driver. So please do try the above combination as well. >>>> >>>> Regards >>>> -Prashant >>>> >>>> >>>> From: David Marchand [mailto:david.marchand@6wind.com] >>>> Sent: Wednesday, March 05, 2014 8:41 PM >>>> To: Prashant Upadhyaya >>>> Cc: Srinivasan J; dev@dpdk.org<mailto:dev@dpdk.org> >>>> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >>>> >>>> Hello Prashant, >>>> >>>> On Wed, Mar 5, 2014 at 3:28 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com<mailto:prashant.upadhyaya@aricent.com>> wrote: >>>> Hi, >>>> >>>> I am also keen to know the answer to the question posted by Srini. >>>> The real question is -- is dpdk1.6.0r1 self-sufficient so that I don't need any extensions etc., or do I still need something from outside like the usermap kernel driver etc. >>>> >>>> Secondly, if I turn on all the debug options for the vmxnet3 pmd in the config file, 1.6.0r1 compilation runs into a problem and reports a function which is defined but not used. >>>> >>>> Can you send your build error ? (maybe in a separate thread ?) >>>> >>>> >>>> I am trying to bring up DPDK inside Fedora18 Guest on ESXi -- when I used DPDK1.6.0r1 (without debug options turned on for vmxnet3 pmd) the igb_uio could take over the vmxnet3 NIC but I encountered a core dump in the dev init function for the vmxnet3 driver -- anybody encountered a similar issue ? >>>> >>>> I encountered these problems as well. >>>> >>>> - igb_uio module does not check if you disable vmxnet3-uio pmds, it will always try to take over vmxnet3 devices. >>>> I have a patch waiting in my working dir to cleanly disable vmxnet3-uio pmd. >>>> >>>> - If you don't bind vmxnet3 devices to uio, but forget to enable vmxnet3-usermap pmd (by specifiying -d librte_pmd_vmxnet3.so), then internal vmxnet3-uio pmd will try to initialise and crash. >>>> I did not look any deeper into this, the easiest way is to disable vmxnet3-uio pmd + apply the patch I will send in a few minutes, as a first workaround. >>>> >>>> >>>> Regards, >>>> -- >>>> David Marchand >>>> >>>> >>>> >>>> >>>> ===================================================================== >>>> = >>>> ========= Please refer to >>>> http://www.aricent.com/legal/email_disclaimer.html >>>> for important disclosures regarding this electronic communication. >>>> ===================================================================== >>>> = >>>> ========= >>>> >>>> >>>> >>>> >>>> ===================================================================== >>>> = >>>> ========= Please refer to >>>> http://www.aricent.com/legal/email_disclaimer.html >>>> for important disclosures regarding this electronic communication. >>>> ===================================================================== >>>> = >>>> ========= >>> >>> >>> >>> >>> ====================================================================== >>> ========= Please refer to >>> http://www.aricent.com/legal/email_disclaimer.html >>> for important disclosures regarding this electronic communication. >>> ====================================================================== >>> ========= >> >> >> >> >> =============================================================================== >> Please refer to http://www.aricent.com/legal/email_disclaimer.html >> for important disclosures regarding this electronic communication. >> =============================================================================== > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [dpdk-dev] On vmxnet-pmd crash in DPDK 1.6.0r1 2014-03-11 6:20 ` Daniel Kan @ 2014-03-11 6:29 ` Daniel Kan 2014-03-21 14:10 ` Thomas Monjalon 0 siblings, 1 reply; 21+ messages in thread From: Daniel Kan @ 2014-03-11 6:29 UTC (permalink / raw) To: dev virtio-pmd has the same pattern. I wonder if vmxnet3-pmd just blindly copied the same paradigm. lib/librte_pmd_virtio/virtio_ethdev.c 473 static struct eth_driver rte_virtio_pmd = { 474 { 475 .name = "rte_virtio_pmd", 476 .id_table = pci_id_virtio_map, 477 #ifdef RTE_EAL_UNBIND_PORTS 478 .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO, 479 #endif 480 }, 481 .eth_dev_init = eth_virtio_dev_init, 482 .dev_private_size = sizeof(struct virtio_adapter), 483 }; On Mar 10, 2014, at 11:20 PM, Daniel Kan <dan@nyansa.com> wrote: > Upon further trace, I know what caused it. The uio hardware resources were never memory mapped when RTE_EAL_UNBIND_PORTS is not enabled. Specifically, pci_dev->mem_resource[] is not mapped. This explains why setting CONFIG_RTE_EAL_UNBIND_PORTS=y fixes the problem. > > lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c > 266 static struct eth_driver rte_vmxnet3_pmd = { > 267 { > 268 .name = "rte_vmxnet3_pmd", > 269 .id_table = pci_id_vmxnet3_map, > 270 #ifdef RTE_EAL_UNBIND_PORTS > 271 .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO, > 272 #endif > 273 }, > 274 .eth_dev_init = eth_vmxnet3_dev_init, > 275 .dev_private_size = sizeof(struct vmxnet3_adapter), > 276 }; > > Note drv_flags will be 0. > > lib/librte_eal/linuxapp/eal/eal_pci.c > 1039 #ifdef RTE_EAL_UNBIND_PORTS > 1040 if (dr->drv_flags & RTE_PCI_DRV_NEED_IGB_UIO) { > 1041 /* unbind driver and load uio resources for Intel NICs */ > 1042 if (pci_switch_module(dr, dev, 1, IGB_UIO_NAME) < 0) > ... > 1050 #else > 1051 if (dr->drv_flags & RTE_PCI_DRV_NEED_IGB_UIO) > 1052 /* just map resources for Intel NICs */ > 1053 if (pci_uio_map_resource(dev) < 0) > 1054 return -1; > 1055 #endif > > If RTE_EAL_UNBIND_PORTS is defined, pci_switch_module will call pci_uio_map_resource. > > I then looked at the bsdapp’s version, it simply has: > lib/librte_eal/bsdapp/eal/eal_pci.c > 479 /* just map the NIC resources */ > 480 if (pci_uio_map_resource(dev) < 0) > 481 return -1; > > I don’t know the history behind why .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO only for RTE_EAL_UNBIND_PORTS. Can we just ensure pci_uio_map_resource is called in linux just like bsd version? One way is to ensure drv_flags is always set to RTE_PCI_DRV_NEED_IGB_UIO, but I don’t know if this fix will break other parts. > > +++ vmxnet3_ethdev.c 2014-03-10 23:18:02.087742434 -0700 > @@ -267,9 +267,7 @@ > { > .name = "rte_vmxnet3_pmd", > .id_table = pci_id_vmxnet3_map, > -#ifdef RTE_EAL_UNBIND_PORTS > .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO, > -#endif > }, > .eth_dev_init = eth_vmxnet3_dev_init, > .dev_private_size = sizeof(struct vmxnet3_adapter), > > Thanks. > > Dan > > On Mar 10, 2014, at 10:33 PM, Daniel Kan <dan@nyansa.com> wrote: > >> I also got the segfault. Setting CONFIG_RTE_EAL_UNBIND_PORTS=y is not ideal because it would also unbind my VMNET3 management interface. >> >> Does anyone know why the crash is happening? According to the stacktrace, hw_addrX is 0x0 during eth_vmxnet3_dev_init. >> >> I’m running on ubuntu 12.04LTS on ESXi 5.5 with Intel I350 NIC as the physical adapter for VMNET3. >> >> Program received signal SIGSEGV, Segmentation fault. >> 0x00000000004fe0eb in vmxnet3_read_addr (addr=0x0) at /home/dkan/nyansa/3rd-party/dpdk-1.6.0r1/lib/librte_pmd_vmxnet3/vmxnet3_ethdev.h:139 >> (gdb) up >> #1 0x00000000004fe331 in eth_vmxnet3_dev_init (eth_drv=0x799440 <rte_vmxnet3_pmd>, eth_dev=0x7d5280 <rte_eth_devices>) at /home/dkan/nyansa/3rd-party/dpdk-1.6.0r1/lib/librte_pmd_v\ >> mxnet3/vmxnet3_ethdev.c:218 >> (gdb) print *hw >> $12 = {hw_addr0 = 0x0, hw_addr1 = 0x0, back = 0x0, device_id = 1968, vendor_id = 5549, subsystem_device_id = 0, subsystem_vendor_id = 0, adapter_stopped = 0, perm_addr = "\000\000\\ >> 000\000\000", num_tx_queues = 1 '\001', num_rx_queues = 1 '\001', bufs_per_pkt = 1 '\001', cur_mtu = 1500, tqd_start = 0x0, rqd_start = 0x0, shared = 0x0, sharedPA = 0, queueDescPA\ >> = 0, queue_desc_len = 0, rss_conf = 0x0, rss_confPA = 0, mf_table = 0x0} >> >> Thanks in advance. >> >> Dan >> >> On Mar 10, 2014, at 1:00 AM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: >> >>> Hi Srini, >>> >>> Thanks, I could also make it work, thanks to your cue ! >>> >>> Now then, this multi-segment not being supported in vmxnet3 driver is a big party-pooper for me. Unfortunately in my usecase, I do indeed make heavy use of multisegment buffers for sending out the data, so my usecase has failed and I will have to fix that. >>> >>> Also, can you please adivse how much is the max data rates you have been able to achieve with one vmxnet3 10G port. >>> >>> Thanks a lot for the advice once again. >>> >>> Regards >>> -Prashant >>> >>> >>> -----Original Message----- >>> From: Srinivasan J [mailto:srinidpdk@gmail.com] >>> Sent: Sunday, March 09, 2014 12:38 AM >>> To: Prashant Upadhyaya >>> Cc: David Marchand; dev@dpdk.org >>> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >>> >>> Prashant, >>> I was also able to hit the issue your hitting using Esxi >>> 5.1.0 evaluation and Fedora 20 X86_64 guest. I was able to fix the issue by setting CONFIG_RTE_EAL_UNBIND_PORTS=y option in defconfig_x86_64-default-linuxapp-gcc configuration file. >>> >>> Issue seen >>> ======== >>> >>> EAL: PCI device 0000:03:00.0 on NUMA socket -1 >>> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd >>> EAL: Device is blacklisted, not initializing >>> EAL: PCI device 0000:0b:00.0 on NUMA socket -1 >>> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd >>> >>> Program received signal SIGSEGV, Segmentation fault. >>> eth_vmxnet3_dev_init (eth_drv=<optimized out>, eth_dev=0x754480 >>> <rte_eth_devices>) at >>> /root/source/dpdk-1.6.0r1/lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c:218 >>> 218 ver = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_VRRS); >>> Missing separate debuginfos, use: debuginfo-install glibc-2.18-11.fc20.x86_64 >>> (gdb) p hw >>> $1 = (struct vmxnet3_hw *) 0x7fffd8fc1040 >>> (gdb) p *hw >>> $2 = {hw_addr0 = 0x0, hw_addr1 = 0x0, back = 0x0, device_id = 1968, vendor_id = 5549, subsystem_device_id = 0, subsystem_vendor_id = 0, >>> adapter_stopped = 0, perm_addr = "\000\000\000\000\000", num_tx_queues = 1 '\001', num_rx_queues = 1 '\001', bufs_per_pkt = 1 '\001', cur_mtu = 0, >>> tqd_start = 0x0, rqd_start = 0x0, shared = 0x0, sharedPA = 0, queueDescPA = 0, queue_desc_len = 0, rss_conf = 0x0, rss_confPA = 0, mf_table = 0x0} >>> (gdb) >>> >>> #define VMXNET3_PCI_BAR1_REG_ADDR(hw, reg) \ >>> ((volatile uint32_t *)((char *)(hw)->hw_addr1 + (reg))) #define VMXNET3_READ_BAR1_REG(hw, reg) \ >>> vmxnet3_read_addr(VMXNET3_PCI_BAR1_REG_ADDR((hw), (reg))) lib/librte_pmd_vmxnet3/vmxnet3_ethdev.h >>> >>> Issue not seen after enabling CONFIG_RTE_EAL_UNBIND_PORTS=y ================================================== >>> >>> [root@localhost build]# ./l2fwd -c 0xf -b 0000:03:00.0 -n 1 -- -p 0x6 >>> EAL: Detected lcore 0 as core 0 on socket 0 >>> EAL: Detected lcore 1 as core 1 on socket 0 >>> EAL: Detected lcore 2 as core 2 on socket 0 >>> EAL: Detected lcore 3 as core 3 on socket 0 >>> EAL: Skip lcore 4 (not detected) >>> EAL: Skip lcore 5 (not detected) >>> EAL: Skip lcore 6 (not detected) >>> EAL: Skip lcore 7 (not detected) >>> EAL: Skip lcore 8 (not detected) >>> EAL: Skip lcore 9 (not detected) >>> EAL: Skip lcore 10 (not detected) >>> EAL: Skip lcore 11 (not detected) >>> EAL: Skip lcore 12 (not detected) >>> EAL: Skip lcore 13 (not detected) >>> EAL: Skip lcore 14 (not detected) >>> EAL: Skip lcore 15 (not detected) >>> EAL: Skip lcore 16 (not detected) >>> EAL: Skip lcore 17 (not detected) >>> EAL: Skip lcore 18 (not detected) >>> EAL: Skip lcore 19 (not detected) >>> EAL: Skip lcore 20 (not detected) >>> EAL: Skip lcore 21 (not detected) >>> EAL: Skip lcore 22 (not detected) >>> EAL: Skip lcore 23 (not detected) >>> EAL: Skip lcore 24 (not detected) >>> EAL: Skip lcore 25 (not detected) >>> EAL: Skip lcore 26 (not detected) >>> EAL: Skip lcore 27 (not detected) >>> EAL: Skip lcore 28 (not detected) >>> EAL: Skip lcore 29 (not detected) >>> EAL: Skip lcore 30 (not detected) >>> EAL: Skip lcore 31 (not detected) >>> EAL: Skip lcore 32 (not detected) >>> EAL: Skip lcore 33 (not detected) >>> EAL: Skip lcore 34 (not detected) >>> EAL: Skip lcore 35 (not detected) >>> EAL: Skip lcore 36 (not detected) >>> EAL: Skip lcore 37 (not detected) >>> EAL: Skip lcore 38 (not detected) >>> EAL: Skip lcore 39 (not detected) >>> EAL: Skip lcore 40 (not detected) >>> EAL: Skip lcore 41 (not detected) >>> EAL: Skip lcore 42 (not detected) >>> EAL: Skip lcore 43 (not detected) >>> EAL: Skip lcore 44 (not detected) >>> EAL: Skip lcore 45 (not detected) >>> EAL: Skip lcore 46 (not detected) >>> EAL: Skip lcore 47 (not detected) >>> EAL: Skip lcore 48 (not detected) >>> EAL: Skip lcore 49 (not detected) >>> EAL: Skip lcore 50 (not detected) >>> EAL: Skip lcore 51 (not detected) >>> EAL: Skip lcore 52 (not detected) >>> EAL: Skip lcore 53 (not detected) >>> EAL: Skip lcore 54 (not detected) >>> EAL: Skip lcore 55 (not detected) >>> EAL: Skip lcore 56 (not detected) >>> EAL: Skip lcore 57 (not detected) >>> EAL: Skip lcore 58 (not detected) >>> EAL: Skip lcore 59 (not detected) >>> EAL: Skip lcore 60 (not detected) >>> EAL: Skip lcore 61 (not detected) >>> EAL: Skip lcore 62 (not detected) >>> EAL: Skip lcore 63 (not detected) >>> EAL: Setting up memory... >>> EAL: Ask a virtual area of 0x200000 bytes >>> EAL: Virtual area found at 0x7f3a76a00000 (size = 0x200000) >>> EAL: Ask a virtual area of 0x7c00000 bytes >>> EAL: Virtual area found at 0x7f3a6ec00000 (size = 0x7c00000) >>> EAL: Ask a virtual area of 0x200000 bytes >>> EAL: Virtual area found at 0x7f3a6e800000 (size = 0x200000) >>> EAL: Requesting 64 pages of size 2MB from socket 0 >>> EAL: TSC frequency is ~2200000 KHz >>> EAL: Master core 0 is ready (tid=76cfe880) >>> EAL: Core 3 is ready (tid=6cffc700) >>> EAL: Core 1 is ready (tid=6dffe700) >>> EAL: Core 2 is ready (tid=6d7fd700) >>> EAL: PCI device 0000:03:00.0 on NUMA socket -1 >>> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd >>> EAL: Device is blacklisted, not initializing >>> EAL: PCI device 0000:0b:00.0 on NUMA socket -1 >>> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd >>> EAL: PCI memory mapped at 0x7f3a76d10000 >>> EAL: PCI memory mapped at 0x7f3a76d0f000 >>> EAL: PCI memory mapped at 0x7f3a76d0d000 >>> EAL: PCI device 0000:13:00.0 on NUMA socket -1 >>> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd >>> EAL: PCI memory mapped at 0x7f3a76d0c000 >>> EAL: PCI memory mapped at 0x7f3a76d0b000 >>> EAL: PCI memory mapped at 0x7f3a76d09000 >>> Notice: odd number of ports in portmask. >>> Lcore 0: RX port 1 >>> Skipping disabled port 0 >>> Initializing port 1... done: >>> Port 1, MAC address: 00:0C:29:F1:9D:A4 >>> >>> >>> Checking link statusdone >>> Port 1 Link Up - speed 10000 Mbps - full-duplex >>> L2FWD: lcore 1 has nothing to do >>> L2FWD: lcore 2 has nothing to do >>> L2FWD: lcore 3 has nothing to do >>> L2FWD: entering main loop on lcore 0 >>> L2FWD: -- lcoreid=0 portid=1 >>> >>> Port statistics ==================================== >>> Statistics for port 1 ------------------------------ >>> Packets sent: 0 >>> Packets received: 0 >>> Packets dropped: 0 >>> Statistics for port 2 ------------------------------ >>> Packets sent: 0 >>> Packets received: 0 >>> Packets dropped: 0 >>> Aggregate statistics =============================== >>> Total packets sent: 0 >>> Total packets received: 0 >>> Total packets dropped: 0 >>> ==================================================== >>> >>> >>> >>> >>> On Fri, Mar 7, 2014 at 10:43 AM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: >>>> Hi Srini, >>>> >>>> Yes, I am still stuck, I am using ESXi5.1 though and Fedora18 in Guest. >>>> Please let me know what is the guest OS distro you are using in the VM. I'll try with that once. >>>> >>>> Regards >>>> -Prashant >>>> >>>> >>>> -----Original Message----- >>>> From: Srinivasan J [mailto:srinidpdk@gmail.com] >>>> Sent: Friday, March 07, 2014 9:08 AM >>>> To: Prashant Upadhyaya >>>> Cc: David Marchand; dev@dpdk.org >>>> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >>>> >>>> Prashant, >>>> Are you still hitting a crash? Do you have hugetlbfs >>>> enabled? if not try adding enabling hugetlbfs. I was able to get >>>> vxnet3 up using the vmxnet3 pmd alone in DPDK 1.60r1. I did not use >>>> the vmxnet3-usermap kernel driver. I am using Esxi 5.5 >>>> >>>> Eg: echo 1024 > /proc/sys/vm/nr_hugepages; mkdir /huge; mount -t >>>> hugetlbfs nodev /huge; >>>> >>>> On Thu, Mar 6, 2014 at 12:48 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote: >>>>> And if it is any help, here is the startup log -- >>>>> >>>>> EAL: Detected lcore 0 as core 0 on socket 0 >>>>> EAL: Detected lcore 1 as core 1 on socket 0 >>>>> EAL: Detected lcore 2 as core 2 on socket 0 >>>>> EAL: Detected lcore 3 as core 3 on socket 0 >>>>> EAL: Detected lcore 4 as core 4 on socket 0 >>>>> EAL: Detected lcore 5 as core 5 on socket 0 >>>>> EAL: Detected lcore 6 as core 6 on socket 0 >>>>> EAL: Detected lcore 7 as core 7 on socket 0 >>>>> EAL: Skip lcore 8 (not detected) >>>>> EAL: Skip lcore 9 (not detected) >>>>> EAL: Skip lcore 10 (not detected) >>>>> EAL: Skip lcore 11 (not detected) >>>>> EAL: Skip lcore 12 (not detected) >>>>> EAL: Skip lcore 13 (not detected) >>>>> EAL: Skip lcore 14 (not detected) >>>>> EAL: Skip lcore 15 (not detected) >>>>> EAL: Skip lcore 16 (not detected) >>>>> EAL: Skip lcore 17 (not detected) >>>>> EAL: Skip lcore 18 (not detected) >>>>> EAL: Skip lcore 19 (not detected) >>>>> EAL: Skip lcore 20 (not detected) >>>>> EAL: Skip lcore 21 (not detected) >>>>> EAL: Skip lcore 22 (not detected) >>>>> EAL: Skip lcore 23 (not detected) >>>>> EAL: Skip lcore 24 (not detected) >>>>> EAL: Skip lcore 25 (not detected) >>>>> EAL: Skip lcore 26 (not detected) >>>>> EAL: Skip lcore 27 (not detected) >>>>> EAL: Skip lcore 28 (not detected) >>>>> EAL: Skip lcore 29 (not detected) >>>>> EAL: Skip lcore 30 (not detected) >>>>> EAL: Skip lcore 31 (not detected) >>>>> EAL: Skip lcore 32 (not detected) >>>>> EAL: Skip lcore 33 (not detected) >>>>> EAL: Skip lcore 34 (not detected) >>>>> EAL: Skip lcore 35 (not detected) >>>>> EAL: Skip lcore 36 (not detected) >>>>> EAL: Skip lcore 37 (not detected) >>>>> EAL: Skip lcore 38 (not detected) >>>>> EAL: Skip lcore 39 (not detected) >>>>> EAL: Skip lcore 40 (not detected) >>>>> EAL: Skip lcore 41 (not detected) >>>>> EAL: Skip lcore 42 (not detected) >>>>> EAL: Skip lcore 43 (not detected) >>>>> EAL: Skip lcore 44 (not detected) >>>>> EAL: Skip lcore 45 (not detected) >>>>> EAL: Skip lcore 46 (not detected) >>>>> EAL: Skip lcore 47 (not detected) >>>>> EAL: Skip lcore 48 (not detected) >>>>> EAL: Skip lcore 49 (not detected) >>>>> EAL: Skip lcore 50 (not detected) >>>>> EAL: Skip lcore 51 (not detected) >>>>> EAL: Skip lcore 52 (not detected) >>>>> EAL: Skip lcore 53 (not detected) >>>>> EAL: Skip lcore 54 (not detected) >>>>> EAL: Skip lcore 55 (not detected) >>>>> EAL: Skip lcore 56 (not detected) >>>>> EAL: Skip lcore 57 (not detected) >>>>> EAL: Skip lcore 58 (not detected) >>>>> EAL: Skip lcore 59 (not detected) >>>>> EAL: Skip lcore 60 (not detected) >>>>> EAL: Skip lcore 61 (not detected) >>>>> EAL: Skip lcore 62 (not detected) >>>>> EAL: Skip lcore 63 (not detected) >>>>> EAL: Setting up memory... >>>>> EAL: Ask a virtual area of 0x80000000 bytes >>>>> EAL: Virtual area found at 0x7f848ae00000 (size = 0x80000000) >>>>> EAL: Requesting 1024 pages of size 2MB from socket 0 >>>>> EAL: TSC frequency is ~2000000 KHz >>>>> EAL: Master core 0 is ready (tid=b3f3f00) >>>>> EAL: Core 1 is ready (tid=8a1f2700) >>>>> EAL: Core 2 is ready (tid=899f1700) >>>>> EAL: Core 3 is ready (tid=891f0700) >>>>> EAL: Core 4 is ready (tid=889ef700) >>>>> EAL: Core 5 is ready (tid=7bfff700) >>>>> EAL: Core 6 is ready (tid=7b7fe700) >>>>> EAL: Core 7 is ready (tid=7affd700) >>>>> Pool initialized >>>>> Global Variables initialized >>>>> PMD: rte_vmxnet3_pmd_init(): >> >>>>> EAL: PCI device 0000:0b:00.0 on NUMA socket -1 >>>>> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd >>>>> PMD: eth_vmxnet3_dev_init(): >> >>>>> Segmentation fault (core dumped) >>>>> >>>>> -----Original Message----- >>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Prashant >>>>> Upadhyaya >>>>> Sent: Thursday, March 06, 2014 12:20 PM >>>>> To: David Marchand >>>>> Cc: dev@dpdk.org >>>>> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >>>>> >>>>> Hi, >>>>> >>>>> Some further update on the crash I am facing. >>>>> I am using DPDK1.6.0r1 and take over the vmxnet3 with igb_uio and >>>>> then start the application. (so no external ko or vmxnet usermap >>>>> etc.) During the port initializations, the crash is happening in the >>>>> following function - >>>>> >>>>> eth_vmxnet3_dev_init >>>>> >>>>> and the crash is happening at the following line - >>>>> /* Check h/w version compatibility with driver. */ >>>>> ver = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_VRRS); >>>>> >>>>> Any hints regarding what could be wrong ? >>>>> >>>>> Regards >>>>> -Prashant >>>>> >>>>> >>>>> From: Prashant Upadhyaya >>>>> Sent: Wednesday, March 05, 2014 9:01 PM >>>>> To: 'David Marchand' >>>>> Cc: Srinivasan J; dev@dpdk.org >>>>> Subject: RE: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >>>>> >>>>> Hi David, >>>>> >>>>> The compilation error with debug flags on was that the functions -- vmxnet3_txq_dump, vmxnet3_rxq_dump - are defined but not used. >>>>> Not a serious error, I will try to get rid of the compiler flag which generates this. >>>>> >>>>> However, I must reiterate, I _did_ bind my vmxnet3 device with igb_uio (and I did not use any .so, because I was intending to use the builtin vmxnet3 driver of dpdk 1.6.0r1), the bind succeeded, but then when I started the application, the dev init for vmxnet3 gave a core dump. >>>>> Your patch and solution seems to be suggesting the reverse, i.e. when I don't bind with igb_uio but try to use the native driver. So please do try the above combination as well. >>>>> >>>>> Regards >>>>> -Prashant >>>>> >>>>> >>>>> From: David Marchand [mailto:david.marchand@6wind.com] >>>>> Sent: Wednesday, March 05, 2014 8:41 PM >>>>> To: Prashant Upadhyaya >>>>> Cc: Srinivasan J; dev@dpdk.org<mailto:dev@dpdk.org> >>>>> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? >>>>> >>>>> Hello Prashant, >>>>> >>>>> On Wed, Mar 5, 2014 at 3:28 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com<mailto:prashant.upadhyaya@aricent.com>> wrote: >>>>> Hi, >>>>> >>>>> I am also keen to know the answer to the question posted by Srini. >>>>> The real question is -- is dpdk1.6.0r1 self-sufficient so that I don't need any extensions etc., or do I still need something from outside like the usermap kernel driver etc. >>>>> >>>>> Secondly, if I turn on all the debug options for the vmxnet3 pmd in the config file, 1.6.0r1 compilation runs into a problem and reports a function which is defined but not used. >>>>> >>>>> Can you send your build error ? (maybe in a separate thread ?) >>>>> >>>>> >>>>> I am trying to bring up DPDK inside Fedora18 Guest on ESXi -- when I used DPDK1.6.0r1 (without debug options turned on for vmxnet3 pmd) the igb_uio could take over the vmxnet3 NIC but I encountered a core dump in the dev init function for the vmxnet3 driver -- anybody encountered a similar issue ? >>>>> >>>>> I encountered these problems as well. >>>>> >>>>> - igb_uio module does not check if you disable vmxnet3-uio pmds, it will always try to take over vmxnet3 devices. >>>>> I have a patch waiting in my working dir to cleanly disable vmxnet3-uio pmd. >>>>> >>>>> - If you don't bind vmxnet3 devices to uio, but forget to enable vmxnet3-usermap pmd (by specifiying -d librte_pmd_vmxnet3.so), then internal vmxnet3-uio pmd will try to initialise and crash. >>>>> I did not look any deeper into this, the easiest way is to disable vmxnet3-uio pmd + apply the patch I will send in a few minutes, as a first workaround. >>>>> >>>>> >>>>> Regards, >>>>> -- >>>>> David Marchand >>>>> >>>>> >>>>> >>>>> >>>>> ===================================================================== >>>>> = >>>>> ========= Please refer to >>>>> http://www.aricent.com/legal/email_disclaimer.html >>>>> for important disclosures regarding this electronic communication. >>>>> ===================================================================== >>>>> = >>>>> ========= >>>>> >>>>> >>>>> >>>>> >>>>> ===================================================================== >>>>> = >>>>> ========= Please refer to >>>>> http://www.aricent.com/legal/email_disclaimer.html >>>>> for important disclosures regarding this electronic communication. >>>>> ===================================================================== >>>>> = >>>>> ========= >>>> >>>> >>>> >>>> >>>> ====================================================================== >>>> ========= Please refer to >>>> http://www.aricent.com/legal/email_disclaimer.html >>>> for important disclosures regarding this electronic communication. >>>> ====================================================================== >>>> ========= >>> >>> >>> >>> >>> =============================================================================== >>> Please refer to http://www.aricent.com/legal/email_disclaimer.html >>> for important disclosures regarding this electronic communication. >>> =============================================================================== >> > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [dpdk-dev] On vmxnet-pmd crash in DPDK 1.6.0r1 2014-03-11 6:29 ` Daniel Kan @ 2014-03-21 14:10 ` Thomas Monjalon 0 siblings, 0 replies; 21+ messages in thread From: Thomas Monjalon @ 2014-03-21 14:10 UTC (permalink / raw) To: Daniel Kan; +Cc: dev 10/03/2014 23:29, Daniel Kan : > virtio-pmd has the same pattern. I wonder if vmxnet3-pmd just blindly copied > the same paradigm. > > 473 static struct eth_driver rte_virtio_pmd = { > 474 { > 475 .name = "rte_virtio_pmd", > 476 .id_table = pci_id_virtio_map, > 477 #ifdef RTE_EAL_UNBIND_PORTS > 478 .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO, > 479 #endif [...] > > 266 static struct eth_driver rte_vmxnet3_pmd = { > > 267 { > > 268 .name = "rte_vmxnet3_pmd", > > 269 .id_table = pci_id_vmxnet3_map, > > 270 #ifdef RTE_EAL_UNBIND_PORTS > > 271 .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO, > > 272 #endif It was the old way of configuring UIO PMD before this patch: pci: use igb_uio mapping only when needed http://dpdk.org/browse/dpdk/commit/?id=10ed99419b12b8 The goal of this patch was to fix usage of the extensions virtio-net-pmd and vmxnet3-usermap. You can see in this patch that em/igb/ixgbe drivers are updated accordingly. Then virtio_uio and vmxnet3_uio were released without this patch in mind. So they were broken. > > lib/librte_eal/linuxapp/eal/eal_pci.c > > 1039 #ifdef RTE_EAL_UNBIND_PORTS > > 1040 if (dr->drv_flags & RTE_PCI_DRV_NEED_IGB_UIO) { > > 1041 /* unbind driver and load uio resources for Intel NICs */ > > 1042 if (pci_switch_module(dr, dev, 1, IGB_UIO_NAME) < 0) > > ... > > 1050 #else > > 1051 if (dr->drv_flags & RTE_PCI_DRV_NEED_IGB_UIO) > > 1052 /* just map resources for Intel NICs */ > > 1053 if (pci_uio_map_resource(dev) < 0) > > 1054 return -1; > > 1055 #endif > > > > If RTE_EAL_UNBIND_PORTS is defined, pci_switch_module will call > > pci_uio_map_resource. > > > > I then looked at the bsdapp’s version, it simply has: > > lib/librte_eal/bsdapp/eal/eal_pci.c > > 479 /* just map the NIC resources */ > > 480 if (pci_uio_map_resource(dev) < 0) > > 481 return -1; > > > > I don’t know the history behind why .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO > > only for RTE_EAL_UNBIND_PORTS. Can we just ensure pci_uio_map_resource is > > called in linux just like bsd version? One way is to ensure drv_flags is > > always set to RTE_PCI_DRV_NEED_IGB_UIO, but I don’t know if this fix will > > break other parts. I've just sent another patch to update and fix virtio_uio and vmxnet3_uio: http://dpdk.org/ml/archives/dev/2014-March/001647.html Feel free to ack it. Thanks for reporting -- Thomas ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? 2014-03-04 15:23 [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? Srinivasan J 2014-03-05 14:28 ` Prashant Upadhyaya @ 2014-03-05 14:55 ` David Marchand 1 sibling, 0 replies; 21+ messages in thread From: David Marchand @ 2014-03-05 14:55 UTC (permalink / raw) To: Srinivasan J; +Cc: dev Hello, I would say the main advantage is the fact that vmxnet3-usermap does not require uio. Regards, -- David Marchand On Tue, Mar 4, 2014 at 4:23 PM, Srinivasan J <srinidpdk@gmail.com> wrote: > I want to try dpdk with vmxnet3 in Vmware Esxi 5.1. I see that the > latest dpdk 1.6.0r1 includes a vmxnet3 pmd. The > vmxnet3-usermap-1.1.tar.gz as well includes a vmxnet3 pmd driver. > > I'm confused as to which vmxnet3 pmd driver to use along with which > vmxnet3 kernel driver (vmxnet3 native kernel driver or vmxnet3-usermap > kernel driver). I also want to try RSS with vmxnet3 and dpdk. As per > Intel DPDK Programmer's Guide (January 2014) RSS is supported with > vmxnet3 since dpdk version 1.6.0. > > Thanks, > Srini > ^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2014-05-06 5:48 UTC | newest] Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-03-04 15:23 [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? Srinivasan J 2014-03-05 14:28 ` Prashant Upadhyaya 2014-03-05 15:10 ` David Marchand 2014-03-05 15:30 ` Prashant Upadhyaya 2014-03-06 6:50 ` Prashant Upadhyaya 2014-03-06 7:18 ` Prashant Upadhyaya 2014-03-07 3:38 ` Srinivasan J 2014-03-07 5:13 ` Prashant Upadhyaya 2014-03-08 19:08 ` Srinivasan J 2014-03-10 8:00 ` Prashant Upadhyaya 2014-03-10 10:54 ` Prashant Upadhyaya 2014-03-10 15:50 ` Stephen Hemminger 2014-03-11 5:27 ` Prashant Upadhyaya 2014-03-11 8:52 ` Prashant Upadhyaya 2014-03-21 8:11 ` Prashant Upadhyaya 2014-05-06 5:48 ` Prashant Upadhyaya 2014-03-11 5:33 ` [dpdk-dev] On vmxnet-pmd crash in DPDK 1.6.0r1 Daniel Kan 2014-03-11 6:20 ` Daniel Kan 2014-03-11 6:29 ` Daniel Kan 2014-03-21 14:10 ` Thomas Monjalon 2014-03-05 14:55 ` [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x? David Marchand
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).