* Re: [dpdk-users] [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node [not found] <CAGxG5cgvahRNJcfPNqnwvezW2Viv8cQNnxZScVRqHfjZ-Ma-qQ@mail.gmail.com> @ 2018-02-09 11:27 ` Bruce Richardson 2018-02-09 11:53 ` Victor Huertas 0 siblings, 1 reply; 6+ messages in thread From: Bruce Richardson @ 2018-02-09 11:27 UTC (permalink / raw) To: Victor Huertas; +Cc: users Moving to users list, since that is more relevant than dev. On Thu, Feb 08, 2018 at 07:36:30PM +0100, Victor Huertas wrote: > Bruce, > > > My requirements are not that much (500 Mbps and 1 Gbps desirable). > > > Thanks for your references links I will have a look at them. > > > Regarding the NIC detection in the DPDK app by the DPDK EAL initialization > after successfully having loaded the vfio-pci, it happens something strange. > > The nb_ports = rte_eth_dev_count(); is always returning 0. > > > Therefore it shows an error telling that "port 0 is not present on the > board". > > > The EAL seems to detect the VFIOs as the only two logs that shows when > initializing regarding VFIO are: > > > EAL: Probing VFIO support... > > EAL: VFIO support initialized > > > And nothing else... shouldn't the EAL detect the two NICs I associated to > the VFIO? very strange... > > Yes, it should. Can you please send full output on startup. Also, can you try using igb_uio module instead of vfio and see if that works for you. Regards, /Bruce > Regards, > > > PD: I have changed the email address account in order to avoid sending > these disturbing disclaimers. Sorry. > > > -----Mensaje original----- > De: Bruce Richardson [mailto:bruce.richardson@intel.com] > Enviado el: jueves, 08 de febrero de 2018 17:42 > Para: Huertas García, Víctor > CC: dev@dpdk.org > Asunto: Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC > association to Numa node > > > > On Thu, Feb 08, 2018 at 04:27:36PM +0000, Huertas García, Víctor wrote: > > > > > > Hi all, > > > > > > After having tried many ways to make the PCIe NIC card appear associated > to a numa node, I haven't been able to do it. > > > That is, every time I try to look at which numa node belongs it always > returns -1. > > > > > > $ cat /sys/bus/pci/devices/0000\:04\:00.1/numa_node > > > -1 > > > > > > $ cat /sys/bus/pci/devices/0000\:04\:00.0/numa_node > > > -1 > > > > > > Using lstopo, I confirm that all PCI cards are "outside" of any Numa > node. > > > > > > I have read in previous posted messages in dpdk-dev community that this > is normal in Nehalem generation Xeon architecture and there is nothing I > can do about it. Can somebody confirm this? > > > > For that generation architecture, it is indeed expected. The NICs are not > directly connected to any NUMA node. > > > > > If so, what implications could this have on packet capture and > performance? > > > > Unsurprisingly, it's the case that newer platforms will perform better, as > you are missing out on performance benefits from improved cores and also > features like Intel® DDIO [1]. > > However, what I/O throughput are you hoping to get from your system? > > Depending on your requirements, what you have may be enough. Some people > use DPDK on lower-end platforms because that is all that they need. You may > also find the chart on slide 6 of [2] of use to see how the max throughput > of a platform has improved over time (and has improved further since that > chart was published). > > > > > > > > Are the NICs available in my DPDK applications? Do I have to specifically > "add" them by "-w 04:00.1 - w 04:00.0"? > > > > Yes, your NICs will still be available, even without NUMA affinity, and no, > you should not need to explicitly whitelist them - though you can if you > want. So long as they are bound to a uio or vfio driver (e.g. > > igb_uio or vfio-pci), they should be detected by DPDK EAL init and made > available to your app. > > > > > Is RSS supported and usable from the DPDK application? > > > > Yes, at least for Intel NICs, and probably most other DPDK-supported NICs > too. > > > > > > > > Thanks a lot for your attention > > > > > > Victor > > > > > > > /Bruce > > > > [1] https://www.intel.com/content/www/us/en/io/data-direct-i-o-t > echnology.html > > [2] https://dpdksummit.com/Archive/pdf/2016Germany/DPDK-2016- > DPDK_FD_IO_Introduction.pdf > > > > PS: This is a public list, so email disclaimers are rather pointless. > > It's best if they can be removed from mails sent here. > > > > > -- > Victor ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-users] [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node 2018-02-09 11:27 ` [dpdk-users] [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node Bruce Richardson @ 2018-02-09 11:53 ` Victor Huertas 2018-02-09 11:58 ` Richardson, Bruce 0 siblings, 1 reply; 6+ messages in thread From: Victor Huertas @ 2018-02-09 11:53 UTC (permalink / raw) To: Bruce Richardson; +Cc: users Bruce, With igb_uio it happens the same: Network devices using DPDK-compatible driver ============================================ 0000:04:00.0 '82576 Gigabit Network Connection 10c9' drv=igb_uio unused=vfio-pci 0000:04:00.1 '82576 Gigabit Network Connection 10c9' drv=igb_uio unused=vfio-pci Network devices using kernel driver =================================== 0000:02:00.0 '82572EI Gigabit Ethernet Controller (Copper) 107d' if=eth1 drv=e1000e unused=igb_uio,vfio-pci *Active* Even I have linked the rte libs with the flag --whole-archive flag (I found somewhere in google that this solved a similar problem) but with no result. All the output I get when I run the app (it is the l3fwd-acl example) is what you see below: cuda1@cuda1:~/git/dpdk-stable> sudo /home/cuda1/eclipse-workspace/DPDK_Test/Debug/DPDK_Test -c 0x0f -n 2 -- -p 0x3 EAL: Detected 8 lcore(s) EAL: Probing VFIO support... EAL: VFIO support initialized port 0 is not present on the board EAL: Error - exiting with code: 1 Cause: check_port_config failed 2018-02-09 12:27 GMT+01:00 Bruce Richardson <bruce.richardson@intel.com>: > Moving to users list, since that is more relevant than dev. > > On Thu, Feb 08, 2018 at 07:36:30PM +0100, Victor Huertas wrote: > > Bruce, > > > > > > My requirements are not that much (500 Mbps and 1 Gbps desirable). > > > > > > Thanks for your references links I will have a look at them. > > > > > > Regarding the NIC detection in the DPDK app by the DPDK EAL > initialization > > after successfully having loaded the vfio-pci, it happens something > strange. > > > > The nb_ports = rte_eth_dev_count(); is always returning 0. > > > > > > Therefore it shows an error telling that "port 0 is not present on the > > board". > > > > > > The EAL seems to detect the VFIOs as the only two logs that shows when > > initializing regarding VFIO are: > > > > > > EAL: Probing VFIO support... > > > > EAL: VFIO support initialized > > > > > > And nothing else... shouldn't the EAL detect the two NICs I associated to > > the VFIO? very strange... > > > > > > Yes, it should. Can you please send full output on startup. > Also, can you try using igb_uio module instead of vfio and see if that > works for you. > > Regards, > /Bruce > > > Regards, > > > > > > PD: I have changed the email address account in order to avoid sending > > these disturbing disclaimers. Sorry. > > > > > > -----Mensaje original----- > > De: Bruce Richardson [mailto:bruce.richardson@intel.com] > > Enviado el: jueves, 08 de febrero de 2018 17:42 > > Para: Huertas García, Víctor > > CC: dev@dpdk.org > > Asunto: Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC > > association to Numa node > > > > > > > > On Thu, Feb 08, 2018 at 04:27:36PM +0000, Huertas García, Víctor wrote: > > > > > > > > > > Hi all, > > > > > > > > > > After having tried many ways to make the PCIe NIC card appear > associated > > to a numa node, I haven't been able to do it. > > > > > That is, every time I try to look at which numa node belongs it always > > returns -1. > > > > > > > > > > $ cat /sys/bus/pci/devices/0000\:04\:00.1/numa_node > > > > > -1 > > > > > > > > > > $ cat /sys/bus/pci/devices/0000\:04\:00.0/numa_node > > > > > -1 > > > > > > > > > > Using lstopo, I confirm that all PCI cards are "outside" of any Numa > > node. > > > > > > > > > > I have read in previous posted messages in dpdk-dev community that this > > is normal in Nehalem generation Xeon architecture and there is nothing I > > can do about it. Can somebody confirm this? > > > > > > > > For that generation architecture, it is indeed expected. The NICs are not > > directly connected to any NUMA node. > > > > > > > > > If so, what implications could this have on packet capture and > > performance? > > > > > > > > Unsurprisingly, it's the case that newer platforms will perform better, > as > > you are missing out on performance benefits from improved cores and also > > features like Intel® DDIO [1]. > > > > However, what I/O throughput are you hoping to get from your system? > > > > Depending on your requirements, what you have may be enough. Some people > > use DPDK on lower-end platforms because that is all that they need. You > may > > also find the chart on slide 6 of [2] of use to see how the max > throughput > > of a platform has improved over time (and has improved further since that > > chart was published). > > > > > > > > > > > > > > Are the NICs available in my DPDK applications? Do I have to > specifically > > "add" them by "-w 04:00.1 - w 04:00.0"? > > > > > > > > Yes, your NICs will still be available, even without NUMA affinity, and > no, > > you should not need to explicitly whitelist them - though you can if you > > want. So long as they are bound to a uio or vfio driver (e.g. > > > > igb_uio or vfio-pci), they should be detected by DPDK EAL init and made > > available to your app. > > > > > > > > > Is RSS supported and usable from the DPDK application? > > > > > > > > Yes, at least for Intel NICs, and probably most other DPDK-supported NICs > > too. > > > > > > > > > > > > > > Thanks a lot for your attention > > > > > > > > > > Victor > > > > > > > > > > > > > /Bruce > > > > > > > > [1] https://www.intel.com/content/www/us/en/io/data-direct-i-o-t > > echnology.html > > > > [2] https://dpdksummit.com/Archive/pdf/2016Germany/DPDK-2016- > > DPDK_FD_IO_Introduction.pdf > > > > > > > > PS: This is a public list, so email disclaimers are rather pointless. > > > > It's best if they can be removed from mails sent here. > > > > > > > > > > -- > > Victor > -- Victor ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-users] [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node 2018-02-09 11:53 ` Victor Huertas @ 2018-02-09 11:58 ` Richardson, Bruce 2018-02-09 12:01 ` Victor Huertas 0 siblings, 1 reply; 6+ messages in thread From: Richardson, Bruce @ 2018-02-09 11:58 UTC (permalink / raw) To: Victor Huertas; +Cc: users Does it work with testpmd, built as part of DPDK? When you build the l3fwd-acl example do you use the DPDK build system? From: Victor Huertas [mailto:vhuertas@gmail.com] Sent: Friday, February 9, 2018 11:54 AM To: Richardson, Bruce <bruce.richardson@intel.com> Cc: users@dpdk.org Subject: Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node Bruce, With igb_uio it happens the same: Network devices using DPDK-compatible driver ============================================ 0000:04:00.0 '82576 Gigabit Network Connection 10c9' drv=igb_uio unused=vfio-pci 0000:04:00.1 '82576 Gigabit Network Connection 10c9' drv=igb_uio unused=vfio-pci Network devices using kernel driver =================================== 0000:02:00.0 '82572EI Gigabit Ethernet Controller (Copper) 107d' if=eth1 drv=e1000e unused=igb_uio,vfio-pci *Active* Even I have linked the rte libs with the flag --whole-archive flag (I found somewhere in google that this solved a similar problem) but with no result. All the output I get when I run the app (it is the l3fwd-acl example) is what you see below: cuda1@cuda1:~/git/dpdk-stable> sudo /home/cuda1/eclipse-workspace/DPDK_Test/Debug/DPDK_Test -c 0x0f -n 2 -- -p 0x3 EAL: Detected 8 lcore(s) EAL: Probing VFIO support... EAL: VFIO support initialized port 0 is not present on the board EAL: Error - exiting with code: 1 Cause: check_port_config failed 2018-02-09 12:27 GMT+01:00 Bruce Richardson <bruce.richardson@intel.com<mailto:bruce.richardson@intel.com>>: Moving to users list, since that is more relevant than dev. On Thu, Feb 08, 2018 at 07:36:30PM +0100, Victor Huertas wrote: > Bruce, > > > My requirements are not that much (500 Mbps and 1 Gbps desirable). > > > Thanks for your references links I will have a look at them. > > > Regarding the NIC detection in the DPDK app by the DPDK EAL initialization > after successfully having loaded the vfio-pci, it happens something strange. > > The nb_ports = rte_eth_dev_count(); is always returning 0. > > > Therefore it shows an error telling that "port 0 is not present on the > board". > > > The EAL seems to detect the VFIOs as the only two logs that shows when > initializing regarding VFIO are: > > > EAL: Probing VFIO support... > > EAL: VFIO support initialized > > > And nothing else... shouldn't the EAL detect the two NICs I associated to > the VFIO? very strange... > > Yes, it should. Can you please send full output on startup. Also, can you try using igb_uio module instead of vfio and see if that works for you. Regards, /Bruce > Regards, > > > PD: I have changed the email address account in order to avoid sending > these disturbing disclaimers. Sorry. > > > -----Mensaje original----- > De: Bruce Richardson [mailto:bruce.richardson@intel.com<mailto:bruce.richardson@intel.com>] > Enviado el: jueves, 08 de febrero de 2018 17:42 > Para: Huertas García, Víctor > CC: dev@dpdk.org<mailto:dev@dpdk.org> > Asunto: Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC > association to Numa node > > > > On Thu, Feb 08, 2018 at 04:27:36PM +0000, Huertas García, Víctor wrote: > > > > > > Hi all, > > > > > > After having tried many ways to make the PCIe NIC card appear associated > to a numa node, I haven't been able to do it. > > > That is, every time I try to look at which numa node belongs it always > returns -1. > > > > > > $ cat /sys/bus/pci/devices/0000\:04\:00.1/numa_node > > > -1 > > > > > > $ cat /sys/bus/pci/devices/0000\:04\:00.0/numa_node > > > -1 > > > > > > Using lstopo, I confirm that all PCI cards are "outside" of any Numa > node. > > > > > > I have read in previous posted messages in dpdk-dev community that this > is normal in Nehalem generation Xeon architecture and there is nothing I > can do about it. Can somebody confirm this? > > > > For that generation architecture, it is indeed expected. The NICs are not > directly connected to any NUMA node. > > > > > If so, what implications could this have on packet capture and > performance? > > > > Unsurprisingly, it's the case that newer platforms will perform better, as > you are missing out on performance benefits from improved cores and also > features like Intel® DDIO [1]. > > However, what I/O throughput are you hoping to get from your system? > > Depending on your requirements, what you have may be enough. Some people > use DPDK on lower-end platforms because that is all that they need. You may > also find the chart on slide 6 of [2] of use to see how the max throughput > of a platform has improved over time (and has improved further since that > chart was published). > > > > > > > > Are the NICs available in my DPDK applications? Do I have to specifically > "add" them by "-w 04:00.1 - w 04:00.0"? > > > > Yes, your NICs will still be available, even without NUMA affinity, and no, > you should not need to explicitly whitelist them - though you can if you > want. So long as they are bound to a uio or vfio driver (e.g. > > igb_uio or vfio-pci), they should be detected by DPDK EAL init and made > available to your app. > > > > > Is RSS supported and usable from the DPDK application? > > > > Yes, at least for Intel NICs, and probably most other DPDK-supported NICs > too. > > > > > > > > Thanks a lot for your attention > > > > > > Victor > > > > > > > /Bruce > > > > [1] https://www.intel.com/content/www/us/en/io/data-direct-i-o-t > echnology.html > > [2] https://dpdksummit.com/Archive/pdf/2016Germany/DPDK-2016- > DPDK_FD_IO_Introduction.pdf > > > > PS: This is a public list, so email disclaimers are rather pointless. > > It's best if they can be removed from mails sent here. > > > > > -- > Victor -- Victor ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-users] [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node 2018-02-09 11:58 ` Richardson, Bruce @ 2018-02-09 12:01 ` Victor Huertas 2018-02-09 12:03 ` Richardson, Bruce 0 siblings, 1 reply; 6+ messages in thread From: Victor Huertas @ 2018-02-09 12:01 UTC (permalink / raw) To: Richardson, Bruce; +Cc: users Nope, I use Eclipse CDT environment to compile (I add all the static libs manually and the executable is successfully generated). Regards, 2018-02-09 12:58 GMT+01:00 Richardson, Bruce <bruce.richardson@intel.com>: > Does it work with testpmd, built as part of DPDK? > > When you build the l3fwd-acl example do you use the DPDK build system? > > > > *From:* Victor Huertas [mailto:vhuertas@gmail.com] > *Sent:* Friday, February 9, 2018 11:54 AM > *To:* Richardson, Bruce <bruce.richardson@intel.com> > *Cc:* users@dpdk.org > *Subject:* Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe > NIC association to Numa node > > > > Bruce, > > With igb_uio it happens the same: > > Network devices using DPDK-compatible driver > ============================================ > 0000:04:00.0 '82576 Gigabit Network Connection 10c9' drv=igb_uio > unused=vfio-pci > 0000:04:00.1 '82576 Gigabit Network Connection 10c9' drv=igb_uio > unused=vfio-pci > > Network devices using kernel driver > =================================== > 0000:02:00.0 '82572EI Gigabit Ethernet Controller (Copper) 107d' if=eth1 > drv=e1000e unused=igb_uio,vfio-pci *Active* > > Even I have linked the rte libs with the flag --whole-archive flag (I > found somewhere in google that this solved a similar problem) but with no > result. > > All the output I get when I run the app (it is the l3fwd-acl example) is > what you see below: > > cuda1@cuda1:~/git/dpdk-stable> sudo /home/cuda1/eclipse-workspace/DPDK_Test/Debug/DPDK_Test > -c 0x0f -n 2 -- -p 0x3 > EAL: Detected 8 lcore(s) > EAL: Probing VFIO support... > EAL: VFIO support initialized > port 0 is not present on the board > EAL: Error - exiting with code: 1 > Cause: check_port_config failed > > > > 2018-02-09 12:27 GMT+01:00 Bruce Richardson <bruce.richardson@intel.com>: > > Moving to users list, since that is more relevant than dev. > > On Thu, Feb 08, 2018 at 07:36:30PM +0100, Victor Huertas wrote: > > Bruce, > > > > > > My requirements are not that much (500 Mbps and 1 Gbps desirable). > > > > > > Thanks for your references links I will have a look at them. > > > > > > Regarding the NIC detection in the DPDK app by the DPDK EAL > initialization > > after successfully having loaded the vfio-pci, it happens something > strange. > > > > The nb_ports = rte_eth_dev_count(); is always returning 0. > > > > > > Therefore it shows an error telling that "port 0 is not present on the > > board". > > > > > > The EAL seems to detect the VFIOs as the only two logs that shows when > > initializing regarding VFIO are: > > > > > > EAL: Probing VFIO support... > > > > EAL: VFIO support initialized > > > > > > And nothing else... shouldn't the EAL detect the two NICs I associated to > > the VFIO? very strange... > > > > > > Yes, it should. Can you please send full output on startup. > Also, can you try using igb_uio module instead of vfio and see if that > works for you. > > Regards, > /Bruce > > > > Regards, > > > > > > PD: I have changed the email address account in order to avoid sending > > these disturbing disclaimers. Sorry. > > > > > > -----Mensaje original----- > > De: Bruce Richardson [mailto:bruce.richardson@intel.com] > > Enviado el: jueves, 08 de febrero de 2018 17:42 > > Para: Huertas García, Víctor > > CC: dev@dpdk.org > > Asunto: Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC > > association to Numa node > > > > > > > > On Thu, Feb 08, 2018 at 04:27:36PM +0000, Huertas García, Víctor wrote: > > > > > > > > > > Hi all, > > > > > > > > > > After having tried many ways to make the PCIe NIC card appear > associated > > to a numa node, I haven't been able to do it. > > > > > That is, every time I try to look at which numa node belongs it always > > returns -1. > > > > > > > > > > $ cat /sys/bus/pci/devices/0000\:04\:00.1/numa_node > > > > > -1 > > > > > > > > > > $ cat /sys/bus/pci/devices/0000\:04\:00.0/numa_node > > > > > -1 > > > > > > > > > > Using lstopo, I confirm that all PCI cards are "outside" of any Numa > > node. > > > > > > > > > > I have read in previous posted messages in dpdk-dev community that this > > is normal in Nehalem generation Xeon architecture and there is nothing I > > can do about it. Can somebody confirm this? > > > > > > > > For that generation architecture, it is indeed expected. The NICs are not > > directly connected to any NUMA node. > > > > > > > > > If so, what implications could this have on packet capture and > > performance? > > > > > > > > Unsurprisingly, it's the case that newer platforms will perform better, > as > > you are missing out on performance benefits from improved cores and also > > features like Intel® DDIO [1]. > > > > However, what I/O throughput are you hoping to get from your system? > > > > Depending on your requirements, what you have may be enough. Some people > > use DPDK on lower-end platforms because that is all that they need. You > may > > also find the chart on slide 6 of [2] of use to see how the max > throughput > > of a platform has improved over time (and has improved further since that > > chart was published). > > > > > > > > > > > > > > Are the NICs available in my DPDK applications? Do I have to > specifically > > "add" them by "-w 04:00.1 - w 04:00.0"? > > > > > > > > Yes, your NICs will still be available, even without NUMA affinity, and > no, > > you should not need to explicitly whitelist them - though you can if you > > want. So long as they are bound to a uio or vfio driver (e.g. > > > > igb_uio or vfio-pci), they should be detected by DPDK EAL init and made > > available to your app. > > > > > > > > > Is RSS supported and usable from the DPDK application? > > > > > > > > Yes, at least for Intel NICs, and probably most other DPDK-supported NICs > > too. > > > > > > > > > > > > > > Thanks a lot for your attention > > > > > > > > > > Victor > > > > > > > > > > > > > /Bruce > > > > > > > > [1] https://www.intel.com/content/www/us/en/io/data-direct-i-o-t > > echnology.html > > > > [2] https://dpdksummit.com/Archive/pdf/2016Germany/DPDK-2016- > > DPDK_FD_IO_Introduction.pdf > > > > > > > > PS: This is a public list, so email disclaimers are rather pointless. > > > > It's best if they can be removed from mails sent here. > > > > > > > > > > -- > > Victor > > > > > -- > > Victor > -- Victor ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-users] [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node 2018-02-09 12:01 ` Victor Huertas @ 2018-02-09 12:03 ` Richardson, Bruce 2018-02-09 12:17 ` Victor Huertas 0 siblings, 1 reply; 6+ messages in thread From: Richardson, Bruce @ 2018-02-09 12:03 UTC (permalink / raw) To: Victor Huertas; +Cc: users You are probably missing another linker flag so - at a guess "--whole-archive". Please test with the testpmd app built using DPDK itself. /Bruce From: Victor Huertas [mailto:vhuertas@gmail.com] Sent: Friday, February 9, 2018 12:01 PM To: Richardson, Bruce <bruce.richardson@intel.com> Cc: users@dpdk.org Subject: Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node Nope, I use Eclipse CDT environment to compile (I add all the static libs manually and the executable is successfully generated). Regards, 2018-02-09 12:58 GMT+01:00 Richardson, Bruce <bruce.richardson@intel.com<mailto:bruce.richardson@intel.com>>: Does it work with testpmd, built as part of DPDK? When you build the l3fwd-acl example do you use the DPDK build system? From: Victor Huertas [mailto:vhuertas@gmail.com<mailto:vhuertas@gmail.com>] Sent: Friday, February 9, 2018 11:54 AM To: Richardson, Bruce <bruce.richardson@intel.com<mailto:bruce.richardson@intel.com>> Cc: users@dpdk.org<mailto:users@dpdk.org> Subject: Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node Bruce, With igb_uio it happens the same: Network devices using DPDK-compatible driver ============================================ 0000:04:00.0 '82576 Gigabit Network Connection 10c9' drv=igb_uio unused=vfio-pci 0000:04:00.1 '82576 Gigabit Network Connection 10c9' drv=igb_uio unused=vfio-pci Network devices using kernel driver =================================== 0000:02:00.0 '82572EI Gigabit Ethernet Controller (Copper) 107d' if=eth1 drv=e1000e unused=igb_uio,vfio-pci *Active* Even I have linked the rte libs with the flag --whole-archive flag (I found somewhere in google that this solved a similar problem) but with no result. All the output I get when I run the app (it is the l3fwd-acl example) is what you see below: cuda1@cuda1:~/git/dpdk-stable> sudo /home/cuda1/eclipse-workspace/DPDK_Test/Debug/DPDK_Test -c 0x0f -n 2 -- -p 0x3 EAL: Detected 8 lcore(s) EAL: Probing VFIO support... EAL: VFIO support initialized port 0 is not present on the board EAL: Error - exiting with code: 1 Cause: check_port_config failed 2018-02-09 12:27 GMT+01:00 Bruce Richardson <bruce.richardson@intel.com<mailto:bruce.richardson@intel.com>>: Moving to users list, since that is more relevant than dev. On Thu, Feb 08, 2018 at 07:36:30PM +0100, Victor Huertas wrote: > Bruce, > > > My requirements are not that much (500 Mbps and 1 Gbps desirable). > > > Thanks for your references links I will have a look at them. > > > Regarding the NIC detection in the DPDK app by the DPDK EAL initialization > after successfully having loaded the vfio-pci, it happens something strange. > > The nb_ports = rte_eth_dev_count(); is always returning 0. > > > Therefore it shows an error telling that "port 0 is not present on the > board". > > > The EAL seems to detect the VFIOs as the only two logs that shows when > initializing regarding VFIO are: > > > EAL: Probing VFIO support... > > EAL: VFIO support initialized > > > And nothing else... shouldn't the EAL detect the two NICs I associated to > the VFIO? very strange... > > Yes, it should. Can you please send full output on startup. Also, can you try using igb_uio module instead of vfio and see if that works for you. Regards, /Bruce > Regards, > > > PD: I have changed the email address account in order to avoid sending > these disturbing disclaimers. Sorry. > > > -----Mensaje original----- > De: Bruce Richardson [mailto:bruce.richardson@intel.com<mailto:bruce.richardson@intel.com>] > Enviado el: jueves, 08 de febrero de 2018 17:42 > Para: Huertas García, Víctor > CC: dev@dpdk.org<mailto:dev@dpdk.org> > Asunto: Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC > association to Numa node > > > > On Thu, Feb 08, 2018 at 04:27:36PM +0000, Huertas García, Víctor wrote: > > > > > > Hi all, > > > > > > After having tried many ways to make the PCIe NIC card appear associated > to a numa node, I haven't been able to do it. > > > That is, every time I try to look at which numa node belongs it always > returns -1. > > > > > > $ cat /sys/bus/pci/devices/0000\:04\:00.1/numa_node > > > -1 > > > > > > $ cat /sys/bus/pci/devices/0000\:04\:00.0/numa_node > > > -1 > > > > > > Using lstopo, I confirm that all PCI cards are "outside" of any Numa > node. > > > > > > I have read in previous posted messages in dpdk-dev community that this > is normal in Nehalem generation Xeon architecture and there is nothing I > can do about it. Can somebody confirm this? > > > > For that generation architecture, it is indeed expected. The NICs are not > directly connected to any NUMA node. > > > > > If so, what implications could this have on packet capture and > performance? > > > > Unsurprisingly, it's the case that newer platforms will perform better, as > you are missing out on performance benefits from improved cores and also > features like Intel® DDIO [1]. > > However, what I/O throughput are you hoping to get from your system? > > Depending on your requirements, what you have may be enough. Some people > use DPDK on lower-end platforms because that is all that they need. You may > also find the chart on slide 6 of [2] of use to see how the max throughput > of a platform has improved over time (and has improved further since that > chart was published). > > > > > > > > Are the NICs available in my DPDK applications? Do I have to specifically > "add" them by "-w 04:00.1 - w 04:00.0"? > > > > Yes, your NICs will still be available, even without NUMA affinity, and no, > you should not need to explicitly whitelist them - though you can if you > want. So long as they are bound to a uio or vfio driver (e.g. > > igb_uio or vfio-pci), they should be detected by DPDK EAL init and made > available to your app. > > > > > Is RSS supported and usable from the DPDK application? > > > > Yes, at least for Intel NICs, and probably most other DPDK-supported NICs > too. > > > > > > > > Thanks a lot for your attention > > > > > > Victor > > > > > > > /Bruce > > > > [1] https://www.intel.com/content/www/us/en/io/data-direct-i-o-t > echnology.html > > [2] https://dpdksummit.com/Archive/pdf/2016Germany/DPDK-2016- > DPDK_FD_IO_Introduction.pdf > > > > PS: This is a public list, so email disclaimers are rather pointless. > > It's best if they can be removed from mails sent here. > > > > > -- > Victor -- Victor -- Victor ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-users] [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node 2018-02-09 12:03 ` Richardson, Bruce @ 2018-02-09 12:17 ` Victor Huertas 0 siblings, 0 replies; 6+ messages in thread From: Victor Huertas @ 2018-02-09 12:17 UTC (permalink / raw) To: Richardson, Bruce; +Cc: users As you can see in the linker command --whole-archive was already used so link all dpdk libs: g++ -L/home/cuda1/git/dpdk-stable/x86_64-native-linuxapp-gcc/build/lib/librte_acl -L/home/cuda1/git/dpdk-stable/x86_64-native-linuxapp-gcc/build/lib/librte_mbuf -L/home/cuda1/git/dpdk-stable/x86_64-native-linuxapp-gcc/build/lib/librte_ether -L/home/cuda1/git/dpdk-stable/x86_64-native-linuxapp-gcc/build/lib/librte_mempool -L/home/cuda1/git/dpdk-stable/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/eal -o "DPDK_Test" -Wl,--whole-archive -lrte_ethdev -lrte_mbuf -lrte_mempool -lrte_acl -lrte_eal -Wl,--no-whole-archive ./src/DPDK_Test.o -lpthread -lnuma -ldl -Wl,--as-needed I will try to build it using DPDK. Perhaps something is being missed in my Eclipse configuration Regards, 2018-02-09 13:03 GMT+01:00 Richardson, Bruce <bruce.richardson@intel.com>: > You are probably missing another linker flag so - at a guess > "--whole-archive". > > > > Please test with the testpmd app built using DPDK itself. > > > > /Bruce > > > > *From:* Victor Huertas [mailto:vhuertas@gmail.com] > *Sent:* Friday, February 9, 2018 12:01 PM > > *To:* Richardson, Bruce <bruce.richardson@intel.com> > *Cc:* users@dpdk.org > *Subject:* Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe > NIC association to Numa node > > > > Nope, I use Eclipse CDT environment to compile (I add all the static libs > manually and the executable is successfully generated). > > Regards, > > > > 2018-02-09 12:58 GMT+01:00 Richardson, Bruce <bruce.richardson@intel.com>: > > Does it work with testpmd, built as part of DPDK? > > When you build the l3fwd-acl example do you use the DPDK build system? > > > > *From:* Victor Huertas [mailto:vhuertas@gmail.com] > *Sent:* Friday, February 9, 2018 11:54 AM > *To:* Richardson, Bruce <bruce.richardson@intel.com> > *Cc:* users@dpdk.org > *Subject:* Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe > NIC association to Numa node > > > > Bruce, > > With igb_uio it happens the same: > > Network devices using DPDK-compatible driver > ============================================ > 0000:04:00.0 '82576 Gigabit Network Connection 10c9' drv=igb_uio > unused=vfio-pci > 0000:04:00.1 '82576 Gigabit Network Connection 10c9' drv=igb_uio > unused=vfio-pci > > Network devices using kernel driver > =================================== > 0000:02:00.0 '82572EI Gigabit Ethernet Controller (Copper) 107d' if=eth1 > drv=e1000e unused=igb_uio,vfio-pci *Active* > > Even I have linked the rte libs with the flag --whole-archive flag (I > found somewhere in google that this solved a similar problem) but with no > result. > > All the output I get when I run the app (it is the l3fwd-acl example) is > what you see below: > > cuda1@cuda1:~/git/dpdk-stable> sudo /home/cuda1/eclipse-workspace/DPDK_Test/Debug/DPDK_Test > -c 0x0f -n 2 -- -p 0x3 > EAL: Detected 8 lcore(s) > EAL: Probing VFIO support... > EAL: VFIO support initialized > port 0 is not present on the board > EAL: Error - exiting with code: 1 > Cause: check_port_config failed > > > > 2018-02-09 12:27 GMT+01:00 Bruce Richardson <bruce.richardson@intel.com>: > > Moving to users list, since that is more relevant than dev. > > On Thu, Feb 08, 2018 at 07:36:30PM +0100, Victor Huertas wrote: > > Bruce, > > > > > > My requirements are not that much (500 Mbps and 1 Gbps desirable). > > > > > > Thanks for your references links I will have a look at them. > > > > > > Regarding the NIC detection in the DPDK app by the DPDK EAL > initialization > > after successfully having loaded the vfio-pci, it happens something > strange. > > > > The nb_ports = rte_eth_dev_count(); is always returning 0. > > > > > > Therefore it shows an error telling that "port 0 is not present on the > > board". > > > > > > The EAL seems to detect the VFIOs as the only two logs that shows when > > initializing regarding VFIO are: > > > > > > EAL: Probing VFIO support... > > > > EAL: VFIO support initialized > > > > > > And nothing else... shouldn't the EAL detect the two NICs I associated to > > the VFIO? very strange... > > > > > > Yes, it should. Can you please send full output on startup. > Also, can you try using igb_uio module instead of vfio and see if that > works for you. > > Regards, > /Bruce > > > > Regards, > > > > > > PD: I have changed the email address account in order to avoid sending > > these disturbing disclaimers. Sorry. > > > > > > -----Mensaje original----- > > De: Bruce Richardson [mailto:bruce.richardson@intel.com] > > Enviado el: jueves, 08 de febrero de 2018 17:42 > > Para: Huertas García, Víctor > > CC: dev@dpdk.org > > Asunto: Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC > > association to Numa node > > > > > > > > On Thu, Feb 08, 2018 at 04:27:36PM +0000, Huertas García, Víctor wrote: > > > > > > > > > > Hi all, > > > > > > > > > > After having tried many ways to make the PCIe NIC card appear > associated > > to a numa node, I haven't been able to do it. > > > > > That is, every time I try to look at which numa node belongs it always > > returns -1. > > > > > > > > > > $ cat /sys/bus/pci/devices/0000\:04\:00.1/numa_node > > > > > -1 > > > > > > > > > > $ cat /sys/bus/pci/devices/0000\:04\:00.0/numa_node > > > > > -1 > > > > > > > > > > Using lstopo, I confirm that all PCI cards are "outside" of any Numa > > node. > > > > > > > > > > I have read in previous posted messages in dpdk-dev community that this > > is normal in Nehalem generation Xeon architecture and there is nothing I > > can do about it. Can somebody confirm this? > > > > > > > > For that generation architecture, it is indeed expected. The NICs are not > > directly connected to any NUMA node. > > > > > > > > > If so, what implications could this have on packet capture and > > performance? > > > > > > > > Unsurprisingly, it's the case that newer platforms will perform better, > as > > you are missing out on performance benefits from improved cores and also > > features like Intel® DDIO [1]. > > > > However, what I/O throughput are you hoping to get from your system? > > > > Depending on your requirements, what you have may be enough. Some people > > use DPDK on lower-end platforms because that is all that they need. You > may > > also find the chart on slide 6 of [2] of use to see how the max > throughput > > of a platform has improved over time (and has improved further since that > > chart was published). > > > > > > > > > > > > > > Are the NICs available in my DPDK applications? Do I have to > specifically > > "add" them by "-w 04:00.1 - w 04:00.0"? > > > > > > > > Yes, your NICs will still be available, even without NUMA affinity, and > no, > > you should not need to explicitly whitelist them - though you can if you > > want. So long as they are bound to a uio or vfio driver (e.g. > > > > igb_uio or vfio-pci), they should be detected by DPDK EAL init and made > > available to your app. > > > > > > > > > Is RSS supported and usable from the DPDK application? > > > > > > > > Yes, at least for Intel NICs, and probably most other DPDK-supported NICs > > too. > > > > > > > > > > > > > > Thanks a lot for your attention > > > > > > > > > > Victor > > > > > > > > > > > > > /Bruce > > > > > > > > [1] https://www.intel.com/content/www/us/en/io/data-direct-i-o-t > > echnology.html > > > > [2] https://dpdksummit.com/Archive/pdf/2016Germany/DPDK-2016- > > DPDK_FD_IO_Introduction.pdf > > > > > > > > PS: This is a public list, so email disclaimers are rather pointless. > > > > It's best if they can be removed from mails sent here. > > > > > > > > > > -- > > Victor > > > > > -- > > Victor > > > > > -- > > Victor > -- Victor ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-02-09 12:17 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <CAGxG5cgvahRNJcfPNqnwvezW2Viv8cQNnxZScVRqHfjZ-Ma-qQ@mail.gmail.com>
2018-02-09 11:27 ` [dpdk-users] [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node Bruce Richardson
2018-02-09 11:53 ` Victor Huertas
2018-02-09 11:58 ` Richardson, Bruce
2018-02-09 12:01 ` Victor Huertas
2018-02-09 12:03 ` Richardson, Bruce
2018-02-09 12:17 ` Victor Huertas
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).