* Re: [dpdk-users] [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node
[not found] <CAGxG5cgvahRNJcfPNqnwvezW2Viv8cQNnxZScVRqHfjZ-Ma-qQ@mail.gmail.com>
@ 2018-02-09 11:27 ` Bruce Richardson
2018-02-09 11:53 ` Victor Huertas
0 siblings, 1 reply; 6+ messages in thread
From: Bruce Richardson @ 2018-02-09 11:27 UTC (permalink / raw)
To: Victor Huertas; +Cc: users
Moving to users list, since that is more relevant than dev.
On Thu, Feb 08, 2018 at 07:36:30PM +0100, Victor Huertas wrote:
> Bruce,
>
>
> My requirements are not that much (500 Mbps and 1 Gbps desirable).
>
>
> Thanks for your references links I will have a look at them.
>
>
> Regarding the NIC detection in the DPDK app by the DPDK EAL initialization
> after successfully having loaded the vfio-pci, it happens something strange.
>
> The nb_ports = rte_eth_dev_count(); is always returning 0.
>
>
> Therefore it shows an error telling that "port 0 is not present on the
> board".
>
>
> The EAL seems to detect the VFIOs as the only two logs that shows when
> initializing regarding VFIO are:
>
>
> EAL: Probing VFIO support...
>
> EAL: VFIO support initialized
>
>
> And nothing else... shouldn't the EAL detect the two NICs I associated to
> the VFIO? very strange...
>
>
Yes, it should. Can you please send full output on startup.
Also, can you try using igb_uio module instead of vfio and see if that
works for you.
Regards,
/Bruce
> Regards,
>
>
> PD: I have changed the email address account in order to avoid sending
> these disturbing disclaimers. Sorry.
>
>
> -----Mensaje original-----
> De: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Enviado el: jueves, 08 de febrero de 2018 17:42
> Para: Huertas García, Víctor
> CC: dev@dpdk.org
> Asunto: Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC
> association to Numa node
>
>
>
> On Thu, Feb 08, 2018 at 04:27:36PM +0000, Huertas García, Víctor wrote:
>
> >
>
> > Hi all,
>
> >
>
> > After having tried many ways to make the PCIe NIC card appear associated
> to a numa node, I haven't been able to do it.
>
> > That is, every time I try to look at which numa node belongs it always
> returns -1.
>
> >
>
> > $ cat /sys/bus/pci/devices/0000\:04\:00.1/numa_node
>
> > -1
>
> >
>
> > $ cat /sys/bus/pci/devices/0000\:04\:00.0/numa_node
>
> > -1
>
> >
>
> > Using lstopo, I confirm that all PCI cards are "outside" of any Numa
> node.
>
> >
>
> > I have read in previous posted messages in dpdk-dev community that this
> is normal in Nehalem generation Xeon architecture and there is nothing I
> can do about it. Can somebody confirm this?
>
>
>
> For that generation architecture, it is indeed expected. The NICs are not
> directly connected to any NUMA node.
>
>
>
> > If so, what implications could this have on packet capture and
> performance?
>
>
>
> Unsurprisingly, it's the case that newer platforms will perform better, as
> you are missing out on performance benefits from improved cores and also
> features like Intel® DDIO [1].
>
> However, what I/O throughput are you hoping to get from your system?
>
> Depending on your requirements, what you have may be enough. Some people
> use DPDK on lower-end platforms because that is all that they need. You may
> also find the chart on slide 6 of [2] of use to see how the max throughput
> of a platform has improved over time (and has improved further since that
> chart was published).
>
>
>
> >
>
> > Are the NICs available in my DPDK applications? Do I have to specifically
> "add" them by "-w 04:00.1 - w 04:00.0"?
>
>
>
> Yes, your NICs will still be available, even without NUMA affinity, and no,
> you should not need to explicitly whitelist them - though you can if you
> want. So long as they are bound to a uio or vfio driver (e.g.
>
> igb_uio or vfio-pci), they should be detected by DPDK EAL init and made
> available to your app.
>
>
>
> > Is RSS supported and usable from the DPDK application?
>
>
>
> Yes, at least for Intel NICs, and probably most other DPDK-supported NICs
> too.
>
>
>
> >
>
> > Thanks a lot for your attention
>
> >
>
> > Victor
>
> >
>
>
>
> /Bruce
>
>
>
> [1] https://www.intel.com/content/www/us/en/io/data-direct-i-o-t
> echnology.html
>
> [2] https://dpdksummit.com/Archive/pdf/2016Germany/DPDK-2016-
> DPDK_FD_IO_Introduction.pdf
>
>
>
> PS: This is a public list, so email disclaimers are rather pointless.
>
> It's best if they can be removed from mails sent here.
>
>
>
>
> --
> Victor
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-users] [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node
2018-02-09 11:27 ` [dpdk-users] [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node Bruce Richardson
@ 2018-02-09 11:53 ` Victor Huertas
2018-02-09 11:58 ` Richardson, Bruce
0 siblings, 1 reply; 6+ messages in thread
From: Victor Huertas @ 2018-02-09 11:53 UTC (permalink / raw)
To: Bruce Richardson; +Cc: users
Bruce,
With igb_uio it happens the same:
Network devices using DPDK-compatible driver
============================================
0000:04:00.0 '82576 Gigabit Network Connection 10c9' drv=igb_uio
unused=vfio-pci
0000:04:00.1 '82576 Gigabit Network Connection 10c9' drv=igb_uio
unused=vfio-pci
Network devices using kernel driver
===================================
0000:02:00.0 '82572EI Gigabit Ethernet Controller (Copper) 107d' if=eth1
drv=e1000e unused=igb_uio,vfio-pci *Active*
Even I have linked the rte libs with the flag --whole-archive flag (I found
somewhere in google that this solved a similar problem) but with no result.
All the output I get when I run the app (it is the l3fwd-acl example) is
what you see below:
cuda1@cuda1:~/git/dpdk-stable> sudo
/home/cuda1/eclipse-workspace/DPDK_Test/Debug/DPDK_Test -c 0x0f -n 2 -- -p
0x3
EAL: Detected 8 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
port 0 is not present on the board
EAL: Error - exiting with code: 1
Cause: check_port_config failed
2018-02-09 12:27 GMT+01:00 Bruce Richardson <bruce.richardson@intel.com>:
> Moving to users list, since that is more relevant than dev.
>
> On Thu, Feb 08, 2018 at 07:36:30PM +0100, Victor Huertas wrote:
> > Bruce,
> >
> >
> > My requirements are not that much (500 Mbps and 1 Gbps desirable).
> >
> >
> > Thanks for your references links I will have a look at them.
> >
> >
> > Regarding the NIC detection in the DPDK app by the DPDK EAL
> initialization
> > after successfully having loaded the vfio-pci, it happens something
> strange.
> >
> > The nb_ports = rte_eth_dev_count(); is always returning 0.
> >
> >
> > Therefore it shows an error telling that "port 0 is not present on the
> > board".
> >
> >
> > The EAL seems to detect the VFIOs as the only two logs that shows when
> > initializing regarding VFIO are:
> >
> >
> > EAL: Probing VFIO support...
> >
> > EAL: VFIO support initialized
> >
> >
> > And nothing else... shouldn't the EAL detect the two NICs I associated to
> > the VFIO? very strange...
> >
> >
>
> Yes, it should. Can you please send full output on startup.
> Also, can you try using igb_uio module instead of vfio and see if that
> works for you.
>
> Regards,
> /Bruce
>
> > Regards,
> >
> >
> > PD: I have changed the email address account in order to avoid sending
> > these disturbing disclaimers. Sorry.
> >
> >
> > -----Mensaje original-----
> > De: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > Enviado el: jueves, 08 de febrero de 2018 17:42
> > Para: Huertas García, Víctor
> > CC: dev@dpdk.org
> > Asunto: Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC
> > association to Numa node
> >
> >
> >
> > On Thu, Feb 08, 2018 at 04:27:36PM +0000, Huertas García, Víctor wrote:
> >
> > >
> >
> > > Hi all,
> >
> > >
> >
> > > After having tried many ways to make the PCIe NIC card appear
> associated
> > to a numa node, I haven't been able to do it.
> >
> > > That is, every time I try to look at which numa node belongs it always
> > returns -1.
> >
> > >
> >
> > > $ cat /sys/bus/pci/devices/0000\:04\:00.1/numa_node
> >
> > > -1
> >
> > >
> >
> > > $ cat /sys/bus/pci/devices/0000\:04\:00.0/numa_node
> >
> > > -1
> >
> > >
> >
> > > Using lstopo, I confirm that all PCI cards are "outside" of any Numa
> > node.
> >
> > >
> >
> > > I have read in previous posted messages in dpdk-dev community that this
> > is normal in Nehalem generation Xeon architecture and there is nothing I
> > can do about it. Can somebody confirm this?
> >
> >
> >
> > For that generation architecture, it is indeed expected. The NICs are not
> > directly connected to any NUMA node.
> >
> >
> >
> > > If so, what implications could this have on packet capture and
> > performance?
> >
> >
> >
> > Unsurprisingly, it's the case that newer platforms will perform better,
> as
> > you are missing out on performance benefits from improved cores and also
> > features like Intel® DDIO [1].
> >
> > However, what I/O throughput are you hoping to get from your system?
> >
> > Depending on your requirements, what you have may be enough. Some people
> > use DPDK on lower-end platforms because that is all that they need. You
> may
> > also find the chart on slide 6 of [2] of use to see how the max
> throughput
> > of a platform has improved over time (and has improved further since that
> > chart was published).
> >
> >
> >
> > >
> >
> > > Are the NICs available in my DPDK applications? Do I have to
> specifically
> > "add" them by "-w 04:00.1 - w 04:00.0"?
> >
> >
> >
> > Yes, your NICs will still be available, even without NUMA affinity, and
> no,
> > you should not need to explicitly whitelist them - though you can if you
> > want. So long as they are bound to a uio or vfio driver (e.g.
> >
> > igb_uio or vfio-pci), they should be detected by DPDK EAL init and made
> > available to your app.
> >
> >
> >
> > > Is RSS supported and usable from the DPDK application?
> >
> >
> >
> > Yes, at least for Intel NICs, and probably most other DPDK-supported NICs
> > too.
> >
> >
> >
> > >
> >
> > > Thanks a lot for your attention
> >
> > >
> >
> > > Victor
> >
> > >
> >
> >
> >
> > /Bruce
> >
> >
> >
> > [1] https://www.intel.com/content/www/us/en/io/data-direct-i-o-t
> > echnology.html
> >
> > [2] https://dpdksummit.com/Archive/pdf/2016Germany/DPDK-2016-
> > DPDK_FD_IO_Introduction.pdf
> >
> >
> >
> > PS: This is a public list, so email disclaimers are rather pointless.
> >
> > It's best if they can be removed from mails sent here.
> >
> >
> >
> >
> > --
> > Victor
>
--
Victor
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-users] [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node
2018-02-09 11:53 ` Victor Huertas
@ 2018-02-09 11:58 ` Richardson, Bruce
2018-02-09 12:01 ` Victor Huertas
0 siblings, 1 reply; 6+ messages in thread
From: Richardson, Bruce @ 2018-02-09 11:58 UTC (permalink / raw)
To: Victor Huertas; +Cc: users
Does it work with testpmd, built as part of DPDK?
When you build the l3fwd-acl example do you use the DPDK build system?
From: Victor Huertas [mailto:vhuertas@gmail.com]
Sent: Friday, February 9, 2018 11:54 AM
To: Richardson, Bruce <bruce.richardson@intel.com>
Cc: users@dpdk.org
Subject: Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node
Bruce,
With igb_uio it happens the same:
Network devices using DPDK-compatible driver
============================================
0000:04:00.0 '82576 Gigabit Network Connection 10c9' drv=igb_uio unused=vfio-pci
0000:04:00.1 '82576 Gigabit Network Connection 10c9' drv=igb_uio unused=vfio-pci
Network devices using kernel driver
===================================
0000:02:00.0 '82572EI Gigabit Ethernet Controller (Copper) 107d' if=eth1 drv=e1000e unused=igb_uio,vfio-pci *Active*
Even I have linked the rte libs with the flag --whole-archive flag (I found somewhere in google that this solved a similar problem) but with no result.
All the output I get when I run the app (it is the l3fwd-acl example) is what you see below:
cuda1@cuda1:~/git/dpdk-stable> sudo /home/cuda1/eclipse-workspace/DPDK_Test/Debug/DPDK_Test -c 0x0f -n 2 -- -p 0x3
EAL: Detected 8 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
port 0 is not present on the board
EAL: Error - exiting with code: 1
Cause: check_port_config failed
2018-02-09 12:27 GMT+01:00 Bruce Richardson <bruce.richardson@intel.com<mailto:bruce.richardson@intel.com>>:
Moving to users list, since that is more relevant than dev.
On Thu, Feb 08, 2018 at 07:36:30PM +0100, Victor Huertas wrote:
> Bruce,
>
>
> My requirements are not that much (500 Mbps and 1 Gbps desirable).
>
>
> Thanks for your references links I will have a look at them.
>
>
> Regarding the NIC detection in the DPDK app by the DPDK EAL initialization
> after successfully having loaded the vfio-pci, it happens something strange.
>
> The nb_ports = rte_eth_dev_count(); is always returning 0.
>
>
> Therefore it shows an error telling that "port 0 is not present on the
> board".
>
>
> The EAL seems to detect the VFIOs as the only two logs that shows when
> initializing regarding VFIO are:
>
>
> EAL: Probing VFIO support...
>
> EAL: VFIO support initialized
>
>
> And nothing else... shouldn't the EAL detect the two NICs I associated to
> the VFIO? very strange...
>
>
Yes, it should. Can you please send full output on startup.
Also, can you try using igb_uio module instead of vfio and see if that
works for you.
Regards,
/Bruce
> Regards,
>
>
> PD: I have changed the email address account in order to avoid sending
> these disturbing disclaimers. Sorry.
>
>
> -----Mensaje original-----
> De: Bruce Richardson [mailto:bruce.richardson@intel.com<mailto:bruce.richardson@intel.com>]
> Enviado el: jueves, 08 de febrero de 2018 17:42
> Para: Huertas García, Víctor
> CC: dev@dpdk.org<mailto:dev@dpdk.org>
> Asunto: Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC
> association to Numa node
>
>
>
> On Thu, Feb 08, 2018 at 04:27:36PM +0000, Huertas García, Víctor wrote:
>
> >
>
> > Hi all,
>
> >
>
> > After having tried many ways to make the PCIe NIC card appear associated
> to a numa node, I haven't been able to do it.
>
> > That is, every time I try to look at which numa node belongs it always
> returns -1.
>
> >
>
> > $ cat /sys/bus/pci/devices/0000\:04\:00.1/numa_node
>
> > -1
>
> >
>
> > $ cat /sys/bus/pci/devices/0000\:04\:00.0/numa_node
>
> > -1
>
> >
>
> > Using lstopo, I confirm that all PCI cards are "outside" of any Numa
> node.
>
> >
>
> > I have read in previous posted messages in dpdk-dev community that this
> is normal in Nehalem generation Xeon architecture and there is nothing I
> can do about it. Can somebody confirm this?
>
>
>
> For that generation architecture, it is indeed expected. The NICs are not
> directly connected to any NUMA node.
>
>
>
> > If so, what implications could this have on packet capture and
> performance?
>
>
>
> Unsurprisingly, it's the case that newer platforms will perform better, as
> you are missing out on performance benefits from improved cores and also
> features like Intel® DDIO [1].
>
> However, what I/O throughput are you hoping to get from your system?
>
> Depending on your requirements, what you have may be enough. Some people
> use DPDK on lower-end platforms because that is all that they need. You may
> also find the chart on slide 6 of [2] of use to see how the max throughput
> of a platform has improved over time (and has improved further since that
> chart was published).
>
>
>
> >
>
> > Are the NICs available in my DPDK applications? Do I have to specifically
> "add" them by "-w 04:00.1 - w 04:00.0"?
>
>
>
> Yes, your NICs will still be available, even without NUMA affinity, and no,
> you should not need to explicitly whitelist them - though you can if you
> want. So long as they are bound to a uio or vfio driver (e.g.
>
> igb_uio or vfio-pci), they should be detected by DPDK EAL init and made
> available to your app.
>
>
>
> > Is RSS supported and usable from the DPDK application?
>
>
>
> Yes, at least for Intel NICs, and probably most other DPDK-supported NICs
> too.
>
>
>
> >
>
> > Thanks a lot for your attention
>
> >
>
> > Victor
>
> >
>
>
>
> /Bruce
>
>
>
> [1] https://www.intel.com/content/www/us/en/io/data-direct-i-o-t
> echnology.html
>
> [2] https://dpdksummit.com/Archive/pdf/2016Germany/DPDK-2016-
> DPDK_FD_IO_Introduction.pdf
>
>
>
> PS: This is a public list, so email disclaimers are rather pointless.
>
> It's best if they can be removed from mails sent here.
>
>
>
>
> --
> Victor
--
Victor
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-users] [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node
2018-02-09 11:58 ` Richardson, Bruce
@ 2018-02-09 12:01 ` Victor Huertas
2018-02-09 12:03 ` Richardson, Bruce
0 siblings, 1 reply; 6+ messages in thread
From: Victor Huertas @ 2018-02-09 12:01 UTC (permalink / raw)
To: Richardson, Bruce; +Cc: users
Nope, I use Eclipse CDT environment to compile (I add all the static libs
manually and the executable is successfully generated).
Regards,
2018-02-09 12:58 GMT+01:00 Richardson, Bruce <bruce.richardson@intel.com>:
> Does it work with testpmd, built as part of DPDK?
>
> When you build the l3fwd-acl example do you use the DPDK build system?
>
>
>
> *From:* Victor Huertas [mailto:vhuertas@gmail.com]
> *Sent:* Friday, February 9, 2018 11:54 AM
> *To:* Richardson, Bruce <bruce.richardson@intel.com>
> *Cc:* users@dpdk.org
> *Subject:* Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe
> NIC association to Numa node
>
>
>
> Bruce,
>
> With igb_uio it happens the same:
>
> Network devices using DPDK-compatible driver
> ============================================
> 0000:04:00.0 '82576 Gigabit Network Connection 10c9' drv=igb_uio
> unused=vfio-pci
> 0000:04:00.1 '82576 Gigabit Network Connection 10c9' drv=igb_uio
> unused=vfio-pci
>
> Network devices using kernel driver
> ===================================
> 0000:02:00.0 '82572EI Gigabit Ethernet Controller (Copper) 107d' if=eth1
> drv=e1000e unused=igb_uio,vfio-pci *Active*
>
> Even I have linked the rte libs with the flag --whole-archive flag (I
> found somewhere in google that this solved a similar problem) but with no
> result.
>
> All the output I get when I run the app (it is the l3fwd-acl example) is
> what you see below:
>
> cuda1@cuda1:~/git/dpdk-stable> sudo /home/cuda1/eclipse-workspace/DPDK_Test/Debug/DPDK_Test
> -c 0x0f -n 2 -- -p 0x3
> EAL: Detected 8 lcore(s)
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> port 0 is not present on the board
> EAL: Error - exiting with code: 1
> Cause: check_port_config failed
>
>
>
> 2018-02-09 12:27 GMT+01:00 Bruce Richardson <bruce.richardson@intel.com>:
>
> Moving to users list, since that is more relevant than dev.
>
> On Thu, Feb 08, 2018 at 07:36:30PM +0100, Victor Huertas wrote:
> > Bruce,
> >
> >
> > My requirements are not that much (500 Mbps and 1 Gbps desirable).
> >
> >
> > Thanks for your references links I will have a look at them.
> >
> >
> > Regarding the NIC detection in the DPDK app by the DPDK EAL
> initialization
> > after successfully having loaded the vfio-pci, it happens something
> strange.
> >
> > The nb_ports = rte_eth_dev_count(); is always returning 0.
> >
> >
> > Therefore it shows an error telling that "port 0 is not present on the
> > board".
> >
> >
> > The EAL seems to detect the VFIOs as the only two logs that shows when
> > initializing regarding VFIO are:
> >
> >
> > EAL: Probing VFIO support...
> >
> > EAL: VFIO support initialized
> >
> >
> > And nothing else... shouldn't the EAL detect the two NICs I associated to
> > the VFIO? very strange...
> >
> >
>
> Yes, it should. Can you please send full output on startup.
> Also, can you try using igb_uio module instead of vfio and see if that
> works for you.
>
> Regards,
> /Bruce
>
>
> > Regards,
> >
> >
> > PD: I have changed the email address account in order to avoid sending
> > these disturbing disclaimers. Sorry.
> >
> >
> > -----Mensaje original-----
> > De: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > Enviado el: jueves, 08 de febrero de 2018 17:42
> > Para: Huertas García, Víctor
> > CC: dev@dpdk.org
> > Asunto: Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC
> > association to Numa node
> >
> >
> >
> > On Thu, Feb 08, 2018 at 04:27:36PM +0000, Huertas García, Víctor wrote:
> >
> > >
> >
> > > Hi all,
> >
> > >
> >
> > > After having tried many ways to make the PCIe NIC card appear
> associated
> > to a numa node, I haven't been able to do it.
> >
> > > That is, every time I try to look at which numa node belongs it always
> > returns -1.
> >
> > >
> >
> > > $ cat /sys/bus/pci/devices/0000\:04\:00.1/numa_node
> >
> > > -1
> >
> > >
> >
> > > $ cat /sys/bus/pci/devices/0000\:04\:00.0/numa_node
> >
> > > -1
> >
> > >
> >
> > > Using lstopo, I confirm that all PCI cards are "outside" of any Numa
> > node.
> >
> > >
> >
> > > I have read in previous posted messages in dpdk-dev community that this
> > is normal in Nehalem generation Xeon architecture and there is nothing I
> > can do about it. Can somebody confirm this?
> >
> >
> >
> > For that generation architecture, it is indeed expected. The NICs are not
> > directly connected to any NUMA node.
> >
> >
> >
> > > If so, what implications could this have on packet capture and
> > performance?
> >
> >
> >
> > Unsurprisingly, it's the case that newer platforms will perform better,
> as
> > you are missing out on performance benefits from improved cores and also
> > features like Intel® DDIO [1].
> >
> > However, what I/O throughput are you hoping to get from your system?
> >
> > Depending on your requirements, what you have may be enough. Some people
> > use DPDK on lower-end platforms because that is all that they need. You
> may
> > also find the chart on slide 6 of [2] of use to see how the max
> throughput
> > of a platform has improved over time (and has improved further since that
> > chart was published).
> >
> >
> >
> > >
> >
> > > Are the NICs available in my DPDK applications? Do I have to
> specifically
> > "add" them by "-w 04:00.1 - w 04:00.0"?
> >
> >
> >
> > Yes, your NICs will still be available, even without NUMA affinity, and
> no,
> > you should not need to explicitly whitelist them - though you can if you
> > want. So long as they are bound to a uio or vfio driver (e.g.
> >
> > igb_uio or vfio-pci), they should be detected by DPDK EAL init and made
> > available to your app.
> >
> >
> >
> > > Is RSS supported and usable from the DPDK application?
> >
> >
> >
> > Yes, at least for Intel NICs, and probably most other DPDK-supported NICs
> > too.
> >
> >
> >
> > >
> >
> > > Thanks a lot for your attention
> >
> > >
> >
> > > Victor
> >
> > >
> >
> >
> >
> > /Bruce
> >
> >
> >
> > [1] https://www.intel.com/content/www/us/en/io/data-direct-i-o-t
> > echnology.html
> >
> > [2] https://dpdksummit.com/Archive/pdf/2016Germany/DPDK-2016-
> > DPDK_FD_IO_Introduction.pdf
> >
> >
> >
> > PS: This is a public list, so email disclaimers are rather pointless.
> >
> > It's best if they can be removed from mails sent here.
> >
> >
> >
> >
> > --
> > Victor
>
>
>
>
> --
>
> Victor
>
--
Victor
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-users] [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node
2018-02-09 12:01 ` Victor Huertas
@ 2018-02-09 12:03 ` Richardson, Bruce
2018-02-09 12:17 ` Victor Huertas
0 siblings, 1 reply; 6+ messages in thread
From: Richardson, Bruce @ 2018-02-09 12:03 UTC (permalink / raw)
To: Victor Huertas; +Cc: users
You are probably missing another linker flag so - at a guess "--whole-archive".
Please test with the testpmd app built using DPDK itself.
/Bruce
From: Victor Huertas [mailto:vhuertas@gmail.com]
Sent: Friday, February 9, 2018 12:01 PM
To: Richardson, Bruce <bruce.richardson@intel.com>
Cc: users@dpdk.org
Subject: Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node
Nope, I use Eclipse CDT environment to compile (I add all the static libs manually and the executable is successfully generated).
Regards,
2018-02-09 12:58 GMT+01:00 Richardson, Bruce <bruce.richardson@intel.com<mailto:bruce.richardson@intel.com>>:
Does it work with testpmd, built as part of DPDK?
When you build the l3fwd-acl example do you use the DPDK build system?
From: Victor Huertas [mailto:vhuertas@gmail.com<mailto:vhuertas@gmail.com>]
Sent: Friday, February 9, 2018 11:54 AM
To: Richardson, Bruce <bruce.richardson@intel.com<mailto:bruce.richardson@intel.com>>
Cc: users@dpdk.org<mailto:users@dpdk.org>
Subject: Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node
Bruce,
With igb_uio it happens the same:
Network devices using DPDK-compatible driver
============================================
0000:04:00.0 '82576 Gigabit Network Connection 10c9' drv=igb_uio unused=vfio-pci
0000:04:00.1 '82576 Gigabit Network Connection 10c9' drv=igb_uio unused=vfio-pci
Network devices using kernel driver
===================================
0000:02:00.0 '82572EI Gigabit Ethernet Controller (Copper) 107d' if=eth1 drv=e1000e unused=igb_uio,vfio-pci *Active*
Even I have linked the rte libs with the flag --whole-archive flag (I found somewhere in google that this solved a similar problem) but with no result.
All the output I get when I run the app (it is the l3fwd-acl example) is what you see below:
cuda1@cuda1:~/git/dpdk-stable> sudo /home/cuda1/eclipse-workspace/DPDK_Test/Debug/DPDK_Test -c 0x0f -n 2 -- -p 0x3
EAL: Detected 8 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
port 0 is not present on the board
EAL: Error - exiting with code: 1
Cause: check_port_config failed
2018-02-09 12:27 GMT+01:00 Bruce Richardson <bruce.richardson@intel.com<mailto:bruce.richardson@intel.com>>:
Moving to users list, since that is more relevant than dev.
On Thu, Feb 08, 2018 at 07:36:30PM +0100, Victor Huertas wrote:
> Bruce,
>
>
> My requirements are not that much (500 Mbps and 1 Gbps desirable).
>
>
> Thanks for your references links I will have a look at them.
>
>
> Regarding the NIC detection in the DPDK app by the DPDK EAL initialization
> after successfully having loaded the vfio-pci, it happens something strange.
>
> The nb_ports = rte_eth_dev_count(); is always returning 0.
>
>
> Therefore it shows an error telling that "port 0 is not present on the
> board".
>
>
> The EAL seems to detect the VFIOs as the only two logs that shows when
> initializing regarding VFIO are:
>
>
> EAL: Probing VFIO support...
>
> EAL: VFIO support initialized
>
>
> And nothing else... shouldn't the EAL detect the two NICs I associated to
> the VFIO? very strange...
>
>
Yes, it should. Can you please send full output on startup.
Also, can you try using igb_uio module instead of vfio and see if that
works for you.
Regards,
/Bruce
> Regards,
>
>
> PD: I have changed the email address account in order to avoid sending
> these disturbing disclaimers. Sorry.
>
>
> -----Mensaje original-----
> De: Bruce Richardson [mailto:bruce.richardson@intel.com<mailto:bruce.richardson@intel.com>]
> Enviado el: jueves, 08 de febrero de 2018 17:42
> Para: Huertas García, Víctor
> CC: dev@dpdk.org<mailto:dev@dpdk.org>
> Asunto: Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC
> association to Numa node
>
>
>
> On Thu, Feb 08, 2018 at 04:27:36PM +0000, Huertas García, Víctor wrote:
>
> >
>
> > Hi all,
>
> >
>
> > After having tried many ways to make the PCIe NIC card appear associated
> to a numa node, I haven't been able to do it.
>
> > That is, every time I try to look at which numa node belongs it always
> returns -1.
>
> >
>
> > $ cat /sys/bus/pci/devices/0000\:04\:00.1/numa_node
>
> > -1
>
> >
>
> > $ cat /sys/bus/pci/devices/0000\:04\:00.0/numa_node
>
> > -1
>
> >
>
> > Using lstopo, I confirm that all PCI cards are "outside" of any Numa
> node.
>
> >
>
> > I have read in previous posted messages in dpdk-dev community that this
> is normal in Nehalem generation Xeon architecture and there is nothing I
> can do about it. Can somebody confirm this?
>
>
>
> For that generation architecture, it is indeed expected. The NICs are not
> directly connected to any NUMA node.
>
>
>
> > If so, what implications could this have on packet capture and
> performance?
>
>
>
> Unsurprisingly, it's the case that newer platforms will perform better, as
> you are missing out on performance benefits from improved cores and also
> features like Intel® DDIO [1].
>
> However, what I/O throughput are you hoping to get from your system?
>
> Depending on your requirements, what you have may be enough. Some people
> use DPDK on lower-end platforms because that is all that they need. You may
> also find the chart on slide 6 of [2] of use to see how the max throughput
> of a platform has improved over time (and has improved further since that
> chart was published).
>
>
>
> >
>
> > Are the NICs available in my DPDK applications? Do I have to specifically
> "add" them by "-w 04:00.1 - w 04:00.0"?
>
>
>
> Yes, your NICs will still be available, even without NUMA affinity, and no,
> you should not need to explicitly whitelist them - though you can if you
> want. So long as they are bound to a uio or vfio driver (e.g.
>
> igb_uio or vfio-pci), they should be detected by DPDK EAL init and made
> available to your app.
>
>
>
> > Is RSS supported and usable from the DPDK application?
>
>
>
> Yes, at least for Intel NICs, and probably most other DPDK-supported NICs
> too.
>
>
>
> >
>
> > Thanks a lot for your attention
>
> >
>
> > Victor
>
> >
>
>
>
> /Bruce
>
>
>
> [1] https://www.intel.com/content/www/us/en/io/data-direct-i-o-t
> echnology.html
>
> [2] https://dpdksummit.com/Archive/pdf/2016Germany/DPDK-2016-
> DPDK_FD_IO_Introduction.pdf
>
>
>
> PS: This is a public list, so email disclaimers are rather pointless.
>
> It's best if they can be removed from mails sent here.
>
>
>
>
> --
> Victor
--
Victor
--
Victor
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-users] [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node
2018-02-09 12:03 ` Richardson, Bruce
@ 2018-02-09 12:17 ` Victor Huertas
0 siblings, 0 replies; 6+ messages in thread
From: Victor Huertas @ 2018-02-09 12:17 UTC (permalink / raw)
To: Richardson, Bruce; +Cc: users
As you can see in the linker command --whole-archive was already used so
link all dpdk libs:
g++
-L/home/cuda1/git/dpdk-stable/x86_64-native-linuxapp-gcc/build/lib/librte_acl
-L/home/cuda1/git/dpdk-stable/x86_64-native-linuxapp-gcc/build/lib/librte_mbuf
-L/home/cuda1/git/dpdk-stable/x86_64-native-linuxapp-gcc/build/lib/librte_ether
-L/home/cuda1/git/dpdk-stable/x86_64-native-linuxapp-gcc/build/lib/librte_mempool
-L/home/cuda1/git/dpdk-stable/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/eal
-o "DPDK_Test" -Wl,--whole-archive -lrte_ethdev -lrte_mbuf -lrte_mempool
-lrte_acl -lrte_eal -Wl,--no-whole-archive ./src/DPDK_Test.o -lpthread
-lnuma -ldl -Wl,--as-needed
I will try to build it using DPDK. Perhaps something is being missed in my
Eclipse configuration
Regards,
2018-02-09 13:03 GMT+01:00 Richardson, Bruce <bruce.richardson@intel.com>:
> You are probably missing another linker flag so - at a guess
> "--whole-archive".
>
>
>
> Please test with the testpmd app built using DPDK itself.
>
>
>
> /Bruce
>
>
>
> *From:* Victor Huertas [mailto:vhuertas@gmail.com]
> *Sent:* Friday, February 9, 2018 12:01 PM
>
> *To:* Richardson, Bruce <bruce.richardson@intel.com>
> *Cc:* users@dpdk.org
> *Subject:* Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe
> NIC association to Numa node
>
>
>
> Nope, I use Eclipse CDT environment to compile (I add all the static libs
> manually and the executable is successfully generated).
>
> Regards,
>
>
>
> 2018-02-09 12:58 GMT+01:00 Richardson, Bruce <bruce.richardson@intel.com>:
>
> Does it work with testpmd, built as part of DPDK?
>
> When you build the l3fwd-acl example do you use the DPDK build system?
>
>
>
> *From:* Victor Huertas [mailto:vhuertas@gmail.com]
> *Sent:* Friday, February 9, 2018 11:54 AM
> *To:* Richardson, Bruce <bruce.richardson@intel.com>
> *Cc:* users@dpdk.org
> *Subject:* Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe
> NIC association to Numa node
>
>
>
> Bruce,
>
> With igb_uio it happens the same:
>
> Network devices using DPDK-compatible driver
> ============================================
> 0000:04:00.0 '82576 Gigabit Network Connection 10c9' drv=igb_uio
> unused=vfio-pci
> 0000:04:00.1 '82576 Gigabit Network Connection 10c9' drv=igb_uio
> unused=vfio-pci
>
> Network devices using kernel driver
> ===================================
> 0000:02:00.0 '82572EI Gigabit Ethernet Controller (Copper) 107d' if=eth1
> drv=e1000e unused=igb_uio,vfio-pci *Active*
>
> Even I have linked the rte libs with the flag --whole-archive flag (I
> found somewhere in google that this solved a similar problem) but with no
> result.
>
> All the output I get when I run the app (it is the l3fwd-acl example) is
> what you see below:
>
> cuda1@cuda1:~/git/dpdk-stable> sudo /home/cuda1/eclipse-workspace/DPDK_Test/Debug/DPDK_Test
> -c 0x0f -n 2 -- -p 0x3
> EAL: Detected 8 lcore(s)
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> port 0 is not present on the board
> EAL: Error - exiting with code: 1
> Cause: check_port_config failed
>
>
>
> 2018-02-09 12:27 GMT+01:00 Bruce Richardson <bruce.richardson@intel.com>:
>
> Moving to users list, since that is more relevant than dev.
>
> On Thu, Feb 08, 2018 at 07:36:30PM +0100, Victor Huertas wrote:
> > Bruce,
> >
> >
> > My requirements are not that much (500 Mbps and 1 Gbps desirable).
> >
> >
> > Thanks for your references links I will have a look at them.
> >
> >
> > Regarding the NIC detection in the DPDK app by the DPDK EAL
> initialization
> > after successfully having loaded the vfio-pci, it happens something
> strange.
> >
> > The nb_ports = rte_eth_dev_count(); is always returning 0.
> >
> >
> > Therefore it shows an error telling that "port 0 is not present on the
> > board".
> >
> >
> > The EAL seems to detect the VFIOs as the only two logs that shows when
> > initializing regarding VFIO are:
> >
> >
> > EAL: Probing VFIO support...
> >
> > EAL: VFIO support initialized
> >
> >
> > And nothing else... shouldn't the EAL detect the two NICs I associated to
> > the VFIO? very strange...
> >
> >
>
> Yes, it should. Can you please send full output on startup.
> Also, can you try using igb_uio module instead of vfio and see if that
> works for you.
>
> Regards,
> /Bruce
>
>
> > Regards,
> >
> >
> > PD: I have changed the email address account in order to avoid sending
> > these disturbing disclaimers. Sorry.
> >
> >
> > -----Mensaje original-----
> > De: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > Enviado el: jueves, 08 de febrero de 2018 17:42
> > Para: Huertas García, Víctor
> > CC: dev@dpdk.org
> > Asunto: Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC
> > association to Numa node
> >
> >
> >
> > On Thu, Feb 08, 2018 at 04:27:36PM +0000, Huertas García, Víctor wrote:
> >
> > >
> >
> > > Hi all,
> >
> > >
> >
> > > After having tried many ways to make the PCIe NIC card appear
> associated
> > to a numa node, I haven't been able to do it.
> >
> > > That is, every time I try to look at which numa node belongs it always
> > returns -1.
> >
> > >
> >
> > > $ cat /sys/bus/pci/devices/0000\:04\:00.1/numa_node
> >
> > > -1
> >
> > >
> >
> > > $ cat /sys/bus/pci/devices/0000\:04\:00.0/numa_node
> >
> > > -1
> >
> > >
> >
> > > Using lstopo, I confirm that all PCI cards are "outside" of any Numa
> > node.
> >
> > >
> >
> > > I have read in previous posted messages in dpdk-dev community that this
> > is normal in Nehalem generation Xeon architecture and there is nothing I
> > can do about it. Can somebody confirm this?
> >
> >
> >
> > For that generation architecture, it is indeed expected. The NICs are not
> > directly connected to any NUMA node.
> >
> >
> >
> > > If so, what implications could this have on packet capture and
> > performance?
> >
> >
> >
> > Unsurprisingly, it's the case that newer platforms will perform better,
> as
> > you are missing out on performance benefits from improved cores and also
> > features like Intel® DDIO [1].
> >
> > However, what I/O throughput are you hoping to get from your system?
> >
> > Depending on your requirements, what you have may be enough. Some people
> > use DPDK on lower-end platforms because that is all that they need. You
> may
> > also find the chart on slide 6 of [2] of use to see how the max
> throughput
> > of a platform has improved over time (and has improved further since that
> > chart was published).
> >
> >
> >
> > >
> >
> > > Are the NICs available in my DPDK applications? Do I have to
> specifically
> > "add" them by "-w 04:00.1 - w 04:00.0"?
> >
> >
> >
> > Yes, your NICs will still be available, even without NUMA affinity, and
> no,
> > you should not need to explicitly whitelist them - though you can if you
> > want. So long as they are bound to a uio or vfio driver (e.g.
> >
> > igb_uio or vfio-pci), they should be detected by DPDK EAL init and made
> > available to your app.
> >
> >
> >
> > > Is RSS supported and usable from the DPDK application?
> >
> >
> >
> > Yes, at least for Intel NICs, and probably most other DPDK-supported NICs
> > too.
> >
> >
> >
> > >
> >
> > > Thanks a lot for your attention
> >
> > >
> >
> > > Victor
> >
> > >
> >
> >
> >
> > /Bruce
> >
> >
> >
> > [1] https://www.intel.com/content/www/us/en/io/data-direct-i-o-t
> > echnology.html
> >
> > [2] https://dpdksummit.com/Archive/pdf/2016Germany/DPDK-2016-
> > DPDK_FD_IO_Introduction.pdf
> >
> >
> >
> > PS: This is a public list, so email disclaimers are rather pointless.
> >
> > It's best if they can be removed from mails sent here.
> >
> >
> >
> >
> > --
> > Victor
>
>
>
>
> --
>
> Victor
>
>
>
>
> --
>
> Victor
>
--
Victor
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-02-09 12:17 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <CAGxG5cgvahRNJcfPNqnwvezW2Viv8cQNnxZScVRqHfjZ-Ma-qQ@mail.gmail.com>
2018-02-09 11:27 ` [dpdk-users] [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node Bruce Richardson
2018-02-09 11:53 ` Victor Huertas
2018-02-09 11:58 ` Richardson, Bruce
2018-02-09 12:01 ` Victor Huertas
2018-02-09 12:03 ` Richardson, Bruce
2018-02-09 12:17 ` Victor Huertas
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).