Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node

Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node

[Victor Huertas]

Bruce,


My requirements are not that much (500 Mbps and 1 Gbps desirable).


Thanks for your references links I will have a look at them.


Regarding the NIC detection in the DPDK app by the DPDK EAL initialization
after successfully having loaded the vfio-pci, it happens something strange.

The nb_ports = rte_eth_dev_count(); is always returning 0.


Therefore it shows an error telling that "port 0 is not present on the
board".


The EAL seems to detect the VFIOs as the only two logs that shows when
initializing regarding VFIO are:


EAL: Probing VFIO support...

EAL: VFIO support initialized


And nothing else... shouldn't the EAL detect the two NICs I associated to
the VFIO? very strange...


Regards,


PD: I have changed the email address account in order to avoid sending
these disturbing disclaimers. Sorry.


-----Mensaje original-----
De: Bruce Richardson [mailto:bruce.richardson@intel.com]
Enviado el: jueves, 08 de febrero de 2018 17:42
Para: Huertas García, Víctor
CC: dev@dpdk.org
Asunto: Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC
association to Numa node



On Thu, Feb 08, 2018 at 04:27:36PM +0000, Huertas García, Víctor wrote:

>

> Hi all,

>

> After having tried many ways to make the PCIe NIC card appear associated
to a numa node, I haven't been able to do it.

> That is, every time I try to look at which numa node belongs it always
returns -1.

>

> $ cat /sys/bus/pci/devices/0000\:04\:00.1/numa_node

> -1

>

> $ cat /sys/bus/pci/devices/0000\:04\:00.0/numa_node

> -1

>

> Using lstopo, I confirm that all PCI cards are "outside" of  any Numa
node.

>

> I have read in previous posted messages in dpdk-dev community that this
is normal in Nehalem generation Xeon architecture and there is nothing I
can do about it. Can somebody confirm this?



For that generation architecture, it is indeed expected. The NICs are not
directly connected to any NUMA node.



> If so, what implications could this have on packet capture and
performance?



Unsurprisingly, it's the case that newer platforms will perform better, as
you are missing out on performance benefits from improved cores and also
features like Intel® DDIO [1].

However, what I/O throughput are you hoping to get from your system?

Depending on your requirements, what you have may be enough. Some people
use DPDK on lower-end platforms because that is all that they need. You may
also find the chart on slide 6 of [2] of use to see how the max throughput
of a platform has improved over time (and has improved further since that
chart was published).



>

> Are the NICs available in my DPDK applications? Do I have to specifically
"add" them by "-w 04:00.1 - w 04:00.0"?



Yes, your NICs will still be available, even without NUMA affinity, and no,
you should not need to explicitly whitelist them - though you can if you
want. So long as they are bound to a uio or vfio driver (e.g.

igb_uio or vfio-pci), they should be detected by DPDK EAL init and made
available to your app.



> Is RSS supported and usable from the DPDK application?



Yes, at least for Intel NICs, and probably most other DPDK-supported NICs
too.



>

> Thanks a lot for your attention

>

> Victor

>



/Bruce



[1] https://www.intel.com/content/www/us/en/io/data-direct-i-o-t
echnology.html

[2] https://dpdksummit.com/Archive/pdf/2016Germany/DPDK-2016-
DPDK_FD_IO_Introduction.pdf



PS: This is a public list, so email disclaimers are rather pointless.

It's best if they can be removed from mails sent here.




-- 
Victor

Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node

[Bruce Richardson]

On Thu, Feb 08, 2018 at 04:27:36PM +0000, Huertas García, Víctor wrote:
> 
> Hi all,
> 
> After having tried many ways to make the PCIe NIC card appear associated to a numa node, I haven't been able to do it.
> That is, every time I try to look at which numa node belongs it always returns -1.
> 
> $ cat /sys/bus/pci/devices/0000\:04\:00.1/numa_node
> -1
> 
> $ cat /sys/bus/pci/devices/0000\:04\:00.0/numa_node
> -1
> 
> Using lstopo, I confirm that all PCI cards are "outside" of  any Numa node.
> 
> I have read in previous posted messages in dpdk-dev community that this is normal in Nehalem generation Xeon architecture and there is nothing I can do about it. Can somebody confirm this?

For that generation architecture, it is indeed expected. The NICs are
not directly connected to any NUMA node.

> If so, what implications could this have on packet capture and performance?

Unsurprisingly, it's the case that newer platforms will perform better,
as you are missing out on performance benefits from improved cores and
also features like Intel® DDIO [1].
However, what I/O throughput are you hoping to get from your system?
Depending on your requirements, what you have may be enough. Some people
use DPDK on lower-end platforms because that is all that they need. You
may also find the chart on slide 6 of [2] of use to see how the max
throughput of a platform has improved over time (and has improved
further since that chart was published).

> 
> Are the NICs available in my DPDK applications? Do I have to specifically "add" them by "-w 04:00.1 - w 04:00.0"?

Yes, your NICs will still be available, even without NUMA affinity, and
no, you should not need to explicitly whitelist them - though you can if
you want. So long as they are bound to a uio or vfio driver (e.g.
igb_uio or vfio-pci), they should be detected by DPDK EAL init and made
available to your app.

> Is RSS supported and usable from the DPDK application?

Yes, at least for Intel NICs, and probably most other DPDK-supported
NICs too.

> 
> Thanks a lot for your attention
> 
> Victor
> 

/Bruce

[1] https://www.intel.com/content/www/us/en/io/data-direct-i-o-technology.html
[2] https://dpdksummit.com/Archive/pdf/2016Germany/DPDK-2016-DPDK_FD_IO_Introduction.pdf

PS: This is a public list, so email disclaimers are rather pointless.
It's best if they can be removed from mails sent here.

> ________________________________
> 
> Este correo electrónico y, en su caso, cualquier fichero anexo al mismo, contiene información de carácter confidencial exclusivamente dirigida a su destinatario o destinatarios. Si no es vd. el destinatario indicado, queda notificado que la lectura, utilización, divulgación y/o copia sin autorización está prohibida en virtud de la legislación vigente. En el caso de haber recibido este correo electrónico por error, se ruega notificar inmediatamente esta circunstancia mediante reenvío a la dirección electrónica del remitente.
> Evite imprimir este mensaje si no es estrictamente necesario.
> 
> This email and any file attached to it (when applicable) contain(s) confidential information that is exclusively addressed to its recipient(s). If you are not the indicated recipient, you are informed that reading, using, disseminating and/or copying it without authorisation is forbidden in accordance with the legislation in effect. If you have received this email by mistake, please immediately notify the sender of the situation by resending it to their email address.
> Avoid printing this message if it is not absolutely necessary.

[dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node

[Huertas García, Víctor]

Hi all,

After having tried many ways to make the PCIe NIC card appear associated to a numa node, I haven't been able to do it.
That is, every time I try to look at which numa node belongs it always returns -1.

$ cat /sys/bus/pci/devices/0000\:04\:00.1/numa_node
-1

$ cat /sys/bus/pci/devices/0000\:04\:00.0/numa_node
-1

Using lstopo, I confirm that all PCI cards are "outside" of  any Numa node.

I have read in previous posted messages in dpdk-dev community that this is normal in Nehalem generation Xeon architecture and there is nothing I can do about it. Can somebody confirm this?
If so, what implications could this have on packet capture and performance?

Are the NICs available in my DPDK applications? Do I have to specifically "add" them by "-w 04:00.1 - w 04:00.0"?
Is RSS supported and usable from the DPDK application?

Thanks a lot for your attention

Victor

________________________________

Este correo electrónico y, en su caso, cualquier fichero anexo al mismo, contiene información de carácter confidencial exclusivamente dirigida a su destinatario o destinatarios. Si no es vd. el destinatario indicado, queda notificado que la lectura, utilización, divulgación y/o copia sin autorización está prohibida en virtud de la legislación vigente. En el caso de haber recibido este correo electrónico por error, se ruega notificar inmediatamente esta circunstancia mediante reenvío a la dirección electrónica del remitente.
Evite imprimir este mensaje si no es estrictamente necesario.

This email and any file attached to it (when applicable) contain(s) confidential information that is exclusively addressed to its recipient(s). If you are not the indicated recipient, you are informed that reading, using, disseminating and/or copying it without authorisation is forbidden in accordance with the legislation in effect. If you have received this email by mistake, please immediately notify the sender of the situation by resending it to their email address.
Avoid printing this message if it is not absolutely necessary.