[dpdk-users] Question about NUMA node for PCI devices and rte_eal_init

[dpdk-users] Question about NUMA node for PCI devices and rte_eal_init

[Jensen, Allen]

This is my first email to the list so apologies if it’s off-topic or anything – just let me know (flame to live, live to flame…)

Had used version 2.2.0 for a while when upgrading to 17.05.02 we discovered the PCI port scan was now trying to allocate memory only from pools allocated for the NUMA node the PCI device was mapped to. In version 2.2.0, it did not do this and used the equivalent of SOCKET_ID_ANY (-1) for the NUMA node.

We had a scenario in which there was a single interface on numa node 1 and all others were on node 0 and we were just allocating everything on node 0 (the on node 0 was a slow interface anyway so performance was not an issue).

It turns out that now an internal data structure – device.dev_private – cannot be allocated because we were not allocating any memory buffer pools for node 1.

My question is – does this dev_private area really need to be forced onto a specific numa node?

Specifically – the line is:

lib/librte_ether/rte_ethdev_pci.h:
                eth_dev->data->dev_private = rte_zmalloc_socket(name,
                                private_data_size, RTE_CACHE_LINE_SIZE,
                                dev->device.numa_node);

I have been trying to understand what eth_dev->data->dev_private is used for and implications of patching ours to use SOCKET_ID_ANY instead of the numa node….

Does any one else have experience with this or have thoughts on the matter?

--
Allen Jensen
Software Engineer
Arbor Networks
pjensen@arbor.net<mailto:bstpierre@arbor.net>
1 404 538 2759
https://www.arbornetworks.com/