[dpdk-dev] NUMA node/socket

[dpdk-dev] NUMA node/socket

[Thomas Monjalon]

There are some mix between NUMA node and socket IDs in DPDK.
Examples:
	* rte_eth_dev_socket_id() returns the NUMA node.
	* rte_malloc use sockets to allocate the memory

Is it critical?

Re: [dpdk-dev] NUMA node/socket

[Thomas Monjalon]

04/07/2021 03:38, Thomas Monjalon:
> There are some mix between NUMA node and socket IDs in DPDK.
> Examples:
> 	* rte_eth_dev_socket_id() returns the NUMA node.
> 	* rte_malloc use sockets to allocate the memory
> 
> Is it critical?

There is a function, implemented for Windows only,
which distinguishes clearly node and socket
but it assumes there is only 1 node per socket:

unsigned int
eal_socket_numa_node(unsigned int socket_id)
{
    return cpu_map.sockets[socket_id].node_id;
}

Reminder: AMD can be configured to have multiple nodes per socket.

Re: [dpdk-dev] NUMA node/socket

[Andrew Rybchenko]

On 7/4/21 4:53 AM, Thomas Monjalon wrote:
> 04/07/2021 03:38, Thomas Monjalon:
>> There are some mix between NUMA node and socket IDs in DPDK.
>> Examples:
>> 	* rte_eth_dev_socket_id() returns the NUMA node.
>> 	* rte_malloc use sockets to allocate the memory
>>
>> Is it critical?
> 
> There is a function, implemented for Windows only,
> which distinguishes clearly node and socket
> but it assumes there is only 1 node per socket:
> 
> unsigned int
> eal_socket_numa_node(unsigned int socket_id)
> {
>     return cpu_map.sockets[socket_id].node_id;
> }
> 
> Reminder: AMD can be configured to have multiple nodes per socket.

Taking the reminder into account the topic definitely
requires improvements.

I apologize for my ignorance, but
Is socket ID really interesting to anybody in DPDK?
If no, we should just clarify terminology and switch
to NUMA node everywhere.

Re: [dpdk-dev] NUMA node/socket

[Thomas Monjalon]

04/07/2021 10:27, Andrew Rybchenko:
> On 7/4/21 4:53 AM, Thomas Monjalon wrote:
> > 04/07/2021 03:38, Thomas Monjalon:
> >> There are some mix between NUMA node and socket IDs in DPDK.
> >> Examples:
> >> 	* rte_eth_dev_socket_id() returns the NUMA node.
> >> 	* rte_malloc use sockets to allocate the memory
> >>
> >> Is it critical?
> > 
> > There is a function, implemented for Windows only,
> > which distinguishes clearly node and socket
> > but it assumes there is only 1 node per socket:
> > 
> > unsigned int
> > eal_socket_numa_node(unsigned int socket_id)
> > {
> >     return cpu_map.sockets[socket_id].node_id;
> > }
> > 
> > Reminder: AMD can be configured to have multiple nodes per socket.
> 
> Taking the reminder into account the topic definitely
> requires improvements.
> 
> I apologize for my ignorance, but
> Is socket ID really interesting to anybody in DPDK?

I think the socket ID might be interesting for the threads,
but not for memory or devices.

> If no, we should just clarify terminology and switch
> to NUMA node everywhere.

I have the same opinion as Andrew.
If socket ID is required, it could be deduced from the NUMA node
or from the CPU core.

Re: [dpdk-dev] NUMA node/socket

[Stephen Hemminger]

On Sun, 04 Jul 2021 11:28:23 +0200
Thomas Monjalon <thomas@monjalon.net> wrote:

> 04/07/2021 10:27, Andrew Rybchenko:
> > On 7/4/21 4:53 AM, Thomas Monjalon wrote:  
> > > 04/07/2021 03:38, Thomas Monjalon:  
> > >> There are some mix between NUMA node and socket IDs in DPDK.
> > >> Examples:
> > >> 	* rte_eth_dev_socket_id() returns the NUMA node.
> > >> 	* rte_malloc use sockets to allocate the memory
> > >>
> > >> Is it critical?  
> > > 
> > > There is a function, implemented for Windows only,
> > > which distinguishes clearly node and socket
> > > but it assumes there is only 1 node per socket:
> > > 
> > > unsigned int
> > > eal_socket_numa_node(unsigned int socket_id)
> > > {
> > >     return cpu_map.sockets[socket_id].node_id;
> > > }
> > > 
> > > Reminder: AMD can be configured to have multiple nodes per socket.  
> > 
> > Taking the reminder into account the topic definitely
> > requires improvements.
> > 
> > I apologize for my ignorance, but
> > Is socket ID really interesting to anybody in DPDK?  
> 
> I think the socket ID might be interesting for the threads,
> but not for memory or devices.
> 
> > If no, we should just clarify terminology and switch
> > to NUMA node everywhere.  
> 
> I have the same opinion as Andrew.
> If socket ID is required, it could be deduced from the NUMA node
> or from the CPU core.
> 
> 

Agree that NUMA node should be used everywhere.
There can be some confusion because some architectures define NUMA
for lcore's that have slower interconnect even if they are on the same chip.
Also Dell used to have different terminology for socket number on some machines.
The socket id reported by BIOS started with 1!

Re: [dpdk-dev] NUMA node/socket

[Dmitry Kozlyuk]

2021-07-04 11:28 (UTC+0200), Thomas Monjalon:
> 04/07/2021 10:27, Andrew Rybchenko:
> > On 7/4/21 4:53 AM, Thomas Monjalon wrote:  
> > > 04/07/2021 03:38, Thomas Monjalon:  
> > >> There are some mix between NUMA node and socket IDs in DPDK.
> > >> Examples:
> > >> 	* rte_eth_dev_socket_id() returns the NUMA node.
> > >> 	* rte_malloc use sockets to allocate the memory
> > >>
> > >> Is it critical?  
> > > 
> > > There is a function, implemented for Windows only,
> > > which distinguishes clearly node and socket
> > > but it assumes there is only 1 node per socket:
> > > 
> > > unsigned int
> > > eal_socket_numa_node(unsigned int socket_id)
> > > {
> > >     return cpu_map.sockets[socket_id].node_id;
> > > }
> > > 
> > > Reminder: AMD can be configured to have multiple nodes per socket.
> >
> > Taking the reminder into account the topic definitely
> > requires improvements.
> > 
> > I apologize for my ignorance, but
> > Is socket ID really interesting to anybody in DPDK?  
> 
> I think the socket ID might be interesting for the threads,
> but not for memory or devices.
> 
> > If no, we should just clarify terminology and switch
> > to NUMA node everywhere.  
> 
> I have the same opinion as Andrew.
> If socket ID is required, it could be deduced from the NUMA node
> or from the CPU core.

I agree with renaming too.
Everywhere in DPDK "socket ID" really means "NUMA node".

I don't see how exactly socket ID can be deduced from NUMA node or CPU core
(assuming rte_socket_id becomes rte_numa_node_id), but I also can't imagine
why an app would need it. EAL could use NUMA distance info for better memory
management: currently SOCKET_ID_ANY means "current NUMA node or the first one
with enough memory available" while it could be "or the closest one".