[PATCH] net/af_xdp: parse numa node id from sysfs

[PATCH] net/af_xdp: parse numa node id from sysfs

[Frank Du]

Get from /sys/class/net/{if}/device/numa_node.

Signed-off-by: Frank Du <frank.du@intel.com>
---
 drivers/net/af_xdp/rte_eth_af_xdp.c | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/drivers/net/af_xdp/rte_eth_af_xdp.c b/drivers/net/af_xdp/rte_eth_af_xdp.c
index b6ec9bf490..38b9d36ab5 100644
--- a/drivers/net/af_xdp/rte_eth_af_xdp.c
+++ b/drivers/net/af_xdp/rte_eth_af_xdp.c
@@ -39,6 +39,7 @@
 #include <rte_power_intrinsics.h>
 
 #include "compat.h"
+#include "eal_filesystem.h"
 
 #ifndef SO_PREFER_BUSY_POLL
 #define SO_PREFER_BUSY_POLL 69
@@ -2038,9 +2039,6 @@ rte_pmd_af_xdp_probe(struct rte_vdev_device *dev)
 		return -EINVAL;
 	}
 
-	if (dev->device.numa_node == SOCKET_ID_ANY)
-		dev->device.numa_node = rte_socket_id();
-
 	if (parse_parameters(kvlist, if_name, &xsk_start_queue_idx,
 			     &xsk_queue_cnt, &shared_umem, prog_path,
 			     &busy_budget, &force_copy) < 0) {
@@ -2053,6 +2051,19 @@ rte_pmd_af_xdp_probe(struct rte_vdev_device *dev)
 		return -EINVAL;
 	}
 
+	/* get numa node id from net sysfs */
+	if (dev->device.numa_node == SOCKET_ID_ANY) {
+		unsigned long numa = 0;
+		char numa_path[PATH_MAX];
+
+		snprintf(numa_path, sizeof(numa_path), "/sys/class/net/%s/device/numa_node",
+			 if_name);
+		if (eal_parse_sysfs_value(numa_path, &numa) != 0)
+			dev->device.numa_node = rte_socket_id();
+		else
+			dev->device.numa_node = numa;
+	}
+
 	busy_budget = busy_budget == -1 ? ETH_AF_XDP_DFLT_BUSY_BUDGET :
 					busy_budget;
 
-- 
2.34.1

Re: [PATCH] net/af_xdp: parse numa node id from sysfs

[Ferruh Yigit]

On 12/12/2022 12:48 AM, Frank Du wrote:
> Get from /sys/class/net/{if}/device/numa_node.
> 
> Signed-off-by: Frank Du <frank.du@intel.com>
> ---
>  drivers/net/af_xdp/rte_eth_af_xdp.c | 17 ++++++++++++++---
>  1 file changed, 14 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/net/af_xdp/rte_eth_af_xdp.c b/drivers/net/af_xdp/rte_eth_af_xdp.c
> index b6ec9bf490..38b9d36ab5 100644
> --- a/drivers/net/af_xdp/rte_eth_af_xdp.c
> +++ b/drivers/net/af_xdp/rte_eth_af_xdp.c
> @@ -39,6 +39,7 @@
>  #include <rte_power_intrinsics.h>
>  
>  #include "compat.h"
> +#include "eal_filesystem.h"
>  
>  #ifndef SO_PREFER_BUSY_POLL
>  #define SO_PREFER_BUSY_POLL 69
> @@ -2038,9 +2039,6 @@ rte_pmd_af_xdp_probe(struct rte_vdev_device *dev)
>  		return -EINVAL;
>  	}
>  
> -	if (dev->device.numa_node == SOCKET_ID_ANY)
> -		dev->device.numa_node = rte_socket_id();
> -
>  	if (parse_parameters(kvlist, if_name, &xsk_start_queue_idx,
>  			     &xsk_queue_cnt, &shared_umem, prog_path,
>  			     &busy_budget, &force_copy) < 0) {
> @@ -2053,6 +2051,19 @@ rte_pmd_af_xdp_probe(struct rte_vdev_device *dev)
>  		return -EINVAL;
>  	}
>  
> +	/* get numa node id from net sysfs */
> +	if (dev->device.numa_node == SOCKET_ID_ANY) {
> +		unsigned long numa = 0;
> +		char numa_path[PATH_MAX];
> +
> +		snprintf(numa_path, sizeof(numa_path), "/sys/class/net/%s/device/numa_node",
> +			 if_name);
> +		if (eal_parse_sysfs_value(numa_path, &numa) != 0)
> +			dev->device.numa_node = rte_socket_id();
> +		else
> +			dev->device.numa_node = numa;
> +	}
> +
>  	busy_budget = busy_budget == -1 ? ETH_AF_XDP_DFLT_BUSY_BUDGET :
>  					busy_budget;
>  

Hi Frank,

It looks reasonable to set virtual DPDK af_xdp device socket to actual
underlying device socket. And as I checked quickly, it works as expected.

But what is the impact and motivation of the patch? In other words why
you are doing this patch and what output you are expecting as a result?
Did you able to do any performance testing, and are you observing any
difference before and after this test?


Thanks,
ferruh

RE: [PATCH] net/af_xdp: parse numa node id from sysfs

[Du, Frank]

Hi ferruh,

Our application use rte_eth_dev_socket_id to query the socket that a NIC port connected, then allocate lcore/memory according to this affinity.

The remote memory access is really slow compared to local.

Thanks,
Frank

-----Original Message-----
From: Ferruh Yigit, <ferruh.yigit@amd.com> 
Sent: Monday, January 16, 2023 9:15 PM
To: Du, Frank <frank.du@intel.com>; Loftus, Ciara <ciara.loftus@intel.com>
Cc: dev@dpdk.org
Subject: Re: [PATCH] net/af_xdp: parse numa node id from sysfs

On 12/12/2022 12:48 AM, Frank Du wrote:
> Get from /sys/class/net/{if}/device/numa_node.
> 
> Signed-off-by: Frank Du <frank.du@intel.com>
> ---
>  drivers/net/af_xdp/rte_eth_af_xdp.c | 17 ++++++++++++++---
>  1 file changed, 14 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/net/af_xdp/rte_eth_af_xdp.c 
> b/drivers/net/af_xdp/rte_eth_af_xdp.c
> index b6ec9bf490..38b9d36ab5 100644
> --- a/drivers/net/af_xdp/rte_eth_af_xdp.c
> +++ b/drivers/net/af_xdp/rte_eth_af_xdp.c
> @@ -39,6 +39,7 @@
>  #include <rte_power_intrinsics.h>
>  
>  #include "compat.h"
> +#include "eal_filesystem.h"
>  
>  #ifndef SO_PREFER_BUSY_POLL
>  #define SO_PREFER_BUSY_POLL 69
> @@ -2038,9 +2039,6 @@ rte_pmd_af_xdp_probe(struct rte_vdev_device *dev)
>  		return -EINVAL;
>  	}
>  
> -	if (dev->device.numa_node == SOCKET_ID_ANY)
> -		dev->device.numa_node = rte_socket_id();
> -
>  	if (parse_parameters(kvlist, if_name, &xsk_start_queue_idx,
>  			     &xsk_queue_cnt, &shared_umem, prog_path,
>  			     &busy_budget, &force_copy) < 0) { @@ -2053,6 +2051,19 @@ 
> rte_pmd_af_xdp_probe(struct rte_vdev_device *dev)
>  		return -EINVAL;
>  	}
>  
> +	/* get numa node id from net sysfs */
> +	if (dev->device.numa_node == SOCKET_ID_ANY) {
> +		unsigned long numa = 0;
> +		char numa_path[PATH_MAX];
> +
> +		snprintf(numa_path, sizeof(numa_path), "/sys/class/net/%s/device/numa_node",
> +			 if_name);
> +		if (eal_parse_sysfs_value(numa_path, &numa) != 0)
> +			dev->device.numa_node = rte_socket_id();
> +		else
> +			dev->device.numa_node = numa;
> +	}
> +
>  	busy_budget = busy_budget == -1 ? ETH_AF_XDP_DFLT_BUSY_BUDGET :
>  					busy_budget;
>  

Hi Frank,

It looks reasonable to set virtual DPDK af_xdp device socket to actual underlying device socket. And as I checked quickly, it works as expected.

But what is the impact and motivation of the patch? In other words why you are doing this patch and what output you are expecting as a result?
Did you able to do any performance testing, and are you observing any difference before and after this test?


Thanks,
ferruh

Re: [PATCH] net/af_xdp: parse numa node id from sysfs

[Ferruh Yigit]

On 1/17/2023 1:35 AM, Du, Frank wrote:

Moved down, please don't top post.

>> 
>> -----Original Message-----
>> From: Ferruh Yigit, <ferruh.yigit@amd.com> 
>> Sent: Monday, January 16, 2023 9:15 PM
>> To: Du, Frank <frank.du@intel.com>; Loftus, Ciara <ciara.loftus@intel.com>
>> Cc: dev@dpdk.org
>> Subject: Re: [PATCH] net/af_xdp: parse numa node id from sysfs
>> 
>> On 12/12/2022 12:48 AM, Frank Du wrote:
>>> Get from /sys/class/net/{if}/device/numa_node.
>>>
>>> Signed-off-by: Frank Du <frank.du@intel.com>
>>> ---
>>>  drivers/net/af_xdp/rte_eth_af_xdp.c | 17 ++++++++++++++---
>>>  1 file changed, 14 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/drivers/net/af_xdp/rte_eth_af_xdp.c 
>>> b/drivers/net/af_xdp/rte_eth_af_xdp.c
>>> index b6ec9bf490..38b9d36ab5 100644
>>> --- a/drivers/net/af_xdp/rte_eth_af_xdp.c
>>> +++ b/drivers/net/af_xdp/rte_eth_af_xdp.c
>>> @@ -39,6 +39,7 @@
>>>  #include <rte_power_intrinsics.h>
>>>  
>>>  #include "compat.h"
>>> +#include "eal_filesystem.h"
>>>  
>>>  #ifndef SO_PREFER_BUSY_POLL
>>>  #define SO_PREFER_BUSY_POLL 69
>>> @@ -2038,9 +2039,6 @@ rte_pmd_af_xdp_probe(struct rte_vdev_device *dev)
>>>  		return -EINVAL;
>>>  	}
>>>  
>>> -	if (dev->device.numa_node == SOCKET_ID_ANY)
>>> -		dev->device.numa_node = rte_socket_id();
>>> -
>>>  	if (parse_parameters(kvlist, if_name, &xsk_start_queue_idx,
>>>  			     &xsk_queue_cnt, &shared_umem, prog_path,
>>>  			     &busy_budget, &force_copy) < 0) { @@ -2053,6 +2051,19 @@ 
>>> rte_pmd_af_xdp_probe(struct rte_vdev_device *dev)
>>>  		return -EINVAL;
>>>  	}
>>>  
>>> +	/* get numa node id from net sysfs */
>>> +	if (dev->device.numa_node == SOCKET_ID_ANY) {
>>> +		unsigned long numa = 0;
>>> +		char numa_path[PATH_MAX];
>>> +
>>> +		snprintf(numa_path, sizeof(numa_path), "/sys/class/net/%s/device/numa_node",
>>> +			 if_name);
>>> +		if (eal_parse_sysfs_value(numa_path, &numa) != 0)
>>> +			dev->device.numa_node = rte_socket_id();
>>> +		else
>>> +			dev->device.numa_node = numa;
>>> +	}
>>> +
>>>  	busy_budget = busy_budget == -1 ? ETH_AF_XDP_DFLT_BUSY_BUDGET :
>>>  					busy_budget;
>>>  
>> 
>> Hi Frank,
>> 
>> It looks reasonable to set virtual DPDK af_xdp device socket to actual underlying device socket. And as I checked quickly, it works as expected.
>> 
>> But what is the impact and motivation of the patch? In other words why you are doing this patch and what output you are expecting as a result?
>> Did you able to do any performance testing, and are you observing any difference before and after this test?
>> 
> 
> Hi ferruh,
> 
> Our application use rte_eth_dev_socket_id to query the socket that a NIC port connected, then allocate lcore/memory according to this affinity.
> 
> The remote memory access is really slow compared to local.
> 

As you observing any performance gain after change? If so, how much?

RE: [PATCH] net/af_xdp: parse numa node id from sysfs

[Du, Frank]

> -----Original Message-----
> From: Ferruh Yigit <ferruh.yigit@amd.com>
> Sent: Tuesday, January 17, 2023 5:14 PM
> To: Du, Frank <frank.du@intel.com>; Loftus, Ciara <ciara.loftus@intel.com>
> Cc: dev@dpdk.org
> Subject: Re: [PATCH] net/af_xdp: parse numa node id from sysfs
> 
> On 1/17/2023 1:35 AM, Du, Frank wrote:
> 
> Moved down, please don't top post.
> 
> >>
> >> -----Original Message-----
> >> From: Ferruh Yigit, <ferruh.yigit@amd.com>
> >> Sent: Monday, January 16, 2023 9:15 PM
> >> To: Du, Frank <frank.du@intel.com>; Loftus, Ciara
> >> <ciara.loftus@intel.com>
> >> Cc: dev@dpdk.org
> >> Subject: Re: [PATCH] net/af_xdp: parse numa node id from sysfs
> >>
> >> On 12/12/2022 12:48 AM, Frank Du wrote:
> >>> Get from /sys/class/net/{if}/device/numa_node.
> >>>
> >>> Signed-off-by: Frank Du <frank.du@intel.com>
> >>> ---
> >>>  drivers/net/af_xdp/rte_eth_af_xdp.c | 17 ++++++++++++++---
> >>>  1 file changed, 14 insertions(+), 3 deletions(-)
> >>>
> >>> diff --git a/drivers/net/af_xdp/rte_eth_af_xdp.c
> >>> b/drivers/net/af_xdp/rte_eth_af_xdp.c
> >>> index b6ec9bf490..38b9d36ab5 100644
> >>> --- a/drivers/net/af_xdp/rte_eth_af_xdp.c
> >>> +++ b/drivers/net/af_xdp/rte_eth_af_xdp.c
> >>> @@ -39,6 +39,7 @@
> >>>  #include <rte_power_intrinsics.h>
> >>>
> >>>  #include "compat.h"
> >>> +#include "eal_filesystem.h"
> >>>
> >>>  #ifndef SO_PREFER_BUSY_POLL
> >>>  #define SO_PREFER_BUSY_POLL 69
> >>> @@ -2038,9 +2039,6 @@ rte_pmd_af_xdp_probe(struct rte_vdev_device
> *dev)
> >>>  		return -EINVAL;
> >>>  	}
> >>>
> >>> -	if (dev->device.numa_node == SOCKET_ID_ANY)
> >>> -		dev->device.numa_node = rte_socket_id();
> >>> -
> >>>  	if (parse_parameters(kvlist, if_name, &xsk_start_queue_idx,
> >>>  			     &xsk_queue_cnt, &shared_umem, prog_path,
> >>>  			     &busy_budget, &force_copy) < 0) { @@ -2053,6
> +2051,19 @@
> >>> rte_pmd_af_xdp_probe(struct rte_vdev_device *dev)
> >>>  		return -EINVAL;
> >>>  	}
> >>>
> >>> +	/* get numa node id from net sysfs */
> >>> +	if (dev->device.numa_node == SOCKET_ID_ANY) {
> >>> +		unsigned long numa = 0;
> >>> +		char numa_path[PATH_MAX];
> >>> +
> >>> +		snprintf(numa_path, sizeof(numa_path),
> "/sys/class/net/%s/device/numa_node",
> >>> +			 if_name);
> >>> +		if (eal_parse_sysfs_value(numa_path, &numa) != 0)
> >>> +			dev->device.numa_node = rte_socket_id();
> >>> +		else
> >>> +			dev->device.numa_node = numa;
> >>> +	}
> >>> +
> >>>  	busy_budget = busy_budget == -1 ? ETH_AF_XDP_DFLT_BUSY_BUDGET :
> >>>  					busy_budget;
> >>>
> >>
> >> Hi Frank,
> >>
> >> It looks reasonable to set virtual DPDK af_xdp device socket to actual
> underlying device socket. And as I checked quickly, it works as expected.
> >>
> >> But what is the impact and motivation of the patch? In other words why you
> are doing this patch and what output you are expecting as a result?
> >> Did you able to do any performance testing, and are you observing any
> difference before and after this test?
> >>
> >
> > Hi ferruh,
> >
> > Our application use rte_eth_dev_socket_id to query the socket that a NIC port
> connected, then allocate lcore/memory according to this affinity.
> >
> > The remote memory access is really slow compared to local.
> >
> 
> As you observing any performance gain after change? If so, how much?

Hi Ferruh,

The NIC in our setup is on the socket 1, if our workload(both memory and cpu) are running on socket 0, it can get max 12g/s throughput on a single core. With this patch, the workload is running on the correct socket 1 cpus, it can get up to 16g/s on a single core.

Re: [PATCH] net/af_xdp: parse numa node id from sysfs

[Ferruh Yigit]

On 1/18/2023 1:53 AM, Du, Frank wrote:
> 
>> -----Original Message-----
>> From: Ferruh Yigit <ferruh.yigit@amd.com>
>> Sent: Tuesday, January 17, 2023 5:14 PM
>> To: Du, Frank <frank.du@intel.com>; Loftus, Ciara <ciara.loftus@intel.com>
>> Cc: dev@dpdk.org
>> Subject: Re: [PATCH] net/af_xdp: parse numa node id from sysfs
>>
>> On 1/17/2023 1:35 AM, Du, Frank wrote:
>>
>> Moved down, please don't top post.
>>
>>>>
>>>> -----Original Message-----
>>>> From: Ferruh Yigit, <ferruh.yigit@amd.com>
>>>> Sent: Monday, January 16, 2023 9:15 PM
>>>> To: Du, Frank <frank.du@intel.com>; Loftus, Ciara
>>>> <ciara.loftus@intel.com>
>>>> Cc: dev@dpdk.org
>>>> Subject: Re: [PATCH] net/af_xdp: parse numa node id from sysfs
>>>>
>>>> On 12/12/2022 12:48 AM, Frank Du wrote:
>>>>> Get from /sys/class/net/{if}/device/numa_node.
>>>>>
>>>>> Signed-off-by: Frank Du <frank.du@intel.com>
>>>>> ---
>>>>>  drivers/net/af_xdp/rte_eth_af_xdp.c | 17 ++++++++++++++---
>>>>>  1 file changed, 14 insertions(+), 3 deletions(-)
>>>>>
>>>>> diff --git a/drivers/net/af_xdp/rte_eth_af_xdp.c
>>>>> b/drivers/net/af_xdp/rte_eth_af_xdp.c
>>>>> index b6ec9bf490..38b9d36ab5 100644
>>>>> --- a/drivers/net/af_xdp/rte_eth_af_xdp.c
>>>>> +++ b/drivers/net/af_xdp/rte_eth_af_xdp.c
>>>>> @@ -39,6 +39,7 @@
>>>>>  #include <rte_power_intrinsics.h>
>>>>>
>>>>>  #include "compat.h"
>>>>> +#include "eal_filesystem.h"
>>>>>
>>>>>  #ifndef SO_PREFER_BUSY_POLL
>>>>>  #define SO_PREFER_BUSY_POLL 69
>>>>> @@ -2038,9 +2039,6 @@ rte_pmd_af_xdp_probe(struct rte_vdev_device
>> *dev)
>>>>>  		return -EINVAL;
>>>>>  	}
>>>>>
>>>>> -	if (dev->device.numa_node == SOCKET_ID_ANY)
>>>>> -		dev->device.numa_node = rte_socket_id();
>>>>> -
>>>>>  	if (parse_parameters(kvlist, if_name, &xsk_start_queue_idx,
>>>>>  			     &xsk_queue_cnt, &shared_umem, prog_path,
>>>>>  			     &busy_budget, &force_copy) < 0) { @@ -2053,6
>> +2051,19 @@
>>>>> rte_pmd_af_xdp_probe(struct rte_vdev_device *dev)
>>>>>  		return -EINVAL;
>>>>>  	}
>>>>>
>>>>> +	/* get numa node id from net sysfs */
>>>>> +	if (dev->device.numa_node == SOCKET_ID_ANY) {
>>>>> +		unsigned long numa = 0;
>>>>> +		char numa_path[PATH_MAX];
>>>>> +
>>>>> +		snprintf(numa_path, sizeof(numa_path),
>> "/sys/class/net/%s/device/numa_node",
>>>>> +			 if_name);
>>>>> +		if (eal_parse_sysfs_value(numa_path, &numa) != 0)
>>>>> +			dev->device.numa_node = rte_socket_id();
>>>>> +		else
>>>>> +			dev->device.numa_node = numa;
>>>>> +	}
>>>>> +
>>>>>  	busy_budget = busy_budget == -1 ? ETH_AF_XDP_DFLT_BUSY_BUDGET :
>>>>>  					busy_budget;
>>>>>
>>>>
>>>> Hi Frank,
>>>>
>>>> It looks reasonable to set virtual DPDK af_xdp device socket to actual
>> underlying device socket. And as I checked quickly, it works as expected.
>>>>
>>>> But what is the impact and motivation of the patch? In other words why you
>> are doing this patch and what output you are expecting as a result?
>>>> Did you able to do any performance testing, and are you observing any
>> difference before and after this test?
>>>>
>>>
>>> Hi ferruh,
>>>
>>> Our application use rte_eth_dev_socket_id to query the socket that a NIC port
>> connected, then allocate lcore/memory according to this affinity.
>>>
>>> The remote memory access is really slow compared to local.
>>>
>>
>> As you observing any performance gain after change? If so, how much?
> 
> Hi Ferruh,
> 
> The NIC in our setup is on the socket 1, if our workload(both memory and cpu) are running on socket 0, it can get max 12g/s throughput on a single core. With this patch, the workload is running on the correct socket 1 cpus, it can get up to 16g/s on a single core.

Thanks for clarification.

Re: [PATCH] net/af_xdp: parse numa node id from sysfs

[Ferruh Yigit]

On 12/12/2022 12:48 AM, Frank Du wrote:
> Get from /sys/class/net/{if}/device/numa_node.
> 
> Signed-off-by: Frank Du <frank.du@intel.com>

Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>

Applied to dpdk-next-net/main, thanks.