Re: [PATCH] net/af_xdp: parse numa node id from sysfs

[Ferruh Yigit <ferruh.yigit@amd.com>]

On 1/18/2023 1:53 AM, Du, Frank wrote:
> 
>> -----Original Message-----
>> From: Ferruh Yigit <ferruh.yigit@amd.com>
>> Sent: Tuesday, January 17, 2023 5:14 PM
>> To: Du, Frank <frank.du@intel.com>; Loftus, Ciara <ciara.loftus@intel.com>
>> Cc: dev@dpdk.org
>> Subject: Re: [PATCH] net/af_xdp: parse numa node id from sysfs
>>
>> On 1/17/2023 1:35 AM, Du, Frank wrote:
>>
>> Moved down, please don't top post.
>>
>>>>
>>>> -----Original Message-----
>>>> From: Ferruh Yigit, <ferruh.yigit@amd.com>
>>>> Sent: Monday, January 16, 2023 9:15 PM
>>>> To: Du, Frank <frank.du@intel.com>; Loftus, Ciara
>>>> <ciara.loftus@intel.com>
>>>> Cc: dev@dpdk.org
>>>> Subject: Re: [PATCH] net/af_xdp: parse numa node id from sysfs
>>>>
>>>> On 12/12/2022 12:48 AM, Frank Du wrote:
>>>>> Get from /sys/class/net/{if}/device/numa_node.
>>>>>
>>>>> Signed-off-by: Frank Du <frank.du@intel.com>
>>>>> ---
>>>>>  drivers/net/af_xdp/rte_eth_af_xdp.c | 17 ++++++++++++++---
>>>>>  1 file changed, 14 insertions(+), 3 deletions(-)
>>>>>
>>>>> diff --git a/drivers/net/af_xdp/rte_eth_af_xdp.c
>>>>> b/drivers/net/af_xdp/rte_eth_af_xdp.c
>>>>> index b6ec9bf490..38b9d36ab5 100644
>>>>> --- a/drivers/net/af_xdp/rte_eth_af_xdp.c
>>>>> +++ b/drivers/net/af_xdp/rte_eth_af_xdp.c
>>>>> @@ -39,6 +39,7 @@
>>>>>  #include <rte_power_intrinsics.h>
>>>>>
>>>>>  #include "compat.h"
>>>>> +#include "eal_filesystem.h"
>>>>>
>>>>>  #ifndef SO_PREFER_BUSY_POLL
>>>>>  #define SO_PREFER_BUSY_POLL 69
>>>>> @@ -2038,9 +2039,6 @@ rte_pmd_af_xdp_probe(struct rte_vdev_device
>> *dev)
>>>>>  		return -EINVAL;
>>>>>  	}
>>>>>
>>>>> -	if (dev->device.numa_node == SOCKET_ID_ANY)
>>>>> -		dev->device.numa_node = rte_socket_id();
>>>>> -
>>>>>  	if (parse_parameters(kvlist, if_name, &xsk_start_queue_idx,
>>>>>  			     &xsk_queue_cnt, &shared_umem, prog_path,
>>>>>  			     &busy_budget, &force_copy) < 0) { @@ -2053,6
>> +2051,19 @@
>>>>> rte_pmd_af_xdp_probe(struct rte_vdev_device *dev)
>>>>>  		return -EINVAL;
>>>>>  	}
>>>>>
>>>>> +	/* get numa node id from net sysfs */
>>>>> +	if (dev->device.numa_node == SOCKET_ID_ANY) {
>>>>> +		unsigned long numa = 0;
>>>>> +		char numa_path[PATH_MAX];
>>>>> +
>>>>> +		snprintf(numa_path, sizeof(numa_path),
>> "/sys/class/net/%s/device/numa_node",
>>>>> +			 if_name);
>>>>> +		if (eal_parse_sysfs_value(numa_path, &numa) != 0)
>>>>> +			dev->device.numa_node = rte_socket_id();
>>>>> +		else
>>>>> +			dev->device.numa_node = numa;
>>>>> +	}
>>>>> +
>>>>>  	busy_budget = busy_budget == -1 ? ETH_AF_XDP_DFLT_BUSY_BUDGET :
>>>>>  					busy_budget;
>>>>>
>>>>
>>>> Hi Frank,
>>>>
>>>> It looks reasonable to set virtual DPDK af_xdp device socket to actual
>> underlying device socket. And as I checked quickly, it works as expected.
>>>>
>>>> But what is the impact and motivation of the patch? In other words why you
>> are doing this patch and what output you are expecting as a result?
>>>> Did you able to do any performance testing, and are you observing any
>> difference before and after this test?
>>>>
>>>
>>> Hi ferruh,
>>>
>>> Our application use rte_eth_dev_socket_id to query the socket that a NIC port
>> connected, then allocate lcore/memory according to this affinity.
>>>
>>> The remote memory access is really slow compared to local.
>>>
>>
>> As you observing any performance gain after change? If so, how much?
> 
> Hi Ferruh,
> 
> The NIC in our setup is on the socket 1, if our workload(both memory and cpu) are running on socket 0, it can get max 12g/s throughput on a single core. With this patch, the workload is running on the correct socket 1 cpus, it can get up to 16g/s on a single core.

Thanks for clarification.