Re: [PATCH v4] net/af_xdp: re-enable secondary process support

[Ferruh Yigit <ferruh.yigit@intel.com>]

On 2/11/2022 9:26 AM, Loftus, Ciara wrote:
>>>
>>> On 2/10/2022 5:47 PM, Loftus, Ciara wrote:
>>>>> Subject: Re: [PATCH v4] net/af_xdp: re-enable secondary process
>> support
>>>>>
>>>>> On 2/10/2022 3:40 PM, Loftus, Ciara wrote:
>>>>>>> Subject: Re: [PATCH v4] net/af_xdp: re-enable secondary process
>>> support
>>>>>>>
>>>>>>> On 2/9/2022 9:48 AM, Ciara Loftus wrote:
>>>>>>>> Secondary process support had been disabled for the AF_XDP PMD
>>>>>>> because
>>>>>>>> there was no logic in place to share the AF_XDP socket file
>> descriptors
>>>>>>>> between the processes. This commit introduces this logic using the
>>> IPC
>>>>>>>> APIs.
>>>>>>>>
>>>>>>>> Rx and Tx are disabled in the secondary process due to memory
>>> mapping
>>>>> of
>>>>>>>> the AF_XDP rings being assigned by the kernel in the primary
>> process
>>>>> only.
>>>>>>>> However other operations including retrieval of stats are permitted.
>>>>>>>>
>>>>>>>> Signed-off-by: Ciara Loftus <ciara.loftus@intel.com>
>>>>>>>>
>>>>>>>
>>>>>>> Hi Ciara,
>>>>>>>
>>>>>>> When I tried to test the patch getting following error [1], it doesn't
>> look
>>>>>>> related to this patch but can you help to fix the issue, thanks.
>>>>>>>
>>>>>>> [1]
>>>>>>> libxdp: Couldn't find a BPF file with name xsk_def_xdp_prog.o
>>>>>>> xsk_configure(): Failed to create xsk socket.
>>>>>>> eth_rx_queue_setup(): Failed to configure xdp socket
>>>>>>> Fail to configure port 2 rx queues
>>>>>>> EAL: Error - exiting with code: 1
>>>>>>
>>>>>>
>>>>>> Hi Ferruh,
>>>>>>
>>>>>> This file should be generated when libxdp is compiled.
>>>>>> Mine is located @ /usr/local/lib/bpf/xsk_def_xdp_prog.o
>>>>>> Can you check if that file is there for you? It could be in
>>>>> /usr/local/lib64/bpf/ on your machine.
>>>>>> What kernel are you running on?
>>>>>>
>>>>>
>>>>> It is in: /usr/local/lib64/bpf/xsk_def_xdp_prog.o
>>>>>
>>>>> I had to compile libxdp from source because OS package version was old
>>>>> to work with af_xdp.
>>>>> Is something required to point location of this file to af_xdp PMD?
>>>>>
>>>>> I run kernel:
>>>>> 5.15.16-200.fc35.x86_64
>>>>
>>>> I read through the libxdp code to figure out what happens when
>> searching
>>> for the file:
>>>> https://github.com/xdp-project/xdp-
>>> tools/blob/v1.2.2/lib/libxdp/libxdp.c#L1055
>>>>
>>>> secure_getenv(XDP_OBJECT_ENVVAR) is called which according to the
>>> README "defaults to /usr/lib/bpf (or /usr/lib64/bpf on systems using a split
>>> library path)".
>>>> If that fails, BPF_OBJECT_PATH will be searched, which points to
>>> /usr/lib/bpf
>>>>
>>>> I discovered that on my system the getenv() call fails, but the file is
>>> eventually found because luckily BPF_OBJECT_PATH points to the
>>> appropriate place for me (lib):
>>>> https://github.com/xdp-project/xdp-tools/blob/v1.2.2/lib/util/util.h#L24
>>>> I suspect the same failure is happening for you, but since
>>> BPF_OBJECT_PATH points to lib and not lib64, the file is not found.
>>>> As a temporary measure can you create a symlink in /usr/local/lib/bpf/ to
>>> point to /usr/local/lib/bpf/xsk_def_xdp_prog.o
>>>> I will investigate the libxdp issue further. Maybe a change is needed in
>> the
>>> library. If a change or setup recommendation is needed in DPDK I will
>> create a
>>> patch.
>>>>
>>>
>>>
>>> I don't have XDP_OBJECT_ENVVAR or BPF_OBJECT_PATH environment
>>> variables set,
>>> if they should be we should document them.
>>>
>>> When I created '/usr/local/lib/bpf/' link, the BPF file found.
>>> This should be clarified/documented for users.
>>
>> Ok. Ideally we shouldn't have to create the symlink. I will look for a better
>> solution and submit a patch.
>> The symlink might be a temporary solution if another solution is not found.
> 
> Can you please try setting the environment variable LIBXDP_OBJECT_PATH=/usr/local/lib64/bpf/
> And see if your test works without the symlink?
> This worked for me and the getenv succeeded.
> If it works for you too, I'll create a patch for the docs instructing users to do the same.
> 

I confirm it works, and +1 to document it.


btw, when this environment variable is not set (and no symlink), af_xdp fails
and testpmd crashes. I think af_xdp failure shouldn't cause a crash in testpmd,
most probably some error checks are needed in the af_xdp driver.

> Thanks,
> Ciara
> 
>>
>>>
>>>
>>> And still observing following two:
>>>
>>> 1) I don't know what following log means:
>>> Configuring Port 2 (socket 0)
>>> libbpf: elf: skipping unrecognized data section(7) .xdp_run_config
>>> libbpf: elf: skipping unrecognized data section(8) xdp_metadata
>>> libxdp: XDP flag not supported by libxdp.
>>> libbpf: elf: skipping unrecognized data section(8) xdp_metadata
>>> libbpf: elf: skipping unrecognized data section(8) xdp_metadata
>>
>> I reported this and a patch was submitted to libbpf to demote those logs:
>> https://www.spinics.net/lists/bpf/msg49140.html
>> It looks like the patch never made it. I'll chase it up.
>> Anyway, the logs can be ignored as they are not errors.
>>
>>>
>>> 2) When I try to create two af_xdp interface, I only got one:
>>> "--vdev net_af_xdp,iface=enp24s0f1 --vdev net_af_xdp,iface=enp24s0f0"
>>
>> This is also expected as you haven't given each vdev a unique name. Try:
>> "--vdev net_af_xdp0,iface=enp24s0f1 --vdev net_af_xdp1,iface=enp24s0f0"
>>
>> Thank you for the testing.
>>
>> Ciara
>>
>>>
>>>
>>> Thanks,
>>> ferruh