Re: [PATCH v4] net/af_xdp: re-enable secondary process support

[Ferruh Yigit <ferruh.yigit@intel.com>]

On 2/11/2022 1:01 PM, Loftus, Ciara wrote:
>>
>> On 2/11/2022 9:26 AM, Loftus, Ciara wrote:
>>>>>
>>>>> On 2/10/2022 5:47 PM, Loftus, Ciara wrote:
>>>>>>> Subject: Re: [PATCH v4] net/af_xdp: re-enable secondary process
>>>> support
>>>>>>>
>>>>>>> On 2/10/2022 3:40 PM, Loftus, Ciara wrote:
>>>>>>>>> Subject: Re: [PATCH v4] net/af_xdp: re-enable secondary process
>>>>> support
>>>>>>>>>
>>>>>>>>> On 2/9/2022 9:48 AM, Ciara Loftus wrote:
>>>>>>>>>> Secondary process support had been disabled for the AF_XDP
>> PMD
>>>>>>>>> because
>>>>>>>>>> there was no logic in place to share the AF_XDP socket file
>>>> descriptors
>>>>>>>>>> between the processes. This commit introduces this logic using
>> the
>>>>> IPC
>>>>>>>>>> APIs.
>>>>>>>>>>
>>>>>>>>>> Rx and Tx are disabled in the secondary process due to memory
>>>>> mapping
>>>>>>> of
>>>>>>>>>> the AF_XDP rings being assigned by the kernel in the primary
>>>> process
>>>>>>> only.
>>>>>>>>>> However other operations including retrieval of stats are
>> permitted.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Ciara Loftus <ciara.loftus@intel.com>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Ciara,
>>>>>>>>>
>>>>>>>>> When I tried to test the patch getting following error [1], it doesn't
>>>> look
>>>>>>>>> related to this patch but can you help to fix the issue, thanks.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> libxdp: Couldn't find a BPF file with name xsk_def_xdp_prog.o
>>>>>>>>> xsk_configure(): Failed to create xsk socket.
>>>>>>>>> eth_rx_queue_setup(): Failed to configure xdp socket
>>>>>>>>> Fail to configure port 2 rx queues
>>>>>>>>> EAL: Error - exiting with code: 1
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Ferruh,
>>>>>>>>
>>>>>>>> This file should be generated when libxdp is compiled.
>>>>>>>> Mine is located @ /usr/local/lib/bpf/xsk_def_xdp_prog.o
>>>>>>>> Can you check if that file is there for you? It could be in
>>>>>>> /usr/local/lib64/bpf/ on your machine.
>>>>>>>> What kernel are you running on?
>>>>>>>>
>>>>>>>
>>>>>>> It is in: /usr/local/lib64/bpf/xsk_def_xdp_prog.o
>>>>>>>
>>>>>>> I had to compile libxdp from source because OS package version was
>> old
>>>>>>> to work with af_xdp.
>>>>>>> Is something required to point location of this file to af_xdp PMD?
>>>>>>>
>>>>>>> I run kernel:
>>>>>>> 5.15.16-200.fc35.x86_64
>>>>>>
>>>>>> I read through the libxdp code to figure out what happens when
>>>> searching
>>>>> for the file:
>>>>>> https://github.com/xdp-project/xdp-
>>>>> tools/blob/v1.2.2/lib/libxdp/libxdp.c#L1055
>>>>>>
>>>>>> secure_getenv(XDP_OBJECT_ENVVAR) is called which according to the
>>>>> README "defaults to /usr/lib/bpf (or /usr/lib64/bpf on systems using a
>> split
>>>>> library path)".
>>>>>> If that fails, BPF_OBJECT_PATH will be searched, which points to
>>>>> /usr/lib/bpf
>>>>>>
>>>>>> I discovered that on my system the getenv() call fails, but the file is
>>>>> eventually found because luckily BPF_OBJECT_PATH points to the
>>>>> appropriate place for me (lib):
>>>>>> https://github.com/xdp-project/xdp-
>> tools/blob/v1.2.2/lib/util/util.h#L24
>>>>>> I suspect the same failure is happening for you, but since
>>>>> BPF_OBJECT_PATH points to lib and not lib64, the file is not found.
>>>>>> As a temporary measure can you create a symlink in /usr/local/lib/bpf/
>> to
>>>>> point to /usr/local/lib/bpf/xsk_def_xdp_prog.o
>>>>>> I will investigate the libxdp issue further. Maybe a change is needed in
>>>> the
>>>>> library. If a change or setup recommendation is needed in DPDK I will
>>>> create a
>>>>> patch.
>>>>>>
>>>>>
>>>>>
>>>>> I don't have XDP_OBJECT_ENVVAR or BPF_OBJECT_PATH environment
>>>>> variables set,
>>>>> if they should be we should document them.
>>>>>
>>>>> When I created '/usr/local/lib/bpf/' link, the BPF file found.
>>>>> This should be clarified/documented for users.
>>>>
>>>> Ok. Ideally we shouldn't have to create the symlink. I will look for a better
>>>> solution and submit a patch.
>>>> The symlink might be a temporary solution if another solution is not
>> found.
>>>
>>> Can you please try setting the environment variable
>> LIBXDP_OBJECT_PATH=/usr/local/lib64/bpf/
>>> And see if your test works without the symlink?
>>> This worked for me and the getenv succeeded.
>>> If it works for you too, I'll create a patch for the docs instructing users to do
>> the same.
>>>
>>
>> I confirm it works, and +1 to document it.
>>
>>
>> btw, when this environment variable is not set (and no symlink), af_xdp fails
>> and testpmd crashes. I think af_xdp failure shouldn't cause a crash in
>> testpmd,
>> most probably some error checks are needed in the af_xdp driver.
> 
> When I trigger the error case in my environment I get a graceful exit:
> 
> libxdp: Couldn't find a BPF file with name xsk_def_xdp_prog.o
> xsk_configure(): Failed to create xsk socket.
> eth_rx_queue_setup(): Failed to configure xdp socket
> Fail to configure port 0 rx queues
> EAL: Error - exiting with code: 1
>    Cause: Start ports failed
> 
> Can you please provide more info on your crash?
> 

$ ./build/app/dpdk-testpmd --vdev net_af_xdp0,iface=enp24s0f1 --vdev net_af_xdp1,iface=enp94s0f1 -- -i
...
Configuring Port 2 (socket 0)
libxdp: Couldn't find a BPF file with name xsk_def_xdp_prog.o
xsk_configure(): Failed to create xsk socket.
eth_rx_queue_setup(): Failed to configure xdp socket
Fail to configure port 2 rx queues
EAL: Error - exiting with code: 1
   Cause: Start ports failed
Segmentation fault (core dumped)

(I have two physical interfaces too, af_xdp ports are port 2 & 3)