RE: [PATCH v4] net/af_xdp: re-enable secondary process support

["Loftus, Ciara" <ciara.loftus@intel.com>]

> 
> On 2/11/2022 9:26 AM, Loftus, Ciara wrote:
> >>>
> >>> On 2/10/2022 5:47 PM, Loftus, Ciara wrote:
> >>>>> Subject: Re: [PATCH v4] net/af_xdp: re-enable secondary process
> >> support
> >>>>>
> >>>>> On 2/10/2022 3:40 PM, Loftus, Ciara wrote:
> >>>>>>> Subject: Re: [PATCH v4] net/af_xdp: re-enable secondary process
> >>> support
> >>>>>>>
> >>>>>>> On 2/9/2022 9:48 AM, Ciara Loftus wrote:
> >>>>>>>> Secondary process support had been disabled for the AF_XDP
> PMD
> >>>>>>> because
> >>>>>>>> there was no logic in place to share the AF_XDP socket file
> >> descriptors
> >>>>>>>> between the processes. This commit introduces this logic using
> the
> >>> IPC
> >>>>>>>> APIs.
> >>>>>>>>
> >>>>>>>> Rx and Tx are disabled in the secondary process due to memory
> >>> mapping
> >>>>> of
> >>>>>>>> the AF_XDP rings being assigned by the kernel in the primary
> >> process
> >>>>> only.
> >>>>>>>> However other operations including retrieval of stats are
> permitted.
> >>>>>>>>
> >>>>>>>> Signed-off-by: Ciara Loftus <ciara.loftus@intel.com>
> >>>>>>>>
> >>>>>>>
> >>>>>>> Hi Ciara,
> >>>>>>>
> >>>>>>> When I tried to test the patch getting following error [1], it doesn't
> >> look
> >>>>>>> related to this patch but can you help to fix the issue, thanks.
> >>>>>>>
> >>>>>>> [1]
> >>>>>>> libxdp: Couldn't find a BPF file with name xsk_def_xdp_prog.o
> >>>>>>> xsk_configure(): Failed to create xsk socket.
> >>>>>>> eth_rx_queue_setup(): Failed to configure xdp socket
> >>>>>>> Fail to configure port 2 rx queues
> >>>>>>> EAL: Error - exiting with code: 1
> >>>>>>
> >>>>>>
> >>>>>> Hi Ferruh,
> >>>>>>
> >>>>>> This file should be generated when libxdp is compiled.
> >>>>>> Mine is located @ /usr/local/lib/bpf/xsk_def_xdp_prog.o
> >>>>>> Can you check if that file is there for you? It could be in
> >>>>> /usr/local/lib64/bpf/ on your machine.
> >>>>>> What kernel are you running on?
> >>>>>>
> >>>>>
> >>>>> It is in: /usr/local/lib64/bpf/xsk_def_xdp_prog.o
> >>>>>
> >>>>> I had to compile libxdp from source because OS package version was
> old
> >>>>> to work with af_xdp.
> >>>>> Is something required to point location of this file to af_xdp PMD?
> >>>>>
> >>>>> I run kernel:
> >>>>> 5.15.16-200.fc35.x86_64
> >>>>
> >>>> I read through the libxdp code to figure out what happens when
> >> searching
> >>> for the file:
> >>>> https://github.com/xdp-project/xdp-
> >>> tools/blob/v1.2.2/lib/libxdp/libxdp.c#L1055
> >>>>
> >>>> secure_getenv(XDP_OBJECT_ENVVAR) is called which according to the
> >>> README "defaults to /usr/lib/bpf (or /usr/lib64/bpf on systems using a
> split
> >>> library path)".
> >>>> If that fails, BPF_OBJECT_PATH will be searched, which points to
> >>> /usr/lib/bpf
> >>>>
> >>>> I discovered that on my system the getenv() call fails, but the file is
> >>> eventually found because luckily BPF_OBJECT_PATH points to the
> >>> appropriate place for me (lib):
> >>>> https://github.com/xdp-project/xdp-
> tools/blob/v1.2.2/lib/util/util.h#L24
> >>>> I suspect the same failure is happening for you, but since
> >>> BPF_OBJECT_PATH points to lib and not lib64, the file is not found.
> >>>> As a temporary measure can you create a symlink in /usr/local/lib/bpf/
> to
> >>> point to /usr/local/lib/bpf/xsk_def_xdp_prog.o
> >>>> I will investigate the libxdp issue further. Maybe a change is needed in
> >> the
> >>> library. If a change or setup recommendation is needed in DPDK I will
> >> create a
> >>> patch.
> >>>>
> >>>
> >>>
> >>> I don't have XDP_OBJECT_ENVVAR or BPF_OBJECT_PATH environment
> >>> variables set,
> >>> if they should be we should document them.
> >>>
> >>> When I created '/usr/local/lib/bpf/' link, the BPF file found.
> >>> This should be clarified/documented for users.
> >>
> >> Ok. Ideally we shouldn't have to create the symlink. I will look for a better
> >> solution and submit a patch.
> >> The symlink might be a temporary solution if another solution is not
> found.
> >
> > Can you please try setting the environment variable
> LIBXDP_OBJECT_PATH=/usr/local/lib64/bpf/
> > And see if your test works without the symlink?
> > This worked for me and the getenv succeeded.
> > If it works for you too, I'll create a patch for the docs instructing users to do
> the same.
> >
> 
> I confirm it works, and +1 to document it.
> 
> 
> btw, when this environment variable is not set (and no symlink), af_xdp fails
> and testpmd crashes. I think af_xdp failure shouldn't cause a crash in
> testpmd,
> most probably some error checks are needed in the af_xdp driver.

When I trigger the error case in my environment I get a graceful exit:

libxdp: Couldn't find a BPF file with name xsk_def_xdp_prog.o
xsk_configure(): Failed to create xsk socket.
eth_rx_queue_setup(): Failed to configure xdp socket
Fail to configure port 0 rx queues
EAL: Error - exiting with code: 1
  Cause: Start ports failed

Can you please provide more info on your crash?

> 
> > Thanks,
> > Ciara
> >
> >>
> >>>
> >>>
> >>> And still observing following two:
> >>>
> >>> 1) I don't know what following log means:
> >>> Configuring Port 2 (socket 0)
> >>> libbpf: elf: skipping unrecognized data section(7) .xdp_run_config
> >>> libbpf: elf: skipping unrecognized data section(8) xdp_metadata
> >>> libxdp: XDP flag not supported by libxdp.
> >>> libbpf: elf: skipping unrecognized data section(8) xdp_metadata
> >>> libbpf: elf: skipping unrecognized data section(8) xdp_metadata
> >>
> >> I reported this and a patch was submitted to libbpf to demote those logs:
> >> https://www.spinics.net/lists/bpf/msg49140.html
> >> It looks like the patch never made it. I'll chase it up.
> >> Anyway, the logs can be ignored as they are not errors.
> >>
> >>>
> >>> 2) When I try to create two af_xdp interface, I only got one:
> >>> "--vdev net_af_xdp,iface=enp24s0f1 --vdev
> net_af_xdp,iface=enp24s0f0"
> >>
> >> This is also expected as you haven't given each vdev a unique name. Try:
> >> "--vdev net_af_xdp0,iface=enp24s0f1 --vdev
> net_af_xdp1,iface=enp24s0f0"
> >>
> >> Thank you for the testing.
> >>
> >> Ciara
> >>
> >>>
> >>>
> >>> Thanks,
> >>> ferruh