dpdk-testpmd on XDP

dpdk-testpmd on XDP

[Alessio Igor Bogani]

Hi all,

I have a machine that works without a flow using DPDK 22.11.4:
# dpdk-devbind.py --bind=vfio-pci 0000:15:00.0
# dpdk-devbind.py --bind=vfio-pci 0000:15:00.3
# dpdk-testpmd -a 0000:15:00.0 -a 0000:15:00.3 -- --tx-first

The dpdk-testpmd utility works as expected:
+++++++++++++++ Accumulated forward statistics for all ports++++++
RX-packets: 29666849       RX-dropped: 0             RX-total: 29666849
TX-packets: 29666849       TX-dropped: 0             TX-total: 29666849
++++++++++++++++++++++++++++++++++++++++++++++++++++++

I would like to test the same machine with XDP (libbpf 0.7.0 and
libxdp 1.2.8) on the same NICs:
# reboot
# export LIBXDP_OBJECT_PATH=/root/libxdp (*)
# ulimit -l unlimited
# dpdk-testpmd --vdev net_af_xdp0,iface=enp21s0f0 --vdev
net_af_xdp1,iface=enp21s0f3 -- --tx-first

In this case dpdk-testpmd doesn't work:
+++++++++++++++ Accumulated forward statistics for all ports++++
RX-packets: 0              RX-dropped: 0             RX-total: 0
TX-packets: 64             TX-dropped: 0             TX-total: 64
++++++++++++++++++++++++++++++++++++++++++++++++++++

The only suspicious part in the output of the dpdk-testpmd utility is:
[...]
libxdp: XDP flag not supported by libxdp.
libbpf: prog 'xdp_dispatcher': BPF program load failed: Invalid argument
libbpf: prog 'xdp_dispatcher': -- BEGIN PROG LOAD LOG --
Validating prog0() func#1...
btf_vmlinux is malformed
Arg#0 type PTR in prog0() is not supported yet.
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0
peak_states 0 mark_read 0
-- END PROG LOAD LOG --
libbpf: failed to load program 'xdp_dispatcher'
libbpf: failed to load object 'xdp-dispatcher.o'
libxdp: Failed to load dispatcher: Invalid argument
libxdp: Falling back to loading single prog without dispatcher
[...]

I made a mistake for sure but I haven't understood what yet. Do you
have any suggestions for me, please?

Thank you very much!

Ciao,
Alessio

(*) This doesn't seem to have any effect
I copied all file that I thought relevant in /root/libxdp: Makefile,
compat.h, libxdp.c, libxdp.pc, protocol.org, xdp-dispatcher.c,
xdp-dispatcher.ll, xsk_def_xdp_prog.c, xsk_def_xdp_prog.ll,
xsk_def_xdp_prog_5.3.embed.o, README.org, libxdp.3, libxdp.map,
libxdp.pc.template, staticobjs, xdp-dispatcher.c.in, xdp-dispatcher.o,
xsk_def_xdp_prog.embed.o, xsk_def_xdp_prog.o, xsk_def_xdp_prog_5.3.ll,
bpf_instr.h, libxdp.a, libxdp.mk, libxdp_internal.h, tests,
xdp-dispatcher.embed.o, xsk.c, xsk_def_xdp_prog.h,
xsk_def_xdp_prog_5.3.c and xsk_def_xdp_prog_5.3.o.

Re: dpdk-testpmd on XDP

[Stephen Hemminger]

On Tue, 19 Mar 2024 11:48:53 +0100
Alessio Igor Bogani <alessio.bogani@elettra.eu> wrote:

> The only suspicious part in the output of the dpdk-testpmd utility is:
> [...]
> libxdp: XDP flag not supported by libxdp.
> libbpf: prog 'xdp_dispatcher': BPF program load failed: Invalid argument
> libbpf: prog 'xdp_dispatcher': -- BEGIN PROG LOAD LOG --
> Validating prog0() func#1...
> btf_vmlinux is malformed
> Arg#0 type PTR in prog0() is not supported yet.
> processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0
> peak_states 0 mark_read 0
> -- END PROG LOAD LOG --
> libbpf: failed to load program 'xdp_dispatcher'
> libbpf: failed to load object 'xdp-dispatcher.o'
> libxdp: Failed to load dispatcher: Invalid argument
> libxdp: Falling back to loading single prog without dispatcher
> [...]

What distribution and kernel version?
BPF/XDP has changed a lot over last couple of years and not maintained
compatibility.  If you are building your own kernel, likely need to build
the xdp library as well.  If getting from a distro (Fedora, Ubuntu, Debian, etc)
then make sure that xdp and kernel match.

Re: dpdk-testpmd on XDP

[Alessio Igor Bogani]

Stephen,

Thanks for your reply!

On Tue, 19 Mar 2024 at 17:07, Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Tue, 19 Mar 2024 11:48:53 +0100
> Alessio Igor Bogani <alessio.bogani@elettra.eu> wrote:
>
> > The only suspicious part in the output of the dpdk-testpmd utility is:
> > [...]
> > libxdp: XDP flag not supported by libxdp.
> > libbpf: prog 'xdp_dispatcher': BPF program load failed: Invalid argument
> > libbpf: prog 'xdp_dispatcher': -- BEGIN PROG LOAD LOG --
> > Validating prog0() func#1...
> > btf_vmlinux is malformed
> > Arg#0 type PTR in prog0() is not supported yet.
> > processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0
> > peak_states 0 mark_read 0
> > -- END PROG LOAD LOG --
> > libbpf: failed to load program 'xdp_dispatcher'
> > libbpf: failed to load object 'xdp-dispatcher.o'
> > libxdp: Failed to load dispatcher: Invalid argument
> > libxdp: Falling back to loading single prog without dispatcher
> > [...]
>
> What distribution and kernel version?

Custom distribution (Yocto Kirkstone) using 5.10.184.All parts
(kernel, DPDK, libbpf, xdp-tools) are built (cross-compiled) from
source.

> BPF/XDP has changed a lot over last couple of years and not maintained
> compatibility.  If you are building your own kernel, likely need to build
> the xdp library as well.  If getting from a distro (Fedora, Ubuntu, Debian, etc)

Using libbpf 0.7.0 and libxdp 1.2.8, which are provided by meta-intel,
it doesn't work.  I tried the latest version libbpf 1.3.0 and libxdp
1.4.2 and unfortunately It still doesn't work
The output changed a little:
libbpf: prog 'xdp_dispatcher': failed to load: -22
libbpf: failed to load object 'xdp-dispatcher.o'
libxdp: Failed to load dispatcher: Invalid argument
libxdp: Falling back to loading single prog without dispatcher
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libbpf: prog 'xdp_dispatcher': BPF program load failed: Invalid argument
libbpf: prog 'xdp_dispatcher': -- BEGIN PROG LOAD LOG --
Validating prog0() func#1...
btf_vmlinux is malformed
Arg#0 type PTR in prog0() is not supported yet.

> then make sure that xdp and kernel match.

Where can I find that match table between xdp and kernel?

Thanks a lot!

Ciao,
Alessio

Re: dpdk-testpmd on XDP

[Stephen Hemminger]

On Wed, 20 Mar 2024 08:22:05 +0100
Alessio Igor Bogani <alessio.bogani@elettra.eu> wrote:

> Stephen,
> 
> Thanks for your reply!
> 
> On Tue, 19 Mar 2024 at 17:07, Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> >
> > On Tue, 19 Mar 2024 11:48:53 +0100
> > Alessio Igor Bogani <alessio.bogani@elettra.eu> wrote:
> >  
> > > The only suspicious part in the output of the dpdk-testpmd utility is:
> > > [...]
> > > libxdp: XDP flag not supported by libxdp.
> > > libbpf: prog 'xdp_dispatcher': BPF program load failed: Invalid argument
> > > libbpf: prog 'xdp_dispatcher': -- BEGIN PROG LOAD LOG --
> > > Validating prog0() func#1...
> > > btf_vmlinux is malformed
> > > Arg#0 type PTR in prog0() is not supported yet.
> > > processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0
> > > peak_states 0 mark_read 0
> > > -- END PROG LOAD LOG --
> > > libbpf: failed to load program 'xdp_dispatcher'
> > > libbpf: failed to load object 'xdp-dispatcher.o'
> > > libxdp: Failed to load dispatcher: Invalid argument
> > > libxdp: Falling back to loading single prog without dispatcher
> > > [...]  
> >
> > What distribution and kernel version?  
> 
> Custom distribution (Yocto Kirkstone) using 5.10.184.All parts
> (kernel, DPDK, libbpf, xdp-tools) are built (cross-compiled) from
> source.

Then you need to build bpf tools from the same kernel directory and use
them in the DPDK build.  The problem is that DPDK build uses pkg-config
to get the XDP and BPF library versions. That probably won't work as expected
with this type of Yocto build.

Re: dpdk-testpmd on XDP

[Alessio Igor Bogani]

Hi Stephen,

Thank you for your support!

On Wed, 20 Mar 2024 at 17:24, Stephen Hemminger
<stephen@networkplumber.org> wrote:
[...]
> Then you need to build bpf tools from the same kernel directory and use
[...]

Using the right version of libbpf makes all errors disappear!
Unfortunately packets aren't flowing yet between the two interfaces
(things that happen when I use the vfio-pci approach).
Do you have other tips for me?

Thanks!

Ciao,
Alessio

Re: dpdk-testpmd on XDP

[Alessio Igor Bogani]

Hi Stephen,

Here the diff between logs:

# diff vfio.log xdp.log
7,11d6
< EAL: Using IOMMU type 1 (Type 1)
< EAL: Ignore mapping IO port bar(2)
< EAL: Probe PCI driver: net_e1000_igb (8086:1521) device:
0000:15:00.0 (socket 0)
< EAL: Ignore mapping IO port bar(2)
< EAL: Probe PCI driver: net_e1000_igb (8086:1521) device:
0000:15:00.3 (socket 0)
22,23c17,18
< Port 0 Link up at 1 Gbps FDX Autoneg
< Port 1 Link up at 1 Gbps FDX Autoneg
---
> Port 0 Link up at 10 Gbps FDX Autoneg
> Port 1 Link up at 10 Gbps FDX Autoneg
36c31
<       RX desc=512 - RX free threshold=32
---
>       RX desc=0 - RX free threshold=0
40,41c35,36
<       TX desc=512 - TX free threshold=0
<       TX threshold registers: pthresh=8 hthresh=1  wthresh=16
---
>       TX desc=0 - TX free threshold=0
>       TX threshold registers: pthresh=0 hthresh=0  wthresh=0
46c41
<       RX desc=512 - RX free threshold=32
---
>       RX desc=0 - RX free threshold=0
50,51c45,46
<       TX desc=512 - TX free threshold=0
<       TX threshold registers: pthresh=8 hthresh=1  wthresh=16
---
>       TX desc=0 - TX free threshold=0
>       TX threshold registers: pthresh=0 hthresh=0  wthresh=0

I don't know why dpdk-testpmd (using xdp) reports 10 Gbps but NICs are
1 Gbs only...

Ciao,
Alessio

Re: dpdk-testpmd on XDP

[Stephen Hemminger]

On Fri, 22 Mar 2024 18:57:54 +0100
Alessio Igor Bogani <alessio.bogani@elettra.eu> wrote:

> Hi Stephen,
> 
> Thank you for your support!
> 
> On Wed, 20 Mar 2024 at 17:24, Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> [...]
> > Then you need to build bpf tools from the same kernel directory and use  
> [...]
> 
> Using the right version of libbpf makes all errors disappear!
> Unfortunately packets aren't flowing yet between the two interfaces
> (things that happen when I use the vfio-pci approach).
> Do you have other tips for me?
> 
> Thanks!
> 
> Ciao,
> Alessio

You are going to have instrument and debug the internals of XDP.
I am not an expert there.