Failed to load eBPF byte-code on TAP device

Failed to load eBPF byte-code on TAP device

[madhukar mythri]

[-- Attachment #1: Type: text/plain, Size: 1070 bytes --]

Hi all,

On the RHEL9.2 with DPDK 22.11.1 version, DPDK primary application failed
to add RSS flow on TAP sub-device, when loading the TAP BPF byte-code
instructions.

This "struct bpf_insn l3_l4_hash_insns[]" array(from file:
drivers/net/tap/tap_bpf_insns.h) is in eBPF bytecode instructions format,
this eBPF failed to load on TAP PMD with the following error:

=====================
rss_add_actions(): Failed to load BPF section 'l3_l4' (7): Argument list
too long.
net_failsafe: Failed to create a flow on sub_device 1."
=====================
On Kernel-version:  5.15.0 #9 SMP PREEMPT
Arch: x86_64 GNU/Linux

When added some debug logs on Kernel BPF verifier code, we could see that
instruction processed were reached to 1 Million.
But, the Byte code has only 1698 instructions only. Why the Kernel BPF
verifier is processing beyond 1,698 instructions ?

The same byte-code(with DPDK-22.11.1) worked well with RHEL8.x and not
working in RHEL-9.x version.

Does anybody faced such issues ?
Please let me know how to debug such issues on Byte-code.

Thanks,
Madhukar.

[-- Attachment #2: Type: text/html, Size: 1214 bytes --]

Re: Failed to load eBPF byte-code on TAP device

[Stephen Hemminger]

On Wed, 15 Nov 2023 15:38:55 +0530
madhukar mythri <madhukar.mythri@gmail.com> wrote:

> Hi all,
> 
> On the RHEL9.2 with DPDK 22.11.1 version, DPDK primary application failed
> to add RSS flow on TAP sub-device, when loading the TAP BPF byte-code
> instructions.
> 
> This "struct bpf_insn l3_l4_hash_insns[]" array(from file:
> drivers/net/tap/tap_bpf_insns.h) is in eBPF bytecode instructions format,
> this eBPF failed to load on TAP PMD with the following error:
> 
> =====================
> rss_add_actions(): Failed to load BPF section 'l3_l4' (7): Argument list
> too long.
> net_failsafe: Failed to create a flow on sub_device 1."
> =====================
> On Kernel-version:  5.15.0 #9 SMP PREEMPT
> Arch: x86_64 GNU/Linux
> 
> When added some debug logs on Kernel BPF verifier code, we could see that
> instruction processed were reached to 1 Million.
> But, the Byte code has only 1698 instructions only. Why the Kernel BPF
> verifier is processing beyond 1,698 instructions ?
> 
> The same byte-code(with DPDK-22.11.1) worked well with RHEL8.x and not
> working in RHEL-9.x version.
> 
> Does anybody faced such issues ?
> Please let me know how to debug such issues on Byte-code.
> 
> Thanks,
> Madhukar.

Is there anything in the kernel log?

Re: Failed to load eBPF byte-code on TAP device

[madhukar mythri]

[-- Attachment #1: Type: text/plain, Size: 3096 bytes --]

Hi Stephen,

I had added some logs in the BPF verifier of Kernel code, to print the
number of instructions processed and error-code returned as follows:
============================
logs # dmesg |tail -n 20
[   76.318101] #### do_check: instructions Processed 999989 insn
[   76.318102] #### do_check: instructions Processed 999990 insn
[   76.318103] #### do_check: instructions Processed 999991 insn
[   76.318104] #### do_check: instructions Processed 999992 insn
[   76.318105] #### do_check: instructions Processed 999993 insn
[   76.318106] #### do_check: instructions Processed 999994 insn
[   76.318107] #### do_check: instructions Processed 999995 insn
[   76.318108] #### do_check: instructions Processed 999996 insn
[   76.318109] #### do_check: instructions Processed 999997 insn
[   76.318110] #### do_check: instructions Processed 999998 insn
[   76.318111] #### do_check: instructions Processed 999999 insn
[   76.318112] #### do_check: instructions Processed 1000000 insn
[   76.318113] BPF program is too large. Processed 1000001 insn
[   76.318209] ########## bpf_check:  do_check_main done..: ret: -7
[   76.318210] ########## bpf_check:  bpf_prog_offload_finalize done..:
ret: -7
[   76.318212] ########## bpf_check:  check_max_stack_depth done..: ret: -7
[   76.318212] ########## bpf_check:  fixup_call_args done..: ret: -7
[   76.318224] ########## bpf_check:  end..: ret: -7
[   76.318224] ##########  BPF  bpf_check return err: -7..:
=========================

Only these logs which I add in the Kernel-code were printed and do not see
any other Kernel-logs.

Thanks,
Madhuker.

On Wed, Nov 15, 2023 at 8:49 PM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Wed, 15 Nov 2023 15:38:55 +0530
> madhukar mythri <madhukar.mythri@gmail.com> wrote:
>
> > Hi all,
> >
> > On the RHEL9.2 with DPDK 22.11.1 version, DPDK primary application failed
> > to add RSS flow on TAP sub-device, when loading the TAP BPF byte-code
> > instructions.
> >
> > This "struct bpf_insn l3_l4_hash_insns[]" array(from file:
> > drivers/net/tap/tap_bpf_insns.h) is in eBPF bytecode instructions format,
> > this eBPF failed to load on TAP PMD with the following error:
> >
> > =====================
> > rss_add_actions(): Failed to load BPF section 'l3_l4' (7): Argument list
> > too long.
> > net_failsafe: Failed to create a flow on sub_device 1."
> > =====================
> > On Kernel-version:  5.15.0 #9 SMP PREEMPT
> > Arch: x86_64 GNU/Linux
> >
> > When added some debug logs on Kernel BPF verifier code, we could see that
> > instruction processed were reached to 1 Million.
> > But, the Byte code has only 1698 instructions only. Why the Kernel BPF
> > verifier is processing beyond 1,698 instructions ?
> >
> > The same byte-code(with DPDK-22.11.1) worked well with RHEL8.x and not
> > working in RHEL-9.x version.
> >
> > Does anybody faced such issues ?
> > Please let me know how to debug such issues on Byte-code.
> >
> > Thanks,
> > Madhukar.
>
> Is there anything in the kernel log?
>
>
>

[-- Attachment #2: Type: text/html, Size: 3820 bytes --]

Re: Failed to load eBPF byte-code on TAP device

[Stephen Hemminger]

On Thu, 16 Nov 2023 13:41:27 +0530
madhukar mythri <madhukar.mythri@gmail.com> wrote:

> Hi Stephen,
> 
> I had added some logs in the BPF verifier of Kernel code, to print the
> number of instructions processed and error-code returned as follows:
> ============================
> logs # dmesg |tail -n 20
> [   76.318101] #### do_check: instructions Processed 999989 insn
> [   76.318102] #### do_check: instructions Processed 999990 insn
> [   76.318103] #### do_check: instructions Processed 999991 insn
> [   76.318104] #### do_check: instructions Processed 999992 insn
> [   76.318105] #### do_check: instructions Processed 999993 insn
> [   76.318106] #### do_check: instructions Processed 999994 insn
> [   76.318107] #### do_check: instructions Processed 999995 insn
> [   76.318108] #### do_check: instructions Processed 999996 insn
> [   76.318109] #### do_check: instructions Processed 999997 insn
> [   76.318110] #### do_check: instructions Processed 999998 insn
> [   76.318111] #### do_check: instructions Processed 999999 insn
> [   76.318112] #### do_check: instructions Processed 1000000 insn
> [   76.318113] BPF program is too large. Processed 1000001 insn
> [   76.318209] ########## bpf_check:  do_check_main done..: ret: -7
> [   76.318210] ########## bpf_check:  bpf_prog_offload_finalize done..:
> ret: -7
> [   76.318212] ########## bpf_check:  check_max_stack_depth done..: ret: -7
> [   76.318212] ########## bpf_check:  fixup_call_args done..: ret: -7
> [   76.318224] ########## bpf_check:  end..: ret: -7
> [   76.318224] ##########  BPF  bpf_check return err: -7..:
> =========================
> 
> Only these logs which I add in the Kernel-code were printed and do not see
> any other Kernel-logs.
> 
> Thanks,
> Madhuker.
> 
> On Wed, Nov 15, 2023 at 8:49 PM Stephen Hemminger <
> stephen@networkplumber.org> wrote:  
> 
> > On Wed, 15 Nov 2023 15:38:55 +0530
> > madhukar mythri <madhukar.mythri@gmail.com> wrote:
> >  
> > > Hi all,
> > >
> > > On the RHEL9.2 with DPDK 22.11.1 version, DPDK primary application failed
> > > to add RSS flow on TAP sub-device, when loading the TAP BPF byte-code
> > > instructions.
> > >
> > > This "struct bpf_insn l3_l4_hash_insns[]" array(from file:
> > > drivers/net/tap/tap_bpf_insns.h) is in eBPF bytecode instructions format,
> > > this eBPF failed to load on TAP PMD with the following error:
> > >
> > > =====================
> > > rss_add_actions(): Failed to load BPF section 'l3_l4' (7): Argument list
> > > too long.
> > > net_failsafe: Failed to create a flow on sub_device 1."
> > > =====================
> > > On Kernel-version:  5.15.0 #9 SMP PREEMPT
> > > Arch: x86_64 GNU/Linux
> > >
> > > When added some debug logs on Kernel BPF verifier code, we could see that
> > > instruction processed were reached to 1 Million.
> > > But, the Byte code has only 1698 instructions only. Why the Kernel BPF
> > > verifier is processing beyond 1,698 instructions ?
> > >
> > > The same byte-code(with DPDK-22.11.1) worked well with RHEL8.x and not
> > > working in RHEL-9.x version.
> > >
> > > Does anybody faced such issues ?
> > > Please let me know how to debug such issues on Byte-code.
> > >
> > > Thanks,
> > > Madhukar.  
> >
> > Is there anything in the kernel log?

I suspect a kernel bug.
The kernel BPF API is not stable, and RHEL can and does modify the kernel.
Likely a Redhat bug.
Try with recent TAP fixes (in 23.11-rc3).