* Failed to load eBPF byte-code on TAP device
@ 2023-11-15 10:08 madhukar mythri
2023-11-15 15:19 ` Stephen Hemminger
0 siblings, 1 reply; 4+ messages in thread
From: madhukar mythri @ 2023-11-15 10:08 UTC (permalink / raw)
To: users
[-- Attachment #1: Type: text/plain, Size: 1070 bytes --]
Hi all,
On the RHEL9.2 with DPDK 22.11.1 version, DPDK primary application failed
to add RSS flow on TAP sub-device, when loading the TAP BPF byte-code
instructions.
This "struct bpf_insn l3_l4_hash_insns[]" array(from file:
drivers/net/tap/tap_bpf_insns.h) is in eBPF bytecode instructions format,
this eBPF failed to load on TAP PMD with the following error:
=====================
rss_add_actions(): Failed to load BPF section 'l3_l4' (7): Argument list
too long.
net_failsafe: Failed to create a flow on sub_device 1."
=====================
On Kernel-version: 5.15.0 #9 SMP PREEMPT
Arch: x86_64 GNU/Linux
When added some debug logs on Kernel BPF verifier code, we could see that
instruction processed were reached to 1 Million.
But, the Byte code has only 1698 instructions only. Why the Kernel BPF
verifier is processing beyond 1,698 instructions ?
The same byte-code(with DPDK-22.11.1) worked well with RHEL8.x and not
working in RHEL-9.x version.
Does anybody faced such issues ?
Please let me know how to debug such issues on Byte-code.
Thanks,
Madhukar.
[-- Attachment #2: Type: text/html, Size: 1214 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Failed to load eBPF byte-code on TAP device
2023-11-15 10:08 Failed to load eBPF byte-code on TAP device madhukar mythri
@ 2023-11-15 15:19 ` Stephen Hemminger
2023-11-16 8:11 ` madhukar mythri
0 siblings, 1 reply; 4+ messages in thread
From: Stephen Hemminger @ 2023-11-15 15:19 UTC (permalink / raw)
To: madhukar mythri; +Cc: users
On Wed, 15 Nov 2023 15:38:55 +0530
madhukar mythri <madhukar.mythri@gmail.com> wrote:
> Hi all,
>
> On the RHEL9.2 with DPDK 22.11.1 version, DPDK primary application failed
> to add RSS flow on TAP sub-device, when loading the TAP BPF byte-code
> instructions.
>
> This "struct bpf_insn l3_l4_hash_insns[]" array(from file:
> drivers/net/tap/tap_bpf_insns.h) is in eBPF bytecode instructions format,
> this eBPF failed to load on TAP PMD with the following error:
>
> =====================
> rss_add_actions(): Failed to load BPF section 'l3_l4' (7): Argument list
> too long.
> net_failsafe: Failed to create a flow on sub_device 1."
> =====================
> On Kernel-version: 5.15.0 #9 SMP PREEMPT
> Arch: x86_64 GNU/Linux
>
> When added some debug logs on Kernel BPF verifier code, we could see that
> instruction processed were reached to 1 Million.
> But, the Byte code has only 1698 instructions only. Why the Kernel BPF
> verifier is processing beyond 1,698 instructions ?
>
> The same byte-code(with DPDK-22.11.1) worked well with RHEL8.x and not
> working in RHEL-9.x version.
>
> Does anybody faced such issues ?
> Please let me know how to debug such issues on Byte-code.
>
> Thanks,
> Madhukar.
Is there anything in the kernel log?
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Failed to load eBPF byte-code on TAP device
2023-11-15 15:19 ` Stephen Hemminger
@ 2023-11-16 8:11 ` madhukar mythri
2023-11-16 18:39 ` Stephen Hemminger
0 siblings, 1 reply; 4+ messages in thread
From: madhukar mythri @ 2023-11-16 8:11 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: users
[-- Attachment #1: Type: text/plain, Size: 3096 bytes --]
Hi Stephen,
I had added some logs in the BPF verifier of Kernel code, to print the
number of instructions processed and error-code returned as follows:
============================
logs # dmesg |tail -n 20
[ 76.318101] #### do_check: instructions Processed 999989 insn
[ 76.318102] #### do_check: instructions Processed 999990 insn
[ 76.318103] #### do_check: instructions Processed 999991 insn
[ 76.318104] #### do_check: instructions Processed 999992 insn
[ 76.318105] #### do_check: instructions Processed 999993 insn
[ 76.318106] #### do_check: instructions Processed 999994 insn
[ 76.318107] #### do_check: instructions Processed 999995 insn
[ 76.318108] #### do_check: instructions Processed 999996 insn
[ 76.318109] #### do_check: instructions Processed 999997 insn
[ 76.318110] #### do_check: instructions Processed 999998 insn
[ 76.318111] #### do_check: instructions Processed 999999 insn
[ 76.318112] #### do_check: instructions Processed 1000000 insn
[ 76.318113] BPF program is too large. Processed 1000001 insn
[ 76.318209] ########## bpf_check: do_check_main done..: ret: -7
[ 76.318210] ########## bpf_check: bpf_prog_offload_finalize done..:
ret: -7
[ 76.318212] ########## bpf_check: check_max_stack_depth done..: ret: -7
[ 76.318212] ########## bpf_check: fixup_call_args done..: ret: -7
[ 76.318224] ########## bpf_check: end..: ret: -7
[ 76.318224] ########## BPF bpf_check return err: -7..:
=========================
Only these logs which I add in the Kernel-code were printed and do not see
any other Kernel-logs.
Thanks,
Madhuker.
On Wed, Nov 15, 2023 at 8:49 PM Stephen Hemminger <
stephen@networkplumber.org> wrote:
> On Wed, 15 Nov 2023 15:38:55 +0530
> madhukar mythri <madhukar.mythri@gmail.com> wrote:
>
> > Hi all,
> >
> > On the RHEL9.2 with DPDK 22.11.1 version, DPDK primary application failed
> > to add RSS flow on TAP sub-device, when loading the TAP BPF byte-code
> > instructions.
> >
> > This "struct bpf_insn l3_l4_hash_insns[]" array(from file:
> > drivers/net/tap/tap_bpf_insns.h) is in eBPF bytecode instructions format,
> > this eBPF failed to load on TAP PMD with the following error:
> >
> > =====================
> > rss_add_actions(): Failed to load BPF section 'l3_l4' (7): Argument list
> > too long.
> > net_failsafe: Failed to create a flow on sub_device 1."
> > =====================
> > On Kernel-version: 5.15.0 #9 SMP PREEMPT
> > Arch: x86_64 GNU/Linux
> >
> > When added some debug logs on Kernel BPF verifier code, we could see that
> > instruction processed were reached to 1 Million.
> > But, the Byte code has only 1698 instructions only. Why the Kernel BPF
> > verifier is processing beyond 1,698 instructions ?
> >
> > The same byte-code(with DPDK-22.11.1) worked well with RHEL8.x and not
> > working in RHEL-9.x version.
> >
> > Does anybody faced such issues ?
> > Please let me know how to debug such issues on Byte-code.
> >
> > Thanks,
> > Madhukar.
>
> Is there anything in the kernel log?
>
>
>
[-- Attachment #2: Type: text/html, Size: 3820 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Failed to load eBPF byte-code on TAP device
2023-11-16 8:11 ` madhukar mythri
@ 2023-11-16 18:39 ` Stephen Hemminger
0 siblings, 0 replies; 4+ messages in thread
From: Stephen Hemminger @ 2023-11-16 18:39 UTC (permalink / raw)
To: madhukar mythri; +Cc: users
On Thu, 16 Nov 2023 13:41:27 +0530
madhukar mythri <madhukar.mythri@gmail.com> wrote:
> Hi Stephen,
>
> I had added some logs in the BPF verifier of Kernel code, to print the
> number of instructions processed and error-code returned as follows:
> ============================
> logs # dmesg |tail -n 20
> [ 76.318101] #### do_check: instructions Processed 999989 insn
> [ 76.318102] #### do_check: instructions Processed 999990 insn
> [ 76.318103] #### do_check: instructions Processed 999991 insn
> [ 76.318104] #### do_check: instructions Processed 999992 insn
> [ 76.318105] #### do_check: instructions Processed 999993 insn
> [ 76.318106] #### do_check: instructions Processed 999994 insn
> [ 76.318107] #### do_check: instructions Processed 999995 insn
> [ 76.318108] #### do_check: instructions Processed 999996 insn
> [ 76.318109] #### do_check: instructions Processed 999997 insn
> [ 76.318110] #### do_check: instructions Processed 999998 insn
> [ 76.318111] #### do_check: instructions Processed 999999 insn
> [ 76.318112] #### do_check: instructions Processed 1000000 insn
> [ 76.318113] BPF program is too large. Processed 1000001 insn
> [ 76.318209] ########## bpf_check: do_check_main done..: ret: -7
> [ 76.318210] ########## bpf_check: bpf_prog_offload_finalize done..:
> ret: -7
> [ 76.318212] ########## bpf_check: check_max_stack_depth done..: ret: -7
> [ 76.318212] ########## bpf_check: fixup_call_args done..: ret: -7
> [ 76.318224] ########## bpf_check: end..: ret: -7
> [ 76.318224] ########## BPF bpf_check return err: -7..:
> =========================
>
> Only these logs which I add in the Kernel-code were printed and do not see
> any other Kernel-logs.
>
> Thanks,
> Madhuker.
>
> On Wed, Nov 15, 2023 at 8:49 PM Stephen Hemminger <
> stephen@networkplumber.org> wrote:
>
> > On Wed, 15 Nov 2023 15:38:55 +0530
> > madhukar mythri <madhukar.mythri@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > On the RHEL9.2 with DPDK 22.11.1 version, DPDK primary application failed
> > > to add RSS flow on TAP sub-device, when loading the TAP BPF byte-code
> > > instructions.
> > >
> > > This "struct bpf_insn l3_l4_hash_insns[]" array(from file:
> > > drivers/net/tap/tap_bpf_insns.h) is in eBPF bytecode instructions format,
> > > this eBPF failed to load on TAP PMD with the following error:
> > >
> > > =====================
> > > rss_add_actions(): Failed to load BPF section 'l3_l4' (7): Argument list
> > > too long.
> > > net_failsafe: Failed to create a flow on sub_device 1."
> > > =====================
> > > On Kernel-version: 5.15.0 #9 SMP PREEMPT
> > > Arch: x86_64 GNU/Linux
> > >
> > > When added some debug logs on Kernel BPF verifier code, we could see that
> > > instruction processed were reached to 1 Million.
> > > But, the Byte code has only 1698 instructions only. Why the Kernel BPF
> > > verifier is processing beyond 1,698 instructions ?
> > >
> > > The same byte-code(with DPDK-22.11.1) worked well with RHEL8.x and not
> > > working in RHEL-9.x version.
> > >
> > > Does anybody faced such issues ?
> > > Please let me know how to debug such issues on Byte-code.
> > >
> > > Thanks,
> > > Madhukar.
> >
> > Is there anything in the kernel log?
I suspect a kernel bug.
The kernel BPF API is not stable, and RHEL can and does modify the kernel.
Likely a Redhat bug.
Try with recent TAP fixes (in 23.11-rc3).
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2023-11-16 18:39 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-11-15 10:08 Failed to load eBPF byte-code on TAP device madhukar mythri
2023-11-15 15:19 ` Stephen Hemminger
2023-11-16 8:11 ` madhukar mythri
2023-11-16 18:39 ` Stephen Hemminger
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).