From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 65EC443A0E for ; Tue, 30 Jan 2024 14:58:35 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2AE11402CE; Tue, 30 Jan 2024 14:58:35 +0100 (CET) Received: from mail-qv1-f53.google.com (mail-qv1-f53.google.com [209.85.219.53]) by mails.dpdk.org (Postfix) with ESMTP id 9163540275 for ; Tue, 30 Jan 2024 14:58:34 +0100 (CET) Received: by mail-qv1-f53.google.com with SMTP id 6a1803df08f44-68c3d51ecebso22238746d6.3 for ; Tue, 30 Jan 2024 05:58:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706623114; x=1707227914; darn=dpdk.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=3T6voj7revAUWtVWymT1Hed9Aray6HuX/f/j/0KTb/Y=; b=N9Zz/vPi8ri91n+dYZDP7AUF7P8HsviH5fcpnSmRqU+9W5hCtWLSM4nuiz+QMxUsC7 lMzpvP619IitpP9i4Er7Ccr1/fLS2QJHA86MpwZAwlWB+hMLpCsUO66mkWaLe0gtnyiE hGc61B9QKDqh/R8aURmwHoUl9uH30YdBtuCsVl/EKncLgnmRxN/eA+i5OissjfQLinc6 LNCls3x9bvaPMe9OluqO94E79UM1/Lg7TO6MxrBxx/CAcuXSjFdHplarHtRKZnSKBwEp sDLRm2ElhO6LZi30dUDIOWLYz6D2zQab3ngCtwo0r3KAesXhqf2/yHLjb6ixlkO0tX0o btFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706623114; x=1707227914; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=3T6voj7revAUWtVWymT1Hed9Aray6HuX/f/j/0KTb/Y=; b=QeDdfbfAM8FA2IpLlI/Hh77bmRDdTIbeR2cO/gcX6dBkrRJS7w5Gkj2tbtePLO9Mye XP185ZexnGa6K4FFu9Ll8RQqG4ukycLuhxg7uZECa4S4N9VgT5akJxjngtjZ6y+uxqjW DymH/H0m8F4WuShCWwasHYIjz1LuAPphEHHGHnFi0IdWKBXN9Rh1UuSMnxJIgd58420a QrBZtxYDW6scrsa7B9EHjBIcckIXP0Fr8OKLmRPoedAYkxI8Qfvn/qGY7oUi4Ke2RrVW BZK5G8tkQnsI3B+fG8uWVLHBVObtmUVaYfsClao4OKf6GE4d0BCDWn6ekdbR6pWH+Lyh Hxbw== X-Gm-Message-State: AOJu0YyF4Uk7mUZVC88p0KKcM8Ss3cKT4QCL7zR/oX5yMZFCjx0ggSPk gWJ1iU0caaV3H6ctVOyJNYUK0lBwey4p5202ENEBKiJHcucTo1y8iKwfR4FqACQUKjuHZBpvlsK offH12WusLf+zqg0IM9oUFUtQMdTNMW+bLZ8= X-Google-Smtp-Source: AGHT+IHdOuAktqDvzw8LYHgob5y7cxgg+1DKwG8JymzIi9C1zx4MCvcYNZwBFVQY+XF9AKGHLyVN041jLgu2VZrjnbw= X-Received: by 2002:a05:6214:2264:b0:685:2d19:783a with SMTP id gs4-20020a056214226400b006852d19783amr10368718qvb.86.1706623113774; Tue, 30 Jan 2024 05:58:33 -0800 (PST) MIME-Version: 1.0 References: <20240125155304.6816355b@hermes.local> In-Reply-To: From: Pavel Vazharov Date: Tue, 30 Jan 2024 15:58:22 +0200 Message-ID: Subject: Re: Questions about running XDP sockets on top of bonding device or on the physical interfaces behind the bond To: Stephen Hemminger Cc: users Content-Type: multipart/alternative; boundary="000000000000764d7806102a281f" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --000000000000764d7806102a281f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Just for info, if somebody hits the same issue. Forcing the copy of the packets between the kernel and the user space with 'force_copy=3D1' On Fri, Jan 26, 2024 at 4:01=E2=80=AFPM Pavel Vazharov = wrote: > On Fri, Jan 26, 2024 at 1:53=E2=80=AFAM Stephen Hemminger < > stephen@networkplumber.org> wrote: > >> On Thu, 25 Jan 2024 10:48:07 +0200 >> Pavel Vazharov wrote: >> >> > Hi there, >> > >> > I'd like to ask for advice for a weird issue that I'm facing trying to >> run >> > XDP on top of a bonding device (802.3ad) (and also on the physical >> > interfaces behind the bond). >> > >> > I've a DPDK application which runs on top of XDP sockets, using the >> DPDK AF_XDP >> > driver . It was a pure >> DPDK >> > application but lately it was migrated to run on top of XDP sockets >> because >> > we need to split the traffic entering the machine between the DPDK >> > application and other "standard-Linux" applications running on the sam= e >> > machine. >> > The application works fine when running on top of a single interface >> but it >> > has problems when it runs on top of a bonding interface. It needs to b= e >> > able to run with multiple XDP sockets where each socket (or group of X= DP >> > sockets) is/are handled in a separate thread. However, the bonding >> device >> > is reported with a single queue and thus the application can't open mo= re >> > than one XDP socket for it. So I've tried binding the XDP sockets to >> the >> > queues of the physical interfaces. For example: >> > - 3 interfaces each one is set to have 8 queues >> > - I've created 3 virtual af_xdp devices each one with 8 queues i.e. in >> > summary 24 XDP sockets each bound to a separate queue (this >> functionality >> > is provided by the DPDK itself). >> > - I've run the application on 2 threads where the first thread handled >> the >> > first 12 queues (XDP sockets) and the second thread handled the next 1= 2 >> > queues (XDP socket) i.e. the first thread worked with all 8 queues fro= m >> > af_xdp device 0 and the first 4 queues from af_xdp device 1. The secon= d >> > thread worked with the next 4 queues from af_xdp device 1 and all 8 >> queues >> > from af_xdp device 2. I've also tried another distribution scheme (see >> > below). The given threads just call the receve/transmit functions >> provided >> > by the DPDK for the assigned queues. >> > - The problem is that with this scheme the network device on the other >> side >> > reports: "The member of the LACP mode Eth-Trunk interface received an >> > abnormal LACPDU, which may be caused by optical fiber misconnection". >> And >> > this error is always reported for the last device/interface in the >> bonding >> > and the bonding/LACP doesn't work. >> > - Another thing is that if I run the DPDK application on a single >> thread, >> > and the sending/receiving on all queues is handled on a single thread, >> then >> > the bonding seems to work correctly and the above error is not reporte= d. >> > - I've checked the code multiple times and I'm sure that each thread i= s >> > accessing its own group of queues/sockets. >> > - I've tried 2 different schemes of accessing but each one led to the >> same >> > issue. For example (device_idx - queue_idx), I've tried these two >> orders of >> > accessing: >> > Thread 1 Thread2 >> > (0 - 0) (1 - 4) >> > (0 - 1) (1 - 5) >> > ... (1 - 6) >> > ... (1 - 7) >> > (0 - 7) (2 - 0) >> > (1 - 0) (2 - 1) >> > (1 - 1) ... >> > (1 - 2) ... >> > (1 - 3) (2 - 7) >> > >> > Thread 1 Thread2 >> > (0 - 0) (0 - 4) >> > (1 - 0) (1 - 4) >> > (2 - 0) (2 - 4) >> > (0 - 1) (0 - 5) >> > (1 - 1) (1 - 5) >> > (2 - 1) (2 - 5) >> > ... ... >> > (0 - 3) (0 - 7) >> > (1 - 3) (1 - 7) >> > (2 - 3) (2 - 7) >> > >> > And here are my questions based on the above situation: >> > 1. I assumed that it's not possible to run multiple XDP sockets on top >> of >> > the bonding device itself and I need to "bind" the XDP sockets on the >> > physical interfaces behind the bonding device. Am I right about this o= r >> am >> > I missing something? >> > 2. Is the bonding logic (LACP management traffic) affected by the acce= ss >> > pattern of the XDP sockets? >> > 3. Is this scheme supposed to work or it's just that the design is >> wrong? I >> > mean, maybe a group of queues/sockets shouldn't be handled on a given >> > thread but only a single queue should be handled on a given applicatio= n >> > thread. It's just that the physical devices have more queues setup on >> them >> > than the number of threads in the DPDK application and thus multiple >> queues >> > need to be handled on a single application thread. >> > >> > Any ideas are appreciated! >> > >> > Regards, >> > Pavel. >> >> Look at recent discussions on netdev mailing list. >> Linux bonding device still needs more work to fully support XDP. >> > Thank you. Will do so. > Just for info, if somebody hits the same issue. Forcing the copy of the packets between the kernel and the user space with 'force_copy=3D1' fixes the issue explained above. There was another person in the netdev mailing list reporting the same for the case of bonding. And I tried it and it worked in my case too. --000000000000764d7806102a281f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Just for info, if somebody hits the = same issue.
Forcing the copy of the packets between the kernel an= d the user space with 'force_copy=3D1'

On Fri, Jan 26, 2024 at= 4:01=E2=80=AFPM Pavel Vazharov <fr= eakpv@gmail.com> wrote:
On Fri, Jan 26, 2024 at 1:53=E2=80=AFAM Stephen Hemmi= nger <st= ephen@networkplumber.org> wrote:
On Thu, 25 Jan 2024 10:48:07 +0200
Pavel Vazharov <f= reakpv@gmail.com> wrote:

> Hi there,
>
> I'd like to ask for advice for a weird issue that I'm facing t= rying to run
> XDP on top of a bonding device (802.3ad) (and also on the physical
> interfaces behind the bond).
>
> I've a DPDK application which runs on top of XDP sockets, using th= e DPDK AF_XDP
> driver <https://doc.dpdk.org/guides/nics/af_xdp.= html>. It was a pure DPDK
> application but lately it was migrated to run on top of XDP sockets be= cause
> we need to split the traffic entering the machine between the DPDK
> application and other "standard-Linux" applications running = on the same
> machine.
> The application works fine when running on top of a single interface b= ut it
> has problems when it runs on top of a bonding interface. It needs to b= e
> able to run with multiple XDP sockets where each socket (or group of X= DP
> sockets) is/are handled in a separate thread. However, the bonding dev= ice
> is reported with a single queue and thus the application can't ope= n more
> than one=C2=A0 XDP socket for it. So I've tried binding the XDP so= ckets to the
> queues of the physical interfaces. For example:
> - 3 interfaces each one is set to have 8 queues
> - I've created 3 virtual af_xdp devices each one with 8 queues i.e= . in
> summary 24 XDP sockets each bound to a separate queue (this functional= ity
> is provided by the DPDK itself).
> - I've run the application on 2 threads where the first thread han= dled the
> first 12 queues (XDP sockets) and the second thread handled the next 1= 2
> queues (XDP socket) i.e. the first thread worked with all 8 queues fro= m
> af_xdp device 0 and the first 4 queues from af_xdp device 1. The secon= d
> thread worked with the next 4 queues from af_xdp device 1 and all 8 qu= eues
> from af_xdp device 2. I've also tried another distribution scheme = (see
> below). The given threads just call the receve/transmit functions prov= ided
> by the DPDK for the assigned queues.
> - The problem is that with this scheme the network device on the other= side
> reports: "The member of the LACP mode Eth-Trunk interface receive= d an
> abnormal LACPDU, which may be caused by optical fiber misconnection&qu= ot;. And
> this error is always reported for the last device/interface in the bon= ding
> and the bonding/LACP doesn't work.
> - Another thing is that if I run the DPDK application on a single thre= ad,
> and the sending/receiving on all queues is handled on a single thread,= then
> the bonding seems to work correctly and the above error is not reporte= d.
> - I've checked the code multiple times and I'm sure that each = thread is
> accessing its own group of queues/sockets.
> - I've tried 2 different schemes of accessing but each one led to = the same
> issue. For example (device_idx - queue_idx), I've tried these two = orders of
> accessing:
> Thread 1=C2=A0 =C2=A0 =C2=A0 =C2=A0 Thread2
> (0 - 0)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(1 - 4)
> (0 - 1)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(1 - 5)
> ...=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 (1 - 6)
> ...=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 (1 - 7)
> (0 - 7)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(2 - 0)
> (1 - 0)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(2 - 1)
> (1 - 1)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0...
> (1 - 2)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0...
> (1 - 3)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(2 - 7)
>
> Thread 1=C2=A0 =C2=A0 =C2=A0 =C2=A0 Thread2
> (0 - 0)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(0 - 4)
> (1 - 0)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(1 - 4)
> (2 - 0)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(2 - 4)
> (0 - 1)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(0 - 5)
> (1 - 1)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(1 - 5)
> (2 - 1)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(2 - 5)
> ...=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 ...
> (0 - 3)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(0 - 7)
> (1 - 3)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(1 - 7)
> (2 - 3)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(2 - 7)
>
> And here are my questions based on the above situation:
> 1. I assumed that it's not possible to run multiple XDP sockets on= top of
> the bonding device itself and I need to "bind" the XDP socke= ts on the
> physical interfaces behind the bonding device. Am I right about this o= r am
> I missing something?
> 2. Is the bonding logic (LACP management traffic) affected by the acce= ss
> pattern of the XDP sockets?
> 3. Is this scheme supposed to work or it's just that the design is= wrong? I
> mean, maybe a group of queues/sockets shouldn't be handled on a gi= ven
> thread but only a single queue should be handled on a given applicatio= n
> thread. It's just that the physical devices have more queues setup= on them
> than the number of threads in the DPDK application and thus multiple q= ueues
> need to be handled on a single application thread.
>
> Any ideas are appreciated!
>
> Regards,
> Pavel.

Look at recent discussions on netdev mailing list.
Linux bonding device still needs more work to fully support XDP.
Thank you. Will do so.
<= div id=3D"gmail-:zb" class=3D"gmail-Ar gmail-Au gmail-Ao" style=3D"display:= block">
Just for in= fo, if somebody hits the same issue.
Forcing the copy of the pack= ets between the kernel and the user space with 'force_copy=3D1'
=
fixes the issue explained above.
There was another= person in the netdev mailing list reporting the same for the case of bondi= ng.
And I tried it and it worked in my case too.
=C2=A0
--000000000000764d7806102a281f--