From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BAF0DA0093 for ; Mon, 11 Apr 2022 15:45:59 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4D2F340692; Mon, 11 Apr 2022 15:45:59 +0200 (CEST) Received: from mail-vk1-f176.google.com (mail-vk1-f176.google.com [209.85.221.176]) by mails.dpdk.org (Postfix) with ESMTP id 46EF54068C for ; Mon, 11 Apr 2022 15:45:58 +0200 (CEST) Received: by mail-vk1-f176.google.com with SMTP id e2so2702991vkn.4 for ; Mon, 11 Apr 2022 06:45:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=solid-run-com.20210112.gappssmtp.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=dG3RQ7NH8cZtPlpnd1qq5g8aDLxf6GZ45fOVMjVHGfg=; b=o6K1/arzAnRpgjYbWRQCP9ZRnBp1Aol3sJHg6qYyfyGGd4z+geGkNfrkJQSDHLyTbF YSrmzmLdbVyRkdK4WSeJf0ZZNZNHvgDOSj71CRRU+ZJgMNwtfxwNr6XosgNvhXb6+bsc bXl86SxSIH20rn7kO+MJuahGgkHRrdgy5/yTRDrMW1/fLZjQROHPOzBxCGhDtqALHBDQ kxv2WcsRmoAQmvSEsnP5KH6jy8pDxH4ZvXeUBiYJxruXSaYETE/C++HXmtuzSI8oTIpS E8E6rdUgEBoZgAZ+xmqyGKrouzlCBTp+B8ndqUOZZHz1Qp2a/oPdheZ2h1vr/dWuY2cf zWGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=dG3RQ7NH8cZtPlpnd1qq5g8aDLxf6GZ45fOVMjVHGfg=; b=iUdR6T/YB/nezTiPxryn+BZSooShO41Abmz5Zv2LDyopAFn2JvC0Ybqi0SUr4aN3mC /bvtg+Wg5f3HzU5zcmHZgUzlSWs4MxJ8BR5a8xWnwDAfk3rcPpvCBPIpt/XB5rzVyuto rj9g5wVNSyis7tJCaTgGDGcjdnpMLKaDQF0PWULoAaScYrERRkCQzmr8qQ4b3MwtobPp /ZMQRPxXJ62tAKdDPaba8fANrtOdCCbPzKxPCnlsj2Pay6xo54L+lUlPpjgztQKZ2wUv wC6Bqyqd+EEEHxnGNxsdFVPUgAuyxrjNTCaedMOvgFSF90WXqXO8g9ckbQ8WiPLSfCV/ hMIQ== X-Gm-Message-State: AOAM5313wMAXl1jvcooKZXg/qsnIwWjA2GAFREE4G9VRUtbcOYTyht/j vn11YIZ1HvPWpmoP/aditeeB3J2ZbklGiMgHGi+/W8XOCxrvrw== X-Google-Smtp-Source: ABdhPJydhgGTqyXHK3CG52bEStu/WhCDYMdruZxYU02JVTPRwlsrgTx6euSmkQxohwb5jaLBeMSltnIR1OTMn5PJQTI= X-Received: by 2002:a1f:1b56:0:b0:345:4d97:a2e7 with SMTP id b83-20020a1f1b56000000b003454d97a2e7mr2873282vkb.31.1649684757256; Mon, 11 Apr 2022 06:45:57 -0700 (PDT) MIME-Version: 1.0 From: Alvaro Karsz Date: Mon, 11 Apr 2022 16:45:21 +0300 Message-ID: Subject: Poll tx_pkt_burst callback To: users@dpdk.org Content-Type: multipart/alternative; boundary="000000000000f2afc705dc61292d" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --000000000000f2afc705dc61292d Content-Type: text/plain; charset="UTF-8" Hello, I'm developing a PMD for a NIC that I'm developing at the moment. (DPDK version 20.11) The NIC should forward packets between a network device (based on dpaa2 HW) and a PCIe device. I'm running l2fwd application in order to develop the PMD, with the following command: ./dpdk-l2fwd -c 0x3 --vdev=my_pmd0 -- -p 0x3 -T 0 The application uses 2 lcores: - Lcore 0: RX port 0 TX port 1. - Lcore 1: RX port 1 TX port 0. Where port 0 is the physical one, and net_dpaa2 PMD is attached to this port. Port 1 is the virtual one (pci device), and my PMD is attached to it. The problem I'm experiencing is related to the tx_pkt_burst callback. When the tx_pkt_burst callback is called, I start the DMA operations to write the packets, but the problem is that I need to wait for the DMA operations to finish in order to do some more work. I don't want to lock the lcore in my tx_pkt_burst function waiting for the DMA operation to finish, I want to release it as soon as possible, so it will go back polling the physical port. Another issue is that I need to poll the PCI for new available write buffers. I would like to handle all the TX related software work using one core, including polling for new write buffers, starting the DMA operations once new packets arrive and polling DMA until the operation finishes, then doing some more software work. Is there a "correct" way to handle this? What I've tried: tx_pkt_burst just stores the new packets in an array, and rx_pkt_burst handles all the TX tasks (polling for new write buffers, Starts DMA operations, and polls the DMA operations status). This works, but: - The workload is not symmetric, the RX lcore does all the "heavy lifting", while the TX lcore does almost nothing. - spinlocks are needed since 2 lcores access the same data. - Less efficient caches. What I've though: - Using rte_eal_alarm_set to poll the tx_pkt_burst function, for example: uint16_t my_tx_pkt_burst(void *txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) { /* Save mbufs in an array and return number of saved mbufs */ } void tx_poll(void *param) { /* Check for new write buffers*/ /* Start DMA operations for new mbufs from my_tx_pkt_burst */ /* Check DMA status for prev. operations, and do * some more work if needed */ /* Call this function again in 1us */ rte_eal_alarm_set(1, tx_poll, param); } Best regards, Alvaro --000000000000f2afc705dc61292d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello,
I'm developing a PMD for a NIC that I'm developing at the= moment.
(DPDK version 20.11)

The NIC should forward packets between a network device (based o= n dpaa2 HW) and a PCIe device.
I'm running l2fwd applicat= ion in=C2=A0order=C2=A0to=C2=A0develop the PMD, with the following command:=

./dpdk-l2fwd -c 0x3 --vdev=3Dmy_pmd0 -- -p 0x3 -T 0=C2= =A0

The application uses 2 lcores:
  • L= core 0: RX port 0 TX port 1.
  • Lcore 1: RX port 1 TX port 0.

    • Where port 0= is the physical one, and net_dpaa2 PMD is attached to this port.

    P= ort 1 is the virtual one (pci device), and my PMD is attached to it.=


The problem I'm experiencing is related to the=C2=A0tx_pkt_= burst callback.

When the tx_pkt_burst callback is called,= I start the DMA operations to write the packets, but the problem is that I= need to wait for the DMA operations to finish in order to do some more wor= k.

I don't want to lock the lcore in my tx_pkt_burst func= tion waiting for the DMA operation to finish, I want to release it as soon= =C2=A0as possible, so it will go back polling the physical port.

Another issue is that I need to poll the PCI for new available write bu= ffers.

I would like to handle all the=C2=A0TX related softwar= e work using one core,=C2=A0including polling for new write=C2=A0buffers,= =C2=A0starting the=C2=A0DMA operations once new packets arrive and polling = DMA until the operation finishes, then doing some more software=C2=A0work.<= /font>

Is there a "correct" way to handle this?

What I've tried:
tx_pkt_= burst just stores the new packets in an array, and=C2=A0rx_pkt_burst handle= s all the TX tasks (polling for new write buffers, Starts DMA operations, a= nd polls the DMA operations status).
This works, but:
  • = The workload=C2=A0is not symmetric, the RX lcore does all the "heavy l= ifting", while the TX lcore does almost nothing.
  • spinlocks are needed since 2 lcores access the same data= .
  • Less efficient caches.
=C2=A0

What I've thoug= h:
  • Using rte_eal_alarm_set to poll the=C2=A0tx_pkt_burst function, for example:
<= pre class=3D"gmail-lang-c gmail-s-code-block" style=3D"margin-top:0px;paddi= ng:12px;border:0px;font-variant-numeric:inherit;font-variant-east-asian:inh= erit;font-stretch:inherit;line-height:1.30769;font-size:13px;vertical-align= :baseline;box-sizing:inherit;width:auto;max-height:600px;overflow:auto;bord= er-radius:5px">uint16_t my_tx_pkt_burst(void *txq, struct rte_mbuf **tx_pkt= s,
uint16_t nb_pkts)
{

=C2=A0/* Save mbufs in an array and r= eturn number of saved mbufs */
}

void tx_poll(void *param)
{=C2=A0/* Check for new write buffers*/

=C2=A0/* Start DMA operatio= ns for new mbufs from my_tx_pkt_burst */
=C2=A0
=C2=A0/* Check DMA st= atus for prev. operations, and do
=C2=A0 * some more work if needed
= =C2=A0 */
=C2=A0
=C2=A0
=C2=A0/* Call this function again in 1us= */
=C2=A0rte_eal_alarm_set(1, tx_poll, param);
}

Best regards,
Alvaro


--000000000000f2afc705dc61292d--