From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 85B0B4239B for ; Wed, 11 Jan 2023 19:54:44 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 16ACD40FAE; Wed, 11 Jan 2023 19:54:44 +0100 (CET) Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by mails.dpdk.org (Postfix) with ESMTP id 7694040A7D for ; Wed, 11 Jan 2023 19:54:42 +0100 (CET) Received: by mail-pj1-f50.google.com with SMTP id o13so13287375pjg.2 for ; Wed, 11 Jan 2023 10:54:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=Gw805ENlIUOi4tKEDo+1aOtRak5P+g8Di2QR9y3nmqg=; b=pzGKcwSmEwWl9cdChXzGCYrgSTC38tUK2s2yordsGTPOzL/LGa0gZ8UFL8BHZdzyQk HeIRyqTva6DgYDhFXA3Mv9jluELZbrxsmcTCfKcSQjZ2WtR6KUcHInG+FfZnb5qduEuI PFEVFMsoi97A6iNHnEblpd3sucGNJdZ8mt1qEVza7NgDnn+z+84TWyLV8OLPBqiZEWC3 ebM4kcpytq7hne894WhbQSQ8pB/UwtgjmlvNZrOeXgy49hh5F3onEEzFobtnTCNZtQ5f uTnF11pNUIcRr/tEDbmP815p1ib60WMW3I1ELR0qz8MzxPP9WvAedYIZfsF5O0f5BwY9 Ks4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Gw805ENlIUOi4tKEDo+1aOtRak5P+g8Di2QR9y3nmqg=; b=djl8J1HG8iPSBWrX1qllLN6LsmWSvZDXs7bY3R7A5mmvr6Y0aNEKkheKhjQDt0rROc 3bQ7wykTPodcxgF6dPIjZW7Hl/l/wy3AznV2ZQ0rfq3s92YjKIb+Hax3+XMJnwBWrgFC 0NxsktwAjgVhBPWMFjC1GxQS0KJd214voB/gMd3TwSchloau3gMSFxDBxR+kLesi3ZMR jb3oXuJDkBblPQyCC5CIMtgVRWbjTicYW6puAW3wKBlY3kJ+DAIK8RsJNFWquZbQRvmm QQS4bs4TN/h1OjLaAD03Rat4HgRbhXqUt85F3KQTg0K4DXzLrgNfY4qDqkeGYHsoDsQd 9a6w== X-Gm-Message-State: AFqh2kop971Ztp6d5GhgVHSBsDsgaiIYvjUtTEWpSsK2IR3KTNUOaN77 4uR2VOgeyHkc2vkRiHDoYNysBw== X-Google-Smtp-Source: AMrXdXsm97WrP3/sd/oGOcNusnD/SK+A53sbjWcpRYj4GLXCntqxUXm0dUQRH/GvcjeojOYS8NYBtg== X-Received: by 2002:a17:90b:1bc7:b0:226:f951:1068 with SMTP id oa7-20020a17090b1bc700b00226f9511068mr14761021pjb.44.1673463281373; Wed, 11 Jan 2023 10:54:41 -0800 (PST) Received: from hermes.local (204-195-120-218.wavecable.com. [204.195.120.218]) by smtp.gmail.com with ESMTPSA id i5-20020a17090a64c500b0022721df27e9sm4978830pjm.11.2023.01.11.10.54.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Jan 2023 10:54:41 -0800 (PST) Date: Wed, 11 Jan 2023 10:54:38 -0800 From: Stephen Hemminger To: fwefew 4t4tg <7532yahoo@gmail.com> Cc: Dmitry Kozlyuk , users@dpdk.org Subject: Re: DPDK and DMA Message-ID: <20230111105438.40f199ad@hermes.local> In-Reply-To: References: <20230111142600.221cdc2e@sovereign> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On Wed, 11 Jan 2023 13:05:07 -0500 fwefew 4t4tg <7532yahoo@gmail.com> wrote: > Thank you for taking time to provide a nice reply. The upshot here is that > DPDK > already uses DMA in a smart way to move packet data into TXQs. I presume the > reverse also happens: NIC uses DMA to move packets out of its HW RXQs into > the host machine's memory using the mempool associated with it. > > > > On Wed, Jan 11, 2023 at 6:26 AM Dmitry Kozlyuk > wrote: > > > 2023-01-08 16:05 (UTC-0500), fwefew 4t4tg: > > > Consider a valid DPDK TXQ with its mempool of rte_mbufs. Application code > > > will allocate a mbuf from the pool and prepare it with headers, data, and > > > so on. > > > > > > When the mbuf(s) are enqueued to the NIC with rte_eth_tx_burst() does > > DPDK > > > DMA the memory into the NIC? Is this an optimization worth considering? > > > > DPDK is SW running on CPU. > > DMA is a way for HW to access RAM bypassing CPU (thus it is "direct"). > > > > What happens in rte_eth_tx_burst(): > > DPDK fills the packet descriptor and requests the NIC to send the packet. > > The NIC subsequently and asynchronously uses DMA to read the packet data. > > > > Regarding optimizations: > > 1. Even if the NIC has some internal buffer where it stores packet data > > before sending it to the wire, those buffers are not usually exposed. > > 2. If the NIC has on-board memory to store packet data, > > this would be implemented by a mempool driver working with such memory. > > > > > DPDK provides a DMA example here: > > > http://doc.dpdk.org/api/examples_2dma_2dmafwd_8c-example.html > > > > > > Now, to be fair, ultimately whether or not DMA helps must be evidenced > > by a > > > benchmark. Still, is there any serious reason to make mempools and its > > > bufs DMA into and out of the NIC? > > > > DMA devices in DPDK allow the CPU to initiate an operation on RAM > > that will be performed asynchronously by some special HW. > > For example, instead of memset() DPDK can tell DMA device > > to zero a memory block and avoid spending CPU cycles > > (but CPU will need to ensure zeroing completion later). > > The setup and DMA is done in the hardware specific poll mode driver (PMD). There can be some drivers that can't do direct DMA and need to copy data but these are the exception and mostly for virtual devices.