From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 05CCCA04A2 for ; Mon, 17 Jan 2022 01:13:41 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8175140683; Mon, 17 Jan 2022 01:13:41 +0100 (CET) Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) by mails.dpdk.org (Postfix) with ESMTP id 976EC4067B for ; Mon, 17 Jan 2022 01:13:39 +0100 (CET) Received: by mail-pg1-f179.google.com with SMTP id s15so2459151pgs.1 for ; Sun, 16 Jan 2022 16:13:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20210112.gappssmtp.com; s=20210112; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5lKo7vqA0n6Dn1W4a/Lj98sntjiaifKcqaTI+PIg4hk=; b=Jz0qb6smMKbhH1LbdjKFAXz67GFL5aoTlM8SR8rcpABeAAuQylBW5KW1Rbjmf7xLMO g8YVjnf2w52Qn3VV9HBhAD0tv5JcBtdepA/P7YTMffshCTbx5IgNHL96LjgOKDcehNwB YQXe4pKrdDA7W8Cmjhu3P36LfAonPXWmNTza5dmdreNnS7Q/jt6hyp5/E3U3Nj2agB9F 39OQ+ILq7pFJjVnhH4mK/9TWAI7YtQ4neYb8edTRwzOngkYvgYQWfXutOdetjHpThz2g NCVkCYnA/M0OybfPu1m1r8VmNJ1dU5lxZVG16YqWZvK4X7ED9Rb1dHD3wfbBsQ9k48Jb fnCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5lKo7vqA0n6Dn1W4a/Lj98sntjiaifKcqaTI+PIg4hk=; b=c1qFBuMaP4kLPV3k9r1qOWGePXMmB0NI4IJIqIAaq9aDBjR0IYTwHy05xC0alqcysZ 5pu764fXBrTfk6HG7CFJpDfsq6IZ7MdxoyYejObZ0fPxehm+xbz4l/J/4B5aZ4k5Q8sL noxWpGFSpJ8GbEw0rwFOS21Hi4eyt/J3ha86Y+fCInSO3ry+GI3/h5gl3vxtjMutKT16 3eiIJ/ArD7izDjMuQhTde6B+9nctXj4mnBmipjhPmx6P4Noj0Fp/EIMGvSw9uq+SGgOI C9WaX03+qNQ2gdVD4R558tvamek39YWiV+UyNuw5f6HiMHXhImMNKSC5NFGlvMbpYuwf J69g== X-Gm-Message-State: AOAM533J4cQQXUD0rL03Q4RSTkbBlBTJrxaJ87N49gBqI2Oao/wHq0h3 MUz0HcOHfWW4rGdSIwDw4qrTUg== X-Google-Smtp-Source: ABdhPJxIzz+2gij/9/othfGYwjGb6jHrrJBavFVrlnQ+TfOMOBPE18T5unpjaLA+Dkc6N8CPbkLOjQ== X-Received: by 2002:a62:1483:0:b0:4bc:bdd5:f30 with SMTP id 125-20020a621483000000b004bcbdd50f30mr18679707pfu.17.1642378418338; Sun, 16 Jan 2022 16:13:38 -0800 (PST) Received: from hermes.local (204-195-112-199.wavecable.com. [204.195.112.199]) by smtp.gmail.com with ESMTPSA id a21sm11542161pfl.209.2022.01.16.16.13.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Jan 2022 16:13:37 -0800 (PST) Date: Sun, 16 Jan 2022 16:13:35 -0800 From: Stephen Hemminger To: fwefew 4t4tg <7532yahoo@gmail.com> Cc: users@dpdk.org Subject: Re: Question: DPDK thread model Message-ID: <20220116161335.2313ea9e@hermes.local> In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On Sun, 16 Jan 2022 16:27:11 -0500 fwefew 4t4tg <7532yahoo@gmail.com> wrote: > I completed code to initialize an AWS ENA adapter with RX, TX queues. With > this work in hand, DPDK creates one thread pinned to the right core as per > the --lcores argument. So far so good. > The DPDK documentation and example code is fairly clear here. Look at examples/l3fwd > > What's not as clear is how RX packets are handled. As far as I can tell the > canonical way to deal with RX packets is running 'rte_eth_add_rx_callback' > for each RXQ. This allows one to process each received packet (for a given > RXQ) via a provided callback in the same lcore/hardware-thread that DPDK > created for me. As such, there is no need to create additional threads. > Correct? rte_eth_add_rx_callback is not the droid you are looking for! Applications use rte_eth_rx_burst to get packets. Most applications use RSS and/or multiple queues so that each thread reads one or more receive queues. > Furthermore, I hope the mbufs the callback gets somehow correspond to mbufs > associated with the RX descriptors provided to the RXQs so there's no need > for copying packets after the NIC receives them before the callback acts on > it. As far as I can this hope is ill-founded.. A lot of DPDK code I've seen > allocates more mbufs per RXQ than the number of RX descriptors. To me this > seems to imply DPDK's RXQ threads put copies of the > received-off-the-wire-packets into a copy for delivery to app code. You are digging in the wrong place. > TX is less clear to me. > > For TX there seems to be no way to transmit packets (burst or otherwise) > without creating another thread --- that is, another thread beyond what > DPDK makes for me. This other thread must at the appropriate time > prepare mbufs and call 'rte_eth_tx_burst' on the correct TXQ. DPDK seems to > want to keep its thread for it's own work. Yes, DPDK provides > 'rte_eth_add_tx_callback' but that only works after the mbufs have been > created and told to transmit, which is after the fact of creation. Putting > this together, DPDK requires me to create new threads unlike RX. Correct? > > While creating additional threads for TX is not the end of the world, I do > not want the DPDK TX thread to copy mbufs; I want zero-copy. Here, then, I > gather DPDK's TXQ thread takes the mbufs the helper TX thread provides in > the 'rte_eth_tx_burst' call and provides them to the TXQS descriptors so > they go out on the wire without copying. Is this correct? > > Now, it's worth pointing out here that 'rte_eth_tx_queue_setup' unlike the > RX equivalent does not accept a mempool. So in addition to the above > points, those additional TX helper threads (those which call > rte_eth_tx_burst) will need to arrange for its own mempool. That's not hard > to do, but I just want confirmation. The common way to handle Transmit is to have one transmit queue per thread. You don't want to be creating your own threads; those threads would not end up being part of DPDK and it that creates lots of issues.