From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D4098A00C2 for ; Tue, 8 Mar 2022 09:13:02 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C435C4068B; Tue, 8 Mar 2022 09:13:02 +0100 (CET) Received: from mail-yw1-f176.google.com (mail-yw1-f176.google.com [209.85.128.176]) by mails.dpdk.org (Postfix) with ESMTP id 52FA140041 for ; Tue, 8 Mar 2022 09:13:01 +0100 (CET) Received: by mail-yw1-f176.google.com with SMTP id 00721157ae682-2dbc48104beso192584497b3.5 for ; Tue, 08 Mar 2022 00:13:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sarao.ac.za; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=ZH3vKBgeIaxyB+nemqx8t/qcMg1arZMT0YV6Azteki0=; b=gbDTAFab74vcvr0V004TJp6HoKiO/bDYCxYlP5tub3JFCLxh9LsG4kol6oHC71EmO3 bYMMca5cakNu3w/YQGjMKERKgHEwJbBBDfUrlKM1fguHQbUJFjfUKvCmu211610Esmtq 7tYq9Mv3xIcrdCEAOT/vFn5ZIvg1MsnnfUEPZr6F4OvY+RZz0tDVmvsxwKcdnMXPWrdN 3Mdvde2f/tLocHKVQjZCZ0tS9BOpMvMYuGrZ5LWSbneQYjdiTLjRMiEx/nQZPuZc8y4z w4eOqzmVCEUlW8cPtCQ2VxUi81nF7O8c97nVkWLnasxWVWhJ3S5ZAPASf2OEMBsqzsvv dsXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=ZH3vKBgeIaxyB+nemqx8t/qcMg1arZMT0YV6Azteki0=; b=Ds8EOh7+ZKLPL3PbJ62KP5VcTmQSPCRCx3IfIBhPVyeT8cog6GWVkfD+XnlVE9noLi ID0HFwiOkny3dL+eEVPAbzO/1xMk6sdXB3ti1PV25tBz9MMdS9V1K4n/KwX7mw5oTt+W CR3vU1a7GRV9c52a1ZhlOiivALo8nGudsrMD6xWhQboV7fUOyzMNvy40g7zYoQ8Y/keo ooJF8P+F+b9LKeQuOfqXpU50vsYRfVbvOgjFk+KOKoJt8dOc+RiNVnYuTJlgqnddi5HI Um6wyJV5nCVAcm0EYFNlIH7cwACn9jxfil8ZikVuuTNn96zIcf5dOusN2MyVbwspccGB sMWw== X-Gm-Message-State: AOAM5337QG7HPFq+ECYn6C3pkngKiQ8q5f/zoH7CD1a//0XtSQbosk9m 0mVgG0tqlKLPkR9V6+Nav06KhsOqPUhCbRHFvYrq5dRRLTo= X-Google-Smtp-Source: ABdhPJzZLu1hkDTZ8H5HONcVk8Ty+e9M41KupUCTvEcDP1yFrQQmhdXqTtOj2yUY7TA45bmFdSNaVsIqnfGIZ60weUY= X-Received: by 2002:a81:cf02:0:b0:2d0:b68c:cf30 with SMTP id u2-20020a81cf02000000b002d0b68ccf30mr12019998ywi.510.1646727180360; Tue, 08 Mar 2022 00:13:00 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Bruce Merry Date: Tue, 8 Mar 2022 10:12:49 +0200 Message-ID: Subject: Re: Understanding extbufs for transmit To: users@dpdk.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org I can partially answer some of my own questions, for the benefit of future readers, although I definitely haven't figured it all out yet. - When using rte_pktmbuf_attach_extbuf, the original memory of the mbuf is unused. You can create a mempool where the mbufs have zero data room so as not to waste memory, or you can use the space for book-keeping structures (rte_pktmbuf_pool_create_extbuf appears to use it to hold a rte_mbuf_ext_shared_info structure). - You need to use both rte_extmem_register to tell DPDK about the memory in general, as well as rte_dev_dma_map to make it available for a specific device. In both cases you can specify IOVA, and it's not clear to me when you actually need to specify it to which (possible it depends on the IOVA mode and/or the device driver?) Bruce On Wed, 2 Mar 2022 at 11:46, Bruce Merry wrote: > > Hi > > I'm new to DPDK. I'm looking at adding a backend to a high-level > library (https://github.com/ska-sa/spead2) that currently has a > backend using ibverbs raw ethernet QPs for kernel bypass (on mlx5 > hardware). > > One of the features of the high-level library is that when > transmitting data, the user can point at arbitrary memory they've > allocated and get zero copy, provided they pre-register memory regions > first. The ibverbs backend maps the pre-registration to ibv_reg_mr, > and when constructing packets it uses a segmented work request with > one segment pointing at headers (in internally-managed memory) and > other segment points at the user-provided memory for the payload. > > I'm trying to understand if I'll be able to support the same thing in > DPDK, given that it seems to be geared to allocate memory for packets > from mempools. It looks like mbuf's can be chained to construct a > packet from non-contiguous data, and I see there are some functions > like rte_pktmbuf_attach_extbuf that look promising, but I haven't yet > found any high-level documentation that explains how to use it. For > example > > - What happens to the original memory of the mbuf - is it just wasted > memory? If so, should I be creating a mempool with small mbufs (just > enough to hold packet headers)? > - How do I pin the memory? Is rte_extmem_register the equivalent of > ibv_reg_mr? It's not clear what "registering" memory with DPDK > actually does. > - How should I compute the iova? (in ibverbs, the driver is normally > responsible for dealing with any address mapping) > > Thanks > Bruce > -- > Bruce Merry > Senior DSP Engineer > SARAO -- Bruce Merry Senior DSP Engineer SARAO