From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1604AA328D for ; Tue, 22 Oct 2019 17:17:27 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B7E861BEDF; Tue, 22 Oct 2019 17:17:25 +0200 (CEST) Received: from mail-il1-f196.google.com (mail-il1-f196.google.com [209.85.166.196]) by dpdk.org (Postfix) with ESMTP id 73C4F1BEC0 for ; Tue, 22 Oct 2019 17:17:24 +0200 (CEST) Received: by mail-il1-f196.google.com with SMTP id f13so15764913ils.11 for ; Tue, 22 Oct 2019 08:17:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=t4S/MxXVpe+UD9m9xePiHg2Kt7mlyC/FnEaat5XxfI4=; b=S4uX96yfMoYElMpHGruM54D/d/aXT1d/Vhu7xReMXJAmCHSD1P+lXkTV+J958vLEsY RFXOzp1zAJ1Tmrxv7/aUj9+CIamnOQ8BuzbIWPuoIKWdT+uVBeMoktK9M5X7xSH/54Ms /Bzc0lQ/3G8JFDjdbrzWHPSgqonc8m8DvFBXIASdt+TF002oqQgITeKwZOGoVUIYPHjV oqvw22LE5Qa10ubr+2lJhqoLRFJ3i6HaLLEuSJ5g4/Dc4iPmUuV5dPzyXChgEhZElYTg jRzdHcC+a6lleC8egsgTB2pql4PGPbYPBZgJ5JKbzJQK8fGs2+0fPr3KW/Epvj6BestN gCuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=t4S/MxXVpe+UD9m9xePiHg2Kt7mlyC/FnEaat5XxfI4=; b=jMZ8v37KupsjgQjkM1mK/cORu+50Sa4Gxq9OxS+avu43oozG28Tx/5J9kTii3XSRdP KQODvsSQpr+/QtaLn9xmaQz47AVKoaQCwgVcNlKIn8ZA8A6qeBz/ttX7wbGJ7qdP/VV4 YZIcf1P9O9LZ8y11Zgiiiz5d30vP4+uruwETTpDCqR+f+IrjijgIUGYYiDo5NUeupWjT Dcj0BU/GTtcSjSngPAcQD7nEGILw4GEtLnE0BubZho8YjxlvjW8ETMIXFWHpR4vLj14a kE1FmzDK58+UKfFhCa/K2spLZb6JkZIxwNEtS8FyvQLTtGvIuhnBADe0UgI4fRYjs9p7 W04Q== X-Gm-Message-State: APjAAAU3hoXK3DvXSwlJSZrmX7fC10aCtgc43+ryrlF2bGBydG8sYQOB zcAGS3sGMMiJFESU6stwU9Aag5A5odmyia1wKtY= X-Google-Smtp-Source: APXvYqwlaHZwDMLYQmezda9SfqhYgr2Wx+HUPiatZ0IQmjM0YJE72QTKrpJGrV2+f/DfQiQRTGMGTHBuw/mbeWYH5P0= X-Received: by 2002:a92:aa48:: with SMTP id j69mr13778371ili.162.1571757443463; Tue, 22 Oct 2019 08:17:23 -0700 (PDT) MIME-Version: 1.0 References: <20191017072723.36509-1-shahafs@mellanox.com> In-Reply-To: From: Jerin Jacob Date: Tue, 22 Oct 2019 20:47:06 +0530 Message-ID: To: Shahaf Shuler Cc: "dev@dpdk.org" , Thomas Monjalon , "olivier.matz@6wind.com" , "wwasko@nvidia.com" , "spotluri@nvidia.com" , Asaf Penso , Slava Ovsiienko Content-Type: text/plain; charset="UTF-8" Subject: Re: [dpdk-dev] [RFC PATCH 20.02] mbuf: hint PMD not to inline packet X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Tue, Oct 22, 2019 at 11:56 AM Shahaf Shuler wrote: > > Thursday, October 17, 2019 8:19 PM, Jerin Jacob: > > Subject: Re: [dpdk-dev] [RFC PATCH 20.02] mbuf: hint PMD not to inline > > packet > > > > On Thu, Oct 17, 2019 at 4:30 PM Shahaf Shuler > > wrote: > > > > > > Thursday, October 17, 2019 11:17 AM, Jerin Jacob: > > > > Subject: Re: [dpdk-dev] [RFC PATCH 20.02] mbuf: hint PMD not to > > > > inline packet > > > > > > > > On Thu, Oct 17, 2019 at 12:57 PM Shahaf Shuler > > > > > > > > wrote: > > > > > > > > > > Some PMDs inline the mbuf data buffer directly to device. This is > > > > > in order to save the overhead of the PCI headers involved when the > > > > > device DMA read the buffer pointer. For some devices it is > > > > > essential in order to reach the pick BW. > > > > > > > > > > However, there are cases where such inlining is in-efficient. For > > > > > example when the data buffer resides on other device memory (like > > > > > GPU or storage device). attempt to inline such buffer will result > > > > > in high PCI overhead for reading and copying the data from the remote > > device. > > > > > > > > Some questions to understand the use case # Is this use case where > > > > CPU, local DRAM, NW card and GPU memory connected on the coherent > > > > bus > > > > > > Yes. For example one can allocate GPU memory and map it to the GPU bar, > > make it accessible from the host CPU through LD/ST. > > > > > > > # Assuming the CPU needs to touch the buffer prior to Tx, In that > > > > case, it will be useful? > > > > > > If the CPU needs to modify the data then no. it will be more efficient to > > copy the data to CPU and then send it. > > > However there are use cases where the data is DMA w/ zero copy to the > > GPU (for example) , GPU perform the processing on the data, and then CPU > > send the mbuf (w/o touching the data). > > > > OK. If I understanding it correctly it is for offloading the Network/Compute > > functions to GPU from NW card and/or CPU. > > Mostly the compute. The networking on this model is expected to be done by the CPU. > Note this is only one use case. > > > > > > > > > > # How the application knows, The data buffer is in GPU memory in > > > > order to use this flag efficiently? > > > > > > Because it made it happen. For example it attached the mbuf external > > buffer from the other device memory. > > > > > > > # Just an random thought, Does it help, if we create two different > > > > mempools one from local DRAM and one from GPU memory so that the > > > > application can work transparently. > > > > > > But you will still need to teach the PMD which pool it can inline and which > > cannot. > > > IMO it is more generic to have it per mbuf. Moreover, application has this > > info. > > > > IMO, we can not use PKT_TX_DONT_INLINE_HINT flag for generic > > applications, The application usage will be tightly coupled with the platform > > and capabilities of GPU or Host CPU etc. > > > > I think, pushing this logic to the application is bad idea. But if you are writing > > some custom application and the per packet-level you need to control then > > this flag may be the only way. > > Yes. This flag is for custom application who do unique acceleration (by doing Zero copy for compute/compression/encryption accelerators) on specific platforms. > Such application is fully aware to the platform and the location where the data resides hence it is very simple for it to know how to set this flag. # if it is per packet, it will be an implicit requirement to add it mbuf. If so, # Does it makes sense to add through dynamic mbuf? Maybe it is not worth it for a single bit. Since we have only 17 bits (40 - 23) remaining for Rx and Tx and it is custom application requirement, how about adding PKT_PMD_CUSTOM1 flags so that similar requirement by other PMDs can leverage the same bit for such custom applications.(We have a similar use case for smart NIC (not so make much sense for generic applications) but needed for per packet) > > Note, This flag is 0 by default - meaning no hint and generic application works same as today. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > To support a mixed traffic pattern (some buffers from local DRAM, > > > > > some buffers from other devices) with high BW, a hint flag is > > > > > introduced in the mbuf. > > > > > Application will hint the PMD whether or not it should try to > > > > > inline the given mbuf data buffer. PMD should do best effort to > > > > > act upon this request. > > > > > > > > > > Signed-off-by: Shahaf Shuler > > > > > --- > > > > > lib/librte_mbuf/rte_mbuf.h | 9 +++++++++ > > > > > 1 file changed, 9 insertions(+) > > > > > > > > > > diff --git a/lib/librte_mbuf/rte_mbuf.h > > > > > b/lib/librte_mbuf/rte_mbuf.h index 98225ec80b..5934532b7f 100644 > > > > > --- a/lib/librte_mbuf/rte_mbuf.h > > > > > +++ b/lib/librte_mbuf/rte_mbuf.h > > > > > @@ -203,6 +203,15 @@ extern "C" { > > > > > /* add new TX flags here */ > > > > > > > > > > /** > > > > > + * Hint to PMD to not inline the mbuf data buffer to device > > > > > + * rather let the device use its DMA engine to fetch the data > > > > > +with the > > > > > + * provided pointer. > > > > > + * > > > > > + * This flag is a only a hint. PMD should enforce it as best effort. > > > > > + */ > > > > > +#define PKT_TX_DONT_INLINE_HINT (1ULL << 39) > > > > > + > > > > > +/** > > > > > * Indicate that the metadata field in the mbuf is in use. > > > > > */ > > > > > #define PKT_TX_METADATA (1ULL << 40) > > > > > -- > > > > > 2.12.0 > > > > >