From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 906BEA0C47; Sun, 31 Oct 2021 15:01:42 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 151DF4068E; Sun, 31 Oct 2021 15:01:42 +0100 (CET) Received: from mail-io1-f41.google.com (mail-io1-f41.google.com [209.85.166.41]) by mails.dpdk.org (Postfix) with ESMTP id 988DA40689 for ; Sun, 31 Oct 2021 15:01:40 +0100 (CET) Received: by mail-io1-f41.google.com with SMTP id p204so6902353iod.8 for ; Sun, 31 Oct 2021 07:01:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=5ck9u6Cqqhofl8aUp5xJFrmUenl65A498VLtQCZZhO0=; b=AP2sIo2pFuRUdamHMaD0Vyz2hDbOXKYa7xQllXdRAg2iS2kXVxddAJFAOX/zTfG+YM L4uHTSVwmtaxWXPm6zCPalqe2qmlePefaPjCA50koMrk0GVGGB7WFm9qDIzaOGcTuF7y fgFT81FIK2DLuP64K43EHkaJq+zeISAp8zzVEnXEiVrrAEVdP5uWdgki/Lcv0Aq1JCAF QawaNtHQcaEEfINYQUkLxe5/MSzAhd+Bz/GW4ZZIm0lCe7t98vK2uVpkxPLparvovpHx cRPhQ7SKa9slCjY+FH9pQ67q4POT/mqBce6/2O0fI2NwyOL66Xd24DQA19koDCelDMDd AF3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=5ck9u6Cqqhofl8aUp5xJFrmUenl65A498VLtQCZZhO0=; b=7s8yeSNypMsM/RDvNheeVTywn2xb8QNANfpiPYry73HId/xAnt0bs8pY1UWzcAOZVp H2tfB//zeFA87I5lL5XrXvZjp8Q2thx/b5fMV5knSReNbrt17j9xHELAAI0H59khmeJQ f+WxjeAWehYKo3fJbWiY+hxN9Nuu5Y9+tfXtTPi1GPy99sah45y7r/jSQsJvG21/UGos AQ0MzfeDG5cbRMONKFi3ZatItWgtjUBN5qhPUodUuiFKuwE5S0i4xcCVBvmZPRLR2mp0 JWPGUTn/6UBB5qb7W/1J27mDdUHRMWmEFEyE+mewW5DOnETogRi79Rnvr/vzJ1l5A7Sr 3j2g== X-Gm-Message-State: AOAM530ub/Ro30SA0rYl2JM7Y/ZC0ZHDibJSSnUCiK+nDScmpDMqdQ1y KimcjRj7tNDwKIgvkdYCKtiRVBEYl6fXmLt2okY= X-Google-Smtp-Source: ABdhPJyjOfUmhGqdLkgeFoLOdE1LYct/r661BIAIxNTNoEuBGZwCf0LjtNdLI0bPHxNi87GLOSMMagGlw9zniLCnc9A= X-Received: by 2002:a05:6638:3796:: with SMTP id w22mr17567059jal.126.1635688899578; Sun, 31 Oct 2021 07:01:39 -0700 (PDT) MIME-Version: 1.0 References: <20211019181459.1709976-1-jerinj@marvell.com> <35f086cb-bef3-9a11-6a85-7e695c0b0e7c@ericsson.com> In-Reply-To: From: Jerin Jacob Date: Sun, 31 Oct 2021 19:31:13 +0530 Message-ID: To: =?UTF-8?Q?Mattias_R=C3=B6nnblom?= Cc: "jerinj@marvell.com" , "dev@dpdk.org" , "thomas@monjalon.net" , "ferruh.yigit@intel.com" , "ajit.khaparde@broadcom.com" , "aboyer@pensando.io" , "andrew.rybchenko@oktetlabs.ru" , "beilei.xing@intel.com" , "bruce.richardson@intel.com" , "chas3@att.com" , "chenbo.xia@intel.com" , "ciara.loftus@intel.com" , "dsinghrawat@marvell.com" , "ed.czeck@atomicrules.com" , "evgenys@amazon.com" , "grive@u256.net" , "g.singh@nxp.com" , "zhouguoyang@huawei.com" , "haiyue.wang@intel.com" , "hkalra@marvell.com" , "heinrich.kuhn@corigine.com" , "hemant.agrawal@nxp.com" , "hyonkim@cisco.com" , "igorch@amazon.com" , "irusskikh@marvell.com" , "jgrajcia@cisco.com" , "jasvinder.singh@intel.com" , "jianwang@trustnetic.com" , "jiawenwu@trustnetic.com" , "jingjing.wu@intel.com" , "johndale@cisco.com" , "john.miller@atomicrules.com" , "linville@tuxdriver.com" , "keith.wiles@intel.com" , "kirankumark@marvell.com" , "oulijun@huawei.com" , "lironh@marvell.com" , "longli@microsoft.com" , "mw@semihalf.com" , "spinler@cesnet.cz" , "matan@nvidia.com" , "matt.peters@windriver.com" , "maxime.coquelin@redhat.com" , "mk@semihalf.com" , "humin29@huawei.com" , "pnalla@marvell.com" , "ndabilpuram@marvell.com" , "qiming.yang@intel.com" , "qi.z.zhang@intel.com" , "radhac@marvell.com" , "rahul.lakkireddy@chelsio.com" , "rmody@marvell.com" , "rosen.xu@intel.com" , "sachin.saxena@oss.nxp.com" , "skoteshwar@marvell.com" , "shshaikh@marvell.com" , "shaibran@amazon.com" , "shepard.siegel@atomicrules.com" , "asomalap@amd.com" , "somnath.kotur@broadcom.com" , "sthemmin@microsoft.com" , "steven.webster@windriver.com" , "skori@marvell.com" , "mtetsuyah@gmail.com" , "vburru@marvell.com" , "viacheslavo@nvidia.com" , "xiao.w.wang@intel.com" , "cloud.wangxiaoyun@huawei.com" , "yisen.zhuang@huawei.com" , "yongwang@vmware.com" , "xuanziyang2@huawei.com" , "pkapoor@marvell.com" , "nadavh@marvell.com" , "sburla@marvell.com" , "pathreya@marvell.com" , "gakhil@marvell.com" , "mdr@ashroe.eu" , "dmitry.kozliuk@gmail.com" , "anatoly.burakov@intel.com" , "cristian.dumitrescu@intel.com" , "honnappa.nagarahalli@arm.com" , "ruifeng.wang@arm.com" , "drc@linux.vnet.ibm.com" , "konstantin.ananyev@intel.com" , "olivier.matz@6wind.com" , "jay.jayatheerthan@intel.com" , "asekhar@marvell.com" , "pbhagavatula@marvell.com" , Elana Agostini Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-dev] [RFC PATCH 0/1] Dataplane Workload Accelerator library X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Sun, Oct 31, 2021 at 2:48 PM Mattias R=C3=B6nnblom wrote: > > On 2021-10-29 17:51, Jerin Jacob wrote: > > On Fri, Oct 29, 2021 at 5:27 PM Mattias R=C3=B6nnblom > > wrote: > >> On 2021-10-25 11:03, Jerin Jacob wrote: > >>> On Mon, Oct 25, 2021 at 1:05 PM Mattias R=C3=B6nnblom > >>> wrote: > >>>> On 2021-10-19 20:14, jerinj@marvell.com wrote: > >>>>> From: Jerin Jacob > >>>>> > >>>>> > >>>>> Dataplane Workload Accelerator library > >>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > >>>>> > >>>>> Definition of Dataplane Workload Accelerator > >>>>> -------------------------------------------- > >>>>> Dataplane Workload Accelerator(DWA) typically contains a set of CPU= s, > >>>>> Network controllers and programmable data acceleration engines for > >>>>> packet processing, cryptography, regex engines, baseband processing= , etc. > >>>>> This allows DWA to offload compute/packet processing/baseband/ > >>>>> cryptography-related workload from the host CPU to save the cost an= d power. > >>>>> Also to enable scaling the workload by adding DWAs to the Host CPU = as needed. > >>>>> > >>>>> Unlike other devices in DPDK, the DWA device is not fixed-function > >>>>> due to the fact that it has CPUs and programmable HW accelerators. > >>>> There are already several instances of DPDK devices with pure-softwa= re > >>>> implementation. In this regard, a DPU/SmartNIC represents nothing ne= w. > >>>> What's new, it seems to me, is a much-increased need to > >>>> configure/arrange the processing in complex manners, to avoid bounci= ng > >>>> everything to the host CPU. > >>> Yes and No. It will be based on the profile. The TLV type TYPE_USER_P= LANE will > >>> have user plane traffic from/to host. For example, offloading ORAN sp= lit 7.2 > >>> baseband profile. Transport blocks sent to/from host as TYPE_USER_PLA= NE. > >>> > >>>> Something like P4 or rte_flow-based hooks or > >>>> some other kind of extension. The eventdev adapters solve the same > >>>> problem (where on some systems packets go through the host CPU on th= eir > >>>> way to the event device, and others do not) - although on a *much* > >>>> smaller scale. > >>> Yes. Eventdev Adapters only for event device plumbing. > >>> > >>> > >>>> "Not-fixed function" seems to call for more hot plug support in the > >>>> device APIs. Such functionality could then be reused by anything tha= t > >>>> can be reconfigured dynamically (FPGAs, firmware-programmed > >>>> accelerators, etc.), > >>> Yes. > >>> > >>>> but which may not be able to serve as a RPC > >>>> endpoint, like a SmartNIC. > >>> It can. That's the reason for choosing TLVs. So that > >>> any higher level language can use TLVs like https://protect2.fireeye.= com/v1/url?k=3D96886daf-c91357b6-96882d34-8682aaa22bc0-c994a5dcbda5d9e8&q= =3D1&e=3De89c0aca-a3b3-4f72-b616-ba4550b856b6&u=3Dhttps%3A%2F%2Fgithub.com%= 2Fustropo%2Futtlv > >>> to communicate with the accelerator. TLVs follow the request and > >>> response scheme like RPC. So it can warp it under application if need= ed. > >>> > >>>> DWA could be some kind of DPDK-internal framework for managing certa= in > >>>> type of DPUs, but should it be exposed to the user application? > >>> Could you clarify a bit more. > >>> The offload is represented as a set of TLVs in generic fashion. There > >>> is no DPU specific bit in offload representation. See > >>> rte_dwa_profiile_l3fwd.h header file. > >> > >> It seems a bit cumbersome to work with TLVs on the user application > >> side. Would it be an alternative to have the profile API as a set of C > >> APIs instead of TLV-based messaging interface? The underlying > >> implementation could still be - in many or all cases - be TLVs sent ov= er > >> some appropriate transport. > > The reason to pick TLVs is as follows > > > > 1) Very easy to enable ABI compatibility. (Learned from rte_flow) > > > Do you include the TLV-defined profile interface in "ABI"? Or do you > with ABI only mean the C ABI to send/receive TLVs? To me, the former > makes the most sense, since changing the profile will break binary > compatibility with then-existing applications. The TLV payload will be as part of ABI just like rte_flow. If there is ABI breakage on any TLV we can add a new Tag and it is associa= ted payload to enable backward compatibility. i.e old TLV will work without any change > > > > 2) If it needs to be transported over network etc it needs to be > > packed so that way > > it is easy for implementation to do that with TLV also it gives better > > performance in such > > cases by avoiding reformatting or possibly avoiding memcpy etc. > > My question was not "why TLVs", but the more specific "why are TLVs > exposed to the user application." I find it likely the user applications > are going to wrap the TLV serialization and de-serialization into their > own functions. We can stack up the TLVs, unlike traditional function calls. Those things really need if the device supports N profiles so multiple TLVs can be used in a single shot in fastpath. > > > > 3) It is easy to plugin with another high-level programing language as > > just one API > > > Make sense. One note though: the transport is just one API, but then > each profile makes up an API as well, although it's not C, but TLV-based. Yes, > > > > 4) Easy to decouple DWA core library functionalities from profile. > > 5) Easy to enable asynchronous scheme using request and response TLVs. > > 6) Most importantly, We could introduce type notion with TLV > > (connected with the type of message See TYPE_ATTACHED, TYPE_STOPPED, > > TYPE_USER_PLANE etc ), > > That way, we can have a uniform outlook of profiles instead of each pro= file > > coming with a setup of its own APIs and __rules__ on the state machine. > > I think, for a framework to leverage communication mechanisms and other > > aspects between profiles, it's important to have some synergy between p= rofiles. > > > > > > Yes. I agree that a bit more logic is required on the application side > > to use TLV, > > But I think we can have a wrapper function getting req and response str= uctures. > > > Do you think ethdev, eventdev, cryptodev and the other DPDK APIs had > been better off as TLV-based messaging interfaces as well? From a user > point of view, I'm not sure I see what's so special about talking to a > SmartNIC compared to functions implemented in a GPU, FPGA, an > fix-function ASIC, a large array of garden gnomes or some other manner. > More functionality and more need for asynchronicity (if that's a word) > maybe. No. I am trying to avoid creating 1000s of API and it is driver hooks for all profiles and enable symmetry between all the profiles by attaching state, type attributes to TLV so that we can get a unified view. Nothing specific to SmartNIC/GPU/FPGA. Also, TLVs are very common in interoperable solutions like https://scf.io/en/documents/222_5G_FAPI_PHY_API_Specification.php > > >> Such a C API could still be asynchronous, and still be a profile API > >> (rather than a set of new DPDK device types). > >> > >> > >> What I tried to ask during the meeting but where I didn't get an answe= r > >> (or at least one that I could understand) was how the profiles was to = be > >> specified and/or documented. Maybe the above is what you had in mind > >> already. > > Yes. Documentation is easy, please check the RFC header file for Doxyge= n > > meta to express all the attributes of a TLV. > > > > > > +enum rte_dwa_port_host_ethernet { > > + /** > > + * Attribute | Value > > + * ----------|-------- > > + * Tag | RTE_DWA_TAG_PORT_HOST_ETHERNET > > + * Stag | RTE_DWA_STAG_PORT_HOST_ETHERNET_H2D_INFO > > + * Direction | H2D > > + * Type | TYPE_ATTACHED > > + * Payload | NA > > + * Pair TLV | RTE_DWA_STAG_PORT_HOST_ETHERNET_D2H_INFO > > + * > > + * Request DWA host ethernet port information. > > + */ > > + RTE_DWA_STAG_PORT_HOST_ETHERNET_H2D_INFO, > > + /** > > + * Attribute | Value > > + * ----------|--------- > > + * Tag | RTE_DWA_TAG_PORT_HOST_ETHERNET > > + * Stag | RTE_DWA_STAG_PORT_HOST_ETHERNET_D2H_INFO > > + * Direction | H2D > > + * Type | TYPE_ATTACHED > > + * Payload | struct rte_dwa_port_host_ethernet_d2h_info > > + * Pair TLV | RTE_DWA_STAG_PORT_HOST_ETHERNET_H2D_INFO > > + * > > + * Response for DWA host ethernet port information. > > + */ > > + RTE_DWA_STAG_PORT_HOST_ETHERNET_D2H_INFO, > > > Thanks for the pointer. > > > It would make sense to have a machine-readable schema, so you can > generate the (in my view) inevitable wrapper code. Much like what gRPC > is to protobuf, or Sun RPC to XDR. I thought of doing that, I thought it may not be good due to 1) Additional library dependencies 2) Performance overhead of such solutions. Not all the transports are not supported in all the libraries and allow drivers to enable any sort of transport. 3) Keep it simple 4) Better asynchronous support. 5) If someone needs gRPC kind of thing, it can be wrapped over TLV. Since rte_flow already has the TLV concept it may not be new to DPDK. I really liked rte_flow enablement of ABI combability and its ease of addin= g new stuff. Try to follow similar stuff which is proven in DPDK. Ie. New profile creation will very easy, it will be a matter of identifying the TLVs and their type and payload, rather than everyone comes with new APIs in every profile. > > > Why not use protobuf and its IDL to specify the interface? > >