From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 09558A00BE; Mon, 20 Dec 2021 14:21:57 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B783F40040; Mon, 20 Dec 2021 14:21:56 +0100 (CET) Received: from mail-io1-f50.google.com (mail-io1-f50.google.com [209.85.166.50]) by mails.dpdk.org (Postfix) with ESMTP id 302BC4003C for ; Mon, 20 Dec 2021 14:21:55 +0100 (CET) Received: by mail-io1-f50.google.com with SMTP id k21so13224908ioh.4 for ; Mon, 20 Dec 2021 05:21:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=/P3N3nOy2kEPQO7sP8Xlo7/KDtff9ddIowXXOnBcC6E=; b=jeK6BzE3v6UT/cDScRXZla07dnzAJkk4Adk2wLX9Iq9wYqeSq5+Ve/Ansg6pvpHxtj TzfGJOW7M36D7Ualbdcmw7UCesPGP6HvnN+TpQAKJlXQ9J7CQSX06It60Ajdgk8sS/R0 ujKOWE9bd427UdXU4Nq09hSfvJQlIqJZ27GjFgWK3Fkuxcwji7Hs8d2vCJRTlE5y+Tqf PY97om3kXmdUPhZfSt+CCCCA0WOM3jQwUmbP4idaVACJ3vSxA/0OlepE5eXIwrs9CXqx HyHUY9BAApzLAjfnXrH4PjzYrd1GdCIeUjZ8RuINGhrksyTgFvOEUrp83LSNPjwvYS5h IQVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=/P3N3nOy2kEPQO7sP8Xlo7/KDtff9ddIowXXOnBcC6E=; b=RWMjxfjmXDzAgHevxQa9kZY29ob0t4HvXarwKreoEvUFImGVktMcDl/oSRZ0baYng3 l8Y0WFVpBJcF4PoWvsWf771kNqMoTdAdtl/xOv70lZJIXkUncV7A8yumWWmKKmQpMgzw ekUOpPj1HTC8VxmlXaoN+QXSDvAOBgg/cwj9xSPhK1MYprzd+a87Ht23HDOET0LBGh9I bTS23ZQTIOXn+KUfKdL+Q14dmjeCHARjCxfYAg340FF/MAbHlgzdhZHeGxggeDnq7PYi gmIcK0uZt9cBiM8WminjcqiI6enwZcdZzli26oF7ymEdP8089tlyfFO9umnhEZX34PfE ZkqA== X-Gm-Message-State: AOAM530R/fdI426ONfZS4hlj77KYB6B14ZlyuaAM7g39SipRo4i6qLRg MYgsWrEqM/+ZBUTzBBnBRxtcVdN7ebMUlEP6jOTLeQFe X-Google-Smtp-Source: ABdhPJwYqGgd17r9fOnlpk2aFeHy+UD+9XILWuSkCN+wbCIXQzJzAVmiAAO7dPrNHurg8I44m8SmcyclLDsWl8ckDtA= X-Received: by 2002:a05:6638:13d5:: with SMTP id i21mr9549894jaj.79.1640006514506; Mon, 20 Dec 2021 05:21:54 -0800 (PST) MIME-Version: 1.0 References: <20211214141242.3383831-1-ronan.randles@intel.com> <20211214141242.3383831-6-ronan.randles@intel.com> In-Reply-To: From: Jerin Jacob Date: Mon, 20 Dec 2021 18:51:28 +0530 Message-ID: Subject: Re: [PATCH 05/12] gen: add raw packet data API and tests To: "Van Haaren, Harry" Cc: "Randles, Ronan" , Thomas Monjalon , dpdk-dev Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Fri, Dec 17, 2021 at 5:10 PM Van Haaren, Harry wrote: > > +CC Thomas; > > > -----Original Message----- > > From: Jerin Jacob > > Sent: Wednesday, December 15, 2021 12:41 PM > > To: Randles, Ronan > > Cc: dpdk-dev ; Van Haaren, Harry > > > > Subject: Re: [PATCH 05/12] gen: add raw packet data API and tests > > > > On Tue, Dec 14, 2021 at 7:43 PM Ronan Randles > > wrote: > > > > > > From: Harry van Haaren > > > > > > + const uint32_t base_size =3D gen->base_pkt->pkt_len; > > > + const uint8_t *base_data =3D rte_pktmbuf_mtod(gen->base_pkt, = uint8_t > > *); > > > > I think, the very next feature will be generating packets for > > incrementing IP addresses or so. > > Hah, yes! It=E2=80=99s a logical next step, and indeed we have POC code i= nternally that Ronan > and I have worked on that does this :) I've been using this internal POC = of > testing of OVS for ~ a year now, and it provides a pretty nice workflow f= or me. > > > In this case, one packet-based template will not work. > > Why not? I agree that "pre-calculating" all packets will not work, but th= e approach > we have taken for this library is different. See below; > > > May we worth consider that use case into API framework first and add su= pport > > later for implementation as it may change the complete land space of AP= I to have > > better performance. Options like struct rte_gen logical object can have > > N templates instead of one is an option on the table. :-) > > Agree - more complex usages have been designed for too. Let me explain; > > 1) A single gen instance uses a single template, and has "modifiers" that= allow > manipulation of the packet before sending. The gen->base_pkt is copied to= the > destination mbuf, and then updated by the modifiers. This approach is muc= h better > to allow for huge flow-counts (> 1 million?) as pre-calculating and stori= ng 1 million > packets is a waste of memory, and causes a lot of mem-IO for the datapath= core. > > 2) The "modifiers" approach allows any number of things to be changed, wi= th little > mem-IO, and variable CPU cycle cost based on the modifiers themselves. > If the CPU cycle cost of generating packets is too high, just add more co= res :) > > 3) There are also some smarts we can apply for pre-calculating only a sma= ll amount of > data per packet (e.g. uniformly-random distributed src ip). The memory fo= otprint is > lower than pre-calc of whole packets, and the runtime overhead of uniform= -random > is moved to configure time instead of on the datapath. > > 4) Dynamically generating packets by modification of templates allows for= cool things > to be added, e.g. adding timestamps to packets, and calculating latency c= an > be done using the modifier concept and a protocol string "Ether()/IP()/UD= P()/TSC()". > If the packet is being decapped by the target application, the string par= ams can provide > context for where to "retrieve" the TSC from on RX in the generator: "TSC= (rx_offset=3D30)". > I've found this approach to be very flexible and nice, so am a big fan :) > > 5) In order to have multiple streams of totally-different traffic types (= read "from multiple templates") > the user can initialize multiple rte_gen instances. This allows applicati= ons that require multi-stream traffic > to achieve that too, with the same abstraction as a single template strea= m. Initially the generator app is just > providing a single stream, but this application can be expanded to many u= sages over the next year before 22.11 :) OK. I thought "modifiers" will need some sort of critical section in multiple producer use cases. If so, one option could be N streams in one gen instance vs N gen instance. Just my 2c. Anyway, you folks can decide on one option vs another. Only my concern was including such features affect the prototype of existing APIs or not? In either case, No strong opinion. > > I could ramble on a bit more, but mostly diminishing returns I think... I= 'll just use this email as a reply to Thomas' tweet; > https://twitter.com/tmonjalo/status/1337313985662771201 > > Regards, -Harry