From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 819A1A00BE; Wed, 27 Apr 2022 22:34:35 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5BC7A40E78; Wed, 27 Apr 2022 22:34:35 +0200 (CEST) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id 06C9840691 for ; Wed, 27 Apr 2022 22:34:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651091673; x=1682627673; h=date:from:to:cc:subject:message-id:references: mime-version:content-transfer-encoding:in-reply-to; bh=biBy5Ef2lqc3ZsshPwmJ9kISHjF6JkGh0VDLUmQ6Sy4=; b=Ld/gXhY44iDbQVJVsmB7Z0uDk8EMfP8sViYL2BJcwHynbV86HyNPu1ib 6GFb72/c5IjWqXp3Sw63P6AYtxMcZ/q0//6H5eQkgJVHQLKB2NLmzEHN4 jxNMLqgVTEDba2GP5KLNILjTuYjHPL53noz+rMvaqMQWZbBF6OWve/qwf sXKahNi8GKJEeOzuSbZJS6UfqIJMsYYwL1Cw7hzl/CNTve5SxHF/eX+bM BQco9Oq5suXh00IiM/LcpwbJPIoNq2D671KSBjcAP7L4A87vi0qDq3C0J OVgy5Cv8LwCxJ7U3Blc0QuJYxyNWvPzVyjh5q1opAdWi68eF/qVtFmVXW Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10330"; a="265875213" X-IronPort-AV: E=Sophos;i="5.90,294,1643702400"; d="scan'208";a="265875213" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Apr 2022 13:34:31 -0700 X-IronPort-AV: E=Sophos;i="5.90,294,1643702400"; d="scan'208";a="513859499" Received: from bricha3-mobl.ger.corp.intel.com ([10.252.6.60]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-SHA; 27 Apr 2022 13:34:28 -0700 Date: Wed, 27 Apr 2022 21:34:25 +0100 From: Bruce Richardson To: Ilya Maximets Cc: "Mcnamara, John" , "Hu, Jiayu" , Maxime Coquelin , "Van Haaren, Harry" , Morten =?iso-8859-1?Q?Br=F8rup?= , "Pai G, Sunil" , "Stokes, Ian" , "Ferriter, Cian" , "ovs-dev@openvswitch.org" , "dev@dpdk.org" , "O'Driscoll, Tim" , "Finn, Emma" Subject: Re: OVS DPDK DMA-Dev library/Design Discussion Message-ID: References: <98CBD80474FA8B44BF855DF32C47DC35D86F82@smartserver.smartshare.dk> <94d817cb-8151-6644-c577-ed8b42d24337@redhat.com> <55c5a37f-3ee8-394b-8cff-e7daecb59f73@ovn.org> <0a414313f07d4781b9bdd8523c2e06f5@intel.com> <5ba635f6-0e7f-a558-b599-674e272cfd1e@ovn.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Mon, Apr 25, 2022 at 11:46:01PM +0200, Ilya Maximets wrote: > On 4/20/22 18:41, Mcnamara, John wrote: > >> -----Original Message----- > >> From: Ilya Maximets > >> Sent: Friday, April 8, 2022 10:58 AM > >> To: Hu, Jiayu ; Maxime Coquelin > >> ; Van Haaren, Harry > >> ; Morten Brørup ; > >> Richardson, Bruce > >> Cc: i.maximets@ovn.org; Pai G, Sunil ; Stokes, Ian > >> ; Ferriter, Cian ; ovs- > >> dev@openvswitch.org; dev@dpdk.org; Mcnamara, John > >> ; O'Driscoll, Tim ; > >> Finn, Emma > >> Subject: Re: OVS DPDK DMA-Dev library/Design Discussion > >> > >> On 4/8/22 09:13, Hu, Jiayu wrote: > >>> > >>> > >>>> -----Original Message----- > >>>> From: Ilya Maximets > >>>> Sent: Thursday, April 7, 2022 10:40 PM > >>>> To: Maxime Coquelin ; Van Haaren, Harry > >>>> ; Morten Brørup > >>>> ; Richardson, Bruce > >>>> > >>>> Cc: i.maximets@ovn.org; Pai G, Sunil ; Stokes, > >>>> Ian ; Hu, Jiayu ; Ferriter, > >>>> Cian ; ovs-dev@openvswitch.org; > >>>> dev@dpdk.org; Mcnamara, John ; O'Driscoll, > >>>> Tim ; Finn, Emma > >>>> Subject: Re: OVS DPDK DMA-Dev library/Design Discussion > >>>> > >>>> On 4/7/22 16:25, Maxime Coquelin wrote: > >>>>> Hi Harry, > >>>>> > >>>>> On 4/7/22 16:04, Van Haaren, Harry wrote: > >>>>>> Hi OVS & DPDK, Maintainers & Community, > >>>>>> > >>>>>> Top posting overview of discussion as replies to thread become > >> slower: > >>>>>> perhaps it is a good time to review and plan for next steps? > >>>>>> > >>>>>>  From my perspective, it those most vocal in the thread seem to be > >>>>>> in favour of the clean rx/tx split ("defer work"), with the > >>>>>> tradeoff that the application must be aware of handling the async > >>>>>> DMA completions. If there are any concerns opposing upstreaming of > >>>>>> this > >>>> method, please indicate this promptly, and we can continue technical > >>>> discussions here now. > >>>>> > >>>>> Wasn't there some discussions about handling the Virtio completions > >>>>> with the DMA engine? With that, we wouldn't need the deferral of work. > >>>> > >>>> +1 > >>>> > >>>> With the virtio completions handled by DMA itself, the vhost port > >>>> turns almost into a real HW NIC. With that we will not need any > >>>> extra manipulations from the OVS side, i.e. no need to defer any work > >>>> while maintaining clear split between rx and tx operations. > >>> > >>> First, making DMA do 2B copy would sacrifice performance, and I think > >>> we all agree on that. > >> > >> I do not agree with that. Yes, 2B copy by DMA will likely be slower than > >> done by CPU, however CPU is going away for dozens or even hundreds of > >> thousands of cycles to process a new packet batch or service other ports, > >> hence DMA will likely complete the transmission faster than waiting for > >> the CPU thread to come back to that task. In any case, this has to be > >> tested. > >> > >>> Second, this method comes with an issue of ordering. > >>> For example, PMD thread0 enqueue 10 packets to vring0 first, then PMD > >>> thread1 enqueue 20 packets to vring0. If PMD thread0 and threa1 have > >>> own dedicated DMA device dma0 and dma1, flag/index update for the > >>> first 10 packets is done by dma0, and flag/index update for the left > >>> 20 packets is done by dma1. But there is no ordering guarantee among > >>> different DMA devices, so flag/index update may error. If PMD threads > >>> don't have dedicated DMA devices, which means DMA devices are shared > >>> among threads, we need lock and pay for lock contention in data-path. > >>> Or we can allocate DMA devices for vring dynamically to avoid DMA > >>> sharing among threads. But what's the overhead of allocation mechanism? > >> Who does it? Any thoughts? > >> > >> 1. DMA completion was discussed in context of per-queue allocation, so > >> there > >> is no re-ordering in this case. > >> > >> 2. Overhead can be minimal if allocated device can stick to the queue for > >> a > >> reasonable amount of time without re-allocation on every send. You may > >> look at XPS implementation in lib/dpif-netdev.c in OVS for example of > >> such mechanism. For sure it can not be the same, but ideas can be re- > >> used. > >> > >> 3. Locking doesn't mean contention if resources are allocated/distributed > >> thoughtfully. > >> > >> 4. Allocation can be done be either OVS or vhost library itself, I'd vote > >> for doing that inside the vhost library, so any DPDK application and > >> vhost ethdev can use it without re-inventing from scratch. It also > >> should > >> be simpler from the API point of view if allocation and usage are in > >> the same place. But I don't have a strong opinion here as for now, > >> since > >> no real code examples exist, so it's hard to evaluate how they could > >> look > >> like. > >> > >> But I feel like we're starting to run in circles here as I did already say > >> most of that before. > > > > > > Hi, John. > > Just reading this email as I was on PTO for a last 1.5 weeks > and didn't get through all the emails yet. > > > This does seem to be going in circles, especially since there seemed to be technical alignment on the last public call on March 29th. > > I guess, there is a typo in the date here. > It seems to be 26th, not 29th. > > > It is not feasible to do a real world implementation/POC of every design proposal. > > FWIW, I think it makes sense to PoC and test options that are > going to be simply unavailable going forward if not explored now. > Especially because we don't have any good solutions anyway > ("Deferral of Work" is architecturally wrong solution for OVS). > Hi Ilya, for those of us who haven't spent a long time working on OVS, can you perhaps explain a bit more as to why it is architecturally wrong? From my experience with DPDK, use of any lookaside accelerator, not just DMA but any crypto, compression or otherwise, requires asynchronous operation, and therefore some form of setting work aside temporarily to do other tasks. Thanks, /Bruce