From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 7C0945A62 for ; Tue, 19 Jan 2016 19:33:14 +0100 (CET) Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga102.jf.intel.com with ESMTP; 19 Jan 2016 10:33:13 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.22,318,1449561600"; d="scan'208";a="730475009" Received: from orsmsx109.amr.corp.intel.com ([10.22.240.7]) by orsmga003.jf.intel.com with ESMTP; 19 Jan 2016 10:33:13 -0800 Received: from orsmsx102.amr.corp.intel.com ([169.254.3.45]) by ORSMSX109.amr.corp.intel.com ([169.254.11.31]) with mapi id 14.03.0248.002; Tue, 19 Jan 2016 10:33:13 -0800 From: "Polehn, Mike A" To: "Xie, Huawei" , "Tan, Jianfeng" , "dev@dpdk.org" Thread-Topic: [dpdk-dev] [PATCH] vhost: remove lockless enqueue to the virtio ring Thread-Index: AQHRUtYeWCkCzypD6ESv7Cx9E9IEMZ8DIKKg Date: Tue, 19 Jan 2016 18:33:12 +0000 Message-ID: <745DB4B8861F8E4B9849C970520ABBF1498488E5@ORSMSX102.amr.corp.intel.com> References: <1451918787-85887-1-git-send-email-huawei.xie@intel.com> <569E6372.5030200@intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiNWQxNDc4NDQtODUxMi00MzRkLWJiOGEtYTMzZmEyM2NmY2NiIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX0lDIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE1LjQuMTAuMTkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiRUw4dXBDUVN6QytSaklpRStqQlBZR2VmcUNQSHFpb2MrZ1wvQ2pGREl3TmM9In0= x-ctpclassification: CTP_IC x-originating-ip: [10.22.254.138] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "ann.zhuangyanying@huawei.com" Subject: Re: [dpdk-dev] [PATCH] vhost: remove lockless enqueue to the virtio ring X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Jan 2016 18:33:15 -0000 SMP operations can be very expensive, sometimes can impact operations by 10= 0s to 1000s of clock cycles depending on what is the circumstances of the s= ynchronization. It is how you arrange the SMP operations within the tasks a= t hand across the SMP cores that gives methods for top performance. Using = traditional general purpose SMP methods will result in traditional general = purpose performance. Migrating to general libraries (understood by most gen= eral purpose programmers) from expert abilities (understood by much smaller= group of expert programmers focused on performance) will greatly reduce th= e value of DPDK since the end result will be lower performance and/or have = less predictable operation where rate performance, predictability, and low = latency are the primary goals. The best method to date, is to have multiple outputs to a single port is to= use a DPDK queue with multiple producer, single consumer to do an SMP oper= ation for multiple sources to feed a single non SMP task to output to the p= ort (that is why the ports are not SMP protected). Also when considerable c= ontention from multiple sources occur often (data feeding at same time), ha= ving DPDK queue with input and output variables in separate cache lines ca= n have a notable throughput improvement. Mike=20 -----Original Message----- From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Xie, Huawei Sent: Tuesday, January 19, 2016 8:44 AM To: Tan, Jianfeng; dev@dpdk.org Cc: ann.zhuangyanying@huawei.com Subject: Re: [dpdk-dev] [PATCH] vhost: remove lockless enqueue to the virti= o ring On 1/20/2016 12:25 AM, Tan, Jianfeng wrote: > Hi Huawei, > > On 1/4/2016 10:46 PM, Huawei Xie wrote: >> This patch removes the internal lockless enqueue implmentation. >> DPDK doesn't support receiving/transmitting packets from/to the same=20 >> queue. Vhost PMD wraps vhost device as normal DPDK port. DPDK=20 >> applications normally have their own lock implmentation when enqueue=20 >> packets to the same queue of a port. >> >> The atomic cmpset is a costly operation. This patch should help=20 >> performance a bit. >> >> Signed-off-by: Huawei Xie >> --- >> lib/librte_vhost/vhost_rxtx.c | 86 >> +++++++++++++------------------------------ >> 1 file changed, 25 insertions(+), 61 deletions(-) >> >> diff --git a/lib/librte_vhost/vhost_rxtx.c=20 >> b/lib/librte_vhost/vhost_rxtx.c index bbf3fac..26a1b9c 100644 >> --- a/lib/librte_vhost/vhost_rxtx.c >> +++ b/lib/librte_vhost/vhost_rxtx.c > > I think vhost example will not work well with this patch when > vm2vm=3Dsoftware. > > Test case: > Two virtio ports handled by two pmd threads. Thread 0 polls pkts from > physical NIC and sends to virtio0, while thread0 receives pkts from > virtio1 and routes it to virtio0. vhost port will be wrapped as port, by vhost PMD. DPDK APP treats all physical and virtual ports as ports equally. When two DPDK threads try to enqueue to the same port, the APP needs to consider the contention. All the physical PMDs doesn't support concurrent enqueuing/dequeuing. Vhost PMD should expose the same behavior unless absolutely necessary and we expose the difference of different PMD. > >> - >> *(volatile uint16_t *)&vq->used->idx +=3D entry_success; > > Another unrelated question: We ever try to move this assignment out of > loop to save cost as it's a data contention? This operation itself is not that costly, but it has side effect on the cache transfer. It is outside of the loop for non-mergable case. For mergeable case, it is inside the loop. Actually it has pro and cons whether we do this in burst or in a smaller step. I prefer to move it outside of the loop. Let us address this later. > > Thanks, > Jianfeng > >