From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 119D9A0542; Wed, 26 Oct 2022 07:00:50 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A257A40E28; Wed, 26 Oct 2022 07:00:49 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 1ED1F40A7F for ; Wed, 26 Oct 2022 07:00:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1666760447; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vVe/varUdp5n09LYDXy3q9MsyoCfx+iEOTNY+XiHGdY=; b=C+ZRGpUFScBwuh8WUmR21kvsoT9oBRUT8QfDYh4SMm3aeh2H8wt4OJ2d/MdFHxVnjWrbTg WF85Z+xAEhpjk5SvyflS73v5CYFLHhS3zQJKScGKInwVosG2ijiJa5bjVtBwZgoNxoNggC lAXD2T6bfBiLN513NvM8wZLH8AIWC9o= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-663-IaTJTaZkNNi83af7qXzWog-1; Wed, 26 Oct 2022 01:00:41 -0400 X-MC-Unique: IaTJTaZkNNi83af7qXzWog-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C8CF6857F92; Wed, 26 Oct 2022 05:00:40 +0000 (UTC) Received: from [10.39.208.48] (unknown [10.39.208.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D3FA72166B2B; Wed, 26 Oct 2022 05:00:36 +0000 (UTC) Message-ID: Date: Wed, 26 Oct 2022 07:00:34 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.4.0 Subject: Re: [PATCH v5] net/vhost: support asynchronous data path To: "Hu, Jiayu" , "Richardson, Bruce" Cc: "Wang, YuanX" , "Xia, Chenbo" , "dev@dpdk.org" , "Jiang, Cheng1" , "Ma, WenwuX" , "He, Xingguang" , Thomas Monjalon , Ilya Maximets , David Marchand References: <20220814150636.2260317-1-jiayu.hu@intel.com> <20221024151434.1051251-1-yuanx.wang@intel.com> <9336e30f-7daf-3113-cc1d-d9be4269ec3d@redhat.com> From: Maxime Coquelin In-Reply-To: X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Jiayu, On 10/26/22 04:48, Hu, Jiayu wrote: > > >> -----Original Message----- >> From: Maxime Coquelin >> Sent: Wednesday, October 26, 2022 12:05 AM >> To: Richardson, Bruce >> Cc: Hu, Jiayu ; Wang, YuanX ; >> Xia, Chenbo ; dev@dpdk.org; Jiang, Cheng1 >> ; Ma, WenwuX ; He, >> Xingguang ; Thomas Monjalon >> ; Ilya Maximets ; David >> Marchand >> Subject: Re: [PATCH v5] net/vhost: support asynchronous data path >> >> >> >> On 10/25/22 17:44, Bruce Richardson wrote: >>> On Tue, Oct 25, 2022 at 05:33:31PM +0200, Maxime Coquelin wrote: >>>> >>>> >>>> On 10/25/22 11:15, Hu, Jiayu wrote: >>> >>>>>> I think that for Vhost PMD, the Virtio completions should either be >>>>>> performed by DMA engine or by a dedicated thread. >>>>> >>>>> We cannot depend on DMA engine to do completion, as there is no >>>>> ordering guarantee on the HW. For example, given the DMA engine >>>>> issues two updates on the used ring's index, it is possible that the >>>>> second write completes before the first one. >>>> >>>> I'm not sure for Intel hardware, but other vendors may offer ordering >>>> guarantees, it should be exposed as a capability of the DMA device. >>>> If the DMA device offers this capability, it could be used for Vhost. >>>> >>> >>> While I haven't been following this discussion, this particular >>> comment caught my eye. >>> >>> For jobs submitted via a single dmadev device, the "FENCE" flag is >>> provided as part of the dmadev API[1]. Obviously, if the writes come >>> from different dmadevs, then things are rather more complicated. >> > > The cost of "FENCE" is significant, as it requires the HW to stop pipeline and > wait until all previous jobs completed. If ask DMA to update used ring's index, > every updating on used ring's index requires "FENCE", which will make enqueue/ > dequeue become sync operation finally. "FENCE" would slow down DMA and make > DMA become the bottleneck quickly. Also, stalling HW pipeline also means waste > DMA resource. The more efficient way to maintain the ordering is by SW. I don't > think it is acceptable from performance and resource utilization perspective. But on the other hand, there will be less cachelines sharing between the port's Rx and Tx PMD threads. IIRC, Virtio completions by the DMA engines was discussed but never prototyped, so hard to draw a conclusion. Thanks, Maxime > Thanks, > Jiayu > >> Thanks for the clarification Bruce. >> >> In the Vhost PMD case, there is a 1:1 mapping between the virtqueue and >> the DMA channel, so we should be fine. >> >> Regards, >> Maxime >> >>> /Bruce >>> >>> >>> [1] >>> >> https://doc.dpdk.org/api/rte__dmadev_8h.html#a3375e7b956b305505073c4 >> ff >>> 035afe2f >>> >