From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E71A1A0508; Tue, 29 Mar 2022 18:24:17 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8D54C40691; Tue, 29 Mar 2022 18:24:16 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 626E140141 for ; Tue, 29 Mar 2022 18:24:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648571054; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WxM7HAhreFvLIy6lpPf9eWnC4yVIGdc9nPXfOAZBqeM=; b=YsIjQ6c22mTyawiE3BgxaKI/OLjTw/5mmJrUo3Qbo9gHP7+8zbb4x8qAT60rq2j0an3gAu 3wDsJS8rCdCZVcShvd4Et5Jv8YVyDD7b8Zgo1PpCT/I9BFba6w9qWazQ9jV4/7/Z1LmEsC vgul7ZnItzvJ7bmxqhyrbJpIBs2Do/4= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-622-dKq3zlJ4PuaFcDSUqlSUIw-1; Tue, 29 Mar 2022 12:24:12 -0400 X-MC-Unique: dKq3zlJ4PuaFcDSUqlSUIw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A0FA629DD990; Tue, 29 Mar 2022 16:24:11 +0000 (UTC) Received: from [10.39.208.2] (unknown [10.39.208.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B53AF140241B; Tue, 29 Mar 2022 16:24:09 +0000 (UTC) Message-ID: <7968dd0b-8647-8d7b-786f-dc876bcbf3f0@redhat.com> Date: Tue, 29 Mar 2022 18:24:08 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.7.0 Subject: Re: OVS DPDK DMA-Dev library/Design Discussion To: =?UTF-8?Q?Morten_Br=c3=b8rup?= , "Van Haaren, Harry" , "Pai G, Sunil" , "Stokes, Ian" , "Hu, Jiayu" , "Ferriter, Cian" , Ilya Maximets , ovs-dev@openvswitch.org, dev@dpdk.org Cc: "Mcnamara, John" , "O'Driscoll, Tim" , "Finn, Emma" References: <98CBD80474FA8B44BF855DF32C47DC35D86F7C@smartserver.smartshare.dk> <98CBD80474FA8B44BF855DF32C47DC35D86F7D@smartserver.smartshare.dk> From: Maxime Coquelin In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D86F7D@smartserver.smartshare.dk> X-Scanned-By: MIMEDefang 2.85 on 10.11.54.7 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Morten, On 3/29/22 16:44, Morten Brørup wrote: >> From: Van Haaren, Harry [mailto:harry.van.haaren@intel.com] >> Sent: Tuesday, 29 March 2022 15.02 >> >>> From: Morten Brørup >>> Sent: Tuesday, March 29, 2022 1:51 PM >>> >>> Having thought more about it, I think that a completely different architectural approach is required: >>> >>> Many of the DPDK Ethernet PMDs implement a variety of RX and TX packet burst functions, each optimized for different CPU vector instruction sets. The availability of a DMA engine should be treated the same way. So I suggest that PMDs copying packet contents, e.g. memif, pcap, vmxnet3, should implement DMA optimized RX and TX packet burst functions. >>> >>> Similarly for the DPDK vhost library. >>> >>> In such an architecture, it would be the application's job to allocate DMA channels and assign them to the specific PMDs that should use them. But the actual use of the DMA channels would move down below the application and into the DPDK PMDs and libraries. >>> >>> >>> Med venlig hilsen / Kind regards, >>> -Morten Brørup >> >> Hi Morten, >> >> That's *exactly* how this architecture is designed & implemented. >> 1. The DMA configuration and initialization is up to the application (OVS). >> 2. The VHost library is passed the DMA-dev ID, and its new async rx/tx APIs, and uses the DMA device to accelerate the copy. >> >> Looking forward to talking on the call that just started. Regards, -Harry >> > > OK, thanks - as I said on the call, I haven't looked at the patches. > > Then, I suppose that the TX completions can be handled in the TX function, and the RX completions can be handled in the RX function, just like the Ethdev PMDs handle packet descriptors: > > TX_Burst(tx_packet_array): > 1. Clean up descriptors processed by the NIC chip. --> Process TX DMA channel completions. (Effectively, the 2nd pipeline stage.) > 2. Pass on the tx_packet_array to the NIC chip descriptors. --> Pass on the tx_packet_array to the TX DMA channel. (Effectively, the 1st pipeline stage.) The problem is Tx function might not be called again, so enqueued packets in 2. may never be completed from a Virtio point of view. IOW, the packets will be copied to the Virtio descriptors buffers, but the descriptors will not be made available to the Virtio driver. > > RX_burst(rx_packet_array): > 1. Pass on the finished NIC chip RX descriptors to the rx_packet_array. --> Process RX DMA channel completions. (Effectively, the 2nd pipeline stage.) > 2. Replenish NIC chip RX descriptors. --> Start RX DMA channel for any waiting packets. (Effectively, the 1nd pipeline stage.) > > PMD_init(): > - Prepare NIC chip RX descriptors. (In other words: Replenish NIC chip RX descriptors. = RX pipeline stage 1.) > > PS: Rearranged the email, so we can avoid top posting. >