From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1A1A1A0509; Wed, 30 Mar 2022 13:13:01 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A44A840685; Wed, 30 Mar 2022 13:13:00 +0200 (CEST) Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by mails.dpdk.org (Postfix) with ESMTP id 218684013F for ; Wed, 30 Mar 2022 13:12:58 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1648638779; x=1680174779; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=5Cz15+IqUHXdqce8O868oCZttqHQhSmIO5KOUrz4Xdc=; b=dtsQRbXKJ2LAa3LeDwpgIgUfKz9EdLvfUD2MNTm5/CS2VR6wUQNNFtaz Xu4hdOitsqDGes/T3SA4ZTA4jA+cHN1BlSviqKLIMsRZZP68KEMQGjXqA mawQ2CXZEcTJWIJ0gG+awHIOSRbF5U8yMqjskikQbOO3/FbmirAkiRtwb 0ETfrS4C5LnIPHN554Ye9Wr3nFkfn7ncp/OUCtQQb8G/fmTUa4LY0G1BH 6PjaS5cIxDBFslv6ONcVI35B5i/49Y6nR8TJ7cdupmFXKGAg1/N0lz8Yw BAIXGx2tnGCNUlfQLfIR4hpbSugeqHG8QZ7dzB5OpiyK8OMxvnuFdIeqe w==; X-IronPort-AV: E=McAfee;i="6200,9189,10301"; a="320215802" X-IronPort-AV: E=Sophos;i="5.90,222,1643702400"; d="scan'208";a="320215802" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Mar 2022 04:12:57 -0700 X-IronPort-AV: E=Sophos;i="5.90,222,1643702400"; d="scan'208";a="695068333" Received: from bricha3-mobl.ger.corp.intel.com ([10.252.20.222]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-SHA; 30 Mar 2022 04:12:54 -0700 Date: Wed, 30 Mar 2022 12:12:51 +0100 From: Bruce Richardson To: Ilya Maximets Cc: "Pai G, Sunil" , "Stokes, Ian" , "Hu, Jiayu" , "Ferriter, Cian" , "Van Haaren, Harry" , "Maxime Coquelin (maxime.coquelin@redhat.com)" , "ovs-dev@openvswitch.org" , "dev@dpdk.org" , "Mcnamara, John" , "O'Driscoll, Tim" , "Finn, Emma" Subject: Re: OVS DPDK DMA-Dev library/Design Discussion Message-ID: References: <22e3ff73-f3d9-abae-1866-90d133af5528@ovn.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Wed, Mar 30, 2022 at 12:52:15PM +0200, Ilya Maximets wrote: > On 3/30/22 12:41, Ilya Maximets wrote: > > Forking the thread to discuss a memory consistency/ordering model. > > > > AFAICT, dmadev can be anything from part of a CPU to a completely > > separate PCI device. However, I don't see any memory ordering being > > enforced or even described in the dmadev API or documentation. > > Please, point me to the correct documentation, if I somehow missed it. > > > > We have a DMA device (A) and a CPU core (B) writing respectively > > the data and the descriptor info. CPU core (C) is reading the > > descriptor and the data it points too. > > > > A few things about that process: > > > > 1. There is no memory barrier between writes A and B (Did I miss > > them?). Meaning that those operations can be seen by C in a > > different order regardless of barriers issued by C and regardless > > of the nature of devices A and B. > > > > 2. Even if there is a write barrier between A and B, there is > > no guarantee that C will see these writes in the same order > > as C doesn't use real memory barriers because vhost advertises > > s/advertises/does not advertise/ > > > VIRTIO_F_ORDER_PLATFORM. > > > > So, I'm getting to conclusion that there is a missing write barrier > > on the vhost side and vhost itself must not advertise the > > s/must not/must/ > > Sorry, I wrote things backwards. :) > > > VIRTIO_F_ORDER_PLATFORM, so the virtio driver can use actual memory > > barriers. > > > > Would like to hear some thoughts on that topic. Is it a real issue? > > Is it an issue considering all possible CPU architectures and DMA > > HW variants? > > In terms of ordering of operations using dmadev: * Some DMA HW will perform all operations strictly in order e.g. Intel IOAT, while other hardware may not guarantee order of operations/do things in parallel e.g. Intel DSA. Therefore the dmadev API provides the fence operation which allows the order to be enforced. The fence can be thought of as a full memory barrier, meaning no jobs after the barrier can be started until all those before it have completed. Obviously, for HW where order is always enforced, this will be a no-op, but for hardware that parallelizes, we want to reduce the fences to get best performance. * For synchronization between DMA devices and CPUs, where a CPU can only write after a DMA copy has been done, the CPU must wait for the dma completion to guarantee ordering. Once the completion has been returned the completed operation is globally visible to all cores. Hope this is clear. /Bruce