From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mailout1.w1.samsung.com (mailout1.w1.samsung.com [210.118.77.11]) by dpdk.org (Postfix) with ESMTP id D90965942 for ; Tue, 15 Jan 2019 11:23:57 +0100 (CET) Received: from eucas1p1.samsung.com (unknown [182.198.249.206]) by mailout1.w1.samsung.com (KnoxPortal) with ESMTP id 20190115102356euoutp019cfd84358b507e645af2088fbf16266a~5-o7x60hq1074310743euoutp01_ for ; Tue, 15 Jan 2019 10:23:56 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.w1.samsung.com 20190115102356euoutp019cfd84358b507e645af2088fbf16266a~5-o7x60hq1074310743euoutp01_ DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1547547836; bh=LXk/vcci7xT95OkZ8ZU+Po5ka1EbqA3r6Yjg/Phu2Rk=; h=Subject:To:Cc:From:Date:In-Reply-To:References:From; b=E8Vv+TuHU3a2pb19lB4/4Hy1eWuC5Mo3jOztYF28H9R9XEBfz6aZwbZscZBe6ah7j rc+txvfHo+WDcOHkPZYIBmEWFsvw0jeAE7eYdx8prG3uQ8QfcNIu82WwXteDLqSANK cXi7CRy0LpGCr0CBjC67/isJmSrtqZIkrsNZ0x7A= Received: from eusmges2new.samsung.com (unknown [203.254.199.244]) by eucas1p2.samsung.com (KnoxPortal) with ESMTP id 20190115102356eucas1p22b16274f2ecf1eef4c96a935606eac62~5-o7QkxtW1176811768eucas1p2D; Tue, 15 Jan 2019 10:23:56 +0000 (GMT) Received: from eucas1p2.samsung.com ( [182.198.249.207]) by eusmges2new.samsung.com (EUCPMTA) with SMTP id 60.BF.04294.BB4BD3C5; Tue, 15 Jan 2019 10:23:55 +0000 (GMT) Received: from eusmtrp1.samsung.com (unknown [182.198.249.138]) by eucas1p1.samsung.com (KnoxPortal) with ESMTPA id 20190115102355eucas1p1200e14f235e39b63bd45b69ab804fef4~5-o6eqALy3257332573eucas1p1T; Tue, 15 Jan 2019 10:23:55 +0000 (GMT) Received: from eusmgms2.samsung.com (unknown [182.198.249.180]) by eusmtrp1.samsung.com (KnoxPortal) with ESMTP id 20190115102355eusmtrp17c914da87fb18a2756e0bb1533c6fc2d~5-o6P6Zx12141121411eusmtrp1W; Tue, 15 Jan 2019 10:23:55 +0000 (GMT) X-AuditID: cbfec7f4-835ff700000010c6-33-5c3db4bb1b2a Received: from eusmtip1.samsung.com ( [203.254.199.221]) by eusmgms2.samsung.com (EUCPMTA) with SMTP id 2A.FB.04128.BB4BD3C5; Tue, 15 Jan 2019 10:23:55 +0000 (GMT) Received: from [106.109.129.180] (unknown [106.109.129.180]) by eusmtip1.samsung.com (KnoxPortal) with ESMTPA id 20190115102354eusmtip1395d779b46ed5292c9925d56c1e0cccd~5-o5cLV4o1460014600eusmtip1G; Tue, 15 Jan 2019 10:23:54 +0000 (GMT) To: Shahaf Shuler , "Michael S. Tsirkin" Cc: "dev@dpdk.org" , Maxime Coquelin , Xiao Wang , "jfreimann@redhat.com" , Tiwei Bie , Zhihong Wang , Jason Wang , "xiaolong.ye@intel.com" , "alejandro.lucero@netronome.com" , Daniel Marcovitch From: Ilya Maximets Message-ID: <50fd1781-f1da-da44-0d82-c4a9922850ce@samsung.com> Date: Tue, 15 Jan 2019 13:23:53 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: Content-Language: en-GB Content-Transfer-Encoding: 7bit X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrFKsWRmVeSWpSXmKPExsWy7djP87q7t9jGGHx+YWNx7tMyJoub31ex W7z7tJ3J4kr7T3aLZZc+M1mcW7OUxeJY5x4Wi/+/XrFa3J7jZbG14T+Txf7nh9kt/rwxtdh8 cRKTA6/HrwVLWT0W73nJ5PFs+mEmj+ndD5k93u+7yubRt2UVYwBbFJdNSmpOZllqkb5dAlfG kUmLmQsmKVcc/VrZwHhVrIuRg0NCwETi7kR/EFNIYAWjxHa/LkYuIPMLo0Tj/2usEM5nRok1 H2awdzFygtX33OllhkgsZ5To27QHyvnIKHH00kFGkCphgViJFSfbmUBsEYFAiauTjrOAFDEL bGWW+NbbzgySYBPQkTi1+ghYA6+AncTs/wvB4iwCqhIH1k1jA7FFBSIkOu6vZoOoEZQ4OfMJ C4jNCbRgQfMHsHpmAXGJpi8rWSFseYntb+eAXSQhcI9d4teO78wQd7tIPJzTCPWDsMSr41ug bBmJ05N7WCDseon7LS8ZIZo7GCWmH/rHBJGwl9jy+hw7KJSYBTQl1u/Shwg7Svz+uZwFEo58 EjfeCkLcwCcxadt0Zogwr0RHmxBEtYrE74PLoa6Rkrj57jP7BEalWUg+m4Xkm1lIvpmFsHcB I8sqRvHU0uLc9NRio7zUcr3ixNzi0rx0veT83E2MwBR2+t/xLzsYd/1JOsQowMGoxMPrMN8m Rog1say4MvcQowQHs5IIb5kTUIg3JbGyKrUoP76oNCe1+BCjNAeLkjhvNcODaCGB9MSS1OzU 1ILUIpgsEwenVAOj4bmWdzufcMRHaF/tvqC4UiOYMebHgwqfh1ePFec/9Cpz9Fx1Wk5dLEPp 2bXpOay/kqRud88+tNQwLOr0w8LShXvLeAvk3m1+ZMkT18gTGGwc+rm3t2vdlT0R2T9CLvz4 tjvrmerBKbs//PnKeadSbGfWHJ5d/MpKC52Kv8WeNfSwrG9I4/JQYinOSDTUYi4qTgQAJBqr +10DAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrAIsWRmVeSWpSXmKPExsVy+t/xu7q7t9jGGOxo4rE492kZk8XN76vY Ld592s5kcaX9J7vFskufmSzOrVnKYnGscw+Lxf9fr1gtbs/xstja8J/JYv/zw+wWf96YWmy+ OInJgdfj14KlrB6L97xk8ng2/TCTx/Tuh8we7/ddZfPo27KKMYAtSs+mKL+0JFUhI7+4xFYp 2tDCSM/Q0kLPyMRSz9DYPNbKyFRJ384mJTUnsyy1SN8uQS/jyKTFzAWTlCuOfq1sYLwq1sXI ySEhYCLRc6eXuYuRi0NIYCmjxO0LWxkhElISP35dYIWwhSX+XOtigyh6zyix//tddpCEsECs xIqT7UwgtohAoMS1TXPYQYqYBXYySzy+cZsVomMFq0R3zymwsWwCOhKnVh8Bs3kF7CRm/1/I DGKzCKhKHFg3jQ3EFhWIkDj7ch1UjaDEyZlPWEBsTqBtC5o/gNUzC6hL/Jl3CcoWl2j6spIV wpaX2P52DvMERqFZSNpnIWmZhaRlFpKWBYwsqxhFUkuLc9Nzi430ihNzi0vz0vWS83M3MQJj d9uxn1t2MHa9Cz7EKMDBqMTD6zDfJkaINbGsuDL3EKMEB7OSCG+ZE1CINyWxsiq1KD++qDQn tfgQoynQcxOZpUST84FpJa8k3tDU0NzC0tDc2NzYzEJJnPe8QWWUkEB6YklqdmpqQWoRTB8T B6dUA+M62Tdp65q0BEyvb1au5JRbptEhybcss+DxJ+uM19v9tLwNfj2/GJ0Rf/3Dc8Weo3kO PIw5rr2V8gW34t/9yEr5r84UtWt1XfbhhviZoZEhux6++m9Ux8VeHdX5uHZT1ccFy/QV3lz4 1pXcuE4z+OqKM8ejGXbOqJHekly/Uvegf5zLovlTOZRYijMSDbWYi4oTAX6j3mjzAgAA X-CMS-MailID: 20190115102355eucas1p1200e14f235e39b63bd45b69ab804fef4 X-Msg-Generator: CA Content-Type: text/plain; charset="utf-8" X-RootMTR: 20181226163717eucas1p15276eb45e35abe2c9cf3e7c1e0050823 X-EPHeader: CA CMS-TYPE: 201P X-CMS-RootMailID: 20181226163717eucas1p15276eb45e35abe2c9cf3e7c1e0050823 References: <20181214153812.3878-1-i.maximets@samsung.com> <20181226163712.31596-1-i.maximets@samsung.com> <2832c88f-9b43-997c-5937-ef5ae6482fd5@samsung.com> <20190109104855-mutt-send-email-mst@kernel.org> <0e415915-2a36-6950-4b11-adf91bdf2b0b@samsung.com> Subject: Re: [dpdk-dev] [PATCH v2] net/virtio: add platform memory ordering feature support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jan 2019 10:23:58 -0000 On 15.01.2019 11:55, Shahaf Shuler wrote: > Tuesday, January 15, 2019 10:29 AM, Ilya Maximets: >> Subject: Re: [dpdk-dev] [PATCH v2] net/virtio: add platform memory >> ordering feature support >> >> On 15.01.2019 9:33, Shahaf Shuler wrote: >>> Thursday, January 10, 2019 10:37 PM, Shahaf Shuler: >>>> Subject: RE: [dpdk-dev] [PATCH v2] net/virtio: add platform memory >>>> ordering feature support >>>> >>>> Wednesday, January 9, 2019 5:50 PM, Michael S. Tsirkin: >>>>> alejandro.lucero@netronome.com; Daniel Marcovitch >>>>> >>>>> Subject: Re: [dpdk-dev] [PATCH v2] net/virtio: add platform memory >>>>> ordering feature support >>>>> >>>>> On Wed, Jan 09, 2019 at 05:34:38PM +0300, Ilya Maximets wrote: >>>>>> virtio_mb() is really heavy. I'd like to avoid it somehow, but I >>>>>> don't know how to do this yet. >>>>> >>>>> Linux driver doesn't avoid it either. >>>> >>>> I understand v3 was merged but still would like to continue the >>>> discuss and make sure all is clear and agreed. >>>> >>>> Form patch [1] description it is very clear why we need the >>>> rte_smp_mb() barrier. >>>> However I am not sure why this barrier is interoperate into rte_mb in >>>> case of vDPA. In vDPA case, both read of the user ring and write of >>>> the avail index are for local cached memory. >>>> The only write which is to uncachable memory (device memory) is the >>>> notify itself. >>>> >>>> As I mentioned, there is a need to have a store fence before doing >>>> the notify, but from different reasons. So vDPA use case and need Is >>>> a bit different than what presented in [1]. >>> >>> Any answer? >>> It is pity if we add redundant barriers which will degrade the driver >> performance. >> >> Sorry for late responses. Have a lot of work with OVS right now. >> >> Regarding your question. >> Current code looks like this: >> >> 1. Update ring. >> 2. virtio_wmb() >> 3. Update idx. >> 4. virtio_mb() >> 5. read flags. >> 6. notify. >> >> virtio_mb() is here to avoid reordering of steps 3 and 5. >> i.e. we need a full barrier to ensure the order between store (idx update) >> and load (reading the flags). Otherwise we could miss the notification. >> We can't avoid the barrier here, because even x86 does not guarantee the >> ordering of the local load with earlier local store. > > This is clear. You need the rte_smp_mb() here. My question is why you need the rte_mb() in case of vDPA? > As you said, all accesses are local. Sorry, I misunderstood your question. Memory accesses are not local here. We want the ring data/idx be visible to vDPA HW before reading the flags. > > Pasting you commit code: > /* > * Per virtio_ring.h in Linux. > * For virtio_pci on SMP, we don't need to order with respect to MMIO > * accesses through relaxed memory I/O windows, so smp_mb() et al are > * sufficient. > * > * For using virtio to talk to real devices (eg. vDPA) we do need real > * barriers. > */ > static inline void > virtio_mb(uint8_t weak_barriers) > { > if (weak_barriers) > rte_smp_mb(); > else > rte_mb(); > } > >> >>> >>>> >>>> [1] >>>> >> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpa >>>> >> tches.dpdk.org%2Fpatch%2F49545%2F&data=02%7C01%7Cshahafs%40 >> mellan >>>> >> ox.com%7C01907f1b2e0e4002cb7508d67ac38a98%7Ca652971c7d2e4d9ba6a4 >> d1492 >>>> >> 56f461b%7C0%7C0%7C636831377591864200&sdata=TSpc%2Fzyq2aq0N3 >> %2Bh4o >>>> ro4std8ut%2FQU6%2BOeMDeeaQdsM%3D&reserved=0 >>>> >>>>> >>>>> -- >>>>> MST >>> >>>