From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <yskoh@mellanox.com> Received: from EUR02-VE1-obe.outbound.protection.outlook.com (mail-eopbgr20072.outbound.protection.outlook.com [40.107.2.72]) by dpdk.org (Postfix) with ESMTP id 824F9A84E for <dev@dpdk.org>; Thu, 18 Jan 2018 19:14:05 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=lXBN3F9gIiPPWC3Le3ZxZ/g6Wxw+Y0c4QO6d0L8+3Qg=; b=m4fSgd/ytIZlDs/24MK1hewEWhCCa7OseDTc4m5lIkpwqVwoZvDgs+QANHO3zZBMB0F0U3vR7jv8dD9rNf6MyZ+9PJuabh+RNJm8qUToCeLEMc5rO8XcrPWjwV76Pporh77d+bQdLs+XT15CIgTcoCBmCR/laEfVmT5tbY0ZbP8= Received: from VI1PR0501MB2045.eurprd05.prod.outlook.com (10.167.195.147) by VI1PR0501MB2621.eurprd05.prod.outlook.com (10.172.13.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.407.7; Thu, 18 Jan 2018 18:14:03 +0000 Received: from VI1PR0501MB2045.eurprd05.prod.outlook.com ([fe80::2d4b:9768:b21a:e7e9]) by VI1PR0501MB2045.eurprd05.prod.outlook.com ([fe80::2d4b:9768:b21a:e7e9%14]) with mapi id 15.20.0407.012; Thu, 18 Jan 2018 18:14:03 +0000 From: Yongseok Koh <yskoh@mellanox.com> To: Andrew Rybchenko <arybchenko@solarflare.com> CC: Thomas Monjalon <thomas@monjalon.net>, Jianbo Liu <jianbo.liu@arm.com>, "dev@dpdk.org" <dev@dpdk.org>, Adrien Mazarguil <adrien.mazarguil@6wind.com>, =?iso-8859-1?Q?N=E9lio_Laranjeiro?= <nelio.laranjeiro@6wind.com>, "jerin.jacob@caviumnetworks.com" <jerin.jacob@caviumnetworks.com>, "konstantin.ananyev@intel.com" <konstantin.ananyev@intel.com>, "bruce.richardson@intel.com" <bruce.richardson@intel.com>, Chao Zhu <chaozhu@linux.vnet.ibm.com> Thread-Topic: [dpdk-dev] [PATCH v2 1/8] eal: introduce DMA memory barriers Thread-Index: AQHTjmcQ3r4dfdovVEiBLSv+cotiIqN2ICkAgAAWzICAAd9agIAAUeAAgAEho4CAAGmbgA== Date: Thu, 18 Jan 2018 18:14:03 +0000 Message-ID: <AB474107-C38D-417C-B0EE-BBB2E4993693@mellanox.com> References: <20171227042824.33373-1-yskoh@mellanox.com> <af050403-2404-1364-bfa7-d65ab312180c@solarflare.com> <20180116091040.GA15629@arm.com> <3720864.2redlIt54T@xps> <C08EC0A2-C39A-424D-9269-10FCEAE73BED@mellanox.com> <efbc451f-3416-1a98-b994-87f20848e351@solarflare.com> In-Reply-To: <efbc451f-3416-1a98-b994-87f20848e351@solarflare.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; x-originating-ip: [209.116.155.178] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; VI1PR0501MB2621; 7:6E81Kd1ni5akHrhL0crsM36yZNAxSUDs1ZTsPINfAfdb1ADyjR8q7YKC5NVkSeCH46ga0uawGhHMur4yZ2c86aLhsdaa3EuJfk+1f6LD3KmaQ2RRnAM+NNQjHcNJ6sq1pI7G6uoWTOPQHgZGKau+peDe2+4AFKgxOpVD1Vguf/gJ39iKab6QVgTtboUw1HJveETmSg5QURWSrTppZmdNnw/2YwmTqiw1H0sxoRgwJuS4vjSB4YzAwn5b5gEkKlZU x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: be57276d-35f0-455a-d6fc-08d55e9f4161 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(48565401081)(5600026)(4604075)(3008032)(2017052603307)(7153060)(7193020); SRVR:VI1PR0501MB2621; x-ms-traffictypediagnostic: VI1PR0501MB2621: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: <VI1PR0501MB262100CDA936C478FE07684EC3E80@VI1PR0501MB2621.eurprd05.prod.outlook.com> x-exchange-antispam-report-test: UriScan:(189930954265078)(45079756050767); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040470)(2401047)(5005006)(8121501046)(10201501046)(93006095)(93001095)(3231023)(2400065)(944501161)(3002001)(6055026)(6041268)(20161123562045)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123558120)(6072148)(201708071742011); SRVR:VI1PR0501MB2621; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:VI1PR0501MB2621; x-forefront-prvs: 05568D1FF7 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(396003)(376002)(366004)(346002)(39380400002)(39860400002)(189003)(199004)(4326008)(305945005)(966005)(36756003)(8936002)(25786009)(81156014)(316002)(45080400002)(81166006)(8676002)(68736007)(83716003)(14454004)(3280700002)(7736002)(97736004)(66066001)(478600001)(7416002)(3660700001)(26005)(82746002)(5660300001)(6506007)(76176011)(33656002)(93886005)(6306002)(2950100002)(2900100001)(229853002)(575784001)(6512007)(5250100002)(86362001)(6916009)(53936002)(54906003)(6246003)(102836004)(6116002)(106356001)(53546011)(2906002)(105586002)(6486002)(99286004)(3846002)(59450400001)(6436002); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR0501MB2621; H:VI1PR0501MB2045.eurprd05.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: t2bBYqHY40l2CyrQYGlGmvAB+x3/dvetFxHiNoVwb2ybq+euIBVOnEOf49THA8lyrOmUlKJOVW34rAOKVWACcg== spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-ID: <B122BE66F0D0944CAA299D500B1B84B5@eurprd05.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: be57276d-35f0-455a-d6fc-08d55e9f4161 X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Jan 2018 18:14:03.1441 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0501MB2621 Subject: Re: [dpdk-dev] [PATCH v2 1/8] eal: introduce DMA memory barriers X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://dpdk.org/ml/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://dpdk.org/ml/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://dpdk.org/ml/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> X-List-Received-Date: Thu, 18 Jan 2018 18:14:05 -0000 > On Jan 18, 2018, at 3:56 AM, Andrew Rybchenko <arybchenko@solarflare.com>= wrote: >=20 > On 01/17/2018 09:39 PM, Yongseok Koh wrote: >>> On Jan 17, 2018, at 5:46 AM, Thomas Monjalon <thomas@monjalon.net> wrot= e: >>>=20 >>> 16/01/2018 10:10, Jianbo Liu: >>>> The 01/16/2018 10:49, Andrew Rybchenko wrote: >>>>> On 01/16/2018 04:10 AM, Yongseok Koh wrote: >>>>>> This commit introduces rte_dma_wmb() and rte_dma_rmb(), in order to >>>>>> guarantee the ordering of coherent shared memory between the CPU and= a DMA >>>>>> capable device. >>>>>>=20 >>>>>> Signed-off-by: Yongseok Koh <yskoh@mellanox.com> >>>>>> --- >>>>>> lib/librte_eal/common/include/generic/rte_atomic.h | 18 ++++++++++++= ++++++ >>>>>> 1 file changed, 18 insertions(+) >>>>>>=20 >>>>>> diff --git a/lib/librte_eal/common/include/generic/rte_atomic.h b/li= b/librte_eal/common/include/generic/rte_atomic.h >>>>>> index 16af5ca57..2e0503ce6 100644 >>>>>> --- a/lib/librte_eal/common/include/generic/rte_atomic.h >>>>>> +++ b/lib/librte_eal/common/include/generic/rte_atomic.h >>>>>> @@ -98,6 +98,24 @@ static inline void rte_io_wmb(void); >>>>>> */ >>>>>> static inline void rte_io_rmb(void); >>>>>> +/** >>>>>> + * Write memory barrier for coherent memory between lcore and IO de= vice >>>>>> + * >>>>>> + * Guarantees that the STORE operations on coherent memory that >>>>>> + * precede the rte_dma_wmb() call are visible to I/O device before = the >>>>>> + * STORE operations that follow it. >>>>>> + */ >>>>>> +static inline void rte_dma_wmb(void); >>>>>> + >>>>>> +/** >>>>>> + * Read memory barrier for coherent memory between lcore and IO dev= ice >>>>>> + * >>>>>> + * Guarantees that the LOAD operations on coherent memory updated b= y >>>>>> + * IO device that precede the rte_dma_rmb() call are visible to CPU >>>>>> + * before the LOAD operations that follow it. >>>>>> + */ >>>>>> +static inline void rte_dma_rmb(void); >>>>>> + >>>>>> #endif /* __DOXYGEN__ */ >>>>>> /** >>>>> I'm not an ARMv8 expert so, my comments could be a bit ignorant. >>>>> I'd like to understand the difference between io and added here dma >>>>> barriers. >>>>> The difference should be clearly explained. Otherwise we'll constantl= y hit >>>>> on incorrect choice of barrier type. >>>>> Also I don't understand why "dma" name is chosen taking into account >>>>> that definition is bound to coherent memory between lcore and IO devi= ce. >>>> A good explanation can be found here. >>>>=20 >>>> https://emea01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fg= it.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git%2Fc= ommit%2F%3Fid%3D1077fa36f23e259858caf6f269a47393a5aff523&data=3D02%7C01%7Cy= skoh%40mellanox.com%7C7b526265cbf1449f3db208d55db0c55d%7Ca652971c7d2e4d9ba6= a4d149256f461b%7C0%7C0%7C636517936183877836&sdata=3D2%2Fi8Gs2n%2Fnbe9%2FJ3G= Wr22ndPmQVmvM2Xh12r3j1ZWlg%3D&reserved=3D0 >>> I agree that something more is needed to explain when to use rte_io_*. >>> The only difference between rte_io_* and rte_dma_* is "on coherent memo= ry". >> Okay will add more explanation and send out v3 soon. But, please note th= at >> there's no concrete theory when to use which barrier. Actually, it is mo= stly >> for ARMv8 because it provides more options for barriers. For other archs= , as you >> can see in the patches, there's no difference from IO barriers. >=20 > Absence of concrete theory does not make choice of the memory barrier eas= ier. I didn't say that? I just explained it can't be super clear. > I would say it complicates it significantly. I think it is a minimal requ= irement for > the patchset to explain why a new type should be defined instead of just > fixing of the rte_io_* barriers on ARMv8. What's the different? Which cri= teria > should be checked/taken into account to make the right choice? I already said I'll send out v3. > As far as I can see igb_uio and uio_pci_generic do coherent DMA mapping. > It is not that easy with VFIO since in theory it could be non-coherent if > snooping is not supported by IOMMU. Don't know if it is real. > If so, it makes barrier choice UIO driver-dependent. Sounds bad. I'm not sure I understand this comment but it is because of relaxed memory ordering model of some processors. Actually, x86 is the only processor whic= h has the strong memory ordering model. And the link Jianbo shared could be t= he best article to understand it and I'll summarize it in my v3. Thanks, Yongseok