From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 144BAA04B8; Tue, 12 Nov 2019 08:01:53 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0EAF4B62; Tue, 12 Nov 2019 08:01:52 +0100 (CET) Received: from EUR03-VE1-obe.outbound.protection.outlook.com (mail-eopbgr50046.outbound.protection.outlook.com [40.107.5.46]) by dpdk.org (Postfix) with ESMTP id 06EAA91 for ; Tue, 12 Nov 2019 08:01:50 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ti+UGJCNwaDbre/EsY5lbaLsv/FIWD5tnbIubNYviLE=; b=fX8rll/4OdXXYLLK8hcyP6bTpPcorClcR1ZwZzxdoOEWczhY6Kqil1Gd3DchRd9yJrWpAXgJEVlmQxqqnJL+up5BcgghQuoCmqy5/mk4OTwhyJVB301C2P8nCndKS7ECJjn3e05ddYLq+MSaKaWTB+z5sy3nJnLODz5zfMAHrMI= Received: from VI1PR0802CA0031.eurprd08.prod.outlook.com (2603:10a6:800:a9::17) by AM0PR08MB3012.eurprd08.prod.outlook.com (2603:10a6:208:5b::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2451.23; Tue, 12 Nov 2019 07:01:49 +0000 Received: from AM5EUR03FT031.eop-EUR03.prod.protection.outlook.com (2a01:111:f400:7e08::201) by VI1PR0802CA0031.outlook.office365.com (2603:10a6:800:a9::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2430.22 via Frontend Transport; Tue, 12 Nov 2019 07:01:49 +0000 Authentication-Results: spf=fail (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=none action=none header.from=arm.com; Received-SPF: Fail (protection.outlook.com: domain of arm.com does not designate 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM5EUR03FT031.mail.protection.outlook.com (10.152.16.111) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2430.21 via Frontend Transport; Tue, 12 Nov 2019 07:01:49 +0000 Received: ("Tessian outbound 3fba803f6da3:v33"); Tue, 12 Nov 2019 07:01:49 +0000 X-CR-MTA-TID: 64aa7808 Received: from 77b97e764ff7.1 (cr-mta-lb-1.cr-mta-net [104.47.2.57]) by 64aa7808-outbound-1.mta.getcheckrecipient.com id 85DCD54C-7073-4F1D-9FB3-C849A6C9235C.1; Tue, 12 Nov 2019 07:01:44 +0000 Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-db5eur01lp2057.outbound.protection.outlook.com [104.47.2.57]) by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 77b97e764ff7.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Tue, 12 Nov 2019 07:01:44 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eb6ngYazKDIt5JdfwbNrmwz6263KtzbjRq6Y+w9SJyfoP2M+6pOdjlptNMfJMuklo3xSBgtLptKF2o2cV3PPZQ1r2rLOV7QbotQTUDlLIQAL+aC+NneJQS8d0P7oaB/HdXOpEi8JmdCU4W086PumbRlatrOX/QQp9YXHvYjAmrZWXVlHSY4GOL6Ptrhv0gGw+8vmJAF1g7XL5DpjGQbi6In8uOaB3bs5u0G83B2hzAyyso9R8HnibrkcpYn+ROakP1I/Nb2xDIZxRhK1VU+gXp2dOXmzZ3STwmRC8flmotZhhbbbjJybxAYQFLV3iIlziRRb6vxmw/jolgVMkcri4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ti+UGJCNwaDbre/EsY5lbaLsv/FIWD5tnbIubNYviLE=; b=UZSFF7J3jaBlb96E6many/3hLw1xpTkrQKE1PDUnKzoBM8YkGEP0uveDZFkp1qQ/JYLXVY4J/vX81cHoanF3EIOG+JdwR/lnzhNNu4qqQrxjAyPuzSIoJu55hRPnb2+oXyZ1+biKYSJ612l+nSCeVJZWEMdX95n6wI4PfqM+hqPjStV8ArBe8oMoXpnaksxQZBtdsDQq6J1hLWBw/FPZa5WU6zV0AEEeCUQjHSyi6yTLoN1k+VMbq7SY6KQI6hY1/Z8+mGTwoqBPned+wg2WTf/GiileINTXo2gkSZo49dy9O4BODOAYCgq/Gw6dpfJI9aJVJjdMcj17cGjfLOoUWg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ti+UGJCNwaDbre/EsY5lbaLsv/FIWD5tnbIubNYviLE=; b=fX8rll/4OdXXYLLK8hcyP6bTpPcorClcR1ZwZzxdoOEWczhY6Kqil1Gd3DchRd9yJrWpAXgJEVlmQxqqnJL+up5BcgghQuoCmqy5/mk4OTwhyJVB301C2P8nCndKS7ECJjn3e05ddYLq+MSaKaWTB+z5sy3nJnLODz5zfMAHrMI= Received: from VI1PR08MB5376.eurprd08.prod.outlook.com (10.255.196.79) by VI1PR08MB2925.eurprd08.prod.outlook.com (10.170.236.151) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2430.24; Tue, 12 Nov 2019 07:01:41 +0000 Received: from VI1PR08MB5376.eurprd08.prod.outlook.com ([fe80::709e:c2c9:139d:9df3]) by VI1PR08MB5376.eurprd08.prod.outlook.com ([fe80::709e:c2c9:139d:9df3%4]) with mapi id 15.20.2451.023; Tue, 12 Nov 2019 07:01:41 +0000 From: "Gavin Hu (Arm Technology China)" To: "Wangyu (Eric)" , "dev@dpdk.org" CC: "ferruh.yigit@intel.com" , Linuxarm , "humin (Q)" , dengxiaofeng , "Liyuan (Larry)" , nd Thread-Topic: [dpdk-dev] [PATCH v3] bus/pci: resolve multiple NICs address conflicts Thread-Index: AQHVmPjUMUdRWWdTW0+7wJ3Jd2+giKeGzfYwgAA2MiA= Date: Tue, 12 Nov 2019 07:01:41 +0000 Message-ID: References: <1573521821-21173-1-git-send-email-dengxiaofeng@huawei.com> <78A93308629D474AA53B84C5879E84D24B1086FB@DGGEMM533-MBX.china.huawei.com> In-Reply-To: <78A93308629D474AA53B84C5879E84D24B1086FB@DGGEMM533-MBX.china.huawei.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 1dc00a0d-b8fd-4e50-9c75-beda0282dfc1.0 x-checkrecipientchecked: true Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=Gavin.Hu@arm.com; x-originating-ip: [113.29.88.7] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 70cab1b0-8304-4725-a2e8-08d7673e304d X-MS-TrafficTypeDiagnostic: VI1PR08MB2925:|AM0PR08MB3012: X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true x-ms-oob-tlc-oobclassifiers: OLM:6108;OLM:6108; x-forefront-prvs: 021975AE46 X-Forefront-Antispam-Report-Untrusted: SFV:NSPM; SFS:(10009020)(4636009)(396003)(39860400002)(136003)(376002)(366004)(346002)(189003)(199004)(13464003)(110136005)(9686003)(66066001)(486006)(3846002)(6116002)(305945005)(26005)(6436002)(14454004)(7736002)(476003)(5660300002)(229853002)(6246003)(81156014)(86362001)(54906003)(52536014)(316002)(33656002)(81166006)(25786009)(66946007)(71190400001)(66476007)(66446008)(66556008)(71200400001)(76116006)(2906002)(2501003)(446003)(55016002)(4326008)(6506007)(7696005)(53546011)(76176011)(186003)(478600001)(74316002)(102836004)(55236004)(99286004)(14444005)(256004)(64756008)(8936002)(11346002); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR08MB2925; H:VI1PR08MB5376.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: +Ew7OixYlf/g+F+QmOlnWF59y1Kcrzdlq/pCh2ZJ3xiUeJxr190Bw8O0R1oBv4TXbqqPC4U7tbtyAQz8j7PyyiNpDHtmPprtgjaciNS4Lqgo/ElH5AZ0iXiVkZqgav8008GyDdZyJBzhxJVSi/k87sXF9G5MxA9Wlujwo+IiqLUK6y4f+th4mnEdq/FqxRGe9EMIBl87qFfuVcC0NQtNx2nfFt44FYI6MASAytUi4hTxUo6CTW0Vi6QQhdOo2BmTfE4HT+azj/GYYkIbvuTR0g9OaIcfP/TqfN/+aNyf6eh5723IcDAmj+sSWej4vBJvK3e408u9aKgUyMWD1GXIAJ2EJKKZmaEUnUENlm5PXVd8wxZ5/d0oZWfodGxxzgzTeKObeuzPAqmPM49kFE4RADZujEI+vJYqrf2zEeGAwqqCCQ/cBiw08U6G8QjM8zrX x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR08MB2925 Original-Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Gavin.Hu@arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM5EUR03FT031.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; IPV:CAL; SCL:-1; CTRY:IE; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(39860400002)(396003)(346002)(136003)(376002)(1110001)(339900001)(199004)(189003)(13464003)(55016002)(46406003)(9686003)(26826003)(478600001)(54906003)(110136005)(50466002)(316002)(5660300002)(66066001)(6246003)(356004)(86362001)(14454004)(229853002)(11346002)(446003)(336012)(25786009)(126002)(22756006)(476003)(14444005)(6506007)(8936002)(99286004)(2906002)(4326008)(102836004)(33656002)(105606002)(70586007)(70206006)(8746002)(97756001)(6116002)(53546011)(3846002)(76130400001)(23726003)(52536014)(36906005)(74316002)(26005)(81166006)(81156014)(47776003)(186003)(305945005)(7696005)(7736002)(486006)(76176011)(2501003); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR08MB3012; H:64aa7808-outbound-1.mta.getcheckrecipient.com; FPR:; SPF:Fail; LANG:en; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; A:1; MX:1; X-MS-Office365-Filtering-Correlation-Id-Prvs: a8d8bdba-47ba-4160-89c0-08d7673e2bee NoDisclaimer: True X-Forefront-PRVS: 021975AE46 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: QGzKzabHTO0JX7dJRLRT2f8hMEhL9oolmO1wLEg4dnvt6hkv6eXku0VXhSK3cPtd0grq/T+pQysKKAI0qBF3Kss7MeHWYgnqJxLFmjtzNVp5xJxSLd0aOiM8s42/1cFJTnjrQ+EWRdspsJllsFa5+A+hZPwI2SVw0w8IuT54A6etL6YZ7jC1t5/SVsEUU2/a+stbrAId2hZ8E2j6gONRx1KDiHb54qcwgGsVTUl5bSnPn2EDeIhmhkQeA50VGLcxJbh1/VAiIDwvF9Go9KS6iqS7UVkdw80/uzFaF8+DFtSPCecUFRvs//Y3xQ6ad0hG0vA1xm9A8a2KBORZnPsBJX3CqqJedeMr1QblmKGfMBa5cPWXYIJ5gTjmNbnW7ho1NLMmNtQ+x73pFRaFZr4himLypdyF+3QoMjH/Pl63bfZmV5ylMPyZRPJ6SJrfcNO0 X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Nov 2019 07:01:49.1590 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 70cab1b0-8304-4725-a2e8-08d7673e304d X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR08MB3012 Subject: Re: [dpdk-dev] [PATCH v3] bus/pci: resolve multiple NICs address conflicts X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Eric, > -----Original Message----- > From: dev On Behalf Of Wangyu (Eric) > Sent: Tuesday, November 12, 2019 10:23 AM > To: dev@dpdk.org > Cc: ferruh.yigit@intel.com; Linuxarm ; humin (Q) > ; dengxiaofeng ; > Liyuan (Larry) > Subject: [dpdk-dev] [PATCH v3] bus/pci: resolve multiple NICs address con= flicts Change the headline to "bus/pci: fix mapping issue when page size is greate= r than bar"? > NIC address conflicts on 64K pagesize when using multiple NICs, >=20 > as system will mmap 64K pagesize for NIC, >=20 > but dev->mem_resource[i].len is 16K. I would like the problem is described as follows and followed by analysis a= nd fix. When integrating NICs that request memory spaces less than a page size, the= problem hits as follows: " Cannot mmap device resource file %s to address: %p " The root cause is the PCI memory space is mapped at a page size granule, wi= thout considering the page size to calculate the next address for future ma= pping is wrong. The fix is to calculate the next address for future mappings using the real= mapping size. /Gavin >=20 > Signed-off-by: beard-627 > Signed-off-by: Wangyu (Eric) > Acked-by: Wei Hu > Acked-by: Min Hu > --- > drivers/bus/pci/linux/pci_uio.c | 2 ++ drivers/bus/pci/linux/pci_vfio.= c | 3 +++ > 2 files changed, 5 insertions(+) >=20 > diff --git a/drivers/bus/pci/linux/pci_uio.c b/drivers/bus/pci/linux/pci_= uio.c > index 6dca05a..097dc19 100644 > --- a/drivers/bus/pci/linux/pci_uio.c > +++ b/drivers/bus/pci/linux/pci_uio.c > @@ -351,6 +351,8 @@ > pci_map_addr =3D RTE_PTR_ADD(mapaddr, > (size_t)dev->mem_resource[res_idx].len); >=20 > + pci_map_addr =3D RTE_PTR_ALIGN(pci_map_addr, > sysconf(_SC_PAGE_SIZE)); Here "pci_map_addr" is, in effect, exactly the next_addr we talked about in= V2, for use of next mapping, I vote for this fix. /Gavin > maps[map_idx].phaddr =3D dev->mem_resource[res_idx].phys_addr; > maps[map_idx].size =3D dev->mem_resource[res_idx].len; > maps[map_idx].addr =3D mapaddr; > diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci= _vfio.c > index b8faa23..64cd84a 100644 > --- a/drivers/bus/pci/linux/pci_vfio.c > +++ b/drivers/bus/pci/linux/pci_vfio.c > @@ -750,6 +750,9 @@ > bar_addr =3D pci_map_addr; > pci_map_addr =3D RTE_PTR_ADD(bar_addr, (size_t) reg->size); >=20 > + pci_map_addr =3D RTE_PTR_ALIGN(pci_map_addr, > + sysconf(_SC_PAGE_SIZE)); > + > maps[i].addr =3D bar_addr; > maps[i].offset =3D reg->offset; > maps[i].size =3D reg->size; > -- > 1.8.3.1