From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id DB6E9A046B for ; Tue, 23 Jul 2019 07:28:04 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 99C8D1BF55; Tue, 23 Jul 2019 07:28:03 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by dpdk.org (Postfix) with ESMTP id 9F3271BD62 for ; Tue, 23 Jul 2019 07:28:02 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x6N5Qhb4027667; Mon, 22 Jul 2019 22:28:01 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=pfpt0818; bh=7G9RuumhyeJ57FSDdMDnfXCsEQG8j9Gw1UGy67M4yYs=; b=OTWgxPhh8RJHg9jw0B972hACGB2rfVNd/wyyJ6WgFlSmIiJh/efFdoKudWsOOhfH5ywp AoI2q4W826rWNgteIUF3KHu7fzCjKnCjEh7O6x66oiy58kJUpXz+lJb3yAeDQIOqoeUU 9/MHTZmtzQW6VCA20JllpyQSZKfTZsUCBjNn16nUuMEhgD2pl8E6DNng0VSfsYyC5oZS K/NvOJvR/Jhf/91wv8Y72Zmn6o0TSA22qoTn9edJYE04kvdaOe/1cpy/q39097TUlwLV E9x3F4AXgB+MqQKVPZzNNY3GZF0AwrcpoqYOXiSvcJDj4WugYowc/dOxU8yxlerORSM/ Cw== Received: from sc-exch03.marvell.com ([199.233.58.183]) by mx0b-0016f401.pphosted.com with ESMTP id 2tv2fj9fts-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Mon, 22 Jul 2019 22:28:01 -0700 Received: from SC-EXCH03.marvell.com (10.93.176.83) by SC-EXCH03.marvell.com (10.93.176.83) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Mon, 22 Jul 2019 22:28:00 -0700 Received: from NAM01-BY2-obe.outbound.protection.outlook.com (104.47.34.56) by SC-EXCH03.marvell.com (10.93.176.83) with Microsoft SMTP Server (TLS) id 15.0.1367.3 via Frontend Transport; Mon, 22 Jul 2019 22:28:00 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fyNGesRDE3HIdWh6F6/5b1pcVnMc0iTU/wEP2l0/oN4vuhVdUDFvpELC/+9/UFzs4s1zMgAZyvmlGI3Frnz/B9iDcZKD05BDnFnmupPhLvW3UASG7AJXP2AB/UHdAnQ+D08H6BJ3QYP9DTdw+KZaWo8Urm940OgYKrzBKP7u9CHd0/vAMuqIkHZB/PQFgqIbp5sEhpPoUAKkx5a0SLJp2yo24xkQ52MbK3Am07THu+dYxcFVW8//L52myJCqUUu8XUzAUM/z7W6pwEkoe/88JCuKW72sYcR863knFIXPXXntZtZaMqSvpQX4wBtuHnVXRDdWX4vnX3SyKjU6U4jqdA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7G9RuumhyeJ57FSDdMDnfXCsEQG8j9Gw1UGy67M4yYs=; b=gzUlJmFPrNzpgqLzwnlrKHFdhviakMy8Pb4LTqdfXDEID9e+GbgY4V1+E5K2itpYlP1sNZXg0BNe9EwrdALOhewVue77hRSEejCrUANeCcY4S1XdQv68R2X1/f1s+Z00kS7Z/8GwQyXAXPv9exj1lqiA4HWmi9Y/kP28po5PJeAGMxMpnkrtzlTv++PHk8G3lv3DqawwYcbEdZYWxD+PGe9Cw08wBOdTDz+jOku5E/Kzv2LWq6qPr472VNcnjYn3GZKkXYKAcD3Ek1sVm/+kljX5JE5QszhjKzzejTDZ/h75O1XxCsHFc1e/lCdmFMlM7eEj8cS+dIQ0UgHBdrjIgQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1;spf=pass smtp.mailfrom=marvell.com;dmarc=pass action=none header.from=marvell.com;dkim=pass header.d=marvell.com;arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.onmicrosoft.com; s=selector2-marvell-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7G9RuumhyeJ57FSDdMDnfXCsEQG8j9Gw1UGy67M4yYs=; b=WA1w76DwHZd1DZ5lgDu+RxkhEzm0FuY0Y30wVmb1e2RXYYYd2gaGEKP2Y7BjZcpKOlehT8UCXGSFGGyPgXJfcUMSQcZU03tkuSzzkEKWZ2iOXcZi4r98Ij31P7Wfnnx+RlH5IIlit5wGOca558F3jMmyiuvxSw5PZwTTCBveV+M= Received: from BYAPR18MB2424.namprd18.prod.outlook.com (20.179.91.149) by BYAPR18MB2965.namprd18.prod.outlook.com (20.179.59.209) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2094.17; Tue, 23 Jul 2019 05:27:58 +0000 Received: from BYAPR18MB2424.namprd18.prod.outlook.com ([fe80::2d42:12b6:aa2e:2862]) by BYAPR18MB2424.namprd18.prod.outlook.com ([fe80::2d42:12b6:aa2e:2862%4]) with mapi id 15.20.2094.013; Tue, 23 Jul 2019 05:27:58 +0000 From: Jerin Jacob Kollanukkaran To: "Stojaczyk, Dariusz" , Thomas Monjalon , David Marchand , "Burakov, Anatoly" CC: "dev@dpdk.org" Thread-Topic: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode selection Thread-Index: AQHVQQe8sqi9vPlkxkmyxie40lyIBKbXlQIwgAAORwCAAAYfQA== Date: Tue, 23 Jul 2019 05:27:58 +0000 Message-ID: References: <1562795329-16652-1-git-send-email-david.marchand@redhat.com> <1563800213-29839-1-git-send-email-david.marchand@redhat.com> <1735440.496QTsyyT4@xps> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [106.200.248.176] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 5c0ee0e7-6227-437d-86e1-08d70f2e8603 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600148)(711020)(4605104)(1401327)(2017052603328)(7193020); SRVR:BYAPR18MB2965; x-ms-traffictypediagnostic: BYAPR18MB2965: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 0107098B6C x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(4636009)(136003)(396003)(39860400002)(366004)(376002)(346002)(189003)(199004)(13464003)(54534003)(8936002)(14444005)(5024004)(7696005)(76176011)(256004)(6506007)(14454004)(6116002)(3846002)(53546011)(68736007)(99286004)(33656002)(2906002)(26005)(4326008)(11346002)(446003)(486006)(478600001)(6246003)(9686003)(53936002)(186003)(55016002)(66066001)(81156014)(102836004)(81166006)(6436002)(229853002)(52536014)(316002)(5660300002)(74316002)(76116006)(7736002)(305945005)(86362001)(71190400001)(71200400001)(66946007)(66446008)(64756008)(66556008)(66476007)(110136005)(25786009)(476003)(554214002)(8676002); DIR:OUT; SFP:1101; SCL:1; SRVR:BYAPR18MB2965; H:BYAPR18MB2424.namprd18.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: marvell.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: PJboN6PFYm+Y26hq2PI00/i8AOUzicGel7hQNsJR5lcFtR3vAT1P5asggCVmXkfWAbcSDzIRgEFeRHgCfRXs7TgfboWN/IkDgiOvSrm0ACLdGr7iMEoCUNOBZJbb+NEEITayzzFjfoaEDVRfA/xvvztiBV1q1Ba4wlbszwbGKKB1IE6/BnCXm5stAieiCvvP0K4+Xv56aTNGHC/+tj6jyueW7FKfULh5LafHofXr6k3ksMFFfVka4nrUP+30JKpgBVV5Peq7lsjnm8ZkMsHd28b7FFyKr+Q8rfF+/EzGn9Py/H6BxcnTk4G3cOfcl4PZAPrl7NKk0C3GRG8SdQgzfO2cQz0odRJGmB7Ke25IVQnayOD3shZ7wo1mw57HhbV4ANlXJ3F3Qka84xk5dlDZ7x+9YFXS7eUZT98owSphZBE= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 5c0ee0e7-6227-437d-86e1-08d70f2e8603 X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Jul 2019 05:27:58.6385 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 70e1fb47-1155-421d-87fc-2e58f638b6e0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: jerinj@marvell.com X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR18MB2965 X-OriginatorOrg: marvell.com X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:5.22.84,1.0.8 definitions=2019-07-23_03:2019-07-23,2019-07-23 signatures=0 Subject: Re: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode selection X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: Stojaczyk, Dariusz > Sent: Tuesday, July 23, 2019 10:24 AM > To: Jerin Jacob Kollanukkaran ; Thomas Monjalon > ; David Marchand ; > Burakov, Anatoly > Cc: dev@dpdk.org > Subject: [EXT] RE: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode selection >=20 > > -----Original Message----- > > From: Jerin Jacob Kollanukkaran [mailto:jerinj@marvell.com] > > Sent: Tuesday, July 23, 2019 6:19 AM > > > > > -----Original Message----- > > > From: Stojaczyk, Dariusz > > > Sent: Tuesday, July 23, 2019 9:06 AM > > > To: Thomas Monjalon ; David Marchand > > > ; Burakov, Anatoly > > > ; Jerin Jacob Kollanukkaran > > > > > > Cc: dev@dpdk.org > > > Subject: [EXT] RE: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode > > > selection > > > > > > This introduces a regression where uio-bound devies are attached to > > > a > > DPDK > > > app at runtime. > > > > Just to understand the requirements; > > # Is this requirement for SPDK? > > # Is brand new PCI device scanned and attached to DPDK at runtime? > > # Any specific reason for using uio vs vfio? >=20 > Jerin, Stojaczyk, There reason to choose VA incase if bus detects DC is following: - All drivers are expected to work in RTE_IOVA_VA mode, irrespective of physical address availability. - By default, the mempool, first asks for IOVA-contiguous memory using ``RTE_MEMZONE_IOVA_CONTIG``. This is slow in RTE_IOVA_PA mode and it = may affect the application boot time. - It is easy to enable large amount of IOVA-contiguous memory use-cases with IOVA in VA mode. >=20 > It came up in SPDK tests, but it's certainly nothing SPDK-specific, I can= 't give > you the steps but it should be reproducible even with testpmd. >=20 > The PCI device could have been simply hotplugged to the system after DPDK > app start. DPDK didn't know about it at initialization, so it picked > RTE_IOVA_VA and then would fail to attach any UIO-bound device ever > after: >=20 > EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializin= g We have RTE_PCI_DRV_NEED_IOVA_AS_VA devices in DPDK, Which can work Only on VA. If we default 'PA' incase of DC, then what do with hotplugging = on those devices? > EAL: Driver cannot attach the device (0000:00:09.0) > EAL: Failed to attach device on primary process >=20 > UIO is commonly used on systems without IOMMU- including VMs. The latest machines has IOMMU. Which machines you are testing against, Can we detect the machines without IOMMU and switch to PA? >=20 > > > > If it is for SPDK, > > # How about introducing rte_eal_init_with_mode(enum rte_iova_mode)? > > # How about adding dummy bus which returns RTE_IOVA_PA in the > > bus_get_iommus_class() in SPDK code base? >=20 > There's already an --iova=3Dmode option in DPDK that forces the iova mode= . > I'm not concerned about configurability, but the regression in the defaul= t > behavior. >=20 > I can add workarounds to SPDK, sure, but that wouldn't be a very healthy > approach. Nothing like workaround, I am looking for the options for expressing The requirements for PA? >=20 > D. >=20 > > > > > > > > When there are no devices attached at initialization, the only safe > > > default should be RTE_IOVA_PA. With RTE_IOVA_VA we just won't be > > > able to do any DMA to uio-bound PCI devices. > > > > > > Can we revert this patch? > > > > > > D. > > > > > > > -----Original Message----- > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas > > Monjalon > > > > Sent: Monday, July 22, 2019 5:53 PM > > > > To: David Marchand ; Burakov, Anatoly > > > > ; jerinj@marvell.com > > > > Cc: dev@dpdk.org > > > > Subject: Re: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode > > > > selection > > > > > > > > 22/07/2019 14:56, David Marchand: > > > > > Following the issues reported by Jerin and the discussion that > > > > > emerged from it, here are fixes to restore and document the > > > > > behavior of the EAL and the pci bus driver. > > > > > > > > > > I pondered all the arguments and tried to have the less changes > > > > > possible. > > > > > I can't find a need for a flag to just announce support of > > > > > physical addresses from the pmd point of view. > > > > > So it ended up with something really close to what Jerin had > suggested. > > > > > > > > > > But the problem is that this is still unfinished wrt the document= ation. > > > > > I will be offline for 10 days and we need this to move forward, > > > > > so sending anyway. > > > > > > > > > > Changelog since v3: > > > > > - fixed typos in patch 2, > > > > > - updated patch 3 title, > > > > > - moved and reworded comments in the note section in patch 4, > > > > > > > > > > Changelog since v2 (Jerin): > > > > > - Patch 2/4 - Remove personal appeals in log messages(Anatoly) > > > > > - Patch 4/4 - Added documentation (Anatoly) > > > > > > > > > > Changelog since v1 (Jerin): > > > > > - Changed RTE_PCI_DRV_IOVA_AS_VA flag name as > > > > RTE_PCI_DRV_NEED_IOVA_AS_VA > > > > > (patch 3/4) > > > > > - Changed IOVA mode as VA for default case(patch 4/4) with > > > > documentation > > > > > - Tested the patch series on octeontx2 platform > > > > > > > > Applied, thanks Jerin, Anatoly and David for converging on a > > > > documented solution together. > > > > > > > >