From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM04-BN3-obe.outbound.protection.outlook.com (mail-oln040092009070.outbound.protection.outlook.com [40.92.9.70]) by dpdk.org (Postfix) with ESMTP id 4CCCB2082 for ; Tue, 7 May 2019 20:15:55 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=live.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=z8BAsVs1b0pC1xB1Q4ab1fi8mtFbCn4Ca8aA6UbizR0=; b=ZjHoaxZSqmrBfFs53sST042JlcQvR8Bip4uOl2CJIcG8/t71eouAlUbR+JY9AymTbsZdOPwWPwfEbvbNpJ0Lv/pY40FvuIvKIMuaiYkfSLEVDOdA1MSNz2OBe0mwVyOFjOoEyMxdEADRg+qIGoDX3qW3bllt334O+8LtgxJte104ZOKeOG77O8xE/qNN0M2wwwOKlifuAFK6l5fDYC1c/M7xyk8ivMYX52O5AmZqUWwxQzuMhqws52LBj83vDBzFbdPb2ryKTgBxGXSW5d+CGWiqE7ztP0EvVCyQ3gk4XHcb2Uv3s+SA4zGCZA2OTV/iHUv3wba2HSmVmyqhcG2H+w== Received: from CO1NAM04FT046.eop-NAM04.prod.protection.outlook.com (10.152.90.55) by CO1NAM04HT154.eop-NAM04.prod.protection.outlook.com (10.152.90.252) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.1835.13; Tue, 7 May 2019 18:15:53 +0000 Received: from CY4PR02MB2439.namprd02.prod.outlook.com (10.152.90.55) by CO1NAM04FT046.mail.protection.outlook.com (10.152.91.117) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.1835.13 via Frontend Transport; Tue, 7 May 2019 18:15:53 +0000 Received: from CY4PR02MB2439.namprd02.prod.outlook.com ([fe80::7812:b09d:65fc:d2c2]) by CY4PR02MB2439.namprd02.prod.outlook.com ([fe80::7812:b09d:65fc:d2c2%8]) with mapi id 15.20.1856.012; Tue, 7 May 2019 18:15:53 +0000 From: Robert Nie To: "Burakov, Anatoly" , "dev@dpdk.org" Thread-Topic: [dpdk-dev] about IOVA and namespace in DPDK 19.05 Thread-Index: AQHVAHDqWdk4esV5Ckm2kNo4V1OjLqZfmJ2AgABeNPs= Date: Tue, 7 May 2019 18:15:53 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-incomingtopheadermarker: OriginalChecksum:7BB42983C8069220379F8C22F9E945C1C75A49C57AED5E40181F79B9791C06CD; UpperCasedChecksum:EE7AC519AEEB3E542717E24ADEDB704C063AB28489E30CB380782B3131691AC3; SizeAsReceived:6831; Count:43 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [ojdSP1ZchM5GBHISekl+oRL3T8TRi7fE] x-ms-publictraffictype: Email x-incomingheadercount: 43 x-eopattributedmessage: 0 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(5050001)(7020095)(20181119110)(201702061078)(5061506573)(5061507331)(1603103135)(2017031320274)(2017031323274)(2017031324274)(2017031322404)(1601125500)(1603101475)(1701031045); SRVR:CO1NAM04HT154; x-ms-traffictypediagnostic: CO1NAM04HT154: x-microsoft-antispam-message-info: DG4hYulvtpSMpe4bOZGLMNDDXMjcNd5YZ9CxKUpqCo1AfhIkvnWtlAXW3yK9xUJgJRnJu6g2xSDo0jMopv8y5pLM0XMb3UVKx52aZM15CoK3L4ZPnnq6dDVPFEMsS0PMUXHYO72FPC4cpVonFyWEmqjEiamOt0+llLHzqJ4vaTtAjI18e0maKD+kUDbamO8B MIME-Version: 1.0 X-OriginatorOrg: live.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 90743fd7-4921-4ab3-c237-08d6d3180b27 X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-originalarrivaltime: 07 May 2019 18:15:53.8113 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1NAM04HT154 X-Mailman-Approved-At: Thu, 09 May 2019 09:13:43 +0200 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] about IOVA and namespace in DPDK 19.05 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 May 2019 18:15:55 -0000 Anatoly, Thank you very much for the detail elaboration on IOVA. Glad to know the me= mory allocation model is changed since 18.05 which knowledge is fresh to me= . Thanks, Robert ________________________________ From: Burakov, Anatoly Sent: Tuesday, May 7, 2019 5:02 AM To: Robert Nie; dev@dpdk.org Subject: Re: [dpdk-dev] about IOVA and namespace in DPDK 19.05 On 02-May-19 12:31 AM, Robert Nie wrote: > After some digging, I fixed it by using "--iova-mode=3Dva". Would anyone = please let me know if it was safe to use "va" instead of "pa" for veth use = case? Or any performance drops? > That depends. The IOVA addresses are what you would colloquially refer to as "physical" addresses (in other words, the addresses that hardware uses to do DMA requests). The reason we call them IOVA addresses and not physical addresses is because these DMA addresses do not necessarily correspond to actual physical addresses, but can instead be remapped by platform's IOMMU to be arbitrary. The term "IOVA" covers both cases. When IOVA mode is set to "IOVA as PA", real physical addresses are used for DMA requests, while when IOVA mode is set to "IOVA as VA", then virtual addresses are used instead. The issue you were seeing with IOVA as PA mode is most likely caused by the fact that whatever it was that you were allocating, required IOVA-contiguous memory. Since 18.05, DPDK memory subsystem was reworked to be dynamic (i.e. grow and shrink memory usage on the fly), but that change came with a trade-off - IOVA contiguousness is no longer guaranteed, because we have no control over which page addresses the kernel will give us. Using IOVA as VA mode allows to sidestep this limitations, because we are no longer "given" any IOVA addresses - we set them to be whatever we want. If your driver does not depend on *real physical address* contiguousness, then IOVA as VA mode is perfectly safe to use. For example, if your driver is entirely software-based, or if you're relying on some kind of kernel infrastructure (such as what would be the case with PCAP or AF_PACKET driver) to do actual DMA requests, then using IOVA as VA mode is OK. If, on the other hand, your use case requires actual physical addresses to be contiguous (for example, if you are using KNI), it is not safe. I am only passably familiar with our AF_XDP driver, but i'm assuming that since it's technically a software driver, it should work with IOVA as VA mode just fine. However, if you run into problems, you may try running DPDK in legacy mode (using --legacy-mem EAL switch) - this will make it so that DPDK is initialized similarly to how old versions of DPDK worked, and will reserve big amounts of IOVA-contiguous memory at startup (at a cost of losing the ability to grow and shrink memory usage at runtime). -- Thanks, Anatoly From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id A62EEA0096 for ; Thu, 9 May 2019 09:13:44 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 74F2F4C74; Thu, 9 May 2019 09:13:43 +0200 (CEST) Received: from NAM04-BN3-obe.outbound.protection.outlook.com (mail-oln040092009070.outbound.protection.outlook.com [40.92.9.70]) by dpdk.org (Postfix) with ESMTP id 4CCCB2082 for ; Tue, 7 May 2019 20:15:55 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=live.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=z8BAsVs1b0pC1xB1Q4ab1fi8mtFbCn4Ca8aA6UbizR0=; b=ZjHoaxZSqmrBfFs53sST042JlcQvR8Bip4uOl2CJIcG8/t71eouAlUbR+JY9AymTbsZdOPwWPwfEbvbNpJ0Lv/pY40FvuIvKIMuaiYkfSLEVDOdA1MSNz2OBe0mwVyOFjOoEyMxdEADRg+qIGoDX3qW3bllt334O+8LtgxJte104ZOKeOG77O8xE/qNN0M2wwwOKlifuAFK6l5fDYC1c/M7xyk8ivMYX52O5AmZqUWwxQzuMhqws52LBj83vDBzFbdPb2ryKTgBxGXSW5d+CGWiqE7ztP0EvVCyQ3gk4XHcb2Uv3s+SA4zGCZA2OTV/iHUv3wba2HSmVmyqhcG2H+w== Received: from CO1NAM04FT046.eop-NAM04.prod.protection.outlook.com (10.152.90.55) by CO1NAM04HT154.eop-NAM04.prod.protection.outlook.com (10.152.90.252) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.1835.13; Tue, 7 May 2019 18:15:53 +0000 Received: from CY4PR02MB2439.namprd02.prod.outlook.com (10.152.90.55) by CO1NAM04FT046.mail.protection.outlook.com (10.152.91.117) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.1835.13 via Frontend Transport; Tue, 7 May 2019 18:15:53 +0000 Received: from CY4PR02MB2439.namprd02.prod.outlook.com ([fe80::7812:b09d:65fc:d2c2]) by CY4PR02MB2439.namprd02.prod.outlook.com ([fe80::7812:b09d:65fc:d2c2%8]) with mapi id 15.20.1856.012; Tue, 7 May 2019 18:15:53 +0000 From: Robert Nie To: "Burakov, Anatoly" , "dev@dpdk.org" Thread-Topic: [dpdk-dev] about IOVA and namespace in DPDK 19.05 Thread-Index: AQHVAHDqWdk4esV5Ckm2kNo4V1OjLqZfmJ2AgABeNPs= Date: Tue, 7 May 2019 18:15:53 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-incomingtopheadermarker: OriginalChecksum:7BB42983C8069220379F8C22F9E945C1C75A49C57AED5E40181F79B9791C06CD; UpperCasedChecksum:EE7AC519AEEB3E542717E24ADEDB704C063AB28489E30CB380782B3131691AC3; SizeAsReceived:6831; Count:43 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [ojdSP1ZchM5GBHISekl+oRL3T8TRi7fE] x-ms-publictraffictype: Email x-incomingheadercount: 43 x-eopattributedmessage: 0 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(5050001)(7020095)(20181119110)(201702061078)(5061506573)(5061507331)(1603103135)(2017031320274)(2017031323274)(2017031324274)(2017031322404)(1601125500)(1603101475)(1701031045); SRVR:CO1NAM04HT154; x-ms-traffictypediagnostic: CO1NAM04HT154: x-microsoft-antispam-message-info: DG4hYulvtpSMpe4bOZGLMNDDXMjcNd5YZ9CxKUpqCo1AfhIkvnWtlAXW3yK9xUJgJRnJu6g2xSDo0jMopv8y5pLM0XMb3UVKx52aZM15CoK3L4ZPnnq6dDVPFEMsS0PMUXHYO72FPC4cpVonFyWEmqjEiamOt0+llLHzqJ4vaTtAjI18e0maKD+kUDbamO8B MIME-Version: 1.0 X-OriginatorOrg: live.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 90743fd7-4921-4ab3-c237-08d6d3180b27 X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-originalarrivaltime: 07 May 2019 18:15:53.8113 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1NAM04HT154 X-Mailman-Approved-At: Thu, 09 May 2019 09:13:43 +0200 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] about IOVA and namespace in DPDK 19.05 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190507181553.hedajNS-sYMa-ctspLgdo5t56y0WG2i1beZoAm7xr7g@z> Anatoly, Thank you very much for the detail elaboration on IOVA. Glad to know the me= mory allocation model is changed since 18.05 which knowledge is fresh to me= . Thanks, Robert ________________________________ From: Burakov, Anatoly Sent: Tuesday, May 7, 2019 5:02 AM To: Robert Nie; dev@dpdk.org Subject: Re: [dpdk-dev] about IOVA and namespace in DPDK 19.05 On 02-May-19 12:31 AM, Robert Nie wrote: > After some digging, I fixed it by using "--iova-mode=3Dva". Would anyone = please let me know if it was safe to use "va" instead of "pa" for veth use = case? Or any performance drops? > That depends. The IOVA addresses are what you would colloquially refer to as "physical" addresses (in other words, the addresses that hardware uses to do DMA requests). The reason we call them IOVA addresses and not physical addresses is because these DMA addresses do not necessarily correspond to actual physical addresses, but can instead be remapped by platform's IOMMU to be arbitrary. The term "IOVA" covers both cases. When IOVA mode is set to "IOVA as PA", real physical addresses are used for DMA requests, while when IOVA mode is set to "IOVA as VA", then virtual addresses are used instead. The issue you were seeing with IOVA as PA mode is most likely caused by the fact that whatever it was that you were allocating, required IOVA-contiguous memory. Since 18.05, DPDK memory subsystem was reworked to be dynamic (i.e. grow and shrink memory usage on the fly), but that change came with a trade-off - IOVA contiguousness is no longer guaranteed, because we have no control over which page addresses the kernel will give us. Using IOVA as VA mode allows to sidestep this limitations, because we are no longer "given" any IOVA addresses - we set them to be whatever we want. If your driver does not depend on *real physical address* contiguousness, then IOVA as VA mode is perfectly safe to use. For example, if your driver is entirely software-based, or if you're relying on some kind of kernel infrastructure (such as what would be the case with PCAP or AF_PACKET driver) to do actual DMA requests, then using IOVA as VA mode is OK. If, on the other hand, your use case requires actual physical addresses to be contiguous (for example, if you are using KNI), it is not safe. I am only passably familiar with our AF_XDP driver, but i'm assuming that since it's technically a software driver, it should work with IOVA as VA mode just fine. However, if you run into problems, you may try running DPDK in legacy mode (using --legacy-mem EAL switch) - this will make it so that DPDK is initialized similarly to how old versions of DPDK worked, and will reserve big amounts of IOVA-contiguous memory at startup (at a cost of losing the ability to grow and shrink memory usage at runtime). -- Thanks, Anatoly