From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id 882643772 for ; Tue, 7 May 2019 14:02:34 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 07 May 2019 05:02:34 -0700 X-ExtLoop1: 1 Received: from aburakov-mobl1.ger.corp.intel.com (HELO [10.251.90.106]) ([10.251.90.106]) by fmsmga004.fm.intel.com with ESMTP; 07 May 2019 05:02:32 -0700 To: Robert Nie , "dev@dpdk.org" References: From: "Burakov, Anatoly" Message-ID: Date: Tue, 7 May 2019 13:02:31 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] about IOVA and namespace in DPDK 19.05 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 May 2019 12:02:35 -0000 On 02-May-19 12:31 AM, Robert Nie wrote: > After some digging, I fixed it by using "--iova-mode=va". Would anyone please let me know if it was safe to use "va" instead of "pa" for veth use case? Or any performance drops? > That depends. The IOVA addresses are what you would colloquially refer to as "physical" addresses (in other words, the addresses that hardware uses to do DMA requests). The reason we call them IOVA addresses and not physical addresses is because these DMA addresses do not necessarily correspond to actual physical addresses, but can instead be remapped by platform's IOMMU to be arbitrary. The term "IOVA" covers both cases. When IOVA mode is set to "IOVA as PA", real physical addresses are used for DMA requests, while when IOVA mode is set to "IOVA as VA", then virtual addresses are used instead. The issue you were seeing with IOVA as PA mode is most likely caused by the fact that whatever it was that you were allocating, required IOVA-contiguous memory. Since 18.05, DPDK memory subsystem was reworked to be dynamic (i.e. grow and shrink memory usage on the fly), but that change came with a trade-off - IOVA contiguousness is no longer guaranteed, because we have no control over which page addresses the kernel will give us. Using IOVA as VA mode allows to sidestep this limitations, because we are no longer "given" any IOVA addresses - we set them to be whatever we want. If your driver does not depend on *real physical address* contiguousness, then IOVA as VA mode is perfectly safe to use. For example, if your driver is entirely software-based, or if you're relying on some kind of kernel infrastructure (such as what would be the case with PCAP or AF_PACKET driver) to do actual DMA requests, then using IOVA as VA mode is OK. If, on the other hand, your use case requires actual physical addresses to be contiguous (for example, if you are using KNI), it is not safe. I am only passably familiar with our AF_XDP driver, but i'm assuming that since it's technically a software driver, it should work with IOVA as VA mode just fine. However, if you run into problems, you may try running DPDK in legacy mode (using --legacy-mem EAL switch) - this will make it so that DPDK is initialized similarly to how old versions of DPDK worked, and will reserve big amounts of IOVA-contiguous memory at startup (at a cost of losing the ability to grow and shrink memory usage at runtime). -- Thanks, Anatoly From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id 0C280A0096 for ; Tue, 7 May 2019 14:02:36 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id C002537AF; Tue, 7 May 2019 14:02:35 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id 882643772 for ; Tue, 7 May 2019 14:02:34 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 07 May 2019 05:02:34 -0700 X-ExtLoop1: 1 Received: from aburakov-mobl1.ger.corp.intel.com (HELO [10.251.90.106]) ([10.251.90.106]) by fmsmga004.fm.intel.com with ESMTP; 07 May 2019 05:02:32 -0700 To: Robert Nie , "dev@dpdk.org" References: From: "Burakov, Anatoly" Message-ID: Date: Tue, 7 May 2019 13:02:31 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="UTF-8"; format="flowed" Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] about IOVA and namespace in DPDK 19.05 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190507120231.C6wTyHpBjU-NVWoJWSnY0uaSnb-zLom4JeTHv9GoJDI@z> On 02-May-19 12:31 AM, Robert Nie wrote: > After some digging, I fixed it by using "--iova-mode=va". Would anyone please let me know if it was safe to use "va" instead of "pa" for veth use case? Or any performance drops? > That depends. The IOVA addresses are what you would colloquially refer to as "physical" addresses (in other words, the addresses that hardware uses to do DMA requests). The reason we call them IOVA addresses and not physical addresses is because these DMA addresses do not necessarily correspond to actual physical addresses, but can instead be remapped by platform's IOMMU to be arbitrary. The term "IOVA" covers both cases. When IOVA mode is set to "IOVA as PA", real physical addresses are used for DMA requests, while when IOVA mode is set to "IOVA as VA", then virtual addresses are used instead. The issue you were seeing with IOVA as PA mode is most likely caused by the fact that whatever it was that you were allocating, required IOVA-contiguous memory. Since 18.05, DPDK memory subsystem was reworked to be dynamic (i.e. grow and shrink memory usage on the fly), but that change came with a trade-off - IOVA contiguousness is no longer guaranteed, because we have no control over which page addresses the kernel will give us. Using IOVA as VA mode allows to sidestep this limitations, because we are no longer "given" any IOVA addresses - we set them to be whatever we want. If your driver does not depend on *real physical address* contiguousness, then IOVA as VA mode is perfectly safe to use. For example, if your driver is entirely software-based, or if you're relying on some kind of kernel infrastructure (such as what would be the case with PCAP or AF_PACKET driver) to do actual DMA requests, then using IOVA as VA mode is OK. If, on the other hand, your use case requires actual physical addresses to be contiguous (for example, if you are using KNI), it is not safe. I am only passably familiar with our AF_XDP driver, but i'm assuming that since it's technically a software driver, it should work with IOVA as VA mode just fine. However, if you run into problems, you may try running DPDK in legacy mode (using --legacy-mem EAL switch) - this will make it so that DPDK is initialized similarly to how old versions of DPDK worked, and will reserve big amounts of IOVA-contiguous memory at startup (at a cost of losing the ability to grow and shrink memory usage at runtime). -- Thanks, Anatoly