From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 646E9A04AB; Wed, 6 Nov 2019 12:14:55 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 8C6131BFA0; Wed, 6 Nov 2019 12:14:54 +0100 (CET) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 674B71BF98 for ; Wed, 6 Nov 2019 12:14:52 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Nov 2019 03:14:51 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.68,274,1569308400"; d="scan'208";a="212750231" Received: from aburakov-mobl1.ger.corp.intel.com (HELO [10.237.220.92]) ([10.237.220.92]) by fmsmga001.fm.intel.com with ESMTP; 06 Nov 2019 03:14:49 -0800 To: David Marchand , "Wangyu (Eric)" Cc: dengxiaofeng , "dev@dpdk.org" , "ferruh.yigit@intel.com" , Linuxarm , "humin (Q)" , "Liyuan (Larry)" References: <78A93308629D474AA53B84C5879E84D24B102602@DGGEMM533-MBX.china.huawei.com> <78A93308629D474AA53B84C5879E84D24B103B0C@DGGEMM533-MBX.china.huawei.com> From: "Burakov, Anatoly" Message-ID: <8c50d8b6-4149-f101-fe90-b4c83f9b2b40@intel.com> Date: Wed, 6 Nov 2019 11:14:48 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH v2] bus/pci: resolve multiple NICs address conflicts X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 06-Nov-19 7:37 AM, David Marchand wrote: > On Wed, Nov 6, 2019 at 7:16 AM Wangyu (Eric) wrote: >> >> >> In 64K pagesize system, DPDK will read the size NIC need in uio/uio1/maps/map1/size, when the size small than pagesize(e.g.,82599 is 16K), dev->mem_resource[i].len will be 16K, but the mmap function applies for at least 1 page size, which is 64K. >> Then second NIC mmap, start address is first NIC address + 16K, which already used by first NIC. > > Do you see this issue with vfio? > > >> So if change the size to first NIC address + 64K, problem solved. > > You are hacking a description of the device resources to workaround a problem. > This patch is a no go for me. > > Maybe there is something to do with the hint passed to mmap in uio case. > Adding Anatoly to the loop. > > I did a quick code inspection for VFIO and UIO. We do the same thing in both, so both code paths can be for all intents and purposes considered equivalent. To reserve mappings for addresses, we start at some arbitrary address (find_max_va_end()), and start mapping from there. Then, we do an mmap() *and overwrite* whatever address we expected to get, and then the next address is (current.addr + current.len). The mmap() is called without MAP_FIXED, so we get an address the kernel feels comfortable for us to get. Meaning, even if the initial address hint was not page-aligned, the return value from mmap() will be page-aligned. It seems to me that your platform/kernel does not do that, and allows mmap() to return page-unaligned addresses. I would strongly suggest checking the mmap() return address on your platform (in either UIO or VFIO - they both do it about the same way). We could work around that by doing (next_addr = RTE_PTR_ALIGN(current.addr + current.len, pagesize)), but to me it seems like a bug in your kernel/mmap() implementation. This is an easy fix though, and i'm sure we can put in a workaround like i described. -- Thanks, Anatoly