From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 80957A04B9; Mon, 11 Nov 2019 14:07:09 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id ACF91CF3; Mon, 11 Nov 2019 14:07:08 +0100 (CET) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by dpdk.org (Postfix) with ESMTP id BDA59A3 for ; Mon, 11 Nov 2019 14:07:06 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Nov 2019 05:07:05 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.68,293,1569308400"; d="scan'208";a="197664904" Received: from aburakov-mobl1.ger.corp.intel.com (HELO [10.237.220.92]) ([10.237.220.92]) by orsmga008.jf.intel.com with ESMTP; 11 Nov 2019 05:07:03 -0800 To: "Wangyu (Eric)" , David Marchand Cc: "dev@dpdk.org" , "ferruh.yigit@intel.com" , Linuxarm , "humin (Q)" , "Liyuan (Larry)" , dengxiaofeng References: <78A93308629D474AA53B84C5879E84D24B102602@DGGEMM533-MBX.china.huawei.com> <78A93308629D474AA53B84C5879E84D24B103B0C@DGGEMM533-MBX.china.huawei.com> <8c50d8b6-4149-f101-fe90-b4c83f9b2b40@intel.com> <78A93308629D474AA53B84C5879E84D24B104514@DGGEMM533-MBX.china.huawei.com> <9c8852bc-fb47-e06b-5761-40e2a56faeec@intel.com> <78A93308629D474AA53B84C5879E84D24B107AE2@DGGEMM533-MBX.china.huawei.com> From: "Burakov, Anatoly" Message-ID: Date: Mon, 11 Nov 2019 13:07:01 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <78A93308629D474AA53B84C5879E84D24B107AE2@DGGEMM533-MBX.china.huawei.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] =?utf-8?b?562U5aSNOiDnrZTlpI06ICBbUEFUQ0ggdjJdIGJ1cy9w?= =?utf-8?q?ci=3A_resolve_multiple_NICs_address_conflicts?= X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 11-Nov-19 9:37 AM, Wangyu (Eric) wrote: > > Sorry, I didn't explain it clearly, and I will explain this problem step by step. > > Precondition, we have a 64K page size system and two 82599 NICs. The memory required for each NIC is as follows: > Map0 : size = 0x0000000000400000 > Map1 : size = 0x0000000000004000 > > 1. Primary process start, process mmap() first NIC map0, mmap()'s input address is 0x8202000000, and output address is 0x8202000000, size is 0x0000000000400000, next_addr is 0x8202400000, mmap() executed correctly. > > 2. Primary mmap() first NIC map1, mmap()'s input address is 0x8202400000, and output address is 0x8202400000, size is 0x0000000000004000, next_addr is 0x8202404000, now mmap() applied from 0x8202400000 to 0x8202410000 actually(because page size is 64K), and next_addr is 0x8202404000, but mmap() executed correctly. > > 3. Primary mmap() second NIC map0, mmap()'s input address is 0x8202404000, but it's conflict, so output address is 0xffffbcdc0000(system assigned), size is 0x0000000000400000, next_addr is 0xffffbd1c0000, now the address is abnormal, and mmap() executed correctly. > > 4. Primary mmap() second NIC map1, mmap()'s input address is 0xffffbd1c0000, and output address is 0xffffbcdb0000 (system assigned), size is 0x0000000000004000, now the address is abnormal, and mmap() executed correctly. > > 5. Secondary process start, process mmap() first NIC map0, it's normal. > > 6. Secondary process mmap() first NIC map1, it's normal. > > 7. Secondary process mmap() second NIC map0, mmap()'s input address is 0xffffbcdc0000, but it's conflict on secondary process, so we get another address, but secondary will check if the input address is equal with output address, it's not equal, so secondary will exit with " Cannot mmap device resource file %s to address: %p ". > > > Now I use (next_addr = RTE_PTR_ALIGN(current.addr + current.len, pagesize)) to solve the problem, and it worked. If it is right, I will submit a patch later. > > By the way, I made a mistake, the problem won't happen on VFIO, because VFIO don't apply for 16K memory, only apply for 4M size(map0).But I think VFIO also needs to be modified. > OK, that makes more sense. In that case, aligning next address on page boundary is the right approach. -- Thanks, Anatoly