From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id EBF97A058E; Tue, 24 Mar 2020 23:34:41 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id DB3D42BCE; Tue, 24 Mar 2020 23:34:40 +0100 (CET) Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by dpdk.org (Postfix) with ESMTP id 986991E34 for ; Tue, 24 Mar 2020 23:34:39 +0100 (CET) Received: from pps.filterd (m0098413.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 02OMXoGN142634; Tue, 24 Mar 2020 18:34:36 -0400 Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 2ywd2sk1rw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 24 Mar 2020 18:34:36 -0400 Received: from m0098413.ppops.net (m0098413.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 02OMYZUv144139; Tue, 24 Mar 2020 18:34:35 -0400 Received: from ppma04dal.us.ibm.com (7a.29.35a9.ip4.static.sl-reverse.com [169.53.41.122]) by mx0b-001b2d01.pphosted.com with ESMTP id 2ywd2sk1rj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 24 Mar 2020 18:34:35 -0400 Received: from pps.filterd (ppma04dal.us.ibm.com [127.0.0.1]) by ppma04dal.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 02OMNVle008226; Tue, 24 Mar 2020 22:34:35 GMT Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by ppma04dal.us.ibm.com with ESMTP id 2ywawfutes-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 24 Mar 2020 22:34:35 +0000 Received: from b01ledav006.gho.pok.ibm.com (b01ledav006.gho.pok.ibm.com [9.57.199.111]) by b01cxnp23034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 02OMYYXV52298008 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 24 Mar 2020 22:34:34 GMT Received: from b01ledav006.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4310EAC059; Tue, 24 Mar 2020 22:34:34 +0000 (GMT) Received: from b01ledav006.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 41E78AC05B; Tue, 24 Mar 2020 22:34:33 +0000 (GMT) Received: from ltc.linux.ibm.com (unknown [9.16.170.189]) by b01ledav006.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 24 Mar 2020 22:34:33 +0000 (GMT) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Tue, 24 Mar 2020 15:34:32 -0700 From: dwilder To: Jerin Jacob Cc: Aaron Conole , Michael Santana , Thomas Monjalon , Ferruh Yigit , Andrew Rybchenko , dpdk-dev , "Ruifeng Wang (Arm Technology China)" , David Marchand , David Christensen , David Wilder In-Reply-To: References: <20200220225207.30411-1-dwilder@us.ibm.com> <20200220225207.30411-2-dwilder@us.ibm.com> Message-ID: <1ef03cdb4ab3e66820a595cac0332cc3@linux.vnet.ibm.com> X-Sender: dwilder@us.ibm.com User-Agent: Roundcube Webmail/1.0.1 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138, 18.0.645 definitions=2020-03-24_09:2020-03-23, 2020-03-24 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 lowpriorityscore=0 suspectscore=0 spamscore=0 mlxscore=0 adultscore=0 priorityscore=1501 malwarescore=0 phishscore=0 mlxlogscore=999 bulkscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2003240110 Subject: Re: [dpdk-dev] [PATCH v3 1/3] eal/linux: select iova-mode va with no-huge option X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 2020-03-23 23:19, Jerin Jacob wrote: > On Mon, Mar 23, 2020 at 11:11 PM dwilder wrote: >> >> Thanks you for your review Jerin. See my responses are inline. >> >> On 2020-03-20 06:24, Jerin Jacob wrote: >> > On Fri, Feb 21, 2020 at 4:22 AM David Wilder >> > wrote: >> >> >> >> If --no-huge is set and iova-mode has not been specified force VA >> >> mode. >> >> If --no-huge and --iova-mode=PA is requested error out as this is >> >> an impossible configuration. >> >> >> >> Signed-off-by: David Wilder >> >> --- >> >> lib/librte_eal/linux/eal/eal.c | 14 ++++++++++++++ >> >> 1 file changed, 14 insertions(+) >> >> >> >> diff --git a/lib/librte_eal/linux/eal/eal.c >> >> b/lib/librte_eal/linux/eal/eal.c >> >> index 9530ee55f..d3a0a1731 100644 >> >> --- a/lib/librte_eal/linux/eal/eal.c >> >> +++ b/lib/librte_eal/linux/eal/eal.c >> >> @@ -1062,9 +1062,16 @@ rte_eal_init(int argc, char **argv) >> >> >> >> /* if no EAL option "--iova-mode=", use bus IOVA scheme >> >> */ >> >> if (internal_config.iova_mode == RTE_IOVA_DC) { >> >> + >> >> /* autodetect the IOVA mapping mode */ >> >> enum rte_iova_mode iova_mode = >> >> rte_bus_get_iommu_class(); >> >> >> >> + if (iova_mode == RTE_IOVA_PA && >> >> !rte_eal_has_hugepages()) { >> >> + iova_mode = RTE_IOVA_VA; >> >> > >> > What if igb_uio or vfio_nommu has been loaded(i.e no iommu support >> > enabled from the driver)? This would fail. >> >> Yes they would fail. If igb_uio or vfio_nommu (or any driver) cant be >> forced to VA mode it cant be used with out hugepages. Drivers can be >> available but not used therefor we print a warning message. > > I think, the warning will not be enough as the system will fail anyway. > > iova_mode == RTE_IOVA_PA && rte_eal_has_hugepages() == 0 && no_iommu == > 1 > case, we need to return error. > > iova_mode == RTE_IOVA_PA && rte_eal_has_hugepages() == 0 && no_iommu == > 0 > case warning is enough. The current code will skip the bus if iova-mode is not supported, this allow other devices to continu on. The handing of an unsupported iova-mode is done in rte_pci_probe_one_driver(). See also rte_bus_get_iommu_class() if multiple busses cant agree on iova-mode a warning is given. Here I have bound 0002:01:00.1 to igb_uio and forced iova-mode=pa, much as my code did when --no-huge is used. ./dpdk-devbind.py -s 0002:01:00.1 'Ethernet Controller X710/X557-AT 10GBASE-T 1589' drv=igb_uio unused=i40e <....> dpdk-testpmd -c 3 --iova-mode=va -w 0002:01:00.1 -- -ia EAL: Detected 160 lcore(s) EAL: Detected 2 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: No available hugepages reported in hugepages-2048kB EAL: Probing VFIO support... EAL: PCI device 0002:01:00.1 on NUMA socket 0 EAL: probe driver: 8086:1589 net_i40e EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing <<<<<<<<< EAL: Requested device 0002:01:00.1 cannot be used <.....> >> >> > >> >> + RTE_LOG(WARNING, EAL, "Some buses want 'PA' >> >> but forcing 'VA' because --no-huge is requested.\n"); >> >> + RTE_LOG(WARNING, EAL, "Not all buses may be >> >> able to initialize.\n"); >> >> + } >> >> + >> >> if (iova_mode == RTE_IOVA_DC) { >> >> RTE_LOG(DEBUG, EAL, "Buses did not request a >> >> specific IOVA mode.\n"); >> >> >> >> @@ -1111,6 +1118,13 @@ rte_eal_init(int argc, char **argv) >> >> internal_config.iova_mode; >> >> } >> >> >> >> + if (rte_eal_iova_mode() == RTE_IOVA_PA && >> >> + rte_eal_has_hugepages() == 0) { >> >> + rte_eal_init_alert("Cannot use IOVA as 'PA' with >> >> --no-huge"); >> > >> > Top of the tree already detecting this case. am I missing anything? >> > >> > [master]dell[dpdk.org] $ sudo ./build/app/test/dpdk-test -c 0x3 >> > --no-huge --iova-mode=pa >> > EAL: Detected 56 lcore(s) >> > EAL: Detected 2 NUMA nodes >> > EAL: Static memory layout is selected, amount of reserved memory can >> > be adjusted with -m or --socket-mem >> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket >> > EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not >> > available >> > EAL: Cannot use IOVA as 'PA' since physical addresses are not available >> > >> >> The check you reference is reporting that physical address are not >> available, for example no permissions to read /proc/self/pagemap. In >> this case, if --no-huge is set then PA mode is not allowed. There is >> no >> guarantee that physical address are persistent with out using >> hugepages. > > Since this check is under the following, Yes, make sense for the check. > The old command has explicit --iova-mode=pa. So it is in the > different code paths. > > /* if no EAL option "--iova-mode=", use bus IOVA scheme */ > if (internal_config.iova_mode == RTE_IOVA_DC) { > >> >> >> >> + rte_errno = EINVAL; >> >> + return -1; >> >> + } >> >> + >> >> if (rte_eal_iova_mode() == RTE_IOVA_PA && !phys_addrs) { >> >> rte_eal_init_alert("Cannot use IOVA as 'PA' since >> >> physical addresses are not available"); >> >> rte_errno = EINVAL; >> >> -- >> >> 2.25.0 >> >>