From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7D144A0C5A for ; Thu, 4 Nov 2021 18:14:26 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6817A41223; Thu, 4 Nov 2021 18:14:26 +0100 (CET) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by mails.dpdk.org (Postfix) with ESMTP id A8FF5411C9 for ; Thu, 4 Nov 2021 18:14:24 +0100 (CET) Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 1A4H1dhi010244 for ; Thu, 4 Nov 2021 17:14:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=message-id : date : subject : to : cc : references : from : in-reply-to : content-type : content-transfer-encoding : mime-version; s=pp1; bh=ePpoXbAM6bgJ8CvIRYQmzWDYUTrT+XUFZzA5Ek6gHz0=; b=dCz9H4EI8kZ5kOKkdK3A5JSQPkF5orhPuW1hen5ARV9CAr2/81A8HeLbVpkCgBsh4P77 LdkScaEPP/an4JOA9khIYL66lUdu741jUbuqbnERO9QI2fkxSbw2/ezE+6JyDAO4W1fw 22suGdC85ig1OpJOlNP10vwj334ZS0hgRKTxbmjzfsJ8EOlQyUTZ89tEUAcHGcO/FX4o aMV0uLCSEuRQJ/9xajKNU75ngg+kThaDRkb6a3Eksh72kaw2V1vgwStu7Wklu2T5KzCf gxU91Kz9PelFlxC5wPfZtbRB2BmE0JhOZbmoxfTioyNiSBnDkdhYg1kwYqfnT+iodvdY VQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3c4ftup52h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 04 Nov 2021 17:14:23 +0000 Received: from m0098393.ppops.net (m0098393.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 1A4H1ou7011183 for ; Thu, 4 Nov 2021 17:14:23 GMT Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com with ESMTP id 3c4ftup52b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 04 Nov 2021 17:14:23 +0000 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 1A4H88sA017278; Thu, 4 Nov 2021 17:14:22 GMT Received: from b03cxnp07027.gho.boulder.ibm.com (b03cxnp07027.gho.boulder.ibm.com [9.17.130.14]) by ppma01dal.us.ibm.com with ESMTP id 3c0wpdx4dr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 04 Nov 2021 17:14:22 +0000 Received: from b03ledav006.gho.boulder.ibm.com (b03ledav006.gho.boulder.ibm.com [9.17.130.237]) by b03cxnp07027.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 1A4HEKh436700478 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 4 Nov 2021 17:14:20 GMT Received: from b03ledav006.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A763DC605B; Thu, 4 Nov 2021 17:14:20 +0000 (GMT) Received: from b03ledav006.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8117EC6066; Thu, 4 Nov 2021 17:14:20 +0000 (GMT) Received: from [9.211.89.93] (unknown [9.211.89.93]) by b03ledav006.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 4 Nov 2021 17:14:20 +0000 (GMT) Message-ID: Date: Thu, 4 Nov 2021 10:14:19 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.2.1 Subject: Re: If or how one gets an IP address associated with a vfio-pci bound NIC Content-Language: en-US To: fwefew 4t4tg <7532yahoo@gmail.com> Cc: users@dpdk.org References: <0aa050c0-fa95-a4cd-bb58-ba3dfef8146d@linux.vnet.ibm.com> From: David Christensen In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 9EEi8XffDo96Pqz0UalFns_gRo3RUUfg X-Proofpoint-GUID: pqsMOuShfeda3maEtXHRpt6e_614BQHz Content-Transfer-Encoding: 8bit X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.790,Hydra:6.0.425,FMLib:17.0.607.475 definitions=2021-11-04_05,2021-11-03_01,2020-04-07_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 clxscore=1015 mlxscore=0 malwarescore=0 lowpriorityscore=0 adultscore=0 phishscore=0 priorityscore=1501 impostorscore=0 suspectscore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000 definitions=main-2111040064 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org > I'd appreciate one additional bit of information if possible. Once the > DPDK NIC is bound to vfio-pci the DPDK Linux manual at > https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#vfio > mentions > setup steps including: > > Create the desired number of VF devices > echo 2 > /sys/bus/pci/devices/0000:86:00.0/sriov_numvfs > > My question: what is the upper bound on the number of VF devices? What's > the thinking process? For example, > maybe one of these approaches makes sense? > > - VF device count is bound from above by the number or RX/TX queues > - VF device count is bound from above by the amount of on-NIC memory > - VF device count is bound from above by manufacturer. Each NIC has some > max; read specs > - VF device count is like the number of ports on a UNIX: 1000s are > available and what you need depends on software: how many concurrent > connections are needed? Thu upper bound on Virtual Functions (VF) comes from the hardware itself. It's advertised to the OS through the PCIe configuration register space. You can use the lspci utility to discover this information. For example, running "lspci | grep Ethernet" shows the NICs on my system: 0000:01:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] 0000:01:00.1 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] 0003:01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10) 0003:01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10) 0003:01:00.2 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10) 0003:01:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10) 0005:01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01) 0005:01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01) 0030:01:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] 0030:01:00.1 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] 0034:01:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02) 0034:01:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02) Focusing on the Intel XL710 NIC, I can look at the SR-IOV capabilities values: sudo lspci -vvvv -s 0034:01:00.0 0034:01:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02) Subsystem: Intel Corporation Ethernet Converged Network Adapter XL710-Q2 ... Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV) IOVCap: Migration-, Interrupt Message Number: 000 IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ IOVSta: Migration- Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00 VF offset: 16, stride: 1, Device ID: 154c Supported Page Size: 00000553, System Page Size: 00000010 Region 0: Memory at 0006224000000000 (64-bit, prefetchable) Region 3: Memory at 0006224001000000 (64-bit, prefetchable) VF Migration: offset: 00000000, BIR: 0 The "Total VFs" value indicates how many VFs can be enabled for this NIC and indicates the upper bound you can use when enabling VFs with the echo command you mention above. Other NICs may have different values depending on their individual hardware capabilities. > DPDK must have an API that programatically discovers the PFs and VFs per PF. Support for SR-IOV is managed by the Linux kernel, not DPDK. Once a VF is enabled under Linux, DPDK treats it just like a physical function (PF) NIC, assuming the poll-mode driver (PMD) written by the hardware manufacturer supports operating on the VF. > Finally: is a VF device duplex (sends and receives)? Or just RX or just > TX only? In my experience VFs support both send and receive. There is also some Linux support for limiting bandwidth on VFs that support the capability (see "ip link set vf" on https://linux.die.net/man/8/ip). Dave