From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6B859A0C47 for ; Wed, 3 Nov 2021 19:11:31 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0E3C440E0F; Wed, 3 Nov 2021 19:11:31 +0100 (CET) Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by mails.dpdk.org (Postfix) with ESMTP id 66CCD40689 for ; Wed, 3 Nov 2021 19:11:29 +0100 (CET) Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 1A3HagUi031674 for ; Wed, 3 Nov 2021 18:11:28 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=message-id : date : subject : to : references : from : in-reply-to : content-type : content-transfer-encoding : mime-version; s=pp1; bh=NesMiTSao4HkUYV3M3SQj2qcnN6s3i4elG8e6YSKDew=; b=QdA9pSj9ewA5UZVkXTijGrUkscvnUIIJgzZEl57fD4vWzs6KaWmEg0OORqTBCxuYJLZC fHd8ELTbkpD/+vnNzviGmgKT15W9PguIzQa0UfhSfQo0BELKMngcZ31muL23b8PNyrPn jgGBibJNS2okPQ8AdcBMNubOF/Djnv+IHcLZBhEH0pIHMzlECAB7Q3cDg4swrWUxjd0I z8xDwMYF3L6ntRjcqnWW1WRhC6gqEVdaHKAVDrTeGM4/Av3bBW2Qv7s3VLBcD5CMJ9Z+ oKzgmmoGyzn3WM4iiJeJqXZIhUuYpOXWpvC+wZVNf059hHxaueNmABhrw8OKHapl2V9D +w== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 3c3wtp2fx2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 03 Nov 2021 18:11:28 +0000 Received: from m0098416.ppops.net (m0098416.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 1A3HbeLT005496 for ; Wed, 3 Nov 2021 18:11:27 GMT Received: from ppma05wdc.us.ibm.com (1b.90.2fa9.ip4.static.sl-reverse.com [169.47.144.27]) by mx0b-001b2d01.pphosted.com with ESMTP id 3c3wtp2fwp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 03 Nov 2021 18:11:27 +0000 Received: from pps.filterd (ppma05wdc.us.ibm.com [127.0.0.1]) by ppma05wdc.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 1A3HvIDZ003737; Wed, 3 Nov 2021 18:11:27 GMT Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by ppma05wdc.us.ibm.com with ESMTP id 3c0wpby6kt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 03 Nov 2021 18:11:27 +0000 Received: from b01ledav006.gho.pok.ibm.com (b01ledav006.gho.pok.ibm.com [9.57.199.111]) by b01cxnp23034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 1A3IBQiI44892548 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 3 Nov 2021 18:11:26 GMT Received: from b01ledav006.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B24CBAC05B; Wed, 3 Nov 2021 18:11:26 +0000 (GMT) Received: from b01ledav006.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 698F8AC060; Wed, 3 Nov 2021 18:11:26 +0000 (GMT) Received: from [9.211.81.124] (unknown [9.211.81.124]) by b01ledav006.gho.pok.ibm.com (Postfix) with ESMTP; Wed, 3 Nov 2021 18:11:26 +0000 (GMT) Message-ID: <0aa050c0-fa95-a4cd-bb58-ba3dfef8146d@linux.vnet.ibm.com> Date: Wed, 3 Nov 2021 11:11:28 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.2.1 Subject: Re: If or how one gets an IP address associated with a vfio-pci bound NIC Content-Language: en-US To: fwefew 4t4tg <7532yahoo@gmail.com>, users@dpdk.org References: From: David Christensen In-Reply-To: X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: ZA5SIHIwHkNPE3DL8m3FBY5nt7eKIRTa X-Proofpoint-GUID: 4hnB6UXUiGQGAhCJnou1K9n5e29arMAV Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.790,Hydra:6.0.425,FMLib:17.0.607.475 definitions=2021-11-03_06,2021-11-03_01,2020-04-07_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 spamscore=0 impostorscore=0 lowpriorityscore=0 clxscore=1011 priorityscore=1501 mlxlogscore=750 malwarescore=0 suspectscore=0 mlxscore=0 bulkscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000 definitions=main-2111030096 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On 11/2/21 4:14 PM, fwefew 4t4tg wrote: > I'm trying to use DPDK on AWS i3.metal instances. I have the code built > with AWS vfio-patches. In order to be logged into the machine on one NIC > while having a free ENA NIC for DPDK, I attached a second NIC. > ./dpdk-devbind.py is able to ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > ZjQcmQRYFpfptBannerEnd > > I'm trying to use DPDK on AWS i3.metal instances. I have the code built > with AWS vfio-patches. In order to be logged into the machine on one NIC > while having a free ENA NIC for DPDK, I attached a second NIC. > > ./dpdk-devbind.py is able to see the second NIC, and bind to it. *All > that's working fine. However, by default this 2nd NIC does not have an > IP address.* > > Meanwhile code needs a hostname or IP address of the client and server. > How do I get an IP address associated with this 2nd NIC? I don't think you understand the intent behind the DPDK framework. You're passing control of the NIC to a user application. That means you don't receive any benefits of the kernel's networking stack. The user application you use will need to handle all network services, including it's own TCP/IP stack if required. If you're using the bundled DPDK testpmd application then there's no need to assign an IP address to the interface. The testpmd app can build and send/receive ANY type of network packet, though it's mostly only used to verify functionality provided by the DPDK framework. If you're WRITING a network application then DPDK might be what you want, but if you have a specific network function in mind then you're likely looking for an application that USES DPDK. And do I need > to do some sys-admin work to ensure traffic in and out of the DPDK bound > vfio-pci NIC is kept separate from the first NIC? > > As far as I can see the correct approach is to: > > # setup second NIC to have an IP address and make sure UP before > dpdk-devbind: > * sudo ip addr add dev ens1 label ens1:1 > * sudo ip lin set ens1 u[ > > before I do DPDK bind. > > The NIC, when AWS adds it, starts off down without an IP address by default: > > ubuntu$ lspci | grep Ether > > 04:00.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) > 05:00.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) > ubuntu$ sudo ip a > 1: lo: mtu 65536 qdisc noqueue state UNKNOWN > group default qlen 1000 >     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 >     inet 127.0.0.1/8 scope host lo >        valid_lft forever preferred_lft forever >     inet6 ::1/128 scope host >        valid_lft forever preferred_lft forever > 2: ens785: mtu 9001 qdisc mq state UP > group default qlen 1000 >     link/ether 0a:0f:1f:db:ca:73 brd ff:ff:ff:ff:ff:ff >     inet 172.31.17.144/20 brd 172.31.31.255 > scope global dynamic ens785 >        valid_lft 3544sec preferred_lft 3544sec >     inet6 fe80::80f:1fff:fedb:ca73/64 scope link >        valid_lft forever preferred_lft forever > *3: ens1: mtu 1500 qdisc noop state DOWN group > default qlen 1000 >     link/ether 0a:06:15:14:95:05 brd ff:ff:ff:ff:ff:ff > * > Once I bind 'ens1' dpdk-devbind reports it as bound -AND- it no longer > appears in `ip a`: This is expected. You've removed the NIC from the kernel's control and bound it to the vfio_pci driver, which allows the NIC to be controlled entirely by a user application. > > Network devices using DPDK-compatible driver > ============================================ > 0000:05:00.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena > > Network devices using kernel driver > =================================== > 0000:04:00.0 'Elastic Network Adapter (ENA) ec20' if=ens785 drv=ena > unused=vfio-pci *Active* > > $ ip a > ubuntu@ip-172-31-17-144:~/Scripts$ ip a > 1: lo: mtu 65536 qdisc noqueue state UNKNOWN > group default qlen 1000 >     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 >     inet 127.0.0.1/8 scope host lo >        valid_lft forever preferred_lft forever >     inet6 ::1/128 scope host >        valid_lft forever preferred_lft forever > 2: ens785: mtu 9001 qdisc mq state UP > group default qlen 1000 >     link/ether 0a:0f:1f:db:ca:73 brd ff:ff:ff:ff:ff:ff >     inet 172.31.17.144/20 brd 172.31.31.255 > scope global dynamic ens785 >        valid_lft 3314sec preferred_lft 3314sec >     inet6 fe80::80f:1fff:fedb:ca73/64 scope link >        valid_lft forever preferred_lft forever Everything seems in order here. If you can share what you're trying to accomplish with DPDK we might be able to provide better guidance. Dave