Re: If or how one gets an IP address associated with a vfio-pci bound NIC

[David Christensen <drc@linux.vnet.ibm.com>]

On 11/2/21 4:14 PM, fwefew 4t4tg wrote:
> I'm trying to use DPDK on AWS i3.metal instances. I have the code built 
> with AWS vfio-patches. In order to be logged into the machine on one NIC 
> while having a free ENA NIC for DPDK, I attached a second NIC. 
> ./dpdk-devbind.py is able to ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
> ZjQcmQRYFpfptBannerEnd
> 
> I'm trying to use DPDK on AWS i3.metal instances. I have the code built 
> with AWS vfio-patches. In order to be logged into the machine on one NIC 
> while having a free ENA NIC for DPDK, I attached a second NIC.
> 
> ./dpdk-devbind.py is able to see the second NIC, and bind to it. *All 
> that's working fine. However, by default this 2nd NIC does not have an 
> IP address.*
> 
> Meanwhile code needs a hostname or IP address of the client and server. 
> How do I get an IP address associated with this 2nd NIC? 

I don't think you understand the intent behind the DPDK framework. 
You're passing control of the NIC to a user application.  That means you 
don't receive any benefits of the kernel's networking stack.  The user 
application you use will need to handle all network services, including 
it's own TCP/IP stack if required.

If you're using the bundled DPDK testpmd application then there's no 
need to assign an IP address to the interface.  The testpmd app can 
build and send/receive ANY type of network packet, though it's mostly 
only used to verify functionality provided by the DPDK framework.  If 
you're WRITING a network application then DPDK might be what you want, 
but if you have a specific network function in mind then you're likely 
looking for an application that USES DPDK.

And do I need
> to do some sys-admin work to ensure traffic in and out of the DPDK bound 
> vfio-pci NIC is kept separate from the first NIC?
> 
> As far as I can see the correct approach is to:
> 
> # setup second NIC to have an IP address and make sure UP before 
> dpdk-devbind:
> * sudo ip addr add <some-valid-ipv4-addr> dev ens1 label ens1:1
> * sudo ip lin set ens1 u[
> 
> before I do DPDK bind.
> 
> The NIC, when AWS adds it, starts off down without an IP address by default:
> 
> ubuntu$ lspci | grep Ether
> 
> 04:00.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
> 05:00.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
> ubuntu$ sudo ip a
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
> group default qlen 1000
>      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>      inet 127.0.0.1/8 <http://127.0.0.1/8> scope host lo
>         valid_lft forever preferred_lft forever
>      inet6 ::1/128 scope host
>         valid_lft forever preferred_lft forever
> 2: ens785: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP 
> group default qlen 1000
>      link/ether 0a:0f:1f:db:ca:73 brd ff:ff:ff:ff:ff:ff
>      inet 172.31.17.144/20 <http://172.31.17.144/20> brd 172.31.31.255 
> scope global dynamic ens785
>         valid_lft 3544sec preferred_lft 3544sec
>      inet6 fe80::80f:1fff:fedb:ca73/64 scope link
>         valid_lft forever preferred_lft forever
> *3: ens1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group 
> default qlen 1000
>      link/ether 0a:06:15:14:95:05 brd ff:ff:ff:ff:ff:ff
> *
> Once I bind 'ens1' dpdk-devbind reports it as bound -AND- it no longer 
> appears in `ip a`:

This is expected.  You've removed the NIC from the kernel's control and 
bound it to the vfio_pci driver, which allows the NIC to be controlled 
entirely by a user application.

> 
> Network devices using DPDK-compatible driver
> ============================================
> 0000:05:00.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena
> 
> Network devices using kernel driver
> ===================================
> 0000:04:00.0 'Elastic Network Adapter (ENA) ec20' if=ens785 drv=ena 
> unused=vfio-pci *Active*
> 
> $ ip a
> ubuntu@ip-172-31-17-144:~/Scripts$ ip a
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
> group default qlen 1000
>      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>      inet 127.0.0.1/8 <http://127.0.0.1/8> scope host lo
>         valid_lft forever preferred_lft forever
>      inet6 ::1/128 scope host
>         valid_lft forever preferred_lft forever
> 2: ens785: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP 
> group default qlen 1000
>      link/ether 0a:0f:1f:db:ca:73 brd ff:ff:ff:ff:ff:ff
>      inet 172.31.17.144/20 <http://172.31.17.144/20> brd 172.31.31.255 
> scope global dynamic ens785
>         valid_lft 3314sec preferred_lft 3314sec
>      inet6 fe80::80f:1fff:fedb:ca73/64 scope link
>         valid_lft forever preferred_lft forever

Everything seems in order here.  If you can share what you're trying to 
accomplish with DPDK we might be able to provide better guidance.

Dave