If or how one gets an IP address associated with a vfio-pci bound NIC

If or how one gets an IP address associated with a vfio-pci bound NIC

[fwefew 4t4tg]

[-- Attachment #1: Type: text/plain, Size: 3110 bytes --]

I'm trying to use DPDK on AWS i3.metal instances. I have the code built
with AWS vfio-patches. In order to be logged into the machine on one NIC
while having a free ENA NIC for DPDK, I attached a second NIC.

./dpdk-devbind.py is able to see the second NIC, and bind to it. *All
that's working fine. However, by default this 2nd NIC does not have an IP
address.*

Meanwhile code needs a hostname or IP address of the client and server. How
do I get an IP address associated with this 2nd NIC? And do I need to do
some sys-admin work to ensure traffic in and out of the DPDK bound vfio-pci
NIC is kept separate from the first NIC?

As far as I can see the correct approach is to:

# setup second NIC to have an IP address and make sure UP before
dpdk-devbind:
* sudo ip addr add <some-valid-ipv4-addr> dev ens1 label ens1:1
* sudo ip lin set ens1 u[

before I do DPDK bind.

The NIC, when AWS adds it, starts off down without an IP address by default:

ubuntu$ lspci | grep Ether
04:00.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
05:00.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
ubuntu$ sudo ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group
default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens785: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP
group default qlen 1000
    link/ether 0a:0f:1f:db:ca:73 brd ff:ff:ff:ff:ff:ff
    inet 172.31.17.144/20 brd 172.31.31.255 scope global dynamic ens785
       valid_lft 3544sec preferred_lft 3544sec
    inet6 fe80::80f:1fff:fedb:ca73/64 scope link
       valid_lft forever preferred_lft forever


*3: ens1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000    link/ether 0a:06:15:14:95:05 brd ff:ff:ff:ff:ff:ff*
Once I bind 'ens1' dpdk-devbind reports it as bound -AND- it no longer
appears in `ip a`:

Network devices using DPDK-compatible driver
============================================
0000:05:00.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:04:00.0 'Elastic Network Adapter (ENA) ec20' if=ens785 drv=ena
unused=vfio-pci *Active*

$ ip a
ubuntu@ip-172-31-17-144:~/Scripts$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group
default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens785: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP
group default qlen 1000
    link/ether 0a:0f:1f:db:ca:73 brd ff:ff:ff:ff:ff:ff
    inet 172.31.17.144/20 brd 172.31.31.255 scope global dynamic ens785
       valid_lft 3314sec preferred_lft 3314sec
    inet6 fe80::80f:1fff:fedb:ca73/64 scope link
       valid_lft forever preferred_lft forever

[-- Attachment #2: Type: text/html, Size: 4084 bytes --]

Re: If or how one gets an IP address associated with a vfio-pci bound NIC

[David Christensen]

On 11/2/21 4:14 PM, fwefew 4t4tg wrote:
> I'm trying to use DPDK on AWS i3.metal instances. I have the code built 
> with AWS vfio-patches. In order to be logged into the machine on one NIC 
> while having a free ENA NIC for DPDK, I attached a second NIC. 
> ./dpdk-devbind.py is able to ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
> ZjQcmQRYFpfptBannerEnd
> 
> I'm trying to use DPDK on AWS i3.metal instances. I have the code built 
> with AWS vfio-patches. In order to be logged into the machine on one NIC 
> while having a free ENA NIC for DPDK, I attached a second NIC.
> 
> ./dpdk-devbind.py is able to see the second NIC, and bind to it. *All 
> that's working fine. However, by default this 2nd NIC does not have an 
> IP address.*
> 
> Meanwhile code needs a hostname or IP address of the client and server. 
> How do I get an IP address associated with this 2nd NIC? 

I don't think you understand the intent behind the DPDK framework. 
You're passing control of the NIC to a user application.  That means you 
don't receive any benefits of the kernel's networking stack.  The user 
application you use will need to handle all network services, including 
it's own TCP/IP stack if required.

If you're using the bundled DPDK testpmd application then there's no 
need to assign an IP address to the interface.  The testpmd app can 
build and send/receive ANY type of network packet, though it's mostly 
only used to verify functionality provided by the DPDK framework.  If 
you're WRITING a network application then DPDK might be what you want, 
but if you have a specific network function in mind then you're likely 
looking for an application that USES DPDK.

And do I need
> to do some sys-admin work to ensure traffic in and out of the DPDK bound 
> vfio-pci NIC is kept separate from the first NIC?
> 
> As far as I can see the correct approach is to:
> 
> # setup second NIC to have an IP address and make sure UP before 
> dpdk-devbind:
> * sudo ip addr add <some-valid-ipv4-addr> dev ens1 label ens1:1
> * sudo ip lin set ens1 u[
> 
> before I do DPDK bind.
> 
> The NIC, when AWS adds it, starts off down without an IP address by default:
> 
> ubuntu$ lspci | grep Ether
> 
> 04:00.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
> 05:00.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
> ubuntu$ sudo ip a
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
> group default qlen 1000
>      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>      inet 127.0.0.1/8 <http://127.0.0.1/8> scope host lo
>         valid_lft forever preferred_lft forever
>      inet6 ::1/128 scope host
>         valid_lft forever preferred_lft forever
> 2: ens785: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP 
> group default qlen 1000
>      link/ether 0a:0f:1f:db:ca:73 brd ff:ff:ff:ff:ff:ff
>      inet 172.31.17.144/20 <http://172.31.17.144/20> brd 172.31.31.255 
> scope global dynamic ens785
>         valid_lft 3544sec preferred_lft 3544sec
>      inet6 fe80::80f:1fff:fedb:ca73/64 scope link
>         valid_lft forever preferred_lft forever
> *3: ens1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group 
> default qlen 1000
>      link/ether 0a:06:15:14:95:05 brd ff:ff:ff:ff:ff:ff
> *
> Once I bind 'ens1' dpdk-devbind reports it as bound -AND- it no longer 
> appears in `ip a`:

This is expected.  You've removed the NIC from the kernel's control and 
bound it to the vfio_pci driver, which allows the NIC to be controlled 
entirely by a user application.

> 
> Network devices using DPDK-compatible driver
> ============================================
> 0000:05:00.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena
> 
> Network devices using kernel driver
> ===================================
> 0000:04:00.0 'Elastic Network Adapter (ENA) ec20' if=ens785 drv=ena 
> unused=vfio-pci *Active*
> 
> $ ip a
> ubuntu@ip-172-31-17-144:~/Scripts$ ip a
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
> group default qlen 1000
>      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>      inet 127.0.0.1/8 <http://127.0.0.1/8> scope host lo
>         valid_lft forever preferred_lft forever
>      inet6 ::1/128 scope host
>         valid_lft forever preferred_lft forever
> 2: ens785: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP 
> group default qlen 1000
>      link/ether 0a:0f:1f:db:ca:73 brd ff:ff:ff:ff:ff:ff
>      inet 172.31.17.144/20 <http://172.31.17.144/20> brd 172.31.31.255 
> scope global dynamic ens785
>         valid_lft 3314sec preferred_lft 3314sec
>      inet6 fe80::80f:1fff:fedb:ca73/64 scope link
>         valid_lft forever preferred_lft forever

Everything seems in order here.  If you can share what you're trying to 
accomplish with DPDK we might be able to provide better guidance.

Dave

Re: If or how one gets an IP address associated with a vfio-pci bound NIC

[fwefew 4t4tg]

[-- Attachment #1: Type: text/plain, Size: 1240 bytes --]

>
> David wrote:
> >I don't think you understand the intent behind the DPDK framework.
>
> Thank you. Indeed I stand corrected. Associating an IP address with the
DPDK NIC works from a bad frame on the problem.

I'd appreciate one additional bit of information if possible. Once the DPDK
NIC is bound to vfio-pci the DPDK Linux manual at
https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#vfio mentions
setup steps including:

Create the desired number of VF devices
echo 2 > /sys/bus/pci/devices/0000:86:00.0/sriov_numvfs

My question: what is the upper bound on the number of VF devices? What's
the thinking process? For example,
maybe one of these approaches makes sense?

- VF device count is bound from above by the number or RX/TX queues
- VF device count is bound from above by the amount of on-NIC memory
- VF device count is bound from above by manufacturer. Each NIC has some
max; read specs
- VF device count is like the number of ports on a UNIX: 1000s are
available and what you need depends on software: how many concurrent
connections are needed?

DPDK must have an API that programatically discovers the PFs and VFs per PF.

Finally: is a VF device duplex (sends and receives)? Or just RX or just TX
only?

Thank you.

[-- Attachment #2: Type: text/html, Size: 1631 bytes --]

Re: If or how one gets an IP address associated with a vfio-pci bound NIC

[David Christensen]

> I'd appreciate one additional bit of information if possible. Once the 
> DPDK NIC is bound to vfio-pci the DPDK Linux manual at
> https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#vfio 
> <https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#vfio> mentions 
> setup steps including:
> 
> Create the desired number of VF devices
> echo 2 > /sys/bus/pci/devices/0000:86:00.0/sriov_numvfs
> 
> My question: what is the upper bound on the number of VF devices? What's 
> the thinking process? For example,
> maybe one of these approaches makes sense?
> 
> - VF device count is bound from above by the number or RX/TX queues
> - VF device count is bound from above by the amount of on-NIC memory
> - VF device count is bound from above by manufacturer. Each NIC has some 
> max; read specs
> - VF device count is like the number of ports on a UNIX: 1000s are 
> available and what you need depends on software: how many concurrent 
> connections are needed?

Thu upper bound on Virtual Functions (VF) comes from the hardware 
itself.  It's advertised to the OS through the PCIe configuration 
register space.  You can use the lspci utility to discover this 
information.  For example, running "lspci | grep Ethernet" shows the 
NICs on my system:

0000:01:00.0 Ethernet controller: Mellanox Technologies MT28800 Family 
[ConnectX-5 Ex]
0000:01:00.1 Ethernet controller: Mellanox Technologies MT28800 Family 
[ConnectX-5 Ex]
0003:01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries 
NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)
0003:01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries 
NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)
0003:01:00.2 Ethernet controller: Broadcom Inc. and subsidiaries 
NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)
0003:01:00.3 Ethernet controller: Broadcom Inc. and subsidiaries 
NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)
0005:01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries 
NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0005:01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries 
NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0030:01:00.0 Ethernet controller: Mellanox Technologies MT28800 Family 
[ConnectX-5 Ex]
0030:01:00.1 Ethernet controller: Mellanox Technologies MT28800 Family 
[ConnectX-5 Ex]
0034:01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 
XL710 for 40GbE QSFP+ (rev 02)
0034:01:00.1 Ethernet controller: Intel Corporation Ethernet Controller 
XL710 for 40GbE QSFP+ (rev 02)

Focusing on the Intel XL710 NIC, I can look at the SR-IOV capabilities 
values:

sudo lspci -vvvv -s 0034:01:00.0
0034:01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 
XL710 for 40GbE QSFP+ (rev 02)
	Subsystem: Intel Corporation Ethernet Converged Network Adapter XL710-Q2
...
	Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency 
Link: 00
		VF offset: 16, stride: 1, Device ID: 154c
		Supported Page Size: 00000553, System Page Size: 00000010
		Region 0: Memory at 0006224000000000 (64-bit, prefetchable)
		Region 3: Memory at 0006224001000000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0

The "Total VFs" value indicates how many VFs can be enabled for this NIC 
and indicates the upper bound you can use when enabling VFs with the 
echo command you mention above.  Other NICs may have different values 
depending on their individual hardware capabilities.

> DPDK must have an API that programatically discovers the PFs and VFs per PF.

Support for SR-IOV is managed by the Linux kernel, not DPDK.  Once a VF 
is enabled under Linux, DPDK treats it just like a physical function 
(PF) NIC, assuming the poll-mode driver (PMD) written by the hardware 
manufacturer supports operating on the VF.

> Finally: is a VF device duplex (sends and receives)? Or just RX or just 
> TX only?

In my experience VFs support both send and receive.  There is also some 
Linux support for limiting bandwidth on VFs that support the capability 
(see "ip link set vf" on https://linux.die.net/man/8/ip).

Dave