DPDK usage discussions
 help / color / mirror / Atom feed
From: David Christensen <drc@linux.vnet.ibm.com>
To: fwefew 4t4tg <7532yahoo@gmail.com>
Cc: users@dpdk.org
Subject: Re: If or how one gets an IP address associated with a vfio-pci bound NIC
Date: Thu, 4 Nov 2021 10:14:19 -0700
Message-ID: <e6abb000-1b3a-b06a-a08c-2164b29631e7@linux.vnet.ibm.com> (raw)
In-Reply-To: <CA+Tq66WyUpjG-EME-A0FTGCfJdr_0QoyWZ3K3u2qbvh1-FMUag@mail.gmail.com>

> I'd appreciate one additional bit of information if possible. Once the 
> DPDK NIC is bound to vfio-pci the DPDK Linux manual at
> https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#vfio 
> <https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#vfio> mentions 
> setup steps including:
> 
> Create the desired number of VF devices
> echo 2 > /sys/bus/pci/devices/0000:86:00.0/sriov_numvfs
> 
> My question: what is the upper bound on the number of VF devices? What's 
> the thinking process? For example,
> maybe one of these approaches makes sense?
> 
> - VF device count is bound from above by the number or RX/TX queues
> - VF device count is bound from above by the amount of on-NIC memory
> - VF device count is bound from above by manufacturer. Each NIC has some 
> max; read specs
> - VF device count is like the number of ports on a UNIX: 1000s are 
> available and what you need depends on software: how many concurrent 
> connections are needed?

Thu upper bound on Virtual Functions (VF) comes from the hardware 
itself.  It's advertised to the OS through the PCIe configuration 
register space.  You can use the lspci utility to discover this 
information.  For example, running "lspci | grep Ethernet" shows the 
NICs on my system:

0000:01:00.0 Ethernet controller: Mellanox Technologies MT28800 Family 
[ConnectX-5 Ex]
0000:01:00.1 Ethernet controller: Mellanox Technologies MT28800 Family 
[ConnectX-5 Ex]
0003:01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries 
NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)
0003:01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries 
NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)
0003:01:00.2 Ethernet controller: Broadcom Inc. and subsidiaries 
NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)
0003:01:00.3 Ethernet controller: Broadcom Inc. and subsidiaries 
NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)
0005:01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries 
NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0005:01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries 
NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0030:01:00.0 Ethernet controller: Mellanox Technologies MT28800 Family 
[ConnectX-5 Ex]
0030:01:00.1 Ethernet controller: Mellanox Technologies MT28800 Family 
[ConnectX-5 Ex]
0034:01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 
XL710 for 40GbE QSFP+ (rev 02)
0034:01:00.1 Ethernet controller: Intel Corporation Ethernet Controller 
XL710 for 40GbE QSFP+ (rev 02)

Focusing on the Intel XL710 NIC, I can look at the SR-IOV capabilities 
values:

sudo lspci -vvvv -s 0034:01:00.0
0034:01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 
XL710 for 40GbE QSFP+ (rev 02)
	Subsystem: Intel Corporation Ethernet Converged Network Adapter XL710-Q2
...
	Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency 
Link: 00
		VF offset: 16, stride: 1, Device ID: 154c
		Supported Page Size: 00000553, System Page Size: 00000010
		Region 0: Memory at 0006224000000000 (64-bit, prefetchable)
		Region 3: Memory at 0006224001000000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0

The "Total VFs" value indicates how many VFs can be enabled for this NIC 
and indicates the upper bound you can use when enabling VFs with the 
echo command you mention above.  Other NICs may have different values 
depending on their individual hardware capabilities.

> DPDK must have an API that programatically discovers the PFs and VFs per PF.

Support for SR-IOV is managed by the Linux kernel, not DPDK.  Once a VF 
is enabled under Linux, DPDK treats it just like a physical function 
(PF) NIC, assuming the poll-mode driver (PMD) written by the hardware 
manufacturer supports operating on the VF.

> Finally: is a VF device duplex (sends and receives)? Or just RX or just 
> TX only?

In my experience VFs support both send and receive.  There is also some 
Linux support for limiting bandwidth on VFs that support the capability 
(see "ip link set vf" on https://linux.die.net/man/8/ip).

Dave

      reply	other threads:[~2021-11-04 17:14 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-11-02 23:14 fwefew 4t4tg
2021-11-03 18:11 ` David Christensen
2021-11-04  1:40   ` fwefew 4t4tg
2021-11-04 17:14     ` David Christensen [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=e6abb000-1b3a-b06a-a08c-2164b29631e7@linux.vnet.ibm.com \
    --to=drc@linux.vnet.ibm.com \
    --cc=7532yahoo@gmail.com \
    --cc=users@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK usage discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://inbox.dpdk.org/users/0 users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 users users/ http://inbox.dpdk.org/users \
		users@dpdk.org
	public-inbox-index users

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.users


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git