[dpdk-users] identifying physical dpdk-ports for use with OVS

[dpdk-users] identifying physical dpdk-ports for use with OVS

[Shivaram Mysore]

Hello,
I wanted to just share info about how we can identify DPDK ports for use
with OVS.

Initially, when we do:

# *dpdk-devbind --status*
Network devices using kernel driver
===================================
...
0000:82:00.0 'I350 Gigabit Network Connection' if=enp130s0f0 drv=igb
unused=vfio-pci
0000:82:00.1 'I350 Gigabit Network Connection' if=enp130s0f1 drv=igb
unused=vfio-pci
0000:82:00.2 'I350 Gigabit Network Connection' if=enp130s0f2 drv=igb
unused=vfio-pci
0000:82:00.3 'I350 Gigabit Network Connection' if=enp130s0f3 drv=igb
unused=vfio-pci
0000:83:00.0 'I350 Gigabit Network Connection' if=ens2f0 drv=igb
unused=vfio-pci
0000:83:00.1 'I350 Gigabit Network Connection' if=ens2f1 drv=igb
unused=vfio-pci
0000:83:00.2 'I350 Gigabit Network Connection' if=ens2f2 drv=igb
unused=vfio-pci
0000:83:00.3 'I350 Gigabit Network Connection' if=ens2f3 drv=igb
unused=vfio-pci
...

Then we do:
# *dpdk-devbind --bind=vfio-pci 0000:82:00.0 0000:82:00.1 0000:82:00.2
0000:82:00.3 0000:83:00.0 0000:83:00.1 0000:83:00.2 0000:83:00.3*

Now we do
# *dpdk-devbind --status*

Network devices using DPDK-compatible driver
============================================
0000:82:00.0 'I350 Gigabit Network Connection' drv=vfio-pci unused=igb
0000:82:00.1 'I350 Gigabit Network Connection' drv=vfio-pci unused=igb
0000:82:00.2 'I350 Gigabit Network Connection' drv=vfio-pci unused=igb
0000:82:00.3 'I350 Gigabit Network Connection' drv=vfio-pci unused=igb
0000:83:00.0 'I350 Gigabit Network Connection' drv=vfio-pci unused=igb
0000:83:00.1 'I350 Gigabit Network Connection' drv=vfio-pci unused=igb
0000:83:00.2 'I350 Gigabit Network Connection' drv=vfio-pci unused=igb
0000:83:00.3 'I350 Gigabit Network Connection' drv=vfio-pci unused=igb

--- oops we lost information about interface name (ex. ens2f0).  This is
problem to identify physical ports if you don't have information about
PCI/slot address info and interface name mapping.  Also, doing a ip a on
command line won't give you interface names as they are assigned to DPDK!!

So, to identify port by blinking led we cannot do something like:
# *ethtool -p enp130s0f0 5*
Cannot identify NIC: No such device


With the latest OVS (2.7), we can do (*note*: we no longer need "dpdk"
prefix to port name):
# ovs-vsctl add-port ovs-ip64-br0 enp130s0f0 -- set Interface enp130s0f0
type=dpdk options:dpdk-devargs=0000:82:00.0
# ovs-vsctl add-port ovs-ip64-br0 enp130s0f1 -- set Interface enp130s0f1
type=dpdk # options:dpdk-devargs=0000:82:00.1
# ovs-vsctl add-port ovs-ip64-br0 enp130s0f2 -- set Interface enp130s0f2
type=dpdk options:dpdk-devargs=0000:82:00.2
# ovs-vsctl add-port ovs-ip64-br0 enp130s0f3 -- set Interface enp130s0f3
type=dpdk options:dpdk-devargs=0000:82:00.3
# ovs-vsctl add-port ovs-ip64-br0 ens2f0 -- set Interface ens2f0 type=dpdk
options:dpdk-devargs=0000:83:00.0
# ovs-vsctl add-port ovs-ip64-br0 ens2f1 -- set Interface ens2f1 type=dpdk
options:dpdk-devargs=0000:83:00.1
# ovs-vsctl add-port ovs-ip64-br0 ens2f2 -- set Interface ens2f2 type=dpdk
# options:dpdk-devargs=0000:83:00.2
# ovs-vsctl add-port ovs-ip64-br0 ens2f3 -- set Interface ens2f3 type=dpdk
options:dpdk-devargs=0000:83:00.3

Which will result in:
# *ovs-vsctl show*
92e46601-ca42-4743-8d7d-838c8fff515e
    Bridge "ovs-ip64-br0"
        Controller "tcp:10.20.5.5:6653"
            is_connected: true
        Controller "tcp:10.20.5.5:6654"
            is_connected: true
        Port "ens2f0"
            Interface "ens2f0"
                type: dpdk
                options: {dpdk-devargs="0000:83:00.0"}
        Port "enp130s0f2"
            Interface "enp130s0f2"
                type: dpdk
                options: {dpdk-devargs="0000:82:00.2"}
        Port "ovs-ip64-br0"
            Interface "ovs-ip64-br0"
                type: internal
        Port "ens2f1"
            Interface "ens2f1"
                type: dpdk
                options: {dpdk-devargs="0000:83:00.1"}
        Port "ens2f3"
            Interface "ens2f3"
                type: dpdk
                options: {dpdk-devargs="0000:83:00.3"}
        Port "enp130s0f0"
            Interface "enp130s0f0"
                type: dpdk
                options: {dpdk-devargs="0000:82:00.0"}
        Port "ens2f2"
            Interface "ens2f2"
                type: dpdk
                options: {dpdk-devargs="0000:83:00.2"}
        Port "enp130s0f1"
            Interface "enp130s0f1"
                type: dpdk
                options: {dpdk-devargs="0000:82:00.1"}
        Port "enp130s0f3"
            Interface "enp130s0f3"
                type: dpdk
                options: {dpdk-devargs="0000:82:00.3"}
    ovs_version: "2.7.0"

Now we know what ports to connect to.

Hope this helps

/Shivaram