[dpdk-dev] Bug in "rte_eth_dev_attach" while using mlx5 pmd driver?

[dpdk-dev] Bug in "rte_eth_dev_attach" while using mlx5 pmd driver?

[Sam]

Hi all,

I'm using OVS-2.8.0 + DPDK-17.05-rc1, with 2 pf and 1 vf port under mlx5
pmd driver, like this:

$ sudo /usr/local/share/openvswitch/scripts/dpdk_nic_bind --status
> Network devices using DPDK-compatible driver
> ============================================
> <none>
> Network devices using kernel driver
> ===================================
> ...
> 0000:03:00.0 'MT27640 Family' if=eth4 drv=mlx5_core unused=igb_uio
> 0000:03:00.1 'MT27640 Family' if=eth5 drv=mlx5_core unused=igb_uio
> 0000:03:00.2 'MT27641 Family' if=enp3s0f2 drv=mlx5_core unused=igb_uio
> *Active*
> ...
> Other network devices
> =====================
> <none>


Then I start ovs-vswitchd follow these steps:

> mkdir -p /usr/local/etc/openvswitch
> mkdir -p /usr/local/var/run/openvswitch
> [root@dpdk ~]# rm /usr/local/etc/openvswitch/conf.db
> [root@dpdk ~]# ovsdb-tool create /usr/local/etc/openvswitch/conf.db
> /usr/local/share/openvswitch/vswitch.ovsschema
> [root@dpdk ~]# ovsdb-server /usr/local/etc/openvswitch/conf.db \
> -vconsole:emer -vsyslog:err -vfile:info \
> --remote=punix:/usr/local/var/run/openvswitch/db.sock \
> --private-key=db:Open_vSwitch,SSL,private_key \
> --certificate=db:Open_vSwitch,SSL,certificate \
> --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir \
> --log-file=/usr/local/var/log/openvswitch/ovsdb-server.log \
> --pidfile=/usr/local/var/run/openvswitch/ovsdb-server.pid \
> --detach --monitor
> [root@dpdk ~]# ovs-vsctl --no-wait init
> [root@dpdk ~]# ovs-vsctl --no-wait set Open_vSwitch .
> other_config:dpdk-init=true
> [root@dpdk ~]# ovs-vsctl --no-wait set Open_vSwitch .
> other_config:dpdk-socket-mem="1024,0"
> [root@dpdk ~]# ovs-vsctl --no-wait set Open_vSwitch .
> other_config:pmd-cpu-mask=0x04
> [root@dpdk ~]# ovs-vswitchd unix:/usr/local/var/run/openvswitch/db.sock \
> -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir \
> --log-file=/usr/local/var/log/openvswitch/ovs-vswitchd.log \
> --pidfile=/usr/local/var/run/openvswitch/ovs-vswitchd.pid \
> --detach --monitor


Then I add br0 and dpdk0 port follow these steps:

> ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
> ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
> options:dpdk-devargs=0000:03:00.0


After that, I got error in ovs-vswitchd.log like this:

> 2018-03-29T02:44:11.913Z|00071|memory|INFO|handlers:22 ports:1
> revalidators:8 rules:5
> 2018-03-29T02:45:18.658Z|00072|netdev_dpdk|WARN|Error attaching device
> '0000:03:00.0' to DPDK
> 2018-03-29T02:45:18.658Z|00073|netdev|WARN|dpdk0: could not set
> configuration (Invalid argument)


Deep into code, I found this is caused by these code in rte_ethdev.c:

> int
> rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
> {
> int ret = -1;
> int current = rte_eth_dev_count();
> ...
> /* if nothing happened, there is a bug here, since some driver told us
> * it did attach a device, but did not create a port.
> */
> if (current == rte_eth_dev_count()) {
> ret = -1;
> goto err;
> }
> ...
> }


Which is the nb_ports returned by rte_eth_dev_count() keeps the same.
I think this is mlx5 driver probe port count and add into nb_ports while
starting ovs-vswitchd, not adding port by command `ovs-vsctl add port br0
...`. Refer to "mlx5_pci_probe" -> "rte_eth_dev_allocate" -> "eth_dev_get".

So is this a bug or I should use some other way to add port into bridge
under mlx5 driver? Or is this bug fixed in some version? Or how to fix this
bug? Thank you~

Re: [dpdk-dev] Bug in "rte_eth_dev_attach" while using mlx5 pmd driver?

[Sam]

I use a temporary patch to fix this bug:

diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index 4e1e6dc..8026a05 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -479,7 +479,7 @@ struct rte_eth_dev *
 rte_eth_dev_attach(const char *devargs, uint8_t *port_id)
 {
        int ret = -1;
-       int current = rte_eth_dev_count();
+       //int current = rte_eth_dev_count();
        char *name = NULL;
        char *args = NULL;

@@ -506,12 +506,14 @@ struct rte_eth_dev *
        /* if nothing happened, there is a bug here, since some driver told
us
         * it did attach a device, but did not create a port.
         */
-       if (current == rte_eth_dev_count()) {
-               ret = -1;
-               goto err;
-       }
+       //if (current == rte_eth_dev_count()) {
+       //      ret = -1;
+       //      goto err;
+       //}
+
+    rte_eth_dev_get_port_by_name(name, port_id);

-       *port_id = eth_dev_last_created_port;
+       //*port_id = eth_dev_last_created_port;
        ret = 0;

 err:

2018-03-29 15:00 GMT+08:00 Sam <batmanustc@gmail.com>:

> Hi all,
>
> I'm using OVS-2.8.0 + DPDK-17.05-rc1, with 2 pf and 1 vf port under mlx5
> pmd driver, like this:
>
> $ sudo /usr/local/share/openvswitch/scripts/dpdk_nic_bind --status
>> Network devices using DPDK-compatible driver
>> ============================================
>> <none>
>> Network devices using kernel driver
>> ===================================
>> ...
>> 0000:03:00.0 'MT27640 Family' if=eth4 drv=mlx5_core unused=igb_uio
>> 0000:03:00.1 'MT27640 Family' if=eth5 drv=mlx5_core unused=igb_uio
>> 0000:03:00.2 'MT27641 Family' if=enp3s0f2 drv=mlx5_core unused=igb_uio
>> *Active*
>> ...
>> Other network devices
>> =====================
>> <none>
>
>
> Then I start ovs-vswitchd follow these steps:
>
>> mkdir -p /usr/local/etc/openvswitch
>> mkdir -p /usr/local/var/run/openvswitch
>> [root@dpdk ~]# rm /usr/local/etc/openvswitch/conf.db
>> [root@dpdk ~]# ovsdb-tool create /usr/local/etc/openvswitch/conf.db
>> /usr/local/share/openvswitch/vswitch.ovsschema
>> [root@dpdk ~]# ovsdb-server /usr/local/etc/openvswitch/conf.db \
>> -vconsole:emer -vsyslog:err -vfile:info \
>> --remote=punix:/usr/local/var/run/openvswitch/db.sock \
>> --private-key=db:Open_vSwitch,SSL,private_key \
>> --certificate=db:Open_vSwitch,SSL,certificate \
>> --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir \
>> --log-file=/usr/local/var/log/openvswitch/ovsdb-server.log \
>> --pidfile=/usr/local/var/run/openvswitch/ovsdb-server.pid \
>> --detach --monitor
>> [root@dpdk ~]# ovs-vsctl --no-wait init
>> [root@dpdk ~]# ovs-vsctl --no-wait set Open_vSwitch .
>> other_config:dpdk-init=true
>> [root@dpdk ~]# ovs-vsctl --no-wait set Open_vSwitch .
>> other_config:dpdk-socket-mem="1024,0"
>> [root@dpdk ~]# ovs-vsctl --no-wait set Open_vSwitch .
>> other_config:pmd-cpu-mask=0x04
>> [root@dpdk ~]# ovs-vswitchd unix:/usr/local/var/run/openvswitch/db.sock \
>> -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir \
>> --log-file=/usr/local/var/log/openvswitch/ovs-vswitchd.log \
>> --pidfile=/usr/local/var/run/openvswitch/ovs-vswitchd.pid \
>> --detach --monitor
>
>
> Then I add br0 and dpdk0 port follow these steps:
>
>> ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
>> ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
>> options:dpdk-devargs=0000:03:00.0
>
>
> After that, I got error in ovs-vswitchd.log like this:
>
>> 2018-03-29T02:44:11.913Z|00071|memory|INFO|handlers:22 ports:1
>> revalidators:8 rules:5
>> 2018-03-29T02:45:18.658Z|00072|netdev_dpdk|WARN|Error attaching device
>> '0000:03:00.0' to DPDK
>> 2018-03-29T02:45:18.658Z|00073|netdev|WARN|dpdk0: could not set
>> configuration (Invalid argument)
>
>
> Deep into code, I found this is caused by these code in rte_ethdev.c:
>
>> int
>> rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
>> {
>> int ret = -1;
>> int current = rte_eth_dev_count();
>> ...
>> /* if nothing happened, there is a bug here, since some driver told us
>> * it did attach a device, but did not create a port.
>> */
>> if (current == rte_eth_dev_count()) {
>> ret = -1;
>> goto err;
>> }
>> ...
>> }
>
>
> Which is the nb_ports returned by rte_eth_dev_count() keeps the same.
> I think this is mlx5 driver probe port count and add into nb_ports while
> starting ovs-vswitchd, not adding port by command `ovs-vsctl add port br0
> ...`. Refer to "mlx5_pci_probe" -> "rte_eth_dev_allocate" -> "eth_dev_get".
>
> So is this a bug or I should use some other way to add port into bridge
> under mlx5 driver? Or is this bug fixed in some version? Or how to fix this
> bug? Thank you~
>

Re: [dpdk-dev] Bug in "rte_eth_dev_attach" while using mlx5 pmd driver?

[Thomas Monjalon]

Hi,

29/03/2018 09:00, Sam:
> Hi all,
> 
> I'm using OVS-2.8.0 + DPDK-17.05-rc1, with 2 pf and 1 vf port under mlx5
> pmd driver, like this:

I did not really look at your issue because you are using a
DPDK release candidate.

When looking at http://docs.openvswitch.org/en/latest/faq/releases/
we can find the DPDK releases to use for which OVS:
	2.8.x   17.05.2
	2.9.x   17.11

Please update