[dpdk-dev] Mellanox 100GbE MCX516A-CCAT

[dpdk-dev] Mellanox 100GbE MCX516A-CCAT

[Yasuhiro Ohara]

Hi,

I am trying to use Mellanox MCX516A-CCAT in DPDK.
ConnectX-5 EN network interface card, 100GbE dual-port QSFP28,
PCIe3.0 x16, tall bracket, ROHS R6

I noticed it is not supported yet,
(<https://dpdk.org/doc/guides/nics/mlx5.html>)
but how far is it ?

If I setup mlx5 for MCX516A-CCAT, I get these errors.

PMD: net_mlx5: Forcing port 9 link to be up
PMD: net_mlx5: 0x7ff83ffaae00: error occurred while configuring control flows: Invalid argument
WARNING: port[9]: rte_eth_dev_start() failed: error: -11.

I guess this is -EAGAIN from below.
rte_ethdev.c: rte_eth_dev_start():
    1026         diag = (*dev->dev_ops->dev_start)(dev);
mlx5_trigger.c: mlx5_dev_start():
    170         err = priv_force_link_status_change(priv, ETH_LINK_UP);

My configuration:
dpdk-17.11.1.tar.xz
MLNX_OFED_LINUX-4.2-1.2.0.0 (--upstream-libs)
fw_ver: 16.22.1002

Initial log messages:
EAL: PCI device 0000:af:00.1 on NUMA socket 1
EAL:   probe driver: 15b3:1017 net_mlx5
PMD: mlx5.c:615: mlx5_pci_probe(): PCI information matches, using device "mlx5_9" (SR-IOV: false)
PMD: mlx5.c:660: mlx5_pci_probe(): 1 port(s) detected
PMD: mlx5.c:860: mlx5_pci_probe(): Enhanced MPS is enabled
PMD: mlx5.c:887: mlx5_pci_probe(): port 1 MAC address is 50:6b:4b:08:6c:77

If I enable CONFIG_RTE_LIBRTE_MLX5_DEBUG=y,
I get the following crash from an assertion.

dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_trigger.c:283: priv_dev_traffic_enable: Assertion `(mlx5_ctrl_flow(dev, &multicast, &multicast)) == 0' failed.

Thread 1 "progname" received signal SIGABRT, Aborted.
0x00007ffff66c9428 in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/unix/sysv/linux/raise.c:54
54      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007ffff66c9428 in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007ffff66cb02a in __GI_abort () at abort.c:89
#2  0x00007ffff66c1bd7 in __assert_fail_base (fmt=<optimized out>, 
    assertion=assertion@entry=0x824ec0 "(mlx5_ctrl_flow(dev, &multicast, &multicast)) == 0", 
    file=file@entry=0x824c68 "/home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_trigger.c", line=line@entry=283, 
    function=function@entry=0x824f30 <__PRETTY_FUNCTION__.38015> "priv_dev_traffic_enable") at assert.c:92
#3  0x00007ffff66c1c82 in __GI___assert_fail (
    assertion=assertion@entry=0x824ec0 "(mlx5_ctrl_flow(dev, &multicast, &multicast)) == 0", 
    file=file@entry=0x824c68 "/home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_trigger.c", line=line@entry=283, 
    function=function@entry=0x824f30 <__PRETTY_FUNCTION__.38015> "priv_dev_traffic_enable") at assert.c:101
#4  0x00000000006db033 in priv_dev_traffic_enable (
    priv=priv@entry=0x7ff83ffea940, dev=dev@entry=0xd1b580 <rte_eth_devices>)
    at /home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_trigger.c:283
#5  0x00000000006db1aa in priv_link_start (priv=priv@entry=0x7ff83ffea940)
    at /home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_ethdev.c:915
#6  0x00000000006dc780 in priv_link_update (priv=priv@entry=0x7ff83ffea940, 
    wait_to_complete=wait_to_complete@entry=0)
    at /home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_ethdev.c:969
#7  0x00000000006dded9 in priv_force_link_status_change (
    priv=priv@entry=0x7ff83ffea940, status=status@entry=1)
    at /home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_ethdev.c:1000
#8  0x00000000006da92d in mlx5_dev_start (dev=0xd1b580 <rte_eth_devices>)
    at /home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_trigger.c:170
#9  0x000000000051dfbd in rte_eth_dev_start ()

Mellanox(R) ConnectX(R)-4 100G MCX416A-CCAT (2x100G)
seems to be working, by the way, in the same configuration.

Thanks for help in advance.

Best regards,
Yasu

Re: [dpdk-dev] Mellanox 100GbE MCX516A-CCAT

[Olga Shern]

Hi  Yasuhiro,

Please contact Mellanox support and open a case , support@mellanox.com 

Best Regards,
Olga

-------------------------------------------------------
Olga Shern
SW Sr. Director DPDK 
Mellanox Technologies, Raanana, Israel 



-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yasuhiro Ohara
Sent: Saturday, April 14, 2018 9:26 AM
To: dev@dpdk.org
Subject: [dpdk-dev] Mellanox 100GbE MCX516A-CCAT


Hi,

I am trying to use Mellanox MCX516A-CCAT in DPDK.
ConnectX-5 EN network interface card, 100GbE dual-port QSFP28,
PCIe3.0 x16, tall bracket, ROHS R6

I noticed it is not supported yet,
(<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdpdk.org%2Fdoc%2Fguides%2Fnics%2Fmlx5.html&data=02%7C01%7Colgas%40mellanox.com%7Ccccbb71d1eb746f841d608d5a1d090e6%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636592839568352566&sdata=KffKaTX1hmahlN0xoI9Enes2r0s%2F%2BjiO8OLonU%2B0ANg%3D&reserved=0>)
but how far is it ?

If I setup mlx5 for MCX516A-CCAT, I get these errors.

PMD: net_mlx5: Forcing port 9 link to be up
PMD: net_mlx5: 0x7ff83ffaae00: error occurred while configuring control flows: Invalid argument
WARNING: port[9]: rte_eth_dev_start() failed: error: -11.

I guess this is -EAGAIN from below.
rte_ethdev.c: rte_eth_dev_start():
    1026         diag = (*dev->dev_ops->dev_start)(dev);
mlx5_trigger.c: mlx5_dev_start():
    170         err = priv_force_link_status_change(priv, ETH_LINK_UP);

My configuration:
dpdk-17.11.1.tar.xz
MLNX_OFED_LINUX-4.2-1.2.0.0 (--upstream-libs)
fw_ver: 16.22.1002

Initial log messages:
EAL: PCI device 0000:af:00.1 on NUMA socket 1
EAL:   probe driver: 15b3:1017 net_mlx5
PMD: mlx5.c:615: mlx5_pci_probe(): PCI information matches, using device "mlx5_9" (SR-IOV: false)
PMD: mlx5.c:660: mlx5_pci_probe(): 1 port(s) detected
PMD: mlx5.c:860: mlx5_pci_probe(): Enhanced MPS is enabled
PMD: mlx5.c:887: mlx5_pci_probe(): port 1 MAC address is 50:6b:4b:08:6c:77

If I enable CONFIG_RTE_LIBRTE_MLX5_DEBUG=y, I get the following crash from an assertion.

dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_trigger.c:283: priv_dev_traffic_enable: Assertion `(mlx5_ctrl_flow(dev, &multicast, &multicast)) == 0' failed.

Thread 1 "progname" received signal SIGABRT, Aborted.
0x00007ffff66c9428 in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/unix/sysv/linux/raise.c:54
54      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007ffff66c9428 in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007ffff66cb02a in __GI_abort () at abort.c:89
#2  0x00007ffff66c1bd7 in __assert_fail_base (fmt=<optimized out>, 
    assertion=assertion@entry=0x824ec0 "(mlx5_ctrl_flow(dev, &multicast, &multicast)) == 0", 
    file=file@entry=0x824c68 "/home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_trigger.c", line=line@entry=283, 
    function=function@entry=0x824f30 <__PRETTY_FUNCTION__.38015> "priv_dev_traffic_enable") at assert.c:92
#3  0x00007ffff66c1c82 in __GI___assert_fail (
    assertion=assertion@entry=0x824ec0 "(mlx5_ctrl_flow(dev, &multicast, &multicast)) == 0", 
    file=file@entry=0x824c68 "/home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_trigger.c", line=line@entry=283, 
    function=function@entry=0x824f30 <__PRETTY_FUNCTION__.38015> "priv_dev_traffic_enable") at assert.c:101
#4  0x00000000006db033 in priv_dev_traffic_enable (
    priv=priv@entry=0x7ff83ffea940, dev=dev@entry=0xd1b580 <rte_eth_devices>)
    at /home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_trigger.c:283
#5  0x00000000006db1aa in priv_link_start (priv=priv@entry=0x7ff83ffea940)
    at /home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_ethdev.c:915
#6  0x00000000006dc780 in priv_link_update (priv=priv@entry=0x7ff83ffea940, 
    wait_to_complete=wait_to_complete@entry=0)
    at /home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_ethdev.c:969
#7  0x00000000006dded9 in priv_force_link_status_change (
    priv=priv@entry=0x7ff83ffea940, status=status@entry=1)
    at /home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_ethdev.c:1000
#8  0x00000000006da92d in mlx5_dev_start (dev=0xd1b580 <rte_eth_devices>)
    at /home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_trigger.c:170
#9  0x000000000051dfbd in rte_eth_dev_start ()

Mellanox(R) ConnectX(R)-4 100G MCX416A-CCAT (2x100G) seems to be working, by the way, in the same configuration.

Thanks for help in advance.

Best regards,
Yasu

Re: [dpdk-dev] Mellanox 100GbE MCX516A-CCAT

[Yasuhiro Ohara]

Thank you for the suggestion, Olga.

I was in a confusion, but now somehow I get the card to work,
so for the record I share my current status here.

I was confused about my ConnectX-5 type, which was MCX556A-ECAT
rather than MCX516A-CCAT. Currently the card seems to be working
fine with the below configuration.

MCX556A-ECAT ConnectX-5
MCX556A-ECAT ConnectX-5 Mellanox NIC firmware version: 16.22.1002(?)
Mellanox OFED driver version: MLNX_OFED_LINUX-4.2-1.2.0.0 (--upstream-libs)
DPDK version: 17.11.0 (dpdk-17.11.tar.xz)

Below matrix should be important (which I haven't been following yet):
<http://www.mellanox.com/page/mlnx_ofed_matrix?mtag=linux_sw_drivers>

Although I didn't record it properly (because I didn't thought it's
relevant), sometimes /var/tmp is not writable in my test environment.
And I'm guessing that it might have been the source of the problem,
making the ConnectX-5 from functioning.

There's still many unsure things in me, and I will investigate more
and will continue to report in the Mellanox Community.

Thanks.

regards,
Yasu

From: Olga Shern <olgas@mellanox.com>
Subject: Re: [dpdk-dev] Mellanox 100GbE MCX516A-CCAT
Date: Sun, 15 Apr 2018 19:40:19 +0000
Message-ID: <AM6PR0502MB3637DE2096FAA3DE3E6C4D18D3B10@AM6PR0502MB3637.eurprd05.prod.outlook.com>

> Hi  Yasuhiro,
> 
> Please contact Mellanox support and open a case , support@mellanox.com 
> 
> Best Regards,
> Olga
> 
> -------------------------------------------------------
> Olga Shern
> SW Sr. Director DPDK 
> Mellanox Technologies, Raanana, Israel 
> 
> 
> 
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yasuhiro Ohara
> Sent: Saturday, April 14, 2018 9:26 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] Mellanox 100GbE MCX516A-CCAT
> 
> 
> Hi,
> 
> I am trying to use Mellanox MCX516A-CCAT in DPDK.
> ConnectX-5 EN network interface card, 100GbE dual-port QSFP28,
> PCIe3.0 x16, tall bracket, ROHS R6
> 
> I noticed it is not supported yet,
> (<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdpdk.org%2Fdoc%2Fguides%2Fnics%2Fmlx5.html&data=02%7C01%7Colgas%40mellanox.com%7Ccccbb71d1eb746f841d608d5a1d090e6%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636592839568352566&sdata=KffKaTX1hmahlN0xoI9Enes2r0s%2F%2BjiO8OLonU%2B0ANg%3D&reserved=0>)
> but how far is it ?
> 
> If I setup mlx5 for MCX516A-CCAT, I get these errors.
> 
> PMD: net_mlx5: Forcing port 9 link to be up
> PMD: net_mlx5: 0x7ff83ffaae00: error occurred while configuring control flows: Invalid argument
> WARNING: port[9]: rte_eth_dev_start() failed: error: -11.
> 
> I guess this is -EAGAIN from below.
> rte_ethdev.c: rte_eth_dev_start():
>     1026         diag = (*dev->dev_ops->dev_start)(dev);
> mlx5_trigger.c: mlx5_dev_start():
>     170         err = priv_force_link_status_change(priv, ETH_LINK_UP);
> 
> My configuration:
> dpdk-17.11.1.tar.xz
> MLNX_OFED_LINUX-4.2-1.2.0.0 (--upstream-libs)
> fw_ver: 16.22.1002
> 
> Initial log messages:
> EAL: PCI device 0000:af:00.1 on NUMA socket 1
> EAL:   probe driver: 15b3:1017 net_mlx5
> PMD: mlx5.c:615: mlx5_pci_probe(): PCI information matches, using device "mlx5_9" (SR-IOV: false)
> PMD: mlx5.c:660: mlx5_pci_probe(): 1 port(s) detected
> PMD: mlx5.c:860: mlx5_pci_probe(): Enhanced MPS is enabled
> PMD: mlx5.c:887: mlx5_pci_probe(): port 1 MAC address is 50:6b:4b:08:6c:77
> 
> If I enable CONFIG_RTE_LIBRTE_MLX5_DEBUG=y, I get the following crash from an assertion.
> 
> dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_trigger.c:283: priv_dev_traffic_enable: Assertion `(mlx5_ctrl_flow(dev, &multicast, &multicast)) == 0' failed.
> 
> Thread 1 "progname" received signal SIGABRT, Aborted.
> 0x00007ffff66c9428 in __GI_raise (sig=sig@entry=6)
>     at ../sysdeps/unix/sysv/linux/raise.c:54
> 54      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> (gdb) bt
> #0  0x00007ffff66c9428 in __GI_raise (sig=sig@entry=6)
>     at ../sysdeps/unix/sysv/linux/raise.c:54
> #1  0x00007ffff66cb02a in __GI_abort () at abort.c:89
> #2  0x00007ffff66c1bd7 in __assert_fail_base (fmt=<optimized out>, 
>     assertion=assertion@entry=0x824ec0 "(mlx5_ctrl_flow(dev, &multicast, &multicast)) == 0", 
>     file=file@entry=0x824c68 "/home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_trigger.c", line=line@entry=283, 
>     function=function@entry=0x824f30 <__PRETTY_FUNCTION__.38015> "priv_dev_traffic_enable") at assert.c:92
> #3  0x00007ffff66c1c82 in __GI___assert_fail (
>     assertion=assertion@entry=0x824ec0 "(mlx5_ctrl_flow(dev, &multicast, &multicast)) == 0", 
>     file=file@entry=0x824c68 "/home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_trigger.c", line=line@entry=283, 
>     function=function@entry=0x824f30 <__PRETTY_FUNCTION__.38015> "priv_dev_traffic_enable") at assert.c:101
> #4  0x00000000006db033 in priv_dev_traffic_enable (
>     priv=priv@entry=0x7ff83ffea940, dev=dev@entry=0xd1b580 <rte_eth_devices>)
>     at /home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_trigger.c:283
> #5  0x00000000006db1aa in priv_link_start (priv=priv@entry=0x7ff83ffea940)
>     at /home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_ethdev.c:915
> #6  0x00000000006dc780 in priv_link_update (priv=priv@entry=0x7ff83ffea940, 
>     wait_to_complete=wait_to_complete@entry=0)
>     at /home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_ethdev.c:969
> #7  0x00000000006dded9 in priv_force_link_status_change (
>     priv=priv@entry=0x7ff83ffea940, status=status@entry=1)
>     at /home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_ethdev.c:1000
> #8  0x00000000006da92d in mlx5_dev_start (dev=0xd1b580 <rte_eth_devices>)
>     at /home/hspcr/dpdk-stable-17.11.1/drivers/net/mlx5/mlx5_trigger.c:170
> #9  0x000000000051dfbd in rte_eth_dev_start ()
> 
> Mellanox(R) ConnectX(R)-4 100G MCX416A-CCAT (2x100G) seems to be working, by the way, in the same configuration.
> 
> Thanks for help in advance.
> 
> Best regards,
> Yasu
>