[dpdk-dev] testpmd could not start up with vmxnet3 port

[dpdk-dev] testpmd could not start up with vmxnet3 port

[Ding, HengX]

Testpmd will fail to start up with vmxnet3 port. You can reproduce this issue follow these steps:


1.       Right click your VM and click edit settings

2.       Click "Add..." button.

3.       Choose Ethernet Adapter and click next

4.       Choose vmxnet3 as the Adapter Type, and select a network connection. Click next and finish config.

5.       Add another vmxnet3 port to VM follow step 1-4.

6.       Compile dpdk, bind vmxnet3 port to igb_uio and run testpmd: ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 -- -i


Here is the  bisect result:
f598fd063bb17eba9310ff48164014d09f40c759 is the first bad commit
commit f598fd063bb17eba9310ff48164014d09f40c759
Author: Yong Wang <yongwang@vmware.com>
Date:   Tue Jan 12 18:08:35 2016 -0800

    vmxnet3: add Tx L4 checksum offload

    Support TCP/UDP checksum offload.

    Signed-off-by: Yong Wang <yongwang@vmware.com>
    Acked-by: Stephen Hemminger <stephen@networkplumber.org>

:040000 040000 4524306879c4c0093ac54b0c66b8a8f895af8337 8e1473e7cc9052003abe6caae8e8b232810972d4 M      doc
:040000 040000 ad9cd888717dd976a43cfe05aaf33f3b83a2f7bc 9fbd6fefb95f67bb407bb9a7efcf39e707ab21a7 M      drivers

Re: [dpdk-dev] testpmd could not start up with vmxnet3 port

[Xu, Qian Q]

It's a regression, vmxnet3 can work with dpdk2.2, but now it can't work well.
Yong,
Heng found that your patch may be the first one to break the function, could you check it? Thx.

Thanks
Qian

From: Ding, HengX
Sent: Wednesday, March 23, 2016 11:28 AM
To: yongwang@vmware.com
Cc: Xu, Qian Q; dev@dpdk.org
Subject: testpmd could not start up with vmxnet3 port

Testpmd will fail to start up with vmxnet3 port. You can reproduce this issue follow these steps:


1.       Right click your VM and click edit settings

2.       Click "Add..." button.

3.       Choose Ethernet Adapter and click next

4.       Choose vmxnet3 as the Adapter Type, and select a network connection. Click next and finish config.

5.       Add another vmxnet3 port to VM follow step 1-4.

6.       Compile dpdk, bind vmxnet3 port to igb_uio and run testpmd: ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 -- -i


Here is the  bisect result:
f598fd063bb17eba9310ff48164014d09f40c759 is the first bad commit
commit f598fd063bb17eba9310ff48164014d09f40c759
Author: Yong Wang <yongwang@vmware.com<mailto:yongwang@vmware.com>>
Date:   Tue Jan 12 18:08:35 2016 -0800

    vmxnet3: add Tx L4 checksum offload

    Support TCP/UDP checksum offload.

    Signed-off-by: Yong Wang <yongwang@vmware.com<mailto:yongwang@vmware.com>>
    Acked-by: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>

:040000 040000 4524306879c4c0093ac54b0c66b8a8f895af8337 8e1473e7cc9052003abe6caae8e8b232810972d4 M      doc
:040000 040000 ad9cd888717dd976a43cfe05aaf33f3b83a2f7bc 9fbd6fefb95f67bb407bb9a7efcf39e707ab21a7 M      drivers

Re: [dpdk-dev] testpmd could not start up with vmxnet3 port

[Yong Wang]

From: "Xu, Qian Q" <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>
Date: Tuesday, March 22, 2016 at 8:31 PM
To: "Ding, HengX" <hengx.ding@intel.com<mailto:hengx.ding@intel.com>>, Yong <yongwang@vmware.com<mailto:yongwang@vmware.com>>
Cc: "dev@dpdk.org<mailto:dev@dpdk.org>" <dev@dpdk.org<mailto:dev@dpdk.org>>
Subject: RE: testpmd could not start up with vmxnet3 port

It’s a regression, vmxnet3 can work with dpdk2.2, but now it can’t work well.
Yong,
Heng found that your patch may be the first one to break the function, could you check it? Thx.

Thanks
Qian

From: Ding, HengX
Sent: Wednesday, March 23, 2016 11:28 AM
To: yongwang@vmware.com<mailto:yongwang@vmware.com>
Cc: Xu, Qian Q; dev@dpdk.org<mailto:dev@dpdk.org>
Subject: testpmd could not start up with vmxnet3 port

Testpmd will fail to start up with vmxnet3 port. You can reproduce this issue follow these steps:


1.       Right click your VM and click edit settings

2.       Click “Add…” button.

3.       Choose Ethernet Adapter and click next

4.       Choose vmxnet3 as the Adapter Type, and select a network connection. Click next and finish config.

5.       Add another vmxnet3 port to VM follow step 1-4.

6.       Compile dpdk, bind vmxnet3 port to igb_uio and run testpmd: ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 -- -i


Here is the  bisect result:
f598fd063bb17eba9310ff48164014d09f40c759 is the first bad commit
commit f598fd063bb17eba9310ff48164014d09f40c759
Author: Yong Wang <yongwang@vmware.com<mailto:yongwang@vmware.com>>
Date:   Tue Jan 12 18:08:35 2016 -0800

    vmxnet3: add Tx L4 checksum offload

    Support TCP/UDP checksum offload.

    Signed-off-by: Yong Wang <yongwang@vmware.com<mailto:yongwang@vmware.com>>
    Acked-by: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>

:040000 040000 4524306879c4c0093ac54b0c66b8a8f895af8337 8e1473e7cc9052003abe6caae8e8b232810972d4 M      doc
:040000 040000 ad9cd888717dd976a43cfe05aaf33f3b83a2f7bc 9fbd6fefb95f67bb407bb9a7efcf39e707ab21a7 M      drivers

Currently vmxnet3’s default_txconf.txq_flags is set to the following, which is used by testpmd
as there is no explicit txconf passed when initializing tx queue:

        dev_info->default_txconf.txq_flags = ETH_TXQ_FLAGS_NOMULTSEGS |
                                                ETH_TXQ_FLAGS_NOOFFLOADS;

With the referred patch that introduced l4 cksum offload, we should update the default txq
flags check accordingly.  Heng, can you post the error logs to confirm this is indeed the cause
of the error you reported?

Related to this, I saw that the check for NOMULTISEGS has been removed and the check for
NOVLANOFF was never implemented.  Should we just remove the offload flags check as well
as I don’t see much value of this check.  Basically we know that the device does not support
certain offload and we have to set those flags to let the device initialize. But doing this does
nothing to prevent users to request these non-supported offload.  I also saw another thread
discussing better device capability APIs and hopefully this will not be needed then.

Re: [dpdk-dev] testpmd could not start up with vmxnet3 port

[Vincent Li]

> Currently vmxnet3’s default_txconf.txq_flags is set to the following, which is used by testpmd
> as there is no explicit txconf passed when initializing tx queue:
>
>         dev_info->default_txconf.txq_flags = ETH_TXQ_FLAGS_NOMULTSEGS |
>                                                 ETH_TXQ_FLAGS_NOOFFLOADS;
>
> With the referred patch that introduced l4 cksum offload, we should update the default txq
> flags check accordingly.  Heng, can you post the error logs to confirm this is indeed the cause
> of the error you reported?
>
> Related to this, I saw that the check for NOMULTISEGS has been removed and the check for
> NOVLANOFF was never implemented.  Should we just remove the offload flags check as well
> as I don’t see much value of this check.  Basically we know that the device does not support
> certain offload and we have to set those flags to let the device initialize. But doing this does
> nothing to prevent users to request these non-supported offload.  I also saw another thread
> discussing better device capability APIs and hopefully this will not be needed then.

somehow hijack this thread, in my company  ESXi environment, we are
using tagged vlan in ESXi (select "network label" in vsphpere client
when adding VM network adapter. I found the DPDK application packet
running on one ESXi VM can't reach to other ESXi VM, is this because
VLAN offload not implemented in vmxnet3 PMD ? any clue can I address
the tagged vlan in ESXi environment?

Thanks!

Vincent

Re: [dpdk-dev] testpmd could not start up with vmxnet3 port

[Yong Wang]

From: "Xu, Qian Q" <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>
Date: Tuesday, March 22, 2016 at 8:31 PM
To: "Ding, HengX" <hengx.ding@intel.com<mailto:hengx.ding@intel.com>>, Yong <yongwang@vmware.com<mailto:yongwang@vmware.com>>
Cc: "dev@dpdk.org<mailto:dev@dpdk.org>" <dev@dpdk.org<mailto:dev@dpdk.org>>
Subject: RE: testpmd could not start up with vmxnet3 port

It’s a regression, vmxnet3 can work with dpdk2.2, but now it can’t work well.
Yong,
Heng found that your patch may be the first one to break the function, could you check it? Thx.

Thanks
Qian

From: Ding, HengX
Sent: Wednesday, March 23, 2016 11:28 AM
To: yongwang@vmware.com<mailto:yongwang@vmware.com>
Cc: Xu, Qian Q; dev@dpdk.org<mailto:dev@dpdk.org>
Subject: testpmd could not start up with vmxnet3 port

Testpmd will fail to start up with vmxnet3 port. You can reproduce this issue follow these steps:


1.       Right click your VM and click edit settings

2.       Click “Add…” button.

3.       Choose Ethernet Adapter and click next

4.       Choose vmxnet3 as the Adapter Type, and select a network connection. Click next and finish config.

5.       Add another vmxnet3 port to VM follow step 1-4.

6.       Compile dpdk, bind vmxnet3 port to igb_uio and run testpmd: ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 -- -i


Here is the  bisect result:
f598fd063bb17eba9310ff48164014d09f40c759 is the first bad commit
commit f598fd063bb17eba9310ff48164014d09f40c759
Author: Yong Wang <yongwang@vmware.com<mailto:yongwang@vmware.com>>
Date:   Tue Jan 12 18:08:35 2016 -0800

    vmxnet3: add Tx L4 checksum offload

    Support TCP/UDP checksum offload.

    Signed-off-by: Yong Wang <yongwang@vmware.com<mailto:yongwang@vmware.com>>
    Acked-by: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>

:040000 040000 4524306879c4c0093ac54b0c66b8a8f895af8337 8e1473e7cc9052003abe6caae8e8b232810972d4 M      doc
:040000 040000 ad9cd888717dd976a43cfe05aaf33f3b83a2f7bc 9fbd6fefb95f67bb407bb9a7efcf39e707ab21a7 M      drivers

Resend due to formatting issues.

Currently vmxnet3’s default_txconf.txq_flags is set to the following, which is used by testpmd
as there is no explicit txconf passed when initializing tx queue:

        dev_info->default_txconf.txq_flags = ETH_TXQ_FLAGS_NOMULTSEGS |
                                                ETH_TXQ_FLAGS_NOOFFLOADS;

With the referred patch that introduced l4 cksum offload, we should update the default txq
flags check accordingly.  Heng, can you post the error logs to confirm this is indeed the cause
of the error you reported?

Related to this, I saw that the check for NOMULTISEGS has been removed and the check for
NOVLANOFF was never implemented.  Should we just remove the offload flags check as well
as I don’t see much value of this check.  Basically we know that the device does not support
certain offload and we have to set those flags to let the device initialize. But doing this does
nothing to prevent users to request these non-supported offload.  I also saw another thread
discussing better device capability APIs and hopefully this will not be needed then

Re: [dpdk-dev] testpmd could not start up with vmxnet3 port

[Ding, HengX]

Testpmd output is as below:

./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 --  -i
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 0 on socket 0
EAL: Detected lcore 3 as core 1 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 4 lcore(s)
EAL: Probing VFIO support...
EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or directory)
EAL: VFIO modules not loaded, skipping VFIO support...
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x800000 bytes
EAL: Virtual area found at 0x7ff3c5600000 (size = 0x800000)
EAL: Ask a virtual area of 0x600000 bytes
EAL: Virtual area found at 0x7ff3c4e00000 (size = 0x600000)
EAL: Ask a virtual area of 0x56800000 bytes
EAL: Virtual area found at 0x7ff36e400000 (size = 0x56800000)
EAL: Ask a virtual area of 0x28000000 bytes
EAL: Virtual area found at 0x7ff346200000 (size = 0x28000000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7ff345e00000 (size = 0x200000)
EAL: Ask a virtual area of 0x600000 bytes
EAL: Virtual area found at 0x7ff345600000 (size = 0x600000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7ff345200000 (size = 0x200000)
EAL: Requesting 1024 pages of size 2MB from socket 0
EAL: TSC frequency is ~2793268 KHz
EAL: Master lcore 0 is ready (tid=c62c9940;cpuset=[0])
EAL: lcore 3 is ready (tid=435f0700;cpuset=[3])
EAL: lcore 1 is ready (tid=445f2700;cpuset=[1])
EAL: lcore 2 is ready (tid=43df1700;cpuset=[2])
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:0b:00.0 on NUMA socket -1
EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
EAL:   PCI memory mapped at 0x7ff3c5e00000
EAL:   PCI memory mapped at 0x7ff3c5e01000
EAL:   PCI memory mapped at 0x7ff3c5e02000
EAL: PCI device 0000:13:00.0 on NUMA socket -1
EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
EAL:   PCI memory mapped at 0x7ff3c5e04000
EAL:   PCI memory mapped at 0x7ff3c5e05000
EAL:   PCI memory mapped at 0x7ff3c5e06000
Interactive-mode selected
Configuring Port 0 (socket 0)
Fail to configure port 0 tx queues
EAL: Error - exiting with code: 1
  Cause: Start ports failed

From: Yong Wang [mailto:yongwang@vmware.com]
Sent: Wednesday, March 23, 2016 1:58 PM
To: Xu, Qian Q; Ding, HengX
Cc: dev@dpdk.org; Stephen Hemminger
Subject: Re: testpmd could not start up with vmxnet3 port


From: "Xu, Qian Q" <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>
Date: Tuesday, March 22, 2016 at 8:31 PM
To: "Ding, HengX" <hengx.ding@intel.com<mailto:hengx.ding@intel.com>>, Yong <yongwang@vmware.com<mailto:yongwang@vmware.com>>
Cc: "dev@dpdk.org<mailto:dev@dpdk.org>" <dev@dpdk.org<mailto:dev@dpdk.org>>
Subject: RE: testpmd could not start up with vmxnet3 port

It’s a regression, vmxnet3 can work with dpdk2.2, but now it can’t work well.
Yong,
Heng found that your patch may be the first one to break the function, could you check it? Thx.

Thanks
Qian

From: Ding, HengX
Sent: Wednesday, March 23, 2016 11:28 AM
To: yongwang@vmware.com<mailto:yongwang@vmware.com>
Cc: Xu, Qian Q; dev@dpdk.org<mailto:dev@dpdk.org>
Subject: testpmd could not start up with vmxnet3 port

Testpmd will fail to start up with vmxnet3 port. You can reproduce this issue follow these steps:


1.       Right click your VM and click edit settings

2.       Click “Add…” button.

3.       Choose Ethernet Adapter and click next

4.       Choose vmxnet3 as the Adapter Type, and select a network connection. Click next and finish config.

5.       Add another vmxnet3 port to VM follow step 1-4.

6.       Compile dpdk, bind vmxnet3 port to igb_uio and run testpmd: ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 -- -i


Here is the  bisect result:
f598fd063bb17eba9310ff48164014d09f40c759 is the first bad commit
commit f598fd063bb17eba9310ff48164014d09f40c759
Author: Yong Wang <yongwang@vmware.com<mailto:yongwang@vmware.com>>
Date:   Tue Jan 12 18:08:35 2016 -0800

    vmxnet3: add Tx L4 checksum offload

    Support TCP/UDP checksum offload.

    Signed-off-by: Yong Wang <yongwang@vmware.com<mailto:yongwang@vmware.com>>
    Acked-by: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>

:040000 040000 4524306879c4c0093ac54b0c66b8a8f895af8337 8e1473e7cc9052003abe6caae8e8b232810972d4 M      doc
:040000 040000 ad9cd888717dd976a43cfe05aaf33f3b83a2f7bc 9fbd6fefb95f67bb407bb9a7efcf39e707ab21a7 M      drivers

Resend due to formatting issues.

Currently vmxnet3’s default_txconf.txq_flags is set to the following, which is used by testpmd
as there is no explicit txconf passed when initializing tx queue:

        dev_info->default_txconf.txq_flags = ETH_TXQ_FLAGS_NOMULTSEGS |
                                                ETH_TXQ_FLAGS_NOOFFLOADS;

With the referred patch that introduced l4 cksum offload, we should update the default txq
flags check accordingly.  Heng, can you post the error logs to confirm this is indeed the cause
of the error you reported?

Related to this, I saw that the check for NOMULTISEGS has been removed and the check for
NOVLANOFF was never implemented.  Should we just remove the offload flags check as well
as I don’t see much value of this check.  Basically we know that the device does not support
certain offload and we have to set those flags to let the device initialize. But doing this does
nothing to prevent users to request these non-supported offload.  I also saw another thread
discussing better device capability APIs and hopefully this will not be needed then

Re: [dpdk-dev] testpmd could not start up with vmxnet3 port

[Thomas Monjalon]

2016-03-23 05:57, Yong Wang:
> From: Ding, HengX
> > Testpmd will fail to start up with vmxnet3 port.
[...]
> Currently vmxnet3’s default_txconf.txq_flags is set to the following, which is used by testpmd
> as there is no explicit txconf passed when initializing tx queue:
> 
>         dev_info->default_txconf.txq_flags = ETH_TXQ_FLAGS_NOMULTSEGS |
>                                                 ETH_TXQ_FLAGS_NOOFFLOADS;

In vmxnet3_dev_tx_queue_setup:
    if ((tx_conf->txq_flags & ETH_TXQ_FLAGS_NOXSUMS) !=
        ETH_TXQ_FLAGS_NOXSUMSCTP) {
        PMD_INIT_LOG(ERR, "SCTP checksum offload not supported");
        return -EINVAL;
    }
It means we cannot disable TCP or UDP checksum offload.
ETH_TXQ_FLAGS_NOXSUMS = NOXSUMSCTP + NOXSUMUDP + NOXSUMTCP

I think it should be:
    if ((tx_conf->txq_flags & ETH_TXQ_FLAGS_NOXSUMSCTP) !=
        ETH_TXQ_FLAGS_NOXSUMSCTP) {

> With the referred patch that introduced l4 cksum offload, we should update the default txq
> flags check accordingly.  Heng, can you post the error logs to confirm this is indeed the cause
> of the error you reported?

The default conf is ETH_TXQ_FLAGS_NOOFFLOADS (= NOVLANOFFL + NOXSUMS).
Yes you can update the default conf, *and* fix the check above.

> Related to this, I saw that the check for NOMULTISEGS has been removed and the check for
> NOVLANOFF was never implemented.  Should we just remove the offload flags check as well
> as I don’t see much value of this check.  Basically we know that the device does not support
> certain offload and we have to set those flags to let the device initialize. But doing this does
> nothing to prevent users to request these non-supported offload.  I also saw another thread
> discussing better device capability APIs and hopefully this will not be needed then

These checks are important to throw an error if an offload is requested
but not supported.

Re: [dpdk-dev] testpmd could not start up with vmxnet3 port

[Yong Wang]

On 3/23/16, 8:55 AM, "Thomas Monjalon" <thomas.monjalon@6wind.com> wrote:



>2016-03-23 05:57, Yong Wang:
>> From: Ding, HengX
>> > Testpmd will fail to start up with vmxnet3 port.
>[...]
>> Currently vmxnet3’s default_txconf.txq_flags is set to the following, which is used by testpmd
>> as there is no explicit txconf passed when initializing tx queue:
>> 
>>         dev_info->default_txconf.txq_flags = ETH_TXQ_FLAGS_NOMULTSEGS |
>>                                                 ETH_TXQ_FLAGS_NOOFFLOADS;
>
>In vmxnet3_dev_tx_queue_setup:
>    if ((tx_conf->txq_flags & ETH_TXQ_FLAGS_NOXSUMS) !=
>        ETH_TXQ_FLAGS_NOXSUMSCTP) {
>        PMD_INIT_LOG(ERR, "SCTP checksum offload not supported");
>        return -EINVAL;
>    }
>It means we cannot disable TCP or UDP checksum offload.
>ETH_TXQ_FLAGS_NOXSUMS = NOXSUMSCTP + NOXSUMUDP + NOXSUMTCP
>
>I think it should be:
>    if ((tx_conf->txq_flags & ETH_TXQ_FLAGS_NOXSUMSCTP) !=
>        ETH_TXQ_FLAGS_NOXSUMSCTP) {

I understand this. As I pointed out, such checks are not consistently
enforced as far as I can tell.  Other than Intel drivers, no other
drivers claim DEV_TX_OFFLOAD_SCTP_CKSUM but only vmxnet3 really
checks this. Also, what’s not clear to me is that why do we need
this negative check if we have rx_offload_capa and tx_offload_capa?


Anyway, I’ll send out a patch to fix this.

>
>> With the referred patch that introduced l4 cksum offload, we should update the default txq
>> flags check accordingly.  Heng, can you post the error logs to confirm this is indeed the cause
>> of the error you reported?
>
>The default conf is ETH_TXQ_FLAGS_NOOFFLOADS (= NOVLANOFFL + NOXSUMS).
>Yes you can update the default conf, *and* fix the check above.
>
>> Related to this, I saw that the check for NOMULTISEGS has been removed and the check for
>> NOVLANOFF was never implemented.  Should we just remove the offload flags check as well
>> as I don’t see much value of this check.  Basically we know that the device does not support
>> certain offload and we have to set those flags to let the device initialize. But doing this does
>> nothing to prevent users to request these non-supported offload.  I also saw another thread
>> discussing better device capability APIs and hopefully this will not be needed then
>
>These checks are important to throw an error if an offload is requested
>but not supported.