[dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD

[dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD

[Aziz Hajee]

I am using the dpdk1.6.0r1
I could not find a complete clarification, sorry if missed.
VMXNET3 PMD
============
I have enabled the VMXNET3 PMD  in the dpdk.
 # Compile burst-oriented VMXNET3 PMD driver
 #

CONFIG_RTE_LIBRTE_VMXNET3_PMD=y
CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_INIT=y
CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_RX=n
CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_TX=n
CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_TX_FREE=n
The Intel DPDK VMXNET3 PMD driver does not support mbuf chaining, and I
have to set NOMULTSEGS for the vmxnet3 interface init to succeed.
tx_conf.txq_flags =  ETH_TXQ_FLAGS_NOMULTSEGS
Is there a later version of DPDK that supports multiseg for the dpdk
VMXNET3 PMD.

vmware vmxnet3-usermap AND  DPDK VMXNET3 PMD
=========================================
Is the vmxnet3-usermap.ko module driver also needed ? (appears that I need,
otherwise the eal initialise fails.
sudo insmod ./vmxnet3-usermap.ko enable_shm=2,2 num_rqs=1,1 num_rxds=2048
num_txds=2048

I do not understand if VMXNET3 PMD is there, what is the purpose of
/vmxnet3-usermap.ko/vmxnet3-usermap.ko ?

>From some responses i saw that the following ifdef RTE_EAL_UNBIND_PORTS is
also need to be removed in lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c ?
     {
         .name = "rte_vmxnet3_pmd",
         .id_table = pci_id_vmxnet3_map,
-#ifdef RTE_EAL_UNBIND_PORTS
+// #ifdef RTE_EAL_UNBIND_PORTS
         .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO,
-#endif
+// #endif
     },
     .eth_dev_init = eth_vmxnet3_dev_init,
     .dev_private_size = sizeof(struct vmxnet3_adapter),

thanks,
-aziz

Re: [dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD

[Patel, Rashmin N]

Hi Aziz,

Yes, you're right DPDK VMXNET3-PMD in /lib/librte_pmd_vmxnet3 does not support mbuf chaining today. But it's a standalone bsd driver just like any other pmd in that directory, it does not need vmxnet3-usermap.ko module.

Now there is another vmxnet3 solution in a separate branch as a plugin, which must have vmxnet3-usermap.ko linux module(1), and a user space interface piece(2) to tie it to any DPDK application in the main branch. (1) and (2) makes the solution which is known as vmxnet3-plugin. It's been there for a long time just like virtio-plugin, I don't know who uses it, but community can *reply* here if there is still any need of a separate solution that way.

I'm in favor of consolidating all those version into one elegant solution by grabbing best features from all of them and maintain one copy. I'm sure that developers contributing from VMware would also support that idea because then it makes easy to maintain and debug and bug fix and most importantly avoid such confusion in future.

Thanks,
Rashmin

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Aziz Hajee
Sent: Thursday, November 06, 2014 5:47 PM
To: dev@dpdk.org
Subject: [dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD

I am using the dpdk1.6.0r1
I could not find a complete clarification, sorry if missed.
VMXNET3 PMD
============
I have enabled the VMXNET3 PMD  in the dpdk.
 # Compile burst-oriented VMXNET3 PMD driver  #

CONFIG_RTE_LIBRTE_VMXNET3_PMD=y
CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_INIT=y
CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_RX=n
CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_TX=n
CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_TX_FREE=n
The Intel DPDK VMXNET3 PMD driver does not support mbuf chaining, and I have to set NOMULTSEGS for the vmxnet3 interface init to succeed.
tx_conf.txq_flags =  ETH_TXQ_FLAGS_NOMULTSEGS Is there a later version of DPDK that supports multiseg for the dpdk
VMXNET3 PMD.

vmware vmxnet3-usermap AND  DPDK VMXNET3 PMD =========================================
Is the vmxnet3-usermap.ko module driver also needed ? (appears that I need, otherwise the eal initialise fails.
sudo insmod ./vmxnet3-usermap.ko enable_shm=2,2 num_rqs=1,1 num_rxds=2048
num_txds=2048

I do not understand if VMXNET3 PMD is there, what is the purpose of /vmxnet3-usermap.ko/vmxnet3-usermap.ko ?

From some responses i saw that the following ifdef RTE_EAL_UNBIND_PORTS is also need to be removed in lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c ?
     {
         .name = "rte_vmxnet3_pmd",
         .id_table = pci_id_vmxnet3_map, -#ifdef RTE_EAL_UNBIND_PORTS
+// #ifdef RTE_EAL_UNBIND_PORTS
         .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO, -#endif
+// #endif
     },
     .eth_dev_init = eth_vmxnet3_dev_init,
     .dev_private_size = sizeof(struct vmxnet3_adapter),

thanks,
-aziz

Re: [dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD

[Thomas Monjalon]

Hi,

2014-11-07 16:53, Patel, Rashmin N:
> Yes, you're right DPDK VMXNET3-PMD in /lib/librte_pmd_vmxnet3 does not
> support mbuf chaining today. But it's a standalone bsd driver just like
> any other pmd in that directory, it does not need vmxnet3-usermap.ko module.
> 
> Now there is another vmxnet3 solution in a separate branch as a plugin,
> which must have vmxnet3-usermap.ko linux module(1), and a user space
> interface piece(2) to tie it to any DPDK application in the main branch.
> (1) and (2) makes the solution which is known as vmxnet3-plugin. It's been
> there for a long time just like virtio-plugin, I don't know who uses it,
> but community can *reply* here if there is still any need of a separate
> solution that way.

Coming back to last year, 6WIND developped some PMDs for virtio and vmxnet3.
Later, Intel developped their own version using the uio framework.
The versions in the main repository are the Intel ones, whereas the original
ones from 6WIND are released as extensions.
For completeness, it must be noted that Brocade worked on their own approach
of vmxnet3 and contributed to virtio PMD.
It's now time to merge all these implementations.

The 6WIND implementations show that it's possible to avoid the uio framework.
The virtio-net-pmd use port access granted by iopl().
The vmxnet3-usermap reuse the VMware's kernel module with a special mode for
memory mapping. It was a pre-bifurcated logic.

> I'm in favor of consolidating all those version into one elegant solution
> by grabbing best features from all of them and maintain one copy. I'm sure
> that developers contributing from VMware would also support that idea
> because then it makes easy to maintain and debug and bug fix and most
> importantly avoid such confusion in future.


> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Aziz Hajee
> Sent: Thursday, November 06, 2014 5:47 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD
> 
> I am using the dpdk1.6.0r1
> I could not find a complete clarification, sorry if missed.
> VMXNET3 PMD
> ============
> I have enabled the VMXNET3 PMD  in the dpdk.
>  # Compile burst-oriented VMXNET3 PMD driver  #
> 
> CONFIG_RTE_LIBRTE_VMXNET3_PMD=y
> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_INIT=y
> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_RX=n
> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_TX=n
> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_TX_FREE=n
> The Intel DPDK VMXNET3 PMD driver does not support mbuf chaining, and I have to set NOMULTSEGS for the vmxnet3 interface init to succeed.
> tx_conf.txq_flags =  ETH_TXQ_FLAGS_NOMULTSEGS Is there a later version of DPDK that supports multiseg for the dpdk
> VMXNET3 PMD.
> 
> vmware vmxnet3-usermap AND  DPDK VMXNET3 PMD =========================================
> Is the vmxnet3-usermap.ko module driver also needed ? (appears that I need, otherwise the eal initialise fails.
> sudo insmod ./vmxnet3-usermap.ko enable_shm=2,2 num_rqs=1,1 num_rxds=2048
> num_txds=2048
> 
> I do not understand if VMXNET3 PMD is there, what is the purpose of /vmxnet3-usermap.ko/vmxnet3-usermap.ko ?
> 
> From some responses i saw that the following ifdef RTE_EAL_UNBIND_PORTS is also need to be removed in lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c ?
>      {
>          .name = "rte_vmxnet3_pmd",
>          .id_table = pci_id_vmxnet3_map, -#ifdef RTE_EAL_UNBIND_PORTS
> +// #ifdef RTE_EAL_UNBIND_PORTS
>          .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO, -#endif
> +// #endif
>      },
>      .eth_dev_init = eth_vmxnet3_dev_init,
>      .dev_private_size = sizeof(struct vmxnet3_adapter),
> 
> thanks,
> -aziz

Re: [dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD

[Patel, Rashmin N]

I wasn't sure about why anyone would "avoid using UIO just for Virtio even though other devices need uio/vfio in DPDK." I mean I haven't seen anyone using that way and performance wise they both are similar.

But anyways here are the major reasons we had Intel versions:
1. Intel Virtio-PMD version 
	- was elegantly integrated in DPDK code tree and to keep consistency it uses UIO/VFIO framework just like any other PMDs in DPDK/lib 

2. Intel VMXNET3-PMD 
	- does not depend on any other module (i.e. vmxnet3 plugin depends on vmxnet3-usermap.ko module) 	- was elegantly integrated in DPDK code tree and to keep consistency it uses UIO/VFIO framework just like any other PMDs in DPDK/lib
	- is the version VMware contributes today as they think it's a better idea and they would provide ESXi support for that if they need to

Thanks,
Rashmin


-----Original Message-----
From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com] 
Sent: Sunday, November 09, 2014 4:18 PM
To: dev@dpdk.org
Cc: Patel, Rashmin N; Aziz Hajee
Subject: Re: [dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD

Hi,

2014-11-07 16:53, Patel, Rashmin N:
> Yes, you're right DPDK VMXNET3-PMD in /lib/librte_pmd_vmxnet3 does not 
> support mbuf chaining today. But it's a standalone bsd driver just 
> like any other pmd in that directory, it does not need vmxnet3-usermap.ko module.
> 
> Now there is another vmxnet3 solution in a separate branch as a 
> plugin, which must have vmxnet3-usermap.ko linux module(1), and a user 
> space interface piece(2) to tie it to any DPDK application in the main branch.
> (1) and (2) makes the solution which is known as vmxnet3-plugin. It's 
> been there for a long time just like virtio-plugin, I don't know who 
> uses it, but community can *reply* here if there is still any need of 
> a separate solution that way.

Coming back to last year, 6WIND developped some PMDs for virtio and vmxnet3.
Later, Intel developped their own version using the uio framework.
The versions in the main repository are the Intel ones, whereas the original ones from 6WIND are released as extensions.
For completeness, it must be noted that Brocade worked on their own approach of vmxnet3 and contributed to virtio PMD.
It's now time to merge all these implementations.

The 6WIND implementations show that it's possible to avoid the uio framework.
The virtio-net-pmd use port access granted by iopl().
The vmxnet3-usermap reuse the VMware's kernel module with a special mode for memory mapping. It was a pre-bifurcated logic.

> I'm in favor of consolidating all those version into one elegant 
> solution by grabbing best features from all of them and maintain one 
> copy. I'm sure that developers contributing from VMware would also 
> support that idea because then it makes easy to maintain and debug and 
> bug fix and most importantly avoid such confusion in future.


> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Aziz Hajee
> Sent: Thursday, November 06, 2014 5:47 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD
> 
> I am using the dpdk1.6.0r1
> I could not find a complete clarification, sorry if missed.
> VMXNET3 PMD
> ============
> I have enabled the VMXNET3 PMD  in the dpdk.
>  # Compile burst-oriented VMXNET3 PMD driver  #
> 
> CONFIG_RTE_LIBRTE_VMXNET3_PMD=y
> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_INIT=y
> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_RX=n
> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_TX=n
> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_TX_FREE=n
> The Intel DPDK VMXNET3 PMD driver does not support mbuf chaining, and I have to set NOMULTSEGS for the vmxnet3 interface init to succeed.
> tx_conf.txq_flags =  ETH_TXQ_FLAGS_NOMULTSEGS Is there a later version 
> of DPDK that supports multiseg for the dpdk
> VMXNET3 PMD.
> 
> vmware vmxnet3-usermap AND  DPDK VMXNET3 PMD 
> =========================================
> Is the vmxnet3-usermap.ko module driver also needed ? (appears that I need, otherwise the eal initialise fails.
> sudo insmod ./vmxnet3-usermap.ko enable_shm=2,2 num_rqs=1,1 
> num_rxds=2048
> num_txds=2048
> 
> I do not understand if VMXNET3 PMD is there, what is the purpose of /vmxnet3-usermap.ko/vmxnet3-usermap.ko ?
> 
> From some responses i saw that the following ifdef RTE_EAL_UNBIND_PORTS is also need to be removed in lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c ?
>      {
>          .name = "rte_vmxnet3_pmd",
>          .id_table = pci_id_vmxnet3_map, -#ifdef RTE_EAL_UNBIND_PORTS
> +// #ifdef RTE_EAL_UNBIND_PORTS
>          .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO, -#endif
> +// #endif
>      },
>      .eth_dev_init = eth_vmxnet3_dev_init,
>      .dev_private_size = sizeof(struct vmxnet3_adapter),
> 
> thanks,
> -aziz

Re: [dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD

[Aziz Hajee]

Rashmin,
Since I do need the jumbo, I use the vmxnet3-plugin you described, i.e.
(1)
sudo insmod ./vmxnet3-usermap.ko enable_shm=2,2 num_rqs=1,1 num_rxds=2048
num_txds=2048
and (2) when running the application, use in the args list:
"-d", "librte_pmd_vmxnet3.so"

Does the above two piece mean vmxnet3-plugin

I do see my vmxnet3 device from the dump,    rte_eal_pci_dump();
but the 'nb_ports' in DPDK never gets incremented rte_eth_dev_count()
returns zero.
so all the other api fails, if (port_id >= nb_ports) {
                PMD_DEBUG_TRACE("Invalid port_id=%d\n", port_id);
                return;
        }


0000:03:00.0 - vendor:15ad device:7b0
   00000000d2404000 0000000000001000
   00000000d2403000 0000000000001000
   00000000d2400000 0000000000002000
   0000000000000000 0000000000000000
   0000000000000000 0000000000000000
   0000000000000000 0000000000000000
   00000000d4400000 0000000000010000
0000:0b:00.0 - vendor:15ad device:7b0
   00000000d2504000 0000000000001000
   00000000d2503000 0000000000001000
   00000000d2500000 0000000000002000
   0000000000000000 0000000000000000
   0000000000000000 0000000000000000
   0000000000000000 0000000000000000
   00000000d4500000 0000000000010000
DPDK: No Ethernet ports (rte_eth_dev_count() returns zero)
PMD: rte_eth_dev_info_get: Invalid port_id=0
PMD: rte_eth_dev_configure: Invalid port_id=0
PMD: rte_eth_dev_info_get: Invalid port_id=0
PMD: rte_eth_dev_configure: Invalid port_id=0

So when using not using DPDK PMD for VMXNET3, what am i missing, for the
the DPDK to know the nb_ports,
How will the rte_eth_dev_start(portid) in DPDK library know  ?

rte_pmd_init_all() will not have the init the Intel DPDK PMD ,
RTE_LIBRTE_VMXNET3_PMD = n in config.
#ifdef RTE_LIBRTE_VMXNET3_PMD
        if ((ret = rte_vmxnet3_pmd_init()) != 0) {
                RTE_LOG(ERR, PMD, "Cannot init vmxnet3 PMD\n");
                return (ret);
        }

If I make RTE_LIBRTE_VMXNET3_PMD = y, then I am using the Intel DPDK PMD
and no jumbo.

Thanks,
aziz

On Fri, Nov 7, 2014 at 8:53 AM, Patel, Rashmin N <rashmin.n.patel@intel.com>
wrote:

> Hi Aziz,
>
> Yes, you're right DPDK VMXNET3-PMD in /lib/librte_pmd_vmxnet3 does not
> support mbuf chaining today. But it's a standalone bsd driver just like any
> other pmd in that directory, it does not need vmxnet3-usermap.ko module.
>
> Now there is another vmxnet3 solution in a separate branch as a plugin,
> which must have vmxnet3-usermap.ko linux module(1), and a user space
> interface piece(2) to tie it to any DPDK application in the main branch.
> (1) and (2) makes the solution which is known as vmxnet3-plugin. It's been
> there for a long time just like virtio-plugin, I don't know who uses it,
> but community can *reply* here if there is still any need of a separate
> solution that way.
>
> I'm in favor of consolidating all those version into one elegant solution
> by grabbing best features from all of them and maintain one copy. I'm sure
> that developers contributing from VMware would also support that idea
> because then it makes easy to maintain and debug and bug fix and most
> importantly avoid such confusion in future.
>
> Thanks,
> Rashmin
>
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Aziz Hajee
> Sent: Thursday, November 06, 2014 5:47 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD
>
> I am using the dpdk1.6.0r1
> I could not find a complete clarification, sorry if missed.
> VMXNET3 PMD
> ============
> I have enabled the VMXNET3 PMD  in the dpdk.
>  # Compile burst-oriented VMXNET3 PMD driver  #
>
> CONFIG_RTE_LIBRTE_VMXNET3_PMD=y
> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_INIT=y
> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_RX=n
> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_TX=n
> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_TX_FREE=n
> The Intel DPDK VMXNET3 PMD driver does not support mbuf chaining, and I
> have to set NOMULTSEGS for the vmxnet3 interface init to succeed.
> tx_conf.txq_flags =  ETH_TXQ_FLAGS_NOMULTSEGS Is there a later version of
> DPDK that supports multiseg for the dpdk
> VMXNET3 PMD.
>
> vmware vmxnet3-usermap AND  DPDK VMXNET3 PMD
> =========================================
> Is the vmxnet3-usermap.ko module driver also needed ? (appears that I
> need, otherwise the eal initialise fails.
> sudo insmod ./vmxnet3-usermap.ko enable_shm=2,2 num_rqs=1,1 num_rxds=2048
> num_txds=2048
>
> I do not understand if VMXNET3 PMD is there, what is the purpose of
> /vmxnet3-usermap.ko/vmxnet3-usermap.ko ?
>
> From some responses i saw that the following ifdef RTE_EAL_UNBIND_PORTS is
> also need to be removed in lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c ?
>      {
>          .name = "rte_vmxnet3_pmd",
>          .id_table = pci_id_vmxnet3_map, -#ifdef RTE_EAL_UNBIND_PORTS
> +// #ifdef RTE_EAL_UNBIND_PORTS
>          .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO, -#endif
> +// #endif
>      },
>      .eth_dev_init = eth_vmxnet3_dev_init,
>      .dev_private_size = sizeof(struct vmxnet3_adapter),
>
> thanks,
> -aziz
>

Re: [dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD

[Patel, Rashmin N]

Please find comments in-lined.

Thanks,
RP

From: Aziz Hajee [mailto:aziz@saisei.com]
Sent: Monday, November 10, 2014 6:00 PM
To: Patel, Rashmin N
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD

Rashmin,
Since I do need the jumbo, I use the vmxnet3-plugin you described, i.e.
(1)
sudo insmod ./vmxnet3-usermap.ko enable_shm=2,2 num_rqs=1,1 num_rxds=2048 num_txds=2048
and (2) when running the application, use in the args list:
"-d", "librte_pmd_vmxnet3.so"
Does the above two piece mean vmxnet3-plugin
[RP] that’s correct
I do see my vmxnet3 device from the dump,    rte_eal_pci_dump();
but the 'nb_ports' in DPDK never gets incremented rte_eth_dev_count() returns zero.
so all the other api fails, if (port_id >= nb_ports) {
                PMD_DEBUG_TRACE("Invalid port_id=%d\n", port_id);
                return;
        }

0000:03:00.0 - vendor:15ad device:7b0
   00000000d2404000 0000000000001000
   00000000d2403000 0000000000001000
   00000000d2400000 0000000000002000
   0000000000000000 0000000000000000
   0000000000000000 0000000000000000
   0000000000000000 0000000000000000
   00000000d4400000 0000000000010000
0000:0b:00.0 - vendor:15ad device:7b0
   00000000d2504000 0000000000001000
   00000000d2503000 0000000000001000
   00000000d2500000 0000000000002000
   0000000000000000 0000000000000000
   0000000000000000 0000000000000000
   0000000000000000 0000000000000000
   00000000d4500000 0000000000010000
DPDK: No Ethernet ports (rte_eth_dev_count() returns zero)
PMD: rte_eth_dev_info_get: Invalid port_id=0
PMD: rte_eth_dev_configure: Invalid port_id=0
PMD: rte_eth_dev_info_get: Invalid port_id=0
PMD: rte_eth_dev_configure: Invalid port_id=0
So when using not using DPDK PMD for VMXNET3, what am i missing, for the the DPDK to know the nb_ports,
How will the rte_eth_dev_start(portid) in DPDK library know  ?

rte_pmd_init_all() will not have the init the Intel DPDK PMD , RTE_LIBRTE_VMXNET3_PMD = n in config.
[RP] If you’re using the vmxnet3-plugin, you should keep RTE_LIBRTE_VMXNET3_PMD = n in config, and link the shared library with its headers, it should work. I have tried it once a long back.

#ifdef RTE_LIBRTE_VMXNET3_PMD
        if ((ret = rte_vmxnet3_pmd_init()) != 0) {
                RTE_LOG(ERR, PMD, "Cannot init vmxnet3 PMD\n");
                return (ret);
        }
If I make RTE_LIBRTE_VMXNET3_PMD = y, then I am using the Intel DPDK PMD and no jumbo.
[RP] Yes, I understood that part. We need to support jumbo frames in in-tree version of VMXNET3-PMD, we’ll merge all soon, we can discuss in the community conf. call so please do attend the next one on Nov 18 and we can raise concerns there

Thanks,
aziz

On Fri, Nov 7, 2014 at 8:53 AM, Patel, Rashmin N <rashmin.n.patel@intel.com<mailto:rashmin.n.patel@intel.com>> wrote:
Hi Aziz,

Yes, you're right DPDK VMXNET3-PMD in /lib/librte_pmd_vmxnet3 does not support mbuf chaining today. But it's a standalone bsd driver just like any other pmd in that directory, it does not need vmxnet3-usermap.ko module.

Now there is another vmxnet3 solution in a separate branch as a plugin, which must have vmxnet3-usermap.ko linux module(1), and a user space interface piece(2) to tie it to any DPDK application in the main branch. (1) and (2) makes the solution which is known as vmxnet3-plugin. It's been there for a long time just like virtio-plugin, I don't know who uses it, but community can *reply* here if there is still any need of a separate solution that way.

I'm in favor of consolidating all those version into one elegant solution by grabbing best features from all of them and maintain one copy. I'm sure that developers contributing from VMware would also support that idea because then it makes easy to maintain and debug and bug fix and most importantly avoid such confusion in future.

Thanks,
Rashmin

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>] On Behalf Of Aziz Hajee
Sent: Thursday, November 06, 2014 5:47 PM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Subject: [dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD

I am using the dpdk1.6.0r1
I could not find a complete clarification, sorry if missed.
VMXNET3 PMD
============
I have enabled the VMXNET3 PMD  in the dpdk.
 # Compile burst-oriented VMXNET3 PMD driver  #

CONFIG_RTE_LIBRTE_VMXNET3_PMD=y
CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_INIT=y
CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_RX=n
CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_TX=n
CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_TX_FREE=n
The Intel DPDK VMXNET3 PMD driver does not support mbuf chaining, and I have to set NOMULTSEGS for the vmxnet3 interface init to succeed.
tx_conf.txq_flags =  ETH_TXQ_FLAGS_NOMULTSEGS Is there a later version of DPDK that supports multiseg for the dpdk
VMXNET3 PMD.

vmware vmxnet3-usermap AND  DPDK VMXNET3 PMD =========================================
Is the vmxnet3-usermap.ko module driver also needed ? (appears that I need, otherwise the eal initialise fails.
sudo insmod ./vmxnet3-usermap.ko enable_shm=2,2 num_rqs=1,1 num_rxds=2048
num_txds=2048

I do not understand if VMXNET3 PMD is there, what is the purpose of /vmxnet3-usermap.ko/vmxnet3-usermap.ko ?

From some responses i saw that the following ifdef RTE_EAL_UNBIND_PORTS is also need to be removed in lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c ?
     {
         .name = "rte_vmxnet3_pmd",
         .id_table = pci_id_vmxnet3_map, -#ifdef RTE_EAL_UNBIND_PORTS
+// #ifdef RTE_EAL_UNBIND_PORTS
         .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO, -#endif
+// #endif
     },
     .eth_dev_init = eth_vmxnet3_dev_init,
     .dev_private_size = sizeof(struct vmxnet3_adapter),

thanks,
-aziz

Re: [dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD

[Aziz Hajee]

Thanks Rashmin,
I am close, just getting this solib loading undefined symbol error: (I mod
code to see the error). Using dpdk-1.6.0r1_ss

EAL: Setting up memory...
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fc0a8200000 (size = 0x200000)
EAL: Ask a virtual area of 0x5c00000 bytes
EAL: Virtual area found at 0x7fc07e200000 (size = 0x5c00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fc0a1c00000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fc0a1800000 (size = 0x200000)
EAL: Ask a virtual area of 0x19c00000 bytes
EAL: Virtual area found at 0x7fc064400000 (size = 0x19c00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fc0a1400000 (size = 0x200000)
EAL: Requesting 256 pages of size 2MB from socket 0
EAL: TSC frequency is ~2300000 KHz
Cannot load solib librte_pmd_vmxnet3.so DLERROR:./librte_pmd_vmxnet3.so:
undefined symbol: per_lcore__lcore_id
EAL: (null)

coming from eal.c /* Launch threads, called at application init(). */
int
rte_eal_init(int argc, char **argv)

       TAILQ_FOREACH(solib, &solib_list, next) {
                solib->lib_handle = dlopen(solib->name, RTLD_NOW);
                if ((solib->lib_handle == NULL) && (solib->name[0] != '/'))
{
                        /* relative path: try again with "./" prefix */
                        char sopath[PATH_MAX];
                        snprintf(sopath, sizeof(sopath), "./%s",
solib->name);
                        solib->lib_handle = dlopen(sopath, RTLD_NOW);
                }
                if (solib->lib_handle == NULL)
                        printf("Cannot load solib %s DLERROR:%s\n",
solib->name, dlerror());
//                      RTE_LOG(WARNING, EAL, "%s\n", dlerror());

        }


On Tue, Nov 11, 2014 at 9:19 AM, Patel, Rashmin N <rashmin.n.patel@intel.com
> wrote:

>  Please find comments in-lined.
>
>
>
> Thanks,
>
> RP
>
>
>
> *From:* Aziz Hajee [mailto:aziz@saisei.com]
> *Sent:* Monday, November 10, 2014 6:00 PM
> *To:* Patel, Rashmin N
> *Cc:* dev@dpdk.org
> *Subject:* Re: [dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD
>
>
>
> Rashmin,
> Since I do need the jumbo, I use the vmxnet3-plugin you described, i.e.
> (1)
> sudo insmod ./vmxnet3-usermap.ko enable_shm=2,2 num_rqs=1,1 num_rxds=2048
> num_txds=2048
>
> and (2) when running the application, use in the args list:
>
> "-d", "librte_pmd_vmxnet3.so"
>
> Does the above two piece mean vmxnet3-plugin
>
> [RP] that’s correct
>
> I do see my vmxnet3 device from the dump,    rte_eal_pci_dump();
> but the 'nb_ports' in DPDK never gets incremented rte_eth_dev_count()
> returns zero.
> so all the other api fails, if (port_id >= nb_ports) {
>                 PMD_DEBUG_TRACE("Invalid port_id=%d\n", port_id);
>                 return;
>         }
>
>   0000:03:00.0 - vendor:15ad device:7b0
>    00000000d2404000 0000000000001000
>    00000000d2403000 0000000000001000
>    00000000d2400000 0000000000002000
>    0000000000000000 0000000000000000
>    0000000000000000 0000000000000000
>    0000000000000000 0000000000000000
>    00000000d4400000 0000000000010000
> 0000:0b:00.0 - vendor:15ad device:7b0
>    00000000d2504000 0000000000001000
>    00000000d2503000 0000000000001000
>    00000000d2500000 0000000000002000
>    0000000000000000 0000000000000000
>    0000000000000000 0000000000000000
>    0000000000000000 0000000000000000
>    00000000d4500000 0000000000010000
> DPDK: No Ethernet ports (rte_eth_dev_count() returns zero)
> PMD: rte_eth_dev_info_get: Invalid port_id=0
> PMD: rte_eth_dev_configure: Invalid port_id=0
> PMD: rte_eth_dev_info_get: Invalid port_id=0
> PMD: rte_eth_dev_configure: Invalid port_id=0
>
> So when using not using DPDK PMD for VMXNET3, what am i missing, for the
> the DPDK to know the nb_ports,
>
> How will the rte_eth_dev_start(portid) in DPDK library know  ?
>
> rte_pmd_init_all() will not have the init the Intel DPDK PMD ,
> RTE_LIBRTE_VMXNET3_PMD = n in config.
>
> [RP] If you’re using the vmxnet3-plugin, you should keep
> RTE_LIBRTE_VMXNET3_PMD = n in config, and link the shared library with its
> headers, it should work. I have tried it once a long back.
>
>
> #ifdef RTE_LIBRTE_VMXNET3_PMD
>         if ((ret = rte_vmxnet3_pmd_init()) != 0) {
>                 RTE_LOG(ERR, PMD, "Cannot init vmxnet3 PMD\n");
>                 return (ret);
>         }
>
> If I make RTE_LIBRTE_VMXNET3_PMD = y, then I am using the Intel DPDK PMD
> and no jumbo.
>
> [RP] Yes, I understood that part. We need to support jumbo frames in
> in-tree version of VMXNET3-PMD, we’ll merge all soon, we can discuss in the
> community conf. call so please do attend the next one on Nov 18 and we can
> raise concerns there
>
>
>
> Thanks,
>
> aziz
>
>
>
> On Fri, Nov 7, 2014 at 8:53 AM, Patel, Rashmin N <
> rashmin.n.patel@intel.com> wrote:
>
> Hi Aziz,
>
> Yes, you're right DPDK VMXNET3-PMD in /lib/librte_pmd_vmxnet3 does not
> support mbuf chaining today. But it's a standalone bsd driver just like any
> other pmd in that directory, it does not need vmxnet3-usermap.ko module.
>
> Now there is another vmxnet3 solution in a separate branch as a plugin,
> which must have vmxnet3-usermap.ko linux module(1), and a user space
> interface piece(2) to tie it to any DPDK application in the main branch.
> (1) and (2) makes the solution which is known as vmxnet3-plugin. It's been
> there for a long time just like virtio-plugin, I don't know who uses it,
> but community can *reply* here if there is still any need of a separate
> solution that way.
>
> I'm in favor of consolidating all those version into one elegant solution
> by grabbing best features from all of them and maintain one copy. I'm sure
> that developers contributing from VMware would also support that idea
> because then it makes easy to maintain and debug and bug fix and most
> importantly avoid such confusion in future.
>
> Thanks,
> Rashmin
>
>
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Aziz Hajee
> Sent: Thursday, November 06, 2014 5:47 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD
>
> I am using the dpdk1.6.0r1
> I could not find a complete clarification, sorry if missed.
> VMXNET3 PMD
> ============
> I have enabled the VMXNET3 PMD  in the dpdk.
>  # Compile burst-oriented VMXNET3 PMD driver  #
>
> CONFIG_RTE_LIBRTE_VMXNET3_PMD=y
> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_INIT=y
> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_RX=n
> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_TX=n
> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_TX_FREE=n
> The Intel DPDK VMXNET3 PMD driver does not support mbuf chaining, and I
> have to set NOMULTSEGS for the vmxnet3 interface init to succeed.
> tx_conf.txq_flags =  ETH_TXQ_FLAGS_NOMULTSEGS Is there a later version of
> DPDK that supports multiseg for the dpdk
> VMXNET3 PMD.
>
> vmware vmxnet3-usermap AND  DPDK VMXNET3 PMD
> =========================================
> Is the vmxnet3-usermap.ko module driver also needed ? (appears that I
> need, otherwise the eal initialise fails.
> sudo insmod ./vmxnet3-usermap.ko enable_shm=2,2 num_rqs=1,1 num_rxds=2048
> num_txds=2048
>
> I do not understand if VMXNET3 PMD is there, what is the purpose of
> /vmxnet3-usermap.ko/vmxnet3-usermap.ko ?
>
> From some responses i saw that the following ifdef RTE_EAL_UNBIND_PORTS is
> also need to be removed in lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c ?
>      {
>          .name = "rte_vmxnet3_pmd",
>          .id_table = pci_id_vmxnet3_map, -#ifdef RTE_EAL_UNBIND_PORTS
> +// #ifdef RTE_EAL_UNBIND_PORTS
>          .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO, -#endif
> +// #endif
>      },
>      .eth_dev_init = eth_vmxnet3_dev_init,
>      .dev_private_size = sizeof(struct vmxnet3_adapter),
>
> thanks,
> -aziz
>
>
>