[dpdk-dev] DPDK mbuf pool in SR-IOV env and one RX/TX queue

[dpdk-dev] DPDK mbuf pool in SR-IOV env and one RX/TX queue

[Saurabh Mishra]

Hi,


Is it possible for two or more processes to share the same mbuf_pool in
SR-IOV with single rx/tx queue?



 char *eal_argv[] = {"fakeelf",

                        "-c2",

                        "-n4",

                        "--proc-type=primary",};


int ret = rte_eal_init(4, eal_argv);

And for secondary, we are passing --proc-type=secondary and adjust -c
option according to the core number (1 << core_num).

rte_pktmbuf_pool_create(mbuf_pool_name,
NB_MBUFS,

                            MBUF_CACHE_SIZE, 0,
RTE_MBUF_DEFAULT_BUF_SIZE,

                            rte_socket_id());



We have seen that it does not work. However, with PCI pass through of whole
NIC, we can easily share same mbuf_pool with many processes.


What are the limitations with SR-IOV and DPDK? If we create one primary and
don't create any  secondary proc-type, then that seems to work with SR-IOV.
However, the moment we add a secondary process, rte_mempool_lookup()
returns NULL.


Note that we see only 1 queue for rx and tx and multi-queue gets disabled
the moment we enable VFs.


[683877.050219] ixgbe 0000:06:00.0: *Multi*queue Disabled: Rx Queue count =
1, Tx Queue count = 1

[683879.799872] ixgbe 0000:06:00.1: *Multi*queue Disabled: Rx Queue count =
1, Tx Queue count = 1

The device is question is as follows:



06:00.0 *Ether*net controller: Intel Corporation 82599ES 10-Gigabit
SFI/SFP+ Network Connection (rev 01)

06:00.1 *Ether*net controller: Intel Corporation 82599ES 10-Gigabit
SFI/SFP+ Network Connection (rev 01)

06:10.0 *Ether*net controller: Intel Corporation 82599 *Ether*net
Controller Virtual Function (rev 01)

06:10.1 *Ether*net controller: Intel Corporation 82599 *Ether*net
Controller Virtual Function (rev 01)

06:10.2 *Ether*net controller: Intel Corporation 82599 *Ether*net
Controller Virtual Function (rev 01)

06:10.3 *Ether*net controller: Intel Corporation 82599 *Ether*net
Controller Virtual Function (rev 01)



Thanks,

/Saurabh

Re: [dpdk-dev] DPDK mbuf pool in SR-IOV env and one RX/TX queue

[Bruce Richardson]

On Thu, Jan 21, 2016 at 08:35:20PM -0800, Saurabh Mishra wrote:
> Hi,
> 
> 
> Is it possible for two or more processes to share the same mbuf_pool in
> SR-IOV with single rx/tx queue?
> 
>
The sharing of the mbuf pool is not an issue, but sharing of rx/tx queues is.
The ethdev queues are not multi-thread safe, so to share a queue between processes
or threads, you need to put in locks or other access control mechanisms. [This
also implies a performance hit due to the locking]

Regards,
/Bruce

Re: [dpdk-dev] DPDK mbuf pool in SR-IOV env and one RX/TX queue

[Saurabh Mishra]

Hi Bruce --

>The sharing of the mbuf pool is not an issue, but sharing of rx/tx queues
is.
>The ethdev queues are not multi-thread safe, so to share a queue between
processes
>or threads, you need to put in locks or other access control mechanisms.
[This
>also implies a performance hit due to the locking]
>Regards,
>/Bruce

Right. So now we have only one process to do rx/tx on queue 0 if we detect
that max queue support is 1.

However, we have noticed that if our process, which does rx/tx, is not
primary, then we can't transmit the packet out with SR-IOV.

Is there any specific limitation on SR-IOV (the vf driver in dpdk) that
only primary process should receive and transmit packets?

In our model, we have an agent process which monitor links and another
process which does packet processing. If we make our agent process as
primary then our secondary process is not able to send the packets --
rte_eth_tx_burst() succeed but recipient does not receive the packet.

Thanks,
/Saurabh

On Sat, Jan 23, 2016 at 8:09 AM, Bruce Richardson <
bruce.richardson@intel.com> wrote:

> On Thu, Jan 21, 2016 at 08:35:20PM -0800, Saurabh Mishra wrote:
> > Hi,
> >
> >
> > Is it possible for two or more processes to share the same mbuf_pool in
> > SR-IOV with single rx/tx queue?
> >
> >
> The sharing of the mbuf pool is not an issue, but sharing of rx/tx queues
> is.
> The ethdev queues are not multi-thread safe, so to share a queue between
> processes
> or threads, you need to put in locks or other access control mechanisms.
> [This
> also implies a performance hit due to the locking]
>
> Regards,
> /Bruce
>
>

Re: [dpdk-dev] DPDK mbuf pool in SR-IOV env and one RX/TX queue

[Bruce Richardson]

On Mon, Jan 25, 2016 at 04:15:28PM -0800, Saurabh Mishra wrote:
> Hi Bruce --
> 
> >The sharing of the mbuf pool is not an issue, but sharing of rx/tx queues
> is.
> >The ethdev queues are not multi-thread safe, so to share a queue between
> processes
> >or threads, you need to put in locks or other access control mechanisms.
> [This
> >also implies a performance hit due to the locking]
> >Regards,
> >/Bruce
> 
> Right. So now we have only one process to do rx/tx on queue 0 if we detect
> that max queue support is 1.
> 
> However, we have noticed that if our process, which does rx/tx, is not
> primary, then we can't transmit the packet out with SR-IOV.
> 
> Is there any specific limitation on SR-IOV (the vf driver in dpdk) that
> only primary process should receive and transmit packets?
> 
> In our model, we have an agent process which monitor links and another
> process which does packet processing. If we make our agent process as
> primary then our secondary process is not able to send the packets --
> rte_eth_tx_burst() succeed but recipient does not receive the packet.
> 
> Thanks,
> /Saurabh

There should be no restrictions on RX/TX from secondary processes.

/Bruce

Re: [dpdk-dev] DPDK mbuf pool in SR-IOV env and one RX/TX queue

[Saurabh Mishra]

Any clues or hint on how to debug this kind of problem on SR-IOV? Only
primary can send packet but secondary process couldn't. I have verified
host's qprc and qptc counters on PF and they do increment.

SR-IOV with DPDK seems more challenging than PCI pass through of whole NIC.

Saurabh
On Jan 26, 2016 12:19 PM, "Bruce Richardson" <bruce.richardson@intel.com>
wrote:

> On Mon, Jan 25, 2016 at 04:15:28PM -0800, Saurabh Mishra wrote:
> > Hi Bruce --
> >
> > >The sharing of the mbuf pool is not an issue, but sharing of rx/tx
> queues
> > is.
> > >The ethdev queues are not multi-thread safe, so to share a queue between
> > processes
> > >or threads, you need to put in locks or other access control mechanisms.
> > [This
> > >also implies a performance hit due to the locking]
> > >Regards,
> > >/Bruce
> >
> > Right. So now we have only one process to do rx/tx on queue 0 if we
> detect
> > that max queue support is 1.
> >
> > However, we have noticed that if our process, which does rx/tx, is not
> > primary, then we can't transmit the packet out with SR-IOV.
> >
> > Is there any specific limitation on SR-IOV (the vf driver in dpdk) that
> > only primary process should receive and transmit packets?
> >
> > In our model, we have an agent process which monitor links and another
> > process which does packet processing. If we make our agent process as
> > primary then our secondary process is not able to send the packets --
> > rte_eth_tx_burst() succeed but recipient does not receive the packet.
> >
> > Thanks,
> > /Saurabh
>
> There should be no restrictions on RX/TX from secondary processes.
>
> /Bruce
>