[dpdk-users] DPDK 2.2 MLX4: problem with number of TX/RX queues

[dpdk-users] DPDK 2.2 MLX4: problem with number of TX/RX queues

[Non Voglio]

Hello,
  I am new to the DPDK world and I am having some troubles in using some
code (not written by me).

In particular, I bump into this error when invoking the  rte_eth_dev_configure
function:

PMD: librte_pmd_mlx4: 0x10a4d2a0: TX queues number update: 0 -> 16

PMD: librte_pmd_mlx4: 0x10a4d2a0: RX queues number update: 0 -> 16

PMD: librte_pmd_mlx4: 0x10a4d2a0: only a single RX queue can be configured
when hardware doesn't support RSS


On the other hand, it seems that my hardware is configured to have 16 RX/TX
queues (equal to the number of cores)

$ ethtool -l p785p1

Channel parameters for p785p1:

Pre-set maximums:

RX: 128

TX: 32

Other: 0

Combined: 0

Current hardware settings:

RX: 16

TX: 16

Other: 0

Combined: 0

------ Additional info

I am using DPDK 2.2

I have followed the instructions in [1] to install
MLNX_OFED_LINUX-3.1-1.0.3-ubuntu14.04-x86_64
  (3.2 is out, but instructions in [1] refers to 3.1)

My NIC :

05:00.0 Ethernet controller: Mellanox Technologies MT27520 Family
[ConnectX-3 Pro]

My machine runs a 3.19.0-47-generic on Ubuntu 14.04.

CPU is Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz  2x8 cores

------

Note that, in the end, the software I am going to use is intended to use
flow director and not RSS.


Any idea? I have the feeling the solution might be trivial, but I am a noob
in this domain and I have searched around, finding no answer.


Thank you

  --Harold


[1] http://dpdk.org/doc/guides/nics/mlx4.html

Re: [dpdk-users] DPDK 2.2 MLX4: problem with number of TX/RX queues

[Vincent Li]

On Thu, Feb 11, 2016 at 6:24 AM, Non Voglio <harold.demure87@gmail.com> wrote:
> Hello,
>   I am new to the DPDK world and I am having some troubles in using some
> code (not written by me).
>
> In particular, I bump into this error when invoking the  rte_eth_dev_configure
> function:
>
> PMD: librte_pmd_mlx4: 0x10a4d2a0: TX queues number update: 0 -> 16
>
> PMD: librte_pmd_mlx4: 0x10a4d2a0: RX queues number update: 0 -> 16
>
> PMD: librte_pmd_mlx4: 0x10a4d2a0: only a single RX queue can be configured
> when hardware doesn't support RSS
>

it appears you must enable RSS support to use multiple RX queue

        if (!priv->hw_rss) {
                ERROR("%p: only a single RX queue can be configured when"
                      " hardware doesn't support RSS",
                      (void *)dev);
                return EINVAL;
        }

Vincent

Re: [dpdk-users] DPDK 2.2 MLX4: problem with number of TX/RX queues

[Harold Demure]

Hello Vincent,
  Thank you for your reply. I followed the same reasoning as you did,
looking at the source code earlier today.

Can you send me a link that explains how to change RX for my NIC?
I have been looking around for that, and also performed the steps in the
DPDK official guide for MLX4, but apparently nothing changed (at least, the
error was still there).

In addition, I have to describe a new strange behavior: even *without
changing anything rss-wise*, but just enabling the increased debugging
verbosity (CONFIG_RTE_LIBRTE_MLX4_DEBUG=y), the error I had suddenly
disappeared.

I cannot really explain what happened. Can anybody?

At this point (with the debug prints), now my code blocks because
apparently mlx4 does not support FlowDir, so I have to change a bit my
software, which is built to use it.

Still, I would love to know what's going on with my rx/tx queues.

Thank you.
Regards,
   Harold

2016-02-11 20:10 GMT+01:00 Vincent Li <vincent.mc.li@gmail.com>:

> On Thu, Feb 11, 2016 at 6:24 AM, Non Voglio <harold.demure87@gmail.com>
> wrote:
> > Hello,
> >   I am new to the DPDK world and I am having some troubles in using some
> > code (not written by me).
> >
> > In particular, I bump into this error when invoking the
> rte_eth_dev_configure
> > function:
> >
> > PMD: librte_pmd_mlx4: 0x10a4d2a0: TX queues number update: 0 -> 16
> >
> > PMD: librte_pmd_mlx4: 0x10a4d2a0: RX queues number update: 0 -> 16
> >
> > PMD: librte_pmd_mlx4: 0x10a4d2a0: only a single RX queue can be
> configured
> > when hardware doesn't support RSS
> >
>
> it appears you must enable RSS support to use multiple RX queue
>
>         if (!priv->hw_rss) {
>                 ERROR("%p: only a single RX queue can be configured when"
>                       " hardware doesn't support RSS",
>                       (void *)dev);
>                 return EINVAL;
>         }
>
> Vincent
>

Re: [dpdk-users] DPDK 2.2 MLX4: problem with number of TX/RX queues

[Vincent Li]

On Thu, Feb 11, 2016 at 1:36 PM, Harold Demure
<harold.demure87@gmail.com> wrote:
> Hello Vincent,
>   Thank you for your reply. I followed the same reasoning as you did,
> looking at the source code earlier today.
>
> Can you send me a link that explains how to change RX for my NIC?
> I have been looking around for that, and also performed the steps in the
> DPDK official guide for MLX4, but apparently nothing changed (at least, the
> error was still there).
>

I am DPDK newbie too and no MLX4 nic to play with, I think you need to
find out how to enable RSS for MLX4

librte_pmd_mlx4/Makefile has some clue about RSS:

mlx4_autoconf.h: $(RTE_SDK)/scripts/auto-config-h.sh
        $Q $(RM) -f -- '$@'
        $Q sh -- '$<' '$@' \
                RSS_SUPPORT \
                infiniband/verbs.h \
                enum IBV_EXP_DEVICE_UD_RSS $(AUTOCONF_OUTPUT)

maybe you can find the mlx4_autoconf.h and see if you have RSS_SUPPORT in it?

librte_pmd_mlx4/mlx4.c has:

#ifdef RSS_SUPPORT
                if (ibv_exp_query_device(ctx, &exp_device_attr)) {
                        INFO("experimental ibv_exp_query_device");
                        goto port_error;
                }
                if ((exp_device_attr.exp_device_cap_flags &
                     IBV_EXP_DEVICE_QPG) &&
                    (exp_device_attr.exp_device_cap_flags &
                     IBV_EXP_DEVICE_UD_RSS) &&
                    (exp_device_attr.comp_mask &
                     IBV_EXP_DEVICE_ATTR_RSS_TBL_SZ) &&
                    (exp_device_attr.max_rss_tbl_sz > 0)) {
                        priv->hw_qpg = 1;
                        priv->hw_rss = 1;
//<======================================
                        priv->max_rss_tbl_sz = exp_device_attr.max_rss_tbl_sz;
                } else {
                        priv->hw_qpg = 0;
                        priv->hw_rss = 0;
                        priv->max_rss_tbl_sz = 0;
                }
                priv->hw_tss = !!(exp_device_attr.exp_device_cap_flags &
                                  IBV_EXP_DEVICE_UD_TSS);
                DEBUG("device flags: %s%s%s",
                      (priv->hw_qpg ? "IBV_DEVICE_QPG " : ""),
                      (priv->hw_tss ? "IBV_DEVICE_TSS " : ""),
                      (priv->hw_rss ? "IBV_DEVICE_RSS " : ""));
                if (priv->hw_rss)
                        DEBUG("maximum RSS indirection table size: %u",
                              exp_device_attr.max_rss_tbl_sz);
#endif /* RSS_SUPPORT */

make sure you  meet MLX4 library dependencies that has
ibv_exp_query_device to query mlx4 properly to make sure priv->hw_rss
= 1 ?

after you enabled RSS, then figure out how to enable multiple RX
queues or maybe it will be enabled automatically.

or you can ask MLX4 community

Re: [dpdk-users] DPDK 2.2 MLX4: problem with number of TX/RX queues

[Olga Shern]

Hi Harold, 

DPDK 2.2 with MLNX_OFED that you are using and ConnectX-3 support RSS automatically.
You don’t need to configure anything.

Looks there is some bug in the compilation, did you compile  with make -j, we saw some issue with it.
Can you please try without -j flag.

Best Regards,
Olga

-----Original Message-----
From: users [mailto:users-bounces@dpdk.org] On Behalf Of Harold Demure
Sent: Thursday, February 11, 2016 11:37 PM
To: Vincent Li
Cc: users@dpdk.org
Subject: Re: [dpdk-users] DPDK 2.2 MLX4: problem with number of TX/RX queues

Hello Vincent,
  Thank you for your reply. I followed the same reasoning as you did, looking at the source code earlier today.

Can you send me a link that explains how to change RX for my NIC?
I have been looking around for that, and also performed the steps in the DPDK official guide for MLX4, but apparently nothing changed (at least, the error was still there).

In addition, I have to describe a new strange behavior: even *without changing anything rss-wise*, but just enabling the increased debugging verbosity (CONFIG_RTE_LIBRTE_MLX4_DEBUG=y), the error I had suddenly disappeared.

I cannot really explain what happened. Can anybody?

At this point (with the debug prints), now my code blocks because apparently mlx4 does not support FlowDir, so I have to change a bit my software, which is built to use it.

Still, I would love to know what's going on with my rx/tx queues.

Thank you.
Regards,
   Harold

2016-02-11 20:10 GMT+01:00 Vincent Li <vincent.mc.li@gmail.com>:

> On Thu, Feb 11, 2016 at 6:24 AM, Non Voglio 
> <harold.demure87@gmail.com>
> wrote:
> > Hello,
> >   I am new to the DPDK world and I am having some troubles in using 
> > some code (not written by me).
> >
> > In particular, I bump into this error when invoking the
> rte_eth_dev_configure
> > function:
> >
> > PMD: librte_pmd_mlx4: 0x10a4d2a0: TX queues number update: 0 -> 16
> >
> > PMD: librte_pmd_mlx4: 0x10a4d2a0: RX queues number update: 0 -> 16
> >
> > PMD: librte_pmd_mlx4: 0x10a4d2a0: only a single RX queue can be
> configured
> > when hardware doesn't support RSS
> >
>
> it appears you must enable RSS support to use multiple RX queue
>
>         if (!priv->hw_rss) {
>                 ERROR("%p: only a single RX queue can be configured when"
>                       " hardware doesn't support RSS",
>                       (void *)dev);
>                 return EINVAL;
>         }
>
> Vincent
>

Re: [dpdk-users] DPDK 2.2 MLX4: problem with number of TX/RX queues

[Harold Demure]

Dear Olga and Vincent,
  thank you for your reply.
It seems that the problem has now gone away; both with the -j and without
the -j flag. I will try to backtrace the root cause of the problem and
report it if I manage to isolate it.

If I may, I would also like to ask your help for a related problem, which I
have reported also on the mellanox community forum. If you are available,
you may find my question there [1].

I take the occasion to specifically address Olga Shern because I saw that
she has already given support on a related topic [2].
Also, related to [2] but possibly un-related from the questions in [1], I
am failing to attach a "Perfect Match" filter to my mlx4 NIC, via

rte_eth_dev_filter_ctrl(port_id, RTE_ETH_FILTER_FDIR,
RTE_ETH_FILTER_ADD, &filter)

Is this an expected behavior, namely, mlx4 does not support (yet) this kind
of operational mode?

Thank you (and anybody willing to help) for your time.
Best regards,
  Harold Demure


[1] https://community.mellanox.com/thread/3152
[2] http://dpdk.org/ml/archives/dev/2015-April/016392.html

2016-02-15 9:59 GMT+01:00 Olga Shern <olgas@mellanox.com>:

> Hi Harold,
>
> DPDK 2.2 with MLNX_OFED that you are using and ConnectX-3 support RSS
> automatically.
> You don’t need to configure anything.
>
> Looks there is some bug in the compilation, did you compile  with make -j,
> we saw some issue with it.
> Can you please try without -j flag.
>
> Best Regards,
> Olga
>
> -----Original Message-----
> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Harold Demure
> Sent: Thursday, February 11, 2016 11:37 PM
> To: Vincent Li
> Cc: users@dpdk.org
> Subject: Re: [dpdk-users] DPDK 2.2 MLX4: problem with number of TX/RX
> queues
>
> Hello Vincent,
>   Thank you for your reply. I followed the same reasoning as you did,
> looking at the source code earlier today.
>
> Can you send me a link that explains how to change RX for my NIC?
> I have been looking around for that, and also performed the steps in the
> DPDK official guide for MLX4, but apparently nothing changed (at least, the
> error was still there).
>
> In addition, I have to describe a new strange behavior: even *without
> changing anything rss-wise*, but just enabling the increased debugging
> verbosity (CONFIG_RTE_LIBRTE_MLX4_DEBUG=y), the error I had suddenly
> disappeared.
>
> I cannot really explain what happened. Can anybody?
>
> At this point (with the debug prints), now my code blocks because
> apparently mlx4 does not support FlowDir, so I have to change a bit my
> software, which is built to use it.
>
> Still, I would love to know what's going on with my rx/tx queues.
>
> Thank you.
> Regards,
>    Harold
>
> 2016-02-11 20:10 GMT+01:00 Vincent Li <vincent.mc.li@gmail.com>:
>
> > On Thu, Feb 11, 2016 at 6:24 AM, Non Voglio
> > <harold.demure87@gmail.com>
> > wrote:
> > > Hello,
> > >   I am new to the DPDK world and I am having some troubles in using
> > > some code (not written by me).
> > >
> > > In particular, I bump into this error when invoking the
> > rte_eth_dev_configure
> > > function:
> > >
> > > PMD: librte_pmd_mlx4: 0x10a4d2a0: TX queues number update: 0 -> 16
> > >
> > > PMD: librte_pmd_mlx4: 0x10a4d2a0: RX queues number update: 0 -> 16
> > >
> > > PMD: librte_pmd_mlx4: 0x10a4d2a0: only a single RX queue can be
> > configured
> > > when hardware doesn't support RSS
> > >
> >
> > it appears you must enable RSS support to use multiple RX queue
> >
> >         if (!priv->hw_rss) {
> >                 ERROR("%p: only a single RX queue can be configured when"
> >                       " hardware doesn't support RSS",
> >                       (void *)dev);
> >                 return EINVAL;
> >         }
> >
> > Vincent
> >
>

Re: [dpdk-users] DPDK 2.2 MLX4: problem with number of TX/RX queues

[Gilad Berman]

Harold, 

Flow Director features will be supported only on ConnectX-4 (starting from MLNX_DPDK version to be released end of Feb and on). 
So sadly you can't use it with mlx4 (ConnectX-3). 

I answered also the community question, btw. 


Gilad Berman | Staff System Engineer | Business Development | Mellanox Technologies Ltd. 
Work: +972 52 2554262| 6 Ha’Barzel St. Tel Aviv 6971010, Israel


-----Original Message-----
From: users [mailto:users-bounces@dpdk.org] On Behalf Of Harold Demure
Sent: Tuesday, February 16, 2016 6:05 PM
To: Olga Shern <olgas@mellanox.com>
Cc: users@dpdk.org
Subject: Re: [dpdk-users] DPDK 2.2 MLX4: problem with number of TX/RX queues

Dear Olga and Vincent,
  thank you for your reply.
It seems that the problem has now gone away; both with the -j and without the -j flag. I will try to backtrace the root cause of the problem and report it if I manage to isolate it.

If I may, I would also like to ask your help for a related problem, which I have reported also on the mellanox community forum. If you are available, you may find my question there [1].

I take the occasion to specifically address Olga Shern because I saw that she has already given support on a related topic [2].
Also, related to [2] but possibly un-related from the questions in [1], I am failing to attach a "Perfect Match" filter to my mlx4 NIC, via

rte_eth_dev_filter_ctrl(port_id, RTE_ETH_FILTER_FDIR, RTE_ETH_FILTER_ADD, &filter)

Is this an expected behavior, namely, mlx4 does not support (yet) this kind of operational mode?

Thank you (and anybody willing to help) for your time.
Best regards,
  Harold Demure


[1] https://community.mellanox.com/thread/3152
[2] http://dpdk.org/ml/archives/dev/2015-April/016392.html

2016-02-15 9:59 GMT+01:00 Olga Shern <olgas@mellanox.com>:

> Hi Harold,
>
> DPDK 2.2 with MLNX_OFED that you are using and ConnectX-3 support RSS 
> automatically.
> You don’t need to configure anything.
>
> Looks there is some bug in the compilation, did you compile  with make 
> -j, we saw some issue with it.
> Can you please try without -j flag.
>
> Best Regards,
> Olga
>
> -----Original Message-----
> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Harold Demure
> Sent: Thursday, February 11, 2016 11:37 PM
> To: Vincent Li
> Cc: users@dpdk.org
> Subject: Re: [dpdk-users] DPDK 2.2 MLX4: problem with number of TX/RX 
> queues
>
> Hello Vincent,
>   Thank you for your reply. I followed the same reasoning as you did, 
> looking at the source code earlier today.
>
> Can you send me a link that explains how to change RX for my NIC?
> I have been looking around for that, and also performed the steps in 
> the DPDK official guide for MLX4, but apparently nothing changed (at 
> least, the error was still there).
>
> In addition, I have to describe a new strange behavior: even *without 
> changing anything rss-wise*, but just enabling the increased debugging 
> verbosity (CONFIG_RTE_LIBRTE_MLX4_DEBUG=y), the error I had suddenly 
> disappeared.
>
> I cannot really explain what happened. Can anybody?
>
> At this point (with the debug prints), now my code blocks because 
> apparently mlx4 does not support FlowDir, so I have to change a bit my 
> software, which is built to use it.
>
> Still, I would love to know what's going on with my rx/tx queues.
>
> Thank you.
> Regards,
>    Harold
>
> 2016-02-11 20:10 GMT+01:00 Vincent Li <vincent.mc.li@gmail.com>:
>
> > On Thu, Feb 11, 2016 at 6:24 AM, Non Voglio 
> > <harold.demure87@gmail.com>
> > wrote:
> > > Hello,
> > >   I am new to the DPDK world and I am having some troubles in 
> > > using some code (not written by me).
> > >
> > > In particular, I bump into this error when invoking the
> > rte_eth_dev_configure
> > > function:
> > >
> > > PMD: librte_pmd_mlx4: 0x10a4d2a0: TX queues number update: 0 -> 16
> > >
> > > PMD: librte_pmd_mlx4: 0x10a4d2a0: RX queues number update: 0 -> 16
> > >
> > > PMD: librte_pmd_mlx4: 0x10a4d2a0: only a single RX queue can be
> > configured
> > > when hardware doesn't support RSS
> > >
> >
> > it appears you must enable RSS support to use multiple RX queue
> >
> >         if (!priv->hw_rss) {
> >                 ERROR("%p: only a single RX queue can be configured when"
> >                       " hardware doesn't support RSS",
> >                       (void *)dev);
> >                 return EINVAL;
> >         }
> >
> > Vincent
> >
>

Re: [dpdk-users] DPDK 2.2 MLX4: problem with number of TX/RX queues

[Harold Demure]

Dear Gilad,
   Let me thank you also here for your kind help. I have replied on the
forum.
Regards,
  Harold

2016-02-18 11:31 GMT+01:00 Gilad Berman <giladb@mellanox.com>:

> Harold,
>
> Flow Director features will be supported only on ConnectX-4 (starting from
> MLNX_DPDK version to be released end of Feb and on).
> So sadly you can't use it with mlx4 (ConnectX-3).
>
> I answered also the community question, btw.
>
>
> Gilad Berman | Staff System Engineer | Business Development | Mellanox
> Technologies Ltd.
> Work: +972 52 2554262| 6 Ha’Barzel St. Tel Aviv 6971010, Israel
>
>
> -----Original Message-----
> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Harold Demure
> Sent: Tuesday, February 16, 2016 6:05 PM
> To: Olga Shern <olgas@mellanox.com>
> Cc: users@dpdk.org
> Subject: Re: [dpdk-users] DPDK 2.2 MLX4: problem with number of TX/RX
> queues
>
> Dear Olga and Vincent,
>   thank you for your reply.
> It seems that the problem has now gone away; both with the -j and without
> the -j flag. I will try to backtrace the root cause of the problem and
> report it if I manage to isolate it.
>
> If I may, I would also like to ask your help for a related problem, which
> I have reported also on the mellanox community forum. If you are available,
> you may find my question there [1].
>
> I take the occasion to specifically address Olga Shern because I saw that
> she has already given support on a related topic [2].
> Also, related to [2] but possibly un-related from the questions in [1], I
> am failing to attach a "Perfect Match" filter to my mlx4 NIC, via
>
> rte_eth_dev_filter_ctrl(port_id, RTE_ETH_FILTER_FDIR, RTE_ETH_FILTER_ADD,
> &filter)
>
> Is this an expected behavior, namely, mlx4 does not support (yet) this
> kind of operational mode?
>
> Thank you (and anybody willing to help) for your time.
> Best regards,
>   Harold Demure
>
>
> [1] https://community.mellanox.com/thread/3152
> [2] http://dpdk.org/ml/archives/dev/2015-April/016392.html
>
> 2016-02-15 9:59 GMT+01:00 Olga Shern <olgas@mellanox.com>:
>
> > Hi Harold,
> >
> > DPDK 2.2 with MLNX_OFED that you are using and ConnectX-3 support RSS
> > automatically.
> > You don’t need to configure anything.
> >
> > Looks there is some bug in the compilation, did you compile  with make
> > -j, we saw some issue with it.
> > Can you please try without -j flag.
> >
> > Best Regards,
> > Olga
> >
> > -----Original Message-----
> > From: users [mailto:users-bounces@dpdk.org] On Behalf Of Harold Demure
> > Sent: Thursday, February 11, 2016 11:37 PM
> > To: Vincent Li
> > Cc: users@dpdk.org
> > Subject: Re: [dpdk-users] DPDK 2.2 MLX4: problem with number of TX/RX
> > queues
> >
> > Hello Vincent,
> >   Thank you for your reply. I followed the same reasoning as you did,
> > looking at the source code earlier today.
> >
> > Can you send me a link that explains how to change RX for my NIC?
> > I have been looking around for that, and also performed the steps in
> > the DPDK official guide for MLX4, but apparently nothing changed (at
> > least, the error was still there).
> >
> > In addition, I have to describe a new strange behavior: even *without
> > changing anything rss-wise*, but just enabling the increased debugging
> > verbosity (CONFIG_RTE_LIBRTE_MLX4_DEBUG=y), the error I had suddenly
> > disappeared.
> >
> > I cannot really explain what happened. Can anybody?
> >
> > At this point (with the debug prints), now my code blocks because
> > apparently mlx4 does not support FlowDir, so I have to change a bit my
> > software, which is built to use it.
> >
> > Still, I would love to know what's going on with my rx/tx queues.
> >
> > Thank you.
> > Regards,
> >    Harold
> >
> > 2016-02-11 20:10 GMT+01:00 Vincent Li <vincent.mc.li@gmail.com>:
> >
> > > On Thu, Feb 11, 2016 at 6:24 AM, Non Voglio
> > > <harold.demure87@gmail.com>
> > > wrote:
> > > > Hello,
> > > >   I am new to the DPDK world and I am having some troubles in
> > > > using some code (not written by me).
> > > >
> > > > In particular, I bump into this error when invoking the
> > > rte_eth_dev_configure
> > > > function:
> > > >
> > > > PMD: librte_pmd_mlx4: 0x10a4d2a0: TX queues number update: 0 -> 16
> > > >
> > > > PMD: librte_pmd_mlx4: 0x10a4d2a0: RX queues number update: 0 -> 16
> > > >
> > > > PMD: librte_pmd_mlx4: 0x10a4d2a0: only a single RX queue can be
> > > configured
> > > > when hardware doesn't support RSS
> > > >
> > >
> > > it appears you must enable RSS support to use multiple RX queue
> > >
> > >         if (!priv->hw_rss) {
> > >                 ERROR("%p: only a single RX queue can be configured
> when"
> > >                       " hardware doesn't support RSS",
> > >                       (void *)dev);
> > >                 return EINVAL;
> > >         }
> > >
> > > Vincent
> > >
> >
>