[dpdk-dev] ixgbe TX function selection

[dpdk-dev] ixgbe TX function selection

[Zoltan Kiss]

Hi,

I've noticed that ixgbe_set_tx_function() selects the non-SG function 
even if (dev->data->scattered_rx == 1). That seems a bit dangerous, as 
you can turn that on inadvertently when you don't set max_rx_pkt_len and 
buffer size in certain ways. I've learnt it in the hard way, as my 
segmented packets were leaking memory on the TX path, which doesn't 
cries if you send out segmented packets.
How should this case be treated? Assert on the non-SG TX side for the 
'next' pointer? Or turning on SG if RX has it? It doesn't seem to be a 
solid way as other interfaces still can have SG turned on.

Regards,

Zoltan

Re: [dpdk-dev] ixgbe TX function selection

[Lu, Wenzhuo]

Hi Zoltan,

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
> Sent: Wednesday, March 2, 2016 3:19 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] ixgbe TX function selection
> 
> Hi,
> 
> I've noticed that ixgbe_set_tx_function() selects the non-SG function even if
Thanks for let us know the problem. But I don't catch your point. Do you really mean TX here? After a quick look at the code, I don’t find the SG/non-SG functions for TX. Do I miss something?

> (dev->data->scattered_rx == 1). That seems a bit dangerous, as you can turn
> that on inadvertently when you don't set max_rx_pkt_len and buffer size in
> certain ways. I've learnt it in the hard way, as my segmented packets were
> leaking memory on the TX path, which doesn't cries if you send out segmented
> packets.
Which one will cause problem? SG or non-SG packets? And where does the memory leak happen?

> How should this case be treated? Assert on the non-SG TX side for the 'next'
> pointer? Or turning on SG if RX has it? It doesn't seem to be a solid way as other
> interfaces still can have SG turned on.
> 
> Regards,
> 
> Zoltan

Re: [dpdk-dev] ixgbe TX function selection

[Zoltan Kiss]

On 04/03/16 01:47, Lu, Wenzhuo wrote:
> Hi Zoltan,
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
>> Sent: Wednesday, March 2, 2016 3:19 AM
>> To: dev@dpdk.org
>> Subject: [dpdk-dev] ixgbe TX function selection
>>
>> Hi,
>>
>> I've noticed that ixgbe_set_tx_function() selects the non-SG function even if
> Thanks for let us know the problem. But I don't catch your point. Do you really mean TX here? After a quick look at the code, I don’t find the SG/non-SG functions for TX. Do I miss something?

The simple code path doesn't handle multisegmented packets. 
ixgbe_txq_vec_setup() and ixgbe_xmit_pkts_simple() doesn't even check 
the next pointer of the mbuf, just put the first one on the descriptor 
ring, and when TX completion happens, the memory is leaked because it 
just sets ->next to NULL, and calls rte_mempool_put[_bulk]
ixgbe_xmit_pkts() puts all the segments on the descriptor ring, 
therefore when the descriptors are released they are released as well.
This is what these functions supposed to do, but my point is it's very 
easy to send a multisegmented packet to the simple code path.


>
>> (dev->data->scattered_rx == 1). That seems a bit dangerous, as you can turn
>> that on inadvertently when you don't set max_rx_pkt_len and buffer size in
>> certain ways. I've learnt it in the hard way, as my segmented packets were
>> leaking memory on the TX path, which doesn't cries if you send out segmented
>> packets.
> Which one will cause problem? SG or non-SG packets? And where does the memory leak happen?
>
>> How should this case be treated? Assert on the non-SG TX side for the 'next'
>> pointer? Or turning on SG if RX has it? It doesn't seem to be a solid way as other
>> interfaces still can have SG turned on.
>>
>> Regards,
>>
>> Zoltan

Re: [dpdk-dev] ixgbe TX function selection

[Wu, Jingjing]

Hi, Zoltan

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
> Sent: Wednesday, March 2, 2016 3:19 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] ixgbe TX function selection
> 
> Hi,
> 
> I've noticed that ixgbe_set_tx_function() selects the non-SG function
> even if (dev->data->scattered_rx == 1). That seems a bit dangerous, as
> you can turn that on inadvertently when you don't set max_rx_pkt_len and
> buffer size in certain ways. I've learnt it in the hard way, as my
> segmented packets were leaking memory on the TX path, which doesn't
> cries if you send out segmented packets.
> How should this case be treated? Assert on the non-SG TX side for the
> 'next' pointer? Or turning on SG if RX has it? It doesn't seem to be a
> solid way as other interfaces still can have SG turned on.
> 

If you look into the ixgbe_set_tx_function, you will find tx function
selection is decided by the tx_flags on queue configure, which is
passed by rte_eth_txconf. So even you set dev->data->scattered_rx to 1,
if the tx_flags is ETH_TXQ_FLAGS_NOMULTSEGS, ixgbe_xmit_pkts_simple is
still selected as tx function. So, you'd better to set tx_flags=0, and have a try.

> Regards,
> 
> Zoltan

Re: [dpdk-dev] ixgbe TX function selection

[Zoltan Kiss]

On 10/03/16 07:51, Wu, Jingjing wrote:
> Hi, Zoltan
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
>> Sent: Wednesday, March 2, 2016 3:19 AM
>> To: dev@dpdk.org
>> Subject: [dpdk-dev] ixgbe TX function selection
>>
>> Hi,
>>
>> I've noticed that ixgbe_set_tx_function() selects the non-SG function
>> even if (dev->data->scattered_rx == 1). That seems a bit dangerous, as
>> you can turn that on inadvertently when you don't set max_rx_pkt_len and
>> buffer size in certain ways. I've learnt it in the hard way, as my
>> segmented packets were leaking memory on the TX path, which doesn't
>> cries if you send out segmented packets.
>> How should this case be treated? Assert on the non-SG TX side for the
>> 'next' pointer? Or turning on SG if RX has it? It doesn't seem to be a
>> solid way as other interfaces still can have SG turned on.
>>
>
> If you look into the ixgbe_set_tx_function, you will find tx function
> selection is decided by the tx_flags on queue configure, which is
> passed by rte_eth_txconf. So even you set dev->data->scattered_rx to 1,
> if the tx_flags is ETH_TXQ_FLAGS_NOMULTSEGS, ixgbe_xmit_pkts_simple is
> still selected as tx function. So, you'd better to set tx_flags=0, and have a try.

You mean getting default_txconf from rte_eth_dev_info_get() and 
explicitly turn ETH_TXQ_FLAGS_NOMULTSEGS to 0? (filling tx_flags with 
zeros doesn't work very well) That's a way to solve it for me, but I'm 
rather talking about using defaults which doesn't cause memory leak 
quite easily.

>
>> Regards,
>>
>> Zoltan

Re: [dpdk-dev] ixgbe TX function selection

[Lu, Wenzhuo]

Hi Zoltan,


> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
> Sent: Friday, March 18, 2016 1:11 AM
> To: Wu, Jingjing; dev@dpdk.org
> Subject: Re: [dpdk-dev] ixgbe TX function selection
> 
> 
> 
> On 10/03/16 07:51, Wu, Jingjing wrote:
> > Hi, Zoltan
> >
> >> -----Original Message-----
> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
> >> Sent: Wednesday, March 2, 2016 3:19 AM
> >> To: dev@dpdk.org
> >> Subject: [dpdk-dev] ixgbe TX function selection
> >>
> >> Hi,
> >>
> >> I've noticed that ixgbe_set_tx_function() selects the non-SG function
> >> even if (dev->data->scattered_rx == 1). That seems a bit dangerous,
> >> as you can turn that on inadvertently when you don't set
> >> max_rx_pkt_len and buffer size in certain ways. I've learnt it in the
> >> hard way, as my segmented packets were leaking memory on the TX path,
> >> which doesn't cries if you send out segmented packets.
> >> How should this case be treated? Assert on the non-SG TX side for the
> >> 'next' pointer? Or turning on SG if RX has it? It doesn't seem to be
> >> a solid way as other interfaces still can have SG turned on.
> >>
> >
> > If you look into the ixgbe_set_tx_function, you will find tx function
> > selection is decided by the tx_flags on queue configure, which is
> > passed by rte_eth_txconf. So even you set dev->data->scattered_rx to
> > 1, if the tx_flags is ETH_TXQ_FLAGS_NOMULTSEGS, ixgbe_xmit_pkts_simple
> > is still selected as tx function. So, you'd better to set tx_flags=0, and have a try.
> 
> You mean getting default_txconf from rte_eth_dev_info_get() and explicitly turn
> ETH_TXQ_FLAGS_NOMULTSEGS to 0? (filling tx_flags with zeros doesn't work
> very well) That's a way to solve it for me, but I'm rather talking about using
> defaults which doesn't cause memory leak quite easily.
Yes, ETH_TXQ_FLAGS_NOMULTSEGS only can be set to 1 when you know all your packets will not be segmented.
I think that means normally we should use full function path for TX, for we have no knowledge about if the packets will be segmented or not.
You don't need to set tx_flags to 0, only the ETH_TXQ_FLAGS_NOMULTSEGS bit should be 0, the other bits can be 1 if needed.

> 
> >
> >> Regards,
> >>
> >> Zoltan

Re: [dpdk-dev] ixgbe TX function selection

[Zoltan Kiss]

On 18/03/16 00:45, Lu, Wenzhuo wrote:
> Hi Zoltan,
>
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
>> Sent: Friday, March 18, 2016 1:11 AM
>> To: Wu, Jingjing; dev@dpdk.org
>> Subject: Re: [dpdk-dev] ixgbe TX function selection
>>
>>
>>
>> On 10/03/16 07:51, Wu, Jingjing wrote:
>>> Hi, Zoltan
>>>
>>>> -----Original Message-----
>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
>>>> Sent: Wednesday, March 2, 2016 3:19 AM
>>>> To: dev@dpdk.org
>>>> Subject: [dpdk-dev] ixgbe TX function selection
>>>>
>>>> Hi,
>>>>
>>>> I've noticed that ixgbe_set_tx_function() selects the non-SG function
>>>> even if (dev->data->scattered_rx == 1). That seems a bit dangerous,
>>>> as you can turn that on inadvertently when you don't set
>>>> max_rx_pkt_len and buffer size in certain ways. I've learnt it in the
>>>> hard way, as my segmented packets were leaking memory on the TX path,
>>>> which doesn't cries if you send out segmented packets.
>>>> How should this case be treated? Assert on the non-SG TX side for the
>>>> 'next' pointer? Or turning on SG if RX has it? It doesn't seem to be
>>>> a solid way as other interfaces still can have SG turned on.
>>>>
>>>
>>> If you look into the ixgbe_set_tx_function, you will find tx function
>>> selection is decided by the tx_flags on queue configure, which is
>>> passed by rte_eth_txconf. So even you set dev->data->scattered_rx to
>>> 1, if the tx_flags is ETH_TXQ_FLAGS_NOMULTSEGS, ixgbe_xmit_pkts_simple
>>> is still selected as tx function. So, you'd better to set tx_flags=0, and have a try.
>>
>> You mean getting default_txconf from rte_eth_dev_info_get() and explicitly turn
>> ETH_TXQ_FLAGS_NOMULTSEGS to 0? (filling tx_flags with zeros doesn't work
>> very well) That's a way to solve it for me, but I'm rather talking about using
>> defaults which doesn't cause memory leak quite easily.
> Yes, ETH_TXQ_FLAGS_NOMULTSEGS only can be set to 1 when you know all your packets will not be segmented.
> I think that means normally we should use full function path for TX, for we have no knowledge about if the packets will be segmented or not.
> You don't need to set tx_flags to 0, only the ETH_TXQ_FLAGS_NOMULTSEGS bit should be 0, the other bits can be 1 if needed.

So can we agree that the default settings should set 
ETH_TXQ_FLAGS_NOMULTSEGS to 0?


>
>>
>>>
>>>> Regards,
>>>>
>>>> Zoltan

Re: [dpdk-dev] ixgbe TX function selection

[Ananyev, Konstantin]

Hi Zoltan,

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
> Sent: Friday, March 18, 2016 1:33 PM
> To: Lu, Wenzhuo; Wu, Jingjing; dev@dpdk.org
> Subject: Re: [dpdk-dev] ixgbe TX function selection
> 
> 
> 
> On 18/03/16 00:45, Lu, Wenzhuo wrote:
> > Hi Zoltan,
> >
> >
> >> -----Original Message-----
> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
> >> Sent: Friday, March 18, 2016 1:11 AM
> >> To: Wu, Jingjing; dev@dpdk.org
> >> Subject: Re: [dpdk-dev] ixgbe TX function selection
> >>
> >>
> >>
> >> On 10/03/16 07:51, Wu, Jingjing wrote:
> >>> Hi, Zoltan
> >>>
> >>>> -----Original Message-----
> >>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
> >>>> Sent: Wednesday, March 2, 2016 3:19 AM
> >>>> To: dev@dpdk.org
> >>>> Subject: [dpdk-dev] ixgbe TX function selection
> >>>>
> >>>> Hi,
> >>>>
> >>>> I've noticed that ixgbe_set_tx_function() selects the non-SG function
> >>>> even if (dev->data->scattered_rx == 1). That seems a bit dangerous,
> >>>> as you can turn that on inadvertently when you don't set
> >>>> max_rx_pkt_len and buffer size in certain ways. I've learnt it in the
> >>>> hard way, as my segmented packets were leaking memory on the TX path,
> >>>> which doesn't cries if you send out segmented packets.
> >>>> How should this case be treated? Assert on the non-SG TX side for the
> >>>> 'next' pointer? Or turning on SG if RX has it? It doesn't seem to be
> >>>> a solid way as other interfaces still can have SG turned on.
> >>>>
> >>>
> >>> If you look into the ixgbe_set_tx_function, you will find tx function
> >>> selection is decided by the tx_flags on queue configure, which is
> >>> passed by rte_eth_txconf. So even you set dev->data->scattered_rx to
> >>> 1, if the tx_flags is ETH_TXQ_FLAGS_NOMULTSEGS, ixgbe_xmit_pkts_simple
> >>> is still selected as tx function. So, you'd better to set tx_flags=0, and have a try.
> >>
> >> You mean getting default_txconf from rte_eth_dev_info_get() and explicitly turn
> >> ETH_TXQ_FLAGS_NOMULTSEGS to 0? (filling tx_flags with zeros doesn't work
> >> very well) That's a way to solve it for me, but I'm rather talking about using
> >> defaults which doesn't cause memory leak quite easily.
> > Yes, ETH_TXQ_FLAGS_NOMULTSEGS only can be set to 1 when you know all your packets will not be segmented.
> > I think that means normally we should use full function path for TX, for we have no knowledge about if the packets will be
> segmented or not.
> > You don't need to set tx_flags to 0, only the ETH_TXQ_FLAGS_NOMULTSEGS bit should be 0, the other bits can be 1 if needed.
> 
> So can we agree that the default settings should set
> ETH_TXQ_FLAGS_NOMULTSEGS to 0?

I'd prefer to keep things as they are right now.
There always will be arguments and supporters for both alternatives: 
Should the fastest or the most comprehensive path be the default one.
Again default txq_flags can vary from one PMD to another. 
So, I think  the right behaviour for the app would be not to rely on default value
but set it up manually to the desired value. 
Konstantin

> 
> 
> >
> >>
> >>>
> >>>> Regards,
> >>>>
> >>>> Zoltan