[dpdk-dev] : ixgbe: why bulk allocation is not used for a scattered Rx flow?

[dpdk-dev] : ixgbe: why bulk allocation is not used for a scattered Rx flow?

[Vlad Zolotarov]

Hi, I have a question about the "scattered Rx" feature: why enabling it 
disabled "bulk allocation" feature?
There is some unclear comment in the ixgbe_recv_scattered_pkts():

		/*
		 * Descriptor done.
		 *
		 * Allocate a new mbuf to replenish the RX ring descriptor.
		 * If the allocation fails:
		 *    - arrange for that RX descriptor to be the first one
		 *      being parsed the next time the receive function is
		 *      invoked [on the same queue].
		 *
		 *    - Stop parsing the RX ring and return immediately.
		 *
		 * This policy does not drop the packet received in the RX
		 * descriptor for which the allocation of a new mbuf failed.
		 * Thus, it allows that packet to be later retrieved if
		 * mbuf have been freed in the mean time.
		 * As a side effect, holding RX descriptors instead of
		 * systematically giving them back to the NIC may lead to
		 * RX ring exhaustion situations.
		 * However, the NIC can gracefully prevent such situations
		 * to happen by sending specific "back-pressure" flow control
		 * frames to its peer(s).
		 */

Why the same "policy" can't be done in the bulk-context allocation? - 
Don't advance the RDT until u've refilled the ring. What do I miss here?

Another question is about the LRO feature - is there a reason why it's 
not implemented? I've implemented the LRO support in ixgbe PMD to begin 
with - I used a "scattered Rx" as a template and now I'm tuning it 
(things like the stuff above).

Is there any philosophical reason why it hasn't been implemented in 
*any* PMD so far? ;)

thanks,
vlad

Re: [dpdk-dev] : ixgbe: why bulk allocation is not used for a scattered Rx flow?

[Bruce Richardson]

On Wed, Feb 25, 2015 at 11:40:36AM +0200, Vlad Zolotarov wrote:
> Hi, I have a question about the "scattered Rx" feature: why enabling it
> disabled "bulk allocation" feature?

The "bulk-allocation" feature is one where a more optimized RX code path is
used. For the sake of performance, when doing that code path, certain assumptions
were made, one of which was that packets would fit inside a single mbuf. Not
having this assumption makes the receiving of packets much more complicated and
therefore slower. [For similar reasons, the optimized TX routines e.g. vector
TX, are only used if it is guaranteed that no hardware offload features are
going to be used].

Now, it is possible, though challenging, to write optimized code for these more
complicated cases, such as scattered RX, or TX with offloads or scattered packets.
In general, we will always want separate routines for the simple case and the
complicated cases, as the performance hit of checking for the offloads, or
multi-mbuf packets will be significant enough to hit our performance badly when
they are not needed. In the case of the vector PMD for ixgbe - our highest
performance path right now - we have indeed two receive routines, for simple
and scattered cases. For TX, we only have an optimized path for the simple case,
but that is not to say that at some point someone may provide one for the
offload case too.

A final note on scattered packets in particular: if packets are too big to fit
in a single mbuf, then they are not small packets, and the processing time per
packet available is, by definition, larger than for packets that fit in a 
single mbuf. For 64-byte packets, the packet arrival rate is 67ns @ 10G, or
approx 200 cycles at 3GHz. If we assume a standard 2k mbuf, then a packet which
spans two mbufs takes at least 1654ns, and therefore a 3GHz CPU has nearly 5000
cycles to process that same packet. Therefore, since the processing budget is
so much bigger the need to optimize is much less. Therefore it's more important
to focus on the small packet case, which is what we have done.

> There is some unclear comment in the ixgbe_recv_scattered_pkts():
> 
> 		/*
> 		 * Descriptor done.
> 		 *
> 		 * Allocate a new mbuf to replenish the RX ring descriptor.
> 		 * If the allocation fails:
> 		 *    - arrange for that RX descriptor to be the first one
> 		 *      being parsed the next time the receive function is
> 		 *      invoked [on the same queue].
> 		 *
> 		 *    - Stop parsing the RX ring and return immediately.
> 		 *
> 		 * This policy does not drop the packet received in the RX
> 		 * descriptor for which the allocation of a new mbuf failed.
> 		 * Thus, it allows that packet to be later retrieved if
> 		 * mbuf have been freed in the mean time.
> 		 * As a side effect, holding RX descriptors instead of
> 		 * systematically giving them back to the NIC may lead to
> 		 * RX ring exhaustion situations.
> 		 * However, the NIC can gracefully prevent such situations
> 		 * to happen by sending specific "back-pressure" flow control
> 		 * frames to its peer(s).
> 		 */
> 
> Why the same "policy" can't be done in the bulk-context allocation? - Don't
> advance the RDT until u've refilled the ring. What do I miss here?

A lot of the optimizations done in other code paths, such as bulk alloc, may well
be applicable here, it's just that the work has not been done yet, as the focus
is elsewhere. For vector PMD RX, we have now routines that work on both regular
and scattered packets, and both perform much better than the scalar equivalents.
Also to note that in every RX (and TX) routine, the NIC tail pointer update is
always done just once at the end of the function. 

> 
> Another question is about the LRO feature - is there a reason why it's not
> implemented? I've implemented the LRO support in ixgbe PMD to begin with - I
> used a "scattered Rx" as a template and now I'm tuning it (things like the
> stuff above).
> 
> Is there any philosophical reason why it hasn't been implemented in *any*
> PMD so far? ;)

I'm not aware of any philosophical reasons why it hasn't been done. Patches
are welcome, as always. :-)

/Bruce

Re: [dpdk-dev] : ixgbe: why bulk allocation is not used for a scattered Rx flow?

[Vlad Zolotarov]

On 02/25/15 13:02, Bruce Richardson wrote:
> On Wed, Feb 25, 2015 at 11:40:36AM +0200, Vlad Zolotarov wrote:
>> Hi, I have a question about the "scattered Rx" feature: why enabling it
>> disabled "bulk allocation" feature?
> The "bulk-allocation" feature is one where a more optimized RX code path is
> used. For the sake of performance, when doing that code path, certain assumptions
> were made, one of which was that packets would fit inside a single mbuf. Not
> having this assumption makes the receiving of packets much more complicated and
> therefore slower. [For similar reasons, the optimized TX routines e.g. vector
> TX, are only used if it is guaranteed that no hardware offload features are
> going to be used].
>
> Now, it is possible, though challenging, to write optimized code for these more
> complicated cases, such as scattered RX, or TX with offloads or scattered packets.
> In general, we will always want separate routines for the simple case and the
> complicated cases, as the performance hit of checking for the offloads, or
> multi-mbuf packets will be significant enough to hit our performance badly when
> they are not needed. In the case of the vector PMD for ixgbe - our highest
> performance path right now - we have indeed two receive routines, for simple
> and scattered cases. For TX, we only have an optimized path for the simple case,
> but that is not to say that at some point someone may provide one for the
> offload case too.
>
> A final note on scattered packets in particular: if packets are too big to fit
> in a single mbuf, then they are not small packets, and the processing time per
> packet available is, by definition, larger than for packets that fit in a
> single mbuf. For 64-byte packets, the packet arrival rate is 67ns @ 10G, or
> approx 200 cycles at 3GHz. If we assume a standard 2k mbuf, then a packet which
> spans two mbufs takes at least 1654ns, and therefore a 3GHz CPU has nearly 5000
> cycles to process that same packet. Therefore, since the processing budget is
> so much bigger the need to optimize is much less. Therefore it's more important
> to focus on the small packet case, which is what we have done.

Sure. I'm doing my best not to harm the existing code paths: the RSC 
handler is a separate function (i first patched the scalar scattered 
function but now I'm rewriting it as a stand alone routine), I don't 
change the igb_rx_entry (leave it to be a pointer) and keep the 
additional info in separate descriptors in a separate ring that is not 
accessed in a non-RSC flow.

>
>> There is some unclear comment in the ixgbe_recv_scattered_pkts():
>>
>> 		/*
>> 		 * Descriptor done.
>> 		 *
>> 		 * Allocate a new mbuf to replenish the RX ring descriptor.
>> 		 * If the allocation fails:
>> 		 *    - arrange for that RX descriptor to be the first one
>> 		 *      being parsed the next time the receive function is
>> 		 *      invoked [on the same queue].
>> 		 *
>> 		 *    - Stop parsing the RX ring and return immediately.
>> 		 *
>> 		 * This policy does not drop the packet received in the RX
>> 		 * descriptor for which the allocation of a new mbuf failed.
>> 		 * Thus, it allows that packet to be later retrieved if
>> 		 * mbuf have been freed in the mean time.
>> 		 * As a side effect, holding RX descriptors instead of
>> 		 * systematically giving them back to the NIC may lead to
>> 		 * RX ring exhaustion situations.
>> 		 * However, the NIC can gracefully prevent such situations
>> 		 * to happen by sending specific "back-pressure" flow control
>> 		 * frames to its peer(s).
>> 		 */
>>
>> Why the same "policy" can't be done in the bulk-context allocation? - Don't
>> advance the RDT until u've refilled the ring. What do I miss here?
> A lot of the optimizations done in other code paths, such as bulk alloc, may well
> be applicable here, it's just that the work has not been done yet, as the focus
> is elsewhere. For vector PMD RX, we have now routines that work on both regular
> and scattered packets, and both perform much better than the scalar equivalents.
> Also to note that in every RX (and TX) routine, the NIC tail pointer update is
> always done just once at the end of the function.

I see. Thanks for an educated clarification. Although I've spent some 
time with DPDK I still feel sometimes that I don't I fully understand 
the original author's idea and the clarifications like your really help.
I looked at the vectored receive function (_recv_raw_pkts_vec()) and it 
is one cryptic piece of a code! ;) Since u've brought it up - could u 
direct me to the measurements comparing the vectored  and scalar DPDK 
data paths please? I wonder how working without CSUM offload for 
instance may be faster even for small packets like u mentioned above? 
One will have to calculate it in SW in that case and I'm puzzled how 
this may be faster than letting HW do it...

>
>> Another question is about the LRO feature - is there a reason why it's not
>> implemented? I've implemented the LRO support in ixgbe PMD to begin with - I
>> used a "scattered Rx" as a template and now I'm tuning it (things like the
>> stuff above).
>>
>> Is there any philosophical reason why it hasn't been implemented in *any*
>> PMD so far? ;)
> I'm not aware of any philosophical reasons why it hasn't been done. Patches
> are welcome, as always. :-)

Great! So, i'll send what I have once it's ready... ;)

Again, thank for a great clarification.

>
> /Bruce
>