[dpdk-dev] PMD/l3fwd issue

[dpdk-dev] PMD/l3fwd issue

[Harish Patil]

Hello,
Have a question regarding l3fwd application. The l3fwd application expects
the poll mode driver to return packets whose L2 header is 16-byte aligned.
Otherwise, it results in a crash. This is due to use of _mm_load_si128()
and _mm_store_si128() intrinsics which expects the address to be 16-byte
aligned. However, most of the real protocol stack expects packets such
that its IP header be aligned on a 16-byte boundary (not L2). Its not just
for IP but any L3 for that matter.  That’s way we usually see
skb_reserve(skb, NET_IP_ALIGN) calls in linux drivers.

So I’m looking for suggestions here, whether l3wd application or poll mode
driver should be changed to fix that? What is the right thing to do?
Can a check be added in l3fwd to use _mm_loadu_si128/_mm_storeu_si128
instructions instead of mm_load_si128/_mm_store_si128 if address is found
not be 16B aligned?

Thanks,
Harish



________________________________

This message and any attached documents contain information from the sending company or its parent company(s), subsidiaries, divisions or branch offices that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.

Re: [dpdk-dev] PMD/l3fwd issue

[Ananyev, Konstantin]

Hi Patil,

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Harish Patil
> Sent: Thursday, September 03, 2015 4:53 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] PMD/l3fwd issue
> 
> Hello,
> Have a question regarding l3fwd application. The l3fwd application expects
> the poll mode driver to return packets whose L2 header is 16-byte aligned.

Yep, and as I remember, by default PMD returns ti the upper layer mbufs with data offsets
aligned to cahce line size (64B).
Unless you'll change RTE_PKTMBUF_HEADROOM config parameter.

> Otherwise, it results in a crash. This is due to use of _mm_load_si128()
> and _mm_store_si128() intrinsics which expects the address to be 16-byte
> aligned. However, most of the real protocol stack expects packets such
> that its IP header be aligned on a 16-byte boundary (not L2). Its not just
> for IP but any L3 for that matter.  That’s way we usually see
> skb_reserve(skb, NET_IP_ALIGN) calls in linux drivers.

Well, l3fwd is just an example application to demonstrate usage of DPDK API
And max performance it could get for that type of workload.
No-one forces you to use aligned load/store in your own application.

> 
> So I’m looking for suggestions here, whether l3wd application or poll mode
> driver should be changed to fix that? What is the right thing to do?
> Can a check be added in l3fwd to use _mm_loadu_si128/_mm_storeu_si128
> instructions instead of mm_load_si128/_mm_store_si128 if address is found
> not be 16B aligned?

I'd personally just change l3fwd to use to use _mm_loadu_si128/_mm_storeu_si128 unconditionally.
As by default  address is 16B aligned anyway, I think that using MOVDQU instead of MOVDQA here
shouldn't make that big difference. 
But off course testing need to be done to make sure there is no performance drop with that change.
Konstantin

> 
> Thanks,
> Harish
> 
> 
> 
> ________________________________
> 
> This message and any attached documents contain information from the sending company or its parent company(s), subsidiaries,
> divisions or branch offices that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use
> this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete
> this message.

Re: [dpdk-dev] PMD/l3fwd issue

[Harish Patil]

Hi Konstantin,

>Hi Patil,
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Harish Patil
>> Sent: Thursday, September 03, 2015 4:53 PM
>> To: dev@dpdk.org
>> Subject: [dpdk-dev] PMD/l3fwd issue
>>
>> Hello,
>> Have a question regarding l3fwd application. The l3fwd application
>>expects
>> the poll mode driver to return packets whose L2 header is 16-byte
>>aligned.
>
>Yep, and as I remember, by default PMD returns ti the upper layer mbufs
>with data offsets
>aligned to cahce line size (64B).
>Unless you'll change RTE_PKTMBUF_HEADROOM config parameter.
>
>> Otherwise, it results in a crash. This is due to use of _mm_load_si128()
>> and _mm_store_si128() intrinsics which expects the address to be 16-byte
>> aligned. However, most of the real protocol stack expects packets such
>> that its IP header be aligned on a 16-byte boundary (not L2). Its not
>>just
>> for IP but any L3 for that matter.  That’s way we usually see
>> skb_reserve(skb, NET_IP_ALIGN) calls in linux drivers.
>
>Well, l3fwd is just an example application to demonstrate usage of DPDK
>API
>And max performance it could get for that type of workload.
>No-one forces you to use aligned load/store in your own application.

Yes, I agree if its our private application. But l3fwd being widely used
as a benchmarking/testing tool and they may ran into this issue.

>
>>
>> So I’m looking for suggestions here, whether l3wd application or poll
>>mode
>> driver should be changed to fix that? What is the right thing to do?
>> Can a check be added in l3fwd to use _mm_loadu_si128/_mm_storeu_si128
>> instructions instead of mm_load_si128/_mm_store_si128 if address is
>>found
>> not be 16B aligned?
>
>I'd personally just change l3fwd to use to use
>_mm_loadu_si128/_mm_storeu_si128 unconditionally.
>As by default  address is 16B aligned anyway, I think that using MOVDQU
>instead of MOVDQA here
>shouldn't make that big difference.
>But off course testing need to be done to make sure there is no
>performance drop with that change.

I too would just change l3fwd application so that all poll mode drivers
would just work. Are you proposing that we upstream l3fwd change if we
don’t see performance drop?

>Konstantin
>
>>
>> Thanks,
>> Harish
>>
>>
>>
>> ________________________________
>>
>> This message and any attached documents contain information from the
>>sending company or its parent company(s), subsidiaries,
>> divisions or branch offices that may be confidential. If you are not
>>the intended recipient, you may not read, copy, distribute, or use
>> this information. If you have received this transmission in error,
>>please notify the sender immediately by reply e-mail and then delete
>> this message.
>



________________________________

This message and any attached documents contain information from the sending company or its parent company(s), subsidiaries, divisions or branch offices that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.

Re: [dpdk-dev] PMD/l3fwd issue

[Ananyev, Konstantin]

> -----Original Message-----
> From: Harish Patil [mailto:harish.patil@qlogic.com]
> Sent: Friday, September 04, 2015 2:08 PM
> To: Ananyev, Konstantin; dev@dpdk.org
> Cc: Ameen Rahman
> Subject: Re: PMD/l3fwd issue
> 
> Hi Konstantin,
> 
> >Hi Patil,
> >
> >> -----Original Message-----
> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Harish Patil
> >> Sent: Thursday, September 03, 2015 4:53 PM
> >> To: dev@dpdk.org
> >> Subject: [dpdk-dev] PMD/l3fwd issue
> >>
> >> Hello,
> >> Have a question regarding l3fwd application. The l3fwd application
> >>expects
> >> the poll mode driver to return packets whose L2 header is 16-byte
> >>aligned.
> >
> >Yep, and as I remember, by default PMD returns ti the upper layer mbufs
> >with data offsets
> >aligned to cahce line size (64B).
> >Unless you'll change RTE_PKTMBUF_HEADROOM config parameter.
> >
> >> Otherwise, it results in a crash. This is due to use of _mm_load_si128()
> >> and _mm_store_si128() intrinsics which expects the address to be 16-byte
> >> aligned. However, most of the real protocol stack expects packets such
> >> that its IP header be aligned on a 16-byte boundary (not L2). Its not
> >>just
> >> for IP but any L3 for that matter.  That’s way we usually see
> >> skb_reserve(skb, NET_IP_ALIGN) calls in linux drivers.
> >
> >Well, l3fwd is just an example application to demonstrate usage of DPDK
> >API
> >And max performance it could get for that type of workload.
> >No-one forces you to use aligned load/store in your own application.
> 
> Yes, I agree if its our private application. But l3fwd being widely used
> as a benchmarking/testing tool and they may ran into this issue.
> 

If someone would try to run it with RTE_PKTMBUF_HEADROOM non-aligned on 16B, then probably yes.

> >
> >>
> >> So I’m looking for suggestions here, whether l3wd application or poll
> >>mode
> >> driver should be changed to fix that? What is the right thing to do?
> >> Can a check be added in l3fwd to use _mm_loadu_si128/_mm_storeu_si128
> >> instructions instead of mm_load_si128/_mm_store_si128 if address is
> >>found
> >> not be 16B aligned?
> >
> >I'd personally just change l3fwd to use to use
> >_mm_loadu_si128/_mm_storeu_si128 unconditionally.
> >As by default  address is 16B aligned anyway, I think that using MOVDQU
> >instead of MOVDQA here
> >shouldn't make that big difference.
> >But off course testing need to be done to make sure there is no
> >performance drop with that change.
> 
> I too would just change l3fwd application so that all poll mode drivers
> would just work. Are you proposing that we upstream l3fwd change if we
> don’t see performance drop?

Yep, I'd suggest to verify there is no performance difference and submit a patch.
From our side we can do some extra performance testing too.
Thanks
Konstantin

> 
> >Konstantin
> >
> >>
> >> Thanks,
> >> Harish
> >>
> >>
> >>
> >> ________________________________
> >>
> >> This message and any attached documents contain information from the
> >>sending company or its parent company(s), subsidiaries,
> >> divisions or branch offices that may be confidential. If you are not
> >>the intended recipient, you may not read, copy, distribute, or use
> >> this information. If you have received this transmission in error,
> >>please notify the sender immediately by reply e-mail and then delete
> >> this message.
> >
> 
> 
> 
> ________________________________
> 
> This message and any attached documents contain information from the sending company or its parent company(s), subsidiaries,
> divisions or branch offices that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use
> this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete
> this message.

Re: [dpdk-dev] PMD/l3fwd issue

[Harish Patil]

>
>
>> -----Original Message-----
>> From: Harish Patil [mailto:harish.patil@qlogic.com]
>> Sent: Friday, September 04, 2015 2:08 PM
>> To: Ananyev, Konstantin; dev@dpdk.org
>> Cc: Ameen Rahman
>> Subject: Re: PMD/l3fwd issue
>>
>> Hi Konstantin,
>>
>> >Hi Patil,
>> >
>> >> -----Original Message-----
>> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Harish Patil
>> >> Sent: Thursday, September 03, 2015 4:53 PM
>> >> To: dev@dpdk.org
>> >> Subject: [dpdk-dev] PMD/l3fwd issue
>> >>
>> >> Hello,
>> >> Have a question regarding l3fwd application. The l3fwd application
>> >>expects
>> >> the poll mode driver to return packets whose L2 header is 16-byte
>> >>aligned.
>> >
>> >Yep, and as I remember, by default PMD returns ti the upper layer mbufs
>> >with data offsets
>> >aligned to cahce line size (64B).
>> >Unless you'll change RTE_PKTMBUF_HEADROOM config parameter.
>> >
>> >> Otherwise, it results in a crash. This is due to use of
>>_mm_load_si128()
>> >> and _mm_store_si128() intrinsics which expects the address to be
>>16-byte
>> >> aligned. However, most of the real protocol stack expects packets
>>such
>> >> that its IP header be aligned on a 16-byte boundary (not L2). Its not
>> >>just
>> >> for IP but any L3 for that matter.  That’s way we usually see
>> >> skb_reserve(skb, NET_IP_ALIGN) calls in linux drivers.
>> >
>> >Well, l3fwd is just an example application to demonstrate usage of DPDK
>> >API
>> >And max performance it could get for that type of workload.
>> >No-one forces you to use aligned load/store in your own application.
>>
>> Yes, I agree if its our private application. But l3fwd being widely used
>> as a benchmarking/testing tool and they may ran into this issue.
>>
>
>If someone would try to run it with RTE_PKTMBUF_HEADROOM non-aligned on
>16B, then probably yes.
>
>> >
>> >>
>> >> So I’m looking for suggestions here, whether l3wd application or poll
>> >>mode
>> >> driver should be changed to fix that? What is the right thing to do?
>> >> Can a check be added in l3fwd to use _mm_loadu_si128/_mm_storeu_si128
>> >> instructions instead of mm_load_si128/_mm_store_si128 if address is
>> >>found
>> >> not be 16B aligned?
>> >
>> >I'd personally just change l3fwd to use to use
>> >_mm_loadu_si128/_mm_storeu_si128 unconditionally.
>> >As by default  address is 16B aligned anyway, I think that using MOVDQU
>> >instead of MOVDQA here
>> >shouldn't make that big difference.
>> >But off course testing need to be done to make sure there is no
>> >performance drop with that change.
>>
>> I too would just change l3fwd application so that all poll mode drivers
>> would just work. Are you proposing that we upstream l3fwd change if we
>> don’t see performance drop?
>
>Yep, I'd suggest to verify there is no performance difference and submit
>a patch.
>From our side we can do some extra performance testing too.
>Thanks
>Konstantin

Hi Konstantin,
As you suggested, I have submitted the patch.

Thanks,
Harish


>
>>
>> >Konstantin
>> >
>> >>
>> >> Thanks,
>> >> Harish
>> >>
>> >>
>> >>
>> >> ________________________________
>> >>
>> >> This message and any attached documents contain information from the
>> >>sending company or its parent company(s), subsidiaries,
>> >> divisions or branch offices that may be confidential. If you are not
>> >>the intended recipient, you may not read, copy, distribute, or use
>> >> this information. If you have received this transmission in error,
>> >>please notify the sender immediately by reply e-mail and then delete
>> >> this message.
>> >
>>
>>
>>
>> ________________________________
>>
>> This message and any attached documents contain information from the
>>sending company or its parent company(s), subsidiaries,
>> divisions or branch offices that may be confidential. If you are not
>>the intended recipient, you may not read, copy, distribute, or use
>> this information. If you have received this transmission in error,
>>please notify the sender immediately by reply e-mail and then delete
>> this message.
>



________________________________

This message and any attached documents contain information from the sending company or its parent company(s), subsidiaries, divisions or branch offices that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.