[dpdk-users] UDP packet transmission issue - DPDK 18.08

[dpdk-users] UDP packet transmission issue - DPDK 18.08

[Anupama Laxmi]

We have recently upgraded to DPDK 18.08 version. After upgrading to the
latest version observing issue with UDP packets transmission errors for few
packets ( UDP size 736 bytes) .With older version of DPDK  we never faced
this issue.

 Attaching the filtered tcpdump which shows "Bad UDP length > IP payload
length" for one such packet.

No issue observed while transferring UDP packets with size 28 bytes and 48
bytes. I tried to print the packet length calculation in my program just
before sending it out to the Kernel using rte_kni_tx_burst. The packet
length calculation seems correct to me.

1.)
size_udp:48
sizeof(struct udp_hdr):8
size_ApplMsg:40
udphdr->dgram_len:12288
m->data_len:82
size_ip:68
l2_data_shift:14

2.)
size_udp:28
sizeof(struct udp_hdr):8
size_ApplMsg:20
udphdr->dgram_len:7168
 m->data_len:62
 ip->total_length:12288
 size_ip:48
l2_data_shift:14

*** Packets with UDP size 736 are not getting transmitted to the
receiving end and getting dropped.

3.)
 size_udp:736
 sizeof(struct udp_hdr):8
size_ApplMsg:728
udphdr->dgram_len:57346
 m->data_len:770
size_ip:756
l2_data_shift:14

Also MTU is set to 1500 in my program. So it shouldn't be an issue to
transfer 736 bytes UDP data which is less than 1500 bytes MTU.

Looks like the kernel is dropping this packet.

So I tried to increase the kernel buffer size but that didn't help.

netstat -su -> output shows 0 send/receive buffer errors.


What has changed in DPDK 18.08 with respect to UDP packets? Please suggest
if I need to consider tuning udp ( using new API ) , offloading udp
traffic(new offload flags)  to resolve this issue.

Thanks,
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bad_udp_length.png
Type: image/png
Size: 56015 bytes
Desc: not available
URL: <http://mails.dpdk.org/archives/users/attachments/20190420/f5dd6779/attachment.png>

Re: [dpdk-users] UDP packet transmission issue - DPDK 18.08

[hirok borah]

Hi All,

Do we have any resolution for this issue yet ?
This looks like a defect in DPDK 18.08.

Plz share if anyone have any inputs on this issue.

Regards
Hirok

On Sat, Apr 20, 2019 at 9:41 AM Anupama Laxmi <anupamalaxmi4@gmail.com>
wrote:

> We have recently upgraded to DPDK 18.08 version. After upgrading to the
> latest version observing issue with UDP packets transmission errors for few
> packets ( UDP size 736 bytes) .With older version of DPDK  we never faced
> this issue.
>
>  Attaching the filtered tcpdump which shows "Bad UDP length > IP payload
> length" for one such packet.
>
> No issue observed while transferring UDP packets with size 28 bytes and 48
> bytes. I tried to print the packet length calculation in my program just
> before sending it out to the Kernel using rte_kni_tx_burst. The packet
> length calculation seems correct to me.
>
> 1.)
> size_udp:48
> sizeof(struct udp_hdr):8
> size_ApplMsg:40
> udphdr->dgram_len:12288
> m->data_len:82
> size_ip:68
> l2_data_shift:14
>
> 2.)
> size_udp:28
> sizeof(struct udp_hdr):8
> size_ApplMsg:20
> udphdr->dgram_len:7168
>  m->data_len:62
>  ip->total_length:12288
>  size_ip:48
> l2_data_shift:14
>
> *** Packets with UDP size 736 are not getting transmitted to the
> receiving end and getting dropped.
>
> 3.)
>  size_udp:736
>  sizeof(struct udp_hdr):8
> size_ApplMsg:728
> udphdr->dgram_len:57346
>  m->data_len:770
> size_ip:756
> l2_data_shift:14
>
> Also MTU is set to 1500 in my program. So it shouldn't be an issue to
> transfer 736 bytes UDP data which is less than 1500 bytes MTU.
>
> Looks like the kernel is dropping this packet.
>
> So I tried to increase the kernel buffer size but that didn't help.
>
> netstat -su -> output shows 0 send/receive buffer errors.
>
>
> What has changed in DPDK 18.08 with respect to UDP packets? Please suggest
> if I need to consider tuning udp ( using new API ) , offloading udp
> traffic(new offload flags)  to resolve this issue.
>
> Thanks,
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: bad_udp_length.png
> Type: image/png
> Size: 56015 bytes
> Desc: not available
> URL: <
> http://mails.dpdk.org/archives/users/attachments/20190420/f5dd6779/attachment.png
> >
>


-- 
Regards,
-Hirok

Re: [dpdk-users] UDP packet transmission issue - DPDK 18.08

[Wiles, Keith]

> On May 8, 2019, at 1:41 AM, hirok borah <hirok.gen@gmail.com> wrote:
> 
> Hi All,
> 
> Do we have any resolution for this issue yet ?
> This looks like a defect in DPDK 18.08.

DPDK does not understand UDP/TCP or any L4 Packet per say, it only sends the packets constructed in the application. It will send the packets as defined and the only place DPDK tries to determine the packet type is when it receives the packet which in this case is the kernel.

I do not use testpmd much, but use pktgen-dpdk for my work. Please try with testpmd or pktgen or TRex or moongen and see if the problem still exists.
> 
> Plz share if anyone have any inputs on this issue.
> 
> Regards
> Hirok
> 
> On Sat, Apr 20, 2019 at 9:41 AM Anupama Laxmi <anupamalaxmi4@gmail.com>
> wrote:
> 
>> We have recently upgraded to DPDK 18.08 version. After upgrading to the
>> latest version observing issue with UDP packets transmission errors for few
>> packets ( UDP size 736 bytes) .With older version of DPDK  we never faced
>> this issue.
>> 
>> Attaching the filtered tcpdump which shows "Bad UDP length > IP payload
>> length" for one such packet.
>> 
>> No issue observed while transferring UDP packets with size 28 bytes and 48
>> bytes. I tried to print the packet length calculation in my program just
>> before sending it out to the Kernel using rte_kni_tx_burst. The packet
>> length calculation seems correct to me.
>> 
>> 1.)
>> size_udp:48
>> sizeof(struct udp_hdr):8
>> size_ApplMsg:40
>> udphdr->dgram_len:12288
>> m->data_len:82
>> size_ip:68
>> l2_data_shift:14
>> 
>> 2.)
>> size_udp:28
>> sizeof(struct udp_hdr):8
>> size_ApplMsg:20
>> udphdr->dgram_len:7168
>> m->data_len:62
>> ip->total_length:12288
>> size_ip:48
>> l2_data_shift:14
>> 
>> *** Packets with UDP size 736 are not getting transmitted to the
>> receiving end and getting dropped.
>> 
>> 3.)
>> size_udp:736
>> sizeof(struct udp_hdr):8
>> size_ApplMsg:728
>> udphdr->dgram_len:57346
>> m->data_len:770
>> size_ip:756
>> l2_data_shift:14
>> 
>> Also MTU is set to 1500 in my program. So it shouldn't be an issue to
>> transfer 736 bytes UDP data which is less than 1500 bytes MTU.
>> 
>> Looks like the kernel is dropping this packet.
>> 
>> So I tried to increase the kernel buffer size but that didn't help.
>> 
>> netstat -su -> output shows 0 send/receive buffer errors.
>> 
>> 
>> What has changed in DPDK 18.08 with respect to UDP packets? Please suggest
>> if I need to consider tuning udp ( using new API ) , offloading udp
>> traffic(new offload flags)  to resolve this issue.
>> 
>> Thanks,
>> -------------- next part --------------
>> A non-text attachment was scrubbed...
>> Name: bad_udp_length.png
>> Type: image/png
>> Size: 56015 bytes
>> Desc: not available
>> URL: <
>> http://mails.dpdk.org/archives/users/attachments/20190420/f5dd6779/attachment.png
>>> 
>> 
> 
> 
> -- 
> Regards,
> -Hirok

Regards,
Keith

Re: [dpdk-users] UDP packet transmission issue - DPDK 18.08

[hirok borah]

Thanks Keith for sharing the info.

But as also stated by Anupama, the problem here is with KNI.

When UDP packet of certain length, say 730, is written to kni, it is
getting dropped.

Whereas UDP of length say 60, does not have any issues, it goes to kernel.

And this issue was not seen with previous versions, only seen when we
upgraded to v18.08

Regards
-Hirok


On Wed, May 8, 2019, 7:28 PM Wiles, Keith <keith.wiles@intel.com> wrote:

>
>
> > On May 8, 2019, at 1:41 AM, hirok borah <hirok.gen@gmail.com> wrote:
> >
> > Hi All,
> >
> > Do we have any resolution for this issue yet ?
> > This looks like a defect in DPDK 18.08.
>
> DPDK does not understand UDP/TCP or any L4 Packet per say, it only sends
> the packets constructed in the application. It will send the packets as
> defined and the only place DPDK tries to determine the packet type is when
> it receives the packet which in this case is the kernel.
>
> I do not use testpmd much, but use pktgen-dpdk for my work. Please try
> with testpmd or pktgen or TRex or moongen and see if the problem still
> exists.
> >
> > Plz share if anyone have any inputs on this issue.
> >
> > Regards
> > Hirok
> >
> > On Sat, Apr 20, 2019 at 9:41 AM Anupama Laxmi <anupamalaxmi4@gmail.com>
> > wrote:
> >
> >> We have recently upgraded to DPDK 18.08 version. After upgrading to the
> >> latest version observing issue with UDP packets transmission errors for
> few
> >> packets ( UDP size 736 bytes) .With older version of DPDK  we never
> faced
> >> this issue.
> >>
> >> Attaching the filtered tcpdump which shows "Bad UDP length > IP payload
> >> length" for one such packet.
> >>
> >> No issue observed while transferring UDP packets with size 28 bytes and
> 48
> >> bytes. I tried to print the packet length calculation in my program just
> >> before sending it out to the Kernel using rte_kni_tx_burst. The packet
> >> length calculation seems correct to me.
> >>
> >> 1.)
> >> size_udp:48
> >> sizeof(struct udp_hdr):8
> >> size_ApplMsg:40
> >> udphdr->dgram_len:12288
> >> m->data_len:82
> >> size_ip:68
> >> l2_data_shift:14
> >>
> >> 2.)
> >> size_udp:28
> >> sizeof(struct udp_hdr):8
> >> size_ApplMsg:20
> >> udphdr->dgram_len:7168
> >> m->data_len:62
> >> ip->total_length:12288
> >> size_ip:48
> >> l2_data_shift:14
> >>
> >> *** Packets with UDP size 736 are not getting transmitted to the
> >> receiving end and getting dropped.
> >>
> >> 3.)
> >> size_udp:736
> >> sizeof(struct udp_hdr):8
> >> size_ApplMsg:728
> >> udphdr->dgram_len:57346
> >> m->data_len:770
> >> size_ip:756
> >> l2_data_shift:14
> >>
> >> Also MTU is set to 1500 in my program. So it shouldn't be an issue to
> >> transfer 736 bytes UDP data which is less than 1500 bytes MTU.
> >>
> >> Looks like the kernel is dropping this packet.
> >>
> >> So I tried to increase the kernel buffer size but that didn't help.
> >>
> >> netstat -su -> output shows 0 send/receive buffer errors.
> >>
> >>
> >> What has changed in DPDK 18.08 with respect to UDP packets? Please
> suggest
> >> if I need to consider tuning udp ( using new API ) , offloading udp
> >> traffic(new offload flags)  to resolve this issue.
> >>
> >> Thanks,
> >> -------------- next part --------------
> >> A non-text attachment was scrubbed...
> >> Name: bad_udp_length.png
> >> Type: image/png
> >> Size: 56015 bytes
> >> Desc: not available
> >> URL: <
> >>
> http://mails.dpdk.org/archives/users/attachments/20190420/f5dd6779/attachment.png
> >>>
> >>
> >
> >
> > --
> > Regards,
> > -Hirok
>
> Regards,
> Keith
>
>

Re: [dpdk-users] UDP packet transmission issue - DPDK 18.08

[Wiles, Keith]

Hi Hirok,

> On May 8, 2019, at 9:28 AM, hirok borah <hirok.gen@gmail.com> wrote:
> 
> Thanks Keith for sharing the info.
> 
> But as also stated by Anupama, the problem here is with KNI. 
> 
> When UDP packet of certain length, say 730, is written to kni, it is getting dropped.
> 
> Whereas UDP of length say 60, does not have any issues, it goes to kernel.
> 
> And this issue was not seen with previous versions, only seen when we upgraded to v18.08

I have not used KNI per say, but it also does not seem to understand UDP or TCP or any another protocol sent only mbufs. Unless I am wrong that means the application must construct the UDP frame to be sent to the kernel and setup the mbuf headers correctly, right? Even the KNI test code does not seem to understand UDP frames, just looking at the code anyway.

Using testpmd or traffic generators to send a UDP frame to KNI interface and see if the problem still exists does seem like a reasonable direction to eliminate possible problems.

Beyond that I can not help, take my suggestions as you see fit.
Thanks
> 
> Regards
> -Hirok
> 
> 
> On Wed, May 8, 2019, 7:28 PM Wiles, Keith <keith.wiles@intel.com> wrote:
> 
> 
> > On May 8, 2019, at 1:41 AM, hirok borah <hirok.gen@gmail.com> wrote:
> > 
> > Hi All,
> > 
> > Do we have any resolution for this issue yet ?
> > This looks like a defect in DPDK 18.08.
> 
> DPDK does not understand UDP/TCP or any L4 Packet per say, it only sends the packets constructed in the application. It will send the packets as defined and the only place DPDK tries to determine the packet type is when it receives the packet which in this case is the kernel.
> 
> I do not use testpmd much, but use pktgen-dpdk for my work. Please try with testpmd or pktgen or TRex or moongen and see if the problem still exists.
> > 
> > Plz share if anyone have any inputs on this issue.
> > 
> > Regards
> > Hirok
> > 
> > On Sat, Apr 20, 2019 at 9:41 AM Anupama Laxmi <anupamalaxmi4@gmail.com>
> > wrote:
> > 
> >> We have recently upgraded to DPDK 18.08 version. After upgrading to the
> >> latest version observing issue with UDP packets transmission errors for few
> >> packets ( UDP size 736 bytes) .With older version of DPDK  we never faced
> >> this issue.
> >> 
> >> Attaching the filtered tcpdump which shows "Bad UDP length > IP payload
> >> length" for one such packet.
> >> 
> >> No issue observed while transferring UDP packets with size 28 bytes and 48
> >> bytes. I tried to print the packet length calculation in my program just
> >> before sending it out to the Kernel using rte_kni_tx_burst. The packet
> >> length calculation seems correct to me.
> >> 
> >> 1.)
> >> size_udp:48
> >> sizeof(struct udp_hdr):8
> >> size_ApplMsg:40
> >> udphdr->dgram_len:12288
> >> m->data_len:82
> >> size_ip:68
> >> l2_data_shift:14
> >> 
> >> 2.)
> >> size_udp:28
> >> sizeof(struct udp_hdr):8
> >> size_ApplMsg:20
> >> udphdr->dgram_len:7168
> >> m->data_len:62
> >> ip->total_length:12288
> >> size_ip:48
> >> l2_data_shift:14
> >> 
> >> *** Packets with UDP size 736 are not getting transmitted to the
> >> receiving end and getting dropped.
> >> 
> >> 3.)
> >> size_udp:736
> >> sizeof(struct udp_hdr):8
> >> size_ApplMsg:728
> >> udphdr->dgram_len:57346
> >> m->data_len:770
> >> size_ip:756
> >> l2_data_shift:14
> >> 
> >> Also MTU is set to 1500 in my program. So it shouldn't be an issue to
> >> transfer 736 bytes UDP data which is less than 1500 bytes MTU.
> >> 
> >> Looks like the kernel is dropping this packet.
> >> 
> >> So I tried to increase the kernel buffer size but that didn't help.
> >> 
> >> netstat -su -> output shows 0 send/receive buffer errors.
> >> 
> >> 
> >> What has changed in DPDK 18.08 with respect to UDP packets? Please suggest
> >> if I need to consider tuning udp ( using new API ) , offloading udp
> >> traffic(new offload flags)  to resolve this issue.
> >> 
> >> Thanks,
> >> -------------- next part --------------
> >> A non-text attachment was scrubbed...
> >> Name: bad_udp_length.png
> >> Type: image/png
> >> Size: 56015 bytes
> >> Desc: not available
> >> URL: <
> >> http://mails.dpdk.org/archives/users/attachments/20190420/f5dd6779/attachment.png
> >>> 
> >> 
> > 
> > 
> > -- 
> > Regards,
> > -Hirok
> 
> Regards,
> Keith
> 

Regards,
Keith