[dpdk-dev] Weird behavior of DPDK - ongoing problem

[dpdk-dev] Weird behavior of DPDK - ongoing problem

[Meir Tseitlin]

Hi guys,

I am still struggling to find a solution for a problem each time I face
from different angle (already wrote few posts). Now I think it is some kind
of DMA problem.

I am modifying l2fwd example to collect packet flow from high speed
ethernet device and store it in file, this is why I use pcap driver
attached to file as a one side of l2fwd.
In addition I need to send control packets to the device and receive
answers.

Initially I implemented packet injection mechanism using rte_pktmbuf_alloc
(I tried both original mempool or additional one). I checked receive
packets for specific packet type it header, to detect if it is data packet
or an answer to control packet (answers to control packets are  not
forwarded to file).

The problem is that in 30% of the cases data packet enters the path of
control packet instead of expected answer. Which probably means that after
my packet type check, the mbuf is overwritten before handled properly.

To overcome this problem I created a TAP device for control packets
(instead of allocating them with rte_pktmbuf_alloc) and initialized it with
Pcap driver. But I am facing the same problem - on the receive side, when
control answer packet arrives is it skipped and another data packet sent to
TAP device.

It seems like mbuf containing control packet, intended to be sent to TAP
device is overwritten with data packet after the routing mechanism (maybe
with DMA?)

I also tried enabling sanity check before calling to rte_eth_tx_burst and
it fails.

Please help,
It is really frustrating

Meir

Re: [dpdk-dev] Weird behavior of DPDK - ongoing problem

[Tetsuya.Mukawa]

Hi Meir,

(2014/04/13 19:21), Meir Tseitlin wrote:
> The problem is that in 30% of the cases data packet enters the path of
> control packet instead of expected answer. Which probably means that after
> my packet type check, the mbuf is overwritten before handled properly.
If you are using DPDK-1.5, it's nice to update to DPDK-1.6, because pcap
pmd of DPDK-1.5 has buffer overflow while receiving  data from a target
device/file. And the issue was solved with DPDK-1.6 (in both intel
original code and dpdk.org code).

Also if multiple processes/threads run on a same cpu, probably, it could
cause a problem like you're facing.

Regards,
Tetsuya

Re: [dpdk-dev] Weird behavior of DPDK - ongoing problem

[Meir Tseitlin]

Ok, I think I figured it out - this issue was partly my fault.

I do work with 1.6r0 and I also use Pcap driver.
Pcap driver still contains bug which does not properly release all packets
in rte_eth_tx_burst, causing application to run out of memory. I addressed
this problem wrongly by manually releasing them before they actually sent.
This caused early reallocation.

Anyway, thanks for help!
Meir Tseitlin


On Mon, Apr 14, 2014 at 10:20 AM, Tetsuya.Mukawa <mukawa@igel.co.jp> wrote:

> Hi Meir,
>
> (2014/04/13 19:21), Meir Tseitlin wrote:
> > The problem is that in 30% of the cases data packet enters the path of
> > control packet instead of expected answer. Which probably means that
> after
> > my packet type check, the mbuf is overwritten before handled properly.
> If you are using DPDK-1.5, it's nice to update to DPDK-1.6, because pcap
> pmd of DPDK-1.5 has buffer overflow while receiving  data from a target
> device/file. And the issue was solved with DPDK-1.6 (in both intel
> original code and dpdk.org code).
>
> Also if multiple processes/threads run on a same cpu, probably, it could
> cause a problem like you're facing.
>
> Regards,
> Tetsuya
>



-- 
Kind regards,
*Meir Tseitlin*
Software architect*Mobile:* +972.54.7647417
*Fax:* +972.72.2812365
*Email:* meir.tech@gmail.com
 *http://il.linkedin.com/in/meirts <http://il.linkedin.com/in/meirts>*
*Independent consultant*
<http://maps.google.com/maps?q=&hl=en>
See who we know in
common<http://www.linkedin.com/e/wwk/6408776/?hs=false&tok=2g4qg82N8Sb5U1>Want
a signature like
this?<http://www.linkedin.com/e/sig/6408776/?hs=false&tok=20mz9jA_USb5U1>