[dpdk-dev] DPDK - TX from lcore in packet distributor configuration

[dpdk-dev] DPDK - TX from lcore in packet distributor configuration

[deco33000 Jog]

Hello,

-- PROBLEM
I have a AF_PACKET socket which is in promiscuous mode to get all the NIC traffic and let my apps do the whole stuff.

So I have one receiver and need to communicate the packet to different threads/processes (lcore) so that they can process the rest of the packet (tcp/udp...)

-- QUESTION
I read about the packet distributor architecture which seems to answer that need.
http://dpdk.org/doc/guides/prog_guide/packet_distrib_lib.html

BUT i fear that it be slow at resending the packet to the distributor which may already be overloaded by inputs from the net. Why pass back the answer to the distributor if the lcore could send to the wire directly ?

My problem is going back to the distributor after the packet processing. i would a direct send to the tx ring.

Is it possible ? How ?  By passing the TX pointer to the lcore ?

Re: [dpdk-dev] DPDK - TX from lcore in packet distributor configuration

[Bruce Richardson]

On Wed, Jan 21, 2015 at 03:00:50PM +0100, deco33000 Jog wrote:
> Hello,
> 
> -- PROBLEM
> I have a AF_PACKET socket which is in promiscuous mode to get all the NIC traffic and let my apps do the whole stuff.
> 
> So I have one receiver and need to communicate the packet to different threads/processes (lcore) so that they can process the rest of the packet (tcp/udp...)
> 
> -- QUESTION
> I read about the packet distributor architecture which seems to answer that need.
> http://dpdk.org/doc/guides/prog_guide/packet_distrib_lib.html
> 
> BUT i fear that it be slow at resending the packet to the distributor which may already be overloaded by inputs from the net. Why pass back the answer to the distributor if the lcore could send to the wire directly ?
> 
> My problem is going back to the distributor after the packet processing. i would a direct send to the tx ring.
> 
> Is it possible ? How ?  By passing the TX pointer to the lcore ?

If you sent up the NIC so that it has multiple queues for each thread you can
have each worker send directly to the NIC. Without multiple queues, they could
still send directly, they will just have to use locking or some other access
mechanism to mediate access to the common TX queue.
The other problem with this approach is that sending packets individually is
almost always slower than sending them in bursts. To mitigate against this,
you could look to buffer packets inside the workers before transmitting them
back out, but that could lead to packets being sent out of order - not sure if
that is a problem for you or not.

The reason the distributor sample app sends the packets back to the distributor
after worker processing is to overcome these limitations. There is no additional
cross-core round trip involved in sending a packet back to the distributor along
with the request, and having the distributor re-gather the packets ensures ordering
within flows/tags is maintained. Thereafter packets can be burst-sent out an
ethernet port.

Out of interest, given you are using the AF_PACKET driver, what rate of packets
per second are you looking at? 

/Bruce

Re: [dpdk-dev] DPDK - TX from lcore in packet distributor configuration

[deco33000 Jog]

21.01.2015, 20:16, "deco33000 Jog" <deco33000@yandex.com>:
> 21.01.2015, 17:43, "Bruce Richardson" <bruce.richardson@intel.com>:
>>  On Wed, Jan 21, 2015 at 03:00:50PM +0100, deco33000 Jog wrote:
>>>   Hello,
>>>
>>>   -- PROBLEM
>>>   I have a AF_PACKET socket which is in promiscuous mode to get all the NIC traffic and let my apps do the whole stuff.
>>>
>>>   So I have one receiver and need to communicate the packet to different threads/processes (lcore) so that they can process the rest of the packet (tcp/udp...)
>>>
>>>   -- QUESTION
>>>   I read about the packet distributor architecture which seems to answer that need.
>>>   http://dpdk.org/doc/guides/prog_guide/packet_distrib_lib.html
>>>
>>>   BUT i fear that it be slow at resending the packet to the distributor which may already be overloaded by inputs from the net. Why pass back the answer to the distributor if the lcore could send to the wire directly ?
>>>
>>>   My problem is going back to the distributor after the packet processing. i would a direct send to the tx ring.
>>>
>>>   Is it possible ? How ?  By passing the TX pointer to the lcore ?
>>  If you sent up the NIC so that it has multiple queues for each thread you can
>>  have each worker send directly to the NIC. Without multiple queues, they could
>>  still send directly, they will just have to use locking or some other access
>>  mechanism to mediate access to the common TX queue.
>>  The other problem with this approach is that sending packets individually is
>>  almost always slower than sending them in bursts. To mitigate against this,
>>  you could look to buffer packets inside the workers before transmitting them
>>  back out, but that could lead to packets being sent out of order - not sure if
>>  that is a problem for you or not.
>>
>>  The reason the distributor sample app sends the packets back to the distributor
>>  after worker processing is to overcome these limitations. There is no additional
>>  cross-core round trip involved in sending a packet back to the distributor along
>>  with the request, and having the distributor re-gather the packets ensures ordering
>>  within flows/tags is maintained. Thereafter packets can be burst-sent out an
>>  ethernet port.
>>
>>  Out of interest, given you are using the AF_PACKET driver, what rate of packets
>>  per second are you looking at?
>>
>>  /Bruce
>
> Bruce, thanks,
>
> 1)
> I am looking for saturation of a 1 Gb/s link not a 10 Gb/s yet.
> I want to feel at home before going higher.
>
> I work with raw packets for security purposes.
> Do you think AF_PACKET is not good enough ? Do DPDK enables another (more efficient) solution (I may have missed it)?
>
> 2)
> I tried Netmap and PF_RING too; i am currently seeking for my "weapon" :)
> I want to use it in a socket context for a custom webserver.
>
> At first i thought the packet return would incur a cross core operation.
> But the problem remains :
> the distributor has to deal with Rx data and packets returned from the workers as well. Even if it is fast, I think I prefer to buffer + send directly form the worker to release the distributor from doing it.
> i decided to avoid the Nagle algo as much as possible because i want the least latency as possible for my app. I modulate to to balance the trade-off though.