[dpdk-dev] Regarding mbuf allocation/free in secondary process

[dpdk-dev] Regarding mbuf allocation/free in secondary process

[Saravana Kumar]

Hi DPDK community,



I'd like to have DPDK NIC IO operations in (primary) process and
execution logic in (secondary) processes.
Primary process pushes NIC Rx mbufs to Secondary process through S/W ring

Seconary process allocates mbuf for Tx path and pushes down to Primary
process for NIC Tx


I have few doubts here:

1. If Secondary process dies because of SIGKILL then how can the mbufs
allocated in Secondary process can be freed.
   If it is normal signals like SIGINT/SIGTERM then we can be catch
those and free in those respective signal handlers

2. Secondary process needs to poll on the S/W ring. This can consume 100% cpu.
   Is there a way to avoid polling in secondary process for Rx path

Thanks

Sara

Re: [dpdk-dev] Regarding mbuf allocation/free in secondary process

[Bruce Richardson]

On Tue, Feb 09, 2016 at 11:43:19PM -0800, Saravana Kumar wrote:
> Hi DPDK community,
> 
> 
> 
> I'd like to have DPDK NIC IO operations in (primary) process and
> execution logic in (secondary) processes.
> Primary process pushes NIC Rx mbufs to Secondary process through S/W ring
> 
> Seconary process allocates mbuf for Tx path and pushes down to Primary
> process for NIC Tx
> 
> 
> I have few doubts here:
> 
> 1. If Secondary process dies because of SIGKILL then how can the mbufs
> allocated in Secondary process can be freed.
>    If it is normal signals like SIGINT/SIGTERM then we can be catch
> those and free in those respective signal handlers

If a process terminates abnormally then the buffers being used by that process
may well be leaked. The solution you propose of catching signals will certainly
help as you want to try and ensure that a process always frees all its buffers
properly on termination.

> 
> 2. Secondary process needs to poll on the S/W ring. This can consume 100% cpu.
>    Is there a way to avoid polling in secondary process for Rx path

Not using DPDK software rings, no. You'd have to use some kernel constructs such as
fifo's/named pipes to do blocking reads on those. However, the overhead of using
such structures can be severe making them unusable for many packet processing
applications. An alternative might be to use some small sleep calls i.e. nanosleep
between polls of the SW ring in cases where traffic rates are low. That will 
reduce your cpu usage.

/Bruce

Re: [dpdk-dev] Regarding mbuf allocation/free in secondary process

[Saravana Kumar]

Thanks for your response..

Sara

On Wed, Feb 10, 2016 at 2:01 AM, Bruce Richardson <
bruce.richardson@intel.com> wrote:

> On Tue, Feb 09, 2016 at 11:43:19PM -0800, Saravana Kumar wrote:
> > Hi DPDK community,
> >
> >
> >
> > I'd like to have DPDK NIC IO operations in (primary) process and
> > execution logic in (secondary) processes.
> > Primary process pushes NIC Rx mbufs to Secondary process through S/W ring
> >
> > Seconary process allocates mbuf for Tx path and pushes down to Primary
> > process for NIC Tx
> >
> >
> > I have few doubts here:
> >
> > 1. If Secondary process dies because of SIGKILL then how can the mbufs
> > allocated in Secondary process can be freed.
> >    If it is normal signals like SIGINT/SIGTERM then we can be catch
> > those and free in those respective signal handlers
>
> If a process terminates abnormally then the buffers being used by that
> process
> may well be leaked. The solution you propose of catching signals will
> certainly
> help as you want to try and ensure that a process always frees all its
> buffers
> properly on termination.
>
> >
> > 2. Secondary process needs to poll on the S/W ring. This can consume
> 100% cpu.
> >    Is there a way to avoid polling in secondary process for Rx path
>
> Not using DPDK software rings, no. You'd have to use some kernel
> constructs such as
> fifo's/named pipes to do blocking reads on those. However, the overhead of
> using
> such structures can be severe making them unusable for many packet
> processing
> applications. An alternative might be to use some small sleep calls i.e.
> nanosleep
> between polls of the SW ring in cases where traffic rates are low. That
> will
> reduce your cpu usage.
>
> /Bruce
>
>

Re: [dpdk-dev] Regarding mbuf allocation/free in secondary process

[Lawrence MacIntyre]

Sara:

The use of polling is to provide high throughput. At high bandwidths, 
interrupt processing is  a great contributor to latency and can greatly 
decrease available bandwidth. Polling eliminates the overhead of 
interrupts but it will consume an entire CPU. That is why the DPDK 
library makes it easy to put each thread on a separate core. The 
downside is that for low bandwidth applications, you end up using a lot 
of CPU resources spinning in loops waiting for input.

Using the techniques Bruce mentioned will allow you to code a 
low-bandwidth, low CPU utilization app, but it will not be able to 
handle high bandwidth. You will have to choose one or the other. 
Unfortunately, computer hardware has a limited number of interrupts that 
it can handle per second, and those limits make it impractical to handle 
bandwidths much greater than 1 Gb/s (i.e 10, 40, or 100 Gb/s) using 
interrupts to handle I/O.

Lawrence

This one time (02/10/2016 05:01 AM), at band camp, Bruce Richardson wrote:
> On Tue, Feb 09, 2016 at 11:43:19PM -0800, Saravana Kumar wrote:
>> Hi DPDK community,
>>
>>
>>
>> I'd like to have DPDK NIC IO operations in (primary) process and
>> execution logic in (secondary) processes.
>> Primary process pushes NIC Rx mbufs to Secondary process through S/W ring
>>
>> Seconary process allocates mbuf for Tx path and pushes down to Primary
>> process for NIC Tx
>>
>>
>> I have few doubts here:
>>
>> 1. If Secondary process dies because of SIGKILL then how can the mbufs
>> allocated in Secondary process can be freed.
>>     If it is normal signals like SIGINT/SIGTERM then we can be catch
>> those and free in those respective signal handlers
> If a process terminates abnormally then the buffers being used by that process
> may well be leaked. The solution you propose of catching signals will certainly
> help as you want to try and ensure that a process always frees all its buffers
> properly on termination.
>
>> 2. Secondary process needs to poll on the S/W ring. This can consume 100% cpu.
>>     Is there a way to avoid polling in secondary process for Rx path
> Not using DPDK software rings, no. You'd have to use some kernel constructs such as
> fifo's/named pipes to do blocking reads on those. However, the overhead of using
> such structures can be severe making them unusable for many packet processing
> applications. An alternative might be to use some small sleep calls i.e. nanosleep
> between polls of the SW ring in cases where traffic rates are low. That will
> reduce your cpu usage.
>
> /Bruce
>
>

-- 
Lawrence MacIntyre  macintyrelp@ornl.gov  Oak Ridge National Laboratory
  865.574.7401  Cyber Space and Information Intelligence Research Group