How to rte_epoll_wait for IPC?

How to rte_epoll_wait for IPC?

[Morten Brørup]

Dear Harman, Anatoly and David,

I have been looking somewhat into power management, and have a question about rte_epoll:

Can I use rte_epoll to wait for an event (or interrupt/signal) in one EAL thread, generated by another EAL thread?

Here's a simple use case with two EAL threads:

1. The "ingress" thread receives its packets from the NICs, filters the packets and puts them into a an rte_ring for the "processing" thread. The "ingress" thread can sleep and use RX interrupts to wake up, as shown in the l3fwd-power example. All good.

2. The "processing" thread receives its packets from the rte_ring. This thread should sleep until packets are ready for it in the rte_ring.

The "ingress" thread knows when it puts packets into the rte_ring, so it can signal that event to the "processing" thread, to wake it up; either as an interrupt/signal, or through a file descriptor. Is this supported by rte_epoll (or other DPDK APIs), and how?


In a generic scenario, an EAL thread could be rte_epoll_wait'ing for a variety of event sources. This may require a different, more advanced, solution.

A solution to the simple use case suffices.


PS: I'm using the standard acronym IPC in the subject, although I'm asking about Inter Thread Communication, not Inter Process Communication. I'm not looking for a multi process solution.


Med venlig hilsen / Kind regards,
-Morten Brørup

RE: How to rte_epoll_wait for IPC?

[Honnappa Nagarahalli]

> -----Original Message-----
> From: Morten Brørup <mb@smartsharesystems.com>
> Sent: Friday, October 6, 2023 5:04 AM
> To: Harman Kalra <hkalra@marvell.com>; Anatoly Burakov
> <anatoly.burakov@intel.com>; David Hunt <david.hunt@intel.com>
> Cc: dev@dpdk.org
> Subject: How to rte_epoll_wait for IPC?
> 
> Dear Harman, Anatoly and David,
> 
> I have been looking somewhat into power management, and have a question
> about rte_epoll:
> 
> Can I use rte_epoll to wait for an event (or interrupt/signal) in one EAL thread,
> generated by another EAL thread?
> 
> Here's a simple use case with two EAL threads:
> 
> 1. The "ingress" thread receives its packets from the NICs, filters the packets
> and puts them into a an rte_ring for the "processing" thread. The "ingress"
> thread can sleep and use RX interrupts to wake up, as shown in the l3fwd-
> power example. All good.
> 
> 2. The "processing" thread receives its packets from the rte_ring. This thread
> should sleep until packets are ready for it in the rte_ring.
> 
> The "ingress" thread knows when it puts packets into the rte_ring, so it can
> signal that event to the "processing" thread, to wake it up; either as an
> interrupt/signal, or through a file descriptor. Is this supported by rte_epoll (or
> other DPDK APIs), and how?
This feature is supported by using rte_wait_until_equal_xx APIs in rte_ring.

> 
> 
> In a generic scenario, an EAL thread could be rte_epoll_wait'ing for a variety of
> event sources. This may require a different, more advanced, solution.
> 
> A solution to the simple use case suffices.
> 
> 
> PS: I'm using the standard acronym IPC in the subject, although I'm asking
> about Inter Thread Communication, not Inter Process Communication. I'm not
> looking for a multi process solution.
> 
> 
> Med venlig hilsen / Kind regards,
> -Morten Brørup

RE: How to rte_epoll_wait for IPC?

[Morten Brørup]

> From: Honnappa Nagarahalli [mailto:Honnappa.Nagarahalli@arm.com]
> Sent: Friday, 6 October 2023 17.27
> 
> > From: Morten Brørup <mb@smartsharesystems.com>
> > Sent: Friday, October 6, 2023 5:04 AM
> >
> > Dear Harman, Anatoly and David,
> >
> > I have been looking somewhat into power management, and have a
> question
> > about rte_epoll:
> >
> > Can I use rte_epoll to wait for an event (or interrupt/signal) in one
> EAL thread,
> > generated by another EAL thread?
> >
> > Here's a simple use case with two EAL threads:
> >
> > 1. The "ingress" thread receives its packets from the NICs, filters
> the packets
> > and puts them into a an rte_ring for the "processing" thread. The
> "ingress"
> > thread can sleep and use RX interrupts to wake up, as shown in the
> l3fwd-
> > power example. All good.
> >
> > 2. The "processing" thread receives its packets from the rte_ring.
> This thread
> > should sleep until packets are ready for it in the rte_ring.
> >
> > The "ingress" thread knows when it puts packets into the rte_ring, so
> it can
> > signal that event to the "processing" thread, to wake it up; either as
> an
> > interrupt/signal, or through a file descriptor. Is this supported by
> rte_epoll (or
> > other DPDK APIs), and how?
> This feature is supported by using rte_wait_until_equal_xx APIs in
> rte_ring.

Thank you for the suggestion, Honnappa.

rte_wait_until_equal_xx would be an excellent solution for physical appliances.

However, I am looking for O/S level sleeping, like epoll(), so the hypervisor can detect that the thread is idle. (I should have mentioned this!)

> 
> >
> >
> > In a generic scenario, an EAL thread could be rte_epoll_wait'ing for a
> variety of
> > event sources. This may require a different, more advanced, solution.
> >
> > A solution to the simple use case suffices.
> >
> >
> > PS: I'm using the standard acronym IPC in the subject, although I'm
> asking
> > about Inter Thread Communication, not Inter Process Communication. I'm
> not
> > looking for a multi process solution.
> >
> >
> > Med venlig hilsen / Kind regards,
> > -Morten Brørup

Re: How to rte_epoll_wait for IPC?

[Stephen Hemminger]

On Fri, 6 Oct 2023 12:03:46 +0200
Morten Brørup <mb@smartsharesystems.com> wrote:

> 2. The "processing" thread receives its packets from the rte_ring. This thread should sleep until packets are ready for it in the rte_ring.
> 
> The "ingress" thread knows when it puts packets into the rte_ring, so it can signal that event to the "processing" thread, to wake it up; either as an interrupt/signal, or through a file descriptor. Is this supported by rte_epoll (or other DPDK APIs), and how?

When having to do this in applications, I used a eventfd() as well as the rte ring.
The ingress write's to eventfd and the other thread used epoll on the fd.

Optimized this by having the ingress thread only signal via write if ring was in empty state.
And the reader thread only needs to do (epoll/read) if ring became empty during last cycle;
i.e num packets from rte_ring_burst() was 0.

Basically, when using epoll and other SW event models, you need to turn all data sources
into file descriptors. 

RE: How to rte_epoll_wait for IPC?

[Morten Brørup]

> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> Sent: Friday, 6 October 2023 18.03
> 
> On Fri, 6 Oct 2023 12:03:46 +0200
> Morten Brørup <mb@smartsharesystems.com> wrote:
> 
> > 2. The "processing" thread receives its packets from the rte_ring.
> This thread should sleep until packets are ready for it in the rte_ring.
> >
> > The "ingress" thread knows when it puts packets into the rte_ring, so
> it can signal that event to the "processing" thread, to wake it up;
> either as an interrupt/signal, or through a file descriptor. Is this
> supported by rte_epoll (or other DPDK APIs), and how?
> 
> When having to do this in applications, I used a eventfd() as well as
> the rte ring.
> The ingress write's to eventfd and the other thread used epoll on the
> fd.
> 
> Optimized this by having the ingress thread only signal via write if
> ring was in empty state.
> And the reader thread only needs to do (epoll/read) if ring became empty
> during last cycle;
> i.e num packets from rte_ring_burst() was 0.
> 
> Basically, when using epoll and other SW event models, you need to turn
> all data sources
> into file descriptors.

This sounds exactly like what I am looking for.

Are there DPDK APIs for eventfd(), so I can use them with rte_epoll_wait?

Re: How to rte_epoll_wait for IPC?

[Stephen Hemminger]

On Fri, 6 Oct 2023 18:28:10 +0200
Morten Brørup <mb@smartsharesystems.com> wrote:

> > From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> > Sent: Friday, 6 October 2023 18.03
> > 
> > On Fri, 6 Oct 2023 12:03:46 +0200
> > Morten Brørup <mb@smartsharesystems.com> wrote:
> >   
> > > 2. The "processing" thread receives its packets from the rte_ring.  
> > This thread should sleep until packets are ready for it in the rte_ring.  
> > >
> > > The "ingress" thread knows when it puts packets into the rte_ring, so  
> > it can signal that event to the "processing" thread, to wake it up;
> > either as an interrupt/signal, or through a file descriptor. Is this
> > supported by rte_epoll (or other DPDK APIs), and how?
> > 
> > When having to do this in applications, I used a eventfd() as well as
> > the rte ring.
> > The ingress write's to eventfd and the other thread used epoll on the
> > fd.
> > 
> > Optimized this by having the ingress thread only signal via write if
> > ring was in empty state.
> > And the reader thread only needs to do (epoll/read) if ring became empty
> > during last cycle;
> > i.e num packets from rte_ring_burst() was 0.
> > 
> > Basically, when using epoll and other SW event models, you need to turn
> > all data sources
> > into file descriptors.  
> 
> This sounds exactly like what I am looking for.
> 
> Are there DPDK APIs for eventfd(), so I can use them with rte_epoll_wait?

Eventfd is just a system call.

You then add the resulting fd to epoll object with rte_epoll_ctl().

RE: How to rte_epoll_wait for IPC?

[Honnappa Nagarahalli]

> -----Original Message-----
> From: Morten Brørup <mb@smartsharesystems.com>
> Sent: Friday, October 6, 2023 10:53 AM
> To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Harman Kalra
> <hkalra@marvell.com>; Anatoly Burakov <anatoly.burakov@intel.com>; David
> Hunt <david.hunt@intel.com>
> Cc: dev@dpdk.org; nd <nd@arm.com>; nd <nd@arm.com>
> Subject: RE: How to rte_epoll_wait for IPC?
> 
> > From: Honnappa Nagarahalli [mailto:Honnappa.Nagarahalli@arm.com]
> > Sent: Friday, 6 October 2023 17.27
> >
> > > From: Morten Brørup <mb@smartsharesystems.com>
> > > Sent: Friday, October 6, 2023 5:04 AM
> > >
> > > Dear Harman, Anatoly and David,
> > >
> > > I have been looking somewhat into power management, and have a
> > question
> > > about rte_epoll:
> > >
> > > Can I use rte_epoll to wait for an event (or interrupt/signal) in
> > > one
> > EAL thread,
> > > generated by another EAL thread?
> > >
> > > Here's a simple use case with two EAL threads:
> > >
> > > 1. The "ingress" thread receives its packets from the NICs, filters
> > the packets
> > > and puts them into a an rte_ring for the "processing" thread. The
> > "ingress"
> > > thread can sleep and use RX interrupts to wake up, as shown in the
> > l3fwd-
> > > power example. All good.
> > >
> > > 2. The "processing" thread receives its packets from the rte_ring.
> > This thread
> > > should sleep until packets are ready for it in the rte_ring.
> > >
> > > The "ingress" thread knows when it puts packets into the rte_ring,
> > > so
> > it can
> > > signal that event to the "processing" thread, to wake it up; either
> > > as
> > an
> > > interrupt/signal, or through a file descriptor. Is this supported by
> > rte_epoll (or
> > > other DPDK APIs), and how?
> > This feature is supported by using rte_wait_until_equal_xx APIs in
> > rte_ring.
> 
> Thank you for the suggestion, Honnappa.
> 
> rte_wait_until_equal_xx would be an excellent solution for physical appliances.
> 
> However, I am looking for O/S level sleeping, like epoll(), so the hypervisor can
> detect that the thread is idle. (I should have mentioned this!)
WFE is trapped by the hypervisor if there are multiple vCPUs mapped to the same physical core and another vCPU is allowed to run.
If there is only one vCPU mapped, then hypervisor does not trap it. This allows the CPU to go to low power state.

> 
> >
> > >
> > >
> > > In a generic scenario, an EAL thread could be rte_epoll_wait'ing for
> > > a
> > variety of
> > > event sources. This may require a different, more advanced, solution.
> > >
> > > A solution to the simple use case suffices.
> > >
> > >
> > > PS: I'm using the standard acronym IPC in the subject, although I'm
> > asking
> > > about Inter Thread Communication, not Inter Process Communication.
> > > I'm
> > not
> > > looking for a multi process solution.
> > >
> > >
> > > Med venlig hilsen / Kind regards,
> > > -Morten Brørup