graph: dispatch mode with libevent for control plane

graph: dispatch mode with libevent for control plane

[Robin Jarry]

Hi folks,

I had a look at the graph dispatch mode and wondered if it could be 
adapted to be used with event loops for "slow path" processing.

The use case I have in mind is having lcores dedicated for control plane 
operations which are not latency sensitive. It would be nice to allow 
data plane workers to send packets to these lcores while keeping the 
packets in the graph.

The problem I have with this solution for now, is that all CPUs 
processing the graph with rte_graph_walk_mcore_dispatch() are supposed 
to do busy polling. This is not compatible with select/epoll based event 
loops.

Would it be possible to have a way for busy-polling threads to signal 
non-busy-polling threads that there are packets waiting for them in the 
graph ring? In grout, we implemented something outside of the graph 
using pthread_cond_signal().

https://github.com/DPDK/grout/blob/main/modules/infra/datapath/control_output.c
https://github.com/DPDK/grout/blob/main/modules/infra/control/control_output.c

Unless I am mistaken, this does not perform any syscall and does not 
require acquiring any lock. Only the "slow" thread needs to acquire 
locks and wait for the data path threads to call pthread_cond_signal().

This solution can only be used for busy-polling -> epoll-based 
communication, for the other direction epoll-based -> busy-polling there 
would not need any specific signaling required. Only posting the packets 
to the graph ring would be enough.

What do you think? How would you integrate this into rte_graph? What 
would the API look like in your opinion?

Thanks!

-- 
Robin

RE: [EXTERNAL] graph: dispatch mode with libevent for control plane

[Jerin Jacob]

> -----Original Message-----
> From: Robin Jarry <rjarry@redhat.com>
> Sent: Monday, December 9, 2024 2:58 PM
> To: Nitin Saxena <nsaxena@marvell.com>; dev@dpdk.org; grout@dpdk.org
> Cc: Jerin Jacob <jerinj@marvell.com>; David Marchand
> <david.marchand@redhat.com>; Christophe Fontaine <cfontain@redhat.com>;
> Nitin Saxena <nsaxena16@gmail.com>
> Subject: [EXTERNAL] graph: dispatch mode with libevent for control plane
> 
> Hi folks, I had a look at the graph dispatch mode and wondered if it could be
> adapted to be used with event loops for "slow path" processing. The use case I
> have in mind is having lcores dedicated for control plane operations which are
> not latency 
> Hi folks,
> 
> I had a look at the graph dispatch mode and wondered if it could be adapted to
> be used with event loops for "slow path" processing.
> 
> The use case I have in mind is having lcores dedicated for control plane
> operations which are not latency sensitive. It would be nice to allow data plane
> workers to send packets to these lcores while keeping the packets in the graph.
> 
> The problem I have with this solution for now, is that all CPUs processing the
> graph with rte_graph_walk_mcore_dispatch() are supposed to do busy polling.
> This is not compatible with select/epoll based event loops.
> 
> Would it be possible to have a way for busy-polling threads to signal non-busy-
> polling threads that there are packets waiting for them in the graph ring? In
> grout, we implemented something outside of the graph using
> pthread_cond_signal().

Sorry for delay, I was on business travel.

Adding yanzhirun_163@163.com(mcore model maintainer).

If I understand it correctly,  we need three operations.
a)wait object creation.
b)wait object waiting instead of polling.
c)signal wait object from fastpath to wake up (b)

I see two options.

Option A:

In application (without graph change)

1)call (a)
2)
while(1)
{
     Call (b)
     rte_graph_walk_mcore_dispatch()

}
3)Call (c) from fp nodes(Assuming it is out of tree nodes)


Option B

1)Call (a) in  model create or so
2)Add an option in  rte_graph_walk_mcore_dispatch() to call (b)
3)Update rte_.._enqueue() to call (c) after ring enqueue

Option B is need if we need inbuilt fp nodes.



> 
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__github.com_DPDK_grout_blob_main_modules_infra_datapath_control-
> 5Foutput.c&d=DwIFaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=1DGob4H4rxz6H8uIT
> ozGOCa0s5f4wCNtTa4UUKvcsvI&m=sctQnRalWyQEGrI4o2UpcCXD8n8qveibIe4C
> clcMOUmS7Oi0S7MoVryikpo_ZDh1&s=Zm-
> i7a_D6MFxZO362gYHHe9PtKO_Kr6JeFNlmEHpMCg&e=
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__github.com_DPDK_grout_blob_main_modules_infra_control_control-
> 5Foutput.c&d=DwIFaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=1DGob4H4rxz6H8uIT
> ozGOCa0s5f4wCNtTa4UUKvcsvI&m=sctQnRalWyQEGrI4o2UpcCXD8n8qveibIe4C
> clcMOUmS7Oi0S7MoVryikpo_ZDh1&s=09ea7Sa47B5SPe7SjP5zBNMzqem0XzI3
> HnlU-k73N2w&e=
> 
> Unless I am mistaken, this does not perform any syscall and does not require
> acquiring any lock. Only the "slow" thread needs to acquire locks and wait for
> the data path threads to call pthread_cond_signal().
> 
> This solution can only be used for busy-polling -> epoll-based communication,
> for the other direction epoll-based -> busy-polling there would not need any
> specific signaling required. Only posting the packets to the graph ring would be
> enough.
> 
> What do you think? How would you integrate this into rte_graph? What would
> the API look like in your opinion?
> 
> Thanks!
> 
> --
> Robin