External email: Use caution opening links or attachments


> From: Etelson, Gregory
> Sent: Friday, May 02, 2025 1:46 PM
> To: Richardson, Bruce
> Cc: Gregory Etelson; Van Haaren, Harry; dev@dpdk.org; owen.hilyard@unh.edu
> Subject: Re: [PATCH] rust: RFC/demo of safe API for Dpdk Eal, Eth and Rxq
>
> Hello Bruce,

Hi All,

> > Thanks for sharing. However, IMHO using EAL for thread management in rust
> > is the wrong interface to expose.
>
> EAL is a singleton object in DPDK architecture.
> I see it as a hub for other resources.

Yep, i tend to agree here; EAL is central to the rest of DPDK working correctly.
And given EALs implementation is heavily relying on global static variables, it is
certainly a "singleton" instance, yes.

> Following that idea, the EAL structure can be divided to hold the
> "original" resources inherited from librte_eal and new resources
> introduced in Rust EAL.

Here we can look from different perspectives. Should "Rust EAL" even exist?
If so, why? The DPDK C APIs were designed in baremetal/linux days, where
certain "best-practices" didn't exist yet, and Rust language was pre 1.0 release.

Of course, certain parts of Rust API must depend on EAL being initialized.
There is a logical flow to DPDK initialization, these must be kept for correct functionality.

I guess I'm saying, perhaps we can do better than mirroring the concept of
"DPDK EAL in C" in to "DPDK EAL in Rust".

> > Instead, I believe we should be
> > encouraging native rust thread management, and not exposing any DPDK
> > threading APIs except those necessary to have rust threads work with DPDK,
> > i.e. with an lcore ID. Many years ago when DPDK started, and in the C
> > world, having DPDK as a runtime environment made sense, but times have
> > changed and for Rust, there is a whole ecosystem out there already that we
> > need to "play nice with", so having Rust (not DPDK) do all thread
> > management is the way to go (again IMHO).
> >
>
> I'm not sure what exposed DPDK API you refer to.

I think that's the point :) Perhaps the Rust application should decide how/when to
create threads, and how to schedule & pin them. Not the "DPDK crate for Rust".
To give a more concrete examples, lets look at Tokio (or Monoio, or Glommio, or .. )
which are prominent players in the Rust ecosystem, particularly for networking workloads
where request/response patterns are well served by the "async" programming model (e.g HTTP server).

Lets focus on Tokio first: it is an "async runtime" (two links for future readers)
    https://corrode.dev/blog/async/
    https://rust-lang.github.io/async-book/08_ecosystem/00_chapter.html
So an async runtime can run "async" Rust functions (called Futures, or Tasks when run independently..)
There are lots of words/concepts, but I'll focus only on the thread creation/control aspect, given the DPDK EAL lcore context.

Tokio is a work-stealing scheduler. It spawns "worker" threads, and then gives these "tasks"
to various worker cores (similar to how Golang does its work-stealing scheduling). Some
DPDK crate users might like this type of workflow, where e.g. RXQ polling is a task, and the
"tokio runtime" figures out which worker to run it on. "Spawning" a task causes the "Future"
to start executing. (technical Rust note: notice the "Send" bound on Future: https://docs.rs/tokio/latest/tokio/task/fn.spawn.html )

Other users might prefer the "thread-per-core" and CPU pinning approach (like DPDK itself would do).
Monoio and Glommio both serve these use cases (but in slightly different ways!). They both spawn threads and do CPU pinning.
Monoio and Glommio say "tasks will always remain on the local thread". In Rust techie terms: "Futures are !Send and !Sync"
    https://docs.rs/monoio/latest/monoio/fn.spawn.html
    https://docs.rs/glommio/latest/glommio/fn.spawn_local.html

So there are at least 3 different async runtimes (and I haven't even talked about async-std, smol, embassy, ...) which
all have different use-cases, and methods of running "tasks" on threads. These runtimes exist, and are widely used,
and applications make use of their thread-scheduling capabilities.

So "async runtimes" do thread creation (and optionally CPU pinning) for the user.
Other libraries like "Rayon" are thread-pool managers, those also have various CPU thread-create/pinning capabilities.
If DPDK *also* wants to do thread creation/management and CPU-thread-to-core pinning for the user, that creates tension.

> Bruce wrote: "so having Rust (not DPDK) do all thread management is the way to go (again IMHO)."

I think I agree here, in order to make the Rust DPDK crate usable from the Rust ecosystem,
it must align itself with the existing Rust networking ecosystem.

That means, the DPDK Rust crate should not FORCE the usage of lcore pinnings and mappings.
Allowing a Rust application to decide how to best handle threading (via Rayon, Tokio, Monoio, etc)
will allow much more "native" or "ergonomic" integration of DPDK into Rust applications.

> Regards,
> Gregory

Apologies for the long-form, "wall of text" email, but I hope it captures the nuance of threading and
async runtimes, which I believe in the long term will be very nice to capture "async offload" use-cases
for DPDK. To put it another way, lookaside processing can be hidden behind async functions & runtimes,
if we design the APIs right: and that would be really cool for making async-offload code easy to write correctly!

Regards, -Harry