Hello all,
I wanted to chime in as someone who uses DPDK from Rust in quite a few projects.
No snips so the conversation can continue past me for things I don't comment on.
Hello Harry,
>>>> My concern is how to properly maintain Rust crate once DPDK starts to implement
>>>> it's own API.
>>>
>>> I'm not really sure what is meant here. I don't understand what "own" word refers to?
>>>
>>> I see it like this:
>>> - DPDK has the public C API exported (and that stays the same as today, with ABI rules, version.map files, etc)
>>> - The Rust crate consumes the public C API (via bindgen, as done in this patch. More detail about bindgen below.)
>>>
>>
>> Bindgen cannot provide access to all DPDK public API.
>
> Ah - you're referring to C static inline functions, declared in header files, which bindgen doesn't wrap.
> Correct - thanks - I understand your point now.
>
>> A good example here is rte_eth_rx_burst().
>> That function is defined as inline and bindgen does not translate it.
>> Also, the function definition references rte_eth_fp_ops array that is not part of the
>> public DPDK API. That means Rust cannot duplicate rte_eth_rx_burst() "as-is" and
>> the solution can require extensions to existing DPDK API.
>>
>> I added a new public API that exports rte_eth_fp_ops for a given port Id.
>>
>> Rust implementation of rte_eth_rx_burst() does not have to follow the original
>> approach.
>> Usage of rte_eth_fp_ops is good for C, but Rust has different methods.
>
> Agreed there is a certain "mismatch" sometimes, if functions aren't in the
> actually "C ABI" then they can't be called via bindgen.
>
> Agree that elegant solutions (clean, maintainable, and high performance) will have
> to be found here. Many existing solutions just wrap the "static inline" function into
> a "non-static" function, and export it as a public symbol. That allows calling into it
> from Rust (via bindgen-generated header) however causes an actual function call..
> (LTO _might_ fix/inline it, but not everybody compiles with LTO.. link times!)
>
The core DPDK code is maintained as a self-contained pure C project.
What about extending that model and provide direct access to DPDK resources that
are needed for Rust API ?
That can be arranged as part of "native" DPDK API or as extension for
Rust-enabled DPDK only.
I'm currently experimenting with the latter in
There are some performance wins to be found along this path. It's not a lot, but you can "devirtualize" a lot of DPDK if you use Rust's generics to specialize on particular hardware combinations, at the cost of multiplying code size by the number of supported
NICs. I can do this in my bindings since I tend to only need to test a few software PMDs and the hardware I have, with a fallback to "dyn dpdk::EthDev" which pulls data from info structs all of the time, instead of only pulling runtime-only information (ex:
mac addr, rss key). This is paired with a top-level function which does dispatch for different hardware to "lift" the runtime information about what NIC is used to compile time. I think the main win here is from inlining driver functions, but I haven't done
a detailed analysis of the "why". These are marginal improvements, around a half percent for nic_single_core_perf on my hardware, but there may be more wins for less heavily optimized paths through DPDK where the compiler can do more heavy lifting.
> As DPDK uses static-inline functions primarily for "packet-at-a-time" performance reasons,
> it is unfortunate to give-up (some small amounts of..?) performance by having a C->Rust FFI call.
> We have work to do to find/propose the best solution.
What exactly is a small amount of performance degradation ?
For some existing performance oriented projects loosing performance to
eliminate the `unsafe {}` construct is not an option.
If you are willing to compile with clang and use LTO, both rustc and clang will emit LLVM IR, which gets inlined at link time. I haven't been able to measure a performance difference vs C code even when calling "static inline" hot path function wrappers across
the FFI boundary. This does increase compile times by quite a bit, more than LTO with pure C would increase them due to the amount of LLVM IR that rustc emits, but it means that the performance difference barely exists if it does at all.
>
>> For conclusion, Rust DPDK infrastructure cannot relay on bindgen only and needs
>> to provide native implementation for some public DPDK API.
>> It can be easier to maintain Rust files separately.
>
> OK, now I understand your "kind-of-C-DPDK, kind of Rust-DPDK" code, or the "own" code reference above.
> I'm not sure right now where that would be best implemented/maintained, I'll have to think about it and do a POC.
>
>
>
> Thanks for the link - I see you've been pushing code actively! Good to see opportunities,
> and compare approaches and concepts. Have you investigated wrapping the various pointers
> into structs, and providing safer APIs? For example, the [*mut rte_mbuf; 64] array for RX causes
> raw pointers to be handled for all packet-processing - resulting in "unsafe{ /* work here */ }" blocks.
>
This construct can work:
struct PktsBuf<const SIZE: usize> {
buffer: [*mut rte_mbuf; SIZE]
}
I think that they mean a construct that wraps a *mut rte_mbuf and provides a safe interface. Another option, depending on how invasive the bindings are allowed to be, is to make them references with a lifetime tied to the mempool they are allocated from. For
many usecases, mempools exist for the entire lifetime of the program and have static lifetimes, meaning that the references have static lifetimes which greatly simplifies a lot of the code. This also gets rid of unsafe accesses. However, it would require a
wrapper around the buffer with the valid length, something like a fixed-capacity vector. Another option is to make the pointers Option<NonNull<rte_mbuf>>, which has the same ABI as a C *rte_mbuf but forces null checks. This will be annoying to program against,
but for "one packet at a time to completion" processing it's equivalent to "if (buffer[i] != null) { ... }" at the top of the loop. For the pipeline or graph libraries, this latter option will incur a lot of potentially extra null checks, but the upside is
that it's ABI compatible with the burst functions from the C API.
> The code in the above repo feels like "DPDK C code written in Rust".
Agree.
At this stage there is no native Rust DPDK API.
> It is a great step towards better
> understanding, and having something that works is very valuable; thanks for sharing it.
>
> A "top down" application view might help to brainstorm idiomatic/Safe Rust APIs, and then we can
> discuss/understand how to map these high level APIs onto the "DPDK C in Rust" or even "DPDK C API/ABI" layers.
>
> Does that seem like a good method to you, to achieve an ergonomic Safe Rust API as the end result?
>
I think we need to conclude what the safe API means in terms of DPDK project.
Because it's possible to provide native Rust API for DPDK what will use FFI.
Specially, if Rust PMD is not in plans and performance is one of the main goals.
I think most of the value of Rust can be gotten by making a good Rust API available to consumers of DPDK. There is value in having Rust for PMDs, but there is a lot more code written which consumes DPDK than code in new PMDs. It might be better to hold that
conversation until someone wants to write a Rust PMD. I think that there is also value in a version which is "maximum safety" that takes performance hits where necessary. DPDK is far enough ahead of the performance of most other options that "DPDK at 80%"
is still going to be fast enough for many purposes and the extra safety is valuable there from an ease-of-use perspective. There are, of course, applications which need everything DPDK has to offer from a performance perspective, and those should be served
as well, but I think DPDK can offer a spectrum where users get steadily closer to "zero overhead abstraction over the DPDK C API" the more performance they need, possibly sacrificing some safety along the way.
>
>>> Next steps are to "allowlist" more DPDK public API functions, and start building "Safe Rust" APIs over
>>> them, in order to expose an ergonomic and misuse-resistant API. As you note, this is where a network ports,
>>> lcores, mempools, and ethdev configuration are all required. First goal something like "Safe L2 macswap"?
check out for Rx, L2 addr swap, Tx sequence here:
Regards,
Gregory