[dpdk-dev] Minutes of Technical Board Meeting, 2020-11-18

[dpdk-dev] Minutes of Technical Board Meeting, 2020-11-18

[Ananyev, Konstantin]

Minutes of Technical Board Meeting, 2020-11-18

Members Attending
---------------------------
-Bruce
-Ferruh
-Hemant
-Honnappa
-Jerin
-Kevin
-Konstantin (Chair)
-Maxime
-Olivier
-Thomas

NOTE: The technical board meetings every second Wednesday in
https://meet.jit.si/DPDK at 3 pm UTC.
Meetings are public, and DPDK community members  are welcome to attend.
NOTE: Next meeting will be on Wednesday 2020-12-02 @3pm UTC, and will be
chaired by Maxime.

DPDK build configuration - future enhancements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
There are multiple requests (sometimes controversial) for new abilities
to add into DPDK build system.  
In particular, request from few different teams:
  - add ability to enable/disable individual apps/libs
  - override some build settings for specific libs/drivers
As a first step to move forward - produce design doc of current build system.
Discuss further enhancements based on that doc.
While planning changes to the build system backward compatibility
with 20.11 should be considered.
AR to Bruce to create initial version of the DD.

Asia DPDK Event 
~~~~~~~~~~~~~~
Thomas provided quick update.
Few moderators (speaking mandarin) would be required.
 

Re: [dpdk-dev] Minutes of Technical Board Meeting, 2020-11-18

[Morten Brørup]

Bruce,

Here's my input as a developer of hardware appliances. It is my opinion, and as such may contradict the trend towards making DPDK a library, rather than a development kit.

> DPDK build configuration - future enhancements
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> There are multiple requests (sometimes controversial) for new abilities
> to add into DPDK build system.
> In particular, request from few different teams:
>   - add ability to enable/disable individual apps/libs
>   - override some build settings for specific libs/drivers

My wish list, in prioritized order:

1. The ability to remove features to reduce complexity - and thus the likelihood of bugs!

Remember to consider this in application context.

Background: Our previous firmware used the Linux kernel, and some loadable modules. We ran into a lot of extremely rare and unexpected cases where the Linux kernel network stack did something completely unusual, and our firmware needed to consider all these exceptional cases. This is one of the key reasons we switched to DPDK - the fast path libraries are clean and simple, and don't do anything we didn't ask them to do.

DPDK example: If support for segmented packets are considered "required" by DPDK libraries and drivers, is it also required for applications to support segmented packet? If the application doesn’t need segmented packets, can it safely assume that no DPDK libraries or drivers create segmented packets under any circumstances? If support for segmented packets is a compile time option, there is an implicit guarantee that they don't appear.

2. The ability to remove/tweak features to improve *application* performance in specific environments would be good.

E.g. removing support for multiple mbuf pools would free up an mbuf field (m->pool) for application use.
So would removing support for segmented packets (m->nb_segs, m->next).

Both of these modifications would also reduce complexity, although they would increase source code complexity in all the libraries and drivers needing to support a multidimensional matrix of features. (I highly doubt that all libraries support the combination of all features today... I remember having to argue strongly for the DPDK eBPF library to support reading data inside segmented packets.)

3. Removing cruft that has no effect on performance or similar, is "nice to have".

E.g. drivers for hardware that we do not use.

> As a first step to move forward - produce design doc of current build
> system.
> Discuss further enhancements based on that doc.

> While planning changes to the build system backward compatibility
> with 20.11 should be considered.

Backward compatibility is not a high priority for us. It is an extremely rare event for us to upgrade to a new version of any external software (Linux Kernel, DPDK and other libraries) or build tools, because we consider switching any of it to another version high effort (e.g. it requires extensive testing). In this perspective, having to change some details in the build system is a relatively small effort.

With this said, the documentation of each DPDK release should include a chapter describing what an application developer should do different than with the previous release. E.g. the Release Note enumerates the key modifications as bullet points, but it is not always obvious how that affects an application being developed. (DPDK generally has great documentation, but is somewhat lacking in this area.)

I know that ABI Stability is supposed to make much of this go away, but DPDK is clearly not there yet.

> AR to Bruce to create initial version of the DD.
> 

The following may be scope creep, so just consider it me thinking out loud:

Consider a general design documents in the form of a "life of an mbuf" document, describing how mbufs are pre-allocated for driver RX descriptors, and then handed over to the application trough the receive function, and then possibly going through defragmentation and reordering libraries, and then handed over to another driver's transmit function, which uses the mbufs to set up TX descriptors, and after transmission frees the mbufs to their original pool, where they are ultimately allocated again by a driver to refill its RX descriptor pool.

The document can start off with the simple case with a single non-segmented, non-fragmented, in-order packet. And then it can be extended with variations, e.g. adding the description of segmented packets would explain how the m->nb_segs and m->next are being used when the packet is handled by the drivers and libraries.

In the context of being able to enable/disable libraries and features, the purpose of this document would be to help showing interdependencies.


Med venlig hilsen / kind regards
- Morten Brørup

Re: [dpdk-dev] [dpdk-techboard] Minutes of Technical Board Meeting, 2020-11-18

[Thomas Monjalon]

23/11/2020 10:30, Morten Brørup:
> Bruce,
> 
> Here's my input as a developer of hardware appliances. It is my opinion, and as such may contradict the trend towards making DPDK a library, rather than a development kit.
> 
> > DPDK build configuration - future enhancements
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > There are multiple requests (sometimes controversial) for new abilities
> > to add into DPDK build system.
> > In particular, request from few different teams:
> >   - add ability to enable/disable individual apps/libs
> >   - override some build settings for specific libs/drivers
> 
> My wish list, in prioritized order:
> 
> 1. The ability to remove features to reduce complexity - and thus the likelihood of bugs!
> 
> Remember to consider this in application context.
> 
> Background: Our previous firmware used the Linux kernel, and some loadable modules. We ran into a lot of extremely rare and unexpected cases where the Linux kernel network stack did something completely unusual, and our firmware needed to consider all these exceptional cases. This is one of the key reasons we switched to DPDK - the fast path libraries are clean and simple, and don't do anything we didn't ask them to do.
> 
> DPDK example: If support for segmented packets are considered "required" by DPDK libraries and drivers, is it also required for applications to support segmented packet? If the application doesn’t need segmented packets, can it safely assume that no DPDK libraries or drivers create segmented packets under any circumstances? If support for segmented packets is a compile time option, there is an implicit guarantee that they don't appear.

The primary rule in DPDK is that the application remains in control.
If the application does not call the API function for a feature,
it won't be enabled. So no need to remove the unused libraries.


> 2. The ability to remove/tweak features to improve *application* performance in specific environments would be good.
> 
> E.g. removing support for multiple mbuf pools would free up an mbuf field (m->pool) for application use.
> So would removing support for segmented packets (m->nb_segs, m->next).
> 
> Both of these modifications would also reduce complexity, although they would increase source code complexity in all the libraries and drivers needing to support a multidimensional matrix of features. (I highly doubt that all libraries support the combination of all features today... I remember having to argue strongly for the DPDK eBPF library to support reading data inside segmented packets.)

Because code must remain simple, the mbuf layout is fixed
(except dynamic fields).


> 3. Removing cruft that has no effect on performance or similar, is "nice to have".
> 
> E.g. drivers for hardware that we do not use.
> 
> > As a first step to move forward - produce design doc of current build
> > system.
> > Discuss further enhancements based on that doc.
> 
> > While planning changes to the build system backward compatibility
> > with 20.11 should be considered.
> 
> Backward compatibility is not a high priority for us. It is an extremely rare event for us to upgrade to a new version of any external software (Linux Kernel, DPDK and other libraries) or build tools, because we consider switching any of it to another version high effort (e.g. it requires extensive testing). In this perspective, having to change some details in the build system is a relatively small effort.
> 
> With this said, the documentation of each DPDK release should include a chapter describing what an application developer should do different than with the previous release. E.g. the Release Note enumerates the key modifications as bullet points, but it is not always obvious how that affects an application being developed. (DPDK generally has great documentation, but is somewhat lacking in this area.)
> 
> I know that ABI Stability is supposed to make much of this go away, but DPDK is clearly not there yet.
> 
> > AR to Bruce to create initial version of the DD.
> > 
> 
> The following may be scope creep, so just consider it me thinking out loud:
> 
> Consider a general design documents in the form of a "life of an mbuf" document, describing how mbufs are pre-allocated for driver RX descriptors, and then handed over to the application trough the receive function, and then possibly going through defragmentation and reordering libraries, and then handed over to another driver's transmit function, which uses the mbufs to set up TX descriptors, and after transmission frees the mbufs to their original pool, where they are ultimately allocated again by a driver to refill its RX descriptor pool.
> 
> The document can start off with the simple case with a single non-segmented, non-fragmented, in-order packet. And then it can be extended with variations, e.g. adding the description of segmented packets would explain how the m->nb_segs and m->next are being used when the packet is handled by the drivers and libraries.
> 
> In the context of being able to enable/disable libraries and features, the purpose of this document would be to help showing interdependencies.

I agree we need this kind of doc.
It could be part of the prog guide.
Feel free to draft a skeleton.

Re: [dpdk-dev] [dpdk-techboard] Minutes of Technical Board Meeting, 2020-11-18

[Morten Brørup]

> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Monday, November 23, 2020 11:00 AM
> 
> 23/11/2020 10:30, Morten Brørup:
> > Bruce,
> >
> > Here's my input as a developer of hardware appliances. It is my
> opinion, and as such may contradict the trend towards making DPDK a
> library, rather than a development kit.
> >
> > > DPDK build configuration - future enhancements
> > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > There are multiple requests (sometimes controversial) for new
> abilities
> > > to add into DPDK build system.
> > > In particular, request from few different teams:
> > >   - add ability to enable/disable individual apps/libs
> > >   - override some build settings for specific libs/drivers
> >
> > My wish list, in prioritized order:
> >
> > 1. The ability to remove features to reduce complexity - and thus the
> likelihood of bugs!
> >
> > Remember to consider this in application context.
> >
> > Background: Our previous firmware used the Linux kernel, and some
> loadable modules. We ran into a lot of extremely rare and unexpected
> cases where the Linux kernel network stack did something completely
> unusual, and our firmware needed to consider all these exceptional
> cases. This is one of the key reasons we switched to DPDK - the fast
> path libraries are clean and simple, and don't do anything we didn't
> ask them to do.
> >
> > DPDK example: If support for segmented packets are considered
> "required" by DPDK libraries and drivers, is it also required for
> applications to support segmented packet? If the application doesn’t
> need segmented packets, can it safely assume that no DPDK libraries or
> drivers create segmented packets under any circumstances? If support
> for segmented packets is a compile time option, there is an implicit
> guarantee that they don't appear.
> 
> The primary rule in DPDK is that the application remains in control.
> If the application does not call the API function for a feature,
> it won't be enabled. So no need to remove the unused libraries.

I think that this principle - the application remaining in control - is extremely important for DPDK, and we must always remember this principle when adding features to DPDK.

However, being able to disable some features at compile time elevates the certainty that these features are not being unexpectedly used from "trust" to "absolute certainty".

The DPDK core and libraries are growing in complexity, and I am starting to worry about this. Once bitten twice shy.

By the way, I consider the Dynamic MBUF concept a great enhancement in this area. The cleanup part of the Dynamic MBUF patch set made non-essential fields in the mbuf truly optional.

> 
> 
> > 2. The ability to remove/tweak features to improve *application*
> performance in specific environments would be good.
> >
> > E.g. removing support for multiple mbuf pools would free up an mbuf
> field (m->pool) for application use.
> > So would removing support for segmented packets (m->nb_segs, m-
> >next).
> >
> > Both of these modifications would also reduce complexity, although
> they would increase source code complexity in all the libraries and
> drivers needing to support a multidimensional matrix of features. (I
> highly doubt that all libraries support the combination of all features
> today... I remember having to argue strongly for the DPDK eBPF library
> to support reading data inside segmented packets.)
> 
> Because code must remain simple, the mbuf layout is fixed
> (except dynamic fields).

The mbuf layout could remain fixed (so vector implementations can rely on the layout), but the removed fields would become unused and available for application use instead, thus improving the application performance.

In the example of removing support for multiple mbuf pools, the functions free()'ing mbufs in the DPDK mbuf library and the DPDK drivers would be simpler, thus improving the performance. Removing support for segmented packets would also allow simpler (and thus higher performing) implementations of a few DPDK core functions.

It is somewhat difficult to formulate in writing, but I will try rephrasing my original point: Tweaking DPDK can provide performance improvements in the application itself, not only in the DPDK libraries/drivers.

> 
> 
> > 3. Removing cruft that has no effect on performance or similar, is
> "nice to have".
> >
> > E.g. drivers for hardware that we do not use.
> >
> > > As a first step to move forward - produce design doc of current
> build
> > > system.
> > > Discuss further enhancements based on that doc.
> >
> > > While planning changes to the build system backward compatibility
> > > with 20.11 should be considered.
> >
> > Backward compatibility is not a high priority for us. It is an
> extremely rare event for us to upgrade to a new version of any external
> software (Linux Kernel, DPDK and other libraries) or build tools,
> because we consider switching any of it to another version high effort
> (e.g. it requires extensive testing). In this perspective, having to
> change some details in the build system is a relatively small effort.
> >
> > With this said, the documentation of each DPDK release should include
> a chapter describing what an application developer should do different
> than with the previous release. E.g. the Release Note enumerates the
> key modifications as bullet points, but it is not always obvious how
> that affects an application being developed. (DPDK generally has great
> documentation, but is somewhat lacking in this area.)
> >
> > I know that ABI Stability is supposed to make much of this go away,
> but DPDK is clearly not there yet.
> >
> > > AR to Bruce to create initial version of the DD.
> > >
> >
> > The following may be scope creep, so just consider it me thinking out
> loud:
> >
> > Consider a general design documents in the form of a "life of an
> mbuf" document, describing how mbufs are pre-allocated for driver RX
> descriptors, and then handed over to the application trough the receive
> function, and then possibly going through defragmentation and
> reordering libraries, and then handed over to another driver's transmit
> function, which uses the mbufs to set up TX descriptors, and after
> transmission frees the mbufs to their original pool, where they are
> ultimately allocated again by a driver to refill its RX descriptor
> pool.
> >
> > The document can start off with the simple case with a single non-
> segmented, non-fragmented, in-order packet. And then it can be extended
> with variations, e.g. adding the description of segmented packets would
> explain how the m->nb_segs and m->next are being used when the packet
> is handled by the drivers and libraries.
> >
> > In the context of being able to enable/disable libraries and
> features, the purpose of this document would be to help showing
> interdependencies.
> 
> I agree we need this kind of doc.
> It could be part of the prog guide.
> Feel free to draft a skeleton.
> 
> 
>