Re: [dpdk-dev] [RFC PATCH 0/3] introduce IF proxy library

["Morten Brørup" <mb@smartsharesystems.com>]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jerin Jacob
> Sent: Wednesday, January 15, 2020 5:04 PM
> 
> On Wed, Jan 15, 2020 at 9:00 PM Morten Brørup <mb@smartsharesystems.com>
> wrote:
> >
> > > -----Original Message-----
> > > From: Jerin Jacob [mailto:jerinjacobk@gmail.com]
> > > Sent: Wednesday, January 15, 2020 1:57 PM
> > >
> > > On Wed, Jan 15, 2020 at 5:58 PM Morten Brørup
> > > <mb@smartsharesystems.com> wrote:
> > > >
> > > > > -----Original Message-----
> > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce
> > > Richardson
> > > > > Sent: Wednesday, January 15, 2020 11:16 AM
> > > > >
> > > > > On Tue, Jan 14, 2020 at 06:38:37PM +0100, Andrzej Ostruszka wrote:
> > > > > > On 1/14/20 4:16 PM, Morten Brørup wrote:
> > > > > > > Andrzej,
> > > > > >
> > > > > > Hello Morten
> > > > > >
> > > > > > > Basically you are adding a very small subset of the Linux IP
> > > stack>
> > > > > to interface with DPDK applications via callbacks.
> > > > > >
> > > > > > Yes, at the moment this is limited - we'd prefer first to solicit
> > > > > > some input from community.
> > > > > >
> > > > > > > The library also seems to support interfacing to the route
> > > table,
> > > > > > > so it is not "interface proxy" but "IP stack proxy".
> > > > > >
> > > > > > True, to some extent - for example you can bring the interface up
> > > and
> > > > > > down which has nothing to do with IP stack.  As for the name of
> > > the
> > > > > > library - that is actually part where we are completely open.
> > > The
> > > > > proxy
> > > > > > represents port (thus the name) but that is not all, so any
> > > better
> > > > > name
> > > > > > proposals are welcome.
> > > > > >
> > > > > > > You already mention ARP table as future work. How about
> > > namespaces,
> > > > > > > ip tables, and other advanced features... I foresee the Devil
> > > in
> > > > > the
> > > > > > > details for any real use case.
> > > > > >
> > > > > > Right now I don't know what other things are needed.  This idea
> > > is
> > > > > still
> > > > > > early.  However imagine you'd like to use DPDK to speed up packet
> > > > > > processing of IP stack - would you like to implement all the
> > > > > protocols
> > > > > > that are needed?  Or just let the system handle the control path
> > > and
> > > > > > handle the data path and sniff the control params from the
> > > system.
> > > > > >
> > > > > Like Morten, I'd be a bit concerned at the possible scope of the
> > > work
> > > > > if we
> > > > > start pulling in functionality from the IP stack like ARP etc. To
> > > avoid
> > > > > this becoming a massive effort, how useful would it be if we just
> > > > > limited
> > > > > the scope to physical NIC setup only, and did not do anything above
> > > the
> > > > > l2
> > > > > layer?
> > > >
> > > > Think about it... Regardless of scope, this is clearly a control
> > > plane API, not a data plane API.
> > > >
> > > > It provides a proxy API for the O/S control plane (NETLINK in the
> > > case of Linux), so the DPDK application can use the user interface that
> > > the O/S already provides (e.g. "ip link set dev tap1 mtu 1600" etc.)
> > > for its control plane, instead of implementing its own CLI (or GUI or
> > > whatever).
> > >
> > > Yes.
> > >
> > > >
> > > > In order to provide significant value, it will have to grow
> > > massively, so I can use it as imagined: To make a Linux firewall where
> > > the DPDK application handles the data plane, and the normal Linux
> > > commands are used for setting up the firewall, incl. firewall rules,
> > > port forwarding, NAPT, etc.. The Devil is in the details here!
> > >
> > > Yes.
> > > Another use case would be to handle exception where DPDK may not
> > > handle all the traffic, Traffic such ARP can be redirected to OS. This
> > > would enable DP to focus on the real fast path protocols such as IPv4,
> > > UDP etc.
> > >
> >
> > These are use cases for DPDK being used in an environment where the IP
> stack features provided by Linux suffices. It would be great for a simple
> CPE or Wi-Fi router, e.g. OpenWRT with a DPDK data plane replacing the
> Linux kernel's data plane.
> 
> IMO, it not replacing the Linux IP stack, instead, using the slow path
> services from Linux or any OS. The use case would vary from simple
> WiFI router to 5G transport stack.

I only mentioned the special case where a DPDK application uses the Linux slow path services (though your proxy API) and the DPDK application handles the data plane instead of the packets being handled by the Linux kernel's data plane. But I agree, the concept is broader than that.

> 
> >
> > For this use case, I think an example application would be a much more
> useful way to achieve your goal. Implementing it as an application will
> also uncover what is really needed, instead of us all speculating about
> what a proxy library might need to include.
> >
> > But consider an advanced router with VRFs, VLANs, policy based routing,
> multiple WANs provided through network namespaces... the library will be
> huge!
> 
> We thought of adding the infrastructure and the need per basics, we
> can scale it up. There is no such infrastructure now with DPDK.
> At least if someone wishes to contribute to this these area then there
> should be the path to improve things wrt current situation.
> 
> >
> > > >
> > > > Although I like the concept and idea behind it, I don't think a
> > > control plane proxy API belongs in DPDK. But it could possibly be
> > > hosted by the DPDK project if approved as such.
> > >
> > > Why? rte_flow, rte_tm all control plane APIs and it is part of
> > > DPDK.
> >
> > Yes, there are some DPDK libraries leaning more towards control plane
> than data plane. Another example to prove your point: The whole process
> scheduling library has very little to do with packet processing. Vaguely
> related features are creeping in when objections are not strong enough.
> 
> Yes. That's the reason for the control path vs data path argument that
> doesn't have any value.
> If it is useful for packet processing use cases then have it.
> 
> >
> > > IMO, in order to have effective use of data plane, the control
> > > plane has to be integrated together in an OS-independent way.
> > >
> >
> > Also remember that not all DPDK applications need an IP stack resembling
> what Linux has. E.g. the SmartShare StraightShaper is a transparent
> bandwidth optimization appliance, and it doesn't perform any routing, it
> doesn't use any O/S-like features in the data path, and thus it doesn't
> need to integrate with the IP stack in the O/S. (The management interface
> uses the Linux IP stack, but it is completely isolated from the DPDK
> application's data plane.) The same can be said about e.g. T-Rex.
> >
> > Obviously, not all DPDK applications use all DPDK libraries, and since
> I'm not obligated to use it, I'm not strongly opposed against it. I only
> question its usefulness outside of the specific use case of replacing the
> fast path in the Linux kernel.
> 
> Of course, We still follow the "À la carte" model, where we are not
> forcing to use the library in the end-user application. You can always
> use whatever control path that makes sense with the end-user
> applications.
> But if some application wants to write control plane SW that needs to
> work Linux/FreeBSD/Windows or other RTOS then it can be used (Again if
> someone wishes to do so).
> We can also provide the means for NOPing out the callbacks or override
> with something it is the specific end-user library as well, so that
> complete flexibly will be still with the application wrt the usage.
> 

OK, you convinced me that a general API for interfacing to the O/S control plane might be useful. So let me switch from arguing against it to providing some constructive feedback:

You should consider that most DPDK APIs are not thread safe, meaning that their internal structures cannot be manipulated/reconfigured by a control plane thread while data plane threads are accessing them. E.g. a route cannot be added in the DPDK route library while it is also being used by for lookups by a DPDK data plane thread. The same goes for the hash table library. This means that callbacks are probably not the right design pattern.

AFAIK, the DPDK documentation doesn't mention any "best practices" for interaction between the control plane and data plans threads, so I understand why you chose a design pattern similar to the NIC Link Status Change interrupt design pattern.

Furthermore, I have now skimmed the other parts of your patch set. If I got it right, it looks like there's a limit of 64 callbacks; this will probably not suffice in the long run.

And on the administrative side, I assume one of you guys will volunteer as the maintainer of this library?

-Morten