Re: [dpdk-dev] [PATCH v2 0/4] Introduce IF proxy library

[Thomas Monjalon <thomas@monjalon.net>]

Hi Andrzej,

Thanks for the very good explanations in the cover letter.
I have several comments and questions about the design.
I think IF proxy is a good idea which should be part of a bigger plan.


10/03/2020 12:10, Andrzej Ostruszka:
> What is this useful for
> =======================
> 
> Usually, when an ethernet port is assigned to DPDK it vanishes from the
> system and user looses ability to control it via normal configuration
> utilities (e.g. those from iproute2 package).  Moreover by default DPDK
> application is not aware of the network configuration of the system.
> 
> To address both of these issues application needs to:
> - add some command line interface (or other mechanism) allowing for
>   control of the port and its configuration
> - query the status of network configuration and monitor its changes
> 
> The purpose of this library is to help with both of these tasks (as long
> as they remain in domain of configuration available to the system).  In
> other words, if DPDK application has some special needs, that cannot be
> addressed by the normal system configuration utilities, then they need
> to be solved by the application itself.

In any case, the application must be in the loop.
The application should always remain in control.
When querying some information, nothing need to be controlled I guess.
But when adjusting some configuration, the application must be able
to be notified and decide which change is allowed.
Of course, the application might allow being bypassed.

Currently this rule is not respected in the rte_mp IPC system.
I think rte_mp and IF proxy should follow the same path,
keeping the primary application process in control.

I would like not only secondary process and IF proxy be able to use
this control path. It should be generic enough to allow any application
(local or remote) be part of the control path, communicating with
the DPDK application primary process.

As a summary, I propose to target the following goal:
implement a user configuration path as a DPDK standard
that the application can enable.

Do we agree that the exception packet path is out of scope?


[...]
> We create two proxy interfaces (here based on Tap driver) and bind the
> ports to their proxies.  When user issues a command changing MTU for
> Tap1 interface the library notes this and calls "mtu_change" callback
> for the Port1.  Similarly when user adds an IPv4 address to the Tap2
> interface "addr_add" callback is called for the Port2 and the same
> happens for configuration of routing rule pointing to Tap2.

Will it work as well with TC flow configuration converted to rte_flow?


> Apart from
> callbacks this library can notify about changes via adding events to
> notification queues.  See below for more inforamtion about that and
> a complete list of available callbacks.

There is choice between callback in a random context,
or a read from a message queue in a controlled context.
Second option looks better.


> Please note that nothing has been mentioned about forwarding of the
> packets between system and DPDK.  Since the proxies are normal DPDK
> ports you can receive/send to them via usual RX/TX burst API.  However
> since the library is not aware of the structure of packet processing
> used by the application it cannot automatically forward the packets - it
> is responsibility of the application to include proxy ports into its
> packet processing engine.

So IF proxy does nothing special with packets, right?


> As mentioned above the intention of the library is to:
> - provide information about network configuration that would allow
>   application to decide what to do with the packets received on DPDK
>   ports,
> - allow for control of the ports via standard configuration utilities
> 
> Although the library only helps you to identify proxy for given port
> (and vice versa) and calls appropriate callbacks it does open some
> interesting possibilities.  For example you can use the proxy ports to
> forward packets for protocols that you do not wish to handle in DPDK
> application to the system protocol stack and just listen to the
> configuration changes - so that way you can "offload" handling of those
> protocols to the system.

Note that when using a bifurcated driver (af_xdp or mlx),
the exception path in the kernel is not going through DPDK.
Moreover, no proxy is needed for device configuration in such case.


[...]
> The only mandatory requirement for DPDK port to be able to act as
> a proxy is that it is visible in the system - this is checked during
> port to proxy binding by calling rte_eth_dev_info_get() on proxy port
> and inspecting 'if_index' field (it has to be non-zero).

Simple, good :)


> This creates logical binding - as mentioned above there is no automatic
> packet forwarding.  With this binding whenever user changes the state of
> proxy interface in the system (link up/down, change mac/mtu, add/remove
> IPv4/IPv6) you get appropriate notification for the bound port.

When configuring a port via DPDK API, is it mirrored automatically
to the kernel device?


> So far we've mentioned several times that the library calls callbacks.
> They are grouped in 'struct rte_ifpx_callbacks' and user provides them
> to the library via:
> 
>   rte_ifpx_callbacks_register(&cbs);
> 
> It is worth mentioning that the context (lcore/thread) in which these
> callbacks are called is implementation defined.  It might differ between
> different platforms, so the application needs to assume that some kind
> of inter lcore/thread synchronization/communication is required.
> 
> Apart from notification via callbacks this library also supports
> notifying about the changes via adding events to the configured
> notification queues.  The queues are registered via:
> 
>   int rte_ifpx_queue_add(struct rte_ring *r);
> 
> and the actual logic used is: if there is callback registered then it is
> called, if it returns non-zero then event is considered completed,
> otherwise event is added to each configured notification queue.
> That way application can update data structures that are safe to be
> modified by single writer from within callback or do the common
> preprocessing steps (if any needed) in callback and data that is
> replicated can be updated during handling of queued events.

As explained above, the application must control every changes.

One issue is thread safety.
The simplest model is to manage control path from a single thread
in the primary process.

If we create an API to allow the application managing the control path
from external requests, I think it should be a building block
independent of IF proxy. Then IF proxy can plug into this subsystem.
It would allow other control path mechanisms to co-exist.


[...]
> It is worth to mention also that while typical case would be a 1-to-1
> mapping between port and proxy, the 1-to-many mapping is also supported.
> In that case related callbacks will be called for each port bound to
> given proxy interface - it is application responsibility to define
> semantic of such mapping (e.g. all changes apply to all ports, or link
> changes apply to all but other are accepted in "round robin" fashion, or
> some other logic).

I don't get the interest of one-to-many mapping.


[...]

Thanks for the work.
It seems there are some overlaps with telemetry and rte_mp channels.
The same channel could be used also for dynamic tracing command
or for remote control.
Would you be OK to extend it to a global control subsystem,
having IF proxy plugged in?