DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] [RFC PATCH] ethdev: clarify flow attribute and action port ID semantics
@ 2021-09-07 12:51 Ivan Malov
  2021-10-01 13:47 ` [dpdk-dev] [PATCH v1 00/12] ethdev: rework transfer flow API Andrew Rybchenko
  2021-10-05  0:36 ` [dpdk-dev] [PATCH] ethdev: let apps find transfer admin port for a given ethdev Ivan Malov
  0 siblings, 2 replies; 161+ messages in thread
From: Ivan Malov @ 2021-09-07 12:51 UTC (permalink / raw)
  To: dev; +Cc: Ori Kam, Thomas Monjalon, Ferruh Yigit, Andrew Rybchenko

Problems:

1) Existing item PORT_ID and action PORT_ID are ambiguous because
   one can consider the port in question as either an ethdev or
   an embedded switch entity wired to it, as per the use case,
   which is not expressed clearly in code and documentation.

2) Attributes "ingress" and "egress" may not make sense in flows
   with "transfer" attribute for some PMDs. Furthermore, such
   PMDs may face a related problem (see below).

3) A PMD may not be able to handle "transfer" rules on all ethdevs
   it serves. It may have only one (admin) ethdev capable of that.
   Applications should be able to take this into account and
   submit "transfer" rules on that specific ethdev. However,
   meaning of attributes "ingress" and "egress" might be
   skewed in this case as the ethdev used to make flow
   API calls is just a technical entry point.

In order to solve problem (1)¸ one should recognise the existence
of two major application models: traffic consumer / generator and
a vSwitch / forwarder. To the latter, ethdevs used to perform
Rx / Tx burst calls are simply vSwitch ports. Requesting HW
offloads on these ports implies referring to e-switch ports
that correspond to them at the lowest, e-switch, level.
This way, suggested terminology is clear and concise.

The patch suggests using item / action PORT_ID sub-variants to
disambiguate the meaning. In order to avoid breaking existing
behaviour in Open vSwitch DPDK offload, the sub-variant for
e-switch port is declared with enum value of zero.

In order to solve problems (2) and (3), one needs to recognise
the existence of two paradigms of handling "transfer" rules in
PMDs. If the given PMD needs to "proxy" handling of "transfer"
rules via an admin ethdev, this must not be done implicitly
as the application must know the true ethdev responsible
for handling the flows in order to avoid detaching it
before all "transfer" rules get properly dismantled.

The patch suggests to use a generic helper to let applications
discover the paradigm in use and, if need be, communicate
flow rules through the discovered "proxy" ethdev.

Signed-off-by: Ivan Malov <ivan.malov@oktetlabs.ru>
---
This proposal is an alternative to previously suggested RFCs, [1] and [2].
It covers several related problems and suggests clear API contract to
let vSwitch applications use item PORT_ID and action PORT_ID to refer
to e-switch ports thus preserving existing sense.

[1] https://patches.dpdk.org/project/dpdk/patch/20210601111420.5549-1-ivan.malov@oktetlabs.ru/
[2] https://patches.dpdk.org/project/dpdk/patch/20210903074610.313622-1-andrew.rybchenko@oktetlabs.ru/
---
 lib/ethdev/rte_flow.h | 140 ++++++++++++++++++++++++++++++++----------
 1 file changed, 107 insertions(+), 33 deletions(-)

diff --git a/lib/ethdev/rte_flow.h b/lib/ethdev/rte_flow.h
index 70f455d47d..bf2b5e752c 100644
--- a/lib/ethdev/rte_flow.h
+++ b/lib/ethdev/rte_flow.h
@@ -82,22 +82,32 @@ struct rte_flow_attr {
 	uint32_t ingress:1; /**< Rule applies to ingress traffic. */
 	uint32_t egress:1; /**< Rule applies to egress traffic. */
 	/**
-	 * Instead of simply matching the properties of traffic as it would
-	 * appear on a given DPDK port ID, enabling this attribute transfers
-	 * a flow rule to the lowest possible level of any device endpoints
-	 * found in the pattern.
-	 *
-	 * When supported, this effectively enables an application to
-	 * re-route traffic not necessarily intended for it (e.g. coming
-	 * from or addressed to different physical ports, VFs or
-	 * applications) at the device level.
-	 *
-	 * It complements the behavior of some pattern items such as
-	 * RTE_FLOW_ITEM_TYPE_PHY_PORT and is meaningless without them.
-	 *
-	 * When transferring flow rules, ingress and egress attributes keep
-	 * their original meaning, as if processing traffic emitted or
-	 * received by the application.
+	 * This "transfers" the rule from the ethdev level to the embedded
+	 * switch (e-switch) level, where it's possible to match traffic
+	 * not necessarily going to the ethdev where the flow is created
+	 * and redirect it to endpoints otherwise not necessarily
+	 * accessible from rules having no such attribute.
+	 *
+	 * Applications willing to use attribute "transfer" should detect its
+	 * paradigm implemented inside the PMD. The paradigms are as follows:
+	 *
+	 * - The PMD supports handling "transfer" flow rules on any ethdevs
+	 *   it serves. With this paradigm, rte_flow_pick_transfer_proxy()
+	 *   call returns (-ENOTSUP) for all ethdevs backed by the PMD.
+	 *   Attributes "ingress" and "egress" are valid and preserve
+	 *   their original meaning, at application standpoint. Also,
+	 *   these attributes typically set some implicit filtering.
+	 *
+	 * - The PMD only supports handling "transfer" flow rules on some
+	 *   specific ethdev pointed out by rte_flow_pick_transfer_proxy().
+	 *   Typically, it's an admin PF ethdev backing a group of VF
+	 *   representor ethdevs. In this case, attributes "ingress"
+	 *   and "egress" cannot maintain their original meaning as
+	 *   the ethdev used to handle "transfer" flow rules is
+	 *   just a technical entry point and does not mean any
+	 *   implicit filtering. Attribute "egress" is rejected,
+	 *   and "ingress" (redundant) means traffic ingressing
+	 *   the embedded switch from any of its endpoints.
 	 */
 	uint32_t transfer:1;
 	uint32_t reserved:29; /**< Reserved, must be zero. */
@@ -191,8 +201,9 @@ enum rte_flow_item_type {
 	/**
 	 * [META]
 	 *
-	 * Matches traffic originating from (ingress) or going to (egress) a
-	 * given DPDK port ID.
+	 * Matches traffic originating from (ingress) or going to (egress) the
+	 * port which, depending on the item sub-variant, is the given ethdev
+	 * or the opposite end of the "wire" attached to this ethdev.
 	 *
 	 * See struct rte_flow_item_port_id.
 	 */
@@ -679,29 +690,61 @@ static const struct rte_flow_item_phy_port rte_flow_item_phy_port_mask = {
 };
 #endif
 
+/** Port types for use with item PORT_ID and action PORT_ID */
+enum rte_flow_item_port_id_type {
+	/**
+	 * The port in question is an embedded switch entity connected
+	 * to or represented by the given ethdev / vSwitch port.
+	 *
+	 * +--------------------------+---------------------------+
+	 * |  Ethdev / vSwitch Port   |   Embedded Switch Entity  |
+	 * +--------------------------+---------------------------+
+	 * |         PF / VF         <->       Network Port       |
+	 * +------------------------------------------------------+
+	 * |   PF / VF Representor   <->      PF / VF Itself      |
+	 * +------------------------------------------------------+
+	 */
+	RTE_FLOW_PORT_TYPE_ESWITCH = 0,
+
+	/**
+	 * The port in question is an ethdev or, synonymously,
+	 * a DPDK-backed vSwitch port.
+	 */
+	RTE_FLOW_PORT_TYPE_ETHDEV,
+};
+
 /**
  * RTE_FLOW_ITEM_TYPE_PORT_ID
  *
- * Matches traffic originating from (ingress) or going to (egress) a given
- * DPDK port ID.
+ * Matches traffic originating from (ingress) or going to (egress) the
+ * port which, depending on the item sub-variant, is the given ethdev
+ * or the opposite end of the "wire" attached to this ethdev.
  *
- * Normally only supported if the port ID in question is known by the
- * underlying PMD and related to the device the flow rule is created
- * against.
+ * Typically, the ethdev referring to the port in question must be served
+ * by the same PMD as that of the ethdev used to create the flow rule.
+ * Also, the port must normally belong to the same physical board.
  *
- * This must not be confused with @p PHY_PORT which refers to the physical
- * port of a device, whereas @p PORT_ID refers to a struct rte_eth_dev
- * object on the application side (also known as "port representor"
- * depending on the kind of underlying device).
+ * This must not be confused with item @p PHY_PORT
+ * which refers specifically to a physical port.
  */
+RTE_STD_C11
 struct rte_flow_item_port_id {
-	uint32_t id; /**< DPDK port ID. */
+	/** Synonymous defines for ethdev ID property */
+	union {
+		/** Ethdev (vSwitch port) ID */
+		uint32_t ethdev_id;
+		/** Compatibility alias to avoid breaking legacy applications */
+		uint32_t id;
+	};
+
+	/** Port type (item sub-variant) */
+	enum rte_flow_item_port_id_type type;
 };
 
 /** Default mask for RTE_FLOW_ITEM_TYPE_PORT_ID. */
 #ifndef __cplusplus
 static const struct rte_flow_item_port_id rte_flow_item_port_id_mask = {
-	.id = 0xffffffff,
+	.ethdev_id = 0xffffffff,
 };
 #endif
 
@@ -1976,7 +2019,8 @@ enum rte_flow_action_type {
 	RTE_FLOW_ACTION_TYPE_PHY_PORT,
 
 	/**
-	 * Directs matching traffic to a given DPDK port ID.
+	 * Depending on the port type, directs matching traffic either to the
+	 * given ethdev or to the opposite end of the "wire" attached to it.
 	 *
 	 * See struct rte_flow_action_port_id.
 	 */
@@ -2635,14 +2679,26 @@ struct rte_flow_action_phy_port {
 /**
  * RTE_FLOW_ACTION_TYPE_PORT_ID
  *
- * Directs matching traffic to a given DPDK port ID.
+ * Depending on the port type, directs matching traffic either to the
+ * given ethdev or to the opposite end of the "wire" attached to it.
  *
  * @see RTE_FLOW_ITEM_TYPE_PORT_ID
  */
+RTE_STD_C11
 struct rte_flow_action_port_id {
-	uint32_t original:1; /**< Use original DPDK port ID if possible. */
+	uint32_t original:1; /**< Use original ethdev ID if possible. */
 	uint32_t reserved:31; /**< Reserved, must be zero. */
-	uint32_t id; /**< DPDK port ID. */
+
+	/** Synonymous defines for ethdev ID property */
+	union {
+		/** Ethdev (vSwitch port) ID */
+		uint32_t ethdev_id;
+		/** Compatibility alias to avoid breaking legacy applications */
+		uint32_t id;
+	};
+
+	/** Port type (action sub-variant) */
+	enum rte_flow_item_port_id_type type;
 };
 
 /**
@@ -4288,6 +4344,24 @@ rte_flow_tunnel_item_release(uint16_t port_id,
 			     struct rte_flow_item *items,
 			     uint32_t num_of_items,
 			     struct rte_flow_error *error);
+
+/**
+ * Locate the "proxy" ethdev to handle "transfer" flow rules
+ * for the given ethdev. If the API returns (-ENOTSUP), the
+ * caller should assume that no "proxying" is required.
+ *
+ * @param port_id
+ *   Ethdev ID that potentially needs a "proxy"
+ * @param[out] proxy_port_id
+ *   The "proxy" port through which "transfer" rules must be communicated
+ *
+ * @return
+ *   0 on success, a negative error code otherwise
+ */
+__rte_experimental
+int
+rte_flow_pick_transfer_proxy(uint16_t port_id,
+			     uint16_t *proxy_port_id);
 #ifdef __cplusplus
 }
 #endif
-- 
2.20.1


^ permalink raw reply	[flat|nested] 161+ messages in thread
* Re: [dpdk-dev] [PATCH v1 02/12] ethdev: add eswitch port item to flow API
@ 2021-10-06 15:30 Ivan Malov
  2021-10-06 18:00 ` Ivan Malov
  0 siblings, 1 reply; 161+ messages in thread
From: Ivan Malov @ 2021-10-06 15:30 UTC (permalink / raw)
  To: Ori Kam, Andrew Rybchenko, dev; +Cc: Thomas Monjalon

Hi Ori,

By the looks of it, we are starting to run into slight misunderstanding.

As I see it, the main consequence of our Sep 14 gathering in Jitsi was
understanding of the fact that the concept of item / action PORT_ID is
vague and needs a replacement. As a bare minimum, separate items
should be used: one for an ethdev port and another one for
the "represented entity".

This "represented entity" can be network (via network port), or a guest
machine (via a PF / VF plugged to it), or another ethdev (in the case
when the ethdev we are looking at is a PF/VF representor and this
PF/VF is also plugged to the DPDK application).

So, if I get this right, you don't object this summary. Very well.

But, in the current approach, we stick with term "ESWITCH_PORT" for
that "represented entity", and, as I see it, this term is not quite
descriptive when someone tries to understand which exact port of
the embedded switch is implied. Attempts to clarify it by virtue
of terms "external" or "the most remote" are not-so-successful.

I fully understand that.

But the good news is that the original diagram of the new concept can
be improved in a way that can allow to dispose of misleading words.


      [ A ]       <-- ethdev
        |
      [ B ]       <-- embedded switch (logical) port
        |
        |
        |
===============  <-- plane of symmetry
        |
        |
        |
      [ C ]       <-- embedded switch (logical) port
        |
      [ D ]       <-- represented entity


1. The application sees the ethdev (A) and can assume that this
    ethdev is served by some logical port (B) in the embedded
    switch. Precise nature of this port is vendor-specific.

    For example, for a regular (non-representor) ethdev,
    this port can be a PF, or a VF. This is obvious to
    DPDK developers, but the application doesn't need
    to have this knowledge. It only sees the ethdev.

    If this ethdev is a representor, port (B) can be a truly
    separate logical port or, alternatively, some vendors
    may use "PF + metadata" approach. This port (B) is
    just assumed to exist. The rest is vendor-specific.

2. The application fully understands that the "wire" plugged to
    the ethdev it sees has an opposite end. Over there, there's
    some "represented entity" (D). Once again, the application
    does not know the nature of that "represented entity".
    To it, this entity is just some traffic endpoint.

    And the application understands that this "represented entity"
    is connected to the NIC by means of another logical port (C).
    The nature of this port is unknown. The application does not
    need to have this knowledge. To it, this port just exists.

    Examples of precise nature of (C) / (D):
    -- (D) = network;        (C) = network port
    -- (D) = guest machine;  (C) = VF / PF passed to the VM
    -- (D) = another ethdev; (C) = the ethdev's logical port

    For "the ethdev's logical port" - see the explanation in (1).

3. The pair of points (A), (B) is SYMMETRICAL to the pair
    of points (C), (D). With respect to any given ethdev,
    the embedded switch is split by a plane of symmetry.

4. The property of symmetry in this drawing allows to perceive
    point (D) as a reflection of point (A). A mirrored image.


So, the short of it, looking at the problem the way this picture
looks at it, one can easily understand that the "represented
entity" is just a reflection of the ethdev (and vice versa).

No more need for wordy descriptions like "the most remote port".
The symmetry says it all. But the problem is the name for the
new item and action. Instead of ESWITCH_PORT, something else
should be suggested to make readers think of this symmetry.

To me, the most clear term here would be MIRROR_PORT. As per the rules
of English language, "MIRROR" here is like an adjective. It describes
the property of the "PORT" being a reflection of the corresponding
ethdev. And I'd call the latter an ETHDEV_PORT.

So, ETHDEV_PORT and its MIRROR_PORT.

Yes-yes, I do understand that "mirror" may at some point turn
misleading to some readers who may perceive this word as a
verb and confuse the item / action meaning with the concept
of port mirroring. Alright: I'm aware of that. So let me
then suggest another option: REFLEX_PORT.

REFLEX_PORT doesn't seem to sound as good as MIRROR_PORT,
but it also indicates the property of this entity being
a reflection of the ethdev port (and vice versa).

What do you think?  Does it make sense to make another version of
our patch series with the new term in use and with the new
explanation? Should you wish to suggest your own naming
variants, you're welcome to do so.

On 05/10/2021 15:12, Ivan Malov wrote:
 > Hi Ori, Andrew,
 >
 > On 05/10/2021 12:19, Andrew Rybchenko wrote:
 > > Hi Ori,
 > >
 > > On 10/5/21 9:20 AM, Ori Kam wrote:
 > >> Hi Ivan,
 > >>
 > >>> -----Original Message-----
 > >>> From: Ivan Malov <Ivan.Malov at oktetlabs.ru>
 > >>> Cc: dev at dpdk.org
 > >>> Subject: Re: [PATCH v1 02/12] ethdev: add eswitch port item to 
flow API
 > >>>
 > >>> Hi Ori,
 > >>>
 > >>> On 04/10/2021 14:37, Ori Kam wrote:
 > >>>> Hi Ivan,
 > >>>>
 > >>>>> -----Original Message-----
 > >>>>> From: Ivan Malov <Ivan.Malov at oktetlabs.ru>
 > >>>>> Sent: Monday, October 4, 2021 2:06 PM
 > >>>>> Cc: dev at dpdk.org
 > >>>>> Subject: Re: [PATCH v1 02/12] ethdev: add eswitch port item to flow
 > >>>>> API
 > >>>>>
 > >>>>> Hi Ori,
 > >>>>>
 > >>>>> On 04/10/2021 08:45, Ori Kam wrote:
 > >>>>>> Hi Ivan,
 > >>>>>>
 > >>>>>>> -----Original Message-----
 > >>>>>>> From: Ivan Malov <Ivan.Malov at oktetlabs.ru>
 > >>>>>>> Sent: Sunday, October 3, 2021 9:11 PM
 > >>>>>>> Subject: Re: [PATCH v1 02/12] ethdev: add eswitch port item 
to flow
 > >>>>>>> API
 > >>>>>>>
 > >>>>>>>
 > >>>>>>>
 > >>>>>>> On 03/10/2021 15:40, Ori Kam wrote:
 > >>>>>>>> Hi Andrew and Ivan,
 > >>>>>>>>
 > >>>>>>>>> -----Original Message-----
 > >>>>>>>>> From: Andrew Rybchenko <andrew.rybchenko at oktetlabs.ru>
 > >>>>>>>>> Sent: Friday, October 1, 2021 4:47 PM
 > >>>>>>>>> Subject: [PATCH v1 02/12] ethdev: add eswitch port item to flow
 > >>>>>>>>> API
 > >>>>>>>>>
 > >>>>>>>>> From: Ivan Malov <ivan.malov at oktetlabs.ru>
 > >>>>>>>>>
 > >>>>>>>>> For use with "transfer" flows. Supposed to match traffic 
entering
 > >>>>>>>>> the e-switch from the external world (network, guests) via the
 > >>>>>>>>> port which is logically connected with the given ethdev.
 > >>>>>>>>>
 > >>>>>>>>> Must not be combined with attributes "ingress" / "egress".
 > >>>>>>>>>
 > >>>>>>>>> This item is meant to use the same structure as ethdev item.
 > >>>>>>>>>
 > >>>>>>>>
 > >>>>>>>> In case the app is not working with representors, meaning each
 > >>>>>>>> switch port is mapped to ethdev.
 > >>>>>>>> both items (ethdev and eswitch port ) have the same meaning?
 > >>>>>>>
 > >>>>>>> No. Ethdev means ethdev, and e-switch port is the point where 
this
 > >>>>>>> ethdev is plugged to. For example, "transfer + ESWITCH_PORT" 
for a
 > >>>>>>> regular PF ethdev typically means the network port (maybe you can
 > >>>>>>> recall the idea that a PF ethdev "represents" the network 
port it's
 > >>>>> associated with).
 > >>>>>>>
 > >>>>>>> I believe, that diagrams which these patches add to
 > >>>>>>> "doc/guides/prog_guide/rte_flow.rst" may come in handy to
 > >>>>>>> understand the meaning. Also, you can take a look at our larger
 > >>>>>>> diagram from the Sep 14 gathering.
 > >>>>>>>
 > >>>>>>
 > >>>>>> Lets look at the following system:
 > >>>>>> E-Switch has 3 ports - PF, VF1, VF2
 > >>>>>> The ports are distributed as follows:
 > >>>>>> DPDK application:
 > >>>>>> ethdev(0) pf,
 > >>>>>> ethdev(1) representor to VF1
 > >>>>>> ethdev(2) representor to VF2
 > >>>>>> ethdev(3) VF1
 > >>>>>>
 > >>>>>> VM:
 > >>>>>> VF2
 > >>>>>>
 > >>>>>> As we know all representors are realy connected to the PF(at least
 > >>>>>> in this example)
 > >>>>>
 > >>>>> This example tries to say that the e-switch has 3 ports in total,
 > >>>>> and, given your explanation, one may indeed agree that *in this
 > >>>>> example* representors re-use e-switch port of ethdev=0 (with some
 > >>>>> metadata to distinguish packets, etc.). But one can hardly assume
 > >>>>> that *all* representors with any vendor's NIC are connected to the
 > >>>>> e-switch the same way. It's vendor specific. Well, at least,
 > >>>>> applications don't have this knowledge and don't need to.
 > >>>>>
 > >>>>>>
 > >>>>>> So matching on ethdev(3)  means matching on traffic sent from DPDK
 > >>>>>> port
 > >>>>> 3 right?
 > >>>>>
 > >>>>> Item ETHDEV (ethdev_id=3) matches traffic sent by DPDK port 3. 
Looks
 > >>>>> like we're on the same page here.
 > >>>>>
 > >>>>
 > >>>> Good.
 > >>>>
 > >>>>>> And matching on eswitch_port(3) means matching in traffic that 
goes
 > >>>>>> into VF1 which is the same traffic as ethdev(3) right?
 > >>>>>
 > >>>>> I didn't catch the thought about "the same traffic". Direction 
is not the
 > >>> same.
 > >>>>> Item ESWITCH_PORT (ethdev_id=3) matches traffic sent by DPDK 
port 1.
 > >>>>>
 > >>>> This is the critical part for my understanding.
 > >>>> Matching on ethdev_id(3) means matching on traffic that is 
coming from
 > >>> DPDK port3.
 > >>>
 > >>> Right.
 > >>>
 > >>>> So from E-Switch view point it is traffic that goes into VF1?
 > >>>
 > >>> No. Above you clearly say "coming from DPDK port3". That is, from 
the VF1.
 > >>> *Not* going into it. Port 3 (ethdev_id=3) *is* VF1.
 > >>>
 > >> But taffic that goes from DPDK port 3 goes into VF1,
 > >> what am I missing?
 >
 > Terms like "PF", "VF", "PHY_PORT" describe the nature of the e-switch
 > ports. In your example, DPDK port 3 is just based on VF1. The
 > application doesn't have such knowledge. To it, this is a regular ethdev.
 >
 > In order to gain correct understanding, you should imagine an observer
 > standing inside the e-switch. When that observer sees packets entering
 > the e-switch FROM the VF1, we effectively talk about packets sent by the
 > ethdev sitting on top of that VF1, that is, packets coming FROM DPDK
 > port 3. And vice versa: if you say "traffic that goes into VF1", you
 > mean traffic that LEAVES the e-switch via VF1, that is, traffic, going
 > TO DPDK port 3.
 >
 > "VF1" is just a name of a logical "window" between the e-switch and its
 > surroundings. If this VF is passed to a guest machine, then this VF is a
 > "window" between the e-switch and the guest machine. If you plug this VF
 > to the DPDK application (create an ethdev on top of this VF), then this
 > VF is a "window" between the e-switch and the PMD. That's it.
 >
 > >
 > > DPDK port 3 is a VF1 itself. So, is it loopback? I think no.
 > >
 > > Let me repeat your example:
 > >
 > > DPDK application:
 > > ethdev(0) pf,
 > > ethdev(1) representor to VF1
 > > ethdev(2) representor to VF2
 > > ethdev(3) VF1
 > >
 > > VM:
 > > VF2
 > >
 > > Traffic that goes from DPDK port 3 (which is VF1 bound to DPDK)
 > > goes to its representor by default, i.e. ethdev(1).
 >
 > +1
 >
 > >
 > >>
 > >>>> While matching on E-Switch_port(3) means matching on traffic coming
 > >>> from VF1?
 > >>>
 > >>> No. It means matching on traffic coming from ethdev 1. From the VF1's
 > >>> representor.
 > >>>
 > >>>>
 > >>>> And by the same logic matching on ethdev_id(1) means matching on
 > >>>> taffic that was sent from DPDK port 1 and matching on 
E-Switch_port(1)
 > >>>> means matching on traffic coming from
 > >>>> VF1
 > >>>
 > >>> In this case, you've got this right. But please see my above 
notes. By the
 > >>> looks of it, you might have run into confusion over there.
 > >>>
 > >> That is the issue I'm not sure I understand, and I think that
 > >> if I don't underdand this, this miss underdanding will be shared 
by others.
 >
 > Please see my diagram below.
 >
 > >
 > > In order to address the confusion and misunderstanding we
 > > should find out the source of it. We always match on source
 > > (inbound port) and the result is a destination (outgoing port).
 > > So, if item is ETHDEV, the source is DPDK application via
 > > corresponding ethdev port.  If ETHDEV is an action, the
 > > destination is the DPDK application via specified ethdev port.
 > > Above is regardless of the ethdev port type (representor or
 > > not). Always, no exceptions.
 > >
 > >>>>
 > >>>> So in this case eswitch_port(3) is equal ot eswitch_port(1) right?
 > >>>> While ethdev(1) is not equal to ethdev(3)
 > >>>
 > >>> No.
 > >>>
 > >>> Item ETHDEV (ethdev_id=1) equals item ESWITCH_PORT (ethdev_id=3).
 > >>> Item ETHDEV (ethdev_id=3) equals item ESWITCH_PORT (ethdev_id=1).
 > >>>
 > >> I think this was my first undestaning, lets see if I can explain 
it using my words.
 > >> ETHDEV - means that the source port that we are matching on is the 
closest one
 > >> the dpdk application.
 > >
 > > Yes, it sounds right.
 > >
 > >> Example like above ETHDEV(ethdev_id=1) means matching
 > >> on traffic coming from the PF with some metadata that marks this 
DPDK port.
 >
 > Some vendors support allocation of separate logical e-switch ports for
 > representors. Other vendors can't support that, and, in this case, they
 > indeed have to devise "PF + metadata" solution. But DPDK framework (I
 > mean, vendor-agnostic code) can't assume any of these options as a
 > "standard". Neither can applications do that. It's truly vendor-specific.
 >
 > >
 > > No, no, no. It is the problem of too much knowledge and
 > > diving too deep. Neither you nor API user should not
 > > think about it. Representor ethdev(1) is a DPDK ethdev
 > > port that's it. An application can send traffic via the
 > > ethdev port and receive traffic from it. How the traffic
 > > goes inside is vendor-specific implementation detail.
 > > Application just know that if it sends traffic to VF1
 > > representor ethdev(1), the traffic will be received
 > > from VF1 by default. If it sends traffic from VF1,
 > > the traffic will be received from VF1 representor
 > > by default. It is the definition of the representor.
 >
 > +1
 >
 > >
 > >> ETHDEV(ethdev_id=3) means matching on traffic coming from VF1.
 > >
 > > Yes.
 > >
 > >>
 > >> ESWITCH_PORT meaning matching on the port that is connected to the 
DPDK port
 > >> (the other side of the wire). Example ESWITCH_PORT(ethdev_id=1) 
means matching
 > >> on traffic coming from VF1
 > >
 > > Yes, since ethdev(1) is VF1 representor.
 > >
 > >> While matching on ESWITCH_PORT(ethdev_id=3) means matching on PF 
with some
 > >> metadata.
 > >
 > > Again, yes and no. No, since you should not think about it this way. It
 > > is just traffic sent via VF1 representor (since ethdev(3) is VF1 
and its
 > > other side of the logical wire is VF1
 > > representor). No vendor-specific implementation details.
 > >
 > >>
 > >> Everything assume that representors are on the PF and use some 
metadata.
 > >
 > > No, it is a vendor-specific implementation details.
 > > Flow API definitions should not mention it.
 >
 > +1
 >
 > >
 > >>
 > >> Did I get it right?
 >
 > Let's try to look at the overall idea one more time. Just to get us on
 > the same page.
 >
 > In general, for any given ethdev you have:
 >
 >
 > ETHDEV (A)
 > |
 > |    (PMD area)
 > |
 > CLOSEST ESWITCH_PORT (B)
 > |
 > |    (e-switch area)
 > |
 > REMOTE ESWITCH_PORT (C)
 > |
 > X
 >
 >
 > The point "B" is located closest to the ethdev (A). Not to the
 > application in general, but to the given ethdev.
 >
 > The point "C" is the most remote point from the ethdev's perspective.
 > Again, it's not the most remote from the application. It's the most
 > remote from this specific ethdev.
 >
 >  From the application perspective, the nature of (B) and (C) is unknown.
 > But the application knows for sure that these points just exist. To it,
 > these points are some *logical* e-switch ports.
 >
 > Now.
 >
 > When you say "item ETHDEV", you tell the PMD that you want to match
 > packets entering the *PMD area* FROM point (A). But you also add
 > "transfer" attribute. This attribute makes the PMD translate your
 > request to the e-switch viewpoint by "transferring" point "A" one level
 > below. What was "A" becomes "B". This way, an imaginary observer
 > standing inside the *e-switch area* intercepts packets which enter this
 > area FROM (B).
 > Summary: this way, you match packets going *DOWN* from (A).
 >
 > When you say "item ESWITCH_PORT", you tell the PMD that you want to
 > match packets entering the *PMD area* FROM point (B). Once again,
 > "transfer" translates this to the e-switch viewpoint, that is, (B)
 > becomes (C). So, the e-switch knows that it should intercept packets
 > entering its area FROM point (C).
 > Summary: this way, you match packets going *UP* from (X).
 >
 >
 > Some use case examples:
 >
 > - The ethdev (A) is a DPDK port based on a PF associated with a network
 > port. In this case, (B) == PF and (C) == (X) == network port.
 >
 > - The ethdev (A) is a representor of VF1 which is passed to a VM. In
 > this case, (B) is just an e-switch logical port. It's nature is vendor
 > specific. It can be a PF, it can be not. It depends. (C) == VF, (X) 
== VM.
 >
 > - The ethdev (A) is a representor of VF1 which is plugged to the same
 > DPDK application. In this case, (B) is, once again, some logical
 > e-switch port. (C) == VF. But (X) here is another ethdev. So, this way,
 > ethdev (A) is a representor's ethdev and ethdev (X) is a VF's ethdev.
 >
 > In all cases the application doesn't care about the nature of the
 > e-switch ports (B) and (C). It just knows that they exist.
 >
 > >
 > > I think you get overall idea right, but explanations
 > > are too complicated. It is simpler in fact.
 > >
 > >>>>
 > >>>> And just to complete the picture, matching on ethdev(2) will 
result in
 > >>>> traffic coming from the dpdk port and matching on 
eswitch_port(2) will
 > >>>> match on traffic coming from VF2
 > >>>
 > >>> Exactly.
 > >>>
 > >>>
 > >>> But, Ori, let me draw your attention to the following issue. In 
order to
 > >>> simplify understanding, I suggest that we refrain from saying 
"traffic that
 > >>> GOES TO". Where it goes depends on default rules that are 
supposed to be
 > >>> maintained by the PMD when ports get plugged / unplugged.
 > >>>
 > >>> The flow items ETHDEV and ESWITH_PORT define the SOURCE of traffic.
 > >>> That's it. They define where the traffic "goes FROM".
 > >>>
 > >>> Say, the DPDK application sends a packet from ethdev 0. This 
packet enters
 > >>> the e-switch. Match engine sits in the e-switch and intercepts 
the packet. It
 > >>> doesn't care where the packet *would go* if it wasn't 
intercepted. It cares
 > >>> about where the packet comes from. And it comes from ethdev 0. 
So, in the
 > >>> focus, we have the SOURCE of the packet.
 > >>>
 > >>
 > >> Agree with you we should only look at coming from,
 > >> but something in the patch made me thing otherwise (not sure what 
part)
 > >
 > > I've reread the documentation and failed to find,
 > > but it will be hard for me to do it, since I read
 > > it too many times already.
 > >
 > > Thanks,
 > > Andrew.
 > >

-- 
Ivan M

^ permalink raw reply	[flat|nested] 161+ messages in thread
end of thread, other threads:[~2021-10-26 16:16 UTC | newest]

Thread overview: 161+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-07 12:51 [dpdk-dev] [RFC PATCH] ethdev: clarify flow attribute and action port ID semantics Ivan Malov
2021-10-01 13:47 ` [dpdk-dev] [PATCH v1 00/12] ethdev: rework transfer flow API Andrew Rybchenko
2021-10-01 13:47   ` [dpdk-dev] [PATCH v1 01/12] ethdev: add ethdev item to " Andrew Rybchenko
2021-10-03 11:52     ` Ori Kam
2021-10-03 17:40       ` Ivan Malov
2021-10-03 21:09         ` Ori Kam
2021-10-04  0:00           ` Ivan Malov
2021-10-04 10:47             ` Andrew Rybchenko
2021-10-04 11:08               ` Ivan Malov
2021-10-01 13:47   ` [dpdk-dev] [PATCH v1 02/12] ethdev: add eswitch port " Andrew Rybchenko
2021-10-03 12:40     ` Ori Kam
2021-10-03 18:10       ` Ivan Malov
2021-10-04  5:45         ` Ori Kam
2021-10-04 11:05           ` Ivan Malov
2021-10-04 11:37             ` Ori Kam
2021-10-04 11:58               ` Ivan Malov
2021-10-05  6:20                 ` Ori Kam
2021-10-05  9:19                   ` Andrew Rybchenko
2021-10-05 12:12                     ` Ivan Malov
2021-10-01 13:47   ` [dpdk-dev] [PATCH v1 03/12] ethdev: add ethdev action " Andrew Rybchenko
2021-10-01 13:47   ` [dpdk-dev] [PATCH v1 04/12] ethdev: add eswitch port " Andrew Rybchenko
2021-10-01 13:47   ` [dpdk-dev] [PATCH v1 05/12] ethdev: deprecate hard-to-use or ambiguous items and actions Andrew Rybchenko
2021-10-01 13:47   ` [dpdk-dev] [PATCH v1 06/12] ethdev: deprecate direction attributes in transfer flows Andrew Rybchenko
2021-10-01 13:47   ` [dpdk-dev] [PATCH v1 07/12] net/bnxt: support ethdev and E-Switch port flow items Andrew Rybchenko
2021-10-01 13:47   ` [dpdk-dev] [PATCH v1 08/12] net/bnxt: support ethdev and E-Switch port flow actions Andrew Rybchenko
2021-10-01 13:47   ` [dpdk-dev] [PATCH v1 09/12] net/enic: " Andrew Rybchenko
2021-10-01 13:47   ` [dpdk-dev] [PATCH v1 10/12] net/mlx5: support E-Switch port flow action Andrew Rybchenko
2021-10-01 13:47   ` [dpdk-dev] [PATCH v1 11/12] net/octeontx2: support ethdev " Andrew Rybchenko
2021-10-04 11:13     ` [dpdk-dev] [EXT] " Kiran Kumar Kokkilagadda
2021-10-04 12:45       ` Andrew Rybchenko
2021-10-01 13:47   ` [dpdk-dev] [PATCH v1 12/12] net/sfc: support ethdev flow item Andrew Rybchenko
2021-10-10  0:04   ` [dpdk-dev] [PATCH v2 00/12] ethdev: rework transfer flow API Ivan Malov
2021-10-10  0:04     ` [dpdk-dev] [PATCH v2 01/12] ethdev: add port representor item to " Ivan Malov
2021-10-10 11:15       ` Ori Kam
2021-10-10 13:30         ` Ivan Malov
2021-10-10 14:04           ` Ori Kam
2021-10-10 15:02             ` Ivan Malov
2021-10-10  0:04     ` [dpdk-dev] [PATCH v2 02/12] ethdev: add represented port " Ivan Malov
2021-10-10  0:04     ` [dpdk-dev] [PATCH v2 03/12] ethdev: add port representor action " Ivan Malov
2021-10-10  0:04     ` [dpdk-dev] [PATCH v2 04/12] ethdev: add represented port " Ivan Malov
2021-10-10  0:04     ` [dpdk-dev] [PATCH v2 05/12] ethdev: deprecate hard-to-use or ambiguous items and actions Ivan Malov
2021-10-10  0:04     ` [dpdk-dev] [PATCH v2 06/12] ethdev: deprecate direction attributes in transfer flows Ivan Malov
2021-10-10  0:04     ` [dpdk-dev] [PATCH v2 07/12] net/bnxt: support meta flow items to match on traffic source Ivan Malov
2021-10-10  0:04     ` [dpdk-dev] [PATCH v2 08/12] net/bnxt: support meta flow actions to overrule destinations Ivan Malov
2021-10-10  0:05     ` [dpdk-dev] [PATCH v2 09/12] net/enic: " Ivan Malov
2021-10-10  0:05     ` [dpdk-dev] [PATCH v2 10/12] net/mlx5: support represented port flow action Ivan Malov
2021-10-10  0:05     ` [dpdk-dev] [PATCH v2 11/12] net/octeontx2: support port representor " Ivan Malov
2021-10-10  0:05     ` [dpdk-dev] [PATCH v2 12/12] net/sfc: support port representor flow item Ivan Malov
2021-10-10 14:39   ` [dpdk-dev] [PATCH v3 00/12] ethdev: rework transfer flow API Ivan Malov
2021-10-10 14:39     ` [dpdk-dev] [PATCH v3 01/12] ethdev: add port representor item to " Ivan Malov
2021-10-11 11:49       ` Ori Kam
2021-10-12 12:39         ` Andrew Rybchenko
2021-10-12 20:55       ` Slava Ovsiienko
2021-10-12 23:14         ` Ivan Malov
2021-10-13 18:26           ` Slava Ovsiienko
2021-10-13 11:52       ` Ferruh Yigit
2021-10-10 14:39     ` [dpdk-dev] [PATCH v3 02/12] ethdev: add represented port " Ivan Malov
2021-10-11 11:51       ` Ori Kam
2021-10-12 12:39         ` Andrew Rybchenko
2021-10-10 14:39     ` [dpdk-dev] [PATCH v3 03/12] ethdev: add port representor action " Ivan Malov
2021-10-11 18:18       ` Ori Kam
2021-10-12 12:39         ` Andrew Rybchenko
2021-10-10 14:39     ` [dpdk-dev] [PATCH v3 04/12] ethdev: add represented port " Ivan Malov
2021-10-11 18:20       ` Ori Kam
2021-10-12 12:40         ` Andrew Rybchenko
2021-10-10 14:39     ` [dpdk-dev] [PATCH v3 05/12] ethdev: deprecate hard-to-use or ambiguous items and actions Ivan Malov
2021-10-11 18:23       ` Ori Kam
2021-10-12 12:40         ` Andrew Rybchenko
2021-10-13 11:53       ` Ferruh Yigit
2021-10-10 14:39     ` [dpdk-dev] [PATCH v3 06/12] ethdev: deprecate direction attributes in transfer flows Ivan Malov
2021-10-11 18:33       ` Ori Kam
2021-10-12 12:40         ` Andrew Rybchenko
2021-10-10 14:39     ` [dpdk-dev] [PATCH v3 07/12] net/bnxt: support meta flow items to match on traffic source Ivan Malov
2021-10-10 14:39     ` [dpdk-dev] [PATCH v3 08/12] net/bnxt: support meta flow actions to overrule destinations Ivan Malov
2021-10-10 14:39     ` [dpdk-dev] [PATCH v3 09/12] net/enic: " Ivan Malov
2021-10-12 17:03       ` Hyong Youb Kim (hyonkim)
2021-10-10 14:39     ` [dpdk-dev] [PATCH v3 10/12] net/mlx5: support represented port flow action Ivan Malov
2021-10-12 21:23       ` Slava Ovsiienko
2021-10-10 14:39     ` [dpdk-dev] [PATCH v3 11/12] net/octeontx2: support port representor " Ivan Malov
2021-10-10 14:39     ` [dpdk-dev] [PATCH v3 12/12] net/sfc: support port representor flow item Ivan Malov
2021-10-12 14:45     ` [dpdk-dev] [PATCH v3 00/12] ethdev: rework transfer flow API Ferruh Yigit
2021-10-13 11:51     ` Ferruh Yigit
2021-10-13 16:42   ` [dpdk-dev] [PATCH v4 " Ivan Malov
2021-10-13 16:42     ` [dpdk-dev] [PATCH v4 01/12] ethdev: add port representor item to " Ivan Malov
2021-10-13 16:42     ` [dpdk-dev] [PATCH v4 02/12] ethdev: add represented port " Ivan Malov
2021-10-13 16:42     ` [dpdk-dev] [PATCH v4 03/12] ethdev: add port representor action " Ivan Malov
2021-10-13 16:42     ` [dpdk-dev] [PATCH v4 04/12] ethdev: add represented port " Ivan Malov
2021-10-13 16:42     ` [dpdk-dev] [PATCH v4 05/12] ethdev: deprecate hard-to-use or ambiguous items and actions Ivan Malov
2021-10-13 16:42     ` [dpdk-dev] [PATCH v4 06/12] ethdev: deprecate direction attributes in transfer flows Ivan Malov
2021-10-13 16:42     ` [dpdk-dev] [PATCH v4 07/12] net/bnxt: support meta flow items to match on traffic source Ivan Malov
2021-10-13 16:42     ` [dpdk-dev] [PATCH v4 08/12] net/bnxt: support meta flow actions to overrule destinations Ivan Malov
2021-10-13 16:42     ` [dpdk-dev] [PATCH v4 09/12] net/enic: " Ivan Malov
2021-10-13 16:42     ` [dpdk-dev] [PATCH v4 10/12] net/mlx5: support represented port flow action Ivan Malov
2021-10-13 16:42     ` [dpdk-dev] [PATCH v4 11/12] net/octeontx2: support port representor " Ivan Malov
2021-10-13 16:42     ` [dpdk-dev] [PATCH v4 12/12] net/sfc: support port representor flow item Ivan Malov
2021-10-13 16:57   ` [dpdk-dev] [PATCH v4 00/12] ethdev: rework transfer flow API Ivan Malov
2021-10-13 16:57     ` [dpdk-dev] [PATCH v5 01/12] ethdev: add port representor item to " Ivan Malov
2021-10-13 16:57     ` [dpdk-dev] [PATCH v5 02/12] ethdev: add represented port " Ivan Malov
2021-10-13 16:57     ` [dpdk-dev] [PATCH v5 03/12] ethdev: add port representor action " Ivan Malov
2021-10-13 16:57     ` [dpdk-dev] [PATCH v5 04/12] ethdev: add represented port " Ivan Malov
2021-10-13 16:57     ` [dpdk-dev] [PATCH v5 05/12] ethdev: deprecate hard-to-use or ambiguous items and actions Ivan Malov
2021-10-13 16:57     ` [dpdk-dev] [PATCH v5 06/12] ethdev: deprecate direction attributes in transfer flows Ivan Malov
2021-10-13 16:57     ` [dpdk-dev] [PATCH v5 07/12] net/bnxt: support meta flow items to match on traffic source Ivan Malov
2021-10-13 16:57     ` [dpdk-dev] [PATCH v5 08/12] net/bnxt: support meta flow actions to overrule destinations Ivan Malov
2021-10-13 16:57     ` [dpdk-dev] [PATCH v5 09/12] net/enic: " Ivan Malov
2021-10-13 16:57     ` [dpdk-dev] [PATCH v5 10/12] net/mlx5: support represented port flow action Ivan Malov
2021-10-13 16:57     ` [dpdk-dev] [PATCH v5 11/12] net/octeontx2: support port representor " Ivan Malov
2021-10-13 16:57     ` [dpdk-dev] [PATCH v5 12/12] net/sfc: support port representor flow item Ivan Malov
2021-10-13 17:02   ` [dpdk-dev] [PATCH v6 00/12] ethdev: rework transfer flow API Ivan Malov
2021-10-13 17:02     ` [dpdk-dev] [PATCH v6 01/12] ethdev: add port representor item to " Ivan Malov
2021-10-13 17:15       ` Andrew Rybchenko
2021-10-13 17:02     ` [dpdk-dev] [PATCH v6 02/12] ethdev: add represented port " Ivan Malov
2021-10-13 17:02     ` [dpdk-dev] [PATCH v6 03/12] ethdev: add port representor action " Ivan Malov
2021-10-13 17:02     ` [dpdk-dev] [PATCH v6 04/12] ethdev: add represented port " Ivan Malov
2021-10-13 17:02     ` [dpdk-dev] [PATCH v6 05/12] ethdev: deprecate hard-to-use or ambiguous items and actions Ivan Malov
2021-10-13 17:02     ` [dpdk-dev] [PATCH v6 06/12] ethdev: deprecate direction attributes in transfer flows Ivan Malov
2021-10-13 17:02     ` [dpdk-dev] [PATCH v6 07/12] net/bnxt: support meta flow items to match on traffic source Ivan Malov
2021-10-13 17:02     ` [dpdk-dev] [PATCH v6 08/12] net/bnxt: support meta flow actions to overrule destinations Ivan Malov
2021-10-13 17:02     ` [dpdk-dev] [PATCH v6 09/12] net/enic: " Ivan Malov
2021-10-13 17:02     ` [dpdk-dev] [PATCH v6 10/12] net/mlx5: support represented port flow action Ivan Malov
2021-10-13 18:02       ` Slava Ovsiienko
2021-10-13 17:02     ` [dpdk-dev] [PATCH v6 11/12] net/octeontx2: support port representor " Ivan Malov
2021-10-13 17:02     ` [dpdk-dev] [PATCH v6 12/12] net/sfc: support port representor flow item Ivan Malov
2021-10-13 17:34   ` [dpdk-dev] [PATCH v7 00/12] ethdev: rework transfer flow API Ivan Malov
2021-10-13 17:34     ` [dpdk-dev] [PATCH v7 01/12] ethdev: add port representor item to " Ivan Malov
2021-10-13 17:34     ` [dpdk-dev] [PATCH v7 02/12] ethdev: add represented port " Ivan Malov
2021-10-13 17:34     ` [dpdk-dev] [PATCH v7 03/12] ethdev: add port representor action " Ivan Malov
2021-10-13 17:34     ` [dpdk-dev] [PATCH v7 04/12] ethdev: add represented port " Ivan Malov
2021-10-13 17:34     ` [dpdk-dev] [PATCH v7 05/12] ethdev: deprecate hard-to-use or ambiguous items and actions Ivan Malov
2021-10-13 17:34     ` [dpdk-dev] [PATCH v7 06/12] ethdev: deprecate direction attributes in transfer flows Ivan Malov
2021-10-13 18:48       ` Slava Ovsiienko
2021-10-13 17:34     ` [dpdk-dev] [PATCH v7 07/12] net/bnxt: support meta flow items to match on traffic source Ivan Malov
2021-10-13 17:34     ` [dpdk-dev] [PATCH v7 08/12] net/bnxt: support meta flow actions to overrule destinations Ivan Malov
2021-10-13 17:34     ` [dpdk-dev] [PATCH v7 09/12] net/enic: " Ivan Malov
2021-10-13 17:34     ` [dpdk-dev] [PATCH v7 10/12] net/mlx5: support represented port flow action Ivan Malov
2021-10-13 17:34     ` [dpdk-dev] [PATCH v7 11/12] net/octeontx2: support port representor " Ivan Malov
2021-10-20  2:33       ` [dpdk-dev] [EXT] " Kiran Kumar Kokkilagadda
2021-10-13 17:34     ` [dpdk-dev] [PATCH v7 12/12] net/sfc: support port representor flow item Ivan Malov
2021-10-13 21:08     ` [dpdk-dev] [PATCH v7 00/12] ethdev: rework transfer flow API Ferruh Yigit
2021-10-05  0:36 ` [dpdk-dev] [PATCH] ethdev: let apps find transfer admin port for a given ethdev Ivan Malov
2021-10-05  9:22   ` Ori Kam
2021-10-05 12:41     ` Ivan Malov
2021-10-05 16:04       ` Ori Kam
2021-10-05 17:56   ` Thomas Monjalon
2021-10-05 18:22     ` Ivan Malov
2021-10-05 19:04       ` Thomas Monjalon
2021-10-05 21:10   ` [dpdk-dev] [PATCH v2] ethdev: add API to query proxy port to manage transfer flows Ivan Malov
2021-10-06  7:47     ` Andrew Rybchenko
2021-10-06 12:33   ` [dpdk-dev] [PATCH v3] " Ivan Malov
2021-10-14  3:21   ` [dpdk-dev] [PATCH v4] " Ivan Malov
2021-10-14 11:45     ` Ferruh Yigit
2021-10-26 14:46     ` David Marchand
2021-10-26 15:47       ` Ivan Malov
2021-10-26 15:58         ` Thomas Monjalon
2021-10-26 16:15           ` Ferruh Yigit
2021-10-06 15:30 [dpdk-dev] [PATCH v1 02/12] ethdev: add eswitch port item to flow API Ivan Malov
2021-10-06 18:00 ` Ivan Malov
2021-10-06 19:36   ` Ivan Malov
2021-10-07 13:00     ` Ori Kam
2021-10-07 14:35       ` Ivan Malov
2021-10-07 16:06         ` Andrew Rybchenko

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).