From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6DDCFA0524; Wed, 2 Jun 2021 18:26:32 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DB2CD4069F; Wed, 2 Jun 2021 18:26:31 +0200 (CEST) Received: from shelob.oktetlabs.ru (shelob.oktetlabs.ru [91.220.146.113]) by mails.dpdk.org (Postfix) with ESMTP id 9E3EB40689 for ; Wed, 2 Jun 2021 18:26:30 +0200 (CEST) Received: from [192.168.38.17] (aros.oktetlabs.ru [192.168.38.17]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by shelob.oktetlabs.ru (Postfix) with ESMTPSA id 0834F7F4FE; Wed, 2 Jun 2021 19:26:30 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 shelob.oktetlabs.ru 0834F7F4FE DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=oktetlabs.ru; s=default; t=1622651190; bh=7WDpFgpC2iihGy6s8AuRIiFbLKZqDEkvLMWeXowwQes=; h=Subject:To:Cc:References:From:Date:In-Reply-To; b=VTP4NzhKvo0jbfzd1y2BcTVp2gKQHd2kuF9IsCvAjrHEZF2cr+dUl+sAmbOeHBcI0 D+M1XxI6DhqtoduReDb6eiagP2DBAVkFia9Fza6cCV8bAHobgJFj8EIYjsVCcz8Ls8 lAYcboRg4jznSAPPHYg7Ri7FG1/Y5k7q9j3g7TRI= To: Ilya Maximets , Ivan Malov , dev@dpdk.org Cc: Eli Britstein , Smadar Fuks , Hyong Youb Kim , Kishore Padmanabha , Ori Kam , Ajit Khaparde , Jerin Jacob , John Daley , Thomas Monjalon , Ferruh Yigit References: <20210601111420.5549-1-ivan.malov@oktetlabs.ru> <8c4f559e-3430-e2d5-1199-f1d4f4a8546d@ovn.org> <4ffdb830-53ce-6ae6-7271-3eeb546b78de@oktetlabs.ru> From: Andrew Rybchenko Organization: OKTET Labs Message-ID: Date: Wed, 2 Jun 2021 19:26:29 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] [RFC PATCH] ethdev: clarify flow action PORT ID semantics X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 6/2/21 3:46 PM, Ilya Maximets wrote: > On 6/1/21 4:28 PM, Ivan Malov wrote: >> Hi Ilya, >> >> Thank you for reviewing the proposal at such short notice. I'm afraid that prior discussions overlook the simple fact that the whole problem is not limited to just VF representors. Action PORT_ID is also used with respect to the admin PF's ethdev, which "represents itself" (and by no means it represents the underlying physical/network port). In this case, one cannot state that the application treats it as a physical port, just like one states that the application perceives representors as VFs themselves. > > > I don't think that it was overlooked. If device is in a switchdev mode than > there is a PF representor and VF representors. Application typically works > only with representors in this case is it doesn't make much sense to have > representor and the upstream port attached to the same application at the > same time. Configuration that is applied by application to the representor > (PF or VF, it doesn't matter) applies to the corresponding upstream port > (actual PF or VF) by default. PF is not necessarily associated with a network port. It could be many PFs and just one network port on NIC. Extra PFs are like VFs in this case. These PFs may be passed to a VM in a similar way. So, we can have PF representors similar to VF representors. I.e. it is incorrect to say that PF in the case of switchdev is a representor of a network port. If we prefer to talk in representors terminology, we need 4 types of prepresentors: - PF representor for PCIe physical function - VF representor for PCIe virtual function - SF representor for PCIe sub-function (PASID) - network port representor In fact above is PCIe oriented, but there are other buses and ways to deliver traffic to applications. Basically representor for any virtual port in virtual switch which DPDK app can control using transfer rules. > Exactly same thing here with PORT_ID action. You have a packet and action > to send it to the port, but it's not specified if HW needs to send it to > the representor or the upstream port (again, VF or PF, it doesn't matter). > Since there is no extra information, HW should send it to the upstream > port by default. The same as configuration applies by default to the > upstream port. > > Let's look at some workflow examples: > > DPDK Application > | | > | | > +--PF-rep------VF-rep---+ > | | > | NIC (switchdev) | > | | > +---PF---------VF-------+ > | | > | | > External VM or whatever > Network See above. PF <-> External Network is incorrect above since it not always the case. It should be "NP <-> External network" and "NP-rep" above (NP - network port). Sometimes PF is an NP-rep, but sometimes it is not. It is just a question of default rules in switchdev on what to do with traffic incoming from network port. A bit more complicated picture is: +----------------------------------------+ | DPDK Application | +----+---------+---------+---------+-----+ |PF0 |PF1 | | | | | | +--NP1-rep---NP2-rep---PF2-rep---VF-rep--+ | | | NIC (switchdev) | | | +---NP1-------NP2-------PF2--------VF----+ | | | | | | | | External External VM or VM or Network 1 Network 2 whatever whatever So, sometimes PF plays network port representor role (PF0, PF1), sometimes it requires representor itself (PF2). What to do if PF2 itself is attached to application? Can we route traffic to it using PORT_ID action? It has DPDK ethdev port. It is one of arguments why plain PORT_ID should route DPDK application. Of course, some applications would like to see it as (simpler is better): +----------------------------------------+ | DPDK Application | | | +---PF0-------PF1------PF2-rep---VF-rep--+ | | | | | | | | External External VM or VM or Network 1 Network 2 whatever whatever but some, I believe, require full picture. For examples, I'd really like to know how much traffic goes via all 8 switchdev ports and running rte_eth_stats_get(0, ...) (i.e. DPDK port 0 attached to PF0) I'd like to get NP1-rep stats (not NP1 stats). It will match exactly what I see in DPDK application. It is an argument why plain PORT_ID should be treated as a DPDK ethdev port, not a represented (upstream) entity. > a. Workflow for "DPDK Application" to set MAC to VF: > > 1. "DPDK Application" calls rte_set_etheraddr("VF-rep", new_mac); > 2. DPDK sets MAC for "VF". > > b. Workflow for "DPDK Application" to set MAC to PF: > > 1. "DPDK Application" calls rte_set_etheraddr("PF-rep", new_mac); > 2. DPDK sets MAC for "PF". > > c. Workflow for "DPDK Application" to send packet to the external network: > > 1. "DPDK Application" calls rte_eth_tx_burst("PF-rep", packet); > 2. NIC receives the packet from "PF-rep" and sends it to "PF". > 3. packet egresses to the external network from "PF". > > d. Workflow for "DPDK Application" to send packet to the "VM or whatever": > > 1. "DPDK Application" calls rte_eth_tx_burst("VF-rep", packet); > 2. NIC receives the packet from "VF-rep" and sends it to "VF". > 3. "VM or whatever" receives the packet from "VF". > > In two workflows above there is no rte_flow processing on step 2, i.e., > NIC does not perform any lookups/matches/actions, because it's not possible > to configure actions for packets received from "PF-rep" or > "VF-rep" as these ports doesn't own a port id and all the configuration > and rte_flow actions translated and applied for the devices that these > ports represents ("PF" and "VF") and not representors themselves ("PF-rep" > or "VF-rep"). > > e. Workflow for the packet received on PF and PORT_ID action: > > 1. "DPDK Application" configures rte_flow for all packets from "PF-rep" > to execute PORT_ID "VF-rep". > 2. NIC receives packet on "PF". > 3. NIC executes 'PORT_ID "VF-rep"' action by sending packet to "VF". > 4. "VM or whatever" receives the packet from "VF". > > f. Workflow for the packet received on VF and PORT_ID action: > > 1. "DPDK Application" configures rte_flow for all packets from "VF-rep" > to execute 'PORT_ID "PF-rep"'. > 2. NIC receives packet on "VF". > 3. NIC executes 'PORT_ID "PF-rep"' action by sending packet to "PF". > 4. Packet egresses from the "PF" to the external network. > > Above is what, IMHO, the logic should look like and this matches with > the overall switchdev design in kernel. > > I understand that this logic could seem flipped-over from the HW point > of view, but it's perfectly logical from the user's perspective, because > user should not care if the application works with representors or > some real devices. If application configures that all packets from port > A should be sent to port B, user will expect that these packets will > egress from port B once received from port A. That will be highly > inconvenient if the packet will ingress from port B back to the > application instead. > > DPDK Application > | | > | | > port A port B > | | > *****MAGIC***** > | | > External Another Network > Network or VM or whatever > > It should not matter if there is an extra layer between ports A and B > and the external network and VM. Everything should work in exactly the > same way, transparently for the application. > > The point of hardware offloading, and therefore rte_flow API, is to take > what user does in software and make this "magically" work in hardware in > the exactly same way. And this will be broken if user will have to > use different logic based on the mode the hardware works in, i.e. based on > the fact if the application works with ports or their representors. > > If some specific use case requires application to know if it's an > upstream port or the representor and demystify the internals of the switchdev > NIC, there should be a different port id for the representor itself that > could be used in all DPDK APIs including rte_flow API or a special bit for > that matter. IIRC, there was an idea to add a bit directly to the port_id > for that purpose that will flip over behavior in all the workflow scenarios > that I described above. As I understand we're basically on the same page, but just fighting for defaults in DPDK. >> >> Given these facts, it would not be quite right to just align the documentation with the de-facto action meaning assumed by OvS. > > It's not a "meaning assumed by OvS", it's the original design and the > main idea of a switchdev based on a common sense. If so, common sense is not that common :) My "common sense" says me that PORT_ID action should route traffic to DPDK ethdev port to be received by the DPDK application. >> >> On 01/06/2021 15:10, Ilya Maximets wrote: >>> On 6/1/21 1:14 PM, Ivan Malov wrote: >>>> By its very name, action PORT_ID means that packets hit an ethdev with the >>>> given DPDK port ID. At least the current comments don't state the opposite. >>>> That said, since port representors had been adopted, applications like OvS >>>> have been misusing the action. They misread its purpose as sending packets >>>> to the opposite end of the "wire" plugged to the given ethdev, for example, >>>> redirecting packets to the VF itself rather than to its representor ethdev. >>>> Another example: OvS relies on this action with the admin PF's ethdev port >>>> ID specified in it in order to send offloaded packets to the physical port. >>>> >>>> Since there might be applications which use this action in its valid sense, >>>> one can't just change the documentation to greenlight the opposite meaning. >>>> This patch adds an explicit bit to the action configuration which will let >>>> applications, depending on their needs, leverage the two meanings properly. >>>> Applications like OvS, as well as PMDs, will have to be corrected when the >>>> patch has been applied. But the improved clarity of the action is worth it. >>>> >>>> The proposed change is not the only option. One could avoid changes in OvS >>>> and PMDs if the new configuration field had the opposite meaning, with the >>>> action itself meaning delivery to the represented port and not to DPDK one. >>>> Alternatively, one could define a brand new action with the said behaviour. >>> >>> We had already very similar discussions regarding the understanding of what >>> the representor really is from the DPDK API's point of view, and the last >>> time, IIUC, it was concluded by a tech. board that representor should be >>> a "ghost of a VF", i.e. DPDK APIs should apply configuration by default to >>> VF and not to the representor device: >>>    https://patches.dpdk.org/project/dpdk/cover/20191029185051.32203-1-thomas@monjalon.net/#104376 >>> This wasn't enforced though, IIUC, for existing code and semantics is still mixed. >>> >>> I still think that configuration should be applied to VF, and the same applies >>> to rte_flow API.  IMHO, average application should not care if device is >>> a VF itself or its representor.  Everything should work exactly the same. >>> I think this matches with the original idea/design of the switchdev functionality >>> in the linux kernel and also matches with how the average user thinks about >>> representor devices. >>> >>> If some specific use-case requires to distinguish VF from the representor, >>> there should probably be a separate special API/flag for that. >>> >>> Best regards, Ilya Maximets. >>> >>