From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 71D89A0032; Tue, 15 Mar 2022 23:12:11 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4179D40395; Tue, 15 Mar 2022 23:12:11 +0100 (CET) Received: from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net [217.70.183.198]) by mails.dpdk.org (Postfix) with ESMTP id 134E14014F for ; Tue, 15 Mar 2022 23:12:10 +0100 (CET) Received: (Authenticated sender: i.maximets@ovn.org) by mail.gandi.net (Postfix) with ESMTPSA id 70A34C0008; Tue, 15 Mar 2022 22:12:06 +0000 (UTC) Message-ID: <5248c2ca-f2a6-3fb0-38b8-7f659bfa40de@ovn.org> Date: Tue, 15 Mar 2022 23:12:05 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.5.0 Content-Language: en-US Cc: i.maximets@ovn.org, Sriharsha Basavapatna , Gaetan Rivet , Eli Britstein , Ivan Malov , Andrew Rybchenko , Ori Kam , Thomas Monjalon , Ian Stokes From: Ilya Maximets To: dev Subject: rte_flow API change request: tunnel info restoration is too slow Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi, everyone. After implementing support for tunnel offloading in OVS we faced a significant performance issue caused by the requirement to call rte_flow_get_restore_info() function on a per-packet basis. The main problem is that once the tunnel offloading is configured, there is no way for the application to tell if a particular packet was partially processed in the hardware and the tunnel info has to be restored. What we have to do right now is to call the rte_flow_get_restore_info() unconditionally for every packet. The result of that call says if we have the tunnel info or not. rte_flow_get_restore_info() call itself is very heavy. It is at least a couple of indirect function calls and the device lock on the application side (not-really-thread-safety of the rte_flow API is a separate topic). Internal info lookup inside the driver also costs a lot (depends on a driver). It has been measured that having this call on a per-packet basis can reduce application performance (OVS) by a factor of 3 in some cases: https://mail.openvswitch.org/pipermail/ovs-dev/2021-November/389265.html https://github.com/openvswitch/ovs/commit/6e50c1651869de0335eb4b7fd0960059c5505f5c (Above patch avoid the problem in a hacky way for devices that doesn't support tunnel offloading, but it's not applicable to situation where device actually supports it, since the API has to be called.) Another tricky part is that we have to call rte_flow_get_restore_info() before checking other parts of the mbuf, because mlx5 driver, for example, re-uses the mbuf.hash.fdir value for both tunnel info restoration and classification offloading, so the application has no way to tell which one is used right now and has to call the restoration API first in order to find out. What we need: A generic and fast (couple of CPU cycles) API that will clearly say if the heavy rte_flow_get_restore_info() has to be called for a particular packet or not. Ideally, that should be a static mbuf flag that can be easily checked by the application. Calls inside the device driver are way too slow for that purpose, especially if they are not fully thread-safe, or require complex lookups or calculations. I'm assuming here that packets that really need the tunnel info restoration should be fairly rare. Current state: Currently, the get_restore_info() API is implemented only for mlx5 and sfc drivers, AFAICT. SFC driver is already using mbuf flag, but it's dynamic and not exposed to the application. MLX5 driver re-uses mbuf.hash.fdir value and performs a heavy lookup inside the driver. For now OVS doesn't support tunnel offload with DPDK formally, the code in OVS is under the experimental API ifdef and not compiled-in by default. //Let me know if there is more formal way to submit such requests. Best regards, Ilya Maximets.