From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3F66FA0032; Wed, 16 Mar 2022 10:41:26 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2C119410F3; Wed, 16 Mar 2022 10:41:26 +0100 (CET) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by mails.dpdk.org (Postfix) with ESMTP id A94E540395 for ; Wed, 16 Mar 2022 10:41:24 +0100 (CET) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 2E4095C0092; Wed, 16 Mar 2022 05:41:24 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Wed, 16 Mar 2022 05:41:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= cc:cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm1; bh=q1cFQJP8xp2IhP /fEUVG215yRd/BOdMfUbC0XdsSKzA=; b=YoVm/oyGBW/Xx2rJR1DKnTVN7l5vbi sNgK8kUEN/4bdIpILK4IkjbnKf3dHVLef9RYbm3hRNovDx06wt8pfXqTHrekE8v5 lkOLezk0g59sJOHwMNk+I+1Uql/DIi54jMrp9n9EBYgFWhcjV2/pQfD5HEYTiS8v ByYDkrJM8yrVILWT74crGf0AWLl3qPZN5pvczzKLi3Tqe67dN39z0ulpUjJIbgVO yeg3umzNsE9V9M4+9EftPN4MFETTISpaO7YClL3pnog1riHZD5/7XqTXPy+jvsaE zf7Y2fAd6hQ2Gkk7YIocR5iWr9IjAXUFmb9jKLp+zBvuLGEHoLewx1qQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=q1cFQJP8xp2IhP/fEUVG215yRd/BOdMfUbC0XdsSK zA=; b=cHeSN73EtC1xOj7j2bL51ZI/HZKFxZ0LYnrERpWFepv83gNWQT/IGRfaU sRnHe6+f37st8ha5BMODaUFVXQjVl69vZ6C0tvUR0dUO2yGwY+q45Rxb5rbX4N/D jcS85DfLAwQbRFAe7rKi6cJ38K2WFiU0W35zUc5kaON6geyC39OB/ljwq2dvAbiG eFJaJABLSl9Wh/3MAo/vCq/R3lSGpdwzo7YEKAINpA0qEjmH836RIvDzFfvY2qpX hDj9mjaFsja0Fl45o8G/TUVOCKpeeK7Dcd59W1dRdc7OXxa0kt3e0+Sy2WHyjGZ5 wVnQdMVYuLsQyiUk6uUFEy6gICUKQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrudefvddgtdehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkfgjfhgggfgtsehtufertddttddvnecuhfhrohhmpefvhhhomhgr shcuofhonhhjrghlohhnuceothhhohhmrghssehmohhnjhgrlhhonhdrnhgvtheqnecugg ftrfgrthhtvghrnhepjeetgfeludfftdevvdejgeelheehtddvffffhefhudfgiefhffeg udegjeefveetnecuffhomhgrihhnpehophgvnhhvshifihhttghhrdhorhhgpdhgihhthh husgdrtghomhenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhr ohhmpehthhhomhgrshesmhhonhhjrghlohhnrdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 16 Mar 2022 05:41:22 -0400 (EDT) From: Thomas Monjalon To: Ilya Maximets Cc: dev , Sriharsha Basavapatna , Gaetan Rivet , Eli Britstein , Ivan Malov , Andrew Rybchenko , Ori Kam , Ian Stokes Subject: Re: rte_flow API change request: tunnel info restoration is too slow Date: Wed, 16 Mar 2022 10:41:20 +0100 Message-ID: <6043769.DvuYhMxLoT@thomas> In-Reply-To: <5248c2ca-f2a6-3fb0-38b8-7f659bfa40de@ovn.org> References: <5248c2ca-f2a6-3fb0-38b8-7f659bfa40de@ovn.org> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org 15/03/2022 23:12, Ilya Maximets: > Hi, everyone. > > After implementing support for tunnel offloading in OVS we faced a > significant performance issue caused by the requirement to call > rte_flow_get_restore_info() function on a per-packet basis. > > The main problem is that once the tunnel offloading is configured, > there is no way for the application to tell if a particular packet > was partially processed in the hardware and the tunnel info has to > be restored. What we have to do right now is to call the > rte_flow_get_restore_info() unconditionally for every packet. The > result of that call says if we have the tunnel info or not. > > rte_flow_get_restore_info() call itself is very heavy. It is at > least a couple of indirect function calls and the device lock > on the application side (not-really-thread-safety of the rte_flow > API is a separate topic). Internal info lookup inside the driver > also costs a lot (depends on a driver). > > It has been measured that having this call on a per-packet basis can > reduce application performance (OVS) by a factor of 3 in some cases: > https://mail.openvswitch.org/pipermail/ovs-dev/2021-November/389265.html > https://github.com/openvswitch/ovs/commit/6e50c1651869de0335eb4b7fd0960059c5505f5c > (Above patch avoid the problem in a hacky way for devices that doesn't > support tunnel offloading, but it's not applicable to situation > where device actually supports it, since the API has to be called.) > > Another tricky part is that we have to call rte_flow_get_restore_info() > before checking other parts of the mbuf, because mlx5 driver, for > example, re-uses the mbuf.hash.fdir value for both tunnel info > restoration and classification offloading, so the application has > no way to tell which one is used right now and has to call the > restoration API first in order to find out. > > > What we need: > > A generic and fast (couple of CPU cycles) API that will clearly say > if the heavy rte_flow_get_restore_info() has to be called for a > particular packet or not. Ideally, that should be a static mbuf > flag that can be easily checked by the application. A dynamic mbuf flag, defined in the API, looks to be a good idea. > Calls inside the device driver are way too slow for that purpose, > especially if they are not fully thread-safe, or require complex > lookups or calculations. > > I'm assuming here that packets that really need the tunnel info > restoration should be fairly rare. > > > Current state: > > Currently, the get_restore_info() API is implemented only for mlx5 > and sfc drivers, AFAICT. > SFC driver is already using mbuf flag, but > it's dynamic and not exposed to the application. What is this flag? > MLX5 driver re-uses mbuf.hash.fdir value > and performs a heavy lookup inside the driver. We should avoid re-using a field. > For now OVS doesn't support tunnel offload with DPDK formally, the > code in OVS is under the experimental API ifdef and not compiled-in > by default. > > //Let me know if there is more formal way to submit such requests. That's a well written request, thanks. If you are looking for something more formal, it could be a patch in the API.