From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4392745629; Wed, 17 Jul 2024 04:27:54 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B49CB40299; Wed, 17 Jul 2024 04:27:53 +0200 (CEST) Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) by mails.dpdk.org (Postfix) with ESMTP id 9B36B4025F for ; Wed, 17 Jul 2024 04:27:52 +0200 (CEST) Received: by mail-pg1-f181.google.com with SMTP id 41be03b00d2f7-75e7e110e89so3725774a12.3 for ; Tue, 16 Jul 2024 19:27:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20230601.gappssmtp.com; s=20230601; t=1721183272; x=1721788072; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=EYqXws+63PTLAeh8YPJbtgTY8xbK6qbQe61oDa2kMpY=; b=DXKkp5MPemgvzBtFnDmONMatMLa1OLHkEMBj/hecvT/C/YgwVSTJmBRKMpRSrsxV6W qiVOIkRYY+yS2X6cSpVb/dVZhIFW8CSNWG5YxqRBQRIkI9ZLJirkWg7AAJGfc5D1UOwZ tn63KhMAbfix82vVZR2U/p2YmsZMgA8RYa+/pKGNtXJ/s6gBaQZHkkEV3vgwkbd4P5NE x1uSnWgZNgqqgTIcBeCm1L0bKvBJNTHdWWeyK1H0eUcbgtgFUdhg4fPpMzz7Gf1x2Oee cABO+miqoblc/i6Zcgj8Bi9Wp7GQMMqWt9E9w+QpBDM9FZJf1Cw5Abmw2RRDAST4d3Xt UciA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721183272; x=1721788072; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EYqXws+63PTLAeh8YPJbtgTY8xbK6qbQe61oDa2kMpY=; b=Dnhs+56gEpW9w5YYVfcclA+kqILaOO5nJ/xhSG+VB0nFD7y41Zd86UQn2gWkwJ7a/S +ny8cJ+KNLEs5WAMjv7mr+O8v9FsIE/uK/vsFfkRPyUcSzIxF3YkBpHhgvuehpKTd712 BAQ1kYsSs/348Ypo5P3Z3YQUdy26cs+DPEjB9PEq2NBhGVAZRW36leCUP/a8l0sJ0UiY xa1r7DJHZmXPq7Mkfj3EVt4LzUDxpnIA8F0uJsZx51q/Xl2U1MsLOLBqql6l/Ys+wRqi DMiEnX2+/NEn1LqCo5DQxadhJ/2oP0qKP+xZ52KnBbmSkwxPmBNT6oLzdocgbzX51Bjs qQMQ== X-Gm-Message-State: AOJu0YyLj+9pNkBHj2fblQQrbFmt9VlyaQGsq28GzMVfclQH3dplPFZI rGncsuEY9L7a4WlRzr3RPdTRQcOEHAFe8HG42AIu3qQOvTgaNw3bQjG44dLmR3M= X-Google-Smtp-Source: AGHT+IF7CZ5Msvaq+BOrR+4Hr0pecTDk7z228hClCh0F0QXJA+XsZlLf2CbiH1lG9+S00hJtMbCI1Q== X-Received: by 2002:a05:6a20:7f93:b0:1c3:ff33:2837 with SMTP id adf61e73a8af0-1c3ff3328dbmr77351637.54.1721183271688; Tue, 16 Jul 2024 19:27:51 -0700 (PDT) Received: from hermes.local (204-195-96-226.wavecable.com. [204.195.96.226]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2cacd7037d6sm9026519a91.53.2024.07.16.19.27.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 19:27:51 -0700 (PDT) Date: Tue, 16 Jul 2024 19:27:49 -0700 From: Stephen Hemminger To: Wathsala Vithanage Cc: dev@dpdk.org, Thomas Monjalon , Ferruh Yigit , Andrew Rybchenko , nd@arm.com, Dhruv Tripathi Subject: Re: [RFC v2] ethdev: an API for cache stashing hints Message-ID: <20240716192749.57c2cba8@hermes.local> In-Reply-To: <20240715221141.16153-1-wathsala.vithanage@arm.com> References: <20240715221141.16153-1-wathsala.vithanage@arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Mon, 15 Jul 2024 22:11:41 +0000 Wathsala Vithanage wrote: > An application provides cache stashing hints to the ethernet devices to > improve memory access latencies from the CPU and the NIC. This patch > introduces three distinct hints for this purpose. >=20 > The RTE_ETH_DEV_STASH_HINT_HOST_WILLNEED hint indicates that the host > (CPU) requires the data written by the NIC immediately. This implies > that the CPU expects to read data from its local cache rather than LLC > or main memory if possible. This would improve memory access latency in > the Rx path. For PCI devices with TPH capability, these hints translate > into DWHR (Device Writes Host Reads) access pattern. This hint is only > valid for receive queues. >=20 > The RTE_ETH_DEV_STASH_HINT_BI_DIR_DATA hint indicates that the host and > the device access the data structure equally. Rx/Tx queue descriptors > fit the description of such data. This hint applies to both Rx and Tx > directions.=C2=A0 In the PCI TPH context, this hint translates into a > Bi-Directional access pattern. >=20 > RTE_ETH_DEV_STASH_HINT_DEV_ONLY hint indicates that the CPU is not > involved in a given device's receive or transmit paths. This implies > that only devices are involved in the IO path. Depending on the > implementation, this hint may result in data getting placed in a cache > close to the device or not cached at all. For PCI devices with TPH > capability, this hint translates into D*D* (DWDR, DRDW, DWDW, DRDR) > access patterns. This is a bidirectional hint, and it can be applied to > both Rx and Tx queues.=C2=A0=C2=A0 >=20 > The RTE_ETH_DEV_STASH_HINT_HOST_DONTNEED hint indicates that the device > reads data written by the host (CPU) that may still be in the host's > local cache but is not required by the host anytime soon. This hint is > intended to prevent unnecessary cache invalidations that cause > interconnect latencies when a device writes to a buffer already in host > cache memory. In DPDK, this could happen with the recycling of mbufs > where a mbuf is placed in the Tx queue that then gets back into mempool > and gets recycled back into the Rx queue, all while a copy is being held > in the CPU's local cache unnecessarily. By using this hint on supported > platforms, the mbuf will be invalidated after the device completes the > buffer reading, but it will be well before the buffer gets recycled and > updated in the Rx path. This hint is only valid for transmit queues.=C2=A0 >=20 > Applications use three main interfaces in the ethdev library to discover > and set cache stashing hints. rte_eth_dev_stashing_hints_tx interface is > used to set hints on a Tx queue. rte_eth_dev_stashing_hints_rx interface > is used to set hints on an Rx queue. Both of these functions take the > following parameters as inputs: a port_id (the id of the ethernet > device), a cpu_id (the target CPU), a cache_level (the level of the > cache hierarchy the data should be stashed into), a queue_id (the queue > the hints are applied to). In addition to the above list of parameters, > a type parameter indicates the type of the object the application > expects to be stashed by the hardware. Depending on the hardware, these > may vary. Intel E810 NICs support the stashing of Rx/Tx descriptors, > packet headers, and packet payloads. These are indicated by the macros > RTE_ETH_DEV_STASH_TYPE_DESC, RTE_ETH_DEV_STASH_TYPE_HEADER, > RTE_ETH_DEV_STASH_TYPE_PAYLOAD. Hardware capable of stashing data at any > given offset into a packet can use the RTE_ETH_DEV_STASH_TYPE_OFFSET > type. When an offset is used, the offset parameter in the above two > functions should be set appropriately. >=20 > rte_eth_dev_stashing_hints_discover is used to discover the object types > and hints supported in the platform and the device. The function takes > types and hints pointers used as a bit vector to indicate hints and > types supported by the NIC. An application that intends to use stashing > hints should first discover supported hints and types and then use the > functions rte_eth_dev_stashing_hints_tx and > rte_eth_dev_stashing_hints_rx as required to set stashing hints > accordingly. eth_dev_ops structure has been updated with two new ops > that a PMD should implement to support cache stashing hints. A PMD that > intends to support cache stashing hints should initialize the > set_stashing_hints function pointer to a function that issues hints to > the underlying hardware in compliance with platform capabilities. The > same PMD should also implement a function that can return two-bit fields > indicating supported types and hints and then initialize the > discover_stashing_hints function pointer with it. If the NIC supports > cache stashing hints, the NIC should always set the > RTE_ETH_DEV_CAPA_CACHE_STASHING device capability. >=20 > Signed-off-by: Wathsala Vithanage > Reviewed-by: Dhruv Tripathi My initial reaction is negative on this. The DPDK does not need more nerd k= nobs for performance. If it is a performance win, it should be automatic and han= dled by the driver. If you absolutely have to have another flag, then it should be in existing = config (yes, extend the ABI) rather than adding more flags and calls in ethdev.