From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 3E203A00C4;
	Wed, 28 Sep 2022 14:52:57 +0200 (CEST)
Received: from [217.70.189.124] (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id 359AA41140;
	Wed, 28 Sep 2022 14:52:57 +0200 (CEST)
Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com
 [209.85.221.44]) by mails.dpdk.org (Postfix) with ESMTP id B02804280C
 for <dev@dpdk.org>; Wed, 28 Sep 2022 14:52:55 +0200 (CEST)
Received: by mail-wr1-f44.google.com with SMTP id v28so6816518wrd.3
 for <dev@dpdk.org>; Wed, 28 Sep 2022 05:52:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind.com; s=google;
 h=in-reply-to:content-disposition:mime-version:references:message-id
 :subject:cc:to:from:date:from:to:cc:subject:date;
 bh=31wJPIKWg3Fk8Edmpn4Tp3kO4c2IWpqgax/Z6BYFs6I=;
 b=GiaAYkLSJgNI3vnmh+GOaApIsBWFv2dEX3V/rc0rNsl3kH4/ffzdviVjbIeAr99F0L
 qAtwJ7jMbCtF34W5KdnWh+7fg80YuqrZAwhKRpOyBi+Bw8iEcIZf9xcGRVJ+M4MGC03W
 2xgQNCBMRKKOEn1k9+ONqOm+2LpRTBcOuCL8RzCfR7naxDDkRBRhFvNRK0z9m+TwW1BP
 EFZGX7voqarix3oEj07cdkftGv6vujrHI33KfRaKqTW+yxl1llyt+vYRyO198afM2jOP
 e7wuw8V1lAXdaBkASFY0zRspzAtJm4qMu09sD24BpF7a7aWgkRlc5FjH8dyJQpwTuk99
 e4Tw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=in-reply-to:content-disposition:mime-version:references:message-id
 :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date;
 bh=31wJPIKWg3Fk8Edmpn4Tp3kO4c2IWpqgax/Z6BYFs6I=;
 b=ru/3o/Kgdb+Koucbt+KapP02L+8n9pBCfuaijUxgC4HIullBpf3XPX2XwTWzi7kqcP
 RmvLcBVnnO34TBn+qVNPPM8NGfV4ZbGSjGKkyTNxNCZh4P5oBOrPfRPTPzQg4Z2K97NO
 5MEQd6iG52SBocCvZtIAStpS/+MsYq14lzdw4IfTp+K4+taYpJWArG/3Ea0eprgi7cjn
 dIFjnUktIPvXZP1e7Vlp75vyNCACbXLxFwAL9ifnZuSE/wVKv3g/ViK0yihgFodWOUhK
 g4QtNyqNnOOI83oQboXuuD55G46e6an6Tq2Nyd/maKwU9qMugkMesT4ReDpQtcHGNrp5
 WSAw==
X-Gm-Message-State: ACrzQf2OczdHpkSo4t59dQzFCfLY3b0hjH0utK44EAajZBw2yxN6oKoZ
 B1CdO+ziQPhgciPxfgPvDz0YPA==
X-Google-Smtp-Source: AMsMyM4myPYfPAub8hcBzwdYWLUAhyCWl8zKYtknwtZ5h1k/cELowl4vZa6Lsh2u2O5ezPLn517MFQ==
X-Received: by 2002:a5d:46c7:0:b0:22c:c9e5:4e4e with SMTP id
 g7-20020a5d46c7000000b0022cc9e54e4emr1660391wrs.37.1664369575399; 
 Wed, 28 Sep 2022 05:52:55 -0700 (PDT)
Received: from 6wind.com ([2a01:e0a:5ac:6460:c065:401d:87eb:9b25])
 by smtp.gmail.com with ESMTPSA id
 a20-20020a05600c069400b003a5fa79007fsm1607041wmn.7.2022.09.28.05.52.54
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Wed, 28 Sep 2022 05:52:55 -0700 (PDT)
Date: Wed, 28 Sep 2022 14:52:54 +0200
From: Olivier Matz <olivier.matz@6wind.com>
To: Shijith Thotton <sthotton@marvell.com>
Cc: dev@dpdk.org, pbhagavatula@marvell.com, Honnappa.Nagarahalli@arm.com,
 bruce.richardson@intel.com, jerinj@marvell.com,
 mb@smartsharesystems.com, stephen@networkplumber.org,
 thomas@monjalon.net, david.marchand@redhat.com
Subject: Re: [PATCH v3 3/5] lib: move mbuf next pointer to first cache line
Message-ID: <YzRDpibcDrcUh7j8@platinum>
References: <20220907134340.3629224-1-sthotton@marvell.com>
 <cover.1663767715.git.sthotton@marvell.com>
 <7cc8107a4a83dbfe16abd1f24150b0db6cf1041b.1663767715.git.sthotton@marvell.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <7cc8107a4a83dbfe16abd1f24150b0db6cf1041b.1663767715.git.sthotton@marvell.com>
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org

On Wed, Sep 21, 2022 at 07:26:19PM +0530, Shijith Thotton wrote:
> Swapped position of mbuf next pointer and second dynamic field (dynfield2)
> if the build is configured to use IOVA as VA. This is to move the mbuf
> next pointer to first cache line. kni library is disabled for this
> change as it depends on the offset value of next pointer.
> 
> Signed-off-by: Shijith Thotton <sthotton@marvell.com>
> ---
>  lib/mbuf/rte_mbuf_core.h | 29 +++++++++++++++++++++--------
>  lib/meson.build          |  3 +++
>  2 files changed, 24 insertions(+), 8 deletions(-)
> 
> diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
> index 94907f301d..915dcd8653 100644
> --- a/lib/mbuf/rte_mbuf_core.h
> +++ b/lib/mbuf/rte_mbuf_core.h
> @@ -590,11 +590,14 @@ struct rte_mbuf {
>  		 * working on vector drivers easier.
>  		 */
>  		rte_iova_t buf_iova __rte_aligned(sizeof(rte_iova_t));
> +#if RTE_IOVA_AS_VA
>  		/**
> -		 * Reserved for dynamic field in builds where physical address
> -		 * field is invalid.
> +		 * Next segment of scattered packet.
> +		 * This field is valid when physical address field is invalid.
> +		 * Otherwise next pointer in the second cache line will be used.
>  		 */
> -		uint64_t dynfield2;
> +		struct rte_mbuf *next;
> +#endif
>  	};
>  
>  	/* next 8 bytes are initialised on RX descriptor rearm */
> @@ -711,11 +714,21 @@ struct rte_mbuf {
>  	/* second cache line - fields only used in slow path or on TX */
>  	RTE_MARKER cacheline1 __rte_cache_min_aligned;
>  
> -	/**
> -	 * Next segment of scattered packet. Must be NULL in the last segment or
> -	 * in case of non-segmented packet.
> -	 */
> -	struct rte_mbuf *next;
> +	RTE_STD_C11
> +	union {
> +#if !RTE_IOVA_AS_VA
> +		/**
> +		 * Next segment of scattered packet. Must be NULL in the last
> +		 * segment or in case of non-segmented packet.
> +		 */
> +		struct rte_mbuf *next;
> +#endif
> +		/**
> +		 * Reserved for dynamic field when the next pointer is in first
> +		 * cache line (i.e. RTE_IOVA_AS_VA is 1).
> +		 */
> +		uint64_t dynfield2;
> +	};

Same comment than other patches about union vs #if.

>  
>  	/* fields to support TX offloads */
>  	RTE_STD_C11
> diff --git a/lib/meson.build b/lib/meson.build
> index c648f7d800..73d93bc803 100644
> --- a/lib/meson.build
> +++ b/lib/meson.build
> @@ -88,6 +88,9 @@ optional_libs = [
>  disabled_libs = []
>  opt_disabled_libs = run_command(list_dir_globs, get_option('disable_libs'),
>          check: true).stdout().split()
> +if dpdk_conf.get('RTE_IOVA_AS_VA') == 1
> +    opt_disabled_libs += ['kni']
> +endif

I guess this should be in the previous patch instead, since kni uses
m->buf_iova

>  foreach l:opt_disabled_libs
>      if not optional_libs.contains(l)
>          warning('Cannot disable mandatory library "@0@"'.format(l))


After this change, the documentation of RTE_IOVA_AS_VA can be enhanced to
explain that it also moves the next pointer to the first cache line, possibly
increasing the performance.