From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from dpdk.org (dpdk.org [92.243.14.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 62386A04DC;
	Mon, 19 Oct 2020 15:01:46 +0200 (CEST)
Received: from [92.243.14.124] (localhost [127.0.0.1])
	by dpdk.org (Postfix) with ESMTP id 6C797C95A;
	Mon, 19 Oct 2020 15:01:43 +0200 (CEST)
Received: from mga11.intel.com (mga11.intel.com [192.55.52.93])
 by dpdk.org (Postfix) with ESMTP id 38768C92A
 for <dev@dpdk.org>; Mon, 19 Oct 2020 15:01:41 +0200 (CEST)
IronPort-SDR: sq9g7vxxeNI4Lfd66UeYwRYDdJtnz4xp52VUG9UUUHP/mpeFKwG/XS+3r7x/U/HVl1fLksZV4X
 cF+4CuBQBXsw==
X-IronPort-AV: E=McAfee;i="6000,8403,9778"; a="163535754"
X-IronPort-AV: E=Sophos;i="5.77,394,1596524400"; d="scan'208";a="163535754"
X-Amp-Result: SKIPPED(no attachment in message)
X-Amp-File-Uploaded: False
Received: from orsmga006.jf.intel.com ([10.7.209.51])
 by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;
 19 Oct 2020 06:01:38 -0700
IronPort-SDR: CuLgNJCwBpE7RcHFZ1zopI8TKv0u6WSlIim+7hUj+GUbC0aIARdTKWu8TRRP5U/9MVyhZeh8no
 0KcH69++rfPQ==
X-IronPort-AV: E=Sophos;i="5.77,394,1596524400"; d="scan'208";a="320264256"
Received: from bricha3-mobl.ger.corp.intel.com ([10.213.11.189])
 by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-SHA;
 19 Oct 2020 06:01:36 -0700
Date: Mon, 19 Oct 2020 14:01:30 +0100
From: Bruce Richardson <bruce.richardson@intel.com>
To: Ali Alnubani <alialnu@nvidia.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>,
 NBU-Contact-Thomas Monjalon <thomas@monjalon.net>,
 Asaf Penso <asafp@nvidia.com>
Message-ID: <20201019130130.GA663@bricha3-MOBL.ger.corp.intel.com>
References: <DM6PR12MB4618640B2ABCFA4D3B33C445DA020@DM6PR12MB4618.namprd12.prod.outlook.com>
 <20201015170804.GG554@bricha3-MOBL.ger.corp.intel.com>
 <20201016095910.GD1008@bricha3-MOBL.ger.corp.intel.com>
 <DM6PR12MB4618C29C72D9D21C8DB846B0DA1E0@DM6PR12MB4618.namprd12.prod.outlook.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <DM6PR12MB4618C29C72D9D21C8DB846B0DA1E0@DM6PR12MB4618.namprd12.prod.outlook.com>
Subject: Re: [dpdk-dev] performance degradation with fpic
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>

On Mon, Oct 19, 2020 at 11:47:48AM +0000, Ali Alnubani wrote:
> Hi Bruce,
> 
> > -----Original Message-----
> > From: Bruce Richardson <bruce.richardson@intel.com>
> > Sent: Friday, October 16, 2020 12:59 PM
> > To: Ali Alnubani <alialnu@nvidia.com>
> > Cc: dev@dpdk.org; NBU-Contact-Thomas Monjalon
> > <thomas@monjalon.net>; Asaf Penso <asafp@nvidia.com>
> > Subject: Re: [dpdk-dev] performance degradation with fpic
> > 
> > On Thu, Oct 15, 2020 at 06:08:04PM +0100, Bruce Richardson wrote:
> > > On Thu, Oct 15, 2020 at 04:00:44PM +0000, Ali Alnubani wrote:
> > > >    Hi Bruce,
> > > >
> > > >
> > > >    We have been seeing in some cases that the DPDK forwarding
> > performance
> > > >    is up to 9% lower when DPDK is built as static with meson compared to a
> > > >    build with makefiles.
> > > >
> > > >
> > > >    The same degradation can be reproduced with makefiles on older DPDK
> > > >    releases when building with EXTAR_CFLAGS set to “-fPIC”, it can also be
> > > >    resolved in meson when passing “pic: false” to meson’s static_library
> > > >    call (more tweaking needs to be done to prevent building shared
> > > >    libraries because this change breaks them).
> > > >
> > > >
> > > >    I can reproduce this drop with the following cases:
> > > >      * Baremetal / NIC: ConnectX-4 Lx / OS: RHEL7.4 / CPU: Intel(R)
> > > >        Xeon(R) Gold 6154. Testpmd command:
> > > >
> > > >    testpmd -c 0x7ffc0000 -n 4 -w d8:00.1 -w d8:00.0 --socket-
> > mem=2048,2048
> > > >    -- --port-numa-config=0,1,1,1 --socket-num=1 --burst=64 --txd=512
> > > >    --rxd=512 --mbcache=512 --rxq=2 --txq=2 --nb-cores=1 --no-lsc-
> > interrupt
> > > >    -i -a --rss-udp
> > > >      * KVM guest with SR-IOV passthrough / OS: RHEL7.4 / NIC: ConnectX-5
> > /
> > > >        Host’s CPU: Intel(R) Xeon(R) Gold 6154. Testpmd command:
> > > >        testpmd --master-lcore=0 -c 0x1ffff -n 4 -w
> > > >        00:05.0,mprq_en=1,mprq_log_stride_num=6 --socket-mem=2048,0 --
> > > >        --port-numa-config=0,0 --socket-num=0 --burst=64 --txd=1024
> > > >        --rxd=1024 --mbcache=512 --rxq=16 --txq=16 --nb-cores=8
> > > >        --port-topology=chained --forward-mode=macswap --no-lsc-
> > interrupt
> > > >        -i -a --rss-udp
> > > >      * Baremetal / OS: Ubuntu 18.04 / NIC: ConnectX-5 / CPU: Intel(R)
> > > >        Xeon(R) CPU E5-2697A v4. Testpmd command:
> > > >        testpmd -n 4  -w 0000:82:00.0,rxqs_min_mprq=8,mprq_en=1  -w
> > > >        0000:82:00.1,rxqs_min_mprq=8,mprq_en=1 -c 0xff80  -- --burst=64
> > > >        --mbcache=512 -i  --nb-cores=8  --rxq=8 --txq=8 --txd=1024
> > > >        --rxd=1024 --rss-udp --auto-start
> > > >
> > > >    The packets being received and forwarded by testpmd are of IPv4/UDP
> > > >    type and 64B size.
> > > >
> > > >    Should we disable PIC in static builds?
> > > >
> > > >
> > >
> > > Hi Ali,
> > >
> > > thanks for reporting, though it's strange that you see such a big impact.
> > > In my previous tests with i40e driver I never noticed a difference
> > > between make and meson builds, and I and some others here have been
> > > using meson builds for any performance work for over a year now. That
> > > being said let me reverify what I see on my end.
> > >
> > > In terms of solutions, disabling the -fPIC flag globally implies that
> > > we can no longer build static and shared libs from the same sources,
> > > so we would need to revert to doing either a static or a shared
> > > library build but not both. If the issue is limited to only some
> > > drivers or some cases, we can perhaps add in a build option to have
> > > no-fpic-static builds, to be used in a cases where it is problematic.
> > >
> > > However, at this point, I think we need a little more investigation.
> > > Is there any testing you can do to see if it's just in your driver, or
> > > in perhaps a mempool driver/lib that the issue appears, or if it's
> > > just a global slowdown? Do you see the impact with both clang and gcc?
> > > I'll retest things a bit tomorrow on my end to see what I see.
> > >
> > Hi again,
> > 
> > I've done a quick retest with the i40e driver on my system, using the 20.08
> > version so as to have make vs meson direct comparison. [For reference
> > command used was: "sudo </path/to/testpmd>  -c F00000 -w af:00.0 -w
> > b1:00.0 -w da:00.0 -- --rxq=2 --txq=2 --rxd=2048 --txd=512" using 3x40G ports
> > to a single core running @3GHz.] No major performance differences were
> > seen, but if anything the meson build was very slightly faster, as reported to
> > Jerin, maybe 2%, though it's within the margin of error.
> > 
> 
> Thanks for taking the time to investigate this.
> 
> Disabling PIC for net/mlx5 driver alone in drivers/meson.build resolves the issue for me.
> I saw this issue with gcc (tested with 4.8.5, 9.3.0, and 7.5.0). But I see now that disabling PIC with an old clang version (clang 3.4.2, RHEL7.4) causes a drop in performance, not an improvement like with gcc.
> 
That's interesting.

When you just build with and without -fpic with newer clang, do you see the
same perf drop as with gcc? With the older clang, is the shared lib build
faster than the static one?