From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 87542A04DB; Fri, 16 Oct 2020 11:59:24 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 03F6D1EB5E; Fri, 16 Oct 2020 11:59:22 +0200 (CEST) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by dpdk.org (Postfix) with ESMTP id 57E331EB48 for ; Fri, 16 Oct 2020 11:59:18 +0200 (CEST) IronPort-SDR: Q9AIF0MFFIf6/aXE1nhmYO3KQ+WSwqkPeMTESByt1ftEVPj1qVXNeq8M4rDGKnLcRO4EMPR1I3 OKGAptgM30ig== X-IronPort-AV: E=McAfee;i="6000,8403,9775"; a="184133471" X-IronPort-AV: E=Sophos;i="5.77,382,1596524400"; d="scan'208";a="184133471" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Oct 2020 02:59:17 -0700 IronPort-SDR: 4gye4Hiez8fh9CGz93M+n29uYJuu8Qb5t3WofXgul/e1anbpvJki++DpbN2xdwVi7UBa17DhF7 kAONaImIkmFQ== X-IronPort-AV: E=Sophos;i="5.77,382,1596524400"; d="scan'208";a="531658052" Received: from bricha3-mobl.ger.corp.intel.com ([10.252.54.208]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-SHA; 16 Oct 2020 02:59:15 -0700 Date: Fri, 16 Oct 2020 10:59:10 +0100 From: Bruce Richardson To: Ali Alnubani Cc: "dev@dpdk.org" , NBU-Contact-Thomas Monjalon , Asaf Penso Message-ID: <20201016095910.GD1008@bricha3-MOBL.ger.corp.intel.com> References: <20201015170804.GG554@bricha3-MOBL.ger.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20201015170804.GG554@bricha3-MOBL.ger.corp.intel.com> Subject: Re: [dpdk-dev] performance degradation with fpic X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Thu, Oct 15, 2020 at 06:08:04PM +0100, Bruce Richardson wrote: > On Thu, Oct 15, 2020 at 04:00:44PM +0000, Ali Alnubani wrote: > > Hi Bruce, > > > > > > We have been seeing in some cases that the DPDK forwarding performance > > is up to 9% lower when DPDK is built as static with meson compared to a > > build with makefiles. > > > > > > The same degradation can be reproduced with makefiles on older DPDK > > releases when building with EXTAR_CFLAGS set to “-fPIC”, it can also be > > resolved in meson when passing “pic: false” to meson’s static_library > > call (more tweaking needs to be done to prevent building shared > > libraries because this change breaks them). > > > > > > I can reproduce this drop with the following cases: > > * Baremetal / NIC: ConnectX-4 Lx / OS: RHEL7.4 / CPU: Intel(R) > > Xeon(R) Gold 6154. Testpmd command: > > > > testpmd -c 0x7ffc0000 -n 4 -w d8:00.1 -w d8:00.0 --socket-mem=2048,2048 > > -- --port-numa-config=0,1,1,1 --socket-num=1 --burst=64 --txd=512 > > --rxd=512 --mbcache=512 --rxq=2 --txq=2 --nb-cores=1 --no-lsc-interrupt > > -i -a --rss-udp > > * KVM guest with SR-IOV passthrough / OS: RHEL7.4 / NIC: ConnectX-5 / > > Host’s CPU: Intel(R) Xeon(R) Gold 6154. Testpmd command: > > testpmd --master-lcore=0 -c 0x1ffff -n 4 -w > > 00:05.0,mprq_en=1,mprq_log_stride_num=6 --socket-mem=2048,0 -- > > --port-numa-config=0,0 --socket-num=0 --burst=64 --txd=1024 > > --rxd=1024 --mbcache=512 --rxq=16 --txq=16 --nb-cores=8 > > --port-topology=chained --forward-mode=macswap --no-lsc-interrupt > > -i -a --rss-udp > > * Baremetal / OS: Ubuntu 18.04 / NIC: ConnectX-5 / CPU: Intel(R) > > Xeon(R) CPU E5-2697A v4. Testpmd command: > > testpmd -n 4 -w 0000:82:00.0,rxqs_min_mprq=8,mprq_en=1 -w > > 0000:82:00.1,rxqs_min_mprq=8,mprq_en=1 -c 0xff80 -- --burst=64 > > --mbcache=512 -i --nb-cores=8 --rxq=8 --txq=8 --txd=1024 > > --rxd=1024 --rss-udp --auto-start > > > > The packets being received and forwarded by testpmd are of IPv4/UDP > > type and 64B size. > > > > Should we disable PIC in static builds? > > > > > > Hi Ali, > > thanks for reporting, though it's strange that you see such a big impact. > In my previous tests with i40e driver I never noticed a difference between > make and meson builds, and I and some others here have been using meson > builds for any performance work for over a year now. That being said let me > reverify what I see on my end. > > In terms of solutions, disabling the -fPIC flag globally implies that we > can no longer build static and shared libs from the same sources, so we > would need to revert to doing either a static or a shared library build > but not both. If the issue is limited to only some drivers or some cases, > we can perhaps add in a build option to have no-fpic-static builds, to be > used in a cases where it is problematic. > > However, at this point, I think we need a little more investigation. Is > there any testing you can do to see if it's just in your driver, or in > perhaps a mempool driver/lib that the issue appears, or if it's just a > global slowdown? Do you see the impact with both clang and gcc? I'll > retest things a bit tomorrow on my end to see what I see. > Hi again, I've done a quick retest with the i40e driver on my system, using the 20.08 version so as to have make vs meson direct comparison. [For reference command used was: "sudo -c F00000 -w af:00.0 -w b1:00.0 -w da:00.0 -- --rxq=2 --txq=2 --rxd=2048 --txd=512" using 3x40G ports to a single core running @3GHz.] No major performance differences were seen, but if anything the meson build was very slightly faster, as reported to Jerin, maybe 2%, though it's within the margin of error. Can you try adding '-fno-semantic-interposition' to your build, since reading on the internet it appears that fPIC causes GCC to be very conservative about optimizing things, and that may help. Clang may be less conservative so testing with clang would be good too if you can manage it. Regards, /Bruce