DPDK patches and discussions
 help / color / mirror / Atom feed
From: Bruce Richardson <bruce.richardson@intel.com>
To: Ali Alnubani <alialnu@nvidia.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>,
	NBU-Contact-Thomas Monjalon <thomas@monjalon.net>,
	Asaf Penso <asafp@nvidia.com>
Subject: Re: [dpdk-dev] performance degradation with fpic
Date: Fri, 16 Oct 2020 10:59:10 +0100	[thread overview]
Message-ID: <20201016095910.GD1008@bricha3-MOBL.ger.corp.intel.com> (raw)
In-Reply-To: <20201015170804.GG554@bricha3-MOBL.ger.corp.intel.com>

On Thu, Oct 15, 2020 at 06:08:04PM +0100, Bruce Richardson wrote:
> On Thu, Oct 15, 2020 at 04:00:44PM +0000, Ali Alnubani wrote:
> >    Hi Bruce,
> > 
> > 
> >    We have been seeing in some cases that the DPDK forwarding performance
> >    is up to 9% lower when DPDK is built as static with meson compared to a
> >    build with makefiles.
> > 
> > 
> >    The same degradation can be reproduced with makefiles on older DPDK
> >    releases when building with EXTAR_CFLAGS set to “-fPIC”, it can also be
> >    resolved in meson when passing “pic: false” to meson’s static_library
> >    call (more tweaking needs to be done to prevent building shared
> >    libraries because this change breaks them).
> > 
> > 
> >    I can reproduce this drop with the following cases:
> >      * Baremetal / NIC: ConnectX-4 Lx / OS: RHEL7.4 / CPU: Intel(R)
> >        Xeon(R) Gold 6154. Testpmd command:
> > 
> >    testpmd -c 0x7ffc0000 -n 4 -w d8:00.1 -w d8:00.0 --socket-mem=2048,2048
> >    -- --port-numa-config=0,1,1,1 --socket-num=1 --burst=64 --txd=512
> >    --rxd=512 --mbcache=512 --rxq=2 --txq=2 --nb-cores=1 --no-lsc-interrupt
> >    -i -a --rss-udp
> >      * KVM guest with SR-IOV passthrough / OS: RHEL7.4 / NIC: ConnectX-5 /
> >        Host’s CPU: Intel(R) Xeon(R) Gold 6154. Testpmd command:
> >        testpmd --master-lcore=0 -c 0x1ffff -n 4 -w
> >        00:05.0,mprq_en=1,mprq_log_stride_num=6 --socket-mem=2048,0 --
> >        --port-numa-config=0,0 --socket-num=0 --burst=64 --txd=1024
> >        --rxd=1024 --mbcache=512 --rxq=16 --txq=16 --nb-cores=8
> >        --port-topology=chained --forward-mode=macswap --no-lsc-interrupt
> >        -i -a --rss-udp
> >      * Baremetal / OS: Ubuntu 18.04 / NIC: ConnectX-5 / CPU: Intel(R)
> >        Xeon(R) CPU E5-2697A v4. Testpmd command:
> >        testpmd -n 4  -w 0000:82:00.0,rxqs_min_mprq=8,mprq_en=1  -w
> >        0000:82:00.1,rxqs_min_mprq=8,mprq_en=1 -c 0xff80  -- --burst=64
> >        --mbcache=512 -i  --nb-cores=8  --rxq=8 --txq=8 --txd=1024
> >        --rxd=1024 --rss-udp --auto-start
> > 
> >    The packets being received and forwarded by testpmd are of IPv4/UDP
> >    type and 64B size.
> > 
> >    Should we disable PIC in static builds?
> > 
> > 
> 
> Hi Ali,
> 
> thanks for reporting, though it's strange that you see such a big impact.
> In my previous tests with i40e driver I never noticed a difference between
> make and meson builds, and I and some others here have been using meson
> builds for any performance work for over a year now. That being said let me
> reverify what I see on my end.
> 
> In terms of solutions, disabling the -fPIC flag globally implies that we
> can no longer build static and shared libs from the same sources, so we
> would need to revert to doing either a static or a shared library build
> but not both. If the issue is limited to only some drivers or some cases,
> we can perhaps add in a build option to have no-fpic-static builds, to be
> used in a cases where it is problematic.
> 
> However, at this point, I think we need a little more investigation. Is
> there any testing you can do to see if it's just in your driver, or in
> perhaps a mempool driver/lib that the issue appears, or if it's just a
> global slowdown? Do you see the impact with both clang and gcc?  I'll
> retest things a bit tomorrow on my end to see what I see.
> 
Hi again,

I've done a quick retest with the i40e driver on my system, using the 20.08
version so as to have make vs meson direct comparison. [For reference
command used was: "sudo </path/to/testpmd>  -c F00000 -w af:00.0 -w b1:00.0
-w da:00.0 -- --rxq=2 --txq=2 --rxd=2048 --txd=512" using 3x40G ports to a
single core running @3GHz.] No major performance differences were seen, but
if anything the meson build was very slightly faster, as reported to Jerin,
maybe 2%, though it's within the margin of error.

Can you try adding '-fno-semantic-interposition' to your build, since
reading on the internet it appears that fPIC causes GCC to be very
conservative about optimizing things, and that may help. Clang may be less
conservative so testing with clang would be good too if you can manage it.

Regards,
/Bruce

  parent reply	other threads:[~2020-10-16  9:59 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-10-15 16:00 Ali Alnubani
2020-10-15 17:08 ` Bruce Richardson
2020-10-15 17:14   ` Thomas Monjalon
2020-10-15 17:28     ` [dpdk-dev] [EXT] " Jerin Jacob Kollanukkaran
2020-10-16  8:29       ` Bruce Richardson
2020-10-16  8:39         ` Jerin Jacob
2020-10-15 21:44     ` [dpdk-dev] " Stephen Hemminger
2020-10-16  8:35       ` Bruce Richardson
2020-10-16  9:59   ` Bruce Richardson [this message]
2020-10-19 11:47     ` Ali Alnubani
2020-10-19 13:01       ` Bruce Richardson
2020-10-22 13:17         ` Ali Alnubani
2020-10-22 13:57           ` Bruce Richardson
2020-10-22 14:16             ` Ali Alnubani
2020-11-02 10:40               ` Ali Alnubani
2020-11-02 11:01                 ` Luca Boccassi
2020-11-02 15:00                 ` Bruce Richardson
2020-11-03 10:18                   ` Thomas Monjalon
2020-11-03 10:45                     ` Luca Boccassi
2020-11-03 11:23                     ` Bruce Richardson

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20201016095910.GD1008@bricha3-MOBL.ger.corp.intel.com \
    --to=bruce.richardson@intel.com \
    --cc=alialnu@nvidia.com \
    --cc=asafp@nvidia.com \
    --cc=dev@dpdk.org \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).