Re: [dpdk-dev] performance degradation with fpic

[Ali Alnubani <alialnu@nvidia.com>]

> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Thursday, October 22, 2020 4:58 PM
> To: Ali Alnubani <alialnu@nvidia.com>
> Cc: dev@dpdk.org; NBU-Contact-Thomas Monjalon
> <thomas@monjalon.net>; Asaf Penso <asafp@nvidia.com>
> Subject: Re: [dpdk-dev] performance degradation with fpic
> 
> On Thu, Oct 22, 2020 at 01:17:16PM +0000, Ali Alnubani wrote:
> > Hi Bruce,
> > Sorry for the delayed response.
> >
> > > -----Original Message-----
> > > From: Bruce Richardson <bruce.richardson@intel.com>
> > > Sent: Monday, October 19, 2020 4:02 PM
> > > To: Ali Alnubani <alialnu@nvidia.com>
> > > Cc: dev@dpdk.org; NBU-Contact-Thomas Monjalon
> <thomas@monjalon.net>;
> > > Asaf Penso <asafp@nvidia.com>
> > > Subject: Re: [dpdk-dev] performance degradation with fpic
> > >
> > > On Mon, Oct 19, 2020 at 11:47:48AM +0000, Ali Alnubani wrote:
> > > > Hi Bruce,
> > > >
> > > > > -----Original Message-----
> > > > > From: Bruce Richardson <bruce.richardson@intel.com>
> > > > > Sent: Friday, October 16, 2020 12:59 PM
> > > > > To: Ali Alnubani <alialnu@nvidia.com>
> > > > > Cc: dev@dpdk.org; NBU-Contact-Thomas Monjalon
> > > <thomas@monjalon.net>;
> > > > > Asaf Penso <asafp@nvidia.com>
> > > > > Subject: Re: [dpdk-dev] performance degradation with fpic
> > > > >
> > > > > On Thu, Oct 15, 2020 at 06:08:04PM +0100, Bruce Richardson wrote:
> > > > > > On Thu, Oct 15, 2020 at 04:00:44PM +0000, Ali Alnubani wrote:
> > > > > > >    Hi Bruce,
> > > > > > >
> > > > > > >
> > > > > > >    We have been seeing in some cases that the DPDK
> > > > > > > forwarding
> > > > > performance
> > > > > > >    is up to 9% lower when DPDK is built as static with meson
> > > > > > > compared
> > > to a
> > > > > > >    build with makefiles.
> > > > > > >
> > > > > > >
> > > > > > >    The same degradation can be reproduced with makefiles on
> > > > > > > older
> > > DPDK
> > > > > > >    releases when building with EXTAR_CFLAGS set to “-fPIC”,
> > > > > > > it can also
> > > be
> > > > > > >    resolved in meson when passing “pic: false” to meson’s
> > > static_library
> > > > > > >    call (more tweaking needs to be done to prevent building shared
> > > > > > >    libraries because this change breaks them).
> > > > > > >
> > > > > > >
> > > > > > >    I can reproduce this drop with the following cases:
> > > > > > >      * Baremetal / NIC: ConnectX-4 Lx / OS: RHEL7.4 / CPU: Intel(R)
> > > > > > >        Xeon(R) Gold 6154. Testpmd command:
> > > > > > >
> > > > > > >    testpmd -c 0x7ffc0000 -n 4 -w d8:00.1 -w d8:00.0
> > > > > > > --socket-
> > > > > mem=2048,2048
> > > > > > >    -- --port-numa-config=0,1,1,1 --socket-num=1 --burst=64 --
> txd=512
> > > > > > >    --rxd=512 --mbcache=512 --rxq=2 --txq=2 --nb-cores=1
> > > > > > > --no-lsc-
> > > > > interrupt
> > > > > > >    -i -a --rss-udp
> > > > > > >      * KVM guest with SR-IOV passthrough / OS: RHEL7.4 / NIC:
> > > > > > > ConnectX-5
> > > > > /
> > > > > > >        Host’s CPU: Intel(R) Xeon(R) Gold 6154. Testpmd command:
> > > > > > >        testpmd --master-lcore=0 -c 0x1ffff -n 4 -w
> > > > > > >        00:05.0,mprq_en=1,mprq_log_stride_num=6 --socket-
> > > mem=2048,0 --
> > > > > > >        --port-numa-config=0,0 --socket-num=0 --burst=64 --txd=1024
> > > > > > >        --rxd=1024 --mbcache=512 --rxq=16 --txq=16 --nb-cores=8
> > > > > > >        --port-topology=chained --forward-mode=macswap
> > > > > > > --no-lsc-
> > > > > interrupt
> > > > > > >        -i -a --rss-udp
> > > > > > >      * Baremetal / OS: Ubuntu 18.04 / NIC: ConnectX-5 / CPU:
> Intel(R)
> > > > > > >        Xeon(R) CPU E5-2697A v4. Testpmd command:
> > > > > > >        testpmd -n 4  -w 0000:82:00.0,rxqs_min_mprq=8,mprq_en=1  -
> w
> > > > > > >        0000:82:00.1,rxqs_min_mprq=8,mprq_en=1 -c 0xff80  --
> > > > > > > --
> > > burst=64
> > > > > > >        --mbcache=512 -i  --nb-cores=8  --rxq=8 --txq=8 --txd=1024
> > > > > > >        --rxd=1024 --rss-udp --auto-start
> > > > > > >
> > > > > > >    The packets being received and forwarded by testpmd are
> > > > > > > of
> > > IPv4/UDP
> > > > > > >    type and 64B size.
> > > > > > >
> > > > > > >    Should we disable PIC in static builds?
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > Hi Ali,
> > > > > >
> > > > > > thanks for reporting, though it's strange that you see such a big
> impact.
> > > > > > In my previous tests with i40e driver I never noticed a
> > > > > > difference between make and meson builds, and I and some
> > > > > > others here have been using meson builds for any performance
> > > > > > work for over a year now. That being said let me reverify what I see
> on my end.
> > > > > >
> > > > > > In terms of solutions, disabling the -fPIC flag globally
> > > > > > implies that we can no longer build static and shared libs
> > > > > > from the same sources, so we would need to revert to doing
> > > > > > either a static or a shared library build but not both. If the
> > > > > > issue is limited to only some drivers or some cases, we can
> > > > > > perhaps add in a build option to have no-fpic-static builds,
> > > > > > to be used in a cases where it is
> > > problematic.
> > > > > >
> > > > > > However, at this point, I think we need a little more investigation.
> > > > > > Is there any testing you can do to see if it's just in your
> > > > > > driver, or in perhaps a mempool driver/lib that the issue
> > > > > > appears, or if it's just a global slowdown? Do you see the
> > > > > > impact with both clang
> > > and gcc?
> > > > > > I'll retest things a bit tomorrow on my end to see what I see.
> > > > > >
> > > > > Hi again,
> > > > >
> > > > > I've done a quick retest with the i40e driver on my system,
> > > > > using the 20.08 version so as to have make vs meson direct
> comparison.
> > > > > [For reference command used was: "sudo </path/to/testpmd>  -c
> > > > > F00000 -w af:00.0 -w
> > > > > b1:00.0 -w da:00.0 -- --rxq=2 --txq=2 --rxd=2048 --txd=512"
> > > > > using 3x40G ports to a single core running @3GHz.] No major
> > > > > performance differences were seen, but if anything the meson
> > > > > build was very slightly faster, as reported to Jerin, maybe 2%,
> > > > > though it's within the
> > > margin of error.
> > > > >
> > > >
> > > > Thanks for taking the time to investigate this.
> > > >
> > > > Disabling PIC for net/mlx5 driver alone in drivers/meson.build
> > > > resolves the
> > > issue for me.
> > > > I saw this issue with gcc (tested with 4.8.5, 9.3.0, and 7.5.0).
> > > > But I see now
> > > that disabling PIC with an old clang version (clang 3.4.2, RHEL7.4)
> > > causes a drop in performance, not an improvement like with gcc.
> > > >
> > > That's interesting.
> > >
> > > When you just build with and without -fpic with newer clang, do you
> > > see the same perf drop as with gcc? With the older clang, is the
> > > shared lib build faster than the static one?
> >
> > With the older clang on RHEL7.4, the shared lib is about ~2% slower
> compared to the static build.
> > With clang 11 compiled from source on ubuntu 18.04, I'm getting good
> performance with static meson build, same performance as with makefiles
> with gcc, and ~6% better than the static meson gcc build. Disabling PIC on
> clang 11 degrades performance by ~4%.
> > With clang 6.0.0 however, disabling PIC causes a very small drop (~0.1%).
> >
> > This is on v20.08 with KVM ConnectX-5 SR-IOV passthrough. Command:
> "dpdk-testpmd --master-lcore=0 -c 0x1ffff -n 4 -w 00:05.0 --socket-
> mem=2048,0 -- --port-numa-config=0,0 --socket-num=0 --burst=64 --
> txd=1024 --rxd=1024 --mbcache=512 --rxq=8 --txq=8 --nb-cores=4 --port-
> topology=chained --forward-mode=macswap --no-lsc-interrupt -i -a --rss-
> udp".
> >
> 
> So, am I right in saying that it appears the clang builds are all fine here, that
> performance is pretty much as expected in all cases with the default setting
> of PIC enabled? Therefore it appears that the issue is limited to gcc builds at
> this point?
> 
Yes it appears that way.

Regards,
Ali