From: Rahul Lakkireddy <rahul.lakkireddy@chelsio.com>
To: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>, Felix Marti <felix@chelsio.com>,
Nirranjan Kirubaharan <nirranjan@chelsio.com>,
Kumar A S <kumaras@chelsio.com>
Subject: Re: [dpdk-dev] [PATCH 1/6] cxgbe: Optimize forwarding performance for 40G
Date: Wed, 7 Oct 2015 20:57:22 +0530 [thread overview]
Message-ID: <20151007152721.GA2689@scalar.blr.asicdesigners.com> (raw)
In-Reply-To: <20151005150729.GA8809@scalar.blr.asicdesigners.com>
On Monday, October 10/05/15, 2015 at 20:37:31 +0530, Rahul Lakkireddy wrote:
> On Monday, October 10/05/15, 2015 at 07:09:27 -0700, Ananyev, Konstantin wrote:
> > Hi Rahul,
>
> [...]
>
> > > > > This additional check seems redundant for single segment
> > > > > packets since rte_pktmbuf_free_seg also performs rte_mbuf_sanity_check.
> > > > >
> > > > > Several PMDs already prefer to use rte_pktmbuf_free_seg directly over
> > > > > rte_pktmbuf_free as it is faster.
> > > >
> > > > Other PMDs use rte_pktmbuf_free_seg() as each TD has an associated
> > > > with it segment. So as HW is done with the TD, SW frees associated segment.
> > > > In your case I don't see any point in re-implementing rte_pktmbuf_free() manually,
> > > > and I don't think it would be any faster.
> > > >
> > > > Konstantin
> > >
> > > As I mentioned below, I am clearly seeing a difference of 1 Mpps. And 1
> > > Mpps is not a small difference IMHO.
> >
> > Agree with you here - it is a significant difference.
> >
> > >
> > > When running l3fwd with 8 queues, I also collected a perf report.
> > > When using rte_pktmbuf_free, I see that it eats up around 6% cpu as
> > > below in perf top report:-
> > > --------------------
> > > 32.00% l3fwd [.] cxgbe_poll
> > > 22.25% l3fwd [.] t4_eth_xmit
> > > 20.30% l3fwd [.] main_loop
> > > 6.77% l3fwd [.] rte_pktmbuf_free
> > > 4.86% l3fwd [.] refill_fl_usembufs
> > > 2.00% l3fwd [.] write_sgl
> > > .....
> > > --------------------
> > >
> > > While, when using rte_pktmbuf_free_seg directly, I don't see above
> > > problem. perf top report now comes as:-
> > > -------------------
> > > 33.36% l3fwd [.] cxgbe_poll
> > > 32.69% l3fwd [.] t4_eth_xmit
> > > 19.05% l3fwd [.] main_loop
> > > 5.21% l3fwd [.] refill_fl_usembufs
> > > 2.40% l3fwd [.] write_sgl
> > > ....
> > > -------------------
> >
> > I don't think these 6% disappeared anywhere.
> > As I can see, now t4_eth_xmit() increased by roughly same amount
> > (you still have same job to do).
>
> Right.
>
> > To me it looks like in that case compiler didn't really inline rte_pktmbuf_free().
> > Wonder can you add 'always_inline' attribute to the rte_pktmbuf_free(),
> > and see would it make any difference?
> >
> > Konstantin
>
> I will try out above and update further.
>
Tried always_inline and didn't see any difference in performance in
RHEL 6.4 with gcc 4.4.7, but was seeing 1 MPPS improvement with the
above block.
I've moved to latest RHEL 7.1 with gcc 4.8.3 and tried both
always_inline and the above block and I'm not seeing any difference
for both.
Will drop this block and submit a v2.
Thanks for the review Aaron and Konstantin.
Thanks,
Rahul
next prev parent reply other threads:[~2015-10-07 15:27 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-10-02 11:16 [dpdk-dev] [PATCH 0/6] cxgbe: Optimize tx/rx for 40GbE and add Jumbo Frame support for CXGBE PMD Rahul Lakkireddy
2015-10-02 11:16 ` [dpdk-dev] [PATCH 1/6] cxgbe: Optimize forwarding performance for 40G Rahul Lakkireddy
2015-10-02 21:48 ` Aaron Conole
2015-10-05 10:06 ` Rahul Lakkireddy
2015-10-05 11:46 ` Ananyev, Konstantin
2015-10-05 12:42 ` Rahul Lakkireddy
2015-10-05 14:09 ` Ananyev, Konstantin
2015-10-05 15:07 ` Rahul Lakkireddy
2015-10-07 15:27 ` Rahul Lakkireddy [this message]
2015-10-02 11:16 ` [dpdk-dev] [PATCH 2/6] cxgbe: Update device info and perform sanity checks to enable jumbo frames Rahul Lakkireddy
2015-10-02 11:16 ` [dpdk-dev] [PATCH 3/6] cxgbe: Update tx path to transmit " Rahul Lakkireddy
2015-10-02 11:16 ` [dpdk-dev] [PATCH 4/6] cxgbe: Update rx path to receive " Rahul Lakkireddy
2015-10-02 11:16 ` [dpdk-dev] [PATCH 5/6] cxgbe: Allow apps to change mtu Rahul Lakkireddy
2015-10-02 11:16 ` [dpdk-dev] [PATCH 6/6] doc: Update cxgbe documentation and release notes Rahul Lakkireddy
2015-10-08 13:46 ` [dpdk-dev] [PATCH v2 0/6] cxgbe: Optimize tx/rx for 40GbE and add Jumbo Frame support for CXGBE PMD Rahul Lakkireddy
2015-10-08 13:46 ` [dpdk-dev] [PATCH v2 1/6] cxgbe: Optimize forwarding performance for 40G Rahul Lakkireddy
2015-10-08 13:46 ` [dpdk-dev] [PATCH v2 2/6] cxgbe: Update device info and perform sanity checks to enable jumbo frames Rahul Lakkireddy
2015-10-08 13:46 ` [dpdk-dev] [PATCH v2 3/6] cxgbe: Update tx path to transmit " Rahul Lakkireddy
2015-10-08 13:46 ` [dpdk-dev] [PATCH v2 4/6] cxgbe: Update rx path to receive " Rahul Lakkireddy
2015-10-08 13:46 ` [dpdk-dev] [PATCH v2 5/6] cxgbe: Allow apps to change mtu Rahul Lakkireddy
2015-10-08 13:46 ` [dpdk-dev] [PATCH v2 6/6] doc: Update cxgbe documentation and release notes Rahul Lakkireddy
2015-10-20 16:51 ` [dpdk-dev] [PATCH v2 0/6] cxgbe: Optimize tx/rx for 40GbE and add Jumbo Frame support for CXGBE PMD Thomas Monjalon
2015-10-21 6:14 ` Rahul Lakkireddy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20151007152721.GA2689@scalar.blr.asicdesigners.com \
--to=rahul.lakkireddy@chelsio.com \
--cc=dev@dpdk.org \
--cc=felix@chelsio.com \
--cc=konstantin.ananyev@intel.com \
--cc=kumaras@chelsio.com \
--cc=nirranjan@chelsio.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).