DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>
To: Rahul Lakkireddy <rahul.lakkireddy@chelsio.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>, Felix Marti <felix@chelsio.com>,
	Nirranjan Kirubaharan <nirranjan@chelsio.com>,
	Kumar A S <kumaras@chelsio.com>
Subject: Re: [dpdk-dev] [PATCH 1/6] cxgbe: Optimize forwarding performance for 40G
Date: Mon, 5 Oct 2015 14:09:27 +0000	[thread overview]
Message-ID: <2601191342CEEE43887BDE71AB97725836AA37E2@irsmsx105.ger.corp.intel.com> (raw)
In-Reply-To: <20151005124205.GA24533@scalar.blr.asicdesigners.com>

Hi Rahul,

> -----Original Message-----
> From: Rahul Lakkireddy [mailto:rahul.lakkireddy@chelsio.com]
> Sent: Monday, October 05, 2015 1:42 PM
> To: Ananyev, Konstantin
> Cc: Aaron Conole; dev@dpdk.org; Felix Marti; Kumar A S; Nirranjan Kirubaharan
> Subject: Re: [dpdk-dev] [PATCH 1/6] cxgbe: Optimize forwarding performance for 40G
> 
> Hi Konstantin,
> 
> On Monday, October 10/05/15, 2015 at 04:46:40 -0700, Ananyev, Konstantin wrote:
> >
> >
> > > -----Original Message-----
> > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rahul Lakkireddy
> > > Sent: Monday, October 05, 2015 11:06 AM
> > > To: Aaron Conole
> > > Cc: dev@dpdk.org; Felix Marti; Kumar A S; Nirranjan Kirubaharan
> > > Subject: Re: [dpdk-dev] [PATCH 1/6] cxgbe: Optimize forwarding performance for 40G
> > >
> > > Hi Aaron,
> > >
> > > On Friday, October 10/02/15, 2015 at 14:48:28 -0700, Aaron Conole wrote:
> > > > Hi Rahul,
> > > >
> > > > Rahul Lakkireddy <rahul.lakkireddy@chelsio.com> writes:
> > > >
> > > > > Update sge initialization with respect to free-list manager configuration
> > > > > and ingress arbiter. Also update refill logic to refill mbufs only after
> > > > > a certain threshold for rx.  Optimize tx packet prefetch and free.
> > > > <<snip>>
> > > > >  			for (i = 0; i < sd->coalesce.idx; i++) {
> > > > > -				rte_pktmbuf_free(sd->coalesce.mbuf[i]);
> > > > > +				struct rte_mbuf *tmp = sd->coalesce.mbuf[i];
> > > > > +
> > > > > +				do {
> > > > > +					struct rte_mbuf *next = tmp->next;
> > > > > +
> > > > > +					rte_pktmbuf_free_seg(tmp);
> > > > > +					tmp = next;
> > > > > +				} while (tmp);
> > > > >  				sd->coalesce.mbuf[i] = NULL;
> > > > Pardon my ignorance here, but rte_pktmbuf_free does this work. I can't
> > > > actually see much difference between your rewrite of this block, and
> > > > the implementation of rte_pktmbuf_free() (apart from moving your branch
> > > > to the end of the function). Did your microbenchmarking really show this
> > > > as an improvement?
> > > >
> > > > Thanks for your time,
> > > > Aaron
> > >
> > > rte_pktmbuf_free calls rte_mbuf_sanity_check which does a lot of
> > > checks.
> >
> > Only when RTE_LIBRTE_MBUF_DEBUG is enabled in your config.
> > By default it is switched off.
> 
> Right. I clearly missed this.
> I am running with default config only btw.
> 
> >
> > > This additional check seems redundant for single segment
> > > packets since rte_pktmbuf_free_seg also performs rte_mbuf_sanity_check.
> > >
> > > Several PMDs already prefer to use rte_pktmbuf_free_seg directly over
> > > rte_pktmbuf_free as it is faster.
> >
> > Other PMDs use rte_pktmbuf_free_seg() as each TD has an associated
> > with it segment. So as HW is done with the TD, SW frees associated segment.
> > In your case I don't see any point in re-implementing rte_pktmbuf_free() manually,
> > and I don't think it would be any faster.
> >
> > Konstantin
> 
> As I mentioned below, I am clearly seeing a difference of 1 Mpps. And 1
> Mpps is not a small difference IMHO.

Agree with you here - it is a significant difference.

> 
> When running l3fwd with 8 queues, I also collected a perf report.
> When using rte_pktmbuf_free, I see that it eats up around 6% cpu as
> below in perf top report:-
> --------------------
> 32.00%  l3fwd                        [.] cxgbe_poll
> 22.25%  l3fwd                        [.] t4_eth_xmit
> 20.30%  l3fwd                        [.] main_loop
>  6.77%  l3fwd                        [.] rte_pktmbuf_free
>  4.86%  l3fwd                        [.] refill_fl_usembufs
>  2.00%  l3fwd                        [.] write_sgl
> .....
> --------------------
> 
> While, when using rte_pktmbuf_free_seg directly, I don't see above
> problem. perf top report now comes as:-
> -------------------
> 33.36%  l3fwd                        [.] cxgbe_poll
> 32.69%  l3fwd                        [.] t4_eth_xmit
> 19.05%  l3fwd                        [.] main_loop
>  5.21%  l3fwd                        [.] refill_fl_usembufs
>  2.40%  l3fwd                        [.] write_sgl
> ....
> -------------------

I don't think these 6% disappeared anywhere.
As I can see, now t4_eth_xmit() increased by roughly same amount
(you still have same job to do).
To me it looks like in that case compiler didn't really inline rte_pktmbuf_free().
Wonder can you add 'always_inline' attribute to the  rte_pktmbuf_free(),
and see would it make any difference?

Konstantin 

> 
> I obviously missed the debug flag for rte_mbuf_sanity_check.
> However, there is a clear difference of 1 Mpps. I don't know if its the
> change between while construct used in rte_pktmbuf_free and the
> do..while construct that I used - is making the difference.
> 
> 
> >
> > >
> > > The forwarding perf. improvement with only this particular block is
> > > around 1 Mpps for 64B packets when using l3fwd with 8 queues.
> > >
> > > Thanks,
> > > Rahul

  reply	other threads:[~2015-10-05 14:09 UTC|newest]

Thread overview: 23+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-10-02 11:16 [dpdk-dev] [PATCH 0/6] cxgbe: Optimize tx/rx for 40GbE and add Jumbo Frame support for CXGBE PMD Rahul Lakkireddy
2015-10-02 11:16 ` [dpdk-dev] [PATCH 1/6] cxgbe: Optimize forwarding performance for 40G Rahul Lakkireddy
2015-10-02 21:48   ` Aaron Conole
2015-10-05 10:06     ` Rahul Lakkireddy
2015-10-05 11:46       ` Ananyev, Konstantin
2015-10-05 12:42         ` Rahul Lakkireddy
2015-10-05 14:09           ` Ananyev, Konstantin [this message]
2015-10-05 15:07             ` Rahul Lakkireddy
2015-10-07 15:27               ` Rahul Lakkireddy
2015-10-02 11:16 ` [dpdk-dev] [PATCH 2/6] cxgbe: Update device info and perform sanity checks to enable jumbo frames Rahul Lakkireddy
2015-10-02 11:16 ` [dpdk-dev] [PATCH 3/6] cxgbe: Update tx path to transmit " Rahul Lakkireddy
2015-10-02 11:16 ` [dpdk-dev] [PATCH 4/6] cxgbe: Update rx path to receive " Rahul Lakkireddy
2015-10-02 11:16 ` [dpdk-dev] [PATCH 5/6] cxgbe: Allow apps to change mtu Rahul Lakkireddy
2015-10-02 11:16 ` [dpdk-dev] [PATCH 6/6] doc: Update cxgbe documentation and release notes Rahul Lakkireddy
2015-10-08 13:46 ` [dpdk-dev] [PATCH v2 0/6] cxgbe: Optimize tx/rx for 40GbE and add Jumbo Frame support for CXGBE PMD Rahul Lakkireddy
2015-10-08 13:46   ` [dpdk-dev] [PATCH v2 1/6] cxgbe: Optimize forwarding performance for 40G Rahul Lakkireddy
2015-10-08 13:46   ` [dpdk-dev] [PATCH v2 2/6] cxgbe: Update device info and perform sanity checks to enable jumbo frames Rahul Lakkireddy
2015-10-08 13:46   ` [dpdk-dev] [PATCH v2 3/6] cxgbe: Update tx path to transmit " Rahul Lakkireddy
2015-10-08 13:46   ` [dpdk-dev] [PATCH v2 4/6] cxgbe: Update rx path to receive " Rahul Lakkireddy
2015-10-08 13:46   ` [dpdk-dev] [PATCH v2 5/6] cxgbe: Allow apps to change mtu Rahul Lakkireddy
2015-10-08 13:46   ` [dpdk-dev] [PATCH v2 6/6] doc: Update cxgbe documentation and release notes Rahul Lakkireddy
2015-10-20 16:51   ` [dpdk-dev] [PATCH v2 0/6] cxgbe: Optimize tx/rx for 40GbE and add Jumbo Frame support for CXGBE PMD Thomas Monjalon
2015-10-21  6:14     ` Rahul Lakkireddy

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=2601191342CEEE43887BDE71AB97725836AA37E2@irsmsx105.ger.corp.intel.com \
    --to=konstantin.ananyev@intel.com \
    --cc=dev@dpdk.org \
    --cc=felix@chelsio.com \
    --cc=kumaras@chelsio.com \
    --cc=nirranjan@chelsio.com \
    --cc=rahul.lakkireddy@chelsio.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).