From: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>
To: Vladyslav Buslov <Vladyslav.Buslov@harmonicinc.com>,
"Wu, Jingjing" <jingjing.wu@intel.com>,
"Yigit, Ferruh" <ferruh.yigit@intel.com>,
"Zhang, Helin" <helin.zhang@intel.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH] net/i40e: add additional prefetch instructions for bulk rx
Date: Tue, 11 Oct 2016 08:51:04 +0000 [thread overview]
Message-ID: <2601191342CEEE43887BDE71AB9772583F0C0408@irsmsx105.ger.corp.intel.com> (raw)
In-Reply-To: <MWHPR11MB1360602B685F5573E38429859DDB0@MWHPR11MB1360.namprd11.prod.outlook.com>
Hi Vladislav,
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Vladyslav Buslov
> Sent: Monday, October 10, 2016 6:06 PM
> To: Wu, Jingjing <jingjing.wu@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Zhang, Helin <helin.zhang@intel.com>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] net/i40e: add additional prefetch instructions for bulk rx
>
> > -----Original Message-----
> > From: Wu, Jingjing [mailto:jingjing.wu@intel.com]
> > Sent: Monday, October 10, 2016 4:26 PM
> > To: Yigit, Ferruh; Vladyslav Buslov; Zhang, Helin
> > Cc: dev@dpdk.org
> > Subject: RE: [dpdk-dev] [PATCH] net/i40e: add additional prefetch
> > instructions for bulk rx
> >
> >
> >
> > > -----Original Message-----
> > > From: Yigit, Ferruh
> > > Sent: Wednesday, September 14, 2016 9:25 PM
> > > To: Vladyslav Buslov <vladyslav.buslov@harmonicinc.com>; Zhang, Helin
> > > <helin.zhang@intel.com>; Wu, Jingjing <jingjing.wu@intel.com>
> > > Cc: dev@dpdk.org
> > > Subject: Re: [dpdk-dev] [PATCH] net/i40e: add additional prefetch
> > > instructions for bulk rx
> > >
> > > On 7/14/2016 6:27 PM, Vladyslav Buslov wrote:
> > > > Added prefetch of first packet payload cacheline in
> > > > i40e_rx_scan_hw_ring Added prefetch of second mbuf cacheline in
> > > > i40e_rx_alloc_bufs
> > > >
> > > > Signed-off-by: Vladyslav Buslov <vladyslav.buslov@harmonicinc.com>
> > > > ---
> > > > drivers/net/i40e/i40e_rxtx.c | 7 +++++--
> > > > 1 file changed, 5 insertions(+), 2 deletions(-)
> > > >
> > > > diff --git a/drivers/net/i40e/i40e_rxtx.c
> > > > b/drivers/net/i40e/i40e_rxtx.c index d3cfb98..e493fb4 100644
> > > > --- a/drivers/net/i40e/i40e_rxtx.c
> > > > +++ b/drivers/net/i40e/i40e_rxtx.c
> > > > @@ -1003,6 +1003,7 @@ i40e_rx_scan_hw_ring(struct i40e_rx_queue
> > *rxq)
> > > > /* Translate descriptor info to mbuf parameters */
> > > > for (j = 0; j < nb_dd; j++) {
> > > > mb = rxep[j].mbuf;
> > > > + rte_prefetch0(RTE_PTR_ADD(mb->buf_addr,
> > > RTE_PKTMBUF_HEADROOM));
> >
> > Why did prefetch here? I think if application need to deal with packet, it is
> > more suitable to put it in application.
> >
> > > > qword1 = rte_le_to_cpu_64(\
> > > > rxdp[j].wb.qword1.status_error_len);
> > > > pkt_len = ((qword1 &
> > > I40E_RXD_QW1_LENGTH_PBUF_MASK) >>
> > > > @@ -1086,9 +1087,11 @@ i40e_rx_alloc_bufs(struct i40e_rx_queue
> > *rxq)
> > > >
> > > > rxdp = &rxq->rx_ring[alloc_idx];
> > > > for (i = 0; i < rxq->rx_free_thresh; i++) {
> > > > - if (likely(i < (rxq->rx_free_thresh - 1)))
> > > > + if (likely(i < (rxq->rx_free_thresh - 1))) {
> > > > /* Prefetch next mbuf */
> > > > - rte_prefetch0(rxep[i + 1].mbuf);
> > > > + rte_prefetch0(&rxep[i + 1].mbuf->cacheline0);
> > > > + rte_prefetch0(&rxep[i + 1].mbuf->cacheline1);
I think there are rte_mbuf_prefetch_part1/part2 defined in rte_mbuf.h,
specially for that case.
> > > > + }
> > Agree with this change. And when I test it by testpmd with iofwd, no
> > performance increase is observed but minor decrease.
> > Can you share will us when it will benefit the performance in your scenario ?
> >
> >
> > Thanks
> > Jingjing
>
> Hello Jingjing,
>
> Thanks for code review.
>
> My use case: We have simple distributor thread that receives packets from port and distributes them among worker threads according to
> VLAN and MAC address hash.
>
> While working on performance optimization we determined that most of CPU usage of this thread is in DPDK.
> As and optimization we decided to switch to rx burst alloc function, however that caused additional performance degradation compared to
> scatter rx mode.
> In profiler two major culprits were:
> 1. Access to packet data Eth header in application code. (cache miss)
> 2. Setting next packet descriptor field to NULL in DPDK i40e_rx_alloc_bufs code. (this field is in second descriptor cache line that was not
> prefetched)
I wonder what will happen if we'll remove any prefetches here?
Would it make things better or worse (and by how much)?
> After applying my fixes performance improved compared to scatter rx mode.
>
> I assumed that prefetch of first cache line of packet data belongs to DPDK because it is done in scatter rx mode. (in
> i40e_recv_scattered_pkts)
> It can be moved to application side but IMO it is better to be consistent across all rx modes.
I would agree with Jingjing here, probably PMD should avoid to prefetch packet's data.
Konstantin
next prev parent reply other threads:[~2016-10-11 8:51 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-07-14 17:27 [dpdk-dev] [PATCH] Add missing prefetches to i40e bulk rx path Vladyslav Buslov
2016-07-14 17:27 ` [dpdk-dev] [PATCH] net/i40e: add additional prefetch instructions for bulk rx Vladyslav Buslov
2016-09-14 13:24 ` Ferruh Yigit
2016-10-10 13:25 ` Wu, Jingjing
2016-10-10 17:05 ` Vladyslav Buslov
2016-10-11 8:51 ` Ananyev, Konstantin [this message]
2016-10-11 9:24 ` Vladyslav Buslov
2016-10-12 0:04 ` Ananyev, Konstantin
2016-10-13 10:18 ` Bruce Richardson
2016-10-13 10:30 ` Ananyev, Konstantin
2016-11-15 12:19 ` Ferruh Yigit
2016-11-15 13:27 ` Vladyslav Buslov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2601191342CEEE43887BDE71AB9772583F0C0408@irsmsx105.ger.corp.intel.com \
--to=konstantin.ananyev@intel.com \
--cc=Vladyslav.Buslov@harmonicinc.com \
--cc=dev@dpdk.org \
--cc=ferruh.yigit@intel.com \
--cc=helin.zhang@intel.com \
--cc=jingjing.wu@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).