From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 7E7C19192 for ; Thu, 15 Oct 2015 17:43:59 +0200 (CEST) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga102.jf.intel.com with ESMTP; 15 Oct 2015 08:43:58 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.17,686,1437462000"; d="scan'208";a="811818690" Received: from irsmsx105.ger.corp.intel.com ([163.33.3.28]) by fmsmga001.fm.intel.com with ESMTP; 15 Oct 2015 08:43:49 -0700 Received: from irsmsx112.ger.corp.intel.com (10.108.20.5) by irsmsx105.ger.corp.intel.com (163.33.3.28) with Microsoft SMTP Server (TLS) id 14.3.248.2; Thu, 15 Oct 2015 16:43:48 +0100 Received: from irsmsx105.ger.corp.intel.com ([169.254.7.75]) by irsmsx112.ger.corp.intel.com ([169.254.1.218]) with mapi id 14.03.0248.002; Thu, 15 Oct 2015 16:43:48 +0100 From: "Ananyev, Konstantin" To: Zoltan Kiss , "Richardson, Bruce" , "dev@dpdk.org" Thread-Topic: [PATCH] ixgbe: prefetch packet headers in vector PMD receive function Thread-Index: AQHQ5OrcDMYVDKIQM0OQ/IGpqZ+dT54w9hQAgAAIuYCAABXhgIAAByoAgByJi4CAA4aAcIAaL0SAgAA70yCAAPgTAIAAZcdA Date: Thu, 15 Oct 2015 15:43:48 +0000 Message-ID: <2601191342CEEE43887BDE71AB97725836AB00DE@irsmsx105.ger.corp.intel.com> References: <1441135036-7491-1-git-send-email-zoltan.kiss@linaro.org> <55ED8252.1020900@linaro.org> <59AF69C657FD0841A61C55336867B5B0359227DF@IRSMSX103.ger.corp.intel.com> <55ED9BFD.7040009@linaro.org> <59AF69C657FD0841A61C55336867B5B035922A83@IRSMSX103.ger.corp.intel.com> <56059255.5010507@linaro.org> <2601191342CEEE43887BDE71AB97725836A9CE89@irsmsx105.ger.corp.intel.com> <561E7E83.5000401@linaro.org> <2601191342CEEE43887BDE71AB97725836AAFBAB@irsmsx105.ger.corp.intel.com> <561F80CC.4000809@linaro.org> In-Reply-To: <561F80CC.4000809@linaro.org> Accept-Language: en-IE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [163.33.239.180] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH] ixgbe: prefetch packet headers in vector PMD receive function X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2015 15:44:00 -0000 > -----Original Message----- > From: Zoltan Kiss [mailto:zoltan.kiss@linaro.org] > Sent: Thursday, October 15, 2015 11:33 AM > To: Ananyev, Konstantin; Richardson, Bruce; dev@dpdk.org > Subject: Re: [PATCH] ixgbe: prefetch packet headers in vector PMD receive= function >=20 >=20 >=20 > On 15/10/15 00:23, Ananyev, Konstantin wrote: > > > > > >> -----Original Message----- > >> From: Zoltan Kiss [mailto:zoltan.kiss@linaro.org] > >> Sent: Wednesday, October 14, 2015 5:11 PM > >> To: Ananyev, Konstantin; Richardson, Bruce; dev@dpdk.org > >> Subject: Re: [PATCH] ixgbe: prefetch packet headers in vector PMD rece= ive function > >> > >> > >> > >> On 28/09/15 00:19, Ananyev, Konstantin wrote: > >>> > >>> > >>>> -----Original Message----- > >>>> From: Zoltan Kiss [mailto:zoltan.kiss@linaro.org] > >>>> Sent: Friday, September 25, 2015 7:29 PM > >>>> To: Richardson, Bruce; dev@dpdk.org > >>>> Cc: Ananyev, Konstantin > >>>> Subject: Re: [PATCH] ixgbe: prefetch packet headers in vector PMD re= ceive function > >>>> > >>>> On 07/09/15 07:41, Richardson, Bruce wrote: > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: Zoltan Kiss [mailto:zoltan.kiss@linaro.org] > >>>>>> Sent: Monday, September 7, 2015 3:15 PM > >>>>>> To: Richardson, Bruce; dev@dpdk.org > >>>>>> Cc: Ananyev, Konstantin > >>>>>> Subject: Re: [PATCH] ixgbe: prefetch packet headers in vector PMD = receive > >>>>>> function > >>>>>> > >>>>>> > >>>>>> > >>>>>> On 07/09/15 13:57, Richardson, Bruce wrote: > >>>>>>> > >>>>>>>> -----Original Message----- > >>>>>>>> From: Zoltan Kiss [mailto:zoltan.kiss@linaro.org] > >>>>>>>> Sent: Monday, September 7, 2015 1:26 PM > >>>>>>>> To: dev@dpdk.org > >>>>>>>> Cc: Ananyev, Konstantin; Richardson, Bruce > >>>>>>>> Subject: Re: [PATCH] ixgbe: prefetch packet headers in vector PM= D > >>>>>>>> receive function > >>>>>>>> > >>>>>>>> Hi, > >>>>>>>> > >>>>>>>> I just realized I've missed the "[PATCH]" tag from the subject. = Did > >>>>>>>> anyone had time to review this? > >>>>>>>> > >>>>>>> Hi Zoltan, > >>>>>>> > >>>>>>> the big thing that concerns me with this is the addition of new > >>>>>>> instructions for each packet in the fast path. Ideally, this > >>>>>>> prefetching would be better handled in the application itself, as= for > >>>>>>> some apps, e.g. those using pipelining, the core doing the RX fro= m the > >>>>>>> NIC may not touch the packet data at all, and the prefetches will > >>>>>> instead cause a performance slowdown. > >>>>>>> Is it possible to get the same performance increase - or somethin= g > >>>>>>> close to it - by making changes in OVS? > >>>>>> OVS already does a prefetch when it's processing the previous pack= et, but > >>>>>> apparently it's not early enough. At least for my test scenario, w= here I'm > >>>>>> forwarding UDP packets with the least possible overhead. I guess i= n tests > >>>>>> where OVS does more complex processing it should be fine. > >>>>>> I'll try to move the prefetch earlier in OVS codebase, but I'm not= sure if > >>>>>> it'll help. > >>>>> I would suggest trying to prefetch more than one packet ahead. Pref= etching 4 or > >>>>> 8 ahead might work better, depending on the processing being done. > >>>> > >>>> I've moved the prefetch earlier, and it seems to work: > >>>> > >>>> https://patchwork.ozlabs.org/patch/519017/ > >>>> > >>>> However it raises the question: should we remove header prefetch fro= m > >>>> all the other drivers, and the CONFIG_RTE_PMD_PACKET_PREFETCH option= ? > >>> > >>> My vote would be for that. > >>> Konstantin > >> > >> After some further thinking I would rather support the > >> rte_packet_prefetch() macro (prefetch the header in the driver, and > >> configure it through CONFIG_RTE_PMD_PACKET_PREFETCH) > >> > >> - the prefetch can happen earlier, so if an application needs the head= er > >> right away, this is the fastest > >> - if the application doesn't need header prefetch, it can turn it off > >> compile time. Same as if we wouldn't have this option. > >> - if the application has mixed needs (sometimes it needs the header > >> right away, sometimes it doesn't), it can turn it off and do what it > >> needs. Same as if we wouldn't have this option. > >> > >> A harder decision would be whether it should be turned on or off by > >> default. Currently it's on, and half of the receive functions don't us= e it. > > > > Yep, it is sort of a mess right now. > > Another question if we'd like to keep it and standardise it: > > at what moment to prefetch: as soon as we realize that HW is done with = that buffer, > > or as late inside rx_burst() as possible? > > I suppose there is no clear answer for that. > I think if the application needs the driver start the prefetching, it > does it because it's already too late when rte_eth_rx_burst() returns. > So I think it's best to do it as soon as we learn that the HW is done. Probably, but it might be situations when it would be more plausible to do = it later too. Again it might depend on particular HW and your memory access pattern. >=20 > > That's why my thought was to just get rid of it. > > Though if it would be implemented in some standardized way and disabled= by default - > > that's probably ok too. > > BTW, couldn't that be implemented just via rte_ethdev rx callbacks mech= anism? > > Then we can have the code one place and don't need compile option at al= l - > > could be ebabled/disabled dynamically on a per nic basis. > > Or would it be too high overhead for that? >=20 > I think if we go that way, it's better to pass an option to > rte_eth_rx_burst() and tell if you want the header prefetched by the > driver. That would be the most flexible. That would mean ABI breakage for rx_burst()... Probably then better a new parameter in rx_conf or something. Though it still means that each PMD has to implement it on its own. And again there might be PMDs that would just ignore that parameter. While for callback it would be all in one and known place. But again, I didn't measure it, so not sure what will be an impact of the c= allback itself. Might be it not worth it.=20 Konstantin >=20 > > Konstantin > > > > > > > >> > >>> > >>> > >>>> > >>>>> > >>>>> /Bruce > >>>