From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <olivier.matz@6wind.com>
Received: from mail.droids-corp.org (zoll.droids-corp.org [94.23.50.67])
 by dpdk.org (Postfix) with ESMTP id BE510DE6
 for <dev@dpdk.org>; Mon, 15 Jun 2015 16:04:10 +0200 (CEST)
Received: from was59-1-82-226-113-214.fbx.proxad.net ([82.226.113.214]
 helo=[192.168.0.10])
 by mail.droids-corp.org with esmtpsa (TLS1.2:DHE_RSA_AES_128_CBC_SHA1:128)
 (Exim 4.80) (envelope-from <olivier.matz@6wind.com>)
 id 1Z4V4G-000864-2A; Mon, 15 Jun 2015 16:08:55 +0200
Message-ID: <557EDB91.9010503@6wind.com>
Date: Mon, 15 Jun 2015 16:05:05 +0200
From: Olivier MATZ <olivier.matz@6wind.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:31.0) Gecko/20100101 Icedove/31.6.0
MIME-Version: 1.0
To: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>, 
 "Richardson, Bruce" <bruce.richardson@intel.com>
References: <87110795-201A-4A1E-A4CC-A778AA7C8218@cisco.com>
 <557ED116.7040508@6wind.com> <20150615134409.GA7500@bricha3-MOBL3>
 <2601191342CEEE43887BDE71AB97725836A0A838@irsmsx105.ger.corp.intel.com>
In-Reply-To: <2601191342CEEE43887BDE71AB97725836A0A838@irsmsx105.ger.corp.intel.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
Cc: "dev@dpdk.org" <dev@dpdk.org>,
 "Damjan Marion \(damarion\)" <damarion@cisco.com>
Subject: Re: [dpdk-dev] rte_mbuf.next in 2nd cacheline
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Mon, 15 Jun 2015 14:04:11 -0000

Hi,

On 06/15/2015 03:54 PM, Ananyev, Konstantin wrote:
> 
> 
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richardson
>> Sent: Monday, June 15, 2015 2:44 PM
>> To: Olivier MATZ
>> Cc: dev@dpdk.org; Damjan Marion (damarion)
>> Subject: Re: [dpdk-dev] rte_mbuf.next in 2nd cacheline
>>
>> On Mon, Jun 15, 2015 at 03:20:22PM +0200, Olivier MATZ wrote:
>>> Hi Damjan,
>>>
>>> On 06/10/2015 11:47 PM, Damjan Marion (damarion) wrote:
>>>>
>>>> Hi,
>>>>
>>>> We noticed 7% performance improvement by simply moving rte_mbuf.next field to the 1st cache line.
>>>>
>>>> Currently, it falls under /* second cache line - fields only used in slow path or on TX */
>>>> but it is actually used at several places in rx fast path. (e.g.: i40e_rx_alloc_bufs() is setting that field to NULL).
>>>>
>>>> Is there anything we can do here (stop using next field, or move it to 1st cache line)?
>>>
>>> Agree, this is also something I noticed, see:
>>> http://dpdk.org/ml/archives/dev/2015-February/014400.html
>>>
>>> I did not have the time to do performance testing, but it's something
>>> I'd like to do as soon as I can. I don't see any obvious reason not to
>>> do it.
>>>
>>> It seems we currently just have enough room to do it (8 bytes are
>>> remaining in the first cache line when compiled in 64 bits).
>>
>> This, to me, is the obvious reason not to do it! It prevents us from taking in
>> any other offload fields in the RX fast-path into the mbuf.
>>
>> That being said, I can see why we might want to look to move it - but from the
>> work done in the ixgbe driver, I'd be hopeful we can get as much performance with
>> it on the second cache line for most cases, through judicious use of prefetching,
>> or otherwise.
>>
>> It took a lot of work and investigation to get free space in the mbuf - especially
>> in cache line 0, and I'd like to avoid just filling the cache line up again as
>> long as we possibly can!
> 
> Yep, agree with Bruce here.
> Plus, with packet_type going to be 4B and vlan_tci_outer,
> we just don't have 8 free bytes at the first cache line any more.

I don't understand why m->next would not be a better candidate than
rx offload fields to be in the first cache line. For instance, m->next
is mandatory and must be initialized when allocating a mbuf (to be
compared with m->seqn for instance, which is also in the first cache
line). So if we want to do some room in the first cache line, I
think we can.

To me, the only reason for not doing it now is because we don't
have a full performance test report (several use-cases, drivers, ...)
that shows it's better.

Olivier