From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id CACE158DF for ; Thu, 4 Sep 2014 13:51:11 +0200 (CEST) Received: from azsmga001.ch.intel.com ([10.2.17.19]) by orsmga102.jf.intel.com with ESMTP; 04 Sep 2014 04:49:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.04,465,1406617200"; d="scan'208";a="475161994" Received: from sivswdev02.ir.intel.com ([10.237.217.46]) by azsmga001.ch.intel.com with ESMTP; 04 Sep 2014 04:55:49 -0700 Date: Thu, 4 Sep 2014 12:55:49 +0100 From: Bruce Richardson To: Yerden Zhumabekov Message-ID: <20140904115549.GA13208@sivswdev02.ir.intel.com> References: <1409759378-10113-1-git-send-email-bruce.richardson@intel.com> <1409759378-10113-12-git-send-email-bruce.richardson@intel.com> <5407F3E9.6080907@sts.kz> <20140904102744.GA23231@sivswdev02.ir.intel.com> <5408463C.8040805@sts.kz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5408463C.8040805@sts.kz> Organization: Intel Shannon Limited. Registered in Ireland. Registered Office: Collinstown Industrial Park, Leixlip, County Kildare. Registered Number: 308263. Business address: Dromore House, East Park, Shannon, Co. Clare. User-Agent: Mutt/1.5.23 (2014-03-12) Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] [PATCH 11/13] mbuf: move l2_len and l3_len to second cache line X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Sep 2014 11:51:12 -0000 On Thu, Sep 04, 2014 at 05:00:12PM +0600, Yerden Zhumabekov wrote: > I get your point. I've also read throught the code of various PMDs and > have found no indication of setting l2_len/l3_len fields as well. > > As for testing, we'd be happy to test the patchset but we are now in > process of building our testing facilities so we are not ready to > provide enough workload for the hardware/software. I was also wondering > if anyone has run some test and can provide some numbers on that matter. > > Personally, I don't think frag/reassemly app is a perfect example for > evaluating 2nd cache line performance penalty. The offsets to L3 and L4 > headers need to be calculated for all TCP/IP traffic and fragmented > traffic is not representative in this case. Maybe it would be better to > write an app which calculates these offsets for different set of mbufs > and provides some stats. For example, l2fwd/l3fwd + additional l2_len > and l3_len calculation. > > And I'm also figuring out how to rewrite our app/libs (prefetch etc) to > reflect the future changes in mbuf, hence my concerns :) > Just a final point on this. Note that the second cache line is always being read by the TX leg of the code to free back mbufs to their mbuf pool post- transmit. The overall fast-path RX+TX benchmarks show no performance degradation due to that access. For sample apps, you make a good point indeed about the existing app not being very useful as they work on larger packets. I'll see what I can throw together here to make a more realistic test. /Bruce > > 04.09.2014 16:27, Bruce Richardson ??????????: > > Hi Yerden, > > > > I understand your concerns and it's good to have this discussion. > > > > There are a number of reasons why I've moved these particular fields > > to the second cache line. Firstly, the main reason is that, obviously enough, > > not all fields will fit in cache line 0, and we need to prioritize what does > > get stored there. The guiding principle behind what fields get moved or not > > that I've chosen to use for this patch set is to move fields that are not > > used on the receive path (or the fastpath receive path, more specifically - > > so that we can move fields only used by jumbo frames that span mbufs) to the > > second cache line. From a search through the existing codebase, there are no > > drivers which set the l2/l3 length fields on RX, this is only used in > > reassembly libraries/apps and by the drivers on TX. > > > > The other reason for moving it to the second cache line is that it logically > > belongs with all the other length fields that we need to add to enable > > tunneling support. [To get an idea of the extra fields that I propose adding > > to the mbuf, please see the RFC patchset I sent out previously as "[RFC > > PATCH 00/14] Extend the mbuf structure"]. While we probably can fit the 16-bits > > needed for l2/l3 length on the mbuf line 0, there is not enough room for all > > the lengths so we would end up splitting them with other fields in between. > > > > So, in terms of what do to about this particular issue. I would hope that for > > applications that use these fields the impact should be small and/or possible > > to work around e.g. maybe prefetch second cache line on RX in driver. If not, > > then I'm happy to see about withdrawing this particular change and seeing if > > we can keep l2/l3 lengths on cache line zero, with other length fields being > > on cache line 1. > > > > Question: would you consider the ip fragmentation and reassembly example apps > > in the Intel DPDK releases good examples to test to see the impacts of this > > change, or is there some other test you would prefer that I look to do? > > Can you perhaps test out the patch sets for the mbuf that I've upstreamed so > > far and let me know what regressions, if any, you see in your use-case > > scenarios? > > > > Regards, > > /Bruce > > > -- > Sincerely, > > Yerden Zhumabekov > STS, ACI > Astana, KZ >