From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id ABB4D2C7A for ; Fri, 5 Dec 2014 16:23:19 +0100 (CET) Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga103.jf.intel.com with ESMTP; 05 Dec 2014 07:21:35 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.07,522,1413270000"; d="scan'208";a="619247998" Received: from irsmsx107.ger.corp.intel.com ([163.33.3.99]) by orsmga001.jf.intel.com with ESMTP; 05 Dec 2014 07:23:14 -0800 Received: from irsmsx105.ger.corp.intel.com ([169.254.7.144]) by IRSMSX107.ger.corp.intel.com ([169.254.10.85]) with mapi id 14.03.0195.001; Fri, 5 Dec 2014 15:23:14 +0000 From: "Ananyev, Konstantin" To: "Richardson, Bruce" Thread-Topic: [PATCH v2] ixgbe: don't override mbuf buffer length Thread-Index: AQHQD+2aqEhfxV/HVEKDZVlTVgxRRZyAKs/ggACkrwCAAAeQ0IAABlaAgAAFpYCAAAYZAIAANYRg Date: Fri, 5 Dec 2014 15:23:13 +0000 Message-ID: <2601191342CEEE43887BDE71AB977258213BCFFC@IRSMSX105.ger.corp.intel.com> References: <1417716553-1506-1-git-send-email-jean-mickael.guerin@6wind.com> <2601191342CEEE43887BDE71AB977258213BCCB6@IRSMSX105.ger.corp.intel.com> <20141205103857.GA8184@bricha3-MOBL3> <2601191342CEEE43887BDE71AB977258213BCEE2@IRSMSX105.ger.corp.intel.com> <20141205112843.GC8184@bricha3-MOBL3> <2601191342CEEE43887BDE71AB977258213BCF7D@IRSMSX105.ger.corp.intel.com> <20141205121044.GA4084@bricha3-MOBL3> In-Reply-To: <20141205121044.GA4084@bricha3-MOBL3> Accept-Language: en-IE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [163.33.239.180] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] [PATCH v2] ixgbe: don't override mbuf buffer length X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Dec 2014 15:23:20 -0000 > -----Original Message----- > From: Richardson, Bruce > Sent: Friday, December 05, 2014 12:11 PM > To: Ananyev, Konstantin > Cc: Jean-Mickael Guerin; dev@dpdk.org > Subject: Re: [PATCH v2] ixgbe: don't override mbuf buffer length >=20 > On Fri, Dec 05, 2014 at 11:59:50AM +0000, Ananyev, Konstantin wrote: > > > > > > > -----Original Message----- > > > From: Richardson, Bruce > > > Sent: Friday, December 05, 2014 11:29 AM > > > To: Ananyev, Konstantin > > > Cc: Jean-Mickael Guerin; dev@dpdk.org > > > Subject: Re: [PATCH v2] ixgbe: don't override mbuf buffer length > > > > > > On Fri, Dec 05, 2014 at 11:19:11AM +0000, Ananyev, Konstantin wrote: > > > > > > > > > > > > > -----Original Message----- > > > > > From: Richardson, Bruce > > > > > Sent: Friday, December 05, 2014 10:39 AM > > > > > To: Ananyev, Konstantin > > > > > Cc: Jean-Mickael Guerin; dev@dpdk.org > > > > > Subject: Re: [PATCH v2] ixgbe: don't override mbuf buffer length > > > > > > > > > > On Fri, Dec 05, 2014 at 01:10:28AM +0000, Ananyev, Konstantin wro= te: > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Jean-Mickael Guerin [mailto:jean-mickael.guerin@6wind.c= om] > > > > > > > Sent: Thursday, December 04, 2014 6:09 PM > > > > > > > To: dev@dpdk.org > > > > > > > Cc: Richardson, Bruce; Ananyev, Konstantin > > > > > > > Subject: [PATCH v2] ixgbe: don't override mbuf buffer length > > > > > > > > > > > > > > The template mbuf_initializer is hard coded with a buflen whi= ch > > > > > > > might have been set differently by the application at the tim= e of > > > > > > > mbuf pool creation. > > > > > > > > > > > > > > Switch to a mbuf allocation, to fetch the correct default val= ues. > > > > > > > There is no performance impact because this is not a data-pla= ne API. > > > > > > > > > > > > > > Signed-off-by: Jean-Mickael Guerin > > > > > > > Acked-by: David Marchand > > > > > > > Fixes: 0ff3324da2 ("ixgbe: rework vector pmd following mbuf c= hanges") > > > > > > > --- > > > > > > > > > > > > > > v2: check returned value of ixgbe_rxq_vec_setup > > > > > > > > > > > > > > lib/librte_pmd_ixgbe/ixgbe_rxtx.c | 5 ++++- > > > > > > > lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c | 19 ++++++++++++-----= -- > > > > > > > 2 files changed, 16 insertions(+), 8 deletions(-) > > > > > > > > > > > > > > diff --git a/lib/librte_pmd_ixgbe/ixgbe_rxtx.c b/lib/librte_p= md_ixgbe/ixgbe_rxtx.c > > > > > > > index 5c36bff..7994da1 100644 > > > > > > > --- a/lib/librte_pmd_ixgbe/ixgbe_rxtx.c > > > > > > > +++ b/lib/librte_pmd_ixgbe/ixgbe_rxtx.c > > > > > > > @@ -2244,7 +2244,10 @@ ixgbe_dev_rx_queue_setup(struct rte_et= h_dev *dev, > > > > > > > use_def_burst_func =3D check_rx_burst_bulk_alloc_preconditi= ons(rxq); > > > > > > > > > > > > > > #ifdef RTE_IXGBE_INC_VECTOR > > > > > > > - ixgbe_rxq_vec_setup(rxq); > > > > > > > + if (ixgbe_rxq_vec_setup(rxq) < 0) { > > > > > > > + ixgbe_rx_queue_release(rxq); > > > > > > > + return (-ENOMEM); > > > > > > > + } > > > > > > > #endif > > > > > > > /* Check if pre-conditions are satisfied, and no Scattered = Rx */ > > > > > > > if (!use_def_burst_func && !dev->data->scattered_rx) { > > > > > > > diff --git a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c b/lib/libr= te_pmd_ixgbe/ixgbe_rxtx_vec.c > > > > > > > index c1b5a78..f7b02f5 100644 > > > > > > > --- a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c > > > > > > > +++ b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c > > > > > > > @@ -732,17 +732,22 @@ static struct ixgbe_txq_ops vec_txq_ops= =3D { > > > > > > > int > > > > > > > ixgbe_rxq_vec_setup(struct igb_rx_queue *rxq) > > > > > > > { > > > > > > > - struct rte_mbuf mb_def =3D { .buf_addr =3D 0 }; /* zeroed m= buf */ > > > > > > > + struct rte_mbuf *mb_def; > > > > > > > > > > > > > > - mb_def.nb_segs =3D 1; > > > > > > > - mb_def.data_off =3D RTE_PKTMBUF_HEADROOM; > > > > > > > - mb_def.buf_len =3D rxq->mb_pool->elt_size - sizeof(struct r= te_mbuf); > > > > > > > - mb_def.port =3D rxq->port_id; > > > > > > > - rte_mbuf_refcnt_set(&mb_def, 1); > > > > > > > + mb_def =3D rte_pktmbuf_alloc(rxq->mb_pool); > > > > > > > + if (mb_def =3D=3D NULL) { > > > > > > > + PMD_INIT_LOG(ERR, "ixgbe_rxq_vec_setup: could not allocate= one mbuf"); > > > > > > > + return -1; > > > > > > > + } > > > > > > > + /* nb_segs, refcnt, data_off and buf_len are already set */ > > > > > > > + mb_def->port =3D rxq->port_id; > > > > > > > > > > > > > > /* prevent compiler reordering: rearm_data covers previous = fields */ > > > > > > > rte_compiler_barrier(); > > > > > > > - rxq->mbuf_initializer =3D *((uint64_t *)&mb_def.rearm_data)= ; > > > > > > > + rxq->mbuf_initializer =3D *((uint64_t *)&mb_def->rearm_data= ); > > > > > > > + > > > > > > > + rte_pktmbuf_free(mb_def); > > > > > > > + > > > > > > > return 0; > > > > > > > } > > > > > > > > > > > > > > -- > > > > > > > 2.1.3 > > > > > > > > > > > > As I said in another mail, I don't think it is a proper fix. > > > > > > What we did here - just changed one assumption to another. > > > > > > Current assumption - all mbuf obj_init() would setup buf_len in= exactly the same manner as rte_pktmbuf_init() does: > > > > > > buf_len =3D mp->elt_size - sizeof(struct rte_mbuf); > > > > > > New assumption - all mbuf obj_init() would setup buf_len for al= l mbufs in the pool to the same value. > > > > > > Both assumptions, I believe, are not always correct. > > > > > > Though, probably the new one would be true more often. > > > > > > > > > > > > I still think the proper fix is not to update mbuf's buf_len fi= eld at ixgbe_rxq_rearm() at all. > > > > > > We should just leave the original value unmodified. > > > > > > Actually, while looking at ixgbe_rxq_rearm(), I don't see any r= eason why we need to update buf_len field. > > > > > > It is not the data that need to be rearmed. > > > > > > The fields that need to be rearmed are: > > > > > > uint16_t data_off; > > > > > > uint16_t refcnt > > > > > > uint8_t nb_segs; > > > > > > uint8_t port; > > > > > > > > > > > > 6B in total. > > > > > > We probably would like to keep rearming as one 64bit load/store= . > > > > > > Though straight below them we have: > > > > > > uint64_t ol_flags; > > > > > > > > > > > > As RX fully override ol_flags anyway, we can safely overwrite f= irst 2B of it. > > > > > > That would allow us to still read/write whole 64bits and avoid = overwriting buf_len. > > > > > > I am talking about something like patch below. > > > > > > I admit that it looks not so pretty, but I think it is much saf= er and correct. > > > > > > Konstantin > > > > > > > > > > I just don't see this as worthwhile doing. We are looking here at= an mbuf pool > > > > > which is to be used for packet buffers for RX. If the packet buff= ers in that > > > > > pool are all of different sizes then we need to go back and look = at other places > > > > > throughout the code too. For instance, we query the buffer length= of the mbuf > > > > > pool when initializing the RX queues to determine if we need to e= nable scattered > > > > > RX. If the mbufs in the pool can potentially be of different size= s, we need to > > > > > turn off the no-scattered-packets optimization and always use the= scatter > > > > > packets code path - because the assumption that buffers don't get= resized could > > > > > also be false. Similarly here, if the mbufs are going to be of di= fferent > > > > > sizes then the user should disable the vector PMD, and use RX cod= e that doesn't > > > > > override the buf_len each time. > > > > > > > > Yes, all buffers in the pool supposed to be not smaller than some t= hreshold. > > > > But it doesn't mean that buffer can't be bigger. > > > > Let say our usual case - buf_len =3D 2K. > > > > Why it should be prohibited to use for RX mbuf with buf_len =3D=3D = 4K? > > > > There is absolutely nothing wrong with it. > > > > > > > > > > > > > > Furthermore, we still don't have an actual use-case where the use= r would want to > > > > > have different size mbufs in an mbuf pool used for RX. They can s= till have > > > > > variable-sized mbufs in other pools, but having all buffers the s= ame size in the > > > > > pool used to receive packets seems a perfectly fair restriction t= o have. If > > > > > someone has an app that they are creating that needs this functio= nality, I'll > > > > > reconsider this opinion, but right now this is all theoretical. > > > > > > > > Well, the question is why do we need to override buf_len at all? > > > > At first place, it doesn't look correct. > > > > Second - what advantage we are gaining from it? > > > > Performance? > > > > I tried with the changes below and didn't see any performance diffe= rence at all. > > > > So what is the point in keeping that potential point of failure? > > > > > > > > About use cases - I think I gave two examples in my previous mail. > > > > Not sure why you consider them artificial. > > > > From other side, do we have right now any actual use-case where peo= ple setup buf_len to something > > > > different then: mp->elt_size - sizeof(struct rte_mbuf)? > > > > > > > > Konstantin > > > > > > > Ok, thanks for the info Konstantin - I assume there is a current case= where the > > > mbuf is being initialized to something other than the default, on the= basis that > > > this patch was proposed as a fix for it. > > > > > > However, rather than argue this further, I'll defer to whatever the r= est of the > > > community decides. Thomas, Jean-Mickael, David - what are your opinio= ns on this? > > > > That's ok, but I still wonder why do we need to keep buf_len under rear= m_data marker and overwrite it for each RX? > > Honestly, probably there exist some obvious reason, that I am missing? > > To keep code tidy and easier to read? > > Or to keep whole (uint64_t)rearm_data on 64bits boundary? > > That are all good reasons, but I suppose data integrity is more importa= nt, right? > > > Yes, those are the reasons it was put there (easy to read, aligned), but = also > because it was viewed as being constant for a queue, i.e. the case where = you > would have different sized mbufs in a pool was never considered. [The fir= st line > of our documentation for the mempools is "A memory pool is an allocator o= f a > fixed-sized object", and everything is based around that idea of fixed si= ze > object pools.] > When doing the mbuf redesign, I pulled together what were the fields that= were > constant for each queue, and grouped them to a single 8-byte write to giv= e > faster (and shorter in terms of code) mbuf initialization. >=20 Ok, thanks for explanation. I'll submit the fix I am proposing as RFC patch. Konstantin