From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7D8CCA04DD; Thu, 5 Nov 2020 14:55:52 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id AD5F9C80A; Thu, 5 Nov 2020 14:55:50 +0100 (CET) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id A4D69C806 for ; Thu, 5 Nov 2020 14:55:47 +0100 (CET) IronPort-SDR: bDTIrKMoQJzUQdif7xN/Ljky56tmrNXlskxDoM9ytyaXqfczfxyiVCZMICSTXae5JhjMTcfDkD YwGJwfDhsaHg== X-IronPort-AV: E=McAfee;i="6000,8403,9795"; a="169525043" X-IronPort-AV: E=Sophos;i="5.77,453,1596524400"; d="scan'208";a="169525043" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Nov 2020 05:55:45 -0800 IronPort-SDR: MoGViT03hqtfvLRi0bVkeA2z92OiR8yMbiKEgx2Rhjl46GG9nlPUpP60F01ffx2haiN74oPcWp 3ZxUh2K0d/Lg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,453,1596524400"; d="scan'208";a="471657210" Received: from fmsmsx603.amr.corp.intel.com ([10.18.126.83]) by orsmga004.jf.intel.com with ESMTP; 05 Nov 2020 05:55:45 -0800 Received: from fmsmsx611.amr.corp.intel.com (10.18.126.91) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 5 Nov 2020 05:55:45 -0800 Received: from fmsmsx603.amr.corp.intel.com (10.18.126.83) by fmsmsx611.amr.corp.intel.com (10.18.126.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 5 Nov 2020 05:55:44 -0800 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5 via Frontend Transport; Thu, 5 Nov 2020 05:55:44 -0800 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (104.47.74.43) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.1713.5; Thu, 5 Nov 2020 05:55:42 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RTJ9SiWAd5eC8jmMl+JsuCVwHWDVGoTE69ObRNviFtYNMBkECAIxoK7/Qk/soWy0N/tc9GpvMwNwYz5vwpc2/d9rR0k1+3j8fCkMkSjB5GvuXvx5NddLu5Ui8eAQGpOdiz8wytlclPeWMLIO49CipsNO4uTGJl98t/7p/cmYrYlBj+qT7LCYC1F49zKXJqna40bLNNWFjmHpj+1EYP21Doc31CTIIm+Rpk8x0UwtIfisY8rZ1bqCZ+vPQB2gEqzL5hT7N+e33bX9GRc5yBCtBMHO/2sVaer5lPHg5z+wcf1X40C4A/zjLSYWm7lv3JjSKbELvazRG1R3drOJVBJ7OQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iiq7RW24ZBLEfKfAz+wV2OHU25D8gZVDAKZBG2nx/Yw=; b=fxcxTfY9o2lEuy/wRv7kqJCbbq56eYNW4usEML4FcHBm6JWRUJT8iXEabs8vM8PaGPiSyspydqaCAWK2oiG+2kt1TM9wzdXeZy6J4wI4EmMdH08/0+lSiQoVM4l639ywYx5baWHRITpuODOPLpbY2CPTH8wW31LyQ0GFmogHQAlMYjuSRFBtDxF3xuYbRv/uSJjC7dFugIJ70VLPXI94StViNujmY6lZV6OwOYMd5eUE4c9g7V8yV/aBRj51P3K+6PiWb3M9AXkSgySMMgzcYdeUvBZ9NNgX0f+xKG8UgTsJGkhxsqyqa56T2/7y3dXjlHusKfevvjKkVB0EYr9a3w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iiq7RW24ZBLEfKfAz+wV2OHU25D8gZVDAKZBG2nx/Yw=; b=baS1TheUCgNO66aR/f+sS3kSEu8H+oGfciAjtw0/FfgursM4lWjr7H5Sc1AM1ulZvpO10P5ta3rEvfiAJKNEGTDmh1k+RGqrK2udQ29Qtwj/38YwMPY7ibGrHjpDYn/0nN32+eFAJjheSjEFW8FRLMzYZDcuwgWqvAnd56J+Bnc= Received: from BYAPR11MB3301.namprd11.prod.outlook.com (2603:10b6:a03:7f::26) by SJ0PR11MB4845.namprd11.prod.outlook.com (2603:10b6:a03:2d1::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3499.27; Thu, 5 Nov 2020 13:55:39 +0000 Received: from BYAPR11MB3301.namprd11.prod.outlook.com ([fe80::f5a4:3f6b:ade3:296b]) by BYAPR11MB3301.namprd11.prod.outlook.com ([fe80::f5a4:3f6b:ade3:296b%3]) with mapi id 15.20.3499.032; Thu, 5 Nov 2020 13:55:39 +0000 From: "Ananyev, Konstantin" To: Olivier Matz CC: Andrew Rybchenko , "dev@dpdk.org" Thread-Topic: [dpdk-dev] [PATCH] mbuf: fix reset on mbuf free Thread-Index: AQHWssw9/Ru+bbJv2UmxCtlz1ORQn6m4ql1ggAB/kQCAAAtLgIAADCkAgAAdJVCAABslAIAACA7wgAAGsgCAAAdYsA== Date: Thu, 5 Nov 2020 13:55:38 +0000 Message-ID: References: <20201104170007.8026-1-olivier.matz@6wind.com> <20201105074626.GL1898@platinum> <10b675b5-c376-746f-9ae2-210685b0b05d@oktetlabs.ru> <20201105091022.GO1898@platinum> <20201105123150.GP1898@platinum> <20201105132438.GS1898@platinum> In-Reply-To: <20201105132438.GS1898@platinum> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-reaction: no-action dlp-version: 11.5.1.3 authentication-results: 6wind.com; dkim=none (message not signed) header.d=none;6wind.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [46.7.39.127] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 2a550f00-249e-4bf2-a97e-08d881927a56 x-ms-traffictypediagnostic: SJ0PR11MB4845: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: pJjD3biJRk8nyY+HXtLGGCCY0FhEXHwOqrrvIFjFON6b7GExOybsqiOP7HfvH+O7wm3fiS5PCOk+ak7ALlbEfdX3zeMA2GDcMhvwjzWtRvrpGa09yXm6Cn7cgk86sQmbKfQjpWKiOXfqzSErPgr5oGfw/I5FqdVftXPqvqyUDivC2O52DH28DrxsdCVYrQ/VZqLAez+bX8dDXJGAc3LJp2Q8TnaPDONBuo6fUY3TD2u4GwsXTkVkAvo6/utGscoXfhhVOoVAG5XIa3ovuzDZzfKuli8pMq8YRLLW7b8ObDVGsfHxKi0cmgSzdTdTDhm6Tno/WzUyc7V6yrEPjqNk2Q== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR11MB3301.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(66476007)(66946007)(76116006)(2906002)(66446008)(5660300002)(64756008)(66556008)(26005)(8676002)(83380400001)(498600001)(186003)(54906003)(71200400001)(6506007)(8936002)(52536014)(7696005)(9686003)(33656002)(55016002)(86362001)(4326008)(53546011)(6916009); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: SdInCX7UjW3LJnI7pDGPNHe3zS7Q7FhnHx7Wogc4ryRP62/tCoFOvPkLCFwHqceX5XgC6u66skGfKhVyDtlKmZEG4hb/lb9FXGFU0WqwhMOULy5k3sMCb4xg1jjTw5n0HV/qgRDMzRyv9uzqtMgSw52oIFmSLoS9jtGWwZjwJcOlXLUryyEFAhHo+auEiCYJB/OWifO4HfSVMZCLm9n6Yz0ONu0jEOjUNkJPt5oy9HBMLz88toF7/cwQKTUyERtylYEt0ajC9dmP29PIxryp1i+FOM+Y3HGovED3AIV+304HAURis5d8a0UoU/zw5+SoilKfICOsPqycIMQeaPMMwMeAeZCgP+hl3EB9sUvGFpvLmeA7DWd8K4XDVFCT3pBI8SR3osiFJ5NQKuyJeEGu3ookL2XbZi+fHGXAaJAwZhwv/i4KYhM0sDqRXCEIipiuNK+bHWP8x1u+rYOeGx5FlUJak5Y4PNvFcilv5mn/P1K2CixkCOHROyChxXY6ctmNnUWCfttItqceFTIm5Cz6HYj1iF2NGkRrsBRYX9nd6f0raQLZvNMKNwLkynAkmgABoUYdP0Mr2g/BUga3wT0ANDrnSYP07B4NAE/VDHNxRtD6GhEnJGgBnXgVk+vrTwHtKL9h41vu3viQYQHh0bGf8A== x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3301.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2a550f00-249e-4bf2-a97e-08d881927a56 X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Nov 2020 13:55:38.9354 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: TSjq3oXsXCMCO9+GoxbWEBGO1bZ8ZNPmeGpCecv/zPcYhXgUZyVB4ji4Mmr57TqaogQ8CNHtPTxc3yZfzprVlSslGdXZKq64eVkfzH2hbFk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB4845 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] [PATCH] mbuf: fix reset on mbuf free X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > > > > > > > > On Thu, Nov 05, 2020 at 11:34:18AM +0000, Ananyev, Konstantin wrote: > > > > > > > > > > > > > On Thu, Nov 05, 2020 at 11:26:51AM +0300, Andrew Rybchenko wrote: > > > > > > On 11/5/20 10:46 AM, Olivier Matz wrote: > > > > > > > On Thu, Nov 05, 2020 at 12:15:49AM +0000, Ananyev, Konstantin= wrote: > > > > > > >> > > > > > > >> Hi Olivier, > > > > > > >> > > > > > > >>> m->nb_seg must be reset on mbuf free whatever the value of = m->next, > > > > > > >>> because it can happen that m->nb_seg is !=3D 1. For instanc= e in this > > > > > > >>> case: > > > > > > >>> > > > > > > >>> m1 =3D rte_pktmbuf_alloc(mp); > > > > > > >>> rte_pktmbuf_append(m1, 500); > > > > > > >>> m2 =3D rte_pktmbuf_alloc(mp); > > > > > > >>> rte_pktmbuf_append(m2, 500); > > > > > > >>> rte_pktmbuf_chain(m1, m2); > > > > > > >>> m0 =3D rte_pktmbuf_alloc(mp); > > > > > > >>> rte_pktmbuf_append(m0, 500); > > > > > > >>> rte_pktmbuf_chain(m0, m1); > > > > > > >>> > > > > > > >>> As rte_pktmbuf_chain() does not reset nb_seg in the initial= m1 > > > > > > >>> segment (this is not required), after this code the mbuf ch= ain > > > > > > >>> have 3 segments: > > > > > > >>> - m0: next=3Dm1, nb_seg=3D3 > > > > > > >>> - m1: next=3Dm2, nb_seg=3D2 > > > > > > >>> - m2: next=3DNULL, nb_seg=3D1 > > > > > > >>> > > > > > > >>> Freeing this mbuf chain will not restore nb_seg=3D1 in the = second > > > > > > >>> segment. > > > > > > >> > > > > > > >> Hmm, not sure why is that? > > > > > > >> You are talking about freeing m1, right? > > > > > > >> rte_pktmbuf_prefree_seg(struct rte_mbuf *m) > > > > > > >> { > > > > > > >> ... > > > > > > >> if (m->next !=3D NULL) { > > > > > > >> m->next =3D NULL; > > > > > > >> m->nb_segs =3D 1; > > > > > > >> } > > > > > > >> > > > > > > >> m1->next !=3D NULL, so it will enter the if() block, > > > > > > >> and will reset both next and nb_segs. > > > > > > >> What I am missing here? > > > > > > >> Thinking in more generic way, that change: > > > > > > >> - if (m->next !=3D NULL) { > > > > > > >> - m->next =3D NULL; > > > > > > >> - m->nb_segs =3D 1; > > > > > > >> - } > > > > > > >> + m->next =3D NULL; > > > > > > >> + m->nb_segs =3D 1; > > > > > > > > > > > > > > Ah, sorry. I oversimplified the example and now it does not > > > > > > > show the issue... > > > > > > > > > > > > > > The full example also adds a split() to break the mbuf chain > > > > > > > between m1 and m2. The kind of thing that would be done for > > > > > > > software TCP segmentation. > > > > > > > > > > > > > > > > > > > If so, may be the right solution is to care about nb_segs > > > > > > when next is set to NULL on split? Any place when next is set > > > > > > to NULL. Just to keep the optimization in a more generic place. > > > > > > > > > > > > > The problem with that approach is that there are already several > > > > > existing split() or trim() implementations in different DPDK-base= d > > > > > applications. For instance, we have some in 6WINDGate. If we forc= e > > > > > applications to set nb_seg to 1 when resetting next, it has to be > > > > > documented because it is not straightforward. > > > > > > > > I think it is better to go that way. > > > > From my perspective it seems natural to reset nb_seg at same time > > > > we reset next, otherwise inconsistency will occur. > > > > > > While it is not explicitly stated for nb_segs, to me it was clear tha= t > > > nb_segs is only valid in the first segment, like for many fields (por= t, > > > ol_flags, vlan, rss, ...). > > > > > > If we say that nb_segs has to be valid in any segments, it means that > > > chain() or split() will have to update it in all segments, which is n= ot > > > efficient. > > > > Why in all? > > We can state that nb_segs on non-first segment should always equal 1. > > As I understand in that case, both split() and chain() have to update n= b_segs > > only for head mbufs, rest ones will remain untouched. >=20 > Well, anyway, I think it's strange to have a constraint on m->nb_segs for > non-first segment. We don't have that kind of constraints for other field= s. True, we don't. But this is one of the fields we consider critical for proper work of mbuf alloc/free mechanism.=20 >=20 >=20 > > > > > > > > Saying that nb_segs has to be valid for the first and last segment se= ems > > > really odd to me. What is the logic behind that except keeping this t= est > > > as is? > > > > > > In any case, it has to be better documented. > > > > > > > > > Olivier > > > > > > > > > > > I think the approach from > > > > > this patch is safer. > > > > > > > > It might be easier from perspective that changes in less places are= required, > > > > Though I think that this patch will introduce some performance drop= . > > > > As now each mbuf_prefree_seg() will cause update of 2 cache lines u= nconditionally. > > > > > > > > > By the way, for 21.11, if we are able to do some optimizations an= d have > > > > > both pool (index?) and next in the first cache line, we may recon= sider > > > > > the fact that next and nb_segs are already set for new allocated = mbufs, > > > > > because it is not straightforward either. > > > > > > > > My suggestion - let's put future optimization discussion aside for = now, > > > > and concentrate on that particular patch. > > > > > > > > > > > > > > > > After this operation, we have 2 mbuf chain: > > > > > > > - m0 with 2 segments, the last one has next=3DNULL but nb_se= g=3D2 > > > > > > > - new_m with 1 segment > > > > > > > > > > > > > > Freeing m0 will not restore nb_seg=3D1 in the second segment. > > > > > > > > > > > > > >> Assumes that it is ok to have an mbuf with > > > > > > >> nb_seg > 1 and next =3D=3D NULL. > > > > > > >> Which seems wrong to me. > > > > > > > > > > > > > > I don't think it is wrong: nb_seg is just ignored when not in= the first > > > > > > > segment, and there is nothing saying it should be set to 1. T= ypically, > > > > > > > rte_pktmbuf_chain() does not change it, and I guess it's the = same for > > > > > > > many similar functions in applications. > > > > > > > > > > > > > > Olivier > > > > > > > > > > > > > >> > > > > > > >> > > > > > > >>> This is expected that mbufs stored in pool have their > > > > > > >>> nb_seg field set to 1. > > > > > > >>> > > > > > > >>> Fixes: 8f094a9ac5d7 ("mbuf: set mbuf fields while in pool") > > > > > > >>> Cc: stable@dpdk.org > > > > > > >>> > > > > > > >>> Signed-off-by: Olivier Matz > > > > > > >>> --- > > > > > > >>> lib/librte_mbuf/rte_mbuf.c | 6 ++---- > > > > > > >>> lib/librte_mbuf/rte_mbuf.h | 12 ++++-------- > > > > > > >>> 2 files changed, 6 insertions(+), 12 deletions(-) > > > > > > >>> > > > > > > >>> diff --git a/lib/librte_mbuf/rte_mbuf.c b/lib/librte_mbuf/r= te_mbuf.c > > > > > > >>> index 8a456e5e64..e632071c23 100644 > > > > > > >>> --- a/lib/librte_mbuf/rte_mbuf.c > > > > > > >>> +++ b/lib/librte_mbuf/rte_mbuf.c > > > > > > >>> @@ -129,10 +129,8 @@ rte_pktmbuf_free_pinned_extmem(void *a= ddr, void *opaque) > > > > > > >>> > > > > > > >>> rte_mbuf_ext_refcnt_set(m->shinfo, 1); > > > > > > >>> m->ol_flags =3D EXT_ATTACHED_MBUF; > > > > > > >>> - if (m->next !=3D NULL) { > > > > > > >>> - m->next =3D NULL; > > > > > > >>> - m->nb_segs =3D 1; > > > > > > >>> - } > > > > > > >>> + m->next =3D NULL; > > > > > > >>> + m->nb_segs =3D 1; > > > > > > >>> rte_mbuf_raw_free(m); > > > > > > >>> } > > > > > > >>> > > > > > > >>> diff --git a/lib/librte_mbuf/rte_mbuf.h b/lib/librte_mbuf/r= te_mbuf.h > > > > > > >>> index a1414ed7cd..ef5800c8ef 100644 > > > > > > >>> --- a/lib/librte_mbuf/rte_mbuf.h > > > > > > >>> +++ b/lib/librte_mbuf/rte_mbuf.h > > > > > > >>> @@ -1329,10 +1329,8 @@ rte_pktmbuf_prefree_seg(struct rte_m= buf *m) > > > > > > >>> return NULL; > > > > > > >>> } > > > > > > >>> > > > > > > >>> - if (m->next !=3D NULL) { > > > > > > >>> - m->next =3D NULL; > > > > > > >>> - m->nb_segs =3D 1; > > > > > > >>> - } > > > > > > >>> + m->next =3D NULL; > > > > > > >>> + m->nb_segs =3D 1; > > > > > > >>> > > > > > > >>> return m; > > > > > > >>> > > > > > > >>> @@ -1346,10 +1344,8 @@ rte_pktmbuf_prefree_seg(struct rte_m= buf *m) > > > > > > >>> return NULL; > > > > > > >>> } > > > > > > >>> > > > > > > >>> - if (m->next !=3D NULL) { > > > > > > >>> - m->next =3D NULL; > > > > > > >>> - m->nb_segs =3D 1; > > > > > > >>> - } > > > > > > >>> + m->next =3D NULL; > > > > > > >>> + m->nb_segs =3D 1; > > > > > > >>> rte_mbuf_refcnt_set(m, 1); > > > > > > >>> > > > > > > >>> return m; > > > > > > >>> -- > > > > > > >>> 2.25.1 > > > > > > >> > > > > > >