From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR01-HE1-obe.outbound.protection.outlook.com (mail-he1eur01on0083.outbound.protection.outlook.com [104.47.0.83]) by dpdk.org (Postfix) with ESMTP id 294E47CBA for ; Thu, 26 Apr 2018 19:18:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=NlSfROD6erAP5H9iRL/uXhbw4MndE8V+/eZnVjg+Y3g=; b=CkJS9xgUE0S/BA4K2Ir8OQM821wscHBHjSjEqTx/21I/j+cXoJvb4gHbbH2idd+e7wrwKxu/inqHxJtMzQyyseQborBFgmg/Q4edn8V62Fwjvu0HkLt970PlXgUj1IyhI0yn6rLUcGjMrcLJ7wk8UepJefgZCmBJgtNhd4u2h54= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; Received: from yongseok-MBP.local (209.116.155.178) by VI1PR0501MB2045.eurprd05.prod.outlook.com (2603:10a6:800:36::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.696.13; Thu, 26 Apr 2018 17:18:26 +0000 Date: Thu, 26 Apr 2018 10:18:14 -0700 From: Yongseok Koh To: Andrew Rybchenko Cc: wenzhuo.lu@intel.com, jingjing.wu@intel.com, olivier.matz@6wind.com, dev@dpdk.org, konstantin.ananyev@intel.com, stephen@networkplumber.org, thomas@monjalon.net, adrien.mazarguil@6wind.com, nelio.laranjeiro@6wind.com Message-ID: <20180426171813.GA8437@yongseok-MBP.local> References: <20180310012532.15809-1-yskoh@mellanox.com> <20180426011010.28078-1-yskoh@mellanox.com> <1a65f081-4c92-78d5-b00c-08e66fdef5c8@solarflare.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1a65f081-4c92-78d5-b00c-08e66fdef5c8@solarflare.com> User-Agent: Mutt/1.9.3 (2018-01-21) X-Originating-IP: [209.116.155.178] X-ClientProxiedBy: BYAPR07CA0020.namprd07.prod.outlook.com (2603:10b6:a02:bc::33) To VI1PR0501MB2045.eurprd05.prod.outlook.com (2603:10a6:800:36::19) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(48565401081)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:VI1PR0501MB2045; X-Microsoft-Exchange-Diagnostics: 1; VI1PR0501MB2045; 3:tzHU8AFXErXu0N1snW2rtiNcGeIfoovFtDTr0DGnW/yLOgQ7Onq8mDs09E+U+afaFIXbbiU+koxzx98k45AhMg3g7kX5U1x3fvENpFk6Z31xxXh8MoFAb5nSiSFL53JF4hEa4t3rOAxMpinK7emnvwyFyTDzz367Ku4UK2FMJIeDi32YXkesMw/Y8f03zEegXj3xRkRSIJRiPLsp+CTrP6w6YOgDD4GLS3OCg7OEpAncBjx+Ss/Plln8PTF1HUTe; 25:cVL18naGPI/T3AIsjxFkGe2g/S4CD58hTYtAQxkGwHO1sqcpilEigrV0egOeX+LJoRDQhOgJ9Q3b2DS08gmUFG3wGkJj/5eMGTSfrT14FcqfE9vlzwwaz4NLa3SA7cpmTtxihz4R174a2vQiwqjGCudqxw8LwC+aP7gtdk3zNxDUKb+hDPl6qp/uFZjPo4dDA6I2chOHO59dWbZXYBRTA99WGxnNy7/BScG6Fnxuz+ud6S8RO2KPd8aSoAlW3VwwcK5h8Y0yquICgPnoN807hzG09cj9vtbehQaFAjU+v5/bW/nKrrKGdclzsmJp4mdd1fuEG65UL01tJssA602W9Q==; 31:uCp5hsUY9qlslIv32zvqXtuaKQZj5y9lv2DM4+eIoJ7BVVTElf1gmQJa4CTA9Eqw81CJPrxdWOqv7oDhQD5kPRVrJjkbMdj51Z+143LMVc9AGobfiBzmFb5+xvwrLCf/+vyvaV4avzpCHQOc1M6hOP/JY+N0o3u6gByInZRXrsObpWv1Q6KJD2LgwD8tP0J/S/1wcOMqt5dRDt0S1JetxkMtUPCLZvdtei7HAd8iEy0= X-MS-TrafficTypeDiagnostic: VI1PR0501MB2045: X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr X-Microsoft-Exchange-Diagnostics: 1; VI1PR0501MB2045; 20:Qf69FK7egpPrYSFCai/aLwHZllJ7t08vdngCeuwey8QoCGwZ+BYjALabkjOokuf1C/q0sjhehHCWlySyZ4GQ6QWHZ2pj56gp6MitY7UM5DwgWXh6bkhpbQ0gLJqjqOAp0TjgtkfXDXSwtHmd6jzlDDycg5YGKgrqUaBBnCq1CRF1Q71Q7p1S06J7MJuPaSr066XatUXqgQB656YRUPsqN7diFV2pfo28xg1mcKA2IcWNA49oNdxKur1js4Z89dyyTdPJKi9d6KY9hLcvkcXnL3UkX0byvc4RtvGU7vuTGuVHBko5zsPwEhrhW6b+A4c51///8L8YpeVWGYsa/r6mXt2yCILOOhCPPbSuzuJQvnAHJDolOM0vvTNWDCpGo/ZGzaE4SuoHl0JEnxzvqba1WnGo2n96uhCK1grW+mnI/wUHfLy5EHfUnOcddbUFTYKAa0VKiO4zLMh+TJpEOvaBbe94sE+BqaTFNxND2fbGc8GVYBGKYa+IkJF4yPHutkH3; 4:K3wdX5ttGHsqze8kx4nZKB/Wdlv0Wh47unXl+mOS0VndCDilpfvMjzq9c75TTpGAuhQCmJkbTMhreEuP0QzjaaYF/5AIFFEXlBZrSjagtcFHUvY4ZxQF2iw39yEadL9u3SafsIji3UEAE7g8I0Qafyf9y3P+DzvGzb/2eiJKTXkuIoHeiEQpIKHFw/KH+POgr0OGPR6gqjZ0XTsraocIFQm6C2SdUo9wgbqKXM3PTPnmNpAWzqIh3A9LbKRI505pm83OiKHmJV4CnYOUlary8A== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(10201501046)(93006095)(93001095)(3231232)(944501410)(52105095)(3002001)(6055026)(6041310)(20161123562045)(20161123558120)(20161123560045)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011); SRVR:VI1PR0501MB2045; BCL:0; PCL:0; RULEID:; SRVR:VI1PR0501MB2045; X-Forefront-PRVS: 0654257CF5 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(366004)(376002)(39860400002)(396003)(346002)(39380400002)(189003)(199004)(47776003)(86362001)(66066001)(23756003)(3846002)(1076002)(6116002)(16526019)(68736007)(186003)(97736004)(50466002)(33656002)(98436002)(446003)(229853002)(106356001)(105586002)(55016002)(2870700001)(2906002)(476003)(486006)(956004)(53936002)(11346002)(6916009)(5890100001)(25786009)(305945005)(478600001)(5660300001)(7416002)(4326008)(6666003)(9686003)(6246003)(26005)(8936002)(316002)(7736002)(8676002)(81166006)(33896004)(81156014)(6506007)(59450400001)(52116002)(386003)(53546011)(58126008)(76176011)(7696005)(18370500001)(357404004); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR0501MB2045; H:yongseok-MBP.local; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?iso-8859-1?Q?1; VI1PR0501MB2045; 23:LEYEZh7mtzjC7EjjWK+Rdh+NKBGhwhdSyEh6c?= =?iso-8859-1?Q?qz75KGYvK9u82BAcqgz1j2uQPhY74kI+J8gIP0ysa4P3y2dJF073goQEEN?= =?iso-8859-1?Q?9st+SDzP1P4tZfGmfMUcm6Xd3JbVL2B9XhR1C0QYru84RP5NHaGaRd62c5?= =?iso-8859-1?Q?7jhep7RZAsDOucYBaYTI90u1fXjfdmyJ1d84GuABjg//xSy4o5AmSKLSip?= =?iso-8859-1?Q?HQaHxbdy2yCPqi9J2rXELPddCf7APIlDXOgQAPjRTbfeurB+OfT+FOTO/k?= =?iso-8859-1?Q?Y9JsggvmlCMs2Dxex7iUTyJhrO+dW5Ho8UxrUPkmApDgFdRZMFk2ZR2kCO?= =?iso-8859-1?Q?pLx039GaA3m7VyxzCGRpabA/IUjT57WJjIPuD5KP7fpaFQz1NwcFUH4qA6?= =?iso-8859-1?Q?Cx/3QnOxok9+feJrVEpYymmXFIXIs2sBfAyy16idOHSuJlfAh/iuHabLHZ?= =?iso-8859-1?Q?+xwdypZqhMvygl7s+XSEXPvAhv0J8/vRGtC1w862CO1hM3ZJGTnsh4Mhyu?= =?iso-8859-1?Q?jXcDsrmUnaJqT09sy8Z3DLqOWNeH0TxKdn+ZNklfXBmhYF8pmYEM2R7VTD?= =?iso-8859-1?Q?uDz+mvizVdGu/c3qCyZeI+MvyNJ4sgJffq/R/9ZeeJWLsd9aXVVhIZn7Ys?= =?iso-8859-1?Q?1sVjLktSd31763JSWHLbVysc8i23DMpBv0YZHXJOnZVAY7zxbIKZsAv12V?= =?iso-8859-1?Q?Zagd2xN/AkEiPWhwoIbvtJQr6wNMG5Iq7uuYFVAW386dg5jK3wJVaxHQJ+?= =?iso-8859-1?Q?H4+kIvCyOzCwbHYe181aAtjmLL48rO7CO+L8tkwkt7SaB0gJBBwUA6GvZv?= =?iso-8859-1?Q?/nooJzRT1ldg4ynSj+GxqVv22Nd83dT1PRVqO9oVODu4yOM/EWNaf7DxUb?= =?iso-8859-1?Q?3Oz/UNOf56F2GdN0m9YfrGtN/fWn5GfLqPnCDA/nTYMa4Kq3U3T7LsLqHl?= =?iso-8859-1?Q?s6/PFcZAiLGyzqGlC1l9W4iM1/RLd/67ON/IOB42La13uIEccgmlk5nq5f?= =?iso-8859-1?Q?s7zrU3+rjkIitoXJnGRieUiTgRUtLPlLOAnO53vlPbGbGEsJDGijIhs90q?= =?iso-8859-1?Q?BYU/vvKVoDu521nxVSVie5sv1CE5rnLg6tDX/WTOkoiVEGnBOlfsfpFqS2?= =?iso-8859-1?Q?/qq8vaHu82vEZndLNmTKRNit42IF4uJpL+5WHj0l10OX4jzbOeaxG3voIv?= =?iso-8859-1?Q?xqm6YVA9KmFkNkPcR0a0ha2FKsJrQcg8OtYjTueJlJa2pgDW5WpJM5mIyM?= =?iso-8859-1?Q?1t+mlpcjp2+T/Y4gfFN+yhHioJuC+9tAKWfm2jIuTuYypbFFUAoJkzWTqI?= =?iso-8859-1?Q?cvu1k9DlCCJ5OH9sxyGBheeYGpTfYXj1FHPZwfTBISysFfQN/npNBuL5mW?= =?iso-8859-1?Q?seRo3LaTJUoTaD4CiujP8VB0o1bukx1iSl+DxNylCz/CQ602vQntVw01BQ?= =?iso-8859-1?Q?FDCv+kcAqplx7IZrvl297OX+uKIQdbObtlrh8h1wnPOscezvuZyQNwiesN?= =?iso-8859-1?B?dz09?= X-Microsoft-Antispam-Message-Info: gCUjIMwKANktPqkG4sLCfFV/IwG+mmcBMiElInFzOXffMQJf6RFQmIu89cTn6M8GDf21jOh1r57ZV2wWM06TCtYgveog+I937DWdVu/bv8ArqAmNTHd+CJEg5bvn7Ik+TiIhswEmVrQLHtBJzguGVH/pflvyUHrtBk/aHb7hHYWUW3HBMf+eTA0Ishl5oddj X-Microsoft-Exchange-Diagnostics: 1; VI1PR0501MB2045; 6:+swY66uiexhAQuuO1AV39OPFxE5yVqXaVNTDrSHIm8MKLNJHZh7haYAmonh4ngHaQgQIsnMUfokcR8eJKfRlwDZSuh98mrIHXJum6EnDz0rfYiowvagGWeSqNwmkBWjrHE/J9HpbL8DWN6fbOQx8gffyKO5rGxqc9nL5frzs3oqv8BFaAMUKieczqCnUJhkffJHMhR9RABFpkbyHBU1fjA0hkkKpQeNrYrID8R7EyymwoBEq21LSqVapILghRJD2VXO+min/clOq8u0EXk2Efof/wIBWrg+Zgrjtoe3uSaMrKFRt7ygj7sPBsjMGnnjI9l1pvCpm4/b6KijsPjHYoSXCo/9n3WefSCm/4GEF8S5Ns0zUwhSAOAuKTZ+vFDu8HFILfi4p+O016Bu7DZJop/Fvj7K9yOS3G70X8j/0ArmJXWOfC+4Ua0PmnyIxft7AaInFJKbQfnIDz/QPyZzbTg==; 5:R/F2fMyHeAh7XdqVqzog3IJ/3MzUvU8Vk9jUwGTv2QRRYKA6KiI9fCCaYya5fV573SOie5kQKO1ojDknwMSqPFKmfAZO8rN18SY39eSAYejMnqKLoa+TBBjcYAxY02o1gMQkjDAB/kcYHT9ZqMoStZo9gp2FfQf2+Cfoo2V7vVo=; 24:ZSinfKvOFZbGWlCKGKqO5Z+tFal6pv4a1ojSE2KzwFIcGtEsREI/C9ls5nCEEiT4favRvrv8usmzzdc8PzuzxvehXUSpPkcb2c37wzTvO8A= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; VI1PR0501MB2045; 7:+Gi/Pn1ZtxUHs2cEb67M62oMfGxEqp4YVC7ycGP10V1cS7Wxby4hjP+EblEtp7Ls2jmLjTwgL0dNr0giY7IZb0VnP4e8jeoI4n6BHUxWXVNzDSFhdn3mfcJ/aaix7RbsZKJj3KWPkpKB7Ptk9zfNdJMCXfidRNrpIel5odG/iyWnegvcflJqhAGkXwtCyeTX/3q/G7G9zP+BCpPPNKNqgd2fCAQ3MlgapGmAw0r+fnFxpk6bbsfIkyOWMTvdIOjo X-MS-Office365-Filtering-Correlation-Id: c30954da-b08f-42e8-cc37-08d5ab99ba25 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Apr 2018 17:18:26.1365 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c30954da-b08f-42e8-cc37-08d5ab99ba25 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0501MB2045 Subject: Re: [dpdk-dev] [PATCH v6 1/2] mbuf: support attaching external buffer to mbuf X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Apr 2018 17:18:30 -0000 On Thu, Apr 26, 2018 at 07:05:01PM +0300, Andrew Rybchenko wrote: > On 04/26/2018 04:10 AM, Yongseok Koh wrote: > > This patch introduces a new way of attaching an external buffer to a mbuf. > > > > Attaching an external buffer is quite similar to mbuf indirection in > > replacing buffer addresses and length of a mbuf, but a few differences: > > - When an indirect mbuf is attached, refcnt of the direct mbuf would be > > 2 as long as the direct mbuf itself isn't freed after the attachment. > > In such cases, the buffer area of a direct mbuf must be read-only. But > > external buffer has its own refcnt and it starts from 1. Unless > > multiple mbufs are attached to a mbuf having an external buffer, the > > external buffer is writable. > > - There's no need to allocate buffer from a mempool. Any buffer can be > > attached with appropriate free callback. > > - Smaller metadata is required to maintain shared data such as refcnt. > > I'm still unsure how reference counters for external buffer work. > > Let's consider the following example: > >                         |<-mbuf1 buf_len-->|<-mbuf2 buf_len-->| > +--+-----------+----+--------------+----+--------------+---+- - - > |  global      |head|mbuf1 data    |head|mbuf2 data    | | > |  shinfo=2    |room| |room|              |   | > +--+-----------+----+--------------+----+--------------+---+- - - >                ^ ^ > +----------+   |      +----------+ | > | mbuf1    +---+      | mbuf2    +-+ > | refcnt=1 |          | refcnt=1 | > +----------+          +----------+ > > I.e. we have big buffer which is sliced into many small data > buffers referenced from mbufs. > > shinfo reference counter is used to control when big buffer > may be freed. But what controls sharing of each block? > > headroom and following mbuf data (buf_len) is owned by > corresponding mbuf and the mbuf owner can do everything > with the space (prepend data, append data, modify etc). > I.e. it is read-write in above terminology. > > What should happen if mbuf1 is cloned? Right now it will result > in a new mbuf1a with reference counter 1 and incremented shinfo > reference counter. And what does say that corresponding area > is read-only now? It looks like nothing. > > As I understand it should be solved using per data area shinfo > which free callback decrements big buffer reference counter. I have to admit that I was confused at the moment and I mixed two different use-cases. 1) Transmitting a large storage block.                |<--mbuf1 buf_len-->|<--mbuf2 buf_len-->| +--+-----------+----+--------------+----+--------------+---+- - - |  global      |head|mbuf1 data    |head|mbuf2 data    | | |  shinfo=2    |room| |room|              |   | +--+-----------+----+--------------+----+--------------+---+- - -                ^ ^ +----------+   |      +----------+ | | mbuf1    +---+      | mbuf2    +-+ | refcnt=1 |          | refcnt=1 | +----------+          +----------+ ^ ^       |next |next +-----+----+       +----------+ | mbuf1_hdr|      | mbuf2_hdr| | refcnt=1 |          | refcnt=1 | +----------+          +----------+ Yes, in this case, the large external buffer should always be read-only. And necessary network headers should be linked via m->next. Owners of m1 or m2 can't alter any bit in the external buffer because shinfo->refcnt > 1. 2) Slicing a large buffer and provide r-w buffers.                |<--mbuf1 buf_len-->| |<--mbuf2 buf_len-->| +--+-----------+----+--------------+------+----+--------------+------+----+- - - |  user data |head|mbuf1 data    |shinfo|head|mbuf2 data    |shinfo| | |  refc=2    |room| |refc=1|room|              |refc=1|   | +--+-----------+----+--------------+------+----+--------------+------+----+- - -                ^ ^ +----------+   |       +----------+ | | mbuf1    +---+      | mbuf2    +-+ | refcnt=1 |          | refcnt=1 | +----------+           +----------+ Here, the user data for the large chunk isn't rte_mbuf_ext_shared_info but a custom structure managed by user in order to free the whole chunk. free_cb would decrement a custom refcnt in custom way. But librte_mbuf doesn't need to be aware of it. It is user's responsibility. The library is just responsible for calling free_cb when shinfo->refcnt gets to zero. > So, we have two reference counters per each mbuf with external > buffer (plus reference counter per big buffer). > Two reference counters sounds too much and it looks like > mbuf-with-extbuf reference counter is not really used > (since on clone/attach update shinfo refcnt). > It is still two counters to check on free. Each refcnt implies whether it is r or r-w. Even for direct mbuf, if two users are accessing it, refcnt is 2 and it is read-only. This should mean both mbuf metadata and its data area are all read-only. Users can alter neither various length fields nor packet data, for example. For non-direct mbufs, still its refcnt should be used, but refcnt only represents the metadata is shared and read-only if it is more than 1. So, refcnt of mbuf-with-extbuf is still used. Incrementing refcnt means an entity acquired access to the object, including cases of attachment (indirec/extbuf). > Have you considered alternative approach to use mbuf refcnt > as sharing indicator for extbuf data? However, in this case > indirect referencing extbuf would logically look like: > > +----------+    +--------+     +--------+ > | indirect +--->| extbuf +---->|  data  | > |  mbuf    |    |  mbuf  |     |        | > +----------+    +--------+     +--------+ > > It looks like it would allow to avoid two reference counters > per data block as above. Personally I'm not sure which approach > is better and would like to hear what you and other reviewers > think about it. So, I still think this patch is okay. > Some minor notes below as well. > > > Signed-off-by: Yongseok Koh > > --- > > > > ** This patch can pass the mbuf_autotest. ** > > > > Submitting only non-mlx5 patches to meet deadline for RC1. mlx5 patches > > will be submitted separately rebased on a differnet patchset which > > accommodates new memory hotplug design to mlx PMDs. > > > > v6: > > * rte_pktmbuf_attach_extbuf() doesn't take NULL shinfo. Instead, > > rte_pktmbuf_ext_shinfo_init_helper() is added. > > * bug fix in rte_pktmbuf_attach() - shinfo wasn't saved to mi. > > * minor changes from review. > > > > v5: > > * rte_pktmbuf_attach_extbuf() sets headroom to 0. > > * if shinfo is provided when attaching, user should initialize it. > > * minor changes from review. > > > > v4: > > * rte_pktmbuf_attach_extbuf() takes new arguments - buf_iova and shinfo. > > user can pass memory for shared data via shinfo argument. > > * minor changes from review. > > > > v3: > > * implement external buffer attachment instead of introducing buf_off for > > mbuf indirection. > > > > lib/librte_mbuf/rte_mbuf.h | 335 +++++++++++++++++++++++++++++++++++++++++---- > > 1 file changed, 306 insertions(+), 29 deletions(-) > > > > diff --git a/lib/librte_mbuf/rte_mbuf.h b/lib/librte_mbuf/rte_mbuf.h > > index 43aaa9c5f..0a6885281 100644 > > --- a/lib/librte_mbuf/rte_mbuf.h > > +++ b/lib/librte_mbuf/rte_mbuf.h [...] > > /** Mbuf prefetch */ > > #define RTE_MBUF_PREFETCH_TO_FREE(m) do { \ > > if ((m) != NULL) \ > > @@ -1213,11 +1307,157 @@ static inline int rte_pktmbuf_alloc_bulk(struct rte_mempool *pool, > > } > > /** > > + * Initialize shared data at the end of an external buffer before attaching > > + * to a mbuf by ``rte_pktmbuf_attach_extbuf()``. This is not a mandatory > > + * initialization but a helper function to simply spare a few bytes at the > > + * end of the buffer for shared data. If shared data is allocated > > + * separately, this should not be called but application has to properly > > + * initialize the shared data according to its need. > > + * > > + * Free callback and its argument is saved and the refcnt is set to 1. > > + * > > + * @warning > > + * buf_len must be adjusted to RTE_PTR_DIFF(shinfo, buf_addr) after this > > + * initialization. For example, > > May be buf_len should be inout and it should be done by the function? > Just a question since current approach looks fragile. Yeah, I thought about that but I didn't want to alter user's variable, I thought it could be error-prone. Anyway either way is okay to me. Will wait for a day to get input because I will send out a new version (hopefully last :-) to fix the nit you mentioned below. > > + * > > + * struct rte_mbuf_ext_shared_info *shinfo = > > + * rte_pktmbuf_ext_shinfo_init_helpfer(buf_addr, buf_len, > > + * free_cb, fcb_arg); > > + * buf_len = RTE_PTR_DIFF(shinfo, buf_addr); > > + * rte_pktmbuf_attach_extbuf(m, buf_addr, buf_iova, buf_len, shinfo); > > + * > > + * @param buf_addr > > + * The pointer to the external buffer. > > + * @param buf_len > > + * The size of the external buffer. buf_len must be larger than the size > > + * of ``struct rte_mbuf_ext_shared_info`` and padding for alignment. If > > + * not enough, this function will return NULL. > > + * @param free_cb > > + * Free callback function to call when the external buffer needs to be > > + * freed. > > + * @param fcb_opaque > > + * Argument for the free callback function. > > + * > > + * @return > > + * A pointer to the initialized shared data on success, return NULL > > + * otherwise. > > + */ > > +static inline struct rte_mbuf_ext_shared_info * > > +rte_pktmbuf_ext_shinfo_init_helper(void *buf_addr, uint16_t buf_len, > > + rte_mbuf_extbuf_free_callback_t free_cb, void *fcb_opaque) > > +{ > > + struct rte_mbuf_ext_shared_info *shinfo; > > + void *buf_end = RTE_PTR_ADD(buf_addr, buf_len); > > + > > + shinfo = RTE_PTR_ALIGN_FLOOR(RTE_PTR_SUB(buf_end, > > + sizeof(*shinfo)), sizeof(uintptr_t)); > > + if ((void *)shinfo <= buf_addr) > > + return NULL; > > + > > + rte_mbuf_ext_refcnt_set(shinfo, 1); > > + > > + shinfo->free_cb = free_cb; > > + shinfo->fcb_opaque = fcb_opaque; > > Just a nit, but I'd suggest to initialize in the same order as in the > struct. > (if there is no reasons why reference counter should be initialized first) Will do. Thanks, Yongseok