From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR01-HE1-obe.outbound.protection.outlook.com (mail-he1eur01on0066.outbound.protection.outlook.com [104.47.0.66]) by dpdk.org (Postfix) with ESMTP id A0E398E69 for ; Wed, 25 Apr 2018 19:06:58 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=Al3AnZGMj2qKGkJzBy/cSKgR6cb9IbP/6ZjzT0HnlWE=; b=gT7w9Len9pnLJbKUQj+g0nNy6qF+wckLPYv1MnM5gmfjsUcNn0AwJNatIW9p3G89MOMPF2lmN+0CSpab+6xB1rcBoxUXixyiaRss8fsMtRtHvmY7EpVufP2xy/dIChyqEMIvEGpXe+O+4E/rhWqOWyJFDt4HsLMmHj6KA8tlMYM= Received: from yongseok-MBP.local (209.116.155.178) by AM5PR0501MB2036.eurprd05.prod.outlook.com (2603:10a6:203:1a::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.715.18; Wed, 25 Apr 2018 17:06:53 +0000 Date: Wed, 25 Apr 2018 10:06:39 -0700 From: Yongseok Koh To: "Ananyev, Konstantin" Cc: "Lu, Wenzhuo" , "Wu, Jingjing" , "olivier.matz@6wind.com" , "dev@dpdk.org" , "arybchenko@solarflare.com" , "stephen@networkplumber.org" , "thomas@monjalon.net" , "adrien.mazarguil@6wind.com" , "nelio.laranjeiro@6wind.com" Message-ID: <20180425170638.GB3268@yongseok-MBP.local> References: <20180310012532.15809-1-yskoh@mellanox.com> <20180425025341.10590-1-yskoh@mellanox.com> <2601191342CEEE43887BDE71AB977258AEBCF98C@IRSMSX102.ger.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2601191342CEEE43887BDE71AB977258AEBCF98C@IRSMSX102.ger.corp.intel.com> User-Agent: Mutt/1.9.3 (2018-01-21) X-Originating-IP: [209.116.155.178] X-ClientProxiedBy: BYAPR07CA0011.namprd07.prod.outlook.com (2603:10b6:a02:bc::24) To AM5PR0501MB2036.eurprd05.prod.outlook.com (2603:10a6:203:1a::22) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(48565401081)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:AM5PR0501MB2036; X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2036; 3:kZGbqtAxzWRZzkl2vdkGl4M46VnGLKZK1vXuLI2rMjiI11h/wocqByjzxXYCHRAE4SJ0jxqy8v2uX9N6O/h2mjVOjSlklFV0PTFzILgh/8wKroetz0pqvdyGVQyJJ6MZ7X3/vg2tT/7zPOBRhoiFsF2zJA7Ay8EduXVIr/NKiN1dnXLdMIhrgXwhmeA9sxwNclUJIygh/b1kqvfFhk4C8g6TDUrF1jDNj7LuqTeRdJfN3fcT+mKhO0g3lkSAe6YS; 25:c5jQoU5qEYzSHmJUmR/6bvZ8Cjn5OS9mfWM1fL7YoqPy8GvX/BWCUDmLsPaR3acu0+Qe3g6+J42hOn8/FkXFlYM2RP3cCEjMKR9QdrHzqN2F5zaSNvp8whagmuoWwjekERgvqDDlH9z2FlPUWgjz1Uf+Dqy5HbVBDRtFtCd5h0wnurarAXNu2wubT/sWngxMAdrTtofPXH4NKM/u7nADG7KIN6Tlv/vg+nU1tLUaM2bMg8/Bm6nAsTIDwun22xda3LLifezdr3jpz8oRYLhHKoj+OUQnHDWbu+9mem1ElQzpxt6H/yS/qImC5LyX+DeUc4mjJnZHiNjSmIo9aSKaqA==; 31:AGmbp1MMAmNMWWg68Yik29r6yw4GEfDfFg9uyjA91+Ohx1nVyLqwMq+o6AyiONGtPr+jTAbtJEcf5kLBI1DnZoM/xDfl1tq9u8Fi9+wC/SlhiDGM9secPpERQ9J4IwWj/jJyhvZl4j2HlmpYn+BkEXsJfb5x/TBBRatL/uOrexmymdNTz7j/x4JrTFEyKqRLQhB7ugWdS0s/v68ygRvf/CO68Z9JkxVGvhfpxpOrbFo= X-MS-TrafficTypeDiagnostic: AM5PR0501MB2036: Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2036; 20:6heEbpf+3OuThdyGjtvkvMB4py/7QYVkezfK+l+nfJz5cskm4a+Dz7o5rPsOhH2QQ6xkV0ILNvgPK8k0sN4VfLDnkkT8L98+AYunu5aD2hkl19AY/moekaRm/XK67UeQ2A+uQkA+t8B6Icnearg2zcCJBGl+HqYcFP9H1GxYrzc6ORBjliggcml8e020a69oO3S3WqmO00V3pyDrww9ggRx7BOJKnaQAUMIRADhz/gvKJA4tcPNP7lsnYDgh8TeYk+FaJ4dB3x5m7VmRFGbSflHN4I9XLqfqeUnIYLY19p8wPaf37jvWjled4NBAUArK51StKRpSmh8oJXV208dYaRQaQbDxF3U9Iq8HXgjQkTdRtwFldl3i+s9qJqeTCmTGw6ajDe2j0mjSi3xy52qs0+Hve24IJ8GjZ9Z3LaBCpzmz4r9kNGpMrMTYdEBonNvY/BjAyFhBciru/g+0taaqPA673UrKJKbw2wNiy0Z9lKIdX1CXJ4urY9jvBPejtxTz; 4:UedS1u3QKsdPso6cimYyaCDlzrxxaZJJzaZS/Ol1QRqM06nIXofsngwyEAPqZpxnxWnteTAoKjJOowU5AL7XYZl/aHnyOKkHMgu0SO3B17z8nsXu0IdV4BYuG8S58FRz7xhFXJXKHdnFxN1nt/96gjkcM8pyC4mKREZ8SrDI1qB0jElZBovVpjpHDx079CWVrQIUtqcK33P9JIieOwnN1VH6ZiL7F5CyXqyPm/mdykaqugg0NjaneQrD5CPcJx5qz7R0YkQHXiKNne/hfQt6Sp+rG3EDQ5qQ6o9GNANoPc+zGXv8ljsYqGGtXhREd58O X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(17755550239193); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(10201501046)(93006095)(93001095)(3231232)(944501410)(52105095)(3002001)(6055026)(6041310)(20161123562045)(20161123558120)(20161123560045)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011); SRVR:AM5PR0501MB2036; BCL:0; PCL:0; RULEID:; SRVR:AM5PR0501MB2036; X-Forefront-PRVS: 06530126A4 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(396003)(376002)(366004)(39860400002)(346002)(39380400002)(189003)(199004)(33896004)(98436002)(7696005)(386003)(6506007)(59450400001)(81166006)(58126008)(47776003)(81156014)(305945005)(446003)(16526019)(54906003)(33656002)(11346002)(478600001)(52116002)(76176011)(26005)(186003)(6666003)(7736002)(16586007)(316002)(6916009)(4326008)(5890100001)(68736007)(8676002)(66066001)(86362001)(6246003)(8936002)(6116002)(3846002)(97736004)(9686003)(105586002)(106356001)(53936002)(55016002)(7416002)(5660300001)(1076002)(23726003)(50466002)(486006)(476003)(25786009)(2906002)(229853002)(956004)(18370500001); DIR:OUT; SFP:1101; SCL:1; SRVR:AM5PR0501MB2036; H:yongseok-MBP.local; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; AM5PR0501MB2036; 23:9oTFOjk5leonBlPQzVE1AsFhHsFj5B/AwdH+YrC?= =?us-ascii?Q?cvRggPzwGEoQP0yjVsZUaCbtec2Fy898bD9h+57hm+9ys9IiqAIjBARzoUJ6?= =?us-ascii?Q?05cuHhwE5kWu6gt7aOHcVhLEix9U51YrhQaYHyfOZX+mKqGBdzOOJuGAQTQY?= =?us-ascii?Q?BaZXQGh4F5MTWGfIpB7aK1T1xEUL2c9UWgUJeLiFBqiO83DsJ5AeG00tfvS1?= =?us-ascii?Q?b7kFaN/xn+cmUDQ2V/tcgYKsa3KPupGhzxYEWgKaU6cSAMyfcwNRdgIJENai?= =?us-ascii?Q?0eLbULAyl0OE+/cwbzvbyGCcN+mtWxVjKYNmrHrdr79NyIJRJaljRVBXaLY9?= =?us-ascii?Q?GWT6JRLbDbBBwIhHoiv6im4ma52wgLPMxznPdENAOIgTgxSl0c8/TSGhwXMS?= =?us-ascii?Q?hT/6csR32Tdy2ZRiAqF4FQ6c7YQg6qVtkzGfuHD+8XZt5xzN6kp0i2Ys/QED?= =?us-ascii?Q?JqQuKDZSH99wGWxc6vO7FSsM5PHUIJs9zPsbKX4UT5mctRDSBWGxibtwcgCS?= =?us-ascii?Q?cv81iBq7SN+SM0rDSmHnNfOrXDCt1PU2K2pW7p/kxLl+XZmEm32QU0+RR2zO?= =?us-ascii?Q?S/Nh1uvYfAYA6mcPc8pBUkvLXH29I5rboGXg1/AxBn9fpgJnzQ8tgUfkuOUX?= =?us-ascii?Q?a5D8d145fmTxCv3zUj99XjqGrL5BOHJ4ILxNMnJ9BoFns/D3bClrwYPFv58r?= =?us-ascii?Q?YujrXvfQzNipgIHZKnkdBWhdAyKBw1IJZcQpMbqzH8f+dyWhHmc776hSOaUb?= =?us-ascii?Q?1coFp4ut2k5ozPcaKKMoIpvojfQeyhO0dZa3jCCfH8g8/Q/s3yCX/xIycUmw?= =?us-ascii?Q?f1Mi5JN7AK09/dG6xiwuCo9UUaLgacdrxK++E96BbJH/9QP/lzz+d7UyvGiQ?= =?us-ascii?Q?vv2zIZ3W2l7Eeu5DfVZnUL0QmH/SIiftwsL+6a3gQ9YKKnM1dWv9yBmSH5oW?= =?us-ascii?Q?R1tFqIiSSPfDwUQ+nqs5fB8qCXbb9eE51mB+Z01Qv/sVAd0D0QY3R3tQcnWw?= =?us-ascii?Q?FaLWGi85IEFu0L9O5ahJvWNLSUCRBY4wORI9BycEGNCy9R+785rWaOJ8Mek4?= =?us-ascii?Q?5hPNjjomwTmESUH1W50y6pfowDK0PyfDlMfb4faqLR5zeYJX4IrL3cJRf6em?= =?us-ascii?Q?YkYG0uNwuHXPBX/pnmPd3GjniFr9IBVhCq0n3mR318LBCOZM9K0ViIkYP2v6?= =?us-ascii?Q?qhGJlKI49TuVB8CYgkerN1A0FaptP0V9wsxufOyFVSkDgJ5sdaVw0En89hXS?= =?us-ascii?Q?8j4YNH1HBMEzAyrH9HeNT7YSYXKyhAvHjIuM5cxzqkmN/On5/RGcV6AaQkDm?= =?us-ascii?Q?EzJyz2i7ah6yoAmi2WHmWZ2fiKjqcfIIp/oGiAoo0Fi9WpVzhEe4xftdk6RL?= =?us-ascii?Q?7mfAS1Wu/6PeHWwhgXcDtq2R0CgM=3D?= X-Microsoft-Antispam-Message-Info: 8CMdfu5TVlNndlyHn3su+kWxt1lsMWkOutZbEwgFdjClMAyRF+9v6vIdUYrPs9YO6RYKw5TzfX/+prX/tE2q6cGXI985MGDgeTaQBW85qpDL0bQThm7cOFI+AQjZSL/06H6w+YYwxJAM/nMwrSSmSrcz3Q6jpCAFD0v5p82JEkP1ZzlUdoekyBvALqF9JU6w X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2036; 6:h7drpyAtxV6XlcdRiZt2yT5m+9d5RWMResFU8zEGxr3ztTPCtbqjOhR4p5grbH0fQF4VCBwaHdb1HpsYC9S+0/bi3jkkHWCqn8M4rBFGtkx/XB3lAXx4ZyYixRZVUiR9/9R2BcO0rrvrsdjNi/D7MiyRU7q3CKwPKWDG31gh6XIQDBQiLAYTnKwq9G/qx2LA8wsi6z6Y+HodO6NdjrBbpLtRxVzqM7lnZSqUi3vm4/pUc0dDvDBvKQW7eFgyn2cRcUVMJyK3Vbi0L9/SN8Ltom/ABkhW73UF9d8tEmf7ndfDSxaE+watyYM8MWuASJKtZ7u03Ux3iAeW6PVdcW9JMujk1CPZWU4KsSv+g/yXR0CkUzPvau2R8pQGA6vLUVdUkevnUdNJNQbOuxN+CEGW48lmfxjW/W2JPHQMpgZXXCQA/PU1ki10XfkVCbCvqQKDdoRDQB3gzSblPFOjrUlxIQ==; 5:qx7RuupiS/yL5cCTIzwtc8PL6TE2B430v8qzK0zh3dKjfY6E3xnmrX0pyDi3STET4KLlhVWzSXNYdNK24KSE8VapMIRDYO02YxzJKDUQUUVp5ZaSZnT2KhMhM87ki+LlvhnoSCGI1Fap2PxIhFN1OxKZTZZ0SSEuzt5FPiVw2xU=; 24:5V/X+UybhGy/IEw21yPw36VUHEnCqxkH5eE0TlhKrGLbOzkkmH/zPYAXFfZy5Ap7SI308qd9lxap22UQSHNip4V3GI4goo/K+vnOgfITERM= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2036; 7:pEtMkVoMNabD3YEiJsERrDblylToe2fIE9Bhc8hVbClq7QmfxU5V8EEp7sIQ8QRn1LVPL9upa0HgwpDwAtqnV3yVnl+vYR5LOgFN/TwVdarBoFZ/gxbub+PGyYjOloE7+M5N8ocBVK9/cXa6ZuMcXf8/iqxXNsijL+89xPWRvanPAeQaYGeXBGJWYWf+G9yuaWDkFJ7WVhtzqHj1Er1bQAezCLPGESLSnDg9IcUT3RaDvojGPKCZ5sJ0atsAG9UJ X-MS-Office365-Filtering-Correlation-Id: 1767e4c3-9c8e-4404-97cf-08d5aacef2e8 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Apr 2018 17:06:53.9512 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1767e4c3-9c8e-4404-97cf-08d5aacef2e8 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM5PR0501MB2036 Subject: Re: [dpdk-dev] [PATCH v5 1/2] mbuf: support attaching external buffer to mbuf X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Apr 2018 17:06:58 -0000 On Wed, Apr 25, 2018 at 01:31:42PM +0000, Ananyev, Konstantin wrote: [...] > > /** Mbuf prefetch */ > > #define RTE_MBUF_PREFETCH_TO_FREE(m) do { \ > > if ((m) != NULL) \ > > @@ -1213,11 +1306,127 @@ static inline int rte_pktmbuf_alloc_bulk(struct rte_mempool *pool, > > } > > > > /** > > + * Attach an external buffer to a mbuf. > > + * > > + * User-managed anonymous buffer can be attached to an mbuf. When attaching > > + * it, corresponding free callback function and its argument should be > > + * provided. This callback function will be called once all the mbufs are > > + * detached from the buffer. > > + * > > + * The headroom for the attaching mbuf will be set to zero and this can be > > + * properly adjusted after attachment. For example, ``rte_pktmbuf_adj()`` > > + * or ``rte_pktmbuf_reset_headroom()`` can be used. > > + * > > + * More mbufs can be attached to the same external buffer by > > + * ``rte_pktmbuf_attach()`` once the external buffer has been attached by > > + * this API. > > + * > > + * Detachment can be done by either ``rte_pktmbuf_detach_extbuf()`` or > > + * ``rte_pktmbuf_detach()``. > > + * > > + * Attaching an external buffer is quite similar to mbuf indirection in > > + * replacing buffer addresses and length of a mbuf, but a few differences: > > + * - When an indirect mbuf is attached, refcnt of the direct mbuf would be > > + * 2 as long as the direct mbuf itself isn't freed after the attachment. > > + * In such cases, the buffer area of a direct mbuf must be read-only. But > > + * external buffer has its own refcnt and it starts from 1. Unless > > + * multiple mbufs are attached to a mbuf having an external buffer, the > > + * external buffer is writable. > > + * - There's no need to allocate buffer from a mempool. Any buffer can be > > + * attached with appropriate free callback and its IO address. > > + * - Smaller metadata is required to maintain shared data such as refcnt. > > + * > > + * @warning > > + * @b EXPERIMENTAL: This API may change without prior notice. > > + * Once external buffer is enabled by allowing experimental API, > > + * ``RTE_MBUF_DIRECT()`` and ``RTE_MBUF_INDIRECT()`` are no longer > > + * exclusive. A mbuf can be considered direct if it is neither indirect nor > > + * having external buffer. > > + * > > + * @param m > > + * The pointer to the mbuf. > > + * @param buf_addr > > + * The pointer to the external buffer we're attaching to. > > + * @param buf_iova > > + * IO address of the external buffer we're attaching to. > > + * @param buf_len > > + * The size of the external buffer we're attaching to. If memory for > > + * shared data is not provided, buf_len must be larger than the size of > > + * ``struct rte_mbuf_ext_shared_info`` and padding for alignment. If not > > + * enough, this function will return NULL. > > + * @param shinfo > > + * User-provided memory for shared data. If NULL, a few bytes in the > > + * trailer of the provided buffer will be dedicated for shared data and > > + * the shared data will be properly initialized. Otherwise, user must > > + * initialize the content except for free callback and its argument. The > > + * pointer of shared data will be stored in m->shinfo. > > + * @param free_cb > > + * Free callback function to call when the external buffer needs to be > > + * freed. > > + * @param fcb_opaque > > + * Argument for the free callback function. > > + * > > + * @return > > + * A pointer to the new start of the data on success, return NULL > > + * otherwise. > > + */ > > +static inline char * __rte_experimental > > +rte_pktmbuf_attach_extbuf(struct rte_mbuf *m, void *buf_addr, > > + rte_iova_t buf_iova, uint16_t buf_len, > > + struct rte_mbuf_ext_shared_info *shinfo, > > + rte_mbuf_extbuf_free_callback_t free_cb, void *fcb_opaque) > > +{ > > + /* Additional attachment should be done by rte_pktmbuf_attach() */ > > + RTE_ASSERT(!RTE_MBUF_HAS_EXTBUF(m)); > > Shouldn't we have here something like: > RTE_ASSERT(RTE_MBUF_DIRECT(m) && rte_mbuf_refcnt_read(m) == 1); > ? Right. That's better. Attaching mbuf should be direct and writable. > > + > > + m->buf_addr = buf_addr; > > + m->buf_iova = buf_iova; > > + > > + if (shinfo == NULL) { > > Instead of allocating shinfo ourselves - wound's it be better to rely > on caller always allocating afeeling it for us (he can do that at the end/start of buffer, > or whenever he likes to. It is just for convenience. For some users, external attachment could be occasional and casual, e.g. punt control traffic from kernel/hv. For such non-serious cases, it is good to provide this small utility. > Again in that case - caller can provide one shinfo to several mbufs (with different buf_addrs) > and would know for sure that free_cb wouldn't be overwritten by mistake. > I.E. mbuf code will only update refcnt inside shinfo. I think you missed the discussion with other people yesterday. This change is exactly for that purpose. Like I documented above, if this API is called with shinfo being provided, it will use the user-provided shinfo instead of sparing a few byte in the trailer and won't touch the shinfo. This code block happens only if user doesn't provide memory for shared data (shinfo is NULL). > > + void *buf_end = RTE_PTR_ADD(buf_addr, buf_len); > > + > > + shinfo = RTE_PTR_ALIGN_FLOOR(RTE_PTR_SUB(buf_end, > > + sizeof(*shinfo)), sizeof(uintptr_t)); > > + if ((void *)shinfo <= buf_addr) > > + return NULL; > > + > > + m->buf_len = RTE_PTR_DIFF(shinfo, buf_addr); > > + rte_mbuf_ext_refcnt_set(shinfo, 1); > > + } else { > > + m->buf_len = buf_len; > > I think you need to update shinfo>refcnt here too. Like explained above, if shinfo is provided, it doesn't alter anything except for callbacks and its arg. Thanks, Yongseok