From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by dpdk.org (Postfix) with ESMTP id 15A7D4C96 for ; Mon, 25 Mar 2019 22:06:07 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 25 Mar 2019 14:06:07 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,270,1549958400"; d="scan'208";a="145156289" Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203]) by orsmga002.jf.intel.com with ESMTP; 25 Mar 2019 14:06:06 -0700 Received: from fmsmsx126.amr.corp.intel.com (10.18.125.43) by FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS) id 14.3.408.0; Mon, 25 Mar 2019 14:06:04 -0700 Received: from fmsmsx105.amr.corp.intel.com ([169.254.4.162]) by FMSMSX126.amr.corp.intel.com ([169.254.1.215]) with mapi id 14.03.0415.000; Mon, 25 Mar 2019 14:06:04 -0700 From: "Howell, Seth" To: "dev@dpdk.org" CC: "Harris, James R" Thread-Topic: Aligned rte_mempool for storage applications Thread-Index: AdTjSAn8LkaY++b+RHO/HMbypNe5Og== Date: Mon, 25 Mar 2019 21:06:04 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-version: 11.0.400.15 dlp-reaction: no-action x-ctpclassification: CTP_NT x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiZDVkMzVkN2ItM2QxOS00YmQ3LTgzMjYtNjk4NzA0ZWM2ODFhIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiZThsVEdqSVNOQzZcL0R2OHZLend3aWtRdmxpTUUyUk9cL0hwUHFNVEFIcW9HUzZtamNuQlVlMGtyWDR0XC9mem5qNCJ9 x-originating-ip: [10.1.200.106] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: [dpdk-dev] Aligned rte_mempool for storage applications X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Mar 2019 21:06:08 -0000 Hello, In SPDK, we use the rte_mempool struct for many internal structure collecti= ons. The per-thread cache and ease of allocation of mempools are very usefu= l features. Some of the collections we store in SPDK are pools of I/O buffers. Typicall= y, these pools contain elements of at least 4096 bytes, and we would like t= hem to be aligned to 4k for performance reasons. Currently, the rte_mempool API doesn't support aligned mempool objects. Thi= s means that when we allocate a 4k buffer and want it aligned to 4k, we act= ually need to allocate an 8k buffer and calculate an offset into it each ti= me we want to use it. We recently did a proof of concept using the rte_mempool_ops hook where we = allocated a mempool and populated it with aligned entries. This allowed us = to retrieve aligned addresses directly from rte_mempool_get(), but didn't h= elp with the allocation size. Because the rte_mempool struct assumes that each element has a header attac= hed to it, we still need to live up to that assumption for each object we c= reate in a mempool. This means that the actual size of a buffer becomes 4k = + 24 bytes. In order to get to our next aligned address, we need to add abo= ut 4k of padding to each element. Modifying the current rte_mempool struct to allow entries without headers s= eems impossible since it would break rte_mempool_for_obj_iter and rte_mempo= ol_from_obj. However I still think there is a lot of benefit to be gained f= rom a mempool structure that supports aligned objects without headers. I am wondering if DPDK would be open to us introducing an rte_mempool_align= ed structure. This structure would essentially be a wrapper around a regula= r mempool struct. However, it would not require headers or trailers for eac= h object in the pool. This structure would only be applicable to a subset of mempools with the fo= llowing characteristics: 1. mempools for which the following flags were set: MEMPOOL_F_NO_CACHE_ALI= GNED, MEMPOOL_F_NO_IOVA_CONTIG , MEMPOOL_F_NO_SPREAD 2. mempools that do not require the use of the following functions rte_mem= pool_from_obj (requires a pointer to the mp in the header of each obj), rte= _mempool_for_obj_iter. 3. Any attempt to create this object when RTE_LIBRTE_MEMPOOL_DEBUG was ena= bled would necessarily fail since we can't check the header cookies. My thought would be that we could implement this data structure in a header= and it would look something like this: Struct rte_mempool_aligned { Struct rte_mempool mp; Size_t obj_alignment; }; The rest of the functions in the header would primarily be wrappers around = the original functions. Most functions (rte_mempool_alloc, rte_mempool_free= , rte_mempool_enqueue/dequeue, rte_mempool_get_count, etc.) could be implem= ented directly as wrappers, and others such as rte_mempool_create and the p= opulate functions would have to be re-implemented to some degree in the new= header. The remaining functions (check_cookies, for_obj_iter) would not be= implemented in the rte_mempool_aligned.h file.=20 Would the community be welcoming of a new rte_mempool_aligned struct? If yo= u don't feel like this would be the way to go, are there other options in D= PDK for creating a pool of pre-allocated aligned objects?=20 Thank you, Seth Howell From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id 80E58A05D3 for ; Mon, 25 Mar 2019 22:06:11 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 198EB4C9C; Mon, 25 Mar 2019 22:06:10 +0100 (CET) Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by dpdk.org (Postfix) with ESMTP id 15A7D4C96 for ; Mon, 25 Mar 2019 22:06:07 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 25 Mar 2019 14:06:07 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,270,1549958400"; d="scan'208";a="145156289" Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203]) by orsmga002.jf.intel.com with ESMTP; 25 Mar 2019 14:06:06 -0700 Received: from fmsmsx126.amr.corp.intel.com (10.18.125.43) by FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS) id 14.3.408.0; Mon, 25 Mar 2019 14:06:04 -0700 Received: from fmsmsx105.amr.corp.intel.com ([169.254.4.162]) by FMSMSX126.amr.corp.intel.com ([169.254.1.215]) with mapi id 14.03.0415.000; Mon, 25 Mar 2019 14:06:04 -0700 From: "Howell, Seth" To: "dev@dpdk.org" CC: "Harris, James R" Thread-Topic: Aligned rte_mempool for storage applications Thread-Index: AdTjSAn8LkaY++b+RHO/HMbypNe5Og== Date: Mon, 25 Mar 2019 21:06:04 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-version: 11.0.400.15 dlp-reaction: no-action x-ctpclassification: CTP_NT x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiZDVkMzVkN2ItM2QxOS00YmQ3LTgzMjYtNjk4NzA0ZWM2ODFhIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiZThsVEdqSVNOQzZcL0R2OHZLend3aWtRdmxpTUUyUk9cL0hwUHFNVEFIcW9HUzZtamNuQlVlMGtyWDR0XC9mem5qNCJ9 x-originating-ip: [10.1.200.106] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: [dpdk-dev] Aligned rte_mempool for storage applications X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190325210604.vWfTLXVRqnqB5sPlwOS4fcuq8eZqhFqmDn39LX67038@z> Hello, In SPDK, we use the rte_mempool struct for many internal structure collecti= ons. The per-thread cache and ease of allocation of mempools are very usefu= l features. Some of the collections we store in SPDK are pools of I/O buffers. Typicall= y, these pools contain elements of at least 4096 bytes, and we would like t= hem to be aligned to 4k for performance reasons. Currently, the rte_mempool API doesn't support aligned mempool objects. Thi= s means that when we allocate a 4k buffer and want it aligned to 4k, we act= ually need to allocate an 8k buffer and calculate an offset into it each ti= me we want to use it. We recently did a proof of concept using the rte_mempool_ops hook where we = allocated a mempool and populated it with aligned entries. This allowed us = to retrieve aligned addresses directly from rte_mempool_get(), but didn't h= elp with the allocation size. Because the rte_mempool struct assumes that each element has a header attac= hed to it, we still need to live up to that assumption for each object we c= reate in a mempool. This means that the actual size of a buffer becomes 4k = + 24 bytes. In order to get to our next aligned address, we need to add abo= ut 4k of padding to each element. Modifying the current rte_mempool struct to allow entries without headers s= eems impossible since it would break rte_mempool_for_obj_iter and rte_mempo= ol_from_obj. However I still think there is a lot of benefit to be gained f= rom a mempool structure that supports aligned objects without headers. I am wondering if DPDK would be open to us introducing an rte_mempool_align= ed structure. This structure would essentially be a wrapper around a regula= r mempool struct. However, it would not require headers or trailers for eac= h object in the pool. This structure would only be applicable to a subset of mempools with the fo= llowing characteristics: 1. mempools for which the following flags were set: MEMPOOL_F_NO_CACHE_ALI= GNED, MEMPOOL_F_NO_IOVA_CONTIG , MEMPOOL_F_NO_SPREAD 2. mempools that do not require the use of the following functions rte_mem= pool_from_obj (requires a pointer to the mp in the header of each obj), rte= _mempool_for_obj_iter. 3. Any attempt to create this object when RTE_LIBRTE_MEMPOOL_DEBUG was ena= bled would necessarily fail since we can't check the header cookies. My thought would be that we could implement this data structure in a header= and it would look something like this: Struct rte_mempool_aligned { Struct rte_mempool mp; Size_t obj_alignment; }; The rest of the functions in the header would primarily be wrappers around = the original functions. Most functions (rte_mempool_alloc, rte_mempool_free= , rte_mempool_enqueue/dequeue, rte_mempool_get_count, etc.) could be implem= ented directly as wrappers, and others such as rte_mempool_create and the p= opulate functions would have to be re-implemented to some degree in the new= header. The remaining functions (check_cookies, for_obj_iter) would not be= implemented in the rte_mempool_aligned.h file.=20 Would the community be welcoming of a new rte_mempool_aligned struct? If yo= u don't feel like this would be the way to go, are there other options in D= PDK for creating a pool of pre-allocated aligned objects?=20 Thank you, Seth Howell