* [dpdk-dev] [PATCH 0/2] fix mempool when --no-huge option is set @ 2013-07-26 14:39 Damien Millescamps 2013-07-26 14:39 ` [dpdk-dev] [PATCH 1/2] mem: get hugepages config Damien Millescamps 2013-07-26 14:39 ` [dpdk-dev] [PATCH 2/2] mem: fix mempool for --no-huge Damien Millescamps 0 siblings, 2 replies; 7+ messages in thread From: Damien Millescamps @ 2013-07-26 14:39 UTC (permalink / raw) To: dev This set of patch permits to use the --no-huge option to create valid mempools. However, drivers necessitating a physically contiguous memory larger than a standard page won't work in this mode. Only vmxnet3-usermap will work in this mode among the available open source PMD. This can be useful for old kernels (< 2.6.33) and VM with limited amount of mempory. Damien Millescamps (2): mem: get hugepages config mem: fix mempool for --no-huge lib/librte_eal/common/include/rte_eal.h | 13 +++++++- lib/librte_eal/linuxapp/eal/eal.c | 4 ++ lib/librte_eal/linuxapp/eal/eal_memory.c | 2 +- lib/librte_mempool/rte_mempool.c | 54 +++++++++++++++++++++++++++++- lib/librte_mempool/rte_mempool.h | 20 +++++++---- 5 files changed, 83 insertions(+), 10 deletions(-) -- 1.7.2.5 ^ permalink raw reply [flat|nested] 7+ messages in thread
* [dpdk-dev] [PATCH 1/2] mem: get hugepages config 2013-07-26 14:39 [dpdk-dev] [PATCH 0/2] fix mempool when --no-huge option is set Damien Millescamps @ 2013-07-26 14:39 ` Damien Millescamps 2013-07-26 14:59 ` Adrien Mazarguil 2013-07-26 14:39 ` [dpdk-dev] [PATCH 2/2] mem: fix mempool for --no-huge Damien Millescamps 1 sibling, 1 reply; 7+ messages in thread From: Damien Millescamps @ 2013-07-26 14:39 UTC (permalink / raw) To: dev Allow external libraries and applications to know if hugepages are enabled. Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com> Signed-off-by: Damien Millescamps <damien.millescamps@6wind.com> --- lib/librte_eal/common/include/rte_eal.h | 13 ++++++++++++- lib/librte_eal/linuxapp/eal/eal.c | 4 ++++ 2 files changed, 16 insertions(+), 1 deletions(-) diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h index 240530d..58a6f90 100644 --- a/lib/librte_eal/common/include/rte_eal.h +++ b/lib/librte_eal/common/include/rte_eal.h @@ -214,7 +214,18 @@ int rte_eal_init(int argc, char **argv); } \ } while(0) - +/** + * Whether EAL is using huge pages (disabled by --no-huge option). + * The no-huge mode cannot be used with UIO poll-mode drivers like igb/ixgbe. + * It is useful for NIC drivers (e.g. librte_pmd_mlx4, librte_pmd_vmxnet3) or + * crypto drivers (e.g. librte_crypto_nitrox) provided by third-parties such + * as 6WIND. + * + * @return + * Nonzero if hugepages are enabled. + */ +int rte_eal_has_hugepages(void); + #ifdef __cplusplus } #endif diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c index ed0e9b1..52de2b6 100644 --- a/lib/librte_eal/linuxapp/eal/eal.c +++ b/lib/librte_eal/linuxapp/eal/eal.c @@ -945,3 +945,7 @@ rte_eal_process_type(void) return (rte_config.process_type); } +int rte_eal_has_hugepages(void) +{ + return ! internal_config.no_hugetlbfs; +} -- 1.7.2.5 ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-dev] [PATCH 1/2] mem: get hugepages config 2013-07-26 14:39 ` [dpdk-dev] [PATCH 1/2] mem: get hugepages config Damien Millescamps @ 2013-07-26 14:59 ` Adrien Mazarguil 2013-07-26 15:10 ` Thomas Monjalon 0 siblings, 1 reply; 7+ messages in thread From: Adrien Mazarguil @ 2013-07-26 14:59 UTC (permalink / raw) To: Damien Millescamps; +Cc: dev On Fri, Jul 26, 2013 at 04:39:12PM +0200, Damien Millescamps wrote: > Allow external libraries and applications to know if hugepages > are enabled. > > Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com> > Signed-off-by: Damien Millescamps <damien.millescamps@6wind.com> > --- > lib/librte_eal/common/include/rte_eal.h | 13 ++++++++++++- > lib/librte_eal/linuxapp/eal/eal.c | 4 ++++ > 2 files changed, 16 insertions(+), 1 deletions(-) Acked-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> -- Adrien Mazarguil 6WIND ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-dev] [PATCH 1/2] mem: get hugepages config 2013-07-26 14:59 ` Adrien Mazarguil @ 2013-07-26 15:10 ` Thomas Monjalon 0 siblings, 0 replies; 7+ messages in thread From: Thomas Monjalon @ 2013-07-26 15:10 UTC (permalink / raw) To: Adrien Mazarguil; +Cc: dev 26/07/2013 16:59, Adrien Mazarguil : > On Fri, Jul 26, 2013 at 04:39:12PM +0200, Damien Millescamps wrote: > > Allow external libraries and applications to know if hugepages > > are enabled. > > > > Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com> > > Signed-off-by: Damien Millescamps <damien.millescamps@6wind.com> > > --- > > > > lib/librte_eal/common/include/rte_eal.h | 13 ++++++++++++- > > lib/librte_eal/linuxapp/eal/eal.c | 4 ++++ > > 2 files changed, 16 insertions(+), 1 deletions(-) > > Acked-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> applied -- Thomas ^ permalink raw reply [flat|nested] 7+ messages in thread
* [dpdk-dev] [PATCH 2/2] mem: fix mempool for --no-huge 2013-07-26 14:39 [dpdk-dev] [PATCH 0/2] fix mempool when --no-huge option is set Damien Millescamps 2013-07-26 14:39 ` [dpdk-dev] [PATCH 1/2] mem: get hugepages config Damien Millescamps @ 2013-07-26 14:39 ` Damien Millescamps 2013-07-26 14:59 ` Adrien Mazarguil 1 sibling, 1 reply; 7+ messages in thread From: Damien Millescamps @ 2013-07-26 14:39 UTC (permalink / raw) To: dev In --no-huge mode, mempool provides objects with their associated header/trailer fitting in a standard page (usually 4KB). This means all non-UIO driver should work correctly in this mode, since UIO drivers allocate ring sizes that cannot fit in a page. Extend rte_mempool_virt2phy to obtain the correct physical address when elements of the pool are not on the same physically contiguous memory region. This is a first step for enhancement PR #29696. Reason for this patch is to be able to run on a kernel < 2.6.37 without the need to patch it, since all kernel below are either bugged or don't have huge page support at all (< 2.6.28). Signed-off-by: Damien Millescamps <damien.millescamps@6wind.com> --- lib/librte_eal/linuxapp/eal/eal_memory.c | 2 +- lib/librte_mempool/rte_mempool.c | 54 +++++++++++++++++++++++++++++- lib/librte_mempool/rte_mempool.h | 20 +++++++---- 3 files changed, 67 insertions(+), 9 deletions(-) diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c index ce0c2d8..00b5952 100644 --- a/lib/librte_eal/linuxapp/eal/eal_memory.c +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c @@ -780,7 +780,7 @@ rte_eal_hugepage_init(void) /* get pointer to global configuration */ mcfg = rte_eal_get_configuration()->mem_config; - /* for debug purposes, hugetlbfs can be disabled */ + /* hugetlbfs can be disabled */ if (internal_config.no_hugetlbfs) { addr = malloc(internal_config.memory); mcfg->memseg[0].phys_addr = (phys_addr_t)(uintptr_t)addr; diff --git a/lib/librte_mempool/rte_mempool.c b/lib/librte_mempool/rte_mempool.c index aa8e76a..87183df 100644 --- a/lib/librte_mempool/rte_mempool.c +++ b/lib/librte_mempool/rte_mempool.c @@ -36,6 +36,7 @@ #include <string.h> #include <stdint.h> #include <stdarg.h> +#include <unistd.h> #include <inttypes.h> #include <errno.h> #include <sys/queue.h> @@ -139,6 +140,8 @@ rte_mempool_create(const char *name, unsigned n, unsigned elt_size, uint32_t header_size, trailer_size; unsigned i; void *obj; + void *startaddr; + int page_size = getpagesize(); /* compilation-time checks */ RTE_BUILD_BUG_ON((sizeof(struct rte_mempool) & @@ -227,6 +230,20 @@ rte_mempool_create(const char *name, unsigned n, unsigned elt_size, trailer_size); trailer_size = new_size - header_size - elt_size; } + if (! rte_eal_has_hugepages()) { + /* + * compute trailer size so that pool elements fit exactly in + * a standard page + */ + int new_size = page_size - header_size - elt_size; + if (new_size < 0 || (unsigned int)new_size < trailer_size) { + printf("When hugepages are disabled, pool objects " + "can't exceed PAGE_SIZE: %d + %d + %d > %d\n", + header_size, elt_size, trailer_size, page_size); + return NULL; + } + trailer_size = new_size; + } /* this is the size of an object, including header and trailer */ total_elt_size = header_size + elt_size + trailer_size; @@ -235,8 +252,31 @@ rte_mempool_create(const char *name, unsigned n, unsigned elt_size, * cache-aligned */ private_data_size = (private_data_size + CACHE_LINE_MASK) & (~CACHE_LINE_MASK); + + if (! rte_eal_has_hugepages()) { + /* + * expand private data size to a whole page, so that the + * first pool element will start on a new standard page + */ + int head = sizeof(struct rte_mempool); + int new_size = (private_data_size + head) % page_size; + if (new_size) { + private_data_size += page_size - new_size; + } + } + mempool_size = total_elt_size * n + sizeof(struct rte_mempool) + private_data_size; + + if (! rte_eal_has_hugepages()) { + /* + * we want the memory pool to start on a page boundary, + * because pool elements crossing page boundaries would + * result in discontiguous physical addresses + */ + mempool_size += page_size; + } + rte_snprintf(mz_name, sizeof(mz_name), "MP_%s", name); mz = rte_memzone_reserve(mz_name, mempool_size, socket_id, mz_flags); @@ -248,8 +288,20 @@ rte_mempool_create(const char *name, unsigned n, unsigned elt_size, if (mz == NULL) goto exit; + if (rte_eal_has_hugepages()) { + startaddr = (void*)mz->addr; + } else { + /* align memory pool start address on a page boundary */ + unsigned long addr = (unsigned long)mz->addr; + if (addr & (page_size - 1)) { + addr += page_size; + addr &= ~(page_size - 1); + } + startaddr = (void*)addr; + } + /* init the mempool structure */ - mp = mz->addr; + mp = startaddr; memset(mp, 0, sizeof(*mp)); rte_snprintf(mp->name, sizeof(mp->name), "%s", name); mp->phys_addr = mz->phys_addr; diff --git a/lib/librte_mempool/rte_mempool.h b/lib/librte_mempool/rte_mempool.h index 67d7f09..33ce35c 100644 --- a/lib/librte_mempool/rte_mempool.h +++ b/lib/librte_mempool/rte_mempool.h @@ -950,16 +950,22 @@ rte_mempool_empty(const struct rte_mempool *mp) * @return * The physical address of the elt element. */ -static inline phys_addr_t rte_mempool_virt2phy(const struct rte_mempool *mp, - const void *elt) +static inline phys_addr_t +rte_mempool_virt2phy(const struct rte_mempool *mp, const void *elt) { - uintptr_t off; - - off = (const char *)elt - (const char *)mp; - return mp->phys_addr + off; + if (rte_eal_has_hugepages()) { + uintptr_t offset = (const char *)elt - (const char *)mp; + return mp->phys_addr + offset; + } else { + /* + * If huge pages are disabled, we cannot assume the + * memory region to be physically contiguous. + * Lookup for each element. + */ + return rte_mem_virt2phy(elt); + } } - /** * Check the consistency of mempool objects. * -- 1.7.2.5 ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-dev] [PATCH 2/2] mem: fix mempool for --no-huge 2013-07-26 14:39 ` [dpdk-dev] [PATCH 2/2] mem: fix mempool for --no-huge Damien Millescamps @ 2013-07-26 14:59 ` Adrien Mazarguil 2013-07-26 15:10 ` Thomas Monjalon 0 siblings, 1 reply; 7+ messages in thread From: Adrien Mazarguil @ 2013-07-26 14:59 UTC (permalink / raw) To: Damien Millescamps; +Cc: dev On Fri, Jul 26, 2013 at 04:39:13PM +0200, Damien Millescamps wrote: > In --no-huge mode, mempool provides objects with their associated > header/trailer fitting in a standard page (usually 4KB). > This means all non-UIO driver should work correctly in this mode, > since UIO drivers allocate ring sizes that cannot fit in a page. > > Extend rte_mempool_virt2phy to obtain the correct physical address when > elements of the pool are not on the same physically contiguous memory region. > This is a first step for enhancement PR #29696. > > Reason for this patch is to be able to run on a kernel < 2.6.37 without > the need to patch it, since all kernel below are either bugged or don't > have huge page support at all (< 2.6.28). > > Signed-off-by: Damien Millescamps <damien.millescamps@6wind.com> > --- > lib/librte_eal/linuxapp/eal/eal_memory.c | 2 +- > lib/librte_mempool/rte_mempool.c | 54 +++++++++++++++++++++++++++++- > lib/librte_mempool/rte_mempool.h | 20 +++++++---- > 3 files changed, 67 insertions(+), 9 deletions(-) Acked-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> -- Adrien Mazarguil 6WIND ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-dev] [PATCH 2/2] mem: fix mempool for --no-huge 2013-07-26 14:59 ` Adrien Mazarguil @ 2013-07-26 15:10 ` Thomas Monjalon 0 siblings, 0 replies; 7+ messages in thread From: Thomas Monjalon @ 2013-07-26 15:10 UTC (permalink / raw) To: Damien Millescamps; +Cc: dev 26/07/2013 16:59, Adrien Mazarguil : > On Fri, Jul 26, 2013 at 04:39:13PM +0200, Damien Millescamps wrote: > > In --no-huge mode, mempool provides objects with their associated > > header/trailer fitting in a standard page (usually 4KB). > > This means all non-UIO driver should work correctly in this mode, > > since UIO drivers allocate ring sizes that cannot fit in a page. > > > > Extend rte_mempool_virt2phy to obtain the correct physical address when > > elements of the pool are not on the same physically contiguous memory > > region. This is a first step for enhancement PR #29696. > > > > Reason for this patch is to be able to run on a kernel < 2.6.37 without > > the need to patch it, since all kernel below are either bugged or don't > > have huge page support at all (< 2.6.28). > > > > Signed-off-by: Damien Millescamps <damien.millescamps@6wind.com> > > --- > > > > lib/librte_eal/linuxapp/eal/eal_memory.c | 2 +- > > lib/librte_mempool/rte_mempool.c | 54 > > +++++++++++++++++++++++++++++- lib/librte_mempool/rte_mempool.h > > | 20 +++++++---- > > 3 files changed, 67 insertions(+), 9 deletions(-) > > Acked-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> applied -- Thomas ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2013-07-26 15:10 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2013-07-26 14:39 [dpdk-dev] [PATCH 0/2] fix mempool when --no-huge option is set Damien Millescamps 2013-07-26 14:39 ` [dpdk-dev] [PATCH 1/2] mem: get hugepages config Damien Millescamps 2013-07-26 14:59 ` Adrien Mazarguil 2013-07-26 15:10 ` Thomas Monjalon 2013-07-26 14:39 ` [dpdk-dev] [PATCH 2/2] mem: fix mempool for --no-huge Damien Millescamps 2013-07-26 14:59 ` Adrien Mazarguil 2013-07-26 15:10 ` Thomas Monjalon
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).