From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 8DBF14C8F for ; Mon, 1 Apr 2019 15:28:06 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 01 Apr 2019 06:28:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,296,1549958400"; d="scan'208";a="160284198" Received: from aburakov-mobl1.ger.corp.intel.com (HELO [10.237.220.103]) ([10.237.220.103]) by fmsmga001.fm.intel.com with ESMTP; 01 Apr 2019 06:28:02 -0700 To: Shahaf Shuler , wenzhuo.lu@intel.com, jingjing.wu@intel.com, bernard.iremonger@intel.com Cc: dev@dpdk.org, rasland@mellanox.com, thomas@monjalon.net, ferruh.yigit@intel.com References: From: "Burakov, Anatoly" Message-ID: Date: Mon, 1 Apr 2019 14:28:02 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH 3/3] app/testpmd: map anonymous memory for eth devices X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Apr 2019 13:28:07 -0000 On 01-Apr-19 11:34 AM, Shahaf Shuler wrote: > Mempools can be populated with anonymous memory when using command line > parameter --mp-alloc=anon. > > Considering the mempools are going to be used by the net devices, > it is better to DMA map this memory. > > This patch add such mapping now that we have the APIs in place[1]. > > [1] commit c33a675b6276 ("bus: introduce device level DMA memory mapping") > > Signed-off-by: Shahaf Shuler > --- > app/test-pmd/testpmd.c | 62 ++++++++++++++++++++++++++++++++++++++++++++- > 1 file changed, 61 insertions(+), 1 deletion(-) > > diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c > index 8c4ebc774c..6729469c6b 100644 > --- a/app/test-pmd/testpmd.c > +++ b/app/test-pmd/testpmd.c > @@ -834,6 +834,61 @@ setup_extmem(uint32_t nb_mbufs, uint32_t mbuf_sz, bool huge) > > return 0; > } > +static void > +dma_unmap_cb(struct rte_mempool *mp __rte_unused, void *opaque __rte_unused, > + struct rte_mempool_memhdr *memhdr, unsigned mem_idx __rte_unused) > +{ > + uint16_t pid = 0; > + int ret; > + > + ret = rte_extmem_unregister(memhdr->addr, memhdr->len); > + if (ret) { > + TESTPMD_LOG(DEBUG, > + "unable to un-register addr 0x%p\n", memhdr->addr); > + return; > + } > + RTE_ETH_FOREACH_DEV(pid) { > + struct rte_eth_dev *dev = > + &rte_eth_devices[pid]; > + > + ret = rte_dev_dma_unmap(dev->device, memhdr->addr, 0, > + memhdr->len); > + if (ret) { > + TESTPMD_LOG(DEBUG, > + "unable to DMA unmap addr 0x%p\n", > + memhdr->addr); > + } > + } I would expect unmap *then* unregister. > +} > + > +static void > +dma_map_cb(struct rte_mempool *mp __rte_unused, void *opaque __rte_unused, > + struct rte_mempool_memhdr *memhdr, unsigned mem_idx __rte_unused) > +{ > + uint16_t pid = 0; > + size_t page_size = sysconf(_SC_PAGESIZE); > + int ret; > + > + ret = rte_extmem_register(memhdr->addr, memhdr->len, NULL, 0, > + page_size); > + if (ret) { > + TESTPMD_LOG(DEBUG, > + "unable to register addr 0x%p\n", memhdr->addr); > + return; > + } > + RTE_ETH_FOREACH_DEV(pid) { > + struct rte_eth_dev *dev = > + &rte_eth_devices[pid]; > + > + ret = rte_dev_dma_map(dev->device, memhdr->addr, 0, > + memhdr->len); > + if (ret) { > + TESTPMD_LOG(DEBUG, > + "unable to DMA map addr 0x%p\n", > + memhdr->addr); Maybe mention the device/driver name as well? Otherwise it's hard to see which particular DMA map/unmap has failed, should one choose to debug-by-logs. > + } > + } > +} > > /* > * Configuration initialisation done once at init time. > @@ -879,6 +934,7 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf, > } > rte_pktmbuf_pool_init(rte_mp, NULL); > rte_mempool_obj_iter(rte_mp, rte_pktmbuf_init, NULL); > + rte_mempool_mem_iter(rte_mp, dma_map_cb, NULL); > break; > } > case MP_ALLOC_XMEM: > @@ -2460,8 +2516,12 @@ pmd_test_exit(void) > } > } > for (i = 0 ; i < RTE_MAX_NUMA_NODES ; i++) > - if (mempools[i]) > + if (mempools[i]) { > + if (mp_alloc_type == MP_ALLOC_ANON) > + rte_mempool_mem_iter(mempools[i], dma_unmap_cb, > + NULL); > rte_mempool_free(mempools[i]); > + } > > printf("\nBye...\n"); > } > -- Thanks, Anatoly From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id 4F509A0679 for ; Mon, 1 Apr 2019 15:28:08 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 2607A4C99; Mon, 1 Apr 2019 15:28:08 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 8DBF14C8F for ; Mon, 1 Apr 2019 15:28:06 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 01 Apr 2019 06:28:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,296,1549958400"; d="scan'208";a="160284198" Received: from aburakov-mobl1.ger.corp.intel.com (HELO [10.237.220.103]) ([10.237.220.103]) by fmsmga001.fm.intel.com with ESMTP; 01 Apr 2019 06:28:02 -0700 To: Shahaf Shuler , wenzhuo.lu@intel.com, jingjing.wu@intel.com, bernard.iremonger@intel.com Cc: dev@dpdk.org, rasland@mellanox.com, thomas@monjalon.net, ferruh.yigit@intel.com References: From: "Burakov, Anatoly" Message-ID: Date: Mon, 1 Apr 2019 14:28:02 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="UTF-8"; format="flowed" Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH 3/3] app/testpmd: map anonymous memory for eth devices X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190401132802.gfuhJmghXS8lAAj99-Bqc7Pb-e8nquCuShHwJ3u7u5Y@z> On 01-Apr-19 11:34 AM, Shahaf Shuler wrote: > Mempools can be populated with anonymous memory when using command line > parameter --mp-alloc=anon. > > Considering the mempools are going to be used by the net devices, > it is better to DMA map this memory. > > This patch add such mapping now that we have the APIs in place[1]. > > [1] commit c33a675b6276 ("bus: introduce device level DMA memory mapping") > > Signed-off-by: Shahaf Shuler > --- > app/test-pmd/testpmd.c | 62 ++++++++++++++++++++++++++++++++++++++++++++- > 1 file changed, 61 insertions(+), 1 deletion(-) > > diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c > index 8c4ebc774c..6729469c6b 100644 > --- a/app/test-pmd/testpmd.c > +++ b/app/test-pmd/testpmd.c > @@ -834,6 +834,61 @@ setup_extmem(uint32_t nb_mbufs, uint32_t mbuf_sz, bool huge) > > return 0; > } > +static void > +dma_unmap_cb(struct rte_mempool *mp __rte_unused, void *opaque __rte_unused, > + struct rte_mempool_memhdr *memhdr, unsigned mem_idx __rte_unused) > +{ > + uint16_t pid = 0; > + int ret; > + > + ret = rte_extmem_unregister(memhdr->addr, memhdr->len); > + if (ret) { > + TESTPMD_LOG(DEBUG, > + "unable to un-register addr 0x%p\n", memhdr->addr); > + return; > + } > + RTE_ETH_FOREACH_DEV(pid) { > + struct rte_eth_dev *dev = > + &rte_eth_devices[pid]; > + > + ret = rte_dev_dma_unmap(dev->device, memhdr->addr, 0, > + memhdr->len); > + if (ret) { > + TESTPMD_LOG(DEBUG, > + "unable to DMA unmap addr 0x%p\n", > + memhdr->addr); > + } > + } I would expect unmap *then* unregister. > +} > + > +static void > +dma_map_cb(struct rte_mempool *mp __rte_unused, void *opaque __rte_unused, > + struct rte_mempool_memhdr *memhdr, unsigned mem_idx __rte_unused) > +{ > + uint16_t pid = 0; > + size_t page_size = sysconf(_SC_PAGESIZE); > + int ret; > + > + ret = rte_extmem_register(memhdr->addr, memhdr->len, NULL, 0, > + page_size); > + if (ret) { > + TESTPMD_LOG(DEBUG, > + "unable to register addr 0x%p\n", memhdr->addr); > + return; > + } > + RTE_ETH_FOREACH_DEV(pid) { > + struct rte_eth_dev *dev = > + &rte_eth_devices[pid]; > + > + ret = rte_dev_dma_map(dev->device, memhdr->addr, 0, > + memhdr->len); > + if (ret) { > + TESTPMD_LOG(DEBUG, > + "unable to DMA map addr 0x%p\n", > + memhdr->addr); Maybe mention the device/driver name as well? Otherwise it's hard to see which particular DMA map/unmap has failed, should one choose to debug-by-logs. > + } > + } > +} > > /* > * Configuration initialisation done once at init time. > @@ -879,6 +934,7 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf, > } > rte_pktmbuf_pool_init(rte_mp, NULL); > rte_mempool_obj_iter(rte_mp, rte_pktmbuf_init, NULL); > + rte_mempool_mem_iter(rte_mp, dma_map_cb, NULL); > break; > } > case MP_ALLOC_XMEM: > @@ -2460,8 +2516,12 @@ pmd_test_exit(void) > } > } > for (i = 0 ; i < RTE_MAX_NUMA_NODES ; i++) > - if (mempools[i]) > + if (mempools[i]) { > + if (mp_alloc_type == MP_ALLOC_ANON) > + rte_mempool_mem_iter(mempools[i], dma_unmap_cb, > + NULL); > rte_mempool_free(mempools[i]); > + } > > printf("\nBye...\n"); > } > -- Thanks, Anatoly