From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8D48CA04C0; Tue, 29 Sep 2020 11:38:56 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id AD42D1D936; Tue, 29 Sep 2020 11:38:35 +0200 (CEST) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 0711A1D935 for ; Tue, 29 Sep 2020 11:38:32 +0200 (CEST) IronPort-SDR: cWrjIzIsKs2miT9feVwfuWvuJYJwmCF1cW88Cd/6Xh2ESCMC+2Am1ZKPKBYOM1kxIfgzyvWNzC UWOU17Cx2a0Q== X-IronPort-AV: E=McAfee;i="6000,8403,9758"; a="159485204" X-IronPort-AV: E=Sophos;i="5.77,318,1596524400"; d="scan'208";a="159485204" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Sep 2020 02:38:31 -0700 IronPort-SDR: i5RSKgljARUOYiBiG9R73n+CVMPpa7wMkgIQBzyk9KmpM0t5VAV8gNXTZkMnS3NYZYurCbOJnG qisQ7e9WUNZg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,318,1596524400"; d="scan'208";a="457216011" Received: from npg-dpdk-patrickfu-casc2.sh.intel.com ([10.67.119.92]) by orsmga004.jf.intel.com with ESMTP; 29 Sep 2020 02:38:29 -0700 From: Patrick Fu To: dev@dpdk.org, maxime.coquelin@redhat.com, chenbo.xia@intel.com Cc: zhihong.wang@intel.com, cheng1.jiang@intel.com, patrick.fu@intel.com Date: Tue, 29 Sep 2020 17:29:53 +0800 Message-Id: <20200929092955.2848419-3-patrick.fu@intel.com> X-Mailer: git-send-email 2.18.4 In-Reply-To: <20200929092955.2848419-1-patrick.fu@intel.com> References: <20200911015316.1903181-1-patrick.fu@intel.com> <20200929092955.2848419-1-patrick.fu@intel.com> Subject: [dpdk-dev] [PATCH v3 2/4] vhost: dynamically allocate async memory X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Allocate async internal memory buffer by rte_malloc(), replacing array declaration inside vq structure. Dynamic allocation can help to save memory footprint when async path is not registered. Signed-off-by: Patrick Fu --- lib/librte_vhost/vhost.c | 69 ++++++++++++++++++++++++++-------------- lib/librte_vhost/vhost.h | 4 +-- 2 files changed, 47 insertions(+), 26 deletions(-) diff --git a/lib/librte_vhost/vhost.c b/lib/librte_vhost/vhost.c index eca507836..05b578c2f 100644 --- a/lib/librte_vhost/vhost.c +++ b/lib/librte_vhost/vhost.c @@ -324,6 +324,24 @@ cleanup_device(struct virtio_net *dev, int destroy) } } +static void +vhost_free_async_mem(struct vhost_virtqueue *vq) +{ + if (vq->async_pkts_pending) + rte_free(vq->async_pkts_pending); + if (vq->async_pkts_info) + rte_free(vq->async_pkts_info); + if (vq->it_pool) + rte_free(vq->it_pool); + if (vq->vec_pool) + rte_free(vq->vec_pool); + + vq->async_pkts_pending = NULL; + vq->async_pkts_info = NULL; + vq->it_pool = NULL; + vq->vec_pool = NULL; +} + void free_vq(struct virtio_net *dev, struct vhost_virtqueue *vq) { @@ -331,10 +349,7 @@ free_vq(struct virtio_net *dev, struct vhost_virtqueue *vq) rte_free(vq->shadow_used_packed); else { rte_free(vq->shadow_used_split); - if (vq->async_pkts_pending) - rte_free(vq->async_pkts_pending); - if (vq->async_pkts_info) - rte_free(vq->async_pkts_info); + vhost_free_async_mem(vq); } rte_free(vq->batch_copy_elems); rte_mempool_free(vq->iotlb_pool); @@ -1538,6 +1553,7 @@ int rte_vhost_async_channel_register(int vid, uint16_t queue_id, struct vhost_virtqueue *vq; struct virtio_net *dev = get_device(vid); struct rte_vhost_async_features f; + int node; if (dev == NULL || ops == NULL) return -1; @@ -1570,19 +1586,32 @@ int rte_vhost_async_channel_register(int vid, uint16_t queue_id, goto reg_out; } - vq->async_pkts_pending = rte_malloc(NULL, +#ifdef RTE_LIBRTE_VHOST_NUMA + if (get_mempolicy(&node, NULL, 0, vq, MPOL_F_NODE | MPOL_F_ADDR)) { + VHOST_LOG_CONFIG(ERR, + "unable to get numa information in async register. " + "allocating async buffer memory on the caller thread node\n"); + node = SOCKET_ID_ANY; + } +#else + node = SOCKET_ID_ANY; +#endif + + vq->async_pkts_pending = rte_malloc_socket(NULL, vq->size * sizeof(uintptr_t), - RTE_CACHE_LINE_SIZE); - vq->async_pkts_info = rte_malloc(NULL, + RTE_CACHE_LINE_SIZE, node); + vq->async_pkts_info = rte_malloc_socket(NULL, vq->size * sizeof(struct async_inflight_info), - RTE_CACHE_LINE_SIZE); - if (!vq->async_pkts_pending || !vq->async_pkts_info) { - if (vq->async_pkts_pending) - rte_free(vq->async_pkts_pending); - - if (vq->async_pkts_info) - rte_free(vq->async_pkts_info); - + RTE_CACHE_LINE_SIZE, node); + vq->it_pool = rte_malloc_socket(NULL, + VHOST_MAX_ASYNC_IT * sizeof(struct rte_vhost_iov_iter), + RTE_CACHE_LINE_SIZE, node); + vq->vec_pool = rte_malloc_socket(NULL, + VHOST_MAX_ASYNC_VEC * sizeof(struct iovec), + RTE_CACHE_LINE_SIZE, node); + if (!vq->async_pkts_pending || !vq->async_pkts_info || + !vq->it_pool || !vq->vec_pool) { + vhost_free_async_mem(vq); VHOST_LOG_CONFIG(ERR, "async register failed: cannot allocate memory for vq data " "(vid %d, qid: %d)\n", vid, queue_id); @@ -1630,15 +1659,7 @@ int rte_vhost_async_channel_unregister(int vid, uint16_t queue_id) goto out; } - if (vq->async_pkts_pending) { - rte_free(vq->async_pkts_pending); - vq->async_pkts_pending = NULL; - } - - if (vq->async_pkts_info) { - rte_free(vq->async_pkts_info); - vq->async_pkts_info = NULL; - } + vhost_free_async_mem(vq); vq->async_ops.transfer_data = NULL; vq->async_ops.check_completed_copies = NULL; diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h index 155a832c1..f0ee00c73 100644 --- a/lib/librte_vhost/vhost.h +++ b/lib/librte_vhost/vhost.h @@ -218,8 +218,8 @@ struct vhost_virtqueue { /* operation callbacks for async dma */ struct rte_vhost_async_channel_ops async_ops; - struct rte_vhost_iov_iter it_pool[VHOST_MAX_ASYNC_IT]; - struct iovec vec_pool[VHOST_MAX_ASYNC_VEC]; + struct rte_vhost_iov_iter *it_pool; + struct iovec *vec_pool; /* async data transfer status */ uintptr_t **async_pkts_pending; -- 2.18.4