From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B0F2FA00C2; Thu, 8 Dec 2022 20:30:28 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 21A7142D3F; Thu, 8 Dec 2022 20:29:55 +0100 (CET) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 4361942D20 for ; Thu, 8 Dec 2022 20:29:49 +0100 (CET) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B8J8Kj2001346 for ; Thu, 8 Dec 2022 11:29:48 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=UlpO2Cgwjwl9g7bw5cmUpeKLYufyl4K8d8NkbsnahCk=; b=MWe9mtqjuK+IgGaaxIXsS2f2dZzfJjvDDh2L4Q6Xw13AFHZfIRrUnFf8Q895+JVyaiPo AAql8c7nQvCqCjE47fiU5oEu2IhvenukbzLU7agKEJAuqetzL3MjlukXOzNnfoClDEPz /ccUIUwdPDwUxQjHn5ZxeG4Uw9g62a9CVPWSVYxcYQzyNgw0OtVkX+zKaCZHoUYkF99/ 3WSsNL/sfnhV1bc31kmkM3qGsp6gNh5P08EOaLE3FxgZYIoOjIiYOM1W/Ak2M21E9YXU H/bVkEvp4RSo9Fnx7DyS/4MdD9OL05ksaZiKXBa0v5UwgyeUHpNwqnaN4LF/BK3CjvmQ iQ== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3mb22svjux-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Thu, 08 Dec 2022 11:29:48 -0800 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Thu, 8 Dec 2022 11:29:45 -0800 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Thu, 8 Dec 2022 11:29:45 -0800 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 9CD413F7066; Thu, 8 Dec 2022 11:29:45 -0800 (PST) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v3 07/12] app/mldev: enable support for burst inferences Date: Thu, 8 Dec 2022 11:29:13 -0800 Message-ID: <20221208192918.25022-7-syalavarthi@marvell.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221208192918.25022-1-syalavarthi@marvell.com> References: <20221129082109.6809-1-syalavarthi@marvell.com> <20221208192918.25022-1-syalavarthi@marvell.com> MIME-Version: 1.0 Content-Type: text/plain X-Proofpoint-ORIG-GUID: lODJWkRBGXDe1tOTUD7xYNiRDm1O2Xjv X-Proofpoint-GUID: lODJWkRBGXDe1tOTUD7xYNiRDm1O2Xjv X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-08_11,2022-12-08_01,2022-06-22_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Added 'burst_size' support for inference tests. Burst size controls the number of inference requests handled during the burst enqueue and dequeue operations of the test case. Signed-off-by: Srikanth Yalavarthi --- app/test-mldev/ml_options.c | 26 ++-- app/test-mldev/ml_options.h | 2 + app/test-mldev/test_inference_common.c | 159 ++++++++++++++++++++++++- app/test-mldev/test_inference_common.h | 4 + 4 files changed, 181 insertions(+), 10 deletions(-) diff --git a/app/test-mldev/ml_options.c b/app/test-mldev/ml_options.c index 01ea050ee7..1990939200 100644 --- a/app/test-mldev/ml_options.c +++ b/app/test-mldev/ml_options.c @@ -30,6 +30,7 @@ ml_options_default(struct ml_options *opt) opt->socket_id = SOCKET_ID_ANY; opt->nb_filelist = 0; opt->repetitions = 1; + opt->burst_size = 1; opt->debug = false; } @@ -151,6 +152,12 @@ ml_parse_repetitions(struct ml_options *opt, const char *arg) return parser_read_uint64(&opt->repetitions, arg); } +static int +ml_parse_burst_size(struct ml_options *opt, const char *arg) +{ + return parser_read_uint16(&opt->burst_size, arg); +} + static void ml_dump_test_options(const char *testname) { @@ -165,7 +172,8 @@ ml_dump_test_options(const char *testname) if ((strcmp(testname, "inference_ordered") == 0) || (strcmp(testname, "inference_interleave") == 0)) { printf("\t\t--filelist : comma separated list of model, input and output\n" - "\t\t--repetitions : number of inference repetitions\n"); + "\t\t--repetitions : number of inference repetitions\n" + "\t\t--burst_size : inference burst size\n"); printf("\n"); } } @@ -185,10 +193,11 @@ print_usage(char *program) ml_test_dump_names(ml_dump_test_options); } -static struct option lgopts[] = { - {ML_TEST, 1, 0, 0}, {ML_DEVICE_ID, 1, 0, 0}, {ML_SOCKET_ID, 1, 0, 0}, - {ML_MODELS, 1, 0, 0}, {ML_FILELIST, 1, 0, 0}, {ML_REPETITIONS, 1, 0, 0}, - {ML_DEBUG, 0, 0, 0}, {ML_HELP, 0, 0, 0}, {NULL, 0, 0, 0}}; +static struct option lgopts[] = {{ML_TEST, 1, 0, 0}, {ML_DEVICE_ID, 1, 0, 0}, + {ML_SOCKET_ID, 1, 0, 0}, {ML_MODELS, 1, 0, 0}, + {ML_FILELIST, 1, 0, 0}, {ML_REPETITIONS, 1, 0, 0}, + {ML_BURST_SIZE, 1, 0, 0}, {ML_DEBUG, 0, 0, 0}, + {ML_HELP, 0, 0, 0}, {NULL, 0, 0, 0}}; static int ml_opts_parse_long(int opt_idx, struct ml_options *opt) @@ -196,9 +205,10 @@ ml_opts_parse_long(int opt_idx, struct ml_options *opt) unsigned int i; struct long_opt_parser parsermap[] = { - {ML_TEST, ml_parse_test_name}, {ML_DEVICE_ID, ml_parse_dev_id}, - {ML_SOCKET_ID, ml_parse_socket_id}, {ML_MODELS, ml_parse_models}, - {ML_FILELIST, ml_parse_filelist}, {ML_REPETITIONS, ml_parse_repetitions}, + {ML_TEST, ml_parse_test_name}, {ML_DEVICE_ID, ml_parse_dev_id}, + {ML_SOCKET_ID, ml_parse_socket_id}, {ML_MODELS, ml_parse_models}, + {ML_FILELIST, ml_parse_filelist}, {ML_REPETITIONS, ml_parse_repetitions}, + {ML_BURST_SIZE, ml_parse_burst_size}, }; for (i = 0; i < RTE_DIM(parsermap); i++) { diff --git a/app/test-mldev/ml_options.h b/app/test-mldev/ml_options.h index ad8aee5964..305b39629a 100644 --- a/app/test-mldev/ml_options.h +++ b/app/test-mldev/ml_options.h @@ -19,6 +19,7 @@ #define ML_MODELS ("models") #define ML_FILELIST ("filelist") #define ML_REPETITIONS ("repetitions") +#define ML_BURST_SIZE ("burst_size") #define ML_DEBUG ("debug") #define ML_HELP ("help") @@ -35,6 +36,7 @@ struct ml_options { struct ml_filelist filelist[ML_TEST_MAX_MODELS]; uint8_t nb_filelist; uint64_t repetitions; + uint16_t burst_size; bool debug; }; diff --git a/app/test-mldev/test_inference_common.c b/app/test-mldev/test_inference_common.c index f0b15861a0..252d77616c 100644 --- a/app/test-mldev/test_inference_common.c +++ b/app/test-mldev/test_inference_common.c @@ -129,6 +129,131 @@ ml_dequeue_single(void *arg) return 0; } +/* Enqueue inference requests with burst size greater than 1 */ +static int +ml_enqueue_burst(void *arg) +{ + struct test_inference *t = ml_test_priv((struct ml_test *)arg); + struct ml_core_args *args; + uint16_t ops_count; + uint64_t model_enq; + uint16_t burst_enq; + uint32_t lcore_id; + uint16_t pending; + uint16_t idx; + int16_t fid; + uint16_t i; + int ret; + + lcore_id = rte_lcore_id(); + args = &t->args[lcore_id]; + model_enq = 0; + + if (args->nb_reqs == 0) + return 0; + +next_rep: + fid = args->start_fid; + +next_model: + ops_count = RTE_MIN(t->cmn.opt->burst_size, args->nb_reqs - model_enq); + ret = rte_mempool_get_bulk(t->op_pool, (void **)args->enq_ops, ops_count); + if (ret != 0) + goto next_model; + +retry: + ret = rte_mempool_get_bulk(t->model[fid].io_pool, (void **)args->reqs, ops_count); + if (ret != 0) + goto retry; + + for (i = 0; i < ops_count; i++) { + args->enq_ops[i]->model_id = t->model[fid].id; + args->enq_ops[i]->nb_batches = t->model[fid].info.batch_size; + args->enq_ops[i]->mempool = t->op_pool; + + args->enq_ops[i]->input.addr = args->reqs[i]->input; + args->enq_ops[i]->input.length = t->model[fid].inp_qsize; + args->enq_ops[i]->input.next = NULL; + + args->enq_ops[i]->output.addr = args->reqs[i]->output; + args->enq_ops[i]->output.length = t->model[fid].out_qsize; + args->enq_ops[i]->output.next = NULL; + + args->enq_ops[i]->user_ptr = args->reqs[i]; + args->reqs[i]->niters++; + args->reqs[i]->fid = fid; + } + + idx = 0; + pending = ops_count; + +enqueue_reqs: + burst_enq = rte_ml_enqueue_burst(t->cmn.opt->dev_id, 0, &args->enq_ops[idx], pending); + pending = pending - burst_enq; + + if (pending > 0) { + idx = idx + burst_enq; + goto enqueue_reqs; + } + + fid++; + if (fid <= args->end_fid) + goto next_model; + + model_enq = model_enq + ops_count; + if (model_enq < args->nb_reqs) + goto next_rep; + + return 0; +} + +/* Dequeue inference requests with burst size greater than 1 */ +static int +ml_dequeue_burst(void *arg) +{ + struct test_inference *t = ml_test_priv((struct ml_test *)arg); + struct rte_ml_op_error error; + struct ml_core_args *args; + struct ml_request *req; + uint64_t total_deq = 0; + uint16_t burst_deq = 0; + uint8_t nb_filelist; + uint32_t lcore_id; + uint32_t i; + + lcore_id = rte_lcore_id(); + args = &t->args[lcore_id]; + nb_filelist = args->end_fid - args->start_fid + 1; + + if (args->nb_reqs == 0) + return 0; + +dequeue_burst: + burst_deq = + rte_ml_dequeue_burst(t->cmn.opt->dev_id, 0, args->deq_ops, t->cmn.opt->burst_size); + + if (likely(burst_deq > 0)) { + total_deq += burst_deq; + + for (i = 0; i < burst_deq; i++) { + if (unlikely(args->deq_ops[i]->status == RTE_ML_OP_STATUS_ERROR)) { + rte_ml_op_error_get(t->cmn.opt->dev_id, args->deq_ops[i], &error); + ml_err("error_code = 0x%" PRIx64 ", error_message = %s\n", + error.errcode, error.message); + } + req = (struct ml_request *)args->deq_ops[i]->user_ptr; + if (req != NULL) + rte_mempool_put(t->model[req->fid].io_pool, req); + } + rte_mempool_put_bulk(t->op_pool, (void *)args->deq_ops, burst_deq); + } + + if (total_deq < args->nb_reqs * nb_filelist) + goto dequeue_burst; + + return 0; +} + bool test_inference_cap_check(struct ml_options *opt) { @@ -178,6 +303,17 @@ test_inference_opt_check(struct ml_options *opt) return -EINVAL; } + if (opt->burst_size == 0) { + ml_err("Invalid option, burst_size = %u\n", opt->burst_size); + return -EINVAL; + } + + if (opt->burst_size > ML_TEST_MAX_POOL_SIZE) { + ml_err("Invalid option, burst_size = %u (> max supported = %d)\n", opt->burst_size, + ML_TEST_MAX_POOL_SIZE); + return -EINVAL; + } + /* check number of available lcores. */ if (rte_lcore_count() < 3) { ml_err("Insufficient lcores = %u\n", rte_lcore_count()); @@ -198,6 +334,7 @@ test_inference_opt_dump(struct ml_options *opt) /* dump test opts */ ml_dump("repetitions", "%" PRIu64, opt->repetitions); + ml_dump("burst_size", "%u", opt->burst_size); ml_dump_begin("filelist"); for (i = 0; i < opt->nb_filelist; i++) { @@ -213,6 +350,7 @@ test_inference_setup(struct ml_test *test, struct ml_options *opt) { struct test_inference *t; void *test_inference; + uint32_t lcore_id; int ret = 0; uint32_t i; @@ -237,13 +375,30 @@ test_inference_setup(struct ml_test *test, struct ml_options *opt) goto error; } - t->enqueue = ml_enqueue_single; - t->dequeue = ml_dequeue_single; + if (opt->burst_size == 1) { + t->enqueue = ml_enqueue_single; + t->dequeue = ml_dequeue_single; + } else { + t->enqueue = ml_enqueue_burst; + t->dequeue = ml_dequeue_burst; + } /* set model initial state */ for (i = 0; i < opt->nb_filelist; i++) t->model[i].state = MODEL_INITIAL; + for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { + t->args[lcore_id].enq_ops = rte_zmalloc_socket( + "ml_test_enq_ops", opt->burst_size * sizeof(struct rte_ml_op *), + RTE_CACHE_LINE_SIZE, opt->socket_id); + t->args[lcore_id].deq_ops = rte_zmalloc_socket( + "ml_test_deq_ops", opt->burst_size * sizeof(struct rte_ml_op *), + RTE_CACHE_LINE_SIZE, opt->socket_id); + t->args[lcore_id].reqs = rte_zmalloc_socket( + "ml_test_requests", opt->burst_size * sizeof(struct ml_request *), + RTE_CACHE_LINE_SIZE, opt->socket_id); + } + return 0; error: diff --git a/app/test-mldev/test_inference_common.h b/app/test-mldev/test_inference_common.h index b058abada4..75d588308b 100644 --- a/app/test-mldev/test_inference_common.h +++ b/app/test-mldev/test_inference_common.h @@ -27,6 +27,10 @@ struct ml_core_args { uint64_t nb_reqs; int16_t start_fid; int16_t end_fid; + + struct rte_ml_op **enq_ops; + struct rte_ml_op **deq_ops; + struct ml_request **reqs; }; struct test_inference { -- 2.17.1