From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8279441D3D; Fri, 10 Mar 2023 09:10:38 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8C5B442D0B; Fri, 10 Mar 2023 09:10:09 +0100 (CET) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id BA5E940FAE for ; Fri, 10 Mar 2023 09:10:02 +0100 (CET) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 32A7ajpH009832 for ; Fri, 10 Mar 2023 00:10:02 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=9UIktwWX1oougZEmaZcRAVl5lnsm4ttXOT05fFMRSN8=; b=TxniV6i5m5oneGVMxWmsVOse93kDqSq34WowOTqA89kKq9cV+GsGR2FTP6Zwoeu0pD5c lrwIu6nhuILEPX8Y+qlVUZJAaw2srC5Q9IvgMI2HN0on0l1JGFAPoXYskXBj63/kJ4r6 +nHRqdSdoLxv39KT1Q5NWD9paUgRxqWUzZXSvrP5o3DJywySMtbxMwwpWIHrQw4j8vJs rnjITDEfs+YKWjO8r5hgt0bW+SCUki6ir7oXbVQlX2UdCeBmH8t9ntKR5hnKa551LZAW 4DnLr34vTeqdQ+oVOVA/9MEapEcnF6p1RLlv2A4iN04CdVcIYSpAl0fw4B+mM41I8Tn/ +w== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3p7n7dhy4q-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Fri, 10 Mar 2023 00:10:01 -0800 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Fri, 10 Mar 2023 00:09:41 -0800 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.42 via Frontend Transport; Fri, 10 Mar 2023 00:09:41 -0800 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 345A13F7093; Fri, 10 Mar 2023 00:09:41 -0800 (PST) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , , , Subject: [PATCH v5 07/12] app/mldev: enable support for burst inferences Date: Fri, 10 Mar 2023 00:09:29 -0800 Message-ID: <20230310080935.2460-8-syalavarthi@marvell.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230310080935.2460-1-syalavarthi@marvell.com> References: <20221129070746.20396-1-syalavarthi@marvell.com> <20230310080935.2460-1-syalavarthi@marvell.com> MIME-Version: 1.0 Content-Type: text/plain X-Proofpoint-GUID: TiWxCSNrDzExf8uXJ1bJg-HV1Svbgbf1 X-Proofpoint-ORIG-GUID: TiWxCSNrDzExf8uXJ1bJg-HV1Svbgbf1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-10_02,2023-03-09_01,2023-02-09_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Added 'burst_size' support for inference tests. Burst size controls the number of inference requests handled during the burst enqueue and dequeue operations of the test case. Signed-off-by: Srikanth Yalavarthi --- app/test-mldev/ml_options.c | 26 ++-- app/test-mldev/ml_options.h | 2 + app/test-mldev/test_inference_common.c | 159 ++++++++++++++++++++++++- app/test-mldev/test_inference_common.h | 4 + 4 files changed, 181 insertions(+), 10 deletions(-) diff --git a/app/test-mldev/ml_options.c b/app/test-mldev/ml_options.c index f9e3ce8e6f..7fbf9c5678 100644 --- a/app/test-mldev/ml_options.c +++ b/app/test-mldev/ml_options.c @@ -24,6 +24,7 @@ ml_options_default(struct ml_options *opt) opt->socket_id = SOCKET_ID_ANY; opt->nb_filelist = 0; opt->repetitions = 1; + opt->burst_size = 1; opt->debug = false; } @@ -145,6 +146,12 @@ ml_parse_repetitions(struct ml_options *opt, const char *arg) return parser_read_uint64(&opt->repetitions, arg); } +static int +ml_parse_burst_size(struct ml_options *opt, const char *arg) +{ + return parser_read_uint16(&opt->burst_size, arg); +} + static void ml_dump_test_options(const char *testname) { @@ -159,7 +166,8 @@ ml_dump_test_options(const char *testname) if ((strcmp(testname, "inference_ordered") == 0) || (strcmp(testname, "inference_interleave") == 0)) { printf("\t\t--filelist : comma separated list of model, input and output\n" - "\t\t--repetitions : number of inference repetitions\n"); + "\t\t--repetitions : number of inference repetitions\n" + "\t\t--burst_size : inference burst size\n"); printf("\n"); } } @@ -179,10 +187,11 @@ print_usage(char *program) ml_test_dump_names(ml_dump_test_options); } -static struct option lgopts[] = { - {ML_TEST, 1, 0, 0}, {ML_DEVICE_ID, 1, 0, 0}, {ML_SOCKET_ID, 1, 0, 0}, - {ML_MODELS, 1, 0, 0}, {ML_FILELIST, 1, 0, 0}, {ML_REPETITIONS, 1, 0, 0}, - {ML_DEBUG, 0, 0, 0}, {ML_HELP, 0, 0, 0}, {NULL, 0, 0, 0}}; +static struct option lgopts[] = {{ML_TEST, 1, 0, 0}, {ML_DEVICE_ID, 1, 0, 0}, + {ML_SOCKET_ID, 1, 0, 0}, {ML_MODELS, 1, 0, 0}, + {ML_FILELIST, 1, 0, 0}, {ML_REPETITIONS, 1, 0, 0}, + {ML_BURST_SIZE, 1, 0, 0}, {ML_DEBUG, 0, 0, 0}, + {ML_HELP, 0, 0, 0}, {NULL, 0, 0, 0}}; static int ml_opts_parse_long(int opt_idx, struct ml_options *opt) @@ -190,9 +199,10 @@ ml_opts_parse_long(int opt_idx, struct ml_options *opt) unsigned int i; struct long_opt_parser parsermap[] = { - {ML_TEST, ml_parse_test_name}, {ML_DEVICE_ID, ml_parse_dev_id}, - {ML_SOCKET_ID, ml_parse_socket_id}, {ML_MODELS, ml_parse_models}, - {ML_FILELIST, ml_parse_filelist}, {ML_REPETITIONS, ml_parse_repetitions}, + {ML_TEST, ml_parse_test_name}, {ML_DEVICE_ID, ml_parse_dev_id}, + {ML_SOCKET_ID, ml_parse_socket_id}, {ML_MODELS, ml_parse_models}, + {ML_FILELIST, ml_parse_filelist}, {ML_REPETITIONS, ml_parse_repetitions}, + {ML_BURST_SIZE, ml_parse_burst_size}, }; for (i = 0; i < RTE_DIM(parsermap); i++) { diff --git a/app/test-mldev/ml_options.h b/app/test-mldev/ml_options.h index 6a13f97a30..00342d8a0c 100644 --- a/app/test-mldev/ml_options.h +++ b/app/test-mldev/ml_options.h @@ -18,6 +18,7 @@ #define ML_MODELS ("models") #define ML_FILELIST ("filelist") #define ML_REPETITIONS ("repetitions") +#define ML_BURST_SIZE ("burst_size") #define ML_DEBUG ("debug") #define ML_HELP ("help") @@ -34,6 +35,7 @@ struct ml_options { struct ml_filelist filelist[ML_TEST_MAX_MODELS]; uint8_t nb_filelist; uint64_t repetitions; + uint16_t burst_size; bool debug; }; diff --git a/app/test-mldev/test_inference_common.c b/app/test-mldev/test_inference_common.c index ea06c0c1e4..530872def6 100644 --- a/app/test-mldev/test_inference_common.c +++ b/app/test-mldev/test_inference_common.c @@ -123,6 +123,131 @@ ml_dequeue_single(void *arg) return 0; } +/* Enqueue inference requests with burst size greater than 1 */ +static int +ml_enqueue_burst(void *arg) +{ + struct test_inference *t = ml_test_priv((struct ml_test *)arg); + struct ml_core_args *args; + uint16_t ops_count; + uint64_t model_enq; + uint16_t burst_enq; + uint32_t lcore_id; + uint16_t pending; + uint16_t idx; + uint16_t fid; + uint16_t i; + int ret; + + lcore_id = rte_lcore_id(); + args = &t->args[lcore_id]; + model_enq = 0; + + if (args->nb_reqs == 0) + return 0; + +next_rep: + fid = args->start_fid; + +next_model: + ops_count = RTE_MIN(t->cmn.opt->burst_size, args->nb_reqs - model_enq); + ret = rte_mempool_get_bulk(t->op_pool, (void **)args->enq_ops, ops_count); + if (ret != 0) + goto next_model; + +retry: + ret = rte_mempool_get_bulk(t->model[fid].io_pool, (void **)args->reqs, ops_count); + if (ret != 0) + goto retry; + + for (i = 0; i < ops_count; i++) { + args->enq_ops[i]->model_id = t->model[fid].id; + args->enq_ops[i]->nb_batches = t->model[fid].info.batch_size; + args->enq_ops[i]->mempool = t->op_pool; + + args->enq_ops[i]->input.addr = args->reqs[i]->input; + args->enq_ops[i]->input.length = t->model[fid].inp_qsize; + args->enq_ops[i]->input.next = NULL; + + args->enq_ops[i]->output.addr = args->reqs[i]->output; + args->enq_ops[i]->output.length = t->model[fid].out_qsize; + args->enq_ops[i]->output.next = NULL; + + args->enq_ops[i]->user_ptr = args->reqs[i]; + args->reqs[i]->niters++; + args->reqs[i]->fid = fid; + } + + idx = 0; + pending = ops_count; + +enqueue_reqs: + burst_enq = rte_ml_enqueue_burst(t->cmn.opt->dev_id, 0, &args->enq_ops[idx], pending); + pending = pending - burst_enq; + + if (pending > 0) { + idx = idx + burst_enq; + goto enqueue_reqs; + } + + fid++; + if (fid <= args->end_fid) + goto next_model; + + model_enq = model_enq + ops_count; + if (model_enq < args->nb_reqs) + goto next_rep; + + return 0; +} + +/* Dequeue inference requests with burst size greater than 1 */ +static int +ml_dequeue_burst(void *arg) +{ + struct test_inference *t = ml_test_priv((struct ml_test *)arg); + struct rte_ml_op_error error; + struct ml_core_args *args; + struct ml_request *req; + uint64_t total_deq = 0; + uint16_t burst_deq = 0; + uint8_t nb_filelist; + uint32_t lcore_id; + uint32_t i; + + lcore_id = rte_lcore_id(); + args = &t->args[lcore_id]; + nb_filelist = args->end_fid - args->start_fid + 1; + + if (args->nb_reqs == 0) + return 0; + +dequeue_burst: + burst_deq = + rte_ml_dequeue_burst(t->cmn.opt->dev_id, 0, args->deq_ops, t->cmn.opt->burst_size); + + if (likely(burst_deq > 0)) { + total_deq += burst_deq; + + for (i = 0; i < burst_deq; i++) { + if (unlikely(args->deq_ops[i]->status == RTE_ML_OP_STATUS_ERROR)) { + rte_ml_op_error_get(t->cmn.opt->dev_id, args->deq_ops[i], &error); + ml_err("error_code = 0x%" PRIx64 ", error_message = %s\n", + error.errcode, error.message); + } + req = (struct ml_request *)args->deq_ops[i]->user_ptr; + if (req != NULL) + rte_mempool_put(t->model[req->fid].io_pool, req); + } + rte_mempool_put_bulk(t->op_pool, (void *)args->deq_ops, burst_deq); + } + + if (total_deq < args->nb_reqs * nb_filelist) + goto dequeue_burst; + + return 0; +} + bool test_inference_cap_check(struct ml_options *opt) { @@ -172,6 +297,17 @@ test_inference_opt_check(struct ml_options *opt) return -EINVAL; } + if (opt->burst_size == 0) { + ml_err("Invalid option, burst_size = %u\n", opt->burst_size); + return -EINVAL; + } + + if (opt->burst_size > ML_TEST_MAX_POOL_SIZE) { + ml_err("Invalid option, burst_size = %u (> max supported = %d)\n", opt->burst_size, + ML_TEST_MAX_POOL_SIZE); + return -EINVAL; + } + /* check number of available lcores. */ if (rte_lcore_count() < 3) { ml_err("Insufficient lcores = %u\n", rte_lcore_count()); @@ -192,6 +328,7 @@ test_inference_opt_dump(struct ml_options *opt) /* dump test opts */ ml_dump("repetitions", "%" PRIu64, opt->repetitions); + ml_dump("burst_size", "%u", opt->burst_size); ml_dump_begin("filelist"); for (i = 0; i < opt->nb_filelist; i++) { @@ -207,6 +344,7 @@ test_inference_setup(struct ml_test *test, struct ml_options *opt) { struct test_inference *t; void *test_inference; + uint32_t lcore_id; int ret = 0; uint32_t i; @@ -231,13 +369,30 @@ test_inference_setup(struct ml_test *test, struct ml_options *opt) goto error; } - t->enqueue = ml_enqueue_single; - t->dequeue = ml_dequeue_single; + if (opt->burst_size == 1) { + t->enqueue = ml_enqueue_single; + t->dequeue = ml_dequeue_single; + } else { + t->enqueue = ml_enqueue_burst; + t->dequeue = ml_dequeue_burst; + } /* set model initial state */ for (i = 0; i < opt->nb_filelist; i++) t->model[i].state = MODEL_INITIAL; + for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { + t->args[lcore_id].enq_ops = rte_zmalloc_socket( + "ml_test_enq_ops", opt->burst_size * sizeof(struct rte_ml_op *), + RTE_CACHE_LINE_SIZE, opt->socket_id); + t->args[lcore_id].deq_ops = rte_zmalloc_socket( + "ml_test_deq_ops", opt->burst_size * sizeof(struct rte_ml_op *), + RTE_CACHE_LINE_SIZE, opt->socket_id); + t->args[lcore_id].reqs = rte_zmalloc_socket( + "ml_test_requests", opt->burst_size * sizeof(struct ml_request *), + RTE_CACHE_LINE_SIZE, opt->socket_id); + } + return 0; error: diff --git a/app/test-mldev/test_inference_common.h b/app/test-mldev/test_inference_common.h index aee59d11a6..d0307a6432 100644 --- a/app/test-mldev/test_inference_common.h +++ b/app/test-mldev/test_inference_common.h @@ -22,6 +22,10 @@ struct ml_core_args { uint64_t nb_reqs; uint16_t start_fid; uint16_t end_fid; + + struct rte_ml_op **enq_ops; + struct rte_ml_op **deq_ops; + struct ml_request **reqs; }; struct test_inference { -- 2.17.1