From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D142A41EB6; Thu, 16 Mar 2023 22:15:16 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6075742F98; Thu, 16 Mar 2023 22:14:49 +0100 (CET) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 46D0C40DDC for ; Thu, 16 Mar 2023 22:14:44 +0100 (CET) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 32GJMmii016125 for ; Thu, 16 Mar 2023 14:14:43 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=GkutLY+aaRCGuXhkYFBkrDQYWRFJTy712C3XdxImHBU=; b=LY+qb2K7YGqbHIzrD1J+0Row46+oDfeL43Px7U4g3EXS6CRS9DXRT/2uiWnzex0GCB5m rlrUsILtgZjA9QwptmoEEXVwSkGtCt+ZCPf+8oiTsW0CIfmDwL6soD9D6IGgEYY3fydS e7JDZgwj9jHrLEBYh+jkqzbQlSDcxjrvem41jpL4iqkPsdyYX59JuTTM6vSJCozjyHHG kDOghQQYm/vsQhr10COm0iRQL/HF1HOnRvymCOsDlLaS+Se/LZ+iZoq1Er8kEsLozQOt gK7P+3RUbGloxuzo9t0yqCghzx+biqfWq/O4AkhF6E0mIc3tB5I4dOvLx0SnJ6UtLKwE Nw== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3pbs2u3mk5-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Thu, 16 Mar 2023 14:14:43 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Thu, 16 Mar 2023 14:14:40 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.42 via Frontend Transport; Thu, 16 Mar 2023 14:14:40 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id B9B983F7063; Thu, 16 Mar 2023 14:14:40 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , , , Subject: [PATCH v7 07/11] app/mldev: enable support for burst inferences Date: Thu, 16 Mar 2023 14:14:30 -0700 Message-ID: <20230316211434.13409-8-syalavarthi@marvell.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230316211434.13409-1-syalavarthi@marvell.com> References: <20221129070746.20396-1-syalavarthi@marvell.com> <20230316211434.13409-1-syalavarthi@marvell.com> MIME-Version: 1.0 Content-Type: text/plain X-Proofpoint-GUID: xHSY3JtMPdd6xMld9C1SdU-T-7_UB-z9 X-Proofpoint-ORIG-GUID: xHSY3JtMPdd6xMld9C1SdU-T-7_UB-z9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-16_14,2023-03-16_02,2023-02-09_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Added 'burst_size' support for inference tests. Burst size controls the number of inference requests handled during the burst enqueue and dequeue operations of the test case. Signed-off-by: Srikanth Yalavarthi Acked-by: Anup Prabhu --- app/test-mldev/ml_options.c | 12 +- app/test-mldev/ml_options.h | 2 + app/test-mldev/test_inference_common.c | 160 ++++++++++++++++++++++++- app/test-mldev/test_inference_common.h | 4 + doc/guides/tools/testmldev.rst | 24 ++++ 5 files changed, 199 insertions(+), 3 deletions(-) diff --git a/app/test-mldev/ml_options.c b/app/test-mldev/ml_options.c index 649cb9d8d1..19f2e1279e 100644 --- a/app/test-mldev/ml_options.c +++ b/app/test-mldev/ml_options.c @@ -24,6 +24,7 @@ ml_options_default(struct ml_options *opt) opt->socket_id = SOCKET_ID_ANY; opt->nb_filelist = 0; opt->repetitions = 1; + opt->burst_size = 1; opt->debug = false; } @@ -145,6 +146,12 @@ ml_parse_repetitions(struct ml_options *opt, const char *arg) return parser_read_uint64(&opt->repetitions, arg); } +static int +ml_parse_burst_size(struct ml_options *opt, const char *arg) +{ + return parser_read_uint16(&opt->burst_size, arg); +} + static void ml_dump_test_options(const char *testname) { @@ -159,7 +166,8 @@ ml_dump_test_options(const char *testname) if ((strcmp(testname, "inference_ordered") == 0) || (strcmp(testname, "inference_interleave") == 0)) { printf("\t\t--filelist : comma separated list of model, input and output\n" - "\t\t--repetitions : number of inference repetitions\n"); + "\t\t--repetitions : number of inference repetitions\n" + "\t\t--burst_size : inference burst size\n"); printf("\n"); } } @@ -186,6 +194,7 @@ static struct option lgopts[] = { {ML_MODELS, 1, 0, 0}, {ML_FILELIST, 1, 0, 0}, {ML_REPETITIONS, 1, 0, 0}, + {ML_BURST_SIZE, 1, 0, 0}, {ML_DEBUG, 0, 0, 0}, {ML_HELP, 0, 0, 0}, {NULL, 0, 0, 0}}; @@ -202,6 +211,7 @@ ml_opts_parse_long(int opt_idx, struct ml_options *opt) {ML_MODELS, ml_parse_models}, {ML_FILELIST, ml_parse_filelist}, {ML_REPETITIONS, ml_parse_repetitions}, + {ML_BURST_SIZE, ml_parse_burst_size}, }; for (i = 0; i < RTE_DIM(parsermap); i++) { diff --git a/app/test-mldev/ml_options.h b/app/test-mldev/ml_options.h index 6a13f97a30..00342d8a0c 100644 --- a/app/test-mldev/ml_options.h +++ b/app/test-mldev/ml_options.h @@ -18,6 +18,7 @@ #define ML_MODELS ("models") #define ML_FILELIST ("filelist") #define ML_REPETITIONS ("repetitions") +#define ML_BURST_SIZE ("burst_size") #define ML_DEBUG ("debug") #define ML_HELP ("help") @@ -34,6 +35,7 @@ struct ml_options { struct ml_filelist filelist[ML_TEST_MAX_MODELS]; uint8_t nb_filelist; uint64_t repetitions; + uint16_t burst_size; bool debug; }; diff --git a/app/test-mldev/test_inference_common.c b/app/test-mldev/test_inference_common.c index 6a6999d524..35323306de 100644 --- a/app/test-mldev/test_inference_common.c +++ b/app/test-mldev/test_inference_common.c @@ -124,6 +124,132 @@ ml_dequeue_single(void *arg) return 0; } +/* Enqueue inference requests with burst size greater than 1 */ +static int +ml_enqueue_burst(void *arg) +{ + struct test_inference *t = ml_test_priv((struct ml_test *)arg); + struct ml_core_args *args; + uint16_t ops_count; + uint64_t model_enq; + uint16_t burst_enq; + uint32_t lcore_id; + uint16_t pending; + uint16_t idx; + uint16_t fid; + uint16_t i; + int ret; + + lcore_id = rte_lcore_id(); + args = &t->args[lcore_id]; + model_enq = 0; + + if (args->nb_reqs == 0) + return 0; + +next_rep: + fid = args->start_fid; + +next_model: + ops_count = RTE_MIN(t->cmn.opt->burst_size, args->nb_reqs - model_enq); + ret = rte_mempool_get_bulk(t->op_pool, (void **)args->enq_ops, ops_count); + if (ret != 0) + goto next_model; + +retry: + ret = rte_mempool_get_bulk(t->model[fid].io_pool, (void **)args->reqs, ops_count); + if (ret != 0) + goto retry; + + for (i = 0; i < ops_count; i++) { + args->enq_ops[i]->model_id = t->model[fid].id; + args->enq_ops[i]->nb_batches = t->model[fid].info.batch_size; + args->enq_ops[i]->mempool = t->op_pool; + + args->enq_ops[i]->input.addr = args->reqs[i]->input; + args->enq_ops[i]->input.length = t->model[fid].inp_qsize; + args->enq_ops[i]->input.next = NULL; + + args->enq_ops[i]->output.addr = args->reqs[i]->output; + args->enq_ops[i]->output.length = t->model[fid].out_qsize; + args->enq_ops[i]->output.next = NULL; + + args->enq_ops[i]->user_ptr = args->reqs[i]; + args->reqs[i]->niters++; + args->reqs[i]->fid = fid; + } + + idx = 0; + pending = ops_count; + +enqueue_reqs: + burst_enq = rte_ml_enqueue_burst(t->cmn.opt->dev_id, 0, &args->enq_ops[idx], pending); + pending = pending - burst_enq; + + if (pending > 0) { + idx = idx + burst_enq; + goto enqueue_reqs; + } + + fid++; + if (fid <= args->end_fid) + goto next_model; + + model_enq = model_enq + ops_count; + if (model_enq < args->nb_reqs) + goto next_rep; + + return 0; +} + +/* Dequeue inference requests with burst size greater than 1 */ +static int +ml_dequeue_burst(void *arg) +{ + struct test_inference *t = ml_test_priv((struct ml_test *)arg); + struct rte_ml_op_error error; + struct ml_core_args *args; + struct ml_request *req; + uint64_t total_deq = 0; + uint16_t burst_deq = 0; + uint8_t nb_filelist; + uint32_t lcore_id; + uint32_t i; + + lcore_id = rte_lcore_id(); + args = &t->args[lcore_id]; + nb_filelist = args->end_fid - args->start_fid + 1; + + if (args->nb_reqs == 0) + return 0; + +dequeue_burst: + burst_deq = + rte_ml_dequeue_burst(t->cmn.opt->dev_id, 0, args->deq_ops, t->cmn.opt->burst_size); + + if (likely(burst_deq > 0)) { + total_deq += burst_deq; + + for (i = 0; i < burst_deq; i++) { + if (unlikely(args->deq_ops[i]->status == RTE_ML_OP_STATUS_ERROR)) { + rte_ml_op_error_get(t->cmn.opt->dev_id, args->deq_ops[i], &error); + ml_err("error_code = 0x%" PRIx64 ", error_message = %s\n", + error.errcode, error.message); + t->error_count[lcore_id]++; + } + req = (struct ml_request *)args->deq_ops[i]->user_ptr; + if (req != NULL) + rte_mempool_put(t->model[req->fid].io_pool, req); + } + rte_mempool_put_bulk(t->op_pool, (void *)args->deq_ops, burst_deq); + } + + if (total_deq < args->nb_reqs * nb_filelist) + goto dequeue_burst; + + return 0; +} + bool test_inference_cap_check(struct ml_options *opt) { @@ -173,6 +299,17 @@ test_inference_opt_check(struct ml_options *opt) return -EINVAL; } + if (opt->burst_size == 0) { + ml_err("Invalid option, burst_size = %u\n", opt->burst_size); + return -EINVAL; + } + + if (opt->burst_size > ML_TEST_MAX_POOL_SIZE) { + ml_err("Invalid option, burst_size = %u (> max supported = %d)\n", opt->burst_size, + ML_TEST_MAX_POOL_SIZE); + return -EINVAL; + } + /* check number of available lcores. */ if (rte_lcore_count() < 3) { ml_err("Insufficient lcores = %u\n", rte_lcore_count()); @@ -193,6 +330,7 @@ test_inference_opt_dump(struct ml_options *opt) /* dump test opts */ ml_dump("repetitions", "%" PRIu64, opt->repetitions); + ml_dump("burst_size", "%u", opt->burst_size); ml_dump_begin("filelist"); for (i = 0; i < opt->nb_filelist; i++) { @@ -208,6 +346,7 @@ test_inference_setup(struct ml_test *test, struct ml_options *opt) { struct test_inference *t; void *test_inference; + uint32_t lcore_id; int ret = 0; uint32_t i; @@ -233,13 +372,30 @@ test_inference_setup(struct ml_test *test, struct ml_options *opt) goto error; } - t->enqueue = ml_enqueue_single; - t->dequeue = ml_dequeue_single; + if (opt->burst_size == 1) { + t->enqueue = ml_enqueue_single; + t->dequeue = ml_dequeue_single; + } else { + t->enqueue = ml_enqueue_burst; + t->dequeue = ml_dequeue_burst; + } /* set model initial state */ for (i = 0; i < opt->nb_filelist; i++) t->model[i].state = MODEL_INITIAL; + for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { + t->args[lcore_id].enq_ops = rte_zmalloc_socket( + "ml_test_enq_ops", opt->burst_size * sizeof(struct rte_ml_op *), + RTE_CACHE_LINE_SIZE, opt->socket_id); + t->args[lcore_id].deq_ops = rte_zmalloc_socket( + "ml_test_deq_ops", opt->burst_size * sizeof(struct rte_ml_op *), + RTE_CACHE_LINE_SIZE, opt->socket_id); + t->args[lcore_id].reqs = rte_zmalloc_socket( + "ml_test_requests", opt->burst_size * sizeof(struct ml_request *), + RTE_CACHE_LINE_SIZE, opt->socket_id); + } + return 0; error: diff --git a/app/test-mldev/test_inference_common.h b/app/test-mldev/test_inference_common.h index abb20fc9fb..da800f2bd4 100644 --- a/app/test-mldev/test_inference_common.h +++ b/app/test-mldev/test_inference_common.h @@ -22,6 +22,10 @@ struct ml_core_args { uint64_t nb_reqs; uint16_t start_fid; uint16_t end_fid; + + struct rte_ml_op **enq_ops; + struct rte_ml_op **deq_ops; + struct ml_request **reqs; }; struct test_inference { diff --git a/doc/guides/tools/testmldev.rst b/doc/guides/tools/testmldev.rst index 1a1ab7d2bf..eb9081723b 100644 --- a/doc/guides/tools/testmldev.rst +++ b/doc/guides/tools/testmldev.rst @@ -91,6 +91,10 @@ The following are the command-line options supported by the test application. Set the number of inference repetitions to be executed in the test per each model. Default value is `1`. +* ``--burst_size `` + + Set the burst size to be used when enqueuing / dequeuing inferences. Default value is `1`. + * ``--debug`` Enable the tests to run in debug mode. @@ -236,6 +240,7 @@ Supported command line options for inference tests are following:: --socket_id --filelist --repetitions + --burst_size List of files to be used for the inference tests can be specified through the option @@ -244,6 +249,9 @@ List of files to be used for the inference tests can be specified through the op list of files required to test with a single model. Multiple filelist entries are supported by the test, one entry per model. Maximum number of file entries supported by the test is `8`. +When ``--burst_size `` option is specified for the test, enqueue and dequeue burst would +try to enqueue or dequeue ``num`` number of inferences per each call respectively. + .. Note:: * The ``--filelist `` is a mandatory option for running inference tests. @@ -280,6 +288,14 @@ Example command to run inference_ordered test: sudo /app/dpdk-test-mldev -c 0xf -a -- \ --test=inference_ordered --filelist model.bin,input.bin,output.bin +Example command to run inference_ordered test with a specific burst size: + +.. code-block:: console + + sudo /app/dpdk-test-mldev -c 0xf -a -- \ + --test=inference_ordered --filelist model.bin,input.bin,output.bin \ + --burst_size 12 + INFERENCE_INTERLEAVE Test ~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -320,6 +336,14 @@ Example command to run inference_interleave test with multiple models: --test=inference_interleave --filelist model_A.bin,input_A.bin,output_A.bin \ --filelist model_B.bin,input_B.bin,output_B.bin +Example command to run inference_interleave test with a specific burst size: + +.. code-block:: console + + sudo /app/dpdk-test-mldev -c 0xf -a -- \ + --test=inference_interleave --filelist model.bin,input.bin,output.bin \ + --burst_size 16 + Debug mode ---------- -- 2.17.1