From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 52AFC41E74; Sat, 11 Mar 2023 16:10:26 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 59FE742D41; Sat, 11 Mar 2023 16:09:24 +0100 (CET) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 5A09041143 for ; Sat, 11 Mar 2023 16:09:14 +0100 (CET) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 32BDP3A6030644 for ; Sat, 11 Mar 2023 07:09:13 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=u9m68Z76t3e2Kf9V8YloucB+aRzh/Xm+s5bD9hE1Oow=; b=FpxLJ3KSkq8uInAjVJq7Lu7I0V1ehf86lZc/rJVcC96N7wmyRz3ddvF5ekraQ0Q068ZC G9vCd3ptVCgnyvZJdp70MCD/qiCuHFpOVlJLQ7+Z/kBuMc1BE6Xiwfhv172SUcREC6zK l3ls0FODsOf0fozziln00mkFGrOopeRQzNcZJd3cKGhZlCtaoGvO12C9ZAX/SabHSkMb T2Zfr5RG2R/FHVa7Eez78VTCLD8/c7Xj7MQ37/TmH3wLWMfmOW3wI0dMdX4C2XtYNNnt yAETT+ZkPHbtOogXUtg9ibkrmJYZOve4PhcQUAMdCHwxoMcW5bGfRxyIOrCrkFN8DcAw 5Q== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3p8t1t086h-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Sat, 11 Mar 2023 07:09:13 -0800 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Sat, 11 Mar 2023 07:09:10 -0800 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.42 via Frontend Transport; Sat, 11 Mar 2023 07:09:09 -0800 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 99F373F7070; Sat, 11 Mar 2023 07:09:09 -0800 (PST) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , , , Subject: [PATCH v6 09/12] app/mldev: enable support for inference batches Date: Sat, 11 Mar 2023 07:09:02 -0800 Message-ID: <20230311150905.26824-10-syalavarthi@marvell.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230311150905.26824-1-syalavarthi@marvell.com> References: <20221129070746.20396-1-syalavarthi@marvell.com> <20230311150905.26824-1-syalavarthi@marvell.com> MIME-Version: 1.0 Content-Type: text/plain X-Proofpoint-GUID: w1GN4Iil_TT8IyszkwvBgcMG_aCNQmGR X-Proofpoint-ORIG-GUID: w1GN4Iil_TT8IyszkwvBgcMG_aCNQmGR X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-11_04,2023-03-10_01,2023-02-09_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Enabled support to execute multiple batches of inferences per each enqueue request. Input and reference for the test should be appropriately provided for multi-batch run. Number of batches can be specified through "--batches" option. Signed-off-by: Srikanth Yalavarthi Acked-by: Anup Prabhu --- app/test-mldev/ml_options.c | 15 ++++++++++++--- app/test-mldev/ml_options.h | 2 ++ app/test-mldev/test_inference_common.c | 22 +++++++++++++--------- app/test-mldev/test_model_common.c | 6 ++++++ app/test-mldev/test_model_common.h | 1 + 5 files changed, 34 insertions(+), 12 deletions(-) diff --git a/app/test-mldev/ml_options.c b/app/test-mldev/ml_options.c index c81dec6e30..499bfde899 100644 --- a/app/test-mldev/ml_options.c +++ b/app/test-mldev/ml_options.c @@ -27,6 +27,7 @@ ml_options_default(struct ml_options *opt) opt->burst_size = 1; opt->queue_pairs = 1; opt->queue_size = 1; + opt->batches = 0; opt->debug = false; } @@ -170,6 +171,12 @@ ml_parse_queue_size(struct ml_options *opt, const char *arg) return parser_read_uint16(&opt->queue_size, arg); } +static int +ml_parse_batches(struct ml_options *opt, const char *arg) +{ + return parser_read_uint16(&opt->batches, arg); +} + static void ml_dump_test_options(const char *testname) { @@ -190,7 +197,8 @@ ml_dump_test_options(const char *testname) "\t\t--repetitions : number of inference repetitions\n" "\t\t--burst_size : inference burst size\n" "\t\t--queue_pairs : number of queue pairs to create\n" - "\t\t--queue_size : size fo queue-pair\n"); + "\t\t--queue_size : size fo queue-pair\n" + "\t\t--batches : number of batches of input\n"); printf("\n"); } } @@ -214,7 +222,8 @@ static struct option lgopts[] = { {ML_TEST, 1, 0, 0}, {ML_DEVICE_ID, 1, 0, 0}, {ML_SOCKET_ID, 1, 0, 0}, {ML_MODELS, 1, 0, 0}, {ML_FILELIST, 1, 0, 0}, {ML_REPETITIONS, 1, 0, 0}, {ML_BURST_SIZE, 1, 0, 0}, {ML_QUEUE_PAIRS, 1, 0, 0}, {ML_QUEUE_SIZE, 1, 0, 0}, - {ML_DEBUG, 0, 0, 0}, {ML_HELP, 0, 0, 0}, {NULL, 0, 0, 0}}; + {ML_BATCHES, 1, 0, 0}, {ML_DEBUG, 0, 0, 0}, {ML_HELP, 0, 0, 0}, + {NULL, 0, 0, 0}}; static int ml_opts_parse_long(int opt_idx, struct ml_options *opt) @@ -226,7 +235,7 @@ ml_opts_parse_long(int opt_idx, struct ml_options *opt) {ML_SOCKET_ID, ml_parse_socket_id}, {ML_MODELS, ml_parse_models}, {ML_FILELIST, ml_parse_filelist}, {ML_REPETITIONS, ml_parse_repetitions}, {ML_BURST_SIZE, ml_parse_burst_size}, {ML_QUEUE_PAIRS, ml_parse_queue_pairs}, - {ML_QUEUE_SIZE, ml_parse_queue_size}, + {ML_QUEUE_SIZE, ml_parse_queue_size}, {ML_BATCHES, ml_parse_batches}, }; for (i = 0; i < RTE_DIM(parsermap); i++) { diff --git a/app/test-mldev/ml_options.h b/app/test-mldev/ml_options.h index c4018ee9d1..48fe064150 100644 --- a/app/test-mldev/ml_options.h +++ b/app/test-mldev/ml_options.h @@ -21,6 +21,7 @@ #define ML_BURST_SIZE ("burst_size") #define ML_QUEUE_PAIRS ("queue_pairs") #define ML_QUEUE_SIZE ("queue_size") +#define ML_BATCHES ("batches") #define ML_DEBUG ("debug") #define ML_HELP ("help") @@ -40,6 +41,7 @@ struct ml_options { uint16_t burst_size; uint16_t queue_pairs; uint16_t queue_size; + uint16_t batches; bool debug; }; diff --git a/app/test-mldev/test_inference_common.c b/app/test-mldev/test_inference_common.c index b4ad3c4b72..0f281aed6c 100644 --- a/app/test-mldev/test_inference_common.c +++ b/app/test-mldev/test_inference_common.c @@ -50,7 +50,7 @@ ml_enqueue_single(void *arg) goto retry; op->model_id = t->model[fid].id; - op->nb_batches = t->model[fid].info.batch_size; + op->nb_batches = t->model[fid].nb_batches; op->mempool = t->op_pool; op->input.addr = req->input; @@ -163,7 +163,7 @@ ml_enqueue_burst(void *arg) for (i = 0; i < ops_count; i++) { args->enq_ops[i]->model_id = t->model[fid].id; - args->enq_ops[i]->nb_batches = t->model[fid].info.batch_size; + args->enq_ops[i]->nb_batches = t->model[fid].nb_batches; args->enq_ops[i]->mempool = t->op_pool; args->enq_ops[i]->input.addr = args->reqs[i]->input; @@ -359,6 +359,11 @@ test_inference_opt_dump(struct ml_options *opt) ml_dump("queue_pairs", "%u", opt->queue_pairs); ml_dump("queue_size", "%u", opt->queue_size); + if (opt->batches == 0) + ml_dump("batches", "%u (default)", opt->batches); + else + ml_dump("batches", "%u", opt->batches); + ml_dump_begin("filelist"); for (i = 0; i < opt->nb_filelist; i++) { ml_dump_list("model", i, opt->filelist[i].model); @@ -528,8 +533,8 @@ ml_request_initialize(struct rte_mempool *mp, void *opaque, void *obj, unsigned req->niters = 0; /* quantize data */ - rte_ml_io_quantize(t->cmn.opt->dev_id, t->model[t->fid].id, - t->model[t->fid].info.batch_size, t->model[t->fid].input, req->input); + rte_ml_io_quantize(t->cmn.opt->dev_id, t->model[t->fid].id, t->model[t->fid].nb_batches, + t->model[t->fid].input, req->input); } int @@ -547,7 +552,7 @@ ml_inference_iomem_setup(struct ml_test *test, struct ml_options *opt, uint16_t int ret; /* get input buffer size */ - ret = rte_ml_io_input_size_get(opt->dev_id, t->model[fid].id, t->model[fid].info.batch_size, + ret = rte_ml_io_input_size_get(opt->dev_id, t->model[fid].id, t->model[fid].nb_batches, &t->model[fid].inp_qsize, &t->model[fid].inp_dsize); if (ret != 0) { ml_err("Failed to get input size, model : %s\n", opt->filelist[fid].model); @@ -555,9 +560,8 @@ ml_inference_iomem_setup(struct ml_test *test, struct ml_options *opt, uint16_t } /* get output buffer size */ - ret = rte_ml_io_output_size_get(opt->dev_id, t->model[fid].id, - t->model[fid].info.batch_size, &t->model[fid].out_qsize, - &t->model[fid].out_dsize); + ret = rte_ml_io_output_size_get(opt->dev_id, t->model[fid].id, t->model[fid].nb_batches, + &t->model[fid].out_qsize, &t->model[fid].out_dsize); if (ret != 0) { ml_err("Failed to get input size, model : %s\n", opt->filelist[fid].model); return ret; @@ -702,7 +706,7 @@ ml_request_finish(struct rte_mempool *mp, void *opaque, void *obj, unsigned int return; t->nb_used++; - rte_ml_io_dequantize(t->cmn.opt->dev_id, model->id, t->model[req->fid].info.batch_size, + rte_ml_io_dequantize(t->cmn.opt->dev_id, model->id, t->model[req->fid].nb_batches, req->output, model->output); } diff --git a/app/test-mldev/test_model_common.c b/app/test-mldev/test_model_common.c index b94d46154d..c28e452f29 100644 --- a/app/test-mldev/test_model_common.c +++ b/app/test-mldev/test_model_common.c @@ -71,6 +71,12 @@ ml_model_load(struct ml_test *test, struct ml_options *opt, struct ml_model *mod return ret; } + /* Update number of batches */ + if (opt->batches == 0) + model->nb_batches = model->info.batch_size; + else + model->nb_batches = opt->batches; + model->state = MODEL_LOADED; return 0; diff --git a/app/test-mldev/test_model_common.h b/app/test-mldev/test_model_common.h index 5ee975109d..19429ce142 100644 --- a/app/test-mldev/test_model_common.h +++ b/app/test-mldev/test_model_common.h @@ -30,6 +30,7 @@ struct ml_model { uint8_t *output; struct rte_mempool *io_pool; + uint32_t nb_batches; }; int ml_model_load(struct ml_test *test, struct ml_options *opt, struct ml_model *model, -- 2.17.1