From: Jerin Jacob <jerinjacobk@gmail.com>
To: Thomas Monjalon <thomas@monjalon.net>
Cc: Jerin Jacob <jerinj@marvell.com>,
dev@dpdk.org, ferruh.yigit@amd.com, ajit.khaparde@broadcom.com,
aboyer@pensando.io, andrew.rybchenko@oktetlabs.ru,
beilei.xing@intel.com, bruce.richardson@intel.com,
chas3@att.com, chenbo.xia@intel.com, ciara.loftus@intel.com,
dsinghrawat@marvell.com, ed.czeck@atomicrules.com,
evgenys@amazon.com, grive@u256.net, g.singh@nxp.com,
zhouguoyang@huawei.com, haiyue.wang@intel.com,
hkalra@marvell.com, heinrich.kuhn@corigine.com,
hemant.agrawal@nxp.com, hyonkim@cisco.com, igorch@amazon.com,
irusskikh@marvell.com, jgrajcia@cisco.com,
jasvinder.singh@intel.com, jianwang@trustnetic.com,
jiawenwu@trustnetic.com, jingjing.wu@intel.com,
johndale@cisco.com, john.miller@atomicrules.com,
linville@tuxdriver.com, keith.wiles@intel.com,
kirankumark@marvell.com, oulijun@huawei.com, lironh@marvell.com,
longli@microsoft.com, mw@semihalf.com, spinler@cesnet.cz,
matan@nvidia.com, matt.peters@windriver.com,
maxime.coquelin@redhat.com, mk@semihalf.com, humin29@huawei.com,
pnalla@marvell.com, ndabilpuram@marvell.com,
qiming.yang@intel.com, qi.z.zhang@intel.com, radhac@marvell.com,
rahul.lakkireddy@chelsio.com, rmody@marvell.com,
rosen.xu@intel.com, sachin.saxena@oss.nxp.com,
skoteshwar@marvell.com, shshaikh@marvell.com,
shaibran@amazon.com, shepard.siegel@atomicrules.com,
asomalap@amd.com, somnath.kotur@broadcom.com,
sthemmin@microsoft.com, steven.webster@windriver.com,
skori@marvell.com, mtetsuyah@gmail.com, vburru@marvell.com,
viacheslavo@nvidia.com, xiao.w.wang@intel.com,
cloud.wangxiaoyun@huawei.com, yisen.zhuang@huawei.com,
yongwang@vmware.com, xuanziyang2@huawei.com,
pkapoor@marvell.com, nadavh@marvell.com, sburla@marvell.com,
pathreya@marvell.com, gakhil@marvell.com,
dmitry.kozliuk@gmail.com, anatoly.burakov@intel.com,
cristian.dumitrescu@intel.com, honnappa.nagarahalli@arm.com,
mattias.ronnblom@ericsson.com, ruifeng.wang@arm.com,
drc@linux.vnet.ibm.com, konstantin.ananyev@intel.com,
olivier.matz@6wind.com, jay.jayatheerthan@intel.com,
asekhar@marvell.com, pbhagavatula@marvell.com,
eagostini@nvidia.com, syalavarthi@marvell.com,
dchickles@marvell.com, sshankarnara@marvell.com,
david.marchand@redhat.com
Subject: Re: [dpdk-dev] [PATCH v1 00/12] mldev: introduce machine learning device library
Date: Thu, 26 Jan 2023 00:31:43 +0530 [thread overview]
Message-ID: <CALBAE1OFk7j57qefVYaaySi0zSrJR0-BSAEeAmTU2bwVtJLErQ@mail.gmail.com> (raw)
In-Reply-To: <3877164.bqPgKRP4r2@thomas>
On Wed, Jan 25, 2023 at 7:50 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 14/11/2022 13:02, jerinj@marvell.com:
> > From: Jerin Jacob <jerinj@marvell.com>
> >
> > Machine learning inference library
> > ==================================
> >
> > Definition of machine learning inference
> > ----------------------------------------
> > Inference in machine learning is the process of making an output prediction
> > based on new input data using a pre-trained machine learning model.
> >
> > The scope of the RFC would include only inferencing with pre-trained machine learning models,
> > training and building/compiling the ML models is out of scope for this RFC or
> > DPDK mldev API. Use existing machine learning compiler frameworks for model creation.
> >
> > Motivation for the new library
> > ------------------------------
> > Multiple semiconductor vendors are offering accelerator products such as DPU
> > (often called Smart-NIC), FPGA, GPU, etc., which have ML inferencing capabilities
> > integrated as part of the product. Use of ML inferencing is increasing in the domain
> > of packet processing for flow classification, intrusion, malware and anomaly detection.
> >
> > Lack of inferencing support through DPDK APIs will involve complexities and
> > increased latency from moving data across frameworks (i.e, dataplane to
> > non dataplane ML frameworks and vice-versa). Having a standardized DPDK APIs for ML
> > inferencing would enable the dataplane solutions to harness the benefit of inline
> > inferencing supported by the hardware.
> >
> > Contents
> > ---------------
> >
> > A) API specification for:
> >
> > 1) Discovery of ML capabilities (e.g., device specific features) in a vendor
> > independent fashion
> > 2) Definition of functions to handle ML devices, which includes probing,
> > initialization and termination of the devices.
> > 3) Definition of functions to handle ML models used to perform inference operations.
> > 4) Definition of function to handle quantize and dequantize operations
> >
> > B) Common code for above specification
Thanks for the review.
>
> Can we compare this library with WinML?
> https://learn.microsoft.com/en-us/windows/ai/windows-ml/api-reference
Proposed DPDK library supports only inferencing with pre-trained models.
> Is there things we can learn from it?
Comparing to winML, API provide functionality similar to
"LearningModel*" classes provides.
Support related to handling custom operators and Native APIs like
winML is provided through this API.
There may more features which we can add where there are drivers which
supports it.
>
>
> > ML Model: An ML model is an algorithm trained over a dataset. A model consists of
> > procedure/algorithm and data/pattern required to make predictions on live data.
> > Once the model is created and trained outside of the DPDK scope, the model can be loaded
> > via rte_ml_model_load() and then start it using rte_ml_model_start() API.
> > The rte_ml_model_params_update() can be used to update the model parameters such as weight
> > and bias without unloading the model using rte_ml_model_unload().
>
> The fact that the model is prepared outside means the model format is free
> and probably different per mldev driver.
> I think it is OK but it requires a lot of documentation effort to explain
> how to bind the model and its parameters with the DPDK API.
> Also we may need to pass some metadata from the model builder
> to the inference engine in order to enable optimizations prepared in the model.
> And the other way, we may need inference capabilities in order to generate
> an optimized model which can run in the inference engine.
The base API specification kept absolute minimum. Currently, weight and biases
parameters updated through rte_ml_model_params_update(). It can be extended
when there are drivers supports it or if you have any specific
parameter you would like to add
it in rte_ml_model_params_update().
Other metadata data like batch, shapes, formats queried using rte_ml_io_info().
>
>
> [...]
> > Typical application utilisation of the ML API will follow the following
> > programming flow.
> >
> > - rte_ml_dev_configure()
> > - rte_ml_dev_queue_pair_setup()
> > - rte_ml_model_load()
> > - rte_ml_model_start()
> > - rte_ml_model_info()
> > - rte_ml_dev_start()
> > - rte_ml_enqueue_burst()
> > - rte_ml_dequeue_burst()
> > - rte_ml_model_stop()
> > - rte_ml_model_unload()
> > - rte_ml_dev_stop()
> > - rte_ml_dev_close()
>
> Where is parameters update in this flow?
Added the mandatory APIs in the top level flow doc.
rte_ml_model_params_update() used to update the parameters.
> Should we update all parameters at once or can it be done more fine-grain?
Currently, rte_ml_model_params_update() can be used to update weight
and bias via buffer when device is
in stop state and without unloading the model.
>
>
> Question about the memory used by mldev:
> Can we manage where the memory is allocated (host, device, mix, etc)?
Just passing buffer pointers now like other subsystem.
Other EAL infra service can take care of the locality of memory as it
is not specific to ML dev.
+/** ML operation's input and output buffer representation as scatter
gather list
+ */
+struct rte_ml_buff_seg {
+ rte_iova_t iova_addr;
+ /**< IOVA address of segment buffer. */
+ void *addr;
+ /**< Virtual address of segment buffer. */
+ uint32_t length;
+ /**< Segment length. */
+ uint32_t reserved;
+ /**< Reserved for future use. */
+ struct rte_ml_buff_seg *next;
+ /**< Points to next segment. Value NULL represents the last segment. */
+};
>
>
next prev parent reply other threads:[~2023-01-25 19:02 UTC|newest]
Thread overview: 80+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-08-03 13:28 [dpdk-dev] [RFC PATCH 0/1] " jerinj
2022-08-03 13:28 ` [dpdk-dev] [RFC PATCH 1/1] " jerinj
2022-11-14 12:02 ` [dpdk-dev] [PATCH v1 00/12] " jerinj
2022-11-14 12:02 ` [dpdk-dev] [PATCH v1 01/12] " jerinj
2023-02-01 13:34 ` Shivah Shankar Shankar Narayan Rao
2023-02-03 0:25 ` Stephen Hemminger
2023-02-03 8:42 ` Thomas Monjalon
2023-02-03 17:33 ` Stephen Hemminger
2023-02-03 20:18 ` Thomas Monjalon
2023-02-03 20:26 ` Stephen Hemminger
2023-02-03 20:49 ` Thomas Monjalon
2023-02-05 23:41 ` Stephen Hemminger
2023-02-03 10:01 ` Jerin Jacob
2023-02-03 0:25 ` Stephen Hemminger
2023-02-03 10:04 ` Jerin Jacob
2023-02-03 0:28 ` Stephen Hemminger
2023-02-03 10:03 ` Jerin Jacob
2023-02-02 5:26 ` Shivah Shankar Shankar Narayan Rao
2022-11-14 12:02 ` [dpdk-dev] [PATCH v1 02/12] mldev: add PMD functions for ML device jerinj
2022-11-14 12:02 ` [dpdk-dev] [PATCH v1 03/12] mldev: support device handling functions jerinj
2022-11-14 12:02 ` [dpdk-dev] [PATCH v1 04/12] mldev: support device queue-pair setup jerinj
2022-11-14 12:02 ` [dpdk-dev] [PATCH v1 05/12] mldev: support handling ML models jerinj
2022-11-14 12:02 ` [dpdk-dev] [PATCH v1 06/12] mldev: support input and output data handling jerinj
2022-11-14 12:02 ` [dpdk-dev] [PATCH v1 07/12] mldev: support op pool and its operations jerinj
2022-11-14 12:02 ` [dpdk-dev] [PATCH v1 08/12] mldev: support inference enqueue and dequeue jerinj
2022-11-14 12:02 ` [dpdk-dev] [PATCH v1 09/12] mldev: support device statistics jerinj
2022-11-14 12:02 ` [dpdk-dev] [PATCH v1 10/12] mldev: support device extended statistics jerinj
2022-11-14 12:02 ` [dpdk-dev] [PATCH v1 11/12] mldev: support to retrieve error information jerinj
2022-11-14 12:02 ` [dpdk-dev] [PATCH v1 12/12] mldev: support to get debug info and test device jerinj
2023-01-25 14:20 ` [dpdk-dev] [PATCH v1 00/12] mldev: introduce machine learning device library Thomas Monjalon
2023-01-25 19:01 ` Jerin Jacob [this message]
2023-01-26 11:11 ` Thomas Monjalon
2023-01-27 2:33 ` [EXT] " Shivah Shankar Shankar Narayan Rao
2023-01-27 4:29 ` Jerin Jacob
2023-01-27 11:34 ` Thomas Monjalon
2023-01-28 11:27 ` Jerin Jacob
2023-02-01 16:57 ` Thomas Monjalon
2023-02-01 17:33 ` Jerin Jacob
2023-02-06 20:24 ` [dpdk-dev] [PATCH v2 " jerinj
2023-02-06 20:24 ` [dpdk-dev] [PATCH v2 01/12] " jerinj
2023-02-06 20:24 ` [dpdk-dev] [PATCH v2 02/12] mldev: support PMD functions for ML device jerinj
2023-02-06 21:04 ` Stephen Hemminger
2023-02-06 22:17 ` Thomas Monjalon
2023-02-07 5:16 ` Jerin Jacob
2023-02-06 20:24 ` [dpdk-dev] [PATCH v2 03/12] mldev: support ML device handling functions jerinj
2023-02-06 20:24 ` [dpdk-dev] [PATCH v2 04/12] mldev: support ML device queue-pair setup jerinj
2023-02-06 20:24 ` [dpdk-dev] [PATCH v2 05/12] mldev: support handling ML models jerinj
2023-02-06 20:24 ` [dpdk-dev] [PATCH v2 06/12] mldev: support input and output data handling jerinj
2023-02-06 20:24 ` [dpdk-dev] [PATCH v2 07/12] mldev: support ML op pool and ops jerinj
2023-02-06 20:24 ` [dpdk-dev] [PATCH v2 08/12] mldev: support inference enqueue and dequeue jerinj
2023-02-06 20:24 ` [dpdk-dev] [PATCH v2 09/12] mldev: support device statistics jerinj
2023-02-06 20:24 ` [dpdk-dev] [PATCH v2 10/12] mldev: support device extended statistics jerinj
2023-02-06 20:24 ` [dpdk-dev] [PATCH v2 11/12] mldev: support to retrieve error information jerinj
2023-02-06 20:24 ` [dpdk-dev] [PATCH v2 12/12] mldev: support to get debug info and test device jerinj
2023-02-07 15:13 ` [dpdk-dev] [PATCH v3 00/12] mldev: introduce machine learning device library jerinj
2023-02-07 15:13 ` [dpdk-dev] [PATCH v3 01/12] " jerinj
2023-02-07 15:13 ` [dpdk-dev] [PATCH v3 02/12] mldev: support PMD functions for ML device jerinj
2023-02-07 15:13 ` [dpdk-dev] [PATCH v3 03/12] mldev: support ML device handling functions jerinj
2023-02-07 15:13 ` [dpdk-dev] [PATCH v3 04/12] mldev: support ML device queue-pair setup jerinj
2023-02-07 15:13 ` [dpdk-dev] [PATCH v3 05/12] mldev: support handling ML models jerinj
2023-02-07 15:13 ` [dpdk-dev] [PATCH v3 06/12] mldev: support input and output data handling jerinj
2023-02-07 15:13 ` [dpdk-dev] [PATCH v3 07/12] mldev: support ML op pool and ops jerinj
2023-02-07 15:13 ` [dpdk-dev] [PATCH v3 08/12] mldev: support inference enqueue and dequeue jerinj
2023-02-07 15:13 ` [dpdk-dev] [PATCH v3 09/12] mldev: support device statistics jerinj
2023-02-07 15:13 ` [dpdk-dev] [PATCH v3 10/12] mldev: support device extended statistics jerinj
2023-02-07 15:13 ` [dpdk-dev] [PATCH v3 11/12] mldev: support to retrieve error information jerinj
2023-02-07 15:13 ` [dpdk-dev] [PATCH v3 12/12] mldev: support to get debug info and test device jerinj
2023-02-15 12:55 ` [dpdk-dev] [PATCH v3 00/12] mldev: introduce machine learning device library Ferruh Yigit
2023-02-15 17:03 ` Jerin Jacob
2023-03-09 17:33 ` Thomas Monjalon
2022-08-03 15:19 ` [dpdk-dev] [RFC PATCH 0/1] " Stephen Hemminger
2022-08-16 13:13 ` Jerin Jacob
2022-08-16 15:45 ` Morten Brørup
2022-08-16 16:34 ` Honnappa Nagarahalli
2022-08-17 14:53 ` Jerin Jacob
2023-01-25 13:47 ` Thomas Monjalon
2023-01-25 13:54 ` Jerin Jacob
2022-08-17 5:37 ` Jerin Jacob
2022-08-17 6:58 ` Morten Brørup
2023-01-25 13:45 ` Thomas Monjalon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CALBAE1OFk7j57qefVYaaySi0zSrJR0-BSAEeAmTU2bwVtJLErQ@mail.gmail.com \
--to=jerinjacobk@gmail.com \
--cc=aboyer@pensando.io \
--cc=ajit.khaparde@broadcom.com \
--cc=anatoly.burakov@intel.com \
--cc=andrew.rybchenko@oktetlabs.ru \
--cc=asekhar@marvell.com \
--cc=asomalap@amd.com \
--cc=beilei.xing@intel.com \
--cc=bruce.richardson@intel.com \
--cc=chas3@att.com \
--cc=chenbo.xia@intel.com \
--cc=ciara.loftus@intel.com \
--cc=cloud.wangxiaoyun@huawei.com \
--cc=cristian.dumitrescu@intel.com \
--cc=david.marchand@redhat.com \
--cc=dchickles@marvell.com \
--cc=dev@dpdk.org \
--cc=dmitry.kozliuk@gmail.com \
--cc=drc@linux.vnet.ibm.com \
--cc=dsinghrawat@marvell.com \
--cc=eagostini@nvidia.com \
--cc=ed.czeck@atomicrules.com \
--cc=evgenys@amazon.com \
--cc=ferruh.yigit@amd.com \
--cc=g.singh@nxp.com \
--cc=gakhil@marvell.com \
--cc=grive@u256.net \
--cc=haiyue.wang@intel.com \
--cc=heinrich.kuhn@corigine.com \
--cc=hemant.agrawal@nxp.com \
--cc=hkalra@marvell.com \
--cc=honnappa.nagarahalli@arm.com \
--cc=humin29@huawei.com \
--cc=hyonkim@cisco.com \
--cc=igorch@amazon.com \
--cc=irusskikh@marvell.com \
--cc=jasvinder.singh@intel.com \
--cc=jay.jayatheerthan@intel.com \
--cc=jerinj@marvell.com \
--cc=jgrajcia@cisco.com \
--cc=jianwang@trustnetic.com \
--cc=jiawenwu@trustnetic.com \
--cc=jingjing.wu@intel.com \
--cc=john.miller@atomicrules.com \
--cc=johndale@cisco.com \
--cc=keith.wiles@intel.com \
--cc=kirankumark@marvell.com \
--cc=konstantin.ananyev@intel.com \
--cc=linville@tuxdriver.com \
--cc=lironh@marvell.com \
--cc=longli@microsoft.com \
--cc=matan@nvidia.com \
--cc=matt.peters@windriver.com \
--cc=mattias.ronnblom@ericsson.com \
--cc=maxime.coquelin@redhat.com \
--cc=mk@semihalf.com \
--cc=mtetsuyah@gmail.com \
--cc=mw@semihalf.com \
--cc=nadavh@marvell.com \
--cc=ndabilpuram@marvell.com \
--cc=olivier.matz@6wind.com \
--cc=oulijun@huawei.com \
--cc=pathreya@marvell.com \
--cc=pbhagavatula@marvell.com \
--cc=pkapoor@marvell.com \
--cc=pnalla@marvell.com \
--cc=qi.z.zhang@intel.com \
--cc=qiming.yang@intel.com \
--cc=radhac@marvell.com \
--cc=rahul.lakkireddy@chelsio.com \
--cc=rmody@marvell.com \
--cc=rosen.xu@intel.com \
--cc=ruifeng.wang@arm.com \
--cc=sachin.saxena@oss.nxp.com \
--cc=sburla@marvell.com \
--cc=shaibran@amazon.com \
--cc=shepard.siegel@atomicrules.com \
--cc=shshaikh@marvell.com \
--cc=skori@marvell.com \
--cc=skoteshwar@marvell.com \
--cc=somnath.kotur@broadcom.com \
--cc=spinler@cesnet.cz \
--cc=sshankarnara@marvell.com \
--cc=steven.webster@windriver.com \
--cc=sthemmin@microsoft.com \
--cc=syalavarthi@marvell.com \
--cc=thomas@monjalon.net \
--cc=vburru@marvell.com \
--cc=viacheslavo@nvidia.com \
--cc=xiao.w.wang@intel.com \
--cc=xuanziyang2@huawei.com \
--cc=yisen.zhuang@huawei.com \
--cc=yongwang@vmware.com \
--cc=zhouguoyang@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).