From: "Ji, Kai" <kai.ji@intel.com>
To: Akhil Goyal <gakhil@marvell.com>, "dev@dpdk.org" <dev@dpdk.org>
Subject: RE: [EXT] [dpdk-dev v3 1/1] lib/cryptodev: multi-process IPC request handler
Date: Thu, 6 Oct 2022 00:57:25 +0000 [thread overview]
Message-ID: <DM6PR11MB3403AE2F2B85435E29AADD14815C9@DM6PR11MB3403.namprd11.prod.outlook.com> (raw)
In-Reply-To: <CO6PR18MB448477A0CA8E175C84BCE653D85A9@CO6PR18MB4484.namprd18.prod.outlook.com>
Please see my comments inline
> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Tuesday, October 4, 2022 7:13 PM
> To: Ji, Kai <kai.ji@intel.com>; dev@dpdk.org
> Subject: RE: [EXT] [dpdk-dev v3 1/1] lib/cryptodev: multi-process IPC request
> handler
>
> > This patch add a function to support queue-pair configuration request
> > to allow the primary or secondary process to setup/free the queue-pair
> > via IPC handler. Add in queue pair in-used by process id array in
> > rte_cryptodev_data for pid tracking.
> >
> > Signed-off-by: Kai Ji <kai.ji@intel.com>
>
> I had a comment on v1, that you should include all PMD maintainers as well
> When sending any major change in library layer.
> But I think you have not read the comments made on v1.
>
> > ---
> > v3:
> > - addin missing free function for qp_in_use_by_pid
> >
> > v2:
> > - code rework
> > ---
> > lib/cryptodev/cryptodev_pmd.h | 3 +- lib/cryptodev/rte_cryptodev.c
> > | 92 +++++++++++++++++++++++++++++++++++
> > lib/cryptodev/rte_cryptodev.h | 37 ++++++++++++++
> > lib/cryptodev/version.map | 2 +
> > 4 files changed, 133 insertions(+), 1 deletion(-)
> >
> > diff --git a/lib/cryptodev/cryptodev_pmd.h
> > b/lib/cryptodev/cryptodev_pmd.h index 09ba952455..f404604963 100644
> > --- a/lib/cryptodev/cryptodev_pmd.h
> > +++ b/lib/cryptodev/cryptodev_pmd.h
> > @@ -78,7 +78,8 @@ struct rte_cryptodev_data {
> > void **queue_pairs;
> > /** Number of device queue pairs. */
> > uint16_t nb_queue_pairs;
> > -
> > + /** Array of process id used for queue pairs **/
> > + uint16_t *qp_in_use_by_pid;
> Why array? And if an array, how will its depth will be calculated.
>
> I commented on v1, that the in_use pid can be stored inside queue private
> data.
[KJ] the depth can be calculated using the max_nb_queue_pairs in dev_info. By stored this information in rte_cryptodev_data struct, we are no longer need to modify every queue private crypto PMDs, and this pid info is not exposed to userspace neither.
>
> > /** PMD-specific private data */
> > void *dev_private;
> > } __rte_cache_aligned;
> > diff --git a/lib/cryptodev/rte_cryptodev.c
> > b/lib/cryptodev/rte_cryptodev.c index 9e76a1c72d..dab6a37bff 100644
> > --- a/lib/cryptodev/rte_cryptodev.c
> > +++ b/lib/cryptodev/rte_cryptodev.c
> > @@ -49,6 +49,9 @@ struct rte_crypto_fp_ops
> > rte_crypto_fp_ops[RTE_CRYPTO_MAX_DEVS];
> > /* spinlock for crypto device callbacks */ static rte_spinlock_t
> > rte_cryptodev_cb_lock = RTE_SPINLOCK_INITIALIZER;
> >
> > +/* crypto queue pair config */
> > +#define CRYPTODEV_MP_REQ "cryptodev_mp_request"
> > +
> > /**
> > * The user application callback description.
> > *
> > @@ -1050,6 +1053,9 @@ rte_cryptodev_pmd_release_device(struct
> > rte_cryptodev *cryptodev)
> > return ret;
> > }
> >
> > + if (cryptodev->data->qp_in_use_by_pid)
> > + rte_free(cryptodev->data->qp_in_use_by_pid);
[KJ] this is qp_in_use_by_pid get freed
> > +
> > ret = rte_cryptodev_data_free(dev_id,
> > &cryptodev_globals.data[dev_id]);
> > if (ret < 0)
> > return ret;
> > @@ -1138,6 +1144,21 @@ rte_cryptodev_queue_pairs_config(struct
> > rte_cryptodev *dev, uint16_t nb_qpairs,
> >
> > }
> > dev->data->nb_queue_pairs = nb_qpairs;
> > +
> > + if (dev->data->qp_in_use_by_pid == NULL) {
> > + dev->data->qp_in_use_by_pid = rte_zmalloc_socket(
> > + "cryptodev->qp_in_use_by_pid",
> > + sizeof(dev->data->qp_in_use_by_pid[0]) *
> > + dev_info.max_nb_queue_pairs,
> > + RTE_CACHE_LINE_SIZE, socket_id);
[KJ] this is qp_in_use_by_pid get malloced
> > + if (dev->data->qp_in_use_by_pid == NULL) {
> > + CDEV_LOG_ERR("failed to get memory for qp meta
> > data, "
> > + "nb_queues %u",
> > + nb_qpairs);
> > + return -(ENOMEM);
> > + }
> > + }
> > +
> > return 0;
> > }
> >
> > @@ -1400,6 +1421,77 @@ rte_cryptodev_queue_pair_setup(uint8_t
> dev_id,
> > uint16_t queue_pair_id,
> > socket_id);
> > }
> >
> > +static int
> > +rte_cryptodev_ipc_request(const struct rte_mp_msg *mp_msg, const
> void
> > *peer)
> > +{
> > + struct rte_mp_msg mp_res;
> > + struct rte_cryptodev_mp_param *res =
> > + (struct rte_cryptodev_mp_param *)mp_res.param;
> > + const struct rte_cryptodev_mp_param *param =
> > + (const struct rte_cryptodev_mp_param *)mp_msg->param;
> > +
> > + int ret;
> > + struct rte_cryptodev *dev;
> > + uint16_t *qps_in_used_by_pid;
> > + int dev_id = param->dev_id;
> > + int qp_id = param->qp_id;
> > + struct rte_cryptodev_qp_conf *queue_conf = param->queue_conf;
> > +
> > + res->result = -EINVAL;
> > + if (!rte_cryptodev_is_valid_dev(dev_id)) {
> > + CDEV_LOG_ERR("Invalid dev_id=%" PRIu8, dev_id);
> > + goto out;
> > + }
> > +
> > + if (!rte_cryptodev_get_qp_status(dev_id, qp_id))
> > + goto out;
> > +
> > + dev = &rte_crypto_devices[dev_id];
> > + qps_in_used_by_pid = dev->data->qp_in_use_by_pid;
> > +
> > + switch (param->type) {
> > + case RTE_CRYPTODEV_MP_REQ_QP_SET:
> > + ret = rte_cryptodev_queue_pair_setup(dev_id, qp_id,
> > + queue_conf, param->socket_id);
> > + if (!ret)
> > + qps_in_used_by_pid[qp_id] = param->process_id;
> > + res->result = ret;
> > + break;
> > + case RTE_CRYPTODEV_MP_REQ_QP_FREE:
> > + if ((rte_eal_process_type() == RTE_PROC_SECONDARY) &&
> > + (qps_in_used_by_pid[qp_id] != param->process_id)) {
> > + CDEV_LOG_ERR("Unable to release qp_id=%" PRIu8,
> > qp_id);
> > + goto out;
> > + }
> > +
> > + ret = (*dev->dev_ops->queue_pair_release)(dev, qp_id);
> > + if (!ret)
> > + qps_in_used_by_pid[qp_id] = 0;
> > +
> > + res->result = ret;
> > + break;
> > + default:
> > + CDEV_LOG_ERR("invalid mp request type\n");
> > + }
> > +
> > +out:
> > + ret = rte_mp_reply(&mp_res, peer);
> > + return ret;
> > +}
> > +
> > +int rte_cryptodev_mp_request_register(void)
> > +{
> > + RTE_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
> > + return rte_mp_action_register(CRYPTODEV_MP_REQ,
> > + rte_cryptodev_ipc_request);
> > +}
> > +
> > +void rte_cryptodev_mp_request_unregister(void)
> > +{
> > + RTE_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
> > + rte_mp_action_unregister(CRYPTODEV_MP_REQ);
> > +}
> > +
>
> It looks an incomplete patch right now.
> - Documentation is missing on how this feature will be used.
[KJ] v4 updates release notes and commit message for usage of this feature.
> - Test cases?
[KJ]We have a standalone multi-process test case that I can submit but it needs some rework so it probably cant be included in the next revision. I can submit it anyway and we can do a revision of it later, after discussion. Ideally, the test app should be integrate into l2fwd test app.
> - PMD implementation is key part for this feature to work.
[KJ] My understanding is that this design should not impact to any PMDs but offers an alternative way for secondary process to configure the queue-pair. Its up to primary process to call the IPC request register function, otherwise no impact to any cryptodev multi-process functionalities.
>
> I believe it is better to defer this to next release as we don't have a clarity on
> how to use this.
[KJ] We would still like to get this feature into the release since we have a customer already using a version of this patchset. We will work to address your concerns/questions and resubmit.
>
next prev parent reply other threads:[~2022-10-06 0:57 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-07-26 23:08 [dpdk-dev v1] " Kai Ji
2022-07-27 4:25 ` [EXT] " Akhil Goyal
2022-08-05 8:51 ` Zhang, Roy Fan
2022-08-08 7:43 ` Akhil Goyal
2022-08-12 8:06 ` Zhang, Roy Fan
2022-08-12 8:25 ` Akhil Goyal
2022-09-21 18:37 ` Akhil Goyal
2022-10-02 1:43 ` [dpdk-dev v2] " Kai Ji
2022-10-02 18:57 ` [EXT] " Akhil Goyal
2022-10-02 22:44 ` [dpdk-dev v3 1/1] " Kai Ji
2022-10-03 16:39 ` Power, Ciara
2022-10-04 18:12 ` [EXT] " Akhil Goyal
2022-10-06 0:57 ` Ji, Kai [this message]
2022-10-06 8:16 ` [dpdk-dev v4] " Kai Ji
2022-10-06 16:19 ` Power, Ciara
2022-10-06 17:06 ` [dpdk-dev v5] " Kai Ji
2022-10-06 18:49 ` [EXT] " Akhil Goyal
2022-10-06 23:11 ` Ji, Kai
2022-10-07 9:37 ` Zhang, Fan
2022-10-06 22:41 ` Konstantin Ananyev
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DM6PR11MB3403AE2F2B85435E29AADD14815C9@DM6PR11MB3403.namprd11.prod.outlook.com \
--to=kai.ji@intel.com \
--cc=dev@dpdk.org \
--cc=gakhil@marvell.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).