Re: [dpdk-dev] [PATCH v3] eventdev: update crypto adapter metadata structures

["Gujjar, Abhinandan S" <abhinandan.gujjar@intel.com>]

Hi Shijith,

> -----Original Message-----
> From: Shijith Thotton <sthotton@marvell.com>
> Sent: Tuesday, September 7, 2021 4:07 PM
> To: Gujjar, Abhinandan S <abhinandan.gujjar@intel.com>; dev@dpdk.org
> Cc: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Anoob Joseph
> <anoobj@marvell.com>; Pavan Nikhilesh Bhagavatula
> <pbhagavatula@marvell.com>; Akhil Goyal <gakhil@marvell.com>; Ray
> Kinsella <mdr@ashroe.eu>; Ankur Dwivedi <adwivedi@marvell.com>
> Subject: RE: [PATCH v3] eventdev: update crypto adapter metadata
> structures
> 
> Hi Abhinandan,
> 
> >> In crypto adapter metadata, reserved bytes in request info structure
> >> is a space holder for response info. It enforces an order of
> >> operation if the structures are updated using memcpy to avoid
> >> overwriting response info. It is logical to move the reserved space
> >> out of request info. It also solves the ordering issue mentioned before.
> >I would like to understand what kind of ordering issue you have faced
> >with the current approach. Could you please give an example/sequence
> and explain?
> >
> 
> I have seen this issue with crypto adapter autotest (#n215).
> 
> Example:
> rte_memcpy(&m_data.response_info, &response_info,
> sizeof(response_info)); rte_memcpy(&m_data.request_info,
> &request_info, sizeof(request_info));
> 
> Here response info is getting overwritten by request info.
> Above lines can reordered to fix the issue, but can be ignored with this patch.
There is a reason for designing the metadata in this way.
Right now, sizeof (union rte_event_crypto_metadata) is 16 bytes.
So, the session based case needs just 16 bytes to store the data.
Whereas, for sessionless case each crypto_ops requires another 16 bytes.

By changing the struct in the following way you are doubling the memory requirement.
With the changes, for sessionless case, each crypto op requires 32 bytes of space instead of 16 bytes and the mempool will be bigger.
This will have the perf impact too!

You can just copy the individual members(cdev_id & queue_pair_id) after the response_info.
OR You have a better way?

> 
> >>
> >> This patch removes the reserve field from request info and makes
> >> event crypto metadata type to structure from union to make space for
> >> response info.
> >>
> >> App and drivers are updated as per metadata change.
> >>
> >> Signed-off-by: Shijith Thotton <sthotton@marvell.com>
> >> Acked-by: Anoob Joseph <anoobj@marvell.com>
> >> ---
> >> v3:
> >> * Updated ABI section of release notes.
> >>
> >> v2:
> >> * Updated deprecation notice.
> >>
> >> v1:
> >> * Rebased.
> >>
> >>  app/test/test_event_crypto_adapter.c              | 14 +++++++-------
> >>  doc/guides/rel_notes/deprecation.rst              |  6 ------
> >>  doc/guides/rel_notes/release_21_11.rst            |  2 ++
> >>  drivers/crypto/octeontx/otx_cryptodev_ops.c       |  8 ++++----
> >>  drivers/crypto/octeontx2/otx2_cryptodev_ops.c     |  4 ++--
> >>  .../event/octeontx2/otx2_evdev_crypto_adptr_tx.h  |  4 ++--
> >>  lib/eventdev/rte_event_crypto_adapter.c           |  8 ++++----
> >>  lib/eventdev/rte_event_crypto_adapter.h           | 15 +++++----------
> >>  8 files changed, 26 insertions(+), 35 deletions(-)
> >>
> >> diff --git a/app/test/test_event_crypto_adapter.c
> >> b/app/test/test_event_crypto_adapter.c
> >> index 3ad20921e2..0d73694d3a 100644
> >> --- a/app/test/test_event_crypto_adapter.c
> >> +++ b/app/test/test_event_crypto_adapter.c
> >> @@ -168,7 +168,7 @@ test_op_forward_mode(uint8_t session_less)  {
> >>  	struct rte_crypto_sym_xform cipher_xform;
> >>  	struct rte_cryptodev_sym_session *sess;
> >> -	union rte_event_crypto_metadata m_data;
> >> +	struct rte_event_crypto_metadata m_data;
> >>  	struct rte_crypto_sym_op *sym_op;
> >>  	struct rte_crypto_op *op;
> >>  	struct rte_mbuf *m;
> >> @@ -368,7 +368,7 @@ test_op_new_mode(uint8_t session_less)  {
> >>  	struct rte_crypto_sym_xform cipher_xform;
> >>  	struct rte_cryptodev_sym_session *sess;
> >> -	union rte_event_crypto_metadata m_data;
> >> +	struct rte_event_crypto_metadata m_data;
> >>  	struct rte_crypto_sym_op *sym_op;
> >>  	struct rte_crypto_op *op;
> >>  	struct rte_mbuf *m;
> >> @@ -406,7 +406,7 @@ test_op_new_mode(uint8_t session_less)
> >>  		if (cap &
> >> RTE_EVENT_CRYPTO_ADAPTER_CAP_SESSION_PRIVATE_DATA) {
> >>  			/* Fill in private user data information */
> >>  			rte_memcpy(&m_data.response_info,
> &response_info,
> >> -				   sizeof(m_data));
> >> +				   sizeof(response_info));
> >>  			rte_cryptodev_sym_session_set_user_data(sess,
> >>  						&m_data, sizeof(m_data));
> >>  		}
> >> @@ -426,7 +426,7 @@ test_op_new_mode(uint8_t session_less)
> >>  		op->private_data_offset = len;
> >>  		/* Fill in private data information */
> >>  		rte_memcpy(&m_data.response_info, &response_info,
> >> -			   sizeof(m_data));
> >> +			   sizeof(response_info));
> >>  		rte_memcpy((uint8_t *)op + len, &m_data, sizeof(m_data));
> >>  	}
> >>
> >> @@ -519,7 +519,7 @@ configure_cryptodev(void)
> >>  			DEFAULT_NUM_XFORMS *
> >>  			sizeof(struct rte_crypto_sym_xform) +
> >>  			MAXIMUM_IV_LENGTH +
> >> -			sizeof(union rte_event_crypto_metadata),
> >> +			sizeof(struct rte_event_crypto_metadata),
> >>  			rte_socket_id());
> >>  	if (params.op_mpool == NULL) {
> >>  		RTE_LOG(ERR, USER1, "Can't create CRYPTO_OP_POOL\n");
> @@ -549,12
> >> +549,12 @@ configure_cryptodev(void)
> >>  	 * to include the session headers & private data
> >>  	 */
> >>  	session_size =
> >> rte_cryptodev_sym_get_private_session_size(TEST_CDEV_ID);
> >> -	session_size += sizeof(union rte_event_crypto_metadata);
> >> +	session_size += sizeof(struct rte_event_crypto_metadata);
> >>
> >>  	params.session_mpool = rte_cryptodev_sym_session_pool_create(
> >>  			"CRYPTO_ADAPTER_SESSION_MP",
> >>  			MAX_NB_SESSIONS, 0, 0,
> >> -			sizeof(union rte_event_crypto_metadata),
> >> +			sizeof(struct rte_event_crypto_metadata),
> >>  			SOCKET_ID_ANY);
> >>  	TEST_ASSERT_NOT_NULL(params.session_mpool,
> >>  			"session mempool allocation failed\n"); diff --git
> >> a/doc/guides/rel_notes/deprecation.rst
> >> b/doc/guides/rel_notes/deprecation.rst
> >> index 76a4abfd6b..58ee95c020 100644
> >> --- a/doc/guides/rel_notes/deprecation.rst
> >> +++ b/doc/guides/rel_notes/deprecation.rst
> >> @@ -266,12 +266,6 @@ Deprecation Notices
> >>    values to the function ``rte_event_eth_rx_adapter_queue_add`` using
> >>    the structure ``rte_event_eth_rx_adapter_queue_add``.
> >>
> >> -* eventdev: Reserved bytes of ``rte_event_crypto_request`` is a
> >> space holder
> >> -  for ``response_info``. Both should be decoupled for better clarity.
> >> -  New space for ``response_info`` can be made by changing
> >> -  ``rte_event_crypto_metadata`` type to structure from union.
> >> -  This change is targeted for DPDK 21.11.
> >> -
> >>  * metrics: The function ``rte_metrics_init`` will have a non-void return
> >>    in order to notify errors instead of calling ``rte_exit``.
> >>
> >> diff --git a/doc/guides/rel_notes/release_21_11.rst
> >> b/doc/guides/rel_notes/release_21_11.rst
> >> index d707a554ef..ab76d5dd55 100644
> >> --- a/doc/guides/rel_notes/release_21_11.rst
> >> +++ b/doc/guides/rel_notes/release_21_11.rst
> >> @@ -100,6 +100,8 @@ ABI Changes
> >>     Also, make sure to start the actual text at the margin.
> >>
> =======================================================
> >>
> >> +* eventdev: Modified type of ``union rte_event_crypto_metadata`` to
> >> +struct and
> >> +  removed reserved bytes from ``struct rte_event_crypto_request``.
> >>
> >>  Known Issues
> >>  ------------
> >> diff --git a/drivers/crypto/octeontx/otx_cryptodev_ops.c
> >> b/drivers/crypto/octeontx/otx_cryptodev_ops.c
> >> index eac6796cfb..c51be63146 100644
> >> --- a/drivers/crypto/octeontx/otx_cryptodev_ops.c
> >> +++ b/drivers/crypto/octeontx/otx_cryptodev_ops.c
> >> @@ -710,17 +710,17 @@ submit_request_to_sso(struct ssows *ws,
> >> uintptr_t req,
> >>  	ssovf_store_pair(add_work, req, ws->grps[rsp_info->queue_id]);  }
> >>
> >> -static inline union rte_event_crypto_metadata *
> >> +static inline struct rte_event_crypto_metadata *
> >>  get_event_crypto_mdata(struct rte_crypto_op *op)  {
> >> -	union rte_event_crypto_metadata *ec_mdata;
> >> +	struct rte_event_crypto_metadata *ec_mdata;
> >>
> >>  	if (op->sess_type == RTE_CRYPTO_OP_WITH_SESSION)
> >>  		ec_mdata = rte_cryptodev_sym_session_get_user_data(
> >>  							   op->sym->session);
> >>  	else if (op->sess_type == RTE_CRYPTO_OP_SESSIONLESS &&
> >>  		 op->private_data_offset)
> >> -		ec_mdata = (union rte_event_crypto_metadata *)
> >> +		ec_mdata = (struct rte_event_crypto_metadata *)
> >>  			((uint8_t *)op + op->private_data_offset);
> >>  	else
> >>  		return NULL;
> >> @@ -731,7 +731,7 @@ get_event_crypto_mdata(struct rte_crypto_op
> *op)
> >> uint16_t __rte_hot  otx_crypto_adapter_enqueue(void *port, struct
> >> rte_crypto_op *op)  {
> >> -	union rte_event_crypto_metadata *ec_mdata;
> >> +	struct rte_event_crypto_metadata *ec_mdata;
> >>  	struct cpt_instance *instance;
> >>  	struct cpt_request_info *req;
> >>  	struct rte_event *rsp_info;
> >> diff --git a/drivers/crypto/octeontx2/otx2_cryptodev_ops.c
> >> b/drivers/crypto/octeontx2/otx2_cryptodev_ops.c
> >> index 42100154cd..952d1352f4 100644
> >> --- a/drivers/crypto/octeontx2/otx2_cryptodev_ops.c
> >> +++ b/drivers/crypto/octeontx2/otx2_cryptodev_ops.c
> >> @@ -453,7 +453,7 @@ otx2_ca_enqueue_req(const struct otx2_cpt_qp
> *qp,
> >>  		    struct rte_crypto_op *op,
> >>  		    uint64_t cpt_inst_w7)
> >>  {
> >> -	union rte_event_crypto_metadata *m_data;
> >> +	struct rte_event_crypto_metadata *m_data;
> >>  	union cpt_inst_s inst;
> >>  	uint64_t lmt_status;
> >>
> >> @@ -468,7 +468,7 @@ otx2_ca_enqueue_req(const struct otx2_cpt_qp
> *qp,
> >>  		}
> >>  	} else if (op->sess_type == RTE_CRYPTO_OP_SESSIONLESS &&
> >>  		   op->private_data_offset) {
> >> -		m_data = (union rte_event_crypto_metadata *)
> >> +		m_data = (struct rte_event_crypto_metadata *)
> >>  			 ((uint8_t *)op +
> >>  			  op->private_data_offset);
> >>  	} else {
> >> diff --git a/drivers/event/octeontx2/otx2_evdev_crypto_adptr_tx.h
> >> b/drivers/event/octeontx2/otx2_evdev_crypto_adptr_tx.h
> >> index ecf7eb9f56..458e8306d7 100644
> >> --- a/drivers/event/octeontx2/otx2_evdev_crypto_adptr_tx.h
> >> +++ b/drivers/event/octeontx2/otx2_evdev_crypto_adptr_tx.h
> >> @@ -16,7 +16,7 @@
> >>  static inline uint16_t
> >>  otx2_ca_enq(uintptr_t tag_op, const struct rte_event *ev)  {
> >> -	union rte_event_crypto_metadata *m_data;
> >> +	struct rte_event_crypto_metadata *m_data;
> >>  	struct rte_crypto_op *crypto_op;
> >>  	struct rte_cryptodev *cdev;
> >>  	struct otx2_cpt_qp *qp;
> >> @@ -37,7 +37,7 @@ otx2_ca_enq(uintptr_t tag_op, const struct
> >> rte_event
> >> *ev)
> >>  		qp_id = m_data->request_info.queue_pair_id;
> >>  	} else if (crypto_op->sess_type == RTE_CRYPTO_OP_SESSIONLESS
> &&
> >>  		   crypto_op->private_data_offset) {
> >> -		m_data = (union rte_event_crypto_metadata *)
> >> +		m_data = (struct rte_event_crypto_metadata *)
> >>  			 ((uint8_t *)crypto_op +
> >>  			  crypto_op->private_data_offset);
> >>  		cdev_id = m_data->request_info.cdev_id; diff --git
> >> a/lib/eventdev/rte_event_crypto_adapter.c
> >> b/lib/eventdev/rte_event_crypto_adapter.c
> >> index e1d38d383d..6977391ae9 100644
> >> --- a/lib/eventdev/rte_event_crypto_adapter.c
> >> +++ b/lib/eventdev/rte_event_crypto_adapter.c
> >> @@ -333,7 +333,7 @@ eca_enq_to_cryptodev(struct
> >> rte_event_crypto_adapter *adapter,
> >>  		 struct rte_event *ev, unsigned int cnt)  {
> >>  	struct rte_event_crypto_adapter_stats *stats = &adapter-
> >> >crypto_stats;
> >> -	union rte_event_crypto_metadata *m_data = NULL;
> >> +	struct rte_event_crypto_metadata *m_data = NULL;
> >>  	struct crypto_queue_pair_info *qp_info = NULL;
> >>  	struct rte_crypto_op *crypto_op;
> >>  	unsigned int i, n;
> >> @@ -371,7 +371,7 @@ eca_enq_to_cryptodev(struct
> >> rte_event_crypto_adapter *adapter,
> >>  			len++;
> >>  		} else if (crypto_op->sess_type ==
> RTE_CRYPTO_OP_SESSIONLESS &&
> >>  				crypto_op->private_data_offset) {
> >> -			m_data = (union rte_event_crypto_metadata *)
> >> +			m_data = (struct rte_event_crypto_metadata *)
> >>  				 ((uint8_t *)crypto_op +
> >>  					crypto_op->private_data_offset);
> >>  			cdev_id = m_data->request_info.cdev_id; @@ -
> >> 504,7 +504,7 @@ eca_ops_enqueue_burst(struct
> rte_event_crypto_adapter
> >> *adapter,
> >>  		  struct rte_crypto_op **ops, uint16_t num)  {
> >>  	struct rte_event_crypto_adapter_stats *stats = &adapter-
> >> >crypto_stats;
> >> -	union rte_event_crypto_metadata *m_data = NULL;
> >> +	struct rte_event_crypto_metadata *m_data = NULL;
> >>  	uint8_t event_dev_id = adapter->eventdev_id;
> >>  	uint8_t event_port_id = adapter->event_port_id;
> >>  	struct rte_event events[BATCH_SIZE]; @@ -523,7 +523,7 @@
> >> eca_ops_enqueue_burst(struct rte_event_crypto_adapter *adapter,
> >>  					ops[i]->sym->session);
> >>  		} else if (ops[i]->sess_type ==
> >> RTE_CRYPTO_OP_SESSIONLESS &&
> >>  				ops[i]->private_data_offset) {
> >> -			m_data = (union rte_event_crypto_metadata *)
> >> +			m_data = (struct rte_event_crypto_metadata *)
> >>  				 ((uint8_t *)ops[i] +
> >>  				  ops[i]->private_data_offset);
> >>  		}
> >> diff --git a/lib/eventdev/rte_event_crypto_adapter.h
> >> b/lib/eventdev/rte_event_crypto_adapter.h
> >> index f8c6cca87c..3c24d9d9df 100644
> >> --- a/lib/eventdev/rte_event_crypto_adapter.h
> >> +++ b/lib/eventdev/rte_event_crypto_adapter.h
> >> @@ -200,11 +200,6 @@ enum rte_event_crypto_adapter_mode {
> >>   * provide event request information to the adapter.
> >>   */
> >>  struct rte_event_crypto_request {
> >> -	uint8_t resv[8];
> >> -	/**< Overlaps with first 8 bytes of struct rte_event
> >> -	 * that encode the response event information. Application
> >> -	 * is expected to fill in struct rte_event response_info.
> >> -	 */
> >>  	uint16_t cdev_id;
> >>  	/**< cryptodev ID to be used */
> >>  	uint16_t queue_pair_id;
> >> @@ -223,16 +218,16 @@ struct rte_event_crypto_request {
> >>   * operation. If the transfer is done by SW, event response information
> >>   * will be used by the adapter.
> >>   */
> >> -union rte_event_crypto_metadata {
> >> -	struct rte_event_crypto_request request_info;
> >> -	/**< Request information to be filled in by application
> >> -	 * for RTE_EVENT_CRYPTO_ADAPTER_OP_FORWARD mode.
> >> -	 */
> >> +struct rte_event_crypto_metadata {
> >>  	struct rte_event response_info;
> >>  	/**< Response information to be filled in by application
> >>  	 * for RTE_EVENT_CRYPTO_ADAPTER_OP_NEW and
> >>  	 * RTE_EVENT_CRYPTO_ADAPTER_OP_FORWARD mode.
> >>  	 */
> >> +	struct rte_event_crypto_request request_info;
> >> +	/**< Request information to be filled in by application
> >> +	 * for RTE_EVENT_CRYPTO_ADAPTER_OP_FORWARD mode.
> >> +	 */
> >>  };
> >>
> >>  /**
> >> --
> >> 2.25.1