[dpdk-dev] [PATCH 0/8] cryptodev: hide internal strutures

[dpdk-dev] [PATCH 0/8] cryptodev: hide internal strutures

[Akhil Goyal]

Structures rte_cryptodev and rte_cryptodev_data are not
supposed to be directly used by the application. These
are made public as they are used by inline datapath
public APIs.
This patchset, creates a new rte_cryptodev_core.h file
which helps in defining a data structure to hold datapath
APIs in a flat array based on the device identifier which
is filled by the PMD.
The patchset updates all crypto PMDs to use the new
framework and hence the internal structs are made hidden.

Similar series for ethdev and eventdev are also floated on ML.
http://patches.dpdk.org/project/dpdk/list/?series=18382
http://patches.dpdk.org/project/dpdk/list/?series=18422


Akhil Goyal (8):
  cryptodev: separate out internal structures
  cryptodev: move inline APIs into separate structure
  cryptodev: add helper functions for new datapath interface
  cryptodev: use new API for datapath functions
  drivers/crypto: use new framework for datapath
  crypto/scheduler: rename enq-deq functions
  crypto/scheduler: update for new datapath framework
  cryptodev: move device specific structures

 drivers/crypto/aesni_gcm/aesni_gcm_pmd.c      |  10 +-
 drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c    |  11 +-
 drivers/crypto/armv8/rte_armv8_pmd.c          |  11 +-
 drivers/crypto/bcmfs/bcmfs_sym_pmd.c          |  11 +-
 drivers/crypto/caam_jr/caam_jr.c              |  11 +-
 drivers/crypto/ccp/ccp_dev.c                  |   1 +
 drivers/crypto/ccp/rte_ccp_pmd.c              |  11 +-
 drivers/crypto/cnxk/cn10k_cryptodev_ops.c     |   8 +-
 drivers/crypto/cnxk/cn10k_cryptodev_ops.h     |   3 +
 drivers/crypto/cnxk/cn10k_ipsec.c             |   1 +
 drivers/crypto/cnxk/cn9k_cryptodev_ops.c      |   9 +-
 drivers/crypto/cnxk/cn9k_cryptodev_ops.h      |   3 +
 .../crypto/cnxk/cnxk_cryptodev_capabilities.c |   1 +
 drivers/crypto/cnxk/cnxk_cryptodev_sec.c      |   1 +
 drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c   |  12 +-
 drivers/crypto/dpaa_sec/dpaa_sec.c            |  11 +-
 drivers/crypto/kasumi/rte_kasumi_pmd.c        |  11 +-
 drivers/crypto/mlx5/mlx5_crypto.c             |  11 +-
 drivers/crypto/mvsam/rte_mrvl_pmd.c           |  11 +-
 drivers/crypto/nitrox/nitrox_sym.c            |  11 +-
 drivers/crypto/nitrox/nitrox_sym_reqmgr.c     |   1 +
 drivers/crypto/null/null_crypto_pmd.c         |  11 +-
 .../crypto/octeontx/otx_cryptodev_hw_access.c |   1 +
 drivers/crypto/octeontx/otx_cryptodev_ops.c   |  16 +-
 drivers/crypto/octeontx/otx_cryptodev_ops.h   |   5 +
 .../crypto/octeontx2/otx2_cryptodev_mbox.c    |   1 +
 drivers/crypto/octeontx2/otx2_cryptodev_ops.c |  11 +-
 drivers/crypto/openssl/rte_openssl_pmd.c      |  11 +-
 drivers/crypto/qat/qat_asym_pmd.c             |  11 +-
 drivers/crypto/qat/qat_sym_pmd.c              |  10 +-
 drivers/crypto/scheduler/scheduler_failover.c |  35 +-
 .../crypto/scheduler/scheduler_multicore.c    |  32 +-
 .../scheduler/scheduler_pkt_size_distr.c      |  34 +-
 .../crypto/scheduler/scheduler_roundrobin.c   |  34 +-
 drivers/crypto/snow3g/rte_snow3g_pmd.c        |  11 +-
 drivers/crypto/virtio/virtio_cryptodev.c      |  10 +-
 drivers/crypto/virtio/virtio_cryptodev.h      |   2 +
 drivers/crypto/virtio/virtio_rxtx.c           |   2 +
 drivers/crypto/zuc/rte_zuc_pmd.c              |  11 +-
 lib/cryptodev/cryptodev_pmd.c                 |  33 ++
 lib/cryptodev/cryptodev_pmd.h                 | 323 +++++++++++++++++-
 lib/cryptodev/meson.build                     |   4 +-
 lib/cryptodev/rte_cryptodev.c                 |  43 ++-
 lib/cryptodev/rte_cryptodev.h                 | 308 +++++------------
 lib/cryptodev/rte_cryptodev_core.h            |  43 +++
 lib/cryptodev/version.map                     |   8 +
 46 files changed, 854 insertions(+), 316 deletions(-)
 create mode 100644 lib/cryptodev/rte_cryptodev_core.h

-- 
2.25.1

[dpdk-dev] [PATCH 1/8] cryptodev: separate out internal structures

[Akhil Goyal]

A new header file rte_cryptodev_core.h is added and all
internal data structures which need not be exposed directly to
application are moved to this file. These structures are mostly
used by drivers, but they need to be in the public header file
as they are accessed by datapath inline functions for
performance reasons.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
---
 lib/cryptodev/cryptodev_pmd.h      |   6 -
 lib/cryptodev/meson.build          |   4 +-
 lib/cryptodev/rte_cryptodev.h      | 360 ++++++++++++-----------------
 lib/cryptodev/rte_cryptodev_core.h | 100 ++++++++
 4 files changed, 245 insertions(+), 225 deletions(-)
 create mode 100644 lib/cryptodev/rte_cryptodev_core.h

diff --git a/lib/cryptodev/cryptodev_pmd.h b/lib/cryptodev/cryptodev_pmd.h
index ec7bb82be8..f775ba6beb 100644
--- a/lib/cryptodev/cryptodev_pmd.h
+++ b/lib/cryptodev/cryptodev_pmd.h
@@ -96,12 +96,6 @@ __rte_internal
 struct rte_cryptodev *
 rte_cryptodev_pmd_get_named_dev(const char *name);
 
-/**
- * The pool of rte_cryptodev structures.
- */
-extern struct rte_cryptodev *rte_cryptodevs;
-
-
 /**
  * Definitions of all functions exported by a driver through the
  * the generic structure of type *crypto_dev_ops* supplied in the
diff --git a/lib/cryptodev/meson.build b/lib/cryptodev/meson.build
index 735935df4a..f32cc62a78 100644
--- a/lib/cryptodev/meson.build
+++ b/lib/cryptodev/meson.build
@@ -14,7 +14,9 @@ headers = files(
         'rte_crypto_sym.h',
         'rte_crypto_asym.h',
 )
-
+indirect_headers += files(
+        'rte_cryptodev_core.h',
+)
 driver_sdk_headers += files(
         'cryptodev_pmd.h',
 )
diff --git a/lib/cryptodev/rte_cryptodev.h b/lib/cryptodev/rte_cryptodev.h
index 33aac44446..3d99dd1cf5 100644
--- a/lib/cryptodev/rte_cryptodev.h
+++ b/lib/cryptodev/rte_cryptodev.h
@@ -861,17 +861,6 @@ rte_cryptodev_callback_unregister(uint8_t dev_id,
 		enum rte_cryptodev_event_type event,
 		rte_cryptodev_cb_fn cb_fn, void *cb_arg);
 
-typedef uint16_t (*dequeue_pkt_burst_t)(void *qp,
-		struct rte_crypto_op **ops,	uint16_t nb_ops);
-/**< Dequeue processed packets from queue pair of a device. */
-
-typedef uint16_t (*enqueue_pkt_burst_t)(void *qp,
-		struct rte_crypto_op **ops,	uint16_t nb_ops);
-/**< Enqueue packets for processing on queue pair of a device. */
-
-
-
-
 struct rte_cryptodev_callback;
 
 /** Structure to keep track of registered callbacks */
@@ -901,216 +890,9 @@ struct rte_cryptodev_cb_rcu {
 	/**< RCU QSBR variable per queue pair */
 };
 
-/** The data structure associated with each crypto device. */
-struct rte_cryptodev {
-	dequeue_pkt_burst_t dequeue_burst;
-	/**< Pointer to PMD receive function. */
-	enqueue_pkt_burst_t enqueue_burst;
-	/**< Pointer to PMD transmit function. */
-
-	struct rte_cryptodev_data *data;
-	/**< Pointer to device data */
-	struct rte_cryptodev_ops *dev_ops;
-	/**< Functions exported by PMD */
-	uint64_t feature_flags;
-	/**< Feature flags exposes HW/SW features for the given device */
-	struct rte_device *device;
-	/**< Backing device */
-
-	uint8_t driver_id;
-	/**< Crypto driver identifier*/
-
-	struct rte_cryptodev_cb_list link_intr_cbs;
-	/**< User application callback for interrupts if present */
-
-	void *security_ctx;
-	/**< Context for security ops */
-
-	__extension__
-	uint8_t attached : 1;
-	/**< Flag indicating the device is attached */
-
-	struct rte_cryptodev_cb_rcu *enq_cbs;
-	/**< User application callback for pre enqueue processing */
-
-	struct rte_cryptodev_cb_rcu *deq_cbs;
-	/**< User application callback for post dequeue processing */
-} __rte_cache_aligned;
-
 void *
 rte_cryptodev_get_sec_ctx(uint8_t dev_id);
 
-/**
- *
- * The data part, with no function pointers, associated with each device.
- *
- * This structure is safe to place in shared memory to be common among
- * different processes in a multi-process configuration.
- */
-struct rte_cryptodev_data {
-	uint8_t dev_id;
-	/**< Device ID for this instance */
-	uint8_t socket_id;
-	/**< Socket ID where memory is allocated */
-	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
-	/**< Unique identifier name */
-
-	__extension__
-	uint8_t dev_started : 1;
-	/**< Device state: STARTED(1)/STOPPED(0) */
-
-	struct rte_mempool *session_pool;
-	/**< Session memory pool */
-	void **queue_pairs;
-	/**< Array of pointers to queue pairs. */
-	uint16_t nb_queue_pairs;
-	/**< Number of device queue pairs. */
-
-	void *dev_private;
-	/**< PMD-specific private data */
-} __rte_cache_aligned;
-
-extern struct rte_cryptodev *rte_cryptodevs;
-/**
- *
- * Dequeue a burst of processed crypto operations from a queue on the crypto
- * device. The dequeued operation are stored in *rte_crypto_op* structures
- * whose pointers are supplied in the *ops* array.
- *
- * The rte_cryptodev_dequeue_burst() function returns the number of ops
- * actually dequeued, which is the number of *rte_crypto_op* data structures
- * effectively supplied into the *ops* array.
- *
- * A return value equal to *nb_ops* indicates that the queue contained
- * at least *nb_ops* operations, and this is likely to signify that other
- * processed operations remain in the devices output queue. Applications
- * implementing a "retrieve as many processed operations as possible" policy
- * can check this specific case and keep invoking the
- * rte_cryptodev_dequeue_burst() function until a value less than
- * *nb_ops* is returned.
- *
- * The rte_cryptodev_dequeue_burst() function does not provide any error
- * notification to avoid the corresponding overhead.
- *
- * @param	dev_id		The symmetric crypto device identifier
- * @param	qp_id		The index of the queue pair from which to
- *				retrieve processed packets. The value must be
- *				in the range [0, nb_queue_pair - 1] previously
- *				supplied to rte_cryptodev_configure().
- * @param	ops		The address of an array of pointers to
- *				*rte_crypto_op* structures that must be
- *				large enough to store *nb_ops* pointers in it.
- * @param	nb_ops		The maximum number of operations to dequeue.
- *
- * @return
- *   - The number of operations actually dequeued, which is the number
- *   of pointers to *rte_crypto_op* structures effectively supplied to the
- *   *ops* array.
- */
-static inline uint16_t
-rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
-		struct rte_crypto_op **ops, uint16_t nb_ops)
-{
-	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
-
-	rte_cryptodev_trace_dequeue_burst(dev_id, qp_id, (void **)ops, nb_ops);
-	nb_ops = (*dev->dequeue_burst)
-			(dev->data->queue_pairs[qp_id], ops, nb_ops);
-#ifdef RTE_CRYPTO_CALLBACKS
-	if (unlikely(dev->deq_cbs != NULL)) {
-		struct rte_cryptodev_cb_rcu *list;
-		struct rte_cryptodev_cb *cb;
-
-		/* __ATOMIC_RELEASE memory order was used when the
-		 * call back was inserted into the list.
-		 * Since there is a clear dependency between loading
-		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
-		 * not required.
-		 */
-		list = &dev->deq_cbs[qp_id];
-		rte_rcu_qsbr_thread_online(list->qsbr, 0);
-		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
-
-		while (cb != NULL) {
-			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
-					cb->arg);
-			cb = cb->next;
-		};
-
-		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
-	}
-#endif
-	return nb_ops;
-}
-
-/**
- * Enqueue a burst of operations for processing on a crypto device.
- *
- * The rte_cryptodev_enqueue_burst() function is invoked to place
- * crypto operations on the queue *qp_id* of the device designated by
- * its *dev_id*.
- *
- * The *nb_ops* parameter is the number of operations to process which are
- * supplied in the *ops* array of *rte_crypto_op* structures.
- *
- * The rte_cryptodev_enqueue_burst() function returns the number of
- * operations it actually enqueued for processing. A return value equal to
- * *nb_ops* means that all packets have been enqueued.
- *
- * @param	dev_id		The identifier of the device.
- * @param	qp_id		The index of the queue pair which packets are
- *				to be enqueued for processing. The value
- *				must be in the range [0, nb_queue_pairs - 1]
- *				previously supplied to
- *				 *rte_cryptodev_configure*.
- * @param	ops		The address of an array of *nb_ops* pointers
- *				to *rte_crypto_op* structures which contain
- *				the crypto operations to be processed.
- * @param	nb_ops		The number of operations to process.
- *
- * @return
- * The number of operations actually enqueued on the crypto device. The return
- * value can be less than the value of the *nb_ops* parameter when the
- * crypto devices queue is full or if invalid parameters are specified in
- * a *rte_crypto_op*.
- */
-static inline uint16_t
-rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
-		struct rte_crypto_op **ops, uint16_t nb_ops)
-{
-	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
-
-#ifdef RTE_CRYPTO_CALLBACKS
-	if (unlikely(dev->enq_cbs != NULL)) {
-		struct rte_cryptodev_cb_rcu *list;
-		struct rte_cryptodev_cb *cb;
-
-		/* __ATOMIC_RELEASE memory order was used when the
-		 * call back was inserted into the list.
-		 * Since there is a clear dependency between loading
-		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
-		 * not required.
-		 */
-		list = &dev->enq_cbs[qp_id];
-		rte_rcu_qsbr_thread_online(list->qsbr, 0);
-		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
-
-		while (cb != NULL) {
-			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
-					cb->arg);
-			cb = cb->next;
-		};
-
-		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
-	}
-#endif
-
-	rte_cryptodev_trace_enqueue_burst(dev_id, qp_id, (void **)ops, nb_ops);
-	return (*dev->enqueue_burst)(
-			dev->data->queue_pairs[qp_id], ops, nb_ops);
-}
-
-
 /** Cryptodev symmetric crypto session
  * Each session is derived from a fixed xform chain. Therefore each session
  * has a fixed algo, key, op-type, digest_len etc.
@@ -1997,6 +1779,148 @@ int rte_cryptodev_remove_deq_callback(uint8_t dev_id,
 				      uint16_t qp_id,
 				      struct rte_cryptodev_cb *cb);
 
+#include <rte_cryptodev_core.h>
+/**
+ *
+ * Dequeue a burst of processed crypto operations from a queue on the crypto
+ * device. The dequeued operation are stored in *rte_crypto_op* structures
+ * whose pointers are supplied in the *ops* array.
+ *
+ * The rte_cryptodev_dequeue_burst() function returns the number of ops
+ * actually dequeued, which is the number of *rte_crypto_op* data structures
+ * effectively supplied into the *ops* array.
+ *
+ * A return value equal to *nb_ops* indicates that the queue contained
+ * at least *nb_ops* operations, and this is likely to signify that other
+ * processed operations remain in the devices output queue. Applications
+ * implementing a "retrieve as many processed operations as possible" policy
+ * can check this specific case and keep invoking the
+ * rte_cryptodev_dequeue_burst() function until a value less than
+ * *nb_ops* is returned.
+ *
+ * The rte_cryptodev_dequeue_burst() function does not provide any error
+ * notification to avoid the corresponding overhead.
+ *
+ * @param	dev_id		The symmetric crypto device identifier
+ * @param	qp_id		The index of the queue pair from which to
+ *				retrieve processed packets. The value must be
+ *				in the range [0, nb_queue_pair - 1] previously
+ *				supplied to rte_cryptodev_configure().
+ * @param	ops		The address of an array of pointers to
+ *				*rte_crypto_op* structures that must be
+ *				large enough to store *nb_ops* pointers in it.
+ * @param	nb_ops		The maximum number of operations to dequeue.
+ *
+ * @return
+ *   - The number of operations actually dequeued, which is the number
+ *   of pointers to *rte_crypto_op* structures effectively supplied to the
+ *   *ops* array.
+ */
+static inline uint16_t
+rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_crypto_op **ops, uint16_t nb_ops)
+{
+	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
+
+	rte_cryptodev_trace_dequeue_burst(dev_id, qp_id, (void **)ops, nb_ops);
+	nb_ops = (*dev->dequeue_burst)
+			(dev->data->queue_pairs[qp_id], ops, nb_ops);
+#ifdef RTE_CRYPTO_CALLBACKS
+	if (unlikely(dev->deq_cbs != NULL)) {
+		struct rte_cryptodev_cb_rcu *list;
+		struct rte_cryptodev_cb *cb;
+
+		/* __ATOMIC_RELEASE memory order was used when the
+		 * call back was inserted into the list.
+		 * Since there is a clear dependency between loading
+		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
+		 * not required.
+		 */
+		list = &dev->deq_cbs[qp_id];
+		rte_rcu_qsbr_thread_online(list->qsbr, 0);
+		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
+
+		while (cb != NULL) {
+			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
+					cb->arg);
+			cb = cb->next;
+		};
+
+		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
+	}
+#endif
+	return nb_ops;
+}
+
+/**
+ * Enqueue a burst of operations for processing on a crypto device.
+ *
+ * The rte_cryptodev_enqueue_burst() function is invoked to place
+ * crypto operations on the queue *qp_id* of the device designated by
+ * its *dev_id*.
+ *
+ * The *nb_ops* parameter is the number of operations to process which are
+ * supplied in the *ops* array of *rte_crypto_op* structures.
+ *
+ * The rte_cryptodev_enqueue_burst() function returns the number of
+ * operations it actually enqueued for processing. A return value equal to
+ * *nb_ops* means that all packets have been enqueued.
+ *
+ * @param	dev_id		The identifier of the device.
+ * @param	qp_id		The index of the queue pair which packets are
+ *				to be enqueued for processing. The value
+ *				must be in the range [0, nb_queue_pairs - 1]
+ *				previously supplied to
+ *				 *rte_cryptodev_configure*.
+ * @param	ops		The address of an array of *nb_ops* pointers
+ *				to *rte_crypto_op* structures which contain
+ *				the crypto operations to be processed.
+ * @param	nb_ops		The number of operations to process.
+ *
+ * @return
+ * The number of operations actually enqueued on the crypto device. The return
+ * value can be less than the value of the *nb_ops* parameter when the
+ * crypto devices queue is full or if invalid parameters are specified in
+ * a *rte_crypto_op*.
+ */
+static inline uint16_t
+rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_crypto_op **ops, uint16_t nb_ops)
+{
+	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
+
+#ifdef RTE_CRYPTO_CALLBACKS
+	if (unlikely(dev->enq_cbs != NULL)) {
+		struct rte_cryptodev_cb_rcu *list;
+		struct rte_cryptodev_cb *cb;
+
+		/* __ATOMIC_RELEASE memory order was used when the
+		 * call back was inserted into the list.
+		 * Since there is a clear dependency between loading
+		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
+		 * not required.
+		 */
+		list = &dev->enq_cbs[qp_id];
+		rte_rcu_qsbr_thread_online(list->qsbr, 0);
+		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
+
+		while (cb != NULL) {
+			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
+					cb->arg);
+			cb = cb->next;
+		};
+
+		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
+	}
+#endif
+
+	rte_cryptodev_trace_enqueue_burst(dev_id, qp_id, (void **)ops, nb_ops);
+	return (*dev->enqueue_burst)(
+			dev->data->queue_pairs[qp_id], ops, nb_ops);
+}
+
+
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/cryptodev/rte_cryptodev_core.h b/lib/cryptodev/rte_cryptodev_core.h
new file mode 100644
index 0000000000..1633e55889
--- /dev/null
+++ b/lib/cryptodev/rte_cryptodev_core.h
@@ -0,0 +1,100 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2021 Marvell.
+ */
+
+#ifndef _RTE_CRYPTODEV_CORE_H_
+#define _RTE_CRYPTODEV_CORE_H_
+
+/**
+ * @file
+ *
+ * RTE Crypto Device internal header.
+ *
+ * This header contains internal data types. But they are still part of the
+ * public API because they are used by inline functions in the published API.
+ *
+ * Applications should not use these directly.
+ *
+ */
+
+typedef uint16_t (*dequeue_pkt_burst_t)(void *qp,
+		struct rte_crypto_op **ops,	uint16_t nb_ops);
+/**< Dequeue processed packets from queue pair of a device. */
+
+typedef uint16_t (*enqueue_pkt_burst_t)(void *qp,
+		struct rte_crypto_op **ops,	uint16_t nb_ops);
+/**< Enqueue packets for processing on queue pair of a device. */
+
+/**
+ * @internal
+ * The data part, with no function pointers, associated with each device.
+ *
+ * This structure is safe to place in shared memory to be common among
+ * different processes in a multi-process configuration.
+ */
+struct rte_cryptodev_data {
+	uint8_t dev_id;
+	/**< Device ID for this instance */
+	uint8_t socket_id;
+	/**< Socket ID where memory is allocated */
+	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
+	/**< Unique identifier name */
+
+	__extension__
+	uint8_t dev_started : 1;
+	/**< Device state: STARTED(1)/STOPPED(0) */
+
+	struct rte_mempool *session_pool;
+	/**< Session memory pool */
+	void **queue_pairs;
+	/**< Array of pointers to queue pairs. */
+	uint16_t nb_queue_pairs;
+	/**< Number of device queue pairs. */
+
+	void *dev_private;
+	/**< PMD-specific private data */
+} __rte_cache_aligned;
+
+
+/** @internal The data structure associated with each crypto device. */
+struct rte_cryptodev {
+	dequeue_pkt_burst_t dequeue_burst;
+	/**< Pointer to PMD receive function. */
+	enqueue_pkt_burst_t enqueue_burst;
+	/**< Pointer to PMD transmit function. */
+
+	struct rte_cryptodev_data *data;
+	/**< Pointer to device data */
+	struct rte_cryptodev_ops *dev_ops;
+	/**< Functions exported by PMD */
+	uint64_t feature_flags;
+	/**< Feature flags exposes HW/SW features for the given device */
+	struct rte_device *device;
+	/**< Backing device */
+
+	uint8_t driver_id;
+	/**< Crypto driver identifier*/
+
+	struct rte_cryptodev_cb_list link_intr_cbs;
+	/**< User application callback for interrupts if present */
+
+	void *security_ctx;
+	/**< Context for security ops */
+
+	__extension__
+	uint8_t attached : 1;
+	/**< Flag indicating the device is attached */
+
+	struct rte_cryptodev_cb_rcu *enq_cbs;
+	/**< User application callback for pre enqueue processing */
+
+	struct rte_cryptodev_cb_rcu *deq_cbs;
+	/**< User application callback for post dequeue processing */
+} __rte_cache_aligned;
+
+/**
+ * The pool of rte_cryptodev structures.
+ */
+extern struct rte_cryptodev *rte_cryptodevs;
+
+#endif /* _RTE_CRYPTODEV_CORE_H_ */
-- 
2.25.1

[dpdk-dev] [PATCH 2/8] cryptodev: move inline APIs into separate structure

[Akhil Goyal]

Move fastpath inline function pointers from rte_cryptodev into a
separate structure accessed via a flat array.
The intension is to make rte_cryptodev and related structures private
to avoid future API/ABI breakages.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
---
 lib/cryptodev/cryptodev_pmd.c      | 33 ++++++++++++++++++++++++++++++
 lib/cryptodev/cryptodev_pmd.h      |  9 ++++++++
 lib/cryptodev/rte_cryptodev.c      |  3 +++
 lib/cryptodev/rte_cryptodev_core.h | 19 +++++++++++++++++
 lib/cryptodev/version.map          |  4 ++++
 5 files changed, 68 insertions(+)

diff --git a/lib/cryptodev/cryptodev_pmd.c b/lib/cryptodev/cryptodev_pmd.c
index 71e34140cd..46772dc355 100644
--- a/lib/cryptodev/cryptodev_pmd.c
+++ b/lib/cryptodev/cryptodev_pmd.c
@@ -158,3 +158,36 @@ rte_cryptodev_pmd_destroy(struct rte_cryptodev *cryptodev)
 
 	return 0;
 }
+
+static uint16_t
+dummy_crypto_enqueue_burst(__rte_unused uint8_t dev_id,
+			  __rte_unused uint8_t qp_id,
+			  __rte_unused struct rte_crypto_op **ops,
+			  __rte_unused uint16_t nb_ops)
+{
+	CDEV_LOG_ERR(
+		"crypto enqueue burst requested for unconfigured crypto device");
+	return 0;
+}
+
+static uint16_t
+dummy_crypto_dequeue_burst(__rte_unused uint8_t dev_id,
+			  __rte_unused uint8_t qp_id,
+			  __rte_unused struct rte_crypto_op **ops,
+			  __rte_unused uint16_t nb_ops)
+{
+	CDEV_LOG_ERR(
+		"crypto enqueue burst requested for unconfigured crypto device");
+	return 0;
+}
+
+void
+rte_cryptodev_api_reset(struct rte_cryptodev_api *api)
+{
+	static const struct rte_cryptodev_api dummy = {
+		.enqueue_burst = dummy_crypto_enqueue_burst,
+		.dequeue_burst = dummy_crypto_dequeue_burst,
+	};
+
+	*api = dummy;
+}
diff --git a/lib/cryptodev/cryptodev_pmd.h b/lib/cryptodev/cryptodev_pmd.h
index f775ba6beb..eeaea13a23 100644
--- a/lib/cryptodev/cryptodev_pmd.h
+++ b/lib/cryptodev/cryptodev_pmd.h
@@ -520,6 +520,15 @@ RTE_INIT(init_ ##driver_id)\
 	driver_id = rte_cryptodev_allocate_driver(&crypto_drv, &(drv));\
 }
 
+/**
+ * Reset crypto device fastpath APIs to dummy values.
+ *
+ * @param The *api* pointer to reset.
+ */
+__rte_internal
+void
+rte_cryptodev_api_reset(struct rte_cryptodev_api *api);
+
 static inline void *
 get_sym_session_private_data(const struct rte_cryptodev_sym_session *sess,
 		uint8_t driver_id) {
diff --git a/lib/cryptodev/rte_cryptodev.c b/lib/cryptodev/rte_cryptodev.c
index 9fa3aff1d3..26f8390668 100644
--- a/lib/cryptodev/rte_cryptodev.c
+++ b/lib/cryptodev/rte_cryptodev.c
@@ -54,6 +54,9 @@ static struct rte_cryptodev_global cryptodev_globals = {
 		.nb_devs		= 0
 };
 
+/* Public fastpath APIs. */
+struct rte_cryptodev_api *rte_cryptodev_api;
+
 /* spinlock for crypto device callbacks */
 static rte_spinlock_t rte_cryptodev_cb_lock = RTE_SPINLOCK_INITIALIZER;
 
diff --git a/lib/cryptodev/rte_cryptodev_core.h b/lib/cryptodev/rte_cryptodev_core.h
index 1633e55889..ec38f70e0c 100644
--- a/lib/cryptodev/rte_cryptodev_core.h
+++ b/lib/cryptodev/rte_cryptodev_core.h
@@ -25,6 +25,25 @@ typedef uint16_t (*enqueue_pkt_burst_t)(void *qp,
 		struct rte_crypto_op **ops,	uint16_t nb_ops);
 /**< Enqueue packets for processing on queue pair of a device. */
 
+typedef uint16_t (*rte_crypto_dequeue_burst_t)(uint8_t dev_id, uint8_t qp_id,
+					      struct rte_crypto_op **ops,
+					      uint16_t nb_ops);
+/**< @internal Dequeue processed packets from queue pair of a device. */
+typedef uint16_t (*rte_crypto_enqueue_burst_t)(uint8_t dev_id, uint8_t qp_id,
+					      struct rte_crypto_op **ops,
+					      uint16_t nb_ops);
+/**< @internal Enqueue packets for processing on queue pair of a device. */
+
+struct rte_cryptodev_api {
+	rte_crypto_enqueue_burst_t enqueue_burst;
+	/**< PMD enqueue burst function. */
+	rte_crypto_dequeue_burst_t dequeue_burst;
+	/**< PMD dequeue burst function. */
+	uintptr_t reserved[6];
+} __rte_cache_aligned;
+
+extern struct rte_cryptodev_api *rte_cryptodev_api;
+
 /**
  * @internal
  * The data part, with no function pointers, associated with each device.
diff --git a/lib/cryptodev/version.map b/lib/cryptodev/version.map
index 2fdf70002d..050089ae55 100644
--- a/lib/cryptodev/version.map
+++ b/lib/cryptodev/version.map
@@ -57,6 +57,9 @@ DPDK_22 {
 	rte_cryptodev_sym_session_init;
 	rte_cryptodevs;
 
+	#added in 21.11
+	rte_cryptodev_api;
+
 	local: *;
 };
 
@@ -114,6 +117,7 @@ INTERNAL {
 	global:
 
 	rte_cryptodev_allocate_driver;
+	rte_cryptodev_api_reset;
 	rte_cryptodev_pmd_allocate;
 	rte_cryptodev_pmd_callback_process;
 	rte_cryptodev_pmd_create;
-- 
2.25.1

[dpdk-dev] [PATCH 3/8] cryptodev: add helper functions for new datapath interface

[Akhil Goyal]

Add helper functions and macros to help drivers to
transition to new datapath interface.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
---
 lib/cryptodev/cryptodev_pmd.h | 246 ++++++++++++++++++++++++++++++++++
 lib/cryptodev/rte_cryptodev.c |  40 +++++-
 lib/cryptodev/version.map     |   4 +
 3 files changed, 289 insertions(+), 1 deletion(-)

diff --git a/lib/cryptodev/cryptodev_pmd.h b/lib/cryptodev/cryptodev_pmd.h
index eeaea13a23..d40e5cee94 100644
--- a/lib/cryptodev/cryptodev_pmd.h
+++ b/lib/cryptodev/cryptodev_pmd.h
@@ -70,6 +70,13 @@ struct cryptodev_driver {
 	const struct rte_driver *driver;
 	uint8_t id;
 };
+/**
+ * @internal
+ * The pool of *rte_cryptodev* structures. The size of the pool
+ * is configured at compile-time in the <rte_cryptodev.c> file.
+ */
+extern struct rte_cryptodev rte_crypto_devices[];
+
 
 /**
  * Get the rte_cryptodev structure device pointer for the device. Assumes a
@@ -529,6 +536,245 @@ __rte_internal
 void
 rte_cryptodev_api_reset(struct rte_cryptodev_api *api);
 
+/**
+ * @internal
+ * Helper routine for cryptodev_dequeue_burst.
+ * Should be called as first thing on entrance to the PMD's
+ * rte_cryptodev_dequeue_burst implementation.
+ * Does necessary checks and returns pointer to cryptodev identifier.
+ *
+ * @param dev_id
+ *  The device identifier of the crypto device.
+ * @param qp_id
+ *  The index of the queue pair from which processed crypto ops will
+ *  be dequeued.
+ *
+ * @return
+ *  Pointer to device queue pair on success or NULL otherwise.
+ */
+__rte_internal
+static inline void *
+_rte_cryptodev_dequeue_prolog(uint8_t dev_id, uint8_t qp_id)
+{
+	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
+
+	return dev->data->queue_pairs[qp_id];
+}
+
+/**
+ * @internal
+ * Helper routine for crypto driver dequeue API.
+ * Should be called at exit from PMD's rte_cryptodev_dequeue_burst
+ * implementation.
+ * Does necessary post-processing - invokes RX callbacks if any, tracing, etc.
+ *
+ * @param dev_id
+ *  The device identifier of the Crypto device.
+ * @param qp_id
+ *  The index of the queue pair from which to retrieve input crypto_ops.
+ * @param ops
+ *   The address of an array of pointers to *rte_crypto_op* structures that
+ *   have been retrieved from the device.
+ * @param nb_ops
+ *   The number of ops that were retrieved from the device.
+ *
+ * @return
+ *  The number of crypto ops effectively supplied to the *ops* array.
+ */
+__rte_internal
+static inline uint16_t
+_rte_cryptodev_dequeue_epilog(uint16_t dev_id, uint16_t qp_id,
+	struct rte_crypto_op **ops, uint16_t nb_ops)
+{
+#ifdef RTE_CRYPTO_CALLBACKS
+	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
+
+	if (unlikely(dev->deq_cbs != NULL)) {
+		struct rte_cryptodev_cb_rcu *list;
+		struct rte_cryptodev_cb *cb;
+
+		/* __ATOMIC_RELEASE memory order was used when the
+		 * call back was inserted into the list.
+		 * Since there is a clear dependency between loading
+		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
+		 * not required.
+		 */
+		list = &dev->deq_cbs[qp_id];
+		rte_rcu_qsbr_thread_online(list->qsbr, 0);
+		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
+
+		while (cb != NULL) {
+			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
+					cb->arg);
+			cb = cb->next;
+		};
+
+		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
+	}
+#endif
+
+	return nb_ops;
+}
+#define _RTE_CRYPTO_DEQ_FUNC(fn)	_rte_crypto_deq_##fn
+
+/**
+ * @internal
+ * Helper macro to create new API wrappers for existing PMD dequeue functions.
+ */
+#define _RTE_CRYPTO_DEQ_PROTO(fn) \
+	uint16_t _RTE_CRYPTO_DEQ_FUNC(fn)(uint8_t dev_id, uint8_t qp_id, \
+			struct rte_crypto_op **ops, uint16_t nb_ops)
+
+/**
+ * @internal
+ * Helper macro to create new API wrappers for existing PMD dequeue functions.
+ */
+#define _RTE_CRYPTO_DEQ_DEF(fn) \
+_RTE_CRYPTO_DEQ_PROTO(fn) \
+{ \
+	void *qp = _rte_cryptodev_dequeue_prolog(dev_id, qp_id); \
+	if (qp == NULL) \
+		return 0; \
+	nb_ops = fn(qp, ops, nb_ops); \
+	return _rte_cryptodev_dequeue_epilog(dev_id, qp_id, ops, nb_ops); \
+}
+
+/**
+ * @internal
+ * Helper routine for cryptodev_enqueue_burst.
+ * Should be called as first thing on entrance to the PMD's
+ * rte_cryptodev_enqueue_burst implementation.
+ * Does necessary checks and returns pointer to cryptodev queue pair.
+ *
+ * @param dev_id
+ *  The device identifier of the crypto device.
+ * @param qp_id
+ *  The index of the queue pair in which packets will be enqueued.
+ * @param ops
+ *   The address of an array of pointers to *rte_crypto_op* structures that
+ *   will be enqueued to the device.
+ * @param nb_ops
+ *   The number of ops that will be sent to the device.
+ *
+ * @return
+ *  Pointer to device queue pair on success or NULL otherwise.
+ */
+__rte_internal
+static inline void *
+_rte_cryptodev_enqueue_prolog(uint8_t dev_id, uint8_t qp_id,
+		struct rte_crypto_op **ops, uint16_t nb_ops)
+{
+	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
+
+#ifdef RTE_CRYPTO_CALLBACKS
+	if (unlikely(dev->enq_cbs != NULL)) {
+		struct rte_cryptodev_cb_rcu *list;
+		struct rte_cryptodev_cb *cb;
+
+		/* __ATOMIC_RELEASE memory order was used when the
+		 * call back was inserted into the list.
+		 * Since there is a clear dependency between loading
+		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
+		 * not required.
+		 */
+		list = &dev->enq_cbs[qp_id];
+		rte_rcu_qsbr_thread_online(list->qsbr, 0);
+		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
+
+		while (cb != NULL) {
+			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
+					cb->arg);
+			cb = cb->next;
+		};
+
+		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
+	}
+#endif
+	return dev->data->queue_pairs[qp_id];
+}
+
+#define _RTE_CRYPTO_ENQ_FUNC(fn)	_rte_crypto_enq_##fn
+
+/**
+ * @internal
+ * Helper macro to create new API wrappers for existing PMD enqueue functions.
+ */
+#define _RTE_CRYPTO_ENQ_PROTO(fn) \
+	uint16_t _RTE_CRYPTO_ENQ_FUNC(fn)(uint8_t dev_id, uint8_t qp_id, \
+			struct rte_crypto_op **ops, uint16_t nb_ops)
+
+/**
+ * @internal
+ * Helper macro to create new API wrappers for existing PMD enqueue functions.
+ */
+#define _RTE_CRYPTO_ENQ_DEF(fn) \
+_RTE_CRYPTO_ENQ_PROTO(fn) \
+{ \
+	void *qp = _rte_cryptodev_enqueue_prolog(dev_id, qp_id, ops, nb_ops); \
+	if (qp == NULL) \
+		return 0; \
+	return fn(qp, ops, nb_ops); \
+}
+
+/**
+ * @internal
+ * Helper routine to get enqueue burst function of a given device.
+ *
+ * @param dev_id
+ *  The device identifier of the Crypto device.
+ *
+ * @return
+ *  The function if valid else NULL
+ */
+__rte_internal
+rte_crypto_enqueue_burst_t
+rte_crypto_get_enq_burst_fn(uint8_t dev_id);
+
+/**
+ * @internal
+ * Helper routine to get dequeue burst function of a given device.
+ *
+ * @param dev_id
+ *  The device identifier of the Crypto device.
+ *
+ * @return
+ *  The function if valid else NULL
+ */
+__rte_internal
+rte_crypto_dequeue_burst_t
+rte_crypto_get_deq_burst_fn(uint8_t dev_id);
+
+/**
+ * @internal
+ * Helper routine to set enqueue burst function of a given device.
+ *
+ * @param dev_id
+ *  The device identifier of the Crypto device.
+ *
+ * @return
+ *  0		Success.
+ *  -EINVAL	Failure if dev_id or fn are in-valid.
+ */
+__rte_internal
+int
+rte_crypto_set_enq_burst_fn(uint8_t dev_id, rte_crypto_enqueue_burst_t fn);
+
+/**
+ * @internal
+ * Helper routine to set dequeue burst function of a given device.
+ *
+ * @param dev_id
+ *  The device identifier of the Crypto device.
+ *
+ * @return
+ *  0		Success.
+ *  -EINVAL	Failure if dev_id or fn are in-valid.
+ */
+__rte_internal
+int
+rte_crypto_set_deq_burst_fn(uint8_t dev_id, rte_crypto_dequeue_burst_t fn);
+
+
 static inline void *
 get_sym_session_private_data(const struct rte_cryptodev_sym_session *sess,
 		uint8_t driver_id) {
diff --git a/lib/cryptodev/rte_cryptodev.c b/lib/cryptodev/rte_cryptodev.c
index 26f8390668..4ab82d21d0 100644
--- a/lib/cryptodev/rte_cryptodev.c
+++ b/lib/cryptodev/rte_cryptodev.c
@@ -44,7 +44,7 @@
 
 static uint8_t nb_drivers;
 
-static struct rte_cryptodev rte_crypto_devices[RTE_CRYPTO_MAX_DEVS];
+struct rte_cryptodev rte_crypto_devices[RTE_CRYPTO_MAX_DEVS];
 
 struct rte_cryptodev *rte_cryptodevs = rte_crypto_devices;
 
@@ -1270,6 +1270,44 @@ rte_cryptodev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
 			socket_id);
 }
 
+rte_crypto_enqueue_burst_t
+rte_crypto_get_enq_burst_fn(uint8_t dev_id)
+{
+	if (dev_id >= RTE_CRYPTO_MAX_DEVS) {
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	return rte_cryptodev_api[dev_id].enqueue_burst;
+}
+
+rte_crypto_dequeue_burst_t
+rte_crypto_get_deq_burst_fn(uint8_t dev_id)
+{
+	if (dev_id >= RTE_CRYPTO_MAX_DEVS) {
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	return rte_cryptodev_api[dev_id].dequeue_burst;
+}
+
+int
+rte_crypto_set_enq_burst_fn(uint8_t dev_id, rte_crypto_enqueue_burst_t fn)
+{
+	if (dev_id >= RTE_CRYPTO_MAX_DEVS || fn == NULL)
+		return -EINVAL;
+	rte_cryptodev_api[dev_id].enqueue_burst = fn;
+	return 0;
+}
+
+int
+rte_crypto_set_deq_burst_fn(uint8_t dev_id, rte_crypto_dequeue_burst_t fn)
+{
+	if (dev_id >= RTE_CRYPTO_MAX_DEVS || fn == NULL)
+		return -EINVAL;
+	rte_cryptodev_api[dev_id].dequeue_burst = fn;
+	return 0;
+}
+
 struct rte_cryptodev_cb *
 rte_cryptodev_add_enq_callback(uint8_t dev_id,
 			       uint16_t qp_id,
diff --git a/lib/cryptodev/version.map b/lib/cryptodev/version.map
index 050089ae55..b64384cc05 100644
--- a/lib/cryptodev/version.map
+++ b/lib/cryptodev/version.map
@@ -116,6 +116,10 @@ EXPERIMENTAL {
 INTERNAL {
 	global:
 
+	rte_crypto_get_deq_burst_fn;
+	rte_crypto_get_enq_burst_fn;
+	rte_crypto_set_deq_burst_fn;
+	rte_crypto_set_enq_burst_fn;
 	rte_cryptodev_allocate_driver;
 	rte_cryptodev_api_reset;
 	rte_cryptodev_pmd_allocate;
-- 
2.25.1

[dpdk-dev] [PATCH 4/8] cryptodev: use new API for datapath functions

[Akhil Goyal]

The datapath inline APIs (rte_cryptodev_enqueue_burst/
rte_cryptodev_dequeue_burst) are updated to use the new
rte_crytodev_api->enqueue_burst/rte_cryptodev_api->dequeue_burst
APIs based on the dev_id

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
---
 lib/cryptodev/rte_cryptodev.h | 62 +++--------------------------------
 1 file changed, 5 insertions(+), 57 deletions(-)

diff --git a/lib/cryptodev/rte_cryptodev.h b/lib/cryptodev/rte_cryptodev.h
index 3d99dd1cf5..49919a43a4 100644
--- a/lib/cryptodev/rte_cryptodev.h
+++ b/lib/cryptodev/rte_cryptodev.h
@@ -1820,36 +1820,10 @@ static inline uint16_t
 rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
 		struct rte_crypto_op **ops, uint16_t nb_ops)
 {
-	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
-
 	rte_cryptodev_trace_dequeue_burst(dev_id, qp_id, (void **)ops, nb_ops);
-	nb_ops = (*dev->dequeue_burst)
-			(dev->data->queue_pairs[qp_id], ops, nb_ops);
-#ifdef RTE_CRYPTO_CALLBACKS
-	if (unlikely(dev->deq_cbs != NULL)) {
-		struct rte_cryptodev_cb_rcu *list;
-		struct rte_cryptodev_cb *cb;
 
-		/* __ATOMIC_RELEASE memory order was used when the
-		 * call back was inserted into the list.
-		 * Since there is a clear dependency between loading
-		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
-		 * not required.
-		 */
-		list = &dev->deq_cbs[qp_id];
-		rte_rcu_qsbr_thread_online(list->qsbr, 0);
-		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
-
-		while (cb != NULL) {
-			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
-					cb->arg);
-			cb = cb->next;
-		};
-
-		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
-	}
-#endif
-	return nb_ops;
+	return rte_cryptodev_api[dev_id].dequeue_burst(
+				dev_id, qp_id, ops, nb_ops);
 }
 
 /**
@@ -1887,36 +1861,10 @@ static inline uint16_t
 rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
 		struct rte_crypto_op **ops, uint16_t nb_ops)
 {
-	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
-
-#ifdef RTE_CRYPTO_CALLBACKS
-	if (unlikely(dev->enq_cbs != NULL)) {
-		struct rte_cryptodev_cb_rcu *list;
-		struct rte_cryptodev_cb *cb;
-
-		/* __ATOMIC_RELEASE memory order was used when the
-		 * call back was inserted into the list.
-		 * Since there is a clear dependency between loading
-		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
-		 * not required.
-		 */
-		list = &dev->enq_cbs[qp_id];
-		rte_rcu_qsbr_thread_online(list->qsbr, 0);
-		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
-
-		while (cb != NULL) {
-			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
-					cb->arg);
-			cb = cb->next;
-		};
-
-		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
-	}
-#endif
-
 	rte_cryptodev_trace_enqueue_burst(dev_id, qp_id, (void **)ops, nb_ops);
-	return (*dev->enqueue_burst)(
-			dev->data->queue_pairs[qp_id], ops, nb_ops);
+
+	return rte_cryptodev_api[dev_id].enqueue_burst(
+				dev_id, qp_id, ops, nb_ops);
 }
 
 
-- 
2.25.1

[dpdk-dev] [PATCH 5/8] drivers/crypto: use new framework for datapath

[Akhil Goyal]

All crypto drivers are updated to use the new API
for all enqueue and dequeue paths.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
---
 drivers/crypto/aesni_gcm/aesni_gcm_pmd.c         | 10 ++++++++--
 drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c       | 11 +++++++++--
 drivers/crypto/armv8/rte_armv8_pmd.c             | 11 +++++++++--
 drivers/crypto/bcmfs/bcmfs_sym_pmd.c             | 11 +++++++++--
 drivers/crypto/caam_jr/caam_jr.c                 | 11 +++++++++--
 drivers/crypto/ccp/ccp_dev.c                     |  1 +
 drivers/crypto/ccp/rte_ccp_pmd.c                 | 11 +++++++++--
 drivers/crypto/cnxk/cn10k_cryptodev_ops.c        |  8 ++++++--
 drivers/crypto/cnxk/cn10k_cryptodev_ops.h        |  3 +++
 drivers/crypto/cnxk/cn10k_ipsec.c                |  1 +
 drivers/crypto/cnxk/cn9k_cryptodev_ops.c         |  9 +++++++--
 drivers/crypto/cnxk/cn9k_cryptodev_ops.h         |  3 +++
 .../crypto/cnxk/cnxk_cryptodev_capabilities.c    |  1 +
 drivers/crypto/cnxk/cnxk_cryptodev_sec.c         |  1 +
 drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c      | 12 ++++++++++--
 drivers/crypto/dpaa_sec/dpaa_sec.c               | 11 +++++++++--
 drivers/crypto/kasumi/rte_kasumi_pmd.c           | 11 +++++++++--
 drivers/crypto/mlx5/mlx5_crypto.c                | 11 +++++++++--
 drivers/crypto/mvsam/rte_mrvl_pmd.c              | 11 +++++++++--
 drivers/crypto/nitrox/nitrox_sym.c               | 11 +++++++++--
 drivers/crypto/nitrox/nitrox_sym_reqmgr.c        |  1 +
 drivers/crypto/null/null_crypto_pmd.c            | 11 +++++++++--
 .../crypto/octeontx/otx_cryptodev_hw_access.c    |  1 +
 drivers/crypto/octeontx/otx_cryptodev_ops.c      | 16 ++++++++++++----
 drivers/crypto/octeontx/otx_cryptodev_ops.h      |  5 +++++
 drivers/crypto/octeontx2/otx2_cryptodev_mbox.c   |  1 +
 drivers/crypto/octeontx2/otx2_cryptodev_ops.c    | 11 +++++++++--
 drivers/crypto/openssl/rte_openssl_pmd.c         | 11 +++++++++--
 drivers/crypto/qat/qat_asym_pmd.c                | 11 +++++++++--
 drivers/crypto/qat/qat_sym_pmd.c                 | 10 ++++++++--
 drivers/crypto/snow3g/rte_snow3g_pmd.c           | 11 +++++++++--
 drivers/crypto/virtio/virtio_cryptodev.c         | 10 ++++++----
 drivers/crypto/virtio/virtio_cryptodev.h         |  2 ++
 drivers/crypto/virtio/virtio_rxtx.c              |  2 ++
 drivers/crypto/zuc/rte_zuc_pmd.c                 | 11 +++++++++--
 35 files changed, 223 insertions(+), 50 deletions(-)

diff --git a/drivers/crypto/aesni_gcm/aesni_gcm_pmd.c b/drivers/crypto/aesni_gcm/aesni_gcm_pmd.c
index 330aad8157..35c89318fe 100644
--- a/drivers/crypto/aesni_gcm/aesni_gcm_pmd.c
+++ b/drivers/crypto/aesni_gcm/aesni_gcm_pmd.c
@@ -14,6 +14,8 @@
 #include "aesni_gcm_pmd_private.h"
 
 static uint8_t cryptodev_driver_id;
+_RTE_CRYPTO_ENQ_PROTO(aesni_gcm_pmd_enqueue_burst);
+_RTE_CRYPTO_DEQ_PROTO(aesni_gcm_pmd_dequeue_burst);
 
 /* setup session handlers */
 static void
@@ -758,6 +760,7 @@ aesni_gcm_pmd_dequeue_burst(void *queue_pair,
 
 	return i;
 }
+_RTE_CRYPTO_DEQ_DEF(aesni_gcm_pmd_dequeue_burst)
 
 static uint16_t
 aesni_gcm_pmd_enqueue_burst(void *queue_pair,
@@ -773,6 +776,7 @@ aesni_gcm_pmd_enqueue_burst(void *queue_pair,
 
 	return nb_enqueued;
 }
+_RTE_CRYPTO_ENQ_DEF(aesni_gcm_pmd_enqueue_burst)
 
 static int aesni_gcm_remove(struct rte_vdev_device *vdev);
 
@@ -807,8 +811,10 @@ aesni_gcm_create(const char *name,
 	dev->dev_ops = rte_aesni_gcm_pmd_ops;
 
 	/* register rx/tx burst functions for data path */
-	dev->dequeue_burst = aesni_gcm_pmd_dequeue_burst;
-	dev->enqueue_burst = aesni_gcm_pmd_enqueue_burst;
+	rte_crypto_set_enq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(aesni_gcm_pmd_enqueue_burst));
+	rte_crypto_set_deq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(aesni_gcm_pmd_dequeue_burst));
 
 	dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
 			RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING |
diff --git a/drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c b/drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c
index 60963a8208..bd7a928583 100644
--- a/drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c
+++ b/drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c
@@ -30,6 +30,8 @@ static RTE_DEFINE_PER_LCORE(MB_MGR *, sync_mb_mgr);
 typedef void (*hash_one_block_t)(const void *data, void *digest);
 typedef void (*aes_keyexp_t)(const void *key, void *enc_exp_keys, void *dec_exp_keys);
 
+_RTE_CRYPTO_ENQ_PROTO(aesni_mb_pmd_enqueue_burst);
+_RTE_CRYPTO_DEQ_PROTO(aesni_mb_pmd_dequeue_burst);
 /**
  * Calculate the authentication pre-computes
  *
@@ -1005,6 +1007,7 @@ aesni_mb_pmd_enqueue_burst(void *__qp, struct rte_crypto_op **ops,
 
 	return nb_enqueued;
 }
+_RTE_CRYPTO_ENQ_DEF(aesni_mb_pmd_enqueue_burst)
 
 /** Get multi buffer session */
 static inline struct aesni_mb_session *
@@ -1872,6 +1875,8 @@ aesni_mb_pmd_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
 
 	return processed_jobs;
 }
+_RTE_CRYPTO_DEQ_DEF(aesni_mb_pmd_dequeue_burst)
+
 
 static MB_MGR *
 alloc_init_mb_mgr(enum aesni_mb_vector_mode vector_mode)
@@ -2097,8 +2102,10 @@ cryptodev_aesni_mb_create(const char *name,
 	dev->dev_ops = rte_aesni_mb_pmd_ops;
 
 	/* register rx/tx burst functions for data path */
-	dev->dequeue_burst = aesni_mb_pmd_dequeue_burst;
-	dev->enqueue_burst = aesni_mb_pmd_enqueue_burst;
+	rte_crypto_set_enq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(aesni_mb_pmd_enqueue_burst));
+	rte_crypto_set_deq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(aesni_mb_pmd_dequeue_burst));
 
 	dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
 			RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING |
diff --git a/drivers/crypto/armv8/rte_armv8_pmd.c b/drivers/crypto/armv8/rte_armv8_pmd.c
index 36a1a9bb4f..6a283df3b7 100644
--- a/drivers/crypto/armv8/rte_armv8_pmd.c
+++ b/drivers/crypto/armv8/rte_armv8_pmd.c
@@ -18,6 +18,9 @@
 
 static uint8_t cryptodev_driver_id;
 
+_RTE_CRYPTO_ENQ_PROTO(armv8_crypto_pmd_enqueue_burst);
+_RTE_CRYPTO_DEQ_PROTO(armv8_crypto_pmd_dequeue_burst);
+
 static int cryptodev_armv8_crypto_uninit(struct rte_vdev_device *vdev);
 
 /**
@@ -731,6 +734,7 @@ armv8_crypto_pmd_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	qp->stats.enqueue_err_count++;
 	return retval;
 }
+_RTE_CRYPTO_ENQ_DEF(armv8_crypto_pmd_enqueue_burst)
 
 /** Dequeue burst */
 static uint16_t
@@ -747,6 +751,7 @@ armv8_crypto_pmd_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
 
 	return nb_dequeued;
 }
+_RTE_CRYPTO_DEQ_DEF(armv8_crypto_pmd_dequeue_burst)
 
 /** Create ARMv8 crypto device */
 static int
@@ -789,8 +794,10 @@ cryptodev_armv8_crypto_create(const char *name,
 	dev->dev_ops = rte_armv8_crypto_pmd_ops;
 
 	/* register rx/tx burst functions for data path */
-	dev->dequeue_burst = armv8_crypto_pmd_dequeue_burst;
-	dev->enqueue_burst = armv8_crypto_pmd_enqueue_burst;
+	rte_crypto_set_enq_burst_fn(cryptodev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(armv8_crypto_pmd_enqueue_burst));
+	rte_crypto_set_deq_burst_fn(cryptodev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(armv8_crypto_pmd_dequeue_burst));
 
 	dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
 			RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING |
diff --git a/drivers/crypto/bcmfs/bcmfs_sym_pmd.c b/drivers/crypto/bcmfs/bcmfs_sym_pmd.c
index d1dd22823e..f2f6f53e71 100644
--- a/drivers/crypto/bcmfs/bcmfs_sym_pmd.c
+++ b/drivers/crypto/bcmfs/bcmfs_sym_pmd.c
@@ -19,6 +19,9 @@
 
 uint8_t cryptodev_bcmfs_driver_id;
 
+_RTE_CRYPTO_ENQ_PROTO(bcmfs_sym_pmd_enqueue_op_burst);
+_RTE_CRYPTO_DEQ_PROTO(bcmfs_sym_pmd_dequeue_op_burst);
+
 static int bcmfs_sym_qp_release(struct rte_cryptodev *dev,
 				uint16_t queue_pair_id);
 
@@ -298,6 +301,7 @@ bcmfs_sym_pmd_enqueue_op_burst(void *queue_pair,
 
 	return enq;
 }
+_RTE_CRYPTO_ENQ_DEF(bcmfs_sym_pmd_enqueue_op_burst)
 
 static void bcmfs_sym_set_request_status(struct rte_crypto_op *op,
 					 struct bcmfs_sym_request *out)
@@ -339,6 +343,7 @@ bcmfs_sym_pmd_dequeue_op_burst(void *queue_pair,
 
 	return pkts;
 }
+_RTE_CRYPTO_DEQ_DEF(bcmfs_sym_pmd_dequeue_op_burst)
 
 /*
  * An rte_driver is needed in the registration of both the
@@ -380,8 +385,10 @@ bcmfs_sym_dev_create(struct bcmfs_device *fsdev)
 	cryptodev->driver_id = cryptodev_bcmfs_driver_id;
 	cryptodev->dev_ops = &crypto_bcmfs_ops;
 
-	cryptodev->enqueue_burst = bcmfs_sym_pmd_enqueue_op_burst;
-	cryptodev->dequeue_burst = bcmfs_sym_pmd_dequeue_op_burst;
+	rte_crypto_set_enq_burst_fn(cryptodev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(bcmfs_sym_pmd_enqueue_op_burst));
+	rte_crypto_set_deq_burst_fn(cryptodev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(bcmfs_sym_pmd_dequeue_op_burst));
 
 	cryptodev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
 				   RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING |
diff --git a/drivers/crypto/caam_jr/caam_jr.c b/drivers/crypto/caam_jr/caam_jr.c
index 258750afe7..ffc88de1af 100644
--- a/drivers/crypto/caam_jr/caam_jr.c
+++ b/drivers/crypto/caam_jr/caam_jr.c
@@ -36,6 +36,9 @@
 #define CRYPTODEV_NAME_CAAM_JR_PMD	crypto_caam_jr
 static uint8_t cryptodev_driver_id;
 
+_RTE_CRYPTO_ENQ_PROTO(caam_jr_enqueue_burst);
+_RTE_CRYPTO_DEQ_PROTO(caam_jr_dequeue_burst);
+
 /* Lists the states possible for the SEC user space driver. */
 enum sec_driver_state_e {
 	SEC_DRIVER_STATE_IDLE,		/* Driver not initialized */
@@ -697,6 +700,7 @@ caam_jr_dequeue_burst(void *qp, struct rte_crypto_op **ops,
 
 	return num_rx;
 }
+_RTE_CRYPTO_DEQ_DEF(caam_jr_dequeue_burst)
 
 /**
  * packet looks like:
@@ -1485,6 +1489,7 @@ caam_jr_enqueue_burst(void *qp, struct rte_crypto_op **ops,
 
 	return num_tx;
 }
+_RTE_CRYPTO_ENQ_DEF(caam_jr_enqueue_burst)
 
 /* Release queue pair */
 static int
@@ -2333,8 +2338,10 @@ caam_jr_dev_init(const char *name,
 	dev->dev_ops = &caam_jr_ops;
 
 	/* register rx/tx burst functions for data path */
-	dev->dequeue_burst = caam_jr_dequeue_burst;
-	dev->enqueue_burst = caam_jr_enqueue_burst;
+	rte_crypto_set_enq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(caam_jr_enqueue_burst));
+	rte_crypto_set_deq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(caam_jr_dequeue_burst));
 	dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
 			RTE_CRYPTODEV_FF_HW_ACCELERATED |
 			RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING |
diff --git a/drivers/crypto/ccp/ccp_dev.c b/drivers/crypto/ccp/ccp_dev.c
index 0eb1b0328e..60533cb5fc 100644
--- a/drivers/crypto/ccp/ccp_dev.c
+++ b/drivers/crypto/ccp/ccp_dev.c
@@ -12,6 +12,7 @@
 #include <sys/file.h>
 #include <unistd.h>
 
+#include <cryptodev_pmd.h>
 #include <rte_hexdump.h>
 #include <rte_memzone.h>
 #include <rte_malloc.h>
diff --git a/drivers/crypto/ccp/rte_ccp_pmd.c b/drivers/crypto/ccp/rte_ccp_pmd.c
index a54d81de46..67b880f2ca 100644
--- a/drivers/crypto/ccp/rte_ccp_pmd.c
+++ b/drivers/crypto/ccp/rte_ccp_pmd.c
@@ -34,6 +34,9 @@ struct ccp_pmd_init_params {
 #define CCP_CRYPTODEV_PARAM_MAX_NB_QP		("max_nb_queue_pairs")
 #define CCP_CRYPTODEV_PARAM_AUTH_OPT		("ccp_auth_opt")
 
+_RTE_CRYPTO_ENQ_PROTO(ccp_pmd_enqueue_burst);
+_RTE_CRYPTO_DEQ_PROTO(ccp_pmd_dequeue_burst);
+
 const char *ccp_pmd_valid_params[] = {
 	CCP_CRYPTODEV_PARAM_NAME,
 	CCP_CRYPTODEV_PARAM_SOCKET_ID,
@@ -140,6 +143,7 @@ ccp_pmd_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	qp->qp_stats.enqueued_count += enq_cnt;
 	return enq_cnt;
 }
+_RTE_CRYPTO_ENQ_DEF(ccp_pmd_enqueue_burst)
 
 static uint16_t
 ccp_pmd_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
@@ -176,6 +180,7 @@ ccp_pmd_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
 
 	return nb_dequeued;
 }
+_RTE_CRYPTO_DEQ_DEF(ccp_pmd_dequeue_burst)
 
 /*
  * The set of PCI devices this driver supports
@@ -257,8 +262,10 @@ cryptodev_ccp_create(const char *name,
 
 	/* register rx/tx burst functions for data path */
 	dev->dev_ops = ccp_pmd_ops;
-	dev->enqueue_burst = ccp_pmd_enqueue_burst;
-	dev->dequeue_burst = ccp_pmd_dequeue_burst;
+	rte_crypto_set_enq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(ccp_pmd_enqueue_burst));
+	rte_crypto_set_deq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(ccp_pmd_dequeue_burst));
 
 	dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
 			RTE_CRYPTODEV_FF_HW_ACCELERATED |
diff --git a/drivers/crypto/cnxk/cn10k_cryptodev_ops.c b/drivers/crypto/cnxk/cn10k_cryptodev_ops.c
index d9b53128bc..4081bd778c 100644
--- a/drivers/crypto/cnxk/cn10k_cryptodev_ops.c
+++ b/drivers/crypto/cnxk/cn10k_cryptodev_ops.c
@@ -256,6 +256,7 @@ cn10k_cpt_enqueue_burst(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops)
 
 	return count + i;
 }
+_RTE_CRYPTO_ENQ_DEF(cn10k_cpt_enqueue_burst)
 
 static inline void
 cn10k_cpt_sec_post_process(struct rte_crypto_op *cop,
@@ -414,12 +415,15 @@ cn10k_cpt_dequeue_burst(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops)
 
 	return i;
 }
+_RTE_CRYPTO_DEQ_DEF(cn10k_cpt_dequeue_burst)
 
 void
 cn10k_cpt_set_enqdeq_fns(struct rte_cryptodev *dev)
 {
-	dev->enqueue_burst = cn10k_cpt_enqueue_burst;
-	dev->dequeue_burst = cn10k_cpt_dequeue_burst;
+	rte_crypto_set_enq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(cn10k_cpt_enqueue_burst));
+	rte_crypto_set_deq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(cn10k_cpt_dequeue_burst));
 
 	rte_mb();
 }
diff --git a/drivers/crypto/cnxk/cn10k_cryptodev_ops.h b/drivers/crypto/cnxk/cn10k_cryptodev_ops.h
index 198e9ea5bd..05b30e6d0b 100644
--- a/drivers/crypto/cnxk/cn10k_cryptodev_ops.h
+++ b/drivers/crypto/cnxk/cn10k_cryptodev_ops.h
@@ -10,6 +10,9 @@
 
 extern struct rte_cryptodev_ops cn10k_cpt_ops;
 
+_RTE_CRYPTO_ENQ_PROTO(cn10k_cpt_enqueue_burst);
+_RTE_CRYPTO_DEQ_PROTO(cn10k_cpt_dequeue_burst);
+
 void cn10k_cpt_set_enqdeq_fns(struct rte_cryptodev *dev);
 
 #endif /* _CN10K_CRYPTODEV_OPS_H_ */
diff --git a/drivers/crypto/cnxk/cn10k_ipsec.c b/drivers/crypto/cnxk/cn10k_ipsec.c
index 1d567bf188..c51d5d55c6 100644
--- a/drivers/crypto/cnxk/cn10k_ipsec.c
+++ b/drivers/crypto/cnxk/cn10k_ipsec.c
@@ -9,6 +9,7 @@
 #include <rte_security.h>
 #include <rte_security_driver.h>
 #include <rte_udp.h>
+#include <cryptodev_pmd.h>
 
 #include "cnxk_cryptodev.h"
 #include "cnxk_ipsec.h"
diff --git a/drivers/crypto/cnxk/cn9k_cryptodev_ops.c b/drivers/crypto/cnxk/cn9k_cryptodev_ops.c
index 97fbf780fe..6a15974d6f 100644
--- a/drivers/crypto/cnxk/cn9k_cryptodev_ops.c
+++ b/drivers/crypto/cnxk/cn9k_cryptodev_ops.c
@@ -175,6 +175,7 @@ cn9k_cpt_enqueue_burst(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops)
 
 	return count;
 }
+_RTE_CRYPTO_ENQ_DEF(cn9k_cpt_enqueue_burst)
 
 static inline void
 cn9k_cpt_dequeue_post_process(struct cnxk_cpt_qp *qp, struct rte_crypto_op *cop,
@@ -299,11 +300,15 @@ cn9k_cpt_dequeue_burst(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops)
 
 	return i;
 }
+_RTE_CRYPTO_DEQ_DEF(cn9k_cpt_dequeue_burst)
+
 void
 cn9k_cpt_set_enqdeq_fns(struct rte_cryptodev *dev)
 {
-	dev->enqueue_burst = cn9k_cpt_enqueue_burst;
-	dev->dequeue_burst = cn9k_cpt_dequeue_burst;
+	rte_crypto_set_enq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(cn9k_cpt_enqueue_burst));
+	rte_crypto_set_deq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(cn9k_cpt_dequeue_burst));
 
 	rte_mb();
 }
diff --git a/drivers/crypto/cnxk/cn9k_cryptodev_ops.h b/drivers/crypto/cnxk/cn9k_cryptodev_ops.h
index d042d18474..4e6bfe6971 100644
--- a/drivers/crypto/cnxk/cn9k_cryptodev_ops.h
+++ b/drivers/crypto/cnxk/cn9k_cryptodev_ops.h
@@ -9,6 +9,9 @@
 
 extern struct rte_cryptodev_ops cn9k_cpt_ops;
 
+_RTE_CRYPTO_ENQ_PROTO(cn9k_cpt_enqueue_burst);
+_RTE_CRYPTO_DEQ_PROTO(cn9k_cpt_dequeue_burst);
+
 void cn9k_cpt_set_enqdeq_fns(struct rte_cryptodev *dev);
 
 #endif /* _CN9K_CRYPTODEV_OPS_H_ */
diff --git a/drivers/crypto/cnxk/cnxk_cryptodev_capabilities.c b/drivers/crypto/cnxk/cnxk_cryptodev_capabilities.c
index ab37f9c43b..7db388b3d6 100644
--- a/drivers/crypto/cnxk/cnxk_cryptodev_capabilities.c
+++ b/drivers/crypto/cnxk/cnxk_cryptodev_capabilities.c
@@ -4,6 +4,7 @@
 
 #include <rte_cryptodev.h>
 #include <rte_security.h>
+#include <cryptodev_pmd.h>
 
 #include "roc_api.h"
 
diff --git a/drivers/crypto/cnxk/cnxk_cryptodev_sec.c b/drivers/crypto/cnxk/cnxk_cryptodev_sec.c
index 8d04d4b575..293d1a18fe 100644
--- a/drivers/crypto/cnxk/cnxk_cryptodev_sec.c
+++ b/drivers/crypto/cnxk/cnxk_cryptodev_sec.c
@@ -6,6 +6,7 @@
 #include <rte_malloc.h>
 #include <rte_security.h>
 #include <rte_security_driver.h>
+#include <cryptodev_pmd.h>
 
 #include "cnxk_cryptodev_capabilities.h"
 #include "cnxk_cryptodev_sec.h"
diff --git a/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c b/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c
index bf69c61916..26c00b2c3d 100644
--- a/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c
+++ b/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c
@@ -59,6 +59,9 @@
 
 static uint8_t cryptodev_driver_id;
 
+_RTE_CRYPTO_ENQ_PROTO(dpaa2_sec_enqueue_burst);
+_RTE_CRYPTO_DEQ_PROTO(dpaa2_sec_dequeue_burst);
+
 #ifdef RTE_LIB_SECURITY
 static inline int
 build_proto_compound_sg_fd(dpaa2_sec_session *sess,
@@ -1524,6 +1527,7 @@ dpaa2_sec_enqueue_burst(void *qp, struct rte_crypto_op **ops,
 	dpaa2_qp->tx_vq.err_pkts += nb_ops;
 	return num_tx;
 }
+_RTE_CRYPTO_ENQ_DEF(dpaa2_sec_enqueue_burst)
 
 #ifdef RTE_LIB_SECURITY
 static inline struct rte_crypto_op *
@@ -1727,6 +1731,8 @@ dpaa2_sec_dequeue_burst(void *qp, struct rte_crypto_op **ops,
 	return num_rx;
 }
 
+_RTE_CRYPTO_DEQ_DEF(dpaa2_sec_dequeue_burst)
+
 /** Release queue pair */
 static int
 dpaa2_sec_queue_pair_release(struct rte_cryptodev *dev, uint16_t queue_pair_id)
@@ -3881,8 +3887,10 @@ dpaa2_sec_dev_init(struct rte_cryptodev *cryptodev)
 	cryptodev->driver_id = cryptodev_driver_id;
 	cryptodev->dev_ops = &crypto_ops;
 
-	cryptodev->enqueue_burst = dpaa2_sec_enqueue_burst;
-	cryptodev->dequeue_burst = dpaa2_sec_dequeue_burst;
+	rte_crypto_set_enq_burst_fn(cryptodev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(dpaa2_sec_enqueue_burst));
+	rte_crypto_set_deq_burst_fn(cryptodev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(dpaa2_sec_dequeue_burst));
 	cryptodev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
 			RTE_CRYPTODEV_FF_HW_ACCELERATED |
 			RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING |
diff --git a/drivers/crypto/dpaa_sec/dpaa_sec.c b/drivers/crypto/dpaa_sec/dpaa_sec.c
index 3d53746ef1..6e998c589b 100644
--- a/drivers/crypto/dpaa_sec/dpaa_sec.c
+++ b/drivers/crypto/dpaa_sec/dpaa_sec.c
@@ -47,6 +47,9 @@
 
 static uint8_t cryptodev_driver_id;
 
+_RTE_CRYPTO_ENQ_PROTO(dpaa_sec_enqueue_burst);
+_RTE_CRYPTO_DEQ_PROTO(dpaa_sec_dequeue_burst);
+
 static int
 dpaa_sec_attach_sess_q(struct dpaa_sec_qp *qp, dpaa_sec_session *sess);
 
@@ -1916,6 +1919,7 @@ dpaa_sec_enqueue_burst(void *qp, struct rte_crypto_op **ops,
 
 	return num_tx;
 }
+_RTE_CRYPTO_ENQ_DEF(dpaa_sec_enqueue_burst)
 
 static uint16_t
 dpaa_sec_dequeue_burst(void *qp, struct rte_crypto_op **ops,
@@ -1940,6 +1944,7 @@ dpaa_sec_dequeue_burst(void *qp, struct rte_crypto_op **ops,
 
 	return num_rx;
 }
+_RTE_CRYPTO_DEQ_DEF(dpaa_sec_dequeue_burst)
 
 /** Release queue pair */
 static int
@@ -3365,8 +3370,10 @@ dpaa_sec_dev_init(struct rte_cryptodev *cryptodev)
 	cryptodev->driver_id = cryptodev_driver_id;
 	cryptodev->dev_ops = &crypto_ops;
 
-	cryptodev->enqueue_burst = dpaa_sec_enqueue_burst;
-	cryptodev->dequeue_burst = dpaa_sec_dequeue_burst;
+	rte_crypto_set_enq_burst_fn(cryptodev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(dpaa_sec_enqueue_burst));
+	rte_crypto_set_deq_burst_fn(cryptodev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(dpaa_sec_dequeue_burst));
 	cryptodev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
 			RTE_CRYPTODEV_FF_HW_ACCELERATED |
 			RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING |
diff --git a/drivers/crypto/kasumi/rte_kasumi_pmd.c b/drivers/crypto/kasumi/rte_kasumi_pmd.c
index d6f927417a..9cf79b323e 100644
--- a/drivers/crypto/kasumi/rte_kasumi_pmd.c
+++ b/drivers/crypto/kasumi/rte_kasumi_pmd.c
@@ -19,6 +19,9 @@
 
 static uint8_t cryptodev_driver_id;
 
+_RTE_CRYPTO_ENQ_PROTO(kasumi_pmd_enqueue_burst);
+_RTE_CRYPTO_DEQ_PROTO(kasumi_pmd_dequeue_burst);
+
 /** Get xform chain order. */
 static enum kasumi_operation
 kasumi_get_mode(const struct rte_crypto_sym_xform *xform)
@@ -508,6 +511,7 @@ kasumi_pmd_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	qp->qp_stats.enqueue_err_count += nb_ops - enqueued_ops;
 	return enqueued_ops;
 }
+_RTE_CRYPTO_ENQ_DEF(kasumi_pmd_enqueue_burst)
 
 static uint16_t
 kasumi_pmd_dequeue_burst(void *queue_pair,
@@ -523,6 +527,7 @@ kasumi_pmd_dequeue_burst(void *queue_pair,
 
 	return nb_dequeued;
 }
+_RTE_CRYPTO_DEQ_DEF(kasumi_pmd_dequeue_burst)
 
 static int cryptodev_kasumi_remove(struct rte_vdev_device *vdev);
 
@@ -545,8 +550,10 @@ cryptodev_kasumi_create(const char *name,
 	dev->dev_ops = rte_kasumi_pmd_ops;
 
 	/* Register RX/TX burst functions for data path. */
-	dev->dequeue_burst = kasumi_pmd_dequeue_burst;
-	dev->enqueue_burst = kasumi_pmd_enqueue_burst;
+	rte_crypto_set_enq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(kasumi_pmd_enqueue_burst));
+	rte_crypto_set_deq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(kasumi_pmd_dequeue_burst));
 
 	dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
 			RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING |
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index b3d5200ca3..ec054d0863 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -39,6 +39,9 @@ int mlx5_crypto_logtype;
 
 uint8_t mlx5_crypto_driver_id;
 
+_RTE_CRYPTO_ENQ_PROTO(mlx5_crypto_enqueue_burst);
+_RTE_CRYPTO_DEQ_PROTO(mlx5_crypto_dequeue_burst);
+
 const struct rte_cryptodev_capabilities mlx5_crypto_caps[] = {
 	{		/* AES XTS */
 		.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
@@ -523,6 +526,7 @@ mlx5_crypto_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	rte_wmb();
 	return nb_ops;
 }
+_RTE_CRYPTO_ENQ_DEF(mlx5_crypto_enqueue_burst)
 
 static __rte_noinline void
 mlx5_crypto_cqe_err_handle(struct mlx5_crypto_qp *qp, struct rte_crypto_op *op)
@@ -576,6 +580,7 @@ mlx5_crypto_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	}
 	return i;
 }
+_RTE_CRYPTO_DEQ_DEF(mlx5_crypto_dequeue_burst)
 
 static void
 mlx5_crypto_qp_init(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp)
@@ -1041,8 +1046,10 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
 	DRV_LOG(INFO,
 		"Crypto device %s was created successfully.", ibv->name);
 	crypto_dev->dev_ops = &mlx5_crypto_ops;
-	crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
-	crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
+	rte_crypto_set_enq_burst_fn(crypto_dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(mlx5_crypto_enqueue_burst));
+	rte_crypto_set_deq_burst_fn(crypto_dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(mlx5_crypto_dequeue_burst));
 	crypto_dev->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
 	crypto_dev->driver_id = mlx5_crypto_driver_id;
 	priv = crypto_dev->data->dev_private;
diff --git a/drivers/crypto/mvsam/rte_mrvl_pmd.c b/drivers/crypto/mvsam/rte_mrvl_pmd.c
index a72642a772..7c8f3fdde4 100644
--- a/drivers/crypto/mvsam/rte_mrvl_pmd.c
+++ b/drivers/crypto/mvsam/rte_mrvl_pmd.c
@@ -22,6 +22,9 @@
 
 static uint8_t cryptodev_driver_id;
 
+_RTE_CRYPTO_ENQ_PROTO(mrvl_crypto_pmd_enqueue_burst);
+_RTE_CRYPTO_DEQ_PROTO(mrvl_crypto_pmd_dequeue_burst);
+
 struct mrvl_pmd_init_params {
 	struct rte_cryptodev_pmd_init_params common;
 	uint32_t max_nb_sessions;
@@ -981,6 +984,7 @@ mrvl_crypto_pmd_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	qp->stats.enqueued_count += to_enq_sec + to_enq_crp;
 	return consumed;
 }
+_RTE_CRYPTO_ENQ_DEF(mrvl_crypto_pmd_enqueue_burst)
 
 /**
  * Dequeue burst.
@@ -1046,6 +1050,7 @@ mrvl_crypto_pmd_dequeue_burst(void *queue_pair,
 	qp->stats.dequeued_count += nb_ops;
 	return nb_ops;
 }
+_RTE_CRYPTO_DEQ_DEF(mrvl_crypto_pmd_dequeue_burst)
 
 /**
  * Create a new crypto device.
@@ -1077,8 +1082,10 @@ cryptodev_mrvl_crypto_create(const char *name,
 	dev->dev_ops = rte_mrvl_crypto_pmd_ops;
 
 	/* Register rx/tx burst functions for data path. */
-	dev->enqueue_burst = mrvl_crypto_pmd_enqueue_burst;
-	dev->dequeue_burst = mrvl_crypto_pmd_dequeue_burst;
+	rte_crypto_set_enq_burst_fn(cryptodev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(mrvl_crypto_pmd_enqueue_burst));
+	rte_crypto_set_deq_burst_fn(cryptodev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(mrvl_crypto_pmd_dequeue_burst));
 
 	dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
 			RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING |
diff --git a/drivers/crypto/nitrox/nitrox_sym.c b/drivers/crypto/nitrox/nitrox_sym.c
index f8b7edcd69..291b1e5983 100644
--- a/drivers/crypto/nitrox/nitrox_sym.c
+++ b/drivers/crypto/nitrox/nitrox_sym.c
@@ -68,6 +68,9 @@ static const struct rte_driver nitrox_rte_sym_drv = {
 	.alias = nitrox_sym_drv_name
 };
 
+_RTE_CRYPTO_ENQ_PROTO(nitrox_sym_dev_enq_burst);
+_RTE_CRYPTO_DEQ_PROTO(nitrox_sym_dev_deq_burst);
+
 static int nitrox_sym_dev_qp_release(struct rte_cryptodev *cdev,
 				     uint16_t qp_id);
 
@@ -677,6 +680,7 @@ nitrox_sym_dev_enq_burst(void *queue_pair, struct rte_crypto_op **ops,
 
 	return cnt;
 }
+_RTE_CRYPTO_ENQ_DEF(nitrox_sym_dev_enq_burst)
 
 static int
 nitrox_deq_single_op(struct nitrox_qp *qp, struct rte_crypto_op **op_ptr)
@@ -726,6 +730,7 @@ nitrox_sym_dev_deq_burst(void *queue_pair, struct rte_crypto_op **ops,
 
 	return cnt;
 }
+_RTE_CRYPTO_DEQ_DEF(nitrox_sym_dev_deq_burst)
 
 static struct rte_cryptodev_ops nitrox_cryptodev_ops = {
 	.dev_configure		= nitrox_sym_dev_config,
@@ -769,8 +774,10 @@ nitrox_sym_pmd_create(struct nitrox_device *ndev)
 	ndev->rte_sym_dev.name = cdev->data->name;
 	cdev->driver_id = nitrox_sym_drv_id;
 	cdev->dev_ops = &nitrox_cryptodev_ops;
-	cdev->enqueue_burst = nitrox_sym_dev_enq_burst;
-	cdev->dequeue_burst = nitrox_sym_dev_deq_burst;
+	rte_crypto_set_enq_burst_fn(cdev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(nitrox_sym_dev_enq_burst));
+	rte_crypto_set_deq_burst_fn(cdev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(nitrox_sym_dev_deq_burst));
 	cdev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
 		RTE_CRYPTODEV_FF_HW_ACCELERATED |
 		RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING |
diff --git a/drivers/crypto/nitrox/nitrox_sym_reqmgr.c b/drivers/crypto/nitrox/nitrox_sym_reqmgr.c
index fe3ca25a0c..7a6a7fadfe 100644
--- a/drivers/crypto/nitrox/nitrox_sym_reqmgr.c
+++ b/drivers/crypto/nitrox/nitrox_sym_reqmgr.c
@@ -6,6 +6,7 @@
 #include <rte_cryptodev.h>
 #include <rte_cycles.h>
 #include <rte_errno.h>
+#include <cryptodev_pmd.h>
 
 #include "nitrox_sym_reqmgr.h"
 #include "nitrox_logs.h"
diff --git a/drivers/crypto/null/null_crypto_pmd.c b/drivers/crypto/null/null_crypto_pmd.c
index f9935d52cc..9a4c0fa66e 100644
--- a/drivers/crypto/null/null_crypto_pmd.c
+++ b/drivers/crypto/null/null_crypto_pmd.c
@@ -11,6 +11,9 @@
 
 static uint8_t cryptodev_driver_id;
 
+_RTE_CRYPTO_ENQ_PROTO(null_crypto_pmd_enqueue_burst);
+_RTE_CRYPTO_DEQ_PROTO(null_crypto_pmd_dequeue_burst);
+
 /** verify and set session parameters */
 int
 null_crypto_set_session_parameters(
@@ -137,6 +140,7 @@ null_crypto_pmd_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	qp->qp_stats.enqueue_err_count++;
 	return i;
 }
+_RTE_CRYPTO_ENQ_DEF(null_crypto_pmd_enqueue_burst)
 
 /** Dequeue burst */
 static uint16_t
@@ -153,6 +157,7 @@ null_crypto_pmd_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
 
 	return nb_dequeued;
 }
+_RTE_CRYPTO_DEQ_DEF(null_crypto_pmd_dequeue_burst)
 
 /** Create crypto device */
 static int
@@ -172,8 +177,10 @@ cryptodev_null_create(const char *name,
 	dev->dev_ops = null_crypto_pmd_ops;
 
 	/* register rx/tx burst functions for data path */
-	dev->dequeue_burst = null_crypto_pmd_dequeue_burst;
-	dev->enqueue_burst = null_crypto_pmd_enqueue_burst;
+	rte_crypto_set_enq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(null_crypto_pmd_enqueue_burst));
+	rte_crypto_set_deq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(null_crypto_pmd_dequeue_burst));
 
 	dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
 			RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING |
diff --git a/drivers/crypto/octeontx/otx_cryptodev_hw_access.c b/drivers/crypto/octeontx/otx_cryptodev_hw_access.c
index ab335c6a62..8b5f32821a 100644
--- a/drivers/crypto/octeontx/otx_cryptodev_hw_access.c
+++ b/drivers/crypto/octeontx/otx_cryptodev_hw_access.c
@@ -12,6 +12,7 @@
 #include <rte_mempool.h>
 #include <rte_memzone.h>
 #include <rte_string_fns.h>
+#include <cryptodev_pmd.h>
 
 #include "otx_cryptodev_hw_access.h"
 #include "otx_cryptodev_mbox.h"
diff --git a/drivers/crypto/octeontx/otx_cryptodev_ops.c b/drivers/crypto/octeontx/otx_cryptodev_ops.c
index 9b5bde53f8..9d607f35ad 100644
--- a/drivers/crypto/octeontx/otx_cryptodev_ops.c
+++ b/drivers/crypto/octeontx/otx_cryptodev_ops.c
@@ -687,12 +687,14 @@ otx_cpt_enqueue_asym(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops)
 {
 	return otx_cpt_pkt_enqueue(qptr, ops, nb_ops, OP_TYPE_ASYM);
 }
+_RTE_CRYPTO_ENQ_DEF(otx_cpt_enqueue_asym)
 
 static uint16_t
 otx_cpt_enqueue_sym(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops)
 {
 	return otx_cpt_pkt_enqueue(qptr, ops, nb_ops, OP_TYPE_SYM);
 }
+_RTE_CRYPTO_ENQ_DEF(otx_cpt_enqueue_sym)
 
 static __rte_always_inline void
 submit_request_to_sso(struct ssows *ws, uintptr_t req,
@@ -1019,12 +1021,14 @@ otx_cpt_dequeue_asym(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops)
 {
 	return otx_cpt_pkt_dequeue(qptr, ops, nb_ops, OP_TYPE_ASYM);
 }
+_RTE_CRYPTO_DEQ_DEF(otx_cpt_dequeue_asym)
 
 static uint16_t
 otx_cpt_dequeue_sym(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops)
 {
 	return otx_cpt_pkt_dequeue(qptr, ops, nb_ops, OP_TYPE_SYM);
 }
+_RTE_CRYPTO_DEQ_DEF(otx_cpt_dequeue_sym)
 
 uintptr_t __rte_hot
 otx_crypto_adapter_dequeue(uintptr_t get_work1)
@@ -1151,11 +1155,15 @@ otx_cpt_dev_create(struct rte_cryptodev *c_dev)
 	c_dev->dev_ops = &cptvf_ops;
 
 	if (c_dev->feature_flags & RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO) {
-		c_dev->enqueue_burst = otx_cpt_enqueue_sym;
-		c_dev->dequeue_burst = otx_cpt_dequeue_sym;
+		rte_crypto_set_enq_burst_fn(c_dev->data->dev_id,
+				_RTE_CRYPTO_ENQ_FUNC(otx_cpt_enqueue_sym));
+		rte_crypto_set_deq_burst_fn(c_dev->data->dev_id,
+				_RTE_CRYPTO_DEQ_FUNC(otx_cpt_dequeue_sym));
 	} else {
-		c_dev->enqueue_burst = otx_cpt_enqueue_asym;
-		c_dev->dequeue_burst = otx_cpt_dequeue_asym;
+		rte_crypto_set_enq_burst_fn(c_dev->data->dev_id,
+				_RTE_CRYPTO_ENQ_FUNC(otx_cpt_enqueue_asym));
+		rte_crypto_set_deq_burst_fn(c_dev->data->dev_id,
+				_RTE_CRYPTO_DEQ_FUNC(otx_cpt_dequeue_asym));
 	}
 
 	/* Save dev private data */
diff --git a/drivers/crypto/octeontx/otx_cryptodev_ops.h b/drivers/crypto/octeontx/otx_cryptodev_ops.h
index f234f16970..72ab287b5b 100644
--- a/drivers/crypto/octeontx/otx_cryptodev_ops.h
+++ b/drivers/crypto/octeontx/otx_cryptodev_ops.h
@@ -14,6 +14,11 @@
 int
 otx_cpt_dev_create(struct rte_cryptodev *c_dev);
 
+_RTE_CRYPTO_ENQ_PROTO(otx_cpt_enqueue_sym);
+_RTE_CRYPTO_DEQ_PROTO(otx_cpt_dequeue_sym);
+_RTE_CRYPTO_ENQ_PROTO(otx_cpt_enqueue_asym);
+_RTE_CRYPTO_DEQ_PROTO(otx_cpt_dequeue_asym);
+
 __rte_internal
 uint16_t __rte_hot
 otx_crypto_adapter_enqueue(void *port, struct rte_crypto_op *op);
diff --git a/drivers/crypto/octeontx2/otx2_cryptodev_mbox.c b/drivers/crypto/octeontx2/otx2_cryptodev_mbox.c
index 812515fc1b..263e09879d 100644
--- a/drivers/crypto/octeontx2/otx2_cryptodev_mbox.c
+++ b/drivers/crypto/octeontx2/otx2_cryptodev_mbox.c
@@ -3,6 +3,7 @@
  */
 #include <rte_cryptodev.h>
 #include <rte_ethdev.h>
+#include <cryptodev_pmd.h>
 
 #include "otx2_cryptodev.h"
 #include "otx2_cryptodev_hw_access.h"
diff --git a/drivers/crypto/octeontx2/otx2_cryptodev_ops.c b/drivers/crypto/octeontx2/otx2_cryptodev_ops.c
index 723804347f..a646c8f3ef 100644
--- a/drivers/crypto/octeontx2/otx2_cryptodev_ops.c
+++ b/drivers/crypto/octeontx2/otx2_cryptodev_ops.c
@@ -33,6 +33,9 @@ static uint64_t otx2_fpm_iova[CPT_EC_ID_PMAX];
 
 /* Forward declarations */
 
+_RTE_CRYPTO_ENQ_PROTO(otx2_cpt_enqueue_burst);
+_RTE_CRYPTO_DEQ_PROTO(otx2_cpt_dequeue_burst);
+
 static int
 otx2_cpt_queue_pair_release(struct rte_cryptodev *dev, uint16_t qp_id);
 
@@ -826,6 +829,7 @@ otx2_cpt_enqueue_burst(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops)
 
 	return count;
 }
+_RTE_CRYPTO_ENQ_DEF(otx2_cpt_enqueue_burst)
 
 static __rte_always_inline void
 otx2_cpt_asym_rsa_op(struct rte_crypto_op *cop, struct cpt_request_info *req,
@@ -1096,12 +1100,15 @@ otx2_cpt_dequeue_burst(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops)
 
 	return nb_completed;
 }
+_RTE_CRYPTO_DEQ_DEF(otx2_cpt_dequeue_burst)
 
 void
 otx2_cpt_set_enqdeq_fns(struct rte_cryptodev *dev)
 {
-	dev->enqueue_burst = otx2_cpt_enqueue_burst;
-	dev->dequeue_burst = otx2_cpt_dequeue_burst;
+	rte_crypto_set_enq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(otx2_cpt_enqueue_burst));
+	rte_crypto_set_deq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(otx2_cpt_dequeue_burst));
 
 	rte_mb();
 }
diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c
index f149366c2a..9401072760 100644
--- a/drivers/crypto/openssl/rte_openssl_pmd.c
+++ b/drivers/crypto/openssl/rte_openssl_pmd.c
@@ -20,6 +20,9 @@
 
 static uint8_t cryptodev_driver_id;
 
+_RTE_CRYPTO_ENQ_PROTO(openssl_pmd_enqueue_burst);
+_RTE_CRYPTO_DEQ_PROTO(openssl_pmd_dequeue_burst);
+
 #if (OPENSSL_VERSION_NUMBER < 0x10100000L)
 static HMAC_CTX *HMAC_CTX_new(void)
 {
@@ -2159,6 +2162,7 @@ openssl_pmd_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	qp->stats.enqueue_err_count++;
 	return i;
 }
+_RTE_CRYPTO_ENQ_DEF(openssl_pmd_enqueue_burst)
 
 /** Dequeue burst */
 static uint16_t
@@ -2175,6 +2179,7 @@ openssl_pmd_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
 
 	return nb_dequeued;
 }
+_RTE_CRYPTO_DEQ_DEF(openssl_pmd_dequeue_burst)
 
 /** Create OPENSSL crypto device */
 static int
@@ -2195,8 +2200,10 @@ cryptodev_openssl_create(const char *name,
 	dev->dev_ops = rte_openssl_pmd_ops;
 
 	/* register rx/tx burst functions for data path */
-	dev->dequeue_burst = openssl_pmd_dequeue_burst;
-	dev->enqueue_burst = openssl_pmd_enqueue_burst;
+	rte_crypto_set_enq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(openssl_pmd_enqueue_burst));
+	rte_crypto_set_deq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(openssl_pmd_dequeue_burst));
 
 	dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
 			RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING |
diff --git a/drivers/crypto/qat/qat_asym_pmd.c b/drivers/crypto/qat/qat_asym_pmd.c
index e91bb0d317..cbd7768f2c 100644
--- a/drivers/crypto/qat/qat_asym_pmd.c
+++ b/drivers/crypto/qat/qat_asym_pmd.c
@@ -13,6 +13,9 @@
 
 uint8_t qat_asym_driver_id;
 
+_RTE_CRYPTO_ENQ_PROTO(qat_asym_pmd_enqueue_op_burst);
+_RTE_CRYPTO_DEQ_PROTO(qat_asym_pmd_dequeue_op_burst);
+
 static const struct rte_cryptodev_capabilities qat_gen1_asym_capabilities[] = {
 	QAT_BASE_GEN1_ASYM_CAPABILITIES,
 	RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST()
@@ -214,12 +217,14 @@ uint16_t qat_asym_pmd_enqueue_op_burst(void *qp, struct rte_crypto_op **ops,
 {
 	return qat_enqueue_op_burst(qp, (void **)ops, nb_ops);
 }
+_RTE_CRYPTO_ENQ_DEF(qat_asym_pmd_enqueue_op_burst)
 
 uint16_t qat_asym_pmd_dequeue_op_burst(void *qp, struct rte_crypto_op **ops,
 				       uint16_t nb_ops)
 {
 	return qat_dequeue_op_burst(qp, (void **)ops, nb_ops);
 }
+_RTE_CRYPTO_DEQ_DEF(qat_asym_pmd_dequeue_op_burst)
 
 /* An rte_driver is needed in the registration of both the device and the driver
  * with cryptodev.
@@ -292,8 +297,10 @@ qat_asym_dev_create(struct qat_pci_device *qat_pci_dev,
 	cryptodev->driver_id = qat_asym_driver_id;
 	cryptodev->dev_ops = &crypto_qat_ops;
 
-	cryptodev->enqueue_burst = qat_asym_pmd_enqueue_op_burst;
-	cryptodev->dequeue_burst = qat_asym_pmd_dequeue_op_burst;
+	rte_crypto_set_enq_burst_fn(cryptodev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(qat_asym_pmd_enqueue_op_burst));
+	rte_crypto_set_deq_burst_fn(cryptodev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(qat_asym_pmd_dequeue_op_burst));
 
 	cryptodev->feature_flags = RTE_CRYPTODEV_FF_ASYMMETRIC_CRYPTO |
 			RTE_CRYPTODEV_FF_HW_ACCELERATED |
diff --git a/drivers/crypto/qat/qat_sym_pmd.c b/drivers/crypto/qat/qat_sym_pmd.c
index efda921c05..b3ea51a246 100644
--- a/drivers/crypto/qat/qat_sym_pmd.c
+++ b/drivers/crypto/qat/qat_sym_pmd.c
@@ -20,6 +20,8 @@
 #define MIXED_CRYPTO_MIN_FW_VER 0x04090000
 
 uint8_t qat_sym_driver_id;
+_RTE_CRYPTO_ENQ_PROTO(qat_sym_pmd_enqueue_op_burst);
+_RTE_CRYPTO_DEQ_PROTO(qat_sym_pmd_dequeue_op_burst);
 
 static const struct rte_cryptodev_capabilities qat_gen1_sym_capabilities[] = {
 	QAT_BASE_GEN1_SYM_CAPABILITIES,
@@ -319,6 +321,7 @@ qat_sym_pmd_enqueue_op_burst(void *qp, struct rte_crypto_op **ops,
 {
 	return qat_enqueue_op_burst(qp, (void **)ops, nb_ops);
 }
+_RTE_CRYPTO_ENQ_DEF(qat_sym_pmd_enqueue_op_burst)
 
 static uint16_t
 qat_sym_pmd_dequeue_op_burst(void *qp, struct rte_crypto_op **ops,
@@ -326,6 +329,7 @@ qat_sym_pmd_dequeue_op_burst(void *qp, struct rte_crypto_op **ops,
 {
 	return qat_dequeue_op_burst(qp, (void **)ops, nb_ops);
 }
+_RTE_CRYPTO_DEQ_DEF(qat_sym_pmd_dequeue_op_burst)
 
 /* An rte_driver is needed in the registration of both the device and the driver
  * with cryptodev.
@@ -399,8 +403,10 @@ qat_sym_dev_create(struct qat_pci_device *qat_pci_dev,
 	cryptodev->driver_id = qat_sym_driver_id;
 	cryptodev->dev_ops = &crypto_qat_ops;
 
-	cryptodev->enqueue_burst = qat_sym_pmd_enqueue_op_burst;
-	cryptodev->dequeue_burst = qat_sym_pmd_dequeue_op_burst;
+	rte_crypto_set_enq_burst_fn(cryptodev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(qat_sym_pmd_enqueue_op_burst));
+	rte_crypto_set_deq_burst_fn(cryptodev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(qat_sym_pmd_dequeue_op_burst));
 
 	cryptodev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
 			RTE_CRYPTODEV_FF_HW_ACCELERATED |
diff --git a/drivers/crypto/snow3g/rte_snow3g_pmd.c b/drivers/crypto/snow3g/rte_snow3g_pmd.c
index 8284ac0b66..9df3d66df2 100644
--- a/drivers/crypto/snow3g/rte_snow3g_pmd.c
+++ b/drivers/crypto/snow3g/rte_snow3g_pmd.c
@@ -18,6 +18,9 @@
 
 static uint8_t cryptodev_driver_id;
 
+_RTE_CRYPTO_ENQ_PROTO(snow3g_pmd_enqueue_burst);
+_RTE_CRYPTO_DEQ_PROTO(snow3g_pmd_dequeue_burst);
+
 /** Get xform chain order. */
 static enum snow3g_operation
 snow3g_get_mode(const struct rte_crypto_sym_xform *xform)
@@ -520,6 +523,7 @@ snow3g_pmd_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	qp->qp_stats.enqueue_err_count += nb_ops - enqueued_ops;
 	return enqueued_ops;
 }
+_RTE_CRYPTO_ENQ_DEF(snow3g_pmd_enqueue_burst)
 
 static uint16_t
 snow3g_pmd_dequeue_burst(void *queue_pair,
@@ -535,6 +539,7 @@ snow3g_pmd_dequeue_burst(void *queue_pair,
 
 	return nb_dequeued;
 }
+_RTE_CRYPTO_DEQ_DEF(snow3g_pmd_dequeue_burst)
 
 static int cryptodev_snow3g_remove(struct rte_vdev_device *vdev);
 
@@ -557,8 +562,10 @@ cryptodev_snow3g_create(const char *name,
 	dev->dev_ops = rte_snow3g_pmd_ops;
 
 	/* Register RX/TX burst functions for data path. */
-	dev->dequeue_burst = snow3g_pmd_dequeue_burst;
-	dev->enqueue_burst = snow3g_pmd_enqueue_burst;
+	rte_crypto_set_enq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(snow3g_pmd_enqueue_burst));
+	rte_crypto_set_deq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(snow3g_pmd_dequeue_burst));
 
 	dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
 			RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING |
diff --git a/drivers/crypto/virtio/virtio_cryptodev.c b/drivers/crypto/virtio/virtio_cryptodev.c
index 8faa39df4a..a76d014a35 100644
--- a/drivers/crypto/virtio/virtio_cryptodev.c
+++ b/drivers/crypto/virtio/virtio_cryptodev.c
@@ -731,8 +731,10 @@ crypto_virtio_create(const char *name, struct rte_pci_device *pci_dev,
 	cryptodev->driver_id = cryptodev_virtio_driver_id;
 	cryptodev->dev_ops = &virtio_crypto_dev_ops;
 
-	cryptodev->enqueue_burst = virtio_crypto_pkt_tx_burst;
-	cryptodev->dequeue_burst = virtio_crypto_pkt_rx_burst;
+	rte_crypto_set_enq_burst_fn(cryptodev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(virtio_crypto_pkt_tx_burst));
+	rte_crypto_set_deq_burst_fn(cryptodev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(virtio_crypto_pkt_rx_burst));
 
 	cryptodev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
 		RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING |
@@ -773,8 +775,8 @@ virtio_crypto_dev_uninit(struct rte_cryptodev *cryptodev)
 	}
 
 	cryptodev->dev_ops = NULL;
-	cryptodev->enqueue_burst = NULL;
-	cryptodev->dequeue_burst = NULL;
+	rte_crypto_set_enq_burst_fn(cryptodev->data->dev_id, NULL);
+	rte_crypto_set_deq_burst_fn(cryptodev->data->dev_id, NULL);
 
 	/* release control queue */
 	virtio_crypto_queue_release(hw->cvq);
diff --git a/drivers/crypto/virtio/virtio_cryptodev.h b/drivers/crypto/virtio/virtio_cryptodev.h
index 215bce7863..2ca8c35434 100644
--- a/drivers/crypto/virtio/virtio_cryptodev.h
+++ b/drivers/crypto/virtio/virtio_cryptodev.h
@@ -63,4 +63,6 @@ uint16_t virtio_crypto_pkt_rx_burst(void *tx_queue,
 		struct rte_crypto_op **tx_pkts,
 		uint16_t nb_pkts);
 
+_RTE_CRYPTO_ENQ_PROTO(virtio_crypto_pkt_tx_burst);
+_RTE_CRYPTO_DEQ_PROTO(virtio_crypto_pkt_rx_burst);
 #endif /* _VIRTIO_CRYPTODEV_H_ */
diff --git a/drivers/crypto/virtio/virtio_rxtx.c b/drivers/crypto/virtio/virtio_rxtx.c
index a65524a306..c96bb541a2 100644
--- a/drivers/crypto/virtio/virtio_rxtx.c
+++ b/drivers/crypto/virtio/virtio_rxtx.c
@@ -454,6 +454,7 @@ virtio_crypto_pkt_rx_burst(void *tx_queue, struct rte_crypto_op **rx_pkts,
 
 	return nb_rx;
 }
+_RTE_CRYPTO_DEQ_DEF(virtio_crypto_pkt_rx_burst)
 
 uint16_t
 virtio_crypto_pkt_tx_burst(void *tx_queue, struct rte_crypto_op **tx_pkts,
@@ -525,3 +526,4 @@ virtio_crypto_pkt_tx_burst(void *tx_queue, struct rte_crypto_op **tx_pkts,
 
 	return nb_tx;
 }
+_RTE_CRYPTO_ENQ_DEF(virtio_crypto_pkt_tx_burst)
diff --git a/drivers/crypto/zuc/rte_zuc_pmd.c b/drivers/crypto/zuc/rte_zuc_pmd.c
index d4b343a7af..19d1670dad 100644
--- a/drivers/crypto/zuc/rte_zuc_pmd.c
+++ b/drivers/crypto/zuc/rte_zuc_pmd.c
@@ -16,6 +16,9 @@
 
 static uint8_t cryptodev_driver_id;
 
+_RTE_CRYPTO_ENQ_PROTO(zuc_pmd_enqueue_burst);
+_RTE_CRYPTO_DEQ_PROTO(zuc_pmd_dequeue_burst);
+
 /** Get xform chain order. */
 static enum zuc_operation
 zuc_get_mode(const struct rte_crypto_sym_xform *xform)
@@ -444,6 +447,7 @@ zuc_pmd_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	qp->qp_stats.enqueue_err_count += nb_ops - enqueued_ops;
 	return enqueued_ops;
 }
+_RTE_CRYPTO_ENQ_DEF(zuc_pmd_enqueue_burst)
 
 static uint16_t
 zuc_pmd_dequeue_burst(void *queue_pair,
@@ -459,6 +463,7 @@ zuc_pmd_dequeue_burst(void *queue_pair,
 
 	return nb_dequeued;
 }
+_RTE_CRYPTO_DEQ_DEF(zuc_pmd_dequeue_burst)
 
 static int cryptodev_zuc_remove(struct rte_vdev_device *vdev);
 
@@ -505,8 +510,10 @@ cryptodev_zuc_create(const char *name,
 	dev->dev_ops = rte_zuc_pmd_ops;
 
 	/* Register RX/TX burst functions for data path. */
-	dev->dequeue_burst = zuc_pmd_dequeue_burst;
-	dev->enqueue_burst = zuc_pmd_enqueue_burst;
+	rte_crypto_set_enq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(zuc_pmd_enqueue_burst));
+	rte_crypto_set_deq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(zuc_pmd_dequeue_burst));
 
 	internals = dev->data->dev_private;
 	internals->mb_mgr = mb_mgr;
-- 
2.25.1

[dpdk-dev] [PATCH 6/8] crypto/scheduler: rename enq-deq functions

[Akhil Goyal]

scheduler PMD has 4 variants, which uses same
name for all the enqueue and dequeue functions.
This causes multiple definitions of same function
with the new framework of datapath APIs.
Hence the function names are updated to specify the
the variant it is for.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
---
 drivers/crypto/scheduler/scheduler_failover.c | 20 +++++++++----------
 .../crypto/scheduler/scheduler_multicore.c    | 18 ++++++++---------
 .../scheduler/scheduler_pkt_size_distr.c      | 20 +++++++++----------
 .../crypto/scheduler/scheduler_roundrobin.c   | 20 +++++++++----------
 4 files changed, 39 insertions(+), 39 deletions(-)

diff --git a/drivers/crypto/scheduler/scheduler_failover.c b/drivers/crypto/scheduler/scheduler_failover.c
index 844312dd1b..88cc8f05f7 100644
--- a/drivers/crypto/scheduler/scheduler_failover.c
+++ b/drivers/crypto/scheduler/scheduler_failover.c
@@ -37,7 +37,7 @@ failover_worker_enqueue(struct scheduler_worker *worker,
 }
 
 static uint16_t
-schedule_enqueue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
+schedule_fo_enqueue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 {
 	struct fo_scheduler_qp_ctx *qp_ctx =
 			((struct scheduler_qp_ctx *)qp)->private_qp_ctx;
@@ -60,14 +60,14 @@ schedule_enqueue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 
 
 static uint16_t
-schedule_enqueue_ordering(void *qp, struct rte_crypto_op **ops,
+schedule_fo_enqueue_ordering(void *qp, struct rte_crypto_op **ops,
 		uint16_t nb_ops)
 {
 	struct rte_ring *order_ring =
 			((struct scheduler_qp_ctx *)qp)->order_ring;
 	uint16_t nb_ops_to_enq = get_max_enqueue_order_count(order_ring,
 			nb_ops);
-	uint16_t nb_ops_enqd = schedule_enqueue(qp, ops,
+	uint16_t nb_ops_enqd = schedule_fo_enqueue(qp, ops,
 			nb_ops_to_enq);
 
 	scheduler_order_insert(order_ring, ops, nb_ops_enqd);
@@ -76,7 +76,7 @@ schedule_enqueue_ordering(void *qp, struct rte_crypto_op **ops,
 }
 
 static uint16_t
-schedule_dequeue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
+schedule_fo_dequeue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 {
 	struct fo_scheduler_qp_ctx *qp_ctx =
 			((struct scheduler_qp_ctx *)qp)->private_qp_ctx;
@@ -108,13 +108,13 @@ schedule_dequeue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 }
 
 static uint16_t
-schedule_dequeue_ordering(void *qp, struct rte_crypto_op **ops,
+schedule_fo_dequeue_ordering(void *qp, struct rte_crypto_op **ops,
 		uint16_t nb_ops)
 {
 	struct rte_ring *order_ring =
 			((struct scheduler_qp_ctx *)qp)->order_ring;
 
-	schedule_dequeue(qp, ops, nb_ops);
+	schedule_fo_dequeue(qp, ops, nb_ops);
 
 	return scheduler_order_drain(order_ring, ops, nb_ops);
 }
@@ -145,11 +145,11 @@ scheduler_start(struct rte_cryptodev *dev)
 	}
 
 	if (sched_ctx->reordering_enabled) {
-		dev->enqueue_burst = schedule_enqueue_ordering;
-		dev->dequeue_burst = schedule_dequeue_ordering;
+		dev->enqueue_burst = schedule_fo_enqueue_ordering;
+		dev->dequeue_burst = schedule_fo_dequeue_ordering;
 	} else {
-		dev->enqueue_burst = schedule_enqueue;
-		dev->dequeue_burst = schedule_dequeue;
+		dev->enqueue_burst = schedule_fo_enqueue;
+		dev->dequeue_burst = schedule_fo_dequeue;
 	}
 
 	for (i = 0; i < dev->data->nb_queue_pairs; i++) {
diff --git a/drivers/crypto/scheduler/scheduler_multicore.c b/drivers/crypto/scheduler/scheduler_multicore.c
index 1e2e8dbf9f..bf97343e52 100644
--- a/drivers/crypto/scheduler/scheduler_multicore.c
+++ b/drivers/crypto/scheduler/scheduler_multicore.c
@@ -36,7 +36,7 @@ struct mc_scheduler_qp_ctx {
 };
 
 static uint16_t
-schedule_enqueue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
+schedule_mc_enqueue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 {
 	struct mc_scheduler_qp_ctx *mc_qp_ctx =
 			((struct scheduler_qp_ctx *)qp)->private_qp_ctx;
@@ -64,14 +64,14 @@ schedule_enqueue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 }
 
 static uint16_t
-schedule_enqueue_ordering(void *qp, struct rte_crypto_op **ops,
+schedule_mc_enqueue_ordering(void *qp, struct rte_crypto_op **ops,
 		uint16_t nb_ops)
 {
 	struct rte_ring *order_ring =
 			((struct scheduler_qp_ctx *)qp)->order_ring;
 	uint16_t nb_ops_to_enq = get_max_enqueue_order_count(order_ring,
 			nb_ops);
-	uint16_t nb_ops_enqd = schedule_enqueue(qp, ops,
+	uint16_t nb_ops_enqd = schedule_mc_enqueue(qp, ops,
 			nb_ops_to_enq);
 
 	scheduler_order_insert(order_ring, ops, nb_ops_enqd);
@@ -81,7 +81,7 @@ schedule_enqueue_ordering(void *qp, struct rte_crypto_op **ops,
 
 
 static uint16_t
-schedule_dequeue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
+schedule_mc_dequeue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 {
 	struct mc_scheduler_qp_ctx *mc_qp_ctx =
 			((struct scheduler_qp_ctx *)qp)->private_qp_ctx;
@@ -107,7 +107,7 @@ schedule_dequeue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 }
 
 static uint16_t
-schedule_dequeue_ordering(void *qp, struct rte_crypto_op **ops,
+schedule_mc_dequeue_ordering(void *qp, struct rte_crypto_op **ops,
 		uint16_t nb_ops)
 {
 	struct rte_ring *order_ring =
@@ -253,11 +253,11 @@ scheduler_start(struct rte_cryptodev *dev)
 					sched_ctx->wc_pool[i]);
 
 	if (sched_ctx->reordering_enabled) {
-		dev->enqueue_burst = &schedule_enqueue_ordering;
-		dev->dequeue_burst = &schedule_dequeue_ordering;
+		dev->enqueue_burst = &schedule_mc_enqueue_ordering;
+		dev->dequeue_burst = &schedule_mc_dequeue_ordering;
 	} else {
-		dev->enqueue_burst = &schedule_enqueue;
-		dev->dequeue_burst = &schedule_dequeue;
+		dev->enqueue_burst = &schedule_mc_enqueue;
+		dev->dequeue_burst = &schedule_mc_dequeue;
 	}
 
 	for (i = 0; i < dev->data->nb_queue_pairs; i++) {
diff --git a/drivers/crypto/scheduler/scheduler_pkt_size_distr.c b/drivers/crypto/scheduler/scheduler_pkt_size_distr.c
index 57e330a744..b025ab9736 100644
--- a/drivers/crypto/scheduler/scheduler_pkt_size_distr.c
+++ b/drivers/crypto/scheduler/scheduler_pkt_size_distr.c
@@ -34,7 +34,7 @@ struct psd_schedule_op {
 };
 
 static uint16_t
-schedule_enqueue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
+schedule_dist_enqueue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 {
 	struct scheduler_qp_ctx *qp_ctx = qp;
 	struct psd_scheduler_qp_ctx *psd_qp_ctx = qp_ctx->private_qp_ctx;
@@ -171,14 +171,14 @@ schedule_enqueue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 }
 
 static uint16_t
-schedule_enqueue_ordering(void *qp, struct rte_crypto_op **ops,
+schedule_dist_enqueue_ordering(void *qp, struct rte_crypto_op **ops,
 		uint16_t nb_ops)
 {
 	struct rte_ring *order_ring =
 			((struct scheduler_qp_ctx *)qp)->order_ring;
 	uint16_t nb_ops_to_enq = get_max_enqueue_order_count(order_ring,
 			nb_ops);
-	uint16_t nb_ops_enqd = schedule_enqueue(qp, ops,
+	uint16_t nb_ops_enqd = schedule_dist_enqueue(qp, ops,
 			nb_ops_to_enq);
 
 	scheduler_order_insert(order_ring, ops, nb_ops_enqd);
@@ -187,7 +187,7 @@ schedule_enqueue_ordering(void *qp, struct rte_crypto_op **ops,
 }
 
 static uint16_t
-schedule_dequeue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
+schedule_dist_dequeue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 {
 	struct psd_scheduler_qp_ctx *qp_ctx =
 			((struct scheduler_qp_ctx *)qp)->private_qp_ctx;
@@ -224,13 +224,13 @@ schedule_dequeue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 }
 
 static uint16_t
-schedule_dequeue_ordering(void *qp, struct rte_crypto_op **ops,
+schedule_dist_dequeue_ordering(void *qp, struct rte_crypto_op **ops,
 		uint16_t nb_ops)
 {
 	struct rte_ring *order_ring =
 			((struct scheduler_qp_ctx *)qp)->order_ring;
 
-	schedule_dequeue(qp, ops, nb_ops);
+	schedule_dist_dequeue(qp, ops, nb_ops);
 
 	return scheduler_order_drain(order_ring, ops, nb_ops);
 }
@@ -281,11 +281,11 @@ scheduler_start(struct rte_cryptodev *dev)
 	}
 
 	if (sched_ctx->reordering_enabled) {
-		dev->enqueue_burst = &schedule_enqueue_ordering;
-		dev->dequeue_burst = &schedule_dequeue_ordering;
+		dev->enqueue_burst = &schedule_dist_enqueue_ordering;
+		dev->dequeue_burst = &schedule_dist_dequeue_ordering;
 	} else {
-		dev->enqueue_burst = &schedule_enqueue;
-		dev->dequeue_burst = &schedule_dequeue;
+		dev->enqueue_burst = &schedule_dist_enqueue;
+		dev->dequeue_burst = &schedule_dist_dequeue;
 	}
 
 	return 0;
diff --git a/drivers/crypto/scheduler/scheduler_roundrobin.c b/drivers/crypto/scheduler/scheduler_roundrobin.c
index bc4a632106..95e34401ce 100644
--- a/drivers/crypto/scheduler/scheduler_roundrobin.c
+++ b/drivers/crypto/scheduler/scheduler_roundrobin.c
@@ -17,7 +17,7 @@ struct rr_scheduler_qp_ctx {
 };
 
 static uint16_t
-schedule_enqueue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
+schedule_rr_enqueue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 {
 	struct rr_scheduler_qp_ctx *rr_qp_ctx =
 			((struct scheduler_qp_ctx *)qp)->private_qp_ctx;
@@ -43,14 +43,14 @@ schedule_enqueue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 }
 
 static uint16_t
-schedule_enqueue_ordering(void *qp, struct rte_crypto_op **ops,
+schedule_rr_enqueue_ordering(void *qp, struct rte_crypto_op **ops,
 		uint16_t nb_ops)
 {
 	struct rte_ring *order_ring =
 			((struct scheduler_qp_ctx *)qp)->order_ring;
 	uint16_t nb_ops_to_enq = get_max_enqueue_order_count(order_ring,
 			nb_ops);
-	uint16_t nb_ops_enqd = schedule_enqueue(qp, ops,
+	uint16_t nb_ops_enqd = schedule_rr_enqueue(qp, ops,
 			nb_ops_to_enq);
 
 	scheduler_order_insert(order_ring, ops, nb_ops_enqd);
@@ -60,7 +60,7 @@ schedule_enqueue_ordering(void *qp, struct rte_crypto_op **ops,
 
 
 static uint16_t
-schedule_dequeue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
+schedule_rr_dequeue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 {
 	struct rr_scheduler_qp_ctx *rr_qp_ctx =
 			((struct scheduler_qp_ctx *)qp)->private_qp_ctx;
@@ -98,13 +98,13 @@ schedule_dequeue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 }
 
 static uint16_t
-schedule_dequeue_ordering(void *qp, struct rte_crypto_op **ops,
+schedule_rr_dequeue_ordering(void *qp, struct rte_crypto_op **ops,
 		uint16_t nb_ops)
 {
 	struct rte_ring *order_ring =
 			((struct scheduler_qp_ctx *)qp)->order_ring;
 
-	schedule_dequeue(qp, ops, nb_ops);
+	schedule_rr_dequeue(qp, ops, nb_ops);
 
 	return scheduler_order_drain(order_ring, ops, nb_ops);
 }
@@ -130,11 +130,11 @@ scheduler_start(struct rte_cryptodev *dev)
 	uint16_t i;
 
 	if (sched_ctx->reordering_enabled) {
-		dev->enqueue_burst = &schedule_enqueue_ordering;
-		dev->dequeue_burst = &schedule_dequeue_ordering;
+		dev->enqueue_burst = &schedule_rr_enqueue_ordering;
+		dev->dequeue_burst = &schedule_rr_dequeue_ordering;
 	} else {
-		dev->enqueue_burst = &schedule_enqueue;
-		dev->dequeue_burst = &schedule_dequeue;
+		dev->enqueue_burst = &schedule_rr_enqueue;
+		dev->dequeue_burst = &schedule_rr_dequeue;
 	}
 
 	for (i = 0; i < dev->data->nb_queue_pairs; i++) {
-- 
2.25.1

[dpdk-dev] [PATCH 7/8] crypto/scheduler: update for new datapath framework

[Akhil Goyal]

PMD is updated to use the new API for all enqueue
and dequeue paths.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
---
 drivers/crypto/scheduler/scheduler_failover.c | 23 +++++++++++++++----
 .../crypto/scheduler/scheduler_multicore.c    | 22 ++++++++++++++----
 .../scheduler/scheduler_pkt_size_distr.c      | 22 ++++++++++++++----
 .../crypto/scheduler/scheduler_roundrobin.c   | 22 ++++++++++++++----
 4 files changed, 72 insertions(+), 17 deletions(-)

diff --git a/drivers/crypto/scheduler/scheduler_failover.c b/drivers/crypto/scheduler/scheduler_failover.c
index 88cc8f05f7..0ccebfa6d1 100644
--- a/drivers/crypto/scheduler/scheduler_failover.c
+++ b/drivers/crypto/scheduler/scheduler_failover.c
@@ -3,6 +3,7 @@
  */
 
 #include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_malloc.h>
 
 #include "rte_cryptodev_scheduler_operations.h"
@@ -13,6 +14,11 @@
 #define NB_FAILOVER_WORKERS	2
 #define WORKER_SWITCH_MASK	(0x01)
 
+_RTE_CRYPTO_ENQ_PROTO(schedule_fo_enqueue);
+_RTE_CRYPTO_DEQ_PROTO(schedule_fo_dequeue);
+_RTE_CRYPTO_ENQ_PROTO(schedule_fo_enqueue_ordering);
+_RTE_CRYPTO_DEQ_PROTO(schedule_fo_dequeue_ordering);
+
 struct fo_scheduler_qp_ctx {
 	struct scheduler_worker primary_worker;
 	struct scheduler_worker secondary_worker;
@@ -57,7 +63,7 @@ schedule_fo_enqueue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 
 	return enqueued_ops;
 }
-
+_RTE_CRYPTO_ENQ_DEF(schedule_fo_enqueue)
 
 static uint16_t
 schedule_fo_enqueue_ordering(void *qp, struct rte_crypto_op **ops,
@@ -74,6 +80,7 @@ schedule_fo_enqueue_ordering(void *qp, struct rte_crypto_op **ops,
 
 	return nb_ops_enqd;
 }
+_RTE_CRYPTO_ENQ_DEF(schedule_fo_enqueue_ordering)
 
 static uint16_t
 schedule_fo_dequeue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
@@ -106,6 +113,7 @@ schedule_fo_dequeue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 
 	return nb_deq_ops + nb_deq_ops2;
 }
+_RTE_CRYPTO_DEQ_DEF(schedule_fo_dequeue)
 
 static uint16_t
 schedule_fo_dequeue_ordering(void *qp, struct rte_crypto_op **ops,
@@ -118,6 +126,7 @@ schedule_fo_dequeue_ordering(void *qp, struct rte_crypto_op **ops,
 
 	return scheduler_order_drain(order_ring, ops, nb_ops);
 }
+_RTE_CRYPTO_DEQ_DEF(schedule_fo_dequeue_ordering)
 
 static int
 worker_attach(__rte_unused struct rte_cryptodev *dev,
@@ -145,11 +154,15 @@ scheduler_start(struct rte_cryptodev *dev)
 	}
 
 	if (sched_ctx->reordering_enabled) {
-		dev->enqueue_burst = schedule_fo_enqueue_ordering;
-		dev->dequeue_burst = schedule_fo_dequeue_ordering;
+		rte_crypto_set_enq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(schedule_fo_enqueue_ordering));
+		rte_crypto_set_deq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(schedule_fo_dequeue_ordering));
 	} else {
-		dev->enqueue_burst = schedule_fo_enqueue;
-		dev->dequeue_burst = schedule_fo_dequeue;
+		rte_crypto_set_enq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(schedule_fo_enqueue));
+		rte_crypto_set_deq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(schedule_fo_dequeue));
 	}
 
 	for (i = 0; i < dev->data->nb_queue_pairs; i++) {
diff --git a/drivers/crypto/scheduler/scheduler_multicore.c b/drivers/crypto/scheduler/scheduler_multicore.c
index bf97343e52..4c145dae88 100644
--- a/drivers/crypto/scheduler/scheduler_multicore.c
+++ b/drivers/crypto/scheduler/scheduler_multicore.c
@@ -4,6 +4,7 @@
 #include <unistd.h>
 
 #include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_malloc.h>
 
 #include "rte_cryptodev_scheduler_operations.h"
@@ -16,6 +17,11 @@
 
 #define CRYPTO_OP_STATUS_BIT_COMPLETE	0x80
 
+_RTE_CRYPTO_ENQ_PROTO(schedule_mc_enqueue);
+_RTE_CRYPTO_DEQ_PROTO(schedule_mc_dequeue);
+_RTE_CRYPTO_ENQ_PROTO(schedule_mc_enqueue_ordering);
+_RTE_CRYPTO_DEQ_PROTO(schedule_mc_dequeue_ordering);
+
 /** multi-core scheduler context */
 struct mc_scheduler_ctx {
 	uint32_t num_workers;             /**< Number of workers polling */
@@ -62,6 +68,7 @@ schedule_mc_enqueue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 
 	return processed_ops;
 }
+_RTE_CRYPTO_ENQ_DEF(schedule_mc_enqueue)
 
 static uint16_t
 schedule_mc_enqueue_ordering(void *qp, struct rte_crypto_op **ops,
@@ -78,6 +85,7 @@ schedule_mc_enqueue_ordering(void *qp, struct rte_crypto_op **ops,
 
 	return nb_ops_enqd;
 }
+_RTE_CRYPTO_ENQ_DEF(schedule_mc_enqueue_ordering)
 
 
 static uint16_t
@@ -105,6 +113,7 @@ schedule_mc_dequeue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 	return processed_ops;
 
 }
+_RTE_CRYPTO_DEQ_DEF(schedule_mc_dequeue)
 
 static uint16_t
 schedule_mc_dequeue_ordering(void *qp, struct rte_crypto_op **ops,
@@ -130,6 +139,7 @@ schedule_mc_dequeue_ordering(void *qp, struct rte_crypto_op **ops,
 	rte_ring_dequeue_finish(order_ring, nb_ops_to_deq);
 	return nb_ops_to_deq;
 }
+_RTE_CRYPTO_DEQ_DEF(schedule_mc_dequeue_ordering)
 
 static int
 worker_attach(__rte_unused struct rte_cryptodev *dev,
@@ -253,11 +263,15 @@ scheduler_start(struct rte_cryptodev *dev)
 					sched_ctx->wc_pool[i]);
 
 	if (sched_ctx->reordering_enabled) {
-		dev->enqueue_burst = &schedule_mc_enqueue_ordering;
-		dev->dequeue_burst = &schedule_mc_dequeue_ordering;
+		rte_crypto_set_enq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(schedule_mc_enqueue_ordering));
+		rte_crypto_set_deq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(schedule_mc_dequeue_ordering));
 	} else {
-		dev->enqueue_burst = &schedule_mc_enqueue;
-		dev->dequeue_burst = &schedule_mc_dequeue;
+		rte_crypto_set_enq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(schedule_mc_enqueue));
+		rte_crypto_set_deq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(schedule_mc_dequeue));
 	}
 
 	for (i = 0; i < dev->data->nb_queue_pairs; i++) {
diff --git a/drivers/crypto/scheduler/scheduler_pkt_size_distr.c b/drivers/crypto/scheduler/scheduler_pkt_size_distr.c
index b025ab9736..811f30ca0d 100644
--- a/drivers/crypto/scheduler/scheduler_pkt_size_distr.c
+++ b/drivers/crypto/scheduler/scheduler_pkt_size_distr.c
@@ -3,6 +3,7 @@
  */
 
 #include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_malloc.h>
 
 #include "rte_cryptodev_scheduler_operations.h"
@@ -14,6 +15,11 @@
 #define SECONDARY_WORKER_IDX			1
 #define NB_PKT_SIZE_WORKERS			2
 
+_RTE_CRYPTO_ENQ_PROTO(schedule_dist_enqueue);
+_RTE_CRYPTO_DEQ_PROTO(schedule_dist_dequeue);
+_RTE_CRYPTO_ENQ_PROTO(schedule_dist_enqueue_ordering);
+_RTE_CRYPTO_DEQ_PROTO(schedule_dist_dequeue_ordering);
+
 /** pkt size based scheduler context */
 struct psd_scheduler_ctx {
 	uint32_t threshold;
@@ -169,6 +175,7 @@ schedule_dist_enqueue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 
 	return processed_ops_pri + processed_ops_sec;
 }
+_RTE_CRYPTO_ENQ_DEF(schedule_dist_enqueue)
 
 static uint16_t
 schedule_dist_enqueue_ordering(void *qp, struct rte_crypto_op **ops,
@@ -185,6 +192,7 @@ schedule_dist_enqueue_ordering(void *qp, struct rte_crypto_op **ops,
 
 	return nb_ops_enqd;
 }
+_RTE_CRYPTO_ENQ_DEF(schedule_dist_enqueue_ordering)
 
 static uint16_t
 schedule_dist_dequeue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
@@ -222,6 +230,7 @@ schedule_dist_dequeue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 
 	return nb_deq_ops_pri + nb_deq_ops_sec;
 }
+_RTE_CRYPTO_DEQ_DEF(schedule_dist_dequeue)
 
 static uint16_t
 schedule_dist_dequeue_ordering(void *qp, struct rte_crypto_op **ops,
@@ -234,6 +243,7 @@ schedule_dist_dequeue_ordering(void *qp, struct rte_crypto_op **ops,
 
 	return scheduler_order_drain(order_ring, ops, nb_ops);
 }
+_RTE_CRYPTO_DEQ_DEF(schedule_dist_dequeue_ordering)
 
 static int
 worker_attach(__rte_unused struct rte_cryptodev *dev,
@@ -281,11 +291,15 @@ scheduler_start(struct rte_cryptodev *dev)
 	}
 
 	if (sched_ctx->reordering_enabled) {
-		dev->enqueue_burst = &schedule_dist_enqueue_ordering;
-		dev->dequeue_burst = &schedule_dist_dequeue_ordering;
+		rte_crypto_set_enq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(schedule_dist_enqueue_ordering));
+		rte_crypto_set_deq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(schedule_dist_dequeue_ordering));
 	} else {
-		dev->enqueue_burst = &schedule_dist_enqueue;
-		dev->dequeue_burst = &schedule_dist_dequeue;
+		rte_crypto_set_enq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(schedule_dist_enqueue));
+		rte_crypto_set_deq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(schedule_dist_dequeue));
 	}
 
 	return 0;
diff --git a/drivers/crypto/scheduler/scheduler_roundrobin.c b/drivers/crypto/scheduler/scheduler_roundrobin.c
index 95e34401ce..139e227cfe 100644
--- a/drivers/crypto/scheduler/scheduler_roundrobin.c
+++ b/drivers/crypto/scheduler/scheduler_roundrobin.c
@@ -3,11 +3,17 @@
  */
 
 #include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_malloc.h>
 
 #include "rte_cryptodev_scheduler_operations.h"
 #include "scheduler_pmd_private.h"
 
+_RTE_CRYPTO_ENQ_PROTO(schedule_rr_enqueue);
+_RTE_CRYPTO_DEQ_PROTO(schedule_rr_dequeue);
+_RTE_CRYPTO_ENQ_PROTO(schedule_rr_enqueue_ordering);
+_RTE_CRYPTO_DEQ_PROTO(schedule_rr_dequeue_ordering);
+
 struct rr_scheduler_qp_ctx {
 	struct scheduler_worker workers[RTE_CRYPTODEV_SCHEDULER_MAX_NB_WORKERS];
 	uint32_t nb_workers;
@@ -41,6 +47,7 @@ schedule_rr_enqueue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 
 	return processed_ops;
 }
+_RTE_CRYPTO_ENQ_DEF(schedule_rr_enqueue)
 
 static uint16_t
 schedule_rr_enqueue_ordering(void *qp, struct rte_crypto_op **ops,
@@ -57,6 +64,7 @@ schedule_rr_enqueue_ordering(void *qp, struct rte_crypto_op **ops,
 
 	return nb_ops_enqd;
 }
+_RTE_CRYPTO_ENQ_DEF(schedule_rr_enqueue_ordering)
 
 
 static uint16_t
@@ -96,6 +104,7 @@ schedule_rr_dequeue(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops)
 
 	return nb_deq_ops;
 }
+_RTE_CRYPTO_DEQ_DEF(schedule_rr_dequeue)
 
 static uint16_t
 schedule_rr_dequeue_ordering(void *qp, struct rte_crypto_op **ops,
@@ -108,6 +117,7 @@ schedule_rr_dequeue_ordering(void *qp, struct rte_crypto_op **ops,
 
 	return scheduler_order_drain(order_ring, ops, nb_ops);
 }
+_RTE_CRYPTO_DEQ_DEF(schedule_rr_dequeue_ordering)
 
 static int
 worker_attach(__rte_unused struct rte_cryptodev *dev,
@@ -130,11 +140,15 @@ scheduler_start(struct rte_cryptodev *dev)
 	uint16_t i;
 
 	if (sched_ctx->reordering_enabled) {
-		dev->enqueue_burst = &schedule_rr_enqueue_ordering;
-		dev->dequeue_burst = &schedule_rr_dequeue_ordering;
+		rte_crypto_set_enq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(schedule_rr_enqueue_ordering));
+		rte_crypto_set_deq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(schedule_rr_dequeue_ordering));
 	} else {
-		dev->enqueue_burst = &schedule_rr_enqueue;
-		dev->dequeue_burst = &schedule_rr_dequeue;
+		rte_crypto_set_enq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_ENQ_FUNC(schedule_rr_enqueue));
+		rte_crypto_set_deq_burst_fn(dev->data->dev_id,
+			_RTE_CRYPTO_DEQ_FUNC(schedule_rr_dequeue));
 	}
 
 	for (i = 0; i < dev->data->nb_queue_pairs; i++) {
-- 
2.25.1

[dpdk-dev] [PATCH 8/8] cryptodev: move device specific structures

[Akhil Goyal]

The device specific structures - rte_cryptodev
and rte_cryptodev_data are moved to cryptodev_pmd.h
to hide it from the applications.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
---
 lib/cryptodev/cryptodev_pmd.h      | 62 ++++++++++++++++++++++++
 lib/cryptodev/rte_cryptodev_core.h | 76 ------------------------------
 2 files changed, 62 insertions(+), 76 deletions(-)

diff --git a/lib/cryptodev/cryptodev_pmd.h b/lib/cryptodev/cryptodev_pmd.h
index d40e5cee94..00c159c2db 100644
--- a/lib/cryptodev/cryptodev_pmd.h
+++ b/lib/cryptodev/cryptodev_pmd.h
@@ -56,6 +56,68 @@ struct rte_cryptodev_pmd_init_params {
 	unsigned int max_nb_queue_pairs;
 };
 
+/**
+ * @internal
+ * The data part, with no function pointers, associated with each device.
+ *
+ * This structure is safe to place in shared memory to be common among
+ * different processes in a multi-process configuration.
+ */
+struct rte_cryptodev_data {
+	uint8_t dev_id;
+	/**< Device ID for this instance */
+	uint8_t socket_id;
+	/**< Socket ID where memory is allocated */
+	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
+	/**< Unique identifier name */
+
+	__extension__
+	uint8_t dev_started : 1;
+	/**< Device state: STARTED(1)/STOPPED(0) */
+
+	struct rte_mempool *session_pool;
+	/**< Session memory pool */
+	void **queue_pairs;
+	/**< Array of pointers to queue pairs. */
+	uint16_t nb_queue_pairs;
+	/**< Number of device queue pairs. */
+
+	void *dev_private;
+	/**< PMD-specific private data */
+} __rte_cache_aligned;
+
+
+/** @internal The data structure associated with each crypto device. */
+struct rte_cryptodev {
+	struct rte_cryptodev_data *data;
+	/**< Pointer to device data */
+	struct rte_cryptodev_ops *dev_ops;
+	/**< Functions exported by PMD */
+	uint64_t feature_flags;
+	/**< Feature flags exposes HW/SW features for the given device */
+	struct rte_device *device;
+	/**< Backing device */
+
+	uint8_t driver_id;
+	/**< Crypto driver identifier*/
+
+	struct rte_cryptodev_cb_list link_intr_cbs;
+	/**< User application callback for interrupts if present */
+
+	void *security_ctx;
+	/**< Context for security ops */
+
+	__extension__
+	uint8_t attached : 1;
+	/**< Flag indicating the device is attached */
+
+	struct rte_cryptodev_cb_rcu *enq_cbs;
+	/**< User application callback for pre enqueue processing */
+
+	struct rte_cryptodev_cb_rcu *deq_cbs;
+	/**< User application callback for post dequeue processing */
+} __rte_cache_aligned;
+
 /** Global structure used for maintaining state of allocated crypto devices */
 struct rte_cryptodev_global {
 	struct rte_cryptodev *devs;	/**< Device information array */
diff --git a/lib/cryptodev/rte_cryptodev_core.h b/lib/cryptodev/rte_cryptodev_core.h
index ec38f70e0c..88506e8a7b 100644
--- a/lib/cryptodev/rte_cryptodev_core.h
+++ b/lib/cryptodev/rte_cryptodev_core.h
@@ -16,15 +16,6 @@
  * Applications should not use these directly.
  *
  */
-
-typedef uint16_t (*dequeue_pkt_burst_t)(void *qp,
-		struct rte_crypto_op **ops,	uint16_t nb_ops);
-/**< Dequeue processed packets from queue pair of a device. */
-
-typedef uint16_t (*enqueue_pkt_burst_t)(void *qp,
-		struct rte_crypto_op **ops,	uint16_t nb_ops);
-/**< Enqueue packets for processing on queue pair of a device. */
-
 typedef uint16_t (*rte_crypto_dequeue_burst_t)(uint8_t dev_id, uint8_t qp_id,
 					      struct rte_crypto_op **ops,
 					      uint16_t nb_ops);
@@ -44,73 +35,6 @@ struct rte_cryptodev_api {
 
 extern struct rte_cryptodev_api *rte_cryptodev_api;
 
-/**
- * @internal
- * The data part, with no function pointers, associated with each device.
- *
- * This structure is safe to place in shared memory to be common among
- * different processes in a multi-process configuration.
- */
-struct rte_cryptodev_data {
-	uint8_t dev_id;
-	/**< Device ID for this instance */
-	uint8_t socket_id;
-	/**< Socket ID where memory is allocated */
-	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
-	/**< Unique identifier name */
-
-	__extension__
-	uint8_t dev_started : 1;
-	/**< Device state: STARTED(1)/STOPPED(0) */
-
-	struct rte_mempool *session_pool;
-	/**< Session memory pool */
-	void **queue_pairs;
-	/**< Array of pointers to queue pairs. */
-	uint16_t nb_queue_pairs;
-	/**< Number of device queue pairs. */
-
-	void *dev_private;
-	/**< PMD-specific private data */
-} __rte_cache_aligned;
-
-
-/** @internal The data structure associated with each crypto device. */
-struct rte_cryptodev {
-	dequeue_pkt_burst_t dequeue_burst;
-	/**< Pointer to PMD receive function. */
-	enqueue_pkt_burst_t enqueue_burst;
-	/**< Pointer to PMD transmit function. */
-
-	struct rte_cryptodev_data *data;
-	/**< Pointer to device data */
-	struct rte_cryptodev_ops *dev_ops;
-	/**< Functions exported by PMD */
-	uint64_t feature_flags;
-	/**< Feature flags exposes HW/SW features for the given device */
-	struct rte_device *device;
-	/**< Backing device */
-
-	uint8_t driver_id;
-	/**< Crypto driver identifier*/
-
-	struct rte_cryptodev_cb_list link_intr_cbs;
-	/**< User application callback for interrupts if present */
-
-	void *security_ctx;
-	/**< Context for security ops */
-
-	__extension__
-	uint8_t attached : 1;
-	/**< Flag indicating the device is attached */
-
-	struct rte_cryptodev_cb_rcu *enq_cbs;
-	/**< User application callback for pre enqueue processing */
-
-	struct rte_cryptodev_cb_rcu *deq_cbs;
-	/**< User application callback for post dequeue processing */
-} __rte_cache_aligned;
-
 /**
  * The pool of rte_cryptodev structures.
  */
-- 
2.25.1

Re: [dpdk-dev] [PATCH 3/8] cryptodev: add helper functions for new datapath interface

[Zhang, Roy Fan]

Hi Akhil,

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Sunday, August 29, 2021 1:52 PM
> To: dev@dpdk.org
> Cc: anoobj@marvell.com; Nicolau, Radu <radu.nicolau@intel.com>; Doherty,
> Declan <declan.doherty@intel.com>; hemant.agrawal@nxp.com;
> matan@nvidia.com; Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> thomas@monjalon.net; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
> asomalap@amd.com; ruifeng.wang@arm.com;
> ajit.khaparde@broadcom.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> adwivedi@marvell.com; michaelsh@marvell.com;
> rnagadheeraj@marvell.com; jianjay.zhou@huawei.com; jerinj@marvell.com;
> Akhil Goyal <gakhil@marvell.com>
> Subject: [PATCH 3/8] cryptodev: add helper functions for new datapath
> interface
> 
> Add helper functions and macros to help drivers to
> transition to new datapath interface.
> 
> Signed-off-by: Akhil Goyal <gakhil@marvell.com>
> ---
>  lib/cryptodev/cryptodev_pmd.h | 246
> ++++++++++++++++++++++++++++++++++
>  lib/cryptodev/rte_cryptodev.c |  40 +++++-
>  lib/cryptodev/version.map     |   4 +
>  3 files changed, 289 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/cryptodev/cryptodev_pmd.h
> b/lib/cryptodev/cryptodev_pmd.h
> index eeaea13a23..d40e5cee94 100644
> --- a/lib/cryptodev/cryptodev_pmd.h
> +++ b/lib/cryptodev/cryptodev_pmd.h
> @@ -70,6 +70,13 @@ struct cryptodev_driver {
>  	const struct rte_driver *driver;
>  	uint8_t id;
>  };
> +/**
> + * @internal
> + * The pool of *rte_cryptodev* structures. The size of the pool
> + * is configured at compile-time in the <rte_cryptodev.c> file.
> + */
> +extern struct rte_cryptodev rte_crypto_devices[];
> +
> 
>  /**
>   * Get the rte_cryptodev structure device pointer for the device. Assumes a
> @@ -529,6 +536,245 @@ __rte_internal
>  void
>  rte_cryptodev_api_reset(struct rte_cryptodev_api *api);
> 
> +/**
> + * @internal
> + * Helper routine for cryptodev_dequeue_burst.
> + * Should be called as first thing on entrance to the PMD's
> + * rte_cryptodev_dequeue_burst implementation.
> + * Does necessary checks and returns pointer to cryptodev identifier.
> + *
> + * @param dev_id
> + *  The device identifier of the crypto device.
> + * @param qp_id
> + *  The index of the queue pair from which processed crypto ops will
> + *  be dequeued.
> + *
> + * @return
> + *  Pointer to device queue pair on success or NULL otherwise.
> + */
> +__rte_internal
> +static inline void *
> +_rte_cryptodev_dequeue_prolog(uint8_t dev_id, uint8_t qp_id)
> +{
> +	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
> +
> +	return dev->data->queue_pairs[qp_id];
> +}
 
[Fan: the function name looks unclear to me - maybe 
rte_cryptodev_dequeue_prepare?
Also the function didn't do any check as the description suggested - the 
qp is later checked in _RTE_CRYPTO_DEQ_DEF, maybe remove the
description?]

> +
> +/**
> + * @internal
> + * Helper routine for crypto driver dequeue API.
> + * Should be called at exit from PMD's rte_cryptodev_dequeue_burst
> + * implementation.
> + * Does necessary post-processing - invokes RX callbacks if any, tracing, etc.
> + *
> + * @param dev_id
> + *  The device identifier of the Crypto device.
> + * @param qp_id
> + *  The index of the queue pair from which to retrieve input crypto_ops.
> + * @param ops
> + *   The address of an array of pointers to *rte_crypto_op* structures that
> + *   have been retrieved from the device.
> + * @param nb_ops
> + *   The number of ops that were retrieved from the device.
> + *
> + * @return
> + *  The number of crypto ops effectively supplied to the *ops* array.
> + */
> +__rte_internal
> +static inline uint16_t
> +_rte_cryptodev_dequeue_epilog(uint16_t dev_id, uint16_t qp_id,
> +	struct rte_crypto_op **ops, uint16_t nb_ops)
> +{
> +#ifdef RTE_CRYPTO_CALLBACKS
> +	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
> +
> +	if (unlikely(dev->deq_cbs != NULL)) {
> +		struct rte_cryptodev_cb_rcu *list;
> +		struct rte_cryptodev_cb *cb;
> +
> +		/* __ATOMIC_RELEASE memory order was used when the
> +		 * call back was inserted into the list.
> +		 * Since there is a clear dependency between loading
> +		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory
> order is
> +		 * not required.
> +		 */
> +		list = &dev->deq_cbs[qp_id];
> +		rte_rcu_qsbr_thread_online(list->qsbr, 0);
> +		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
> +
> +		while (cb != NULL) {
> +			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
> +					cb->arg);
> +			cb = cb->next;
> +		};
> +
> +		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
> +	}
> +#endif
> +
> +	return nb_ops;
> +}

[Fan: same naming issue - maybe rte_cryptodev_dequeue_post, so
as the enqueue part]

> +#define _RTE_CRYPTO_DEQ_FUNC(fn)	_rte_crypto_deq_##fn
> +
> +/**
> + * @internal
> + * Helper macro to create new API wrappers for existing PMD dequeue
> functions.
> + */
> +#define _RTE_CRYPTO_DEQ_PROTO(fn) \
> +	uint16_t _RTE_CRYPTO_DEQ_FUNC(fn)(uint8_t dev_id, uint8_t
> qp_id, \
> +			struct rte_crypto_op **ops, uint16_t nb_ops)
> +
> +/**
> + * @internal
> + * Helper macro to create new API wrappers for existing PMD dequeue
> functions.
> + */
> +#define _RTE_CRYPTO_DEQ_DEF(fn) \
> +_RTE_CRYPTO_DEQ_PROTO(fn) \
> +{ \
> +	void *qp = _rte_cryptodev_dequeue_prolog(dev_id, qp_id); \
> +	if (qp == NULL) \
[Fan: suggest to add "unlikely" above] 
> +		return 0; \
> +	nb_ops = fn(qp, ops, nb_ops); \
> +	return _rte_cryptodev_dequeue_epilog(dev_id, qp_id, ops,
> nb_ops); \
> +}
> +
> +/**
> + * @internal
> + * Helper routine for cryptodev_enqueue_burst.
> + * Should be called as first thing on entrance to the PMD's
> + * rte_cryptodev_enqueue_burst implementation.
> + * Does necessary checks and returns pointer to cryptodev queue pair.
> + *
> + * @param dev_id
> + *  The device identifier of the crypto device.
> + * @param qp_id
> + *  The index of the queue pair in which packets will be enqueued.
> + * @param ops
> + *   The address of an array of pointers to *rte_crypto_op* structures that
> + *   will be enqueued to the device.
> + * @param nb_ops
> + *   The number of ops that will be sent to the device.
> + *
> + * @return
> + *  Pointer to device queue pair on success or NULL otherwise.
> + */
> +__rte_internal
> +static inline void *
> +_rte_cryptodev_enqueue_prolog(uint8_t dev_id, uint8_t qp_id,
> +		struct rte_crypto_op **ops, uint16_t nb_ops)
> +{
> +	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
> +
> +#ifdef RTE_CRYPTO_CALLBACKS
> +	if (unlikely(dev->enq_cbs != NULL)) {
> +		struct rte_cryptodev_cb_rcu *list;
> +		struct rte_cryptodev_cb *cb;
> +
> +		/* __ATOMIC_RELEASE memory order was used when the
> +		 * call back was inserted into the list.
> +		 * Since there is a clear dependency between loading
> +		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory
> order is
> +		 * not required.
> +		 */
> +		list = &dev->enq_cbs[qp_id];
> +		rte_rcu_qsbr_thread_online(list->qsbr, 0);
> +		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
> +
> +		while (cb != NULL) {
> +			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
> +					cb->arg);
> +			cb = cb->next;
> +		};
> +
> +		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
> +	}
> +#endif
> +	return dev->data->queue_pairs[qp_id];
> +}
> +
> +#define _RTE_CRYPTO_ENQ_FUNC(fn)	_rte_crypto_enq_##fn
> +
> +/**
> + * @internal
> + * Helper macro to create new API wrappers for existing PMD enqueue
> functions.
> + */
> +#define _RTE_CRYPTO_ENQ_PROTO(fn) \
> +	uint16_t _RTE_CRYPTO_ENQ_FUNC(fn)(uint8_t dev_id, uint8_t
> qp_id, \
> +			struct rte_crypto_op **ops, uint16_t nb_ops)
> +
> +/**
> + * @internal
> + * Helper macro to create new API wrappers for existing PMD enqueue
> functions.
> + */
> +#define _RTE_CRYPTO_ENQ_DEF(fn) \
> +_RTE_CRYPTO_ENQ_PROTO(fn) \
> +{ \
> +	void *qp = _rte_cryptodev_enqueue_prolog(dev_id, qp_id, ops,
> nb_ops); \
> +	if (qp == NULL) \
> +		return 0; \
> +	return fn(qp, ops, nb_ops); \
> +}
> +
> +/**
> + * @internal
> + * Helper routine to get enqueue burst function of a given device.
> + *
> + * @param dev_id
> + *  The device identifier of the Crypto device.
> + *
> + * @return
> + *  The function if valid else NULL
> + */
> +__rte_internal
> +rte_crypto_enqueue_burst_t
> +rte_crypto_get_enq_burst_fn(uint8_t dev_id);
> +
> +/**
> + * @internal
> + * Helper routine to get dequeue burst function of a given device.
> + *
> + * @param dev_id
> + *  The device identifier of the Crypto device.
> + *
> + * @return
> + *  The function if valid else NULL
> + */
> +__rte_internal
> +rte_crypto_dequeue_burst_t
> +rte_crypto_get_deq_burst_fn(uint8_t dev_id);
> +
> +/**
> + * @internal
> + * Helper routine to set enqueue burst function of a given device.
> + *
> + * @param dev_id
> + *  The device identifier of the Crypto device.
> + *
> + * @return
> + *  0		Success.
> + *  -EINVAL	Failure if dev_id or fn are in-valid.
> + */
> +__rte_internal
> +int
> +rte_crypto_set_enq_burst_fn(uint8_t dev_id,
> rte_crypto_enqueue_burst_t fn);
> +
> +/**
> + * @internal
> + * Helper routine to set dequeue burst function of a given device.
> + *
> + * @param dev_id
> + *  The device identifier of the Crypto device.
> + *
> + * @return
> + *  0		Success.
> + *  -EINVAL	Failure if dev_id or fn are in-valid.
> + */
> +__rte_internal
> +int
> +rte_crypto_set_deq_burst_fn(uint8_t dev_id,
> rte_crypto_dequeue_burst_t fn);
> +
> +
>  static inline void *
>  get_sym_session_private_data(const struct rte_cryptodev_sym_session
> *sess,
>  		uint8_t driver_id) {
> diff --git a/lib/cryptodev/rte_cryptodev.c b/lib/cryptodev/rte_cryptodev.c
> index 26f8390668..4ab82d21d0 100644
> --- a/lib/cryptodev/rte_cryptodev.c
> +++ b/lib/cryptodev/rte_cryptodev.c
> @@ -44,7 +44,7 @@
> 
>  static uint8_t nb_drivers;
> 
> -static struct rte_cryptodev rte_crypto_devices[RTE_CRYPTO_MAX_DEVS];
> +struct rte_cryptodev rte_crypto_devices[RTE_CRYPTO_MAX_DEVS];
> 
>  struct rte_cryptodev *rte_cryptodevs = rte_crypto_devices;
> 
> @@ -1270,6 +1270,44 @@ rte_cryptodev_queue_pair_setup(uint8_t dev_id,
> uint16_t queue_pair_id,
>  			socket_id);
>  }
> 
> +rte_crypto_enqueue_burst_t
> +rte_crypto_get_enq_burst_fn(uint8_t dev_id)
> +{
> +	if (dev_id >= RTE_CRYPTO_MAX_DEVS) {
> +		rte_errno = EINVAL;
> +		return NULL;
> +	}
> +	return rte_cryptodev_api[dev_id].enqueue_burst;
> +}
> +
> +rte_crypto_dequeue_burst_t
> +rte_crypto_get_deq_burst_fn(uint8_t dev_id)
> +{
> +	if (dev_id >= RTE_CRYPTO_MAX_DEVS) {
> +		rte_errno = EINVAL;
> +		return NULL;
> +	}
> +	return rte_cryptodev_api[dev_id].dequeue_burst;
> +}
> +
> +int
> +rte_crypto_set_enq_burst_fn(uint8_t dev_id,
> rte_crypto_enqueue_burst_t fn)
> +{
> +	if (dev_id >= RTE_CRYPTO_MAX_DEVS || fn == NULL)
> +		return -EINVAL;
> +	rte_cryptodev_api[dev_id].enqueue_burst = fn;
> +	return 0;
> +}
> +
> +int
> +rte_crypto_set_deq_burst_fn(uint8_t dev_id,
> rte_crypto_dequeue_burst_t fn)
> +{
> +	if (dev_id >= RTE_CRYPTO_MAX_DEVS || fn == NULL)
> +		return -EINVAL;
> +	rte_cryptodev_api[dev_id].dequeue_burst = fn;
> +	return 0;
> +}
> +
>  struct rte_cryptodev_cb *
>  rte_cryptodev_add_enq_callback(uint8_t dev_id,
>  			       uint16_t qp_id,
> diff --git a/lib/cryptodev/version.map b/lib/cryptodev/version.map
> index 050089ae55..b64384cc05 100644
> --- a/lib/cryptodev/version.map
> +++ b/lib/cryptodev/version.map
> @@ -116,6 +116,10 @@ EXPERIMENTAL {
>  INTERNAL {
>  	global:
> 
> +	rte_crypto_get_deq_burst_fn;
> +	rte_crypto_get_enq_burst_fn;
> +	rte_crypto_set_deq_burst_fn;
> +	rte_crypto_set_enq_burst_fn;
>  	rte_cryptodev_allocate_driver;
>  	rte_cryptodev_api_reset;
>  	rte_cryptodev_pmd_allocate;
> --
> 2.25.1

Regards,
Fan

Re: [dpdk-dev] [PATCH 3/8] cryptodev: add helper functions for new datapath interface

[Akhil Goyal]

Hi Fan,
> Hi Akhil,
> 
> > +__rte_internal
> > +static inline void *
> > +_rte_cryptodev_dequeue_prolog(uint8_t dev_id, uint8_t qp_id)
> > +{
> > +	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
> > +
> > +	return dev->data->queue_pairs[qp_id];
> > +}
> 
> [Fan: the function name looks unclear to me - maybe
> rte_cryptodev_dequeue_prepare?

The naming convention is same as Konstantin did for ethdev and
Subsequently by Pavan in eventdev. I think it is better to align all
With similar naming.

> Also the function didn't do any check as the description suggested - the
> qp is later checked in _RTE_CRYPTO_DEQ_DEF, maybe remove the
> description?]
> 

Ok will update the description in next version

> > +_rte_cryptodev_dequeue_epilog(uint16_t dev_id, uint16_t qp_id,
> > +	struct rte_crypto_op **ops, uint16_t nb_ops)
> > +{
> > +#ifdef RTE_CRYPTO_CALLBACKS
> > +	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
> > +
> > +	if (unlikely(dev->deq_cbs != NULL)) {
> > +		struct rte_cryptodev_cb_rcu *list;
> > +		struct rte_cryptodev_cb *cb;
> > +
> > +		/* __ATOMIC_RELEASE memory order was used when the
> > +		 * call back was inserted into the list.
> > +		 * Since there is a clear dependency between loading
> > +		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory
> > order is
> > +		 * not required.
> > +		 */
> > +		list = &dev->deq_cbs[qp_id];
> > +		rte_rcu_qsbr_thread_online(list->qsbr, 0);
> > +		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
> > +
> > +		while (cb != NULL) {
> > +			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
> > +					cb->arg);
> > +			cb = cb->next;
> > +		};
> > +
> > +		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
> > +	}
> > +#endif
> > +
> > +	return nb_ops;
> > +}
> 
> [Fan: same naming issue - maybe rte_cryptodev_dequeue_post, so
> as the enqueue part]

Same comment as above. Aligned with ethdev and eventdev changes.

> 
> > +#define _RTE_CRYPTO_DEQ_FUNC(fn)	_rte_crypto_deq_##fn
> > +
> > +/**
> > + * @internal
> > + * Helper macro to create new API wrappers for existing PMD dequeue
> > functions.
> > + */
> > +#define _RTE_CRYPTO_DEQ_PROTO(fn) \
> > +	uint16_t _RTE_CRYPTO_DEQ_FUNC(fn)(uint8_t dev_id, uint8_t
> > qp_id, \
> > +			struct rte_crypto_op **ops, uint16_t nb_ops)
> > +
> > +/**
> > + * @internal
> > + * Helper macro to create new API wrappers for existing PMD dequeue
> > functions.
> > + */
> > +#define _RTE_CRYPTO_DEQ_DEF(fn) \
> > +_RTE_CRYPTO_DEQ_PROTO(fn) \
> > +{ \
> > +	void *qp = _rte_cryptodev_dequeue_prolog(dev_id, qp_id); \
> > +	if (qp == NULL) \
> [Fan: suggest to add "unlikely" above]

Ok

Re: [dpdk-dev] [PATCH 0/8] cryptodev: hide internal strutures

[Akhil Goyal]

Hi All,

Please review this series as soon as possible.
Would like to merge this series as early as possible so that we have
Lesser conflicts for other series.

Thanks and Regards,
Akhil

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Sunday, August 29, 2021 6:22 PM
> To: dev@dpdk.org
> Cc: Anoob Joseph <anoobj@marvell.com>; radu.nicolau@intel.com;
> declan.doherty@intel.com; hemant.agrawal@nxp.com; matan@nvidia.com;
> konstantin.ananyev@intel.com; thomas@monjalon.net;
> roy.fan.zhang@intel.com; asomalap@amd.com; ruifeng.wang@arm.com;
> ajit.khaparde@broadcom.com; pablo.de.lara.guarch@intel.com;
> fiona.trahe@intel.com; Ankur Dwivedi <adwivedi@marvell.com>; Michael
> Shamis <michaelsh@marvell.com>; Nagadheeraj Rottela
> <rnagadheeraj@marvell.com>; jianjay.zhou@huawei.com; Jerin Jacob
> Kollanukkaran <jerinj@marvell.com>; Akhil Goyal <gakhil@marvell.com>
> Subject: [PATCH 0/8] cryptodev: hide internal strutures
> 
> Structures rte_cryptodev and rte_cryptodev_data are not
> supposed to be directly used by the application. These
> are made public as they are used by inline datapath
> public APIs.
> This patchset, creates a new rte_cryptodev_core.h file
> which helps in defining a data structure to hold datapath
> APIs in a flat array based on the device identifier which
> is filled by the PMD.
> The patchset updates all crypto PMDs to use the new
> framework and hence the internal structs are made hidden.
> 
> Similar series for ethdev and eventdev are also floated on ML.
> http://patches.dpdk.org/project/dpdk/list/?series=18382
> http://patches.dpdk.org/project/dpdk/list/?series=18422
> 
> 
> Akhil Goyal (8):
>   cryptodev: separate out internal structures
>   cryptodev: move inline APIs into separate structure
>   cryptodev: add helper functions for new datapath interface
>   cryptodev: use new API for datapath functions
>   drivers/crypto: use new framework for datapath
>   crypto/scheduler: rename enq-deq functions
>   crypto/scheduler: update for new datapath framework
>   cryptodev: move device specific structures
> 
>  drivers/crypto/aesni_gcm/aesni_gcm_pmd.c      |  10 +-
>  drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c    |  11 +-
>  drivers/crypto/armv8/rte_armv8_pmd.c          |  11 +-
>  drivers/crypto/bcmfs/bcmfs_sym_pmd.c          |  11 +-
>  drivers/crypto/caam_jr/caam_jr.c              |  11 +-
>  drivers/crypto/ccp/ccp_dev.c                  |   1 +
>  drivers/crypto/ccp/rte_ccp_pmd.c              |  11 +-
>  drivers/crypto/cnxk/cn10k_cryptodev_ops.c     |   8 +-
>  drivers/crypto/cnxk/cn10k_cryptodev_ops.h     |   3 +
>  drivers/crypto/cnxk/cn10k_ipsec.c             |   1 +
>  drivers/crypto/cnxk/cn9k_cryptodev_ops.c      |   9 +-
>  drivers/crypto/cnxk/cn9k_cryptodev_ops.h      |   3 +
>  .../crypto/cnxk/cnxk_cryptodev_capabilities.c |   1 +
>  drivers/crypto/cnxk/cnxk_cryptodev_sec.c      |   1 +
>  drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c   |  12 +-
>  drivers/crypto/dpaa_sec/dpaa_sec.c            |  11 +-
>  drivers/crypto/kasumi/rte_kasumi_pmd.c        |  11 +-
>  drivers/crypto/mlx5/mlx5_crypto.c             |  11 +-
>  drivers/crypto/mvsam/rte_mrvl_pmd.c           |  11 +-
>  drivers/crypto/nitrox/nitrox_sym.c            |  11 +-
>  drivers/crypto/nitrox/nitrox_sym_reqmgr.c     |   1 +
>  drivers/crypto/null/null_crypto_pmd.c         |  11 +-
>  .../crypto/octeontx/otx_cryptodev_hw_access.c |   1 +
>  drivers/crypto/octeontx/otx_cryptodev_ops.c   |  16 +-
>  drivers/crypto/octeontx/otx_cryptodev_ops.h   |   5 +
>  .../crypto/octeontx2/otx2_cryptodev_mbox.c    |   1 +
>  drivers/crypto/octeontx2/otx2_cryptodev_ops.c |  11 +-
>  drivers/crypto/openssl/rte_openssl_pmd.c      |  11 +-
>  drivers/crypto/qat/qat_asym_pmd.c             |  11 +-
>  drivers/crypto/qat/qat_sym_pmd.c              |  10 +-
>  drivers/crypto/scheduler/scheduler_failover.c |  35 +-
>  .../crypto/scheduler/scheduler_multicore.c    |  32 +-
>  .../scheduler/scheduler_pkt_size_distr.c      |  34 +-
>  .../crypto/scheduler/scheduler_roundrobin.c   |  34 +-
>  drivers/crypto/snow3g/rte_snow3g_pmd.c        |  11 +-
>  drivers/crypto/virtio/virtio_cryptodev.c      |  10 +-
>  drivers/crypto/virtio/virtio_cryptodev.h      |   2 +
>  drivers/crypto/virtio/virtio_rxtx.c           |   2 +
>  drivers/crypto/zuc/rte_zuc_pmd.c              |  11 +-
>  lib/cryptodev/cryptodev_pmd.c                 |  33 ++
>  lib/cryptodev/cryptodev_pmd.h                 | 323 +++++++++++++++++-
>  lib/cryptodev/meson.build                     |   4 +-
>  lib/cryptodev/rte_cryptodev.c                 |  43 ++-
>  lib/cryptodev/rte_cryptodev.h                 | 308 +++++------------
>  lib/cryptodev/rte_cryptodev_core.h            |  43 +++
>  lib/cryptodev/version.map                     |   8 +
>  46 files changed, 854 insertions(+), 316 deletions(-)
>  create mode 100644 lib/cryptodev/rte_cryptodev_core.h
> 
> --
> 2.25.1

Re: [dpdk-dev] [PATCH 1/8] cryptodev: separate out internal structures

[Anoob Joseph]

Hi Akhil,

Please see inline.

Thanks,
Anoob

> Subject: [PATCH 1/8] cryptodev: separate out internal structures
> 
> A new header file rte_cryptodev_core.h is added and all internal data
> structures which need not be exposed directly to application are moved to
> this file. These structures are mostly used by drivers, but they need to be in
> the public header file as they are accessed by datapath inline functions for
> performance reasons.
> 
> Signed-off-by: Akhil Goyal <gakhil@marvell.com>
> ---
>  lib/cryptodev/cryptodev_pmd.h      |   6 -
>  lib/cryptodev/meson.build          |   4 +-
>  lib/cryptodev/rte_cryptodev.h      | 360 ++++++++++++-----------------
>  lib/cryptodev/rte_cryptodev_core.h | 100 ++++++++
>  4 files changed, 245 insertions(+), 225 deletions(-)  create mode 100644
> lib/cryptodev/rte_cryptodev_core.h
> 
> diff --git a/lib/cryptodev/cryptodev_pmd.h
> b/lib/cryptodev/cryptodev_pmd.h index ec7bb82be8..f775ba6beb 100644
> --- a/lib/cryptodev/cryptodev_pmd.h
> +++ b/lib/cryptodev/cryptodev_pmd.h
> @@ -96,12 +96,6 @@ __rte_internal
>  struct rte_cryptodev *
>  rte_cryptodev_pmd_get_named_dev(const char *name);
> 
> -/**
> - * The pool of rte_cryptodev structures.
> - */
> -extern struct rte_cryptodev *rte_cryptodevs;
> -
> -
>  /**
>   * Definitions of all functions exported by a driver through the
>   * the generic structure of type *crypto_dev_ops* supplied in the diff --git
> a/lib/cryptodev/meson.build b/lib/cryptodev/meson.build index
> 735935df4a..f32cc62a78 100644
> --- a/lib/cryptodev/meson.build
> +++ b/lib/cryptodev/meson.build
> @@ -14,7 +14,9 @@ headers = files(
>          'rte_crypto_sym.h',
>          'rte_crypto_asym.h',
>  )
> -
> +indirect_headers += files(
> +        'rte_cryptodev_core.h',
> +)
>  driver_sdk_headers += files(
>          'cryptodev_pmd.h',
>  )
> diff --git a/lib/cryptodev/rte_cryptodev.h b/lib/cryptodev/rte_cryptodev.h
> index 33aac44446..3d99dd1cf5 100644
> --- a/lib/cryptodev/rte_cryptodev.h
> +++ b/lib/cryptodev/rte_cryptodev.h
> @@ -861,17 +861,6 @@ rte_cryptodev_callback_unregister(uint8_t dev_id,
>  		enum rte_cryptodev_event_type event,
>  		rte_cryptodev_cb_fn cb_fn, void *cb_arg);
> 
> -typedef uint16_t (*dequeue_pkt_burst_t)(void *qp,
> -		struct rte_crypto_op **ops,	uint16_t nb_ops);
> -/**< Dequeue processed packets from queue pair of a device. */
> -
> -typedef uint16_t (*enqueue_pkt_burst_t)(void *qp,
> -		struct rte_crypto_op **ops,	uint16_t nb_ops);
> -/**< Enqueue packets for processing on queue pair of a device. */
> -
> -
> -
> -
>  struct rte_cryptodev_callback;
> 
>  /** Structure to keep track of registered callbacks */ @@ -901,216 +890,9
> @@ struct rte_cryptodev_cb_rcu {
>  	/**< RCU QSBR variable per queue pair */  };
> 
> -/** The data structure associated with each crypto device. */ -struct
> rte_cryptodev {
> -	dequeue_pkt_burst_t dequeue_burst;
> -	/**< Pointer to PMD receive function. */
> -	enqueue_pkt_burst_t enqueue_burst;
> -	/**< Pointer to PMD transmit function. */
> -
> -	struct rte_cryptodev_data *data;
> -	/**< Pointer to device data */
> -	struct rte_cryptodev_ops *dev_ops;
> -	/**< Functions exported by PMD */
> -	uint64_t feature_flags;
> -	/**< Feature flags exposes HW/SW features for the given device */
> -	struct rte_device *device;
> -	/**< Backing device */
> -
> -	uint8_t driver_id;
> -	/**< Crypto driver identifier*/
> -
> -	struct rte_cryptodev_cb_list link_intr_cbs;
> -	/**< User application callback for interrupts if present */
> -
> -	void *security_ctx;
> -	/**< Context for security ops */
> -
> -	__extension__
> -	uint8_t attached : 1;
> -	/**< Flag indicating the device is attached */
> -
> -	struct rte_cryptodev_cb_rcu *enq_cbs;
> -	/**< User application callback for pre enqueue processing */
> -
> -	struct rte_cryptodev_cb_rcu *deq_cbs;
> -	/**< User application callback for post dequeue processing */
> -} __rte_cache_aligned;
> -
>  void *
>  rte_cryptodev_get_sec_ctx(uint8_t dev_id);
> 
> -/**
> - *
> - * The data part, with no function pointers, associated with each device.
> - *
> - * This structure is safe to place in shared memory to be common among
> - * different processes in a multi-process configuration.
> - */
> -struct rte_cryptodev_data {
> -	uint8_t dev_id;
> -	/**< Device ID for this instance */
> -	uint8_t socket_id;
> -	/**< Socket ID where memory is allocated */
> -	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
> -	/**< Unique identifier name */
> -
> -	__extension__
> -	uint8_t dev_started : 1;
> -	/**< Device state: STARTED(1)/STOPPED(0) */
> -
> -	struct rte_mempool *session_pool;
> -	/**< Session memory pool */
> -	void **queue_pairs;
> -	/**< Array of pointers to queue pairs. */
> -	uint16_t nb_queue_pairs;
> -	/**< Number of device queue pairs. */
> -
> -	void *dev_private;
> -	/**< PMD-specific private data */
> -} __rte_cache_aligned;
> -
> -extern struct rte_cryptodev *rte_cryptodevs;
> -/**
> - *
> - * Dequeue a burst of processed crypto operations from a queue on the
> crypto
> - * device. The dequeued operation are stored in *rte_crypto_op*
> structures
> - * whose pointers are supplied in the *ops* array.
> - *
> - * The rte_cryptodev_dequeue_burst() function returns the number of ops
> - * actually dequeued, which is the number of *rte_crypto_op* data
> structures
> - * effectively supplied into the *ops* array.
> - *
> - * A return value equal to *nb_ops* indicates that the queue contained
> - * at least *nb_ops* operations, and this is likely to signify that other
> - * processed operations remain in the devices output queue. Applications
> - * implementing a "retrieve as many processed operations as possible"
> policy
> - * can check this specific case and keep invoking the
> - * rte_cryptodev_dequeue_burst() function until a value less than
> - * *nb_ops* is returned.
> - *
> - * The rte_cryptodev_dequeue_burst() function does not provide any error
> - * notification to avoid the corresponding overhead.
> - *
> - * @param	dev_id		The symmetric crypto device identifier
> - * @param	qp_id		The index of the queue pair from which to
> - *				retrieve processed packets. The value must
> be
> - *				in the range [0, nb_queue_pair - 1]
> previously
> - *				supplied to rte_cryptodev_configure().
> - * @param	ops		The address of an array of pointers to
> - *				*rte_crypto_op* structures that must be
> - *				large enough to store *nb_ops* pointers in
> it.
> - * @param	nb_ops		The maximum number of operations to
> dequeue.
> - *
> - * @return
> - *   - The number of operations actually dequeued, which is the number
> - *   of pointers to *rte_crypto_op* structures effectively supplied to the
> - *   *ops* array.
> - */
> -static inline uint16_t
> -rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
> -		struct rte_crypto_op **ops, uint16_t nb_ops)
> -{
> -	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
> -
> -	rte_cryptodev_trace_dequeue_burst(dev_id, qp_id, (void **)ops,
> nb_ops);
> -	nb_ops = (*dev->dequeue_burst)
> -			(dev->data->queue_pairs[qp_id], ops, nb_ops);
> -#ifdef RTE_CRYPTO_CALLBACKS
> -	if (unlikely(dev->deq_cbs != NULL)) {
> -		struct rte_cryptodev_cb_rcu *list;
> -		struct rte_cryptodev_cb *cb;
> -
> -		/* __ATOMIC_RELEASE memory order was used when the
> -		 * call back was inserted into the list.
> -		 * Since there is a clear dependency between loading
> -		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory
> order is
> -		 * not required.
> -		 */
> -		list = &dev->deq_cbs[qp_id];
> -		rte_rcu_qsbr_thread_online(list->qsbr, 0);
> -		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
> -
> -		while (cb != NULL) {
> -			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
> -					cb->arg);
> -			cb = cb->next;
> -		};
> -
> -		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
> -	}
> -#endif
> -	return nb_ops;
> -}
> -
> -/**
> - * Enqueue a burst of operations for processing on a crypto device.
> - *
> - * The rte_cryptodev_enqueue_burst() function is invoked to place
> - * crypto operations on the queue *qp_id* of the device designated by
> - * its *dev_id*.
> - *
> - * The *nb_ops* parameter is the number of operations to process which
> are
> - * supplied in the *ops* array of *rte_crypto_op* structures.
> - *
> - * The rte_cryptodev_enqueue_burst() function returns the number of
> - * operations it actually enqueued for processing. A return value equal to
> - * *nb_ops* means that all packets have been enqueued.
> - *
> - * @param	dev_id		The identifier of the device.
> - * @param	qp_id		The index of the queue pair which packets
> are
> - *				to be enqueued for processing. The value
> - *				must be in the range [0, nb_queue_pairs - 1]
> - *				previously supplied to
> - *				 *rte_cryptodev_configure*.
> - * @param	ops		The address of an array of *nb_ops* pointers
> - *				to *rte_crypto_op* structures which contain
> - *				the crypto operations to be processed.
> - * @param	nb_ops		The number of operations to process.
> - *
> - * @return
> - * The number of operations actually enqueued on the crypto device. The
> return
> - * value can be less than the value of the *nb_ops* parameter when the
> - * crypto devices queue is full or if invalid parameters are specified in
> - * a *rte_crypto_op*.
> - */
> -static inline uint16_t
> -rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
> -		struct rte_crypto_op **ops, uint16_t nb_ops)
> -{
> -	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
> -
> -#ifdef RTE_CRYPTO_CALLBACKS
> -	if (unlikely(dev->enq_cbs != NULL)) {
> -		struct rte_cryptodev_cb_rcu *list;
> -		struct rte_cryptodev_cb *cb;
> -
> -		/* __ATOMIC_RELEASE memory order was used when the
> -		 * call back was inserted into the list.
> -		 * Since there is a clear dependency between loading
> -		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory
> order is
> -		 * not required.
> -		 */
> -		list = &dev->enq_cbs[qp_id];
> -		rte_rcu_qsbr_thread_online(list->qsbr, 0);
> -		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
> -
> -		while (cb != NULL) {
> -			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
> -					cb->arg);
> -			cb = cb->next;
> -		};
> -
> -		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
> -	}
> -#endif
> -
> -	rte_cryptodev_trace_enqueue_burst(dev_id, qp_id, (void **)ops,
> nb_ops);
> -	return (*dev->enqueue_burst)(
> -			dev->data->queue_pairs[qp_id], ops, nb_ops);
> -}
> -
> -
>  /** Cryptodev symmetric crypto session
>   * Each session is derived from a fixed xform chain. Therefore each session
>   * has a fixed algo, key, op-type, digest_len etc.
> @@ -1997,6 +1779,148 @@ int rte_cryptodev_remove_deq_callback(uint8_t
> dev_id,
>  				      uint16_t qp_id,
>  				      struct rte_cryptodev_cb *cb);
> 
> +#include <rte_cryptodev_core.h>

[Anoob] Is this intentional? Shouldn't we keep includes together in the top of the file?
 
> +/**
> + *

[Anoob] Is there an extra blank line?
 
> + * Dequeue a burst of processed crypto operations from a queue on the
> +crypto
> + * device. The dequeued operation are stored in *rte_crypto_op*
> +structures
> + * whose pointers are supplied in the *ops* array.
> + *
> + * The rte_cryptodev_dequeue_burst() function returns the number of ops
> + * actually dequeued, which is the number of *rte_crypto_op* data
> +structures
> + * effectively supplied into the *ops* array.
> + *
> + * A return value equal to *nb_ops* indicates that the queue contained
> + * at least *nb_ops* operations, and this is likely to signify that
> +other
> + * processed operations remain in the devices output queue.
> +Applications
> + * implementing a "retrieve as many processed operations as possible"
> +policy
> + * can check this specific case and keep invoking the
> + * rte_cryptodev_dequeue_burst() function until a value less than
> + * *nb_ops* is returned.
> + *
> + * The rte_cryptodev_dequeue_burst() function does not provide any
> +error
> + * notification to avoid the corresponding overhead.
> + *
> + * @param	dev_id		The symmetric crypto device identifier

[Anoob] I do realize this is copied from existing code, but "symmetric" should not be there in the above string, right? The API is equally applicable for all cryptodev operations, right?
 
> + * @param	qp_id		The index of the queue pair from which to
> + *				retrieve processed packets. The value must
> be
> + *				in the range [0, nb_queue_pair - 1]
> previously
> + *				supplied to rte_cryptodev_configure().
> + * @param	ops		The address of an array of pointers to
> + *				*rte_crypto_op* structures that must be
> + *				large enough to store *nb_ops* pointers in
> it.
> + * @param	nb_ops		The maximum number of operations to
> dequeue.
> + *
> + * @return
> + *   - The number of operations actually dequeued, which is the number
> + *   of pointers to *rte_crypto_op* structures effectively supplied to the
> + *   *ops* array.
> + */
> +static inline uint16_t
> +rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
> +		struct rte_crypto_op **ops, uint16_t nb_ops) {
> +	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
> +
> +	rte_cryptodev_trace_dequeue_burst(dev_id, qp_id, (void **)ops,
> nb_ops);
> +	nb_ops = (*dev->dequeue_burst)
> +			(dev->data->queue_pairs[qp_id], ops, nb_ops);
> #ifdef
> +RTE_CRYPTO_CALLBACKS
> +	if (unlikely(dev->deq_cbs != NULL)) {
> +		struct rte_cryptodev_cb_rcu *list;
> +		struct rte_cryptodev_cb *cb;
> +
> +		/* __ATOMIC_RELEASE memory order was used when the
> +		 * call back was inserted into the list.
> +		 * Since there is a clear dependency between loading
> +		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory
> order is
> +		 * not required.
> +		 */
> +		list = &dev->deq_cbs[qp_id];
> +		rte_rcu_qsbr_thread_online(list->qsbr, 0);
> +		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
> +
> +		while (cb != NULL) {
> +			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
> +					cb->arg);
> +			cb = cb->next;
> +		};
> +
> +		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
> +	}
> +#endif
> +	return nb_ops;
> +}
> +
> +/**
> + * Enqueue a burst of operations for processing on a crypto device.
> + *
> + * The rte_cryptodev_enqueue_burst() function is invoked to place
> + * crypto operations on the queue *qp_id* of the device designated by
> + * its *dev_id*.
> + *
> + * The *nb_ops* parameter is the number of operations to process which
> +are
> + * supplied in the *ops* array of *rte_crypto_op* structures.
> + *
> + * The rte_cryptodev_enqueue_burst() function returns the number of
> + * operations it actually enqueued for processing. A return value equal
> +to
> + * *nb_ops* means that all packets have been enqueued.
> + *
> + * @param	dev_id		The identifier of the device.
> + * @param	qp_id		The index of the queue pair which packets
> are
> + *				to be enqueued for processing. The value
> + *				must be in the range [0, nb_queue_pairs - 1]
> + *				previously supplied to
> + *				 *rte_cryptodev_configure*.
> + * @param	ops		The address of an array of *nb_ops* pointers
> + *				to *rte_crypto_op* structures which contain
> + *				the crypto operations to be processed.
> + * @param	nb_ops		The number of operations to process.
> + *
> + * @return
> + * The number of operations actually enqueued on the crypto device. The
> +return
> + * value can be less than the value of the *nb_ops* parameter when the
> + * crypto devices queue is full or if invalid parameters are specified
> +in
> + * a *rte_crypto_op*.
> + */
> +static inline uint16_t
> +rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
> +		struct rte_crypto_op **ops, uint16_t nb_ops) {
> +	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
> +
> +#ifdef RTE_CRYPTO_CALLBACKS
> +	if (unlikely(dev->enq_cbs != NULL)) {
> +		struct rte_cryptodev_cb_rcu *list;
> +		struct rte_cryptodev_cb *cb;
> +
> +		/* __ATOMIC_RELEASE memory order was used when the
> +		 * call back was inserted into the list.
> +		 * Since there is a clear dependency between loading
> +		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory
> order is
> +		 * not required.
> +		 */
> +		list = &dev->enq_cbs[qp_id];
> +		rte_rcu_qsbr_thread_online(list->qsbr, 0);
> +		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
> +
> +		while (cb != NULL) {
> +			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
> +					cb->arg);
> +			cb = cb->next;
> +		};
> +
> +		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
> +	}
> +#endif
> +
> +	rte_cryptodev_trace_enqueue_burst(dev_id, qp_id, (void **)ops,
> nb_ops);
> +	return (*dev->enqueue_burst)(
> +			dev->data->queue_pairs[qp_id], ops, nb_ops); }
> +
> +

[Anoob] Couple of extra blank lines can be removed?
 
> +
>  #ifdef __cplusplus
>  }
>  #endif
> diff --git a/lib/cryptodev/rte_cryptodev_core.h
> b/lib/cryptodev/rte_cryptodev_core.h
> new file mode 100644
> index 0000000000..1633e55889
> --- /dev/null
> +++ b/lib/cryptodev/rte_cryptodev_core.h
> @@ -0,0 +1,100 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(C) 2021 Marvell.

[Anoob] Since the code is mostly a copy from rte_cryptodev.h shouldn't we retain original licenses also?
 
> + */
> +
> +#ifndef _RTE_CRYPTODEV_CORE_H_
> +#define _RTE_CRYPTODEV_CORE_H_

[Anoob] We are not including any of the dependent headers in this file. So the caller would be required to include all the dependent headers. Might be better to include atleast basic required ones (for rte_crypto_op etc).
 
> +
> +/**
> + * @file
> + *
> + * RTE Crypto Device internal header.
> + *
> + * This header contains internal data types. But they are still part of
> +the
> + * public API because they are used by inline functions in the published API.
> + *
> + * Applications should not use these directly.
> + *
> + */
> +
> +typedef uint16_t (*dequeue_pkt_burst_t)(void *qp,
> +		struct rte_crypto_op **ops,	uint16_t nb_ops);
> +/**< Dequeue processed packets from queue pair of a device. */
> +
> +typedef uint16_t (*enqueue_pkt_burst_t)(void *qp,
> +		struct rte_crypto_op **ops,	uint16_t nb_ops);
> +/**< Enqueue packets for processing on queue pair of a device. */
> +
> +/**
> + * @internal
> + * The data part, with no function pointers, associated with each device.
> + *
> + * This structure is safe to place in shared memory to be common among
> + * different processes in a multi-process configuration.
> + */
> +struct rte_cryptodev_data {
> +	uint8_t dev_id;
> +	/**< Device ID for this instance */
> +	uint8_t socket_id;
> +	/**< Socket ID where memory is allocated */
> +	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
> +	/**< Unique identifier name */
> +
> +	__extension__
> +	uint8_t dev_started : 1;
> +	/**< Device state: STARTED(1)/STOPPED(0) */
> +
> +	struct rte_mempool *session_pool;
> +	/**< Session memory pool */
> +	void **queue_pairs;
> +	/**< Array of pointers to queue pairs. */
> +	uint16_t nb_queue_pairs;
> +	/**< Number of device queue pairs. */
> +
> +	void *dev_private;
> +	/**< PMD-specific private data */
> +} __rte_cache_aligned;
> +
> +
> +/** @internal The data structure associated with each crypto device. */
> +struct rte_cryptodev {
> +	dequeue_pkt_burst_t dequeue_burst;
> +	/**< Pointer to PMD receive function. */
> +	enqueue_pkt_burst_t enqueue_burst;
> +	/**< Pointer to PMD transmit function. */

[Anoob] Transmit & receive are more ethdev specific terminology, right? Why not enqueue/dequeue itself?
 
> +
> +	struct rte_cryptodev_data *data;
> +	/**< Pointer to device data */
> +	struct rte_cryptodev_ops *dev_ops;
> +	/**< Functions exported by PMD */
> +	uint64_t feature_flags;
> +	/**< Feature flags exposes HW/SW features for the given device */
> +	struct rte_device *device;
> +	/**< Backing device */
> +
> +	uint8_t driver_id;
> +	/**< Crypto driver identifier*/
> +
> +	struct rte_cryptodev_cb_list link_intr_cbs;
> +	/**< User application callback for interrupts if present */
> +
> +	void *security_ctx;
> +	/**< Context for security ops */
> +
> +	__extension__
> +	uint8_t attached : 1;
> +	/**< Flag indicating the device is attached */
> +
> +	struct rte_cryptodev_cb_rcu *enq_cbs;
> +	/**< User application callback for pre enqueue processing */
> +
> +	struct rte_cryptodev_cb_rcu *deq_cbs;
> +	/**< User application callback for post dequeue processing */ }
> +__rte_cache_aligned;
> +
> +/**
> + * The pool of rte_cryptodev structures.
> + */
> +extern struct rte_cryptodev *rte_cryptodevs;
> +
> +#endif /* _RTE_CRYPTODEV_CORE_H_ */
> --
> 2.25.1

Re: [dpdk-dev] [PATCH 1/8] cryptodev: separate out internal structures

[Akhil Goyal]

Hi Anoob,
Please see inline.

> >
> > +#include <rte_cryptodev_core.h>
> 
> [Anoob] Is this intentional? Shouldn't we keep includes together in the top of
> the file?
> 
It is intentional, and will be removed in a later patch, when the new framework is ready.
> > +/**
> > + *
> 
> [Anoob] Is there an extra blank line?
> 
> > + * Dequeue a burst of processed crypto operations from a queue on the
> > +crypto
> > + * device. The dequeued operation are stored in *rte_crypto_op*
> > +structures
> > + * whose pointers are supplied in the *ops* array.
> > + *
> > + * The rte_cryptodev_dequeue_burst() function returns the number of
> ops
> > + * actually dequeued, which is the number of *rte_crypto_op* data
> > +structures
> > + * effectively supplied into the *ops* array.
> > + *
> > + * A return value equal to *nb_ops* indicates that the queue contained
> > + * at least *nb_ops* operations, and this is likely to signify that
> > +other
> > + * processed operations remain in the devices output queue.
> > +Applications
> > + * implementing a "retrieve as many processed operations as possible"
> > +policy
> > + * can check this specific case and keep invoking the
> > + * rte_cryptodev_dequeue_burst() function until a value less than
> > + * *nb_ops* is returned.
> > + *
> > + * The rte_cryptodev_dequeue_burst() function does not provide any
> > +error
> > + * notification to avoid the corresponding overhead.
> > + *
> > + * @param	dev_id		The symmetric crypto device identifier
> 
> [Anoob] I do realize this is copied from existing code, but "symmetric" should
> not be there in the above string, right? The API is equally applicable for all
> cryptodev operations, right?
Agreed, I did not realize it was like this till now. This code will be removed
In a later patch of the series, but will check the final thing.

> 
> > +
> > +
> 
> [Anoob] Couple of extra blank lines can be removed?
This code is removed in a later patch.
> 
> > +
> >  #ifdef __cplusplus
> >  }
> >  #endif
> > diff --git a/lib/cryptodev/rte_cryptodev_core.h
> > b/lib/cryptodev/rte_cryptodev_core.h
> > new file mode 100644
> > index 0000000000..1633e55889
> > --- /dev/null
> > +++ b/lib/cryptodev/rte_cryptodev_core.h
> > @@ -0,0 +1,100 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(C) 2021 Marvell.
> 
> [Anoob] Since the code is mostly a copy from rte_cryptodev.h shouldn't we
> retain original licenses also?

This is an intermediate patch. At the end of this patchset, the new file will be 
A very small file with stuff only related to the new framework and will not have
Anything copied from other files, hence did not mention previous licenses.

> 
> > + */
> > +
> > +#ifndef _RTE_CRYPTODEV_CORE_H_
> > +#define _RTE_CRYPTODEV_CORE_H_
> 
> [Anoob] We are not including any of the dependent headers in this file. So
> the caller would be required to include all the dependent headers. Might be
> better to include atleast basic required ones (for rte_crypto_op etc).

This file is not required to be included directly as mentioned at the top of the file.
Hence no need to include other header files.
> 


> > +/** @internal The data structure associated with each crypto device. */
> > +struct rte_cryptodev {
> > +	dequeue_pkt_burst_t dequeue_burst;
> > +	/**< Pointer to PMD receive function. */
> > +	enqueue_pkt_burst_t enqueue_burst;
> > +	/**< Pointer to PMD transmit function. */
> 
> [Anoob] Transmit & receive are more ethdev specific terminology, right? Why
> not enqueue/dequeue itself?
This is kind of a legacy code, I just copied. We need not correct here, as this code
Will be removed in a later patch.

Re: [dpdk-dev] [PATCH 0/8] cryptodev: hide internal strutures

[Zhang, Roy Fan]

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Sunday, August 29, 2021 1:52 PM
> To: dev@dpdk.org
> Cc: anoobj@marvell.com; Nicolau, Radu <radu.nicolau@intel.com>; Doherty,
> Declan <declan.doherty@intel.com>; hemant.agrawal@nxp.com;
> matan@nvidia.com; Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> thomas@monjalon.net; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
> asomalap@amd.com; ruifeng.wang@arm.com;
> ajit.khaparde@broadcom.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> adwivedi@marvell.com; michaelsh@marvell.com;
> rnagadheeraj@marvell.com; jianjay.zhou@huawei.com; jerinj@marvell.com;
> Akhil Goyal <gakhil@marvell.com>
> Subject: [PATCH 0/8] cryptodev: hide internal strutures
> 
> Structures rte_cryptodev and rte_cryptodev_data are not
> supposed to be directly used by the application. These
> are made public as they are used by inline datapath
> public APIs.
> This patchset, creates a new rte_cryptodev_core.h file
> which helps in defining a data structure to hold datapath
> APIs in a flat array based on the device identifier which
> is filled by the PMD.
> The patchset updates all crypto PMDs to use the new
> framework and hence the internal structs are made hidden.
> 
> Similar series for ethdev and eventdev are also floated on ML.
> http://patches.dpdk.org/project/dpdk/list/?series=18382
> http://patches.dpdk.org/project/dpdk/list/?series=18422
> 
> 
> Akhil Goyal (8):
>   cryptodev: separate out internal structures
>   cryptodev: move inline APIs into separate structure
>   cryptodev: add helper functions for new datapath interface
>   cryptodev: use new API for datapath functions
>   drivers/crypto: use new framework for datapath
>   crypto/scheduler: rename enq-deq functions
>   crypto/scheduler: update for new datapath framework
>   cryptodev: move device specific structures
> 
>  drivers/crypto/aesni_gcm/aesni_gcm_pmd.c      |  10 +-
>  drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c    |  11 +-
>  drivers/crypto/armv8/rte_armv8_pmd.c          |  11 +-
>  drivers/crypto/bcmfs/bcmfs_sym_pmd.c          |  11 +-
>  drivers/crypto/caam_jr/caam_jr.c              |  11 +-
>  drivers/crypto/ccp/ccp_dev.c                  |   1 +
>  drivers/crypto/ccp/rte_ccp_pmd.c              |  11 +-
>  drivers/crypto/cnxk/cn10k_cryptodev_ops.c     |   8 +-
>  drivers/crypto/cnxk/cn10k_cryptodev_ops.h     |   3 +
>  drivers/crypto/cnxk/cn10k_ipsec.c             |   1 +
>  drivers/crypto/cnxk/cn9k_cryptodev_ops.c      |   9 +-
>  drivers/crypto/cnxk/cn9k_cryptodev_ops.h      |   3 +
>  .../crypto/cnxk/cnxk_cryptodev_capabilities.c |   1 +
>  drivers/crypto/cnxk/cnxk_cryptodev_sec.c      |   1 +
>  drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c   |  12 +-
>  drivers/crypto/dpaa_sec/dpaa_sec.c            |  11 +-
>  drivers/crypto/kasumi/rte_kasumi_pmd.c        |  11 +-
>  drivers/crypto/mlx5/mlx5_crypto.c             |  11 +-
>  drivers/crypto/mvsam/rte_mrvl_pmd.c           |  11 +-
>  drivers/crypto/nitrox/nitrox_sym.c            |  11 +-
>  drivers/crypto/nitrox/nitrox_sym_reqmgr.c     |   1 +
>  drivers/crypto/null/null_crypto_pmd.c         |  11 +-
>  .../crypto/octeontx/otx_cryptodev_hw_access.c |   1 +
>  drivers/crypto/octeontx/otx_cryptodev_ops.c   |  16 +-
>  drivers/crypto/octeontx/otx_cryptodev_ops.h   |   5 +
>  .../crypto/octeontx2/otx2_cryptodev_mbox.c    |   1 +
>  drivers/crypto/octeontx2/otx2_cryptodev_ops.c |  11 +-
>  drivers/crypto/openssl/rte_openssl_pmd.c      |  11 +-
>  drivers/crypto/qat/qat_asym_pmd.c             |  11 +-
>  drivers/crypto/qat/qat_sym_pmd.c              |  10 +-
>  drivers/crypto/scheduler/scheduler_failover.c |  35 +-
>  .../crypto/scheduler/scheduler_multicore.c    |  32 +-
>  .../scheduler/scheduler_pkt_size_distr.c      |  34 +-
>  .../crypto/scheduler/scheduler_roundrobin.c   |  34 +-
>  drivers/crypto/snow3g/rte_snow3g_pmd.c        |  11 +-
>  drivers/crypto/virtio/virtio_cryptodev.c      |  10 +-
>  drivers/crypto/virtio/virtio_cryptodev.h      |   2 +
>  drivers/crypto/virtio/virtio_rxtx.c           |   2 +
>  drivers/crypto/zuc/rte_zuc_pmd.c              |  11 +-
>  lib/cryptodev/cryptodev_pmd.c                 |  33 ++
>  lib/cryptodev/cryptodev_pmd.h                 | 323 +++++++++++++++++-
>  lib/cryptodev/meson.build                     |   4 +-
>  lib/cryptodev/rte_cryptodev.c                 |  43 ++-
>  lib/cryptodev/rte_cryptodev.h                 | 308 +++++------------
>  lib/cryptodev/rte_cryptodev_core.h            |  43 +++
>  lib/cryptodev/version.map                     |   8 +
>  46 files changed, 854 insertions(+), 316 deletions(-)
>  create mode 100644 lib/cryptodev/rte_cryptodev_core.h
> 
> --
> 2.25.1

Series-tested-by: Rebecca Troy <rebecca.troy@intel.com>
Series-acked-by: Fan Zhang <roy.fan.zhang@intel.com>

Re: [dpdk-dev] [PATCH 1/8] cryptodev: separate out internal structures

[Zhang, Roy Fan]

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Sunday, August 29, 2021 1:52 PM
> To: dev@dpdk.org
> Cc: anoobj@marvell.com; Nicolau, Radu <radu.nicolau@intel.com>; Doherty,
> Declan <declan.doherty@intel.com>; hemant.agrawal@nxp.com;
> matan@nvidia.com; Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> thomas@monjalon.net; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
> asomalap@amd.com; ruifeng.wang@arm.com;
> ajit.khaparde@broadcom.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> adwivedi@marvell.com; michaelsh@marvell.com;
> rnagadheeraj@marvell.com; jianjay.zhou@huawei.com; jerinj@marvell.com;
> Akhil Goyal <gakhil@marvell.com>
> Subject: [PATCH 1/8] cryptodev: separate out internal structures
> 
Tested-by: Rebecca Troy <rebecca.troy@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>

Re: [dpdk-dev] [PATCH 2/8] cryptodev: move inline APIs into separate structure

[Zhang, Roy Fan]

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Sunday, August 29, 2021 1:52 PM
> To: dev@dpdk.org
> Cc: anoobj@marvell.com; Nicolau, Radu <radu.nicolau@intel.com>; Doherty,
> Declan <declan.doherty@intel.com>; hemant.agrawal@nxp.com;
> matan@nvidia.com; Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> thomas@monjalon.net; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
> asomalap@amd.com; ruifeng.wang@arm.com;
> ajit.khaparde@broadcom.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> adwivedi@marvell.com; michaelsh@marvell.com;
> rnagadheeraj@marvell.com; jianjay.zhou@huawei.com; jerinj@marvell.com;
> Akhil Goyal <gakhil@marvell.com>
> Subject: [PATCH 2/8] cryptodev: move inline APIs into separate structure
> 
> Move fastpath inline function pointers from rte_cryptodev into a
> separate structure accessed via a flat array.
> The intension is to make rte_cryptodev and related structures private
> to avoid future API/ABI breakages.
> 
> Signed-off-by: Akhil Goyal <gakhil@marvell.com>
> ---
Tested-by: Rebecca Troy <rebecca.troy@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>

Re: [dpdk-dev] [PATCH 3/8] cryptodev: add helper functions for new datapath interface

[Zhang, Roy Fan]

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Sunday, August 29, 2021 1:52 PM
> To: dev@dpdk.org
> Cc: anoobj@marvell.com; Nicolau, Radu <radu.nicolau@intel.com>; Doherty,
> Declan <declan.doherty@intel.com>; hemant.agrawal@nxp.com;
> matan@nvidia.com; Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> thomas@monjalon.net; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
> asomalap@amd.com; ruifeng.wang@arm.com;
> ajit.khaparde@broadcom.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> adwivedi@marvell.com; michaelsh@marvell.com;
> rnagadheeraj@marvell.com; jianjay.zhou@huawei.com; jerinj@marvell.com;
> Akhil Goyal <gakhil@marvell.com>
> Subject: [PATCH 3/8] cryptodev: add helper functions for new datapath
> interface
> 
> Add helper functions and macros to help drivers to
> transition to new datapath interface.
> 
> Signed-off-by: Akhil Goyal <gakhil@marvell.com>
> ---

Tested-by: Rebecca Troy <rebecca.troy@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>

Re: [dpdk-dev] [PATCH 4/8] cryptodev: use new API for datapath functions

[Zhang, Roy Fan]

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Sunday, August 29, 2021 1:52 PM
> To: dev@dpdk.org
> Cc: anoobj@marvell.com; Nicolau, Radu <radu.nicolau@intel.com>; Doherty,
> Declan <declan.doherty@intel.com>; hemant.agrawal@nxp.com;
> matan@nvidia.com; Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> thomas@monjalon.net; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
> asomalap@amd.com; ruifeng.wang@arm.com;
> ajit.khaparde@broadcom.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> adwivedi@marvell.com; michaelsh@marvell.com;
> rnagadheeraj@marvell.com; jianjay.zhou@huawei.com; jerinj@marvell.com;
> Akhil Goyal <gakhil@marvell.com>
> Subject: [PATCH 4/8] cryptodev: use new API for datapath functions
> 
> The datapath inline APIs (rte_cryptodev_enqueue_burst/
> rte_cryptodev_dequeue_burst) are updated to use the new
> rte_crytodev_api->enqueue_burst/rte_cryptodev_api->dequeue_burst
> APIs based on the dev_id
> 
> Signed-off-by: Akhil Goyal <gakhil@marvell.com>
Tested-by: Rebecca Troy <rebecca.troy@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>

Re: [dpdk-dev] [PATCH 5/8] drivers/crypto: use new framework for datapath

[Zhang, Roy Fan]

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Sunday, August 29, 2021 1:52 PM
> To: dev@dpdk.org
> Cc: anoobj@marvell.com; Nicolau, Radu <radu.nicolau@intel.com>; Doherty,
> Declan <declan.doherty@intel.com>; hemant.agrawal@nxp.com;
> matan@nvidia.com; Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> thomas@monjalon.net; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
> asomalap@amd.com; ruifeng.wang@arm.com;
> ajit.khaparde@broadcom.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> adwivedi@marvell.com; michaelsh@marvell.com;
> rnagadheeraj@marvell.com; jianjay.zhou@huawei.com; jerinj@marvell.com;
> Akhil Goyal <gakhil@marvell.com>
> Subject: [PATCH 5/8] drivers/crypto: use new framework for datapath
> 
> All crypto drivers are updated to use the new API
> for all enqueue and dequeue paths.
> 
> Signed-off-by: Akhil Goyal <gakhil@marvell.com>
Tested-by: Rebecca Troy <rebecca.troy@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>

Re: [dpdk-dev] [PATCH 6/8] crypto/scheduler: rename enq-deq functions

[Zhang, Roy Fan]

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Sunday, August 29, 2021 1:52 PM
> To: dev@dpdk.org
> Cc: anoobj@marvell.com; Nicolau, Radu <radu.nicolau@intel.com>; Doherty,
> Declan <declan.doherty@intel.com>; hemant.agrawal@nxp.com;
> matan@nvidia.com; Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> thomas@monjalon.net; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
> asomalap@amd.com; ruifeng.wang@arm.com;
> ajit.khaparde@broadcom.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> adwivedi@marvell.com; michaelsh@marvell.com;
> rnagadheeraj@marvell.com; jianjay.zhou@huawei.com; jerinj@marvell.com;
> Akhil Goyal <gakhil@marvell.com>
> Subject: [PATCH 6/8] crypto/scheduler: rename enq-deq functions
> 
> scheduler PMD has 4 variants, which uses same
> name for all the enqueue and dequeue functions.
> This causes multiple definitions of same function
> with the new framework of datapath APIs.
> Hence the function names are updated to specify the
> the variant it is for.
> 
> Signed-off-by: Akhil Goyal <gakhil@marvell.com>
> ---
Tested-by: Rebecca Troy <rebecca.troy@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>

Re: [dpdk-dev] [PATCH 7/8] crypto/scheduler: update for new datapath framework

[Zhang, Roy Fan]

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Sunday, August 29, 2021 1:52 PM
> To: dev@dpdk.org
> Cc: anoobj@marvell.com; Nicolau, Radu <radu.nicolau@intel.com>; Doherty,
> Declan <declan.doherty@intel.com>; hemant.agrawal@nxp.com;
> matan@nvidia.com; Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> thomas@monjalon.net; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
> asomalap@amd.com; ruifeng.wang@arm.com;
> ajit.khaparde@broadcom.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> adwivedi@marvell.com; michaelsh@marvell.com;
> rnagadheeraj@marvell.com; jianjay.zhou@huawei.com; jerinj@marvell.com;
> Akhil Goyal <gakhil@marvell.com>
> Subject: [PATCH 7/8] crypto/scheduler: update for new datapath framework
> 
> PMD is updated to use the new API for all enqueue
> and dequeue paths.
> 
Tested-by: Rebecca Troy <rebecca.troy@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>

Re: [dpdk-dev] [PATCH 8/8] cryptodev: move device specific structures

[Zhang, Roy Fan]

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Sunday, August 29, 2021 1:52 PM
> To: dev@dpdk.org
> Cc: anoobj@marvell.com; Nicolau, Radu <radu.nicolau@intel.com>; Doherty,
> Declan <declan.doherty@intel.com>; hemant.agrawal@nxp.com;
> matan@nvidia.com; Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> thomas@monjalon.net; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
> asomalap@amd.com; ruifeng.wang@arm.com;
> ajit.khaparde@broadcom.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> adwivedi@marvell.com; michaelsh@marvell.com;
> rnagadheeraj@marvell.com; jianjay.zhou@huawei.com; jerinj@marvell.com;
> Akhil Goyal <gakhil@marvell.com>
> Subject: [PATCH 8/8] cryptodev: move device specific structures
> 
> The device specific structures - rte_cryptodev
> and rte_cryptodev_data are moved to cryptodev_pmd.h
> to hide it from the applications.
> 
> Signed-off-by: Akhil Goyal <gakhil@marvell.com>
> ---
Tested-by: Rebecca Troy <rebecca.troy@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>

Re: [dpdk-dev] [PATCH 2/8] cryptodev: move inline APIs into separate structure

[Ananyev, Konstantin]

Hi Akhil,

Overall, looks good to me.
Few comments below.
Konstantin

> Move fastpath inline function pointers from rte_cryptodev into a
> separate structure accessed via a flat array.
> The intension is to make rte_cryptodev and related structures private
> to avoid future API/ABI breakages.
> 
> Signed-off-by: Akhil Goyal <gakhil@marvell.com>
> ---
>  lib/cryptodev/cryptodev_pmd.c      | 33 ++++++++++++++++++++++++++++++
>  lib/cryptodev/cryptodev_pmd.h      |  9 ++++++++
>  lib/cryptodev/rte_cryptodev.c      |  3 +++
>  lib/cryptodev/rte_cryptodev_core.h | 19 +++++++++++++++++
>  lib/cryptodev/version.map          |  4 ++++
>  5 files changed, 68 insertions(+)
> 
> diff --git a/lib/cryptodev/cryptodev_pmd.c b/lib/cryptodev/cryptodev_pmd.c
> index 71e34140cd..46772dc355 100644
> --- a/lib/cryptodev/cryptodev_pmd.c
> +++ b/lib/cryptodev/cryptodev_pmd.c
> @@ -158,3 +158,36 @@ rte_cryptodev_pmd_destroy(struct rte_cryptodev *cryptodev)
> 
>  	return 0;
>  }
> +
> +static uint16_t
> +dummy_crypto_enqueue_burst(__rte_unused uint8_t dev_id,
> +			  __rte_unused uint8_t qp_id,
> +			  __rte_unused struct rte_crypto_op **ops,
> +			  __rte_unused uint16_t nb_ops)
> +{
> +	CDEV_LOG_ERR(
> +		"crypto enqueue burst requested for unconfigured crypto device");
> +	return 0;
> +}
> +
> +static uint16_t
> +dummy_crypto_dequeue_burst(__rte_unused uint8_t dev_id,
> +			  __rte_unused uint8_t qp_id,
> +			  __rte_unused struct rte_crypto_op **ops,
> +			  __rte_unused uint16_t nb_ops)
> +{
> +	CDEV_LOG_ERR(
> +		"crypto enqueue burst requested for unconfigured crypto device");
> +	return 0;
> +}
> +
> +void
> +rte_cryptodev_api_reset(struct rte_cryptodev_api *api)
> +{
> +	static const struct rte_cryptodev_api dummy = {
> +		.enqueue_burst = dummy_crypto_enqueue_burst,
> +		.dequeue_burst = dummy_crypto_dequeue_burst,
> +	};
> +
> +	*api = dummy;
> +}
> diff --git a/lib/cryptodev/cryptodev_pmd.h b/lib/cryptodev/cryptodev_pmd.h
> index f775ba6beb..eeaea13a23 100644
> --- a/lib/cryptodev/cryptodev_pmd.h
> +++ b/lib/cryptodev/cryptodev_pmd.h
> @@ -520,6 +520,15 @@ RTE_INIT(init_ ##driver_id)\
>  	driver_id = rte_cryptodev_allocate_driver(&crypto_drv, &(drv));\
>  }
> 
> +/**
> + * Reset crypto device fastpath APIs to dummy values.
> + *
> + * @param The *api* pointer to reset.
> + */
> +__rte_internal
> +void
> +rte_cryptodev_api_reset(struct rte_cryptodev_api *api);
> +
>  static inline void *
>  get_sym_session_private_data(const struct rte_cryptodev_sym_session *sess,
>  		uint8_t driver_id) {
> diff --git a/lib/cryptodev/rte_cryptodev.c b/lib/cryptodev/rte_cryptodev.c
> index 9fa3aff1d3..26f8390668 100644
> --- a/lib/cryptodev/rte_cryptodev.c
> +++ b/lib/cryptodev/rte_cryptodev.c
> @@ -54,6 +54,9 @@ static struct rte_cryptodev_global cryptodev_globals = {
>  		.nb_devs		= 0
>  };
> 
> +/* Public fastpath APIs. */
> +struct rte_cryptodev_api *rte_cryptodev_api;

I think it has to be an statically allocated array:
struct rte_cryptodev_api rte_cryptodev_api[RTE_CRYPTO_MAX_DEVS];
Other alternative would be to allocate space for it at some RTE_INIT() function,
but not sure is it really worth it.

Another thing - to make things a bit more error prone - probably
need to fill it with dummy pointers at RTE_INIT timeframe.
Also, I think we need to call rte_cryptodev_api_reset() at 
rte_cryptodev_pmd_destroy() or so.

> +
>  /* spinlock for crypto device callbacks */
>  static rte_spinlock_t rte_cryptodev_cb_lock = RTE_SPINLOCK_INITIALIZER;
> 
> diff --git a/lib/cryptodev/rte_cryptodev_core.h b/lib/cryptodev/rte_cryptodev_core.h
> index 1633e55889..ec38f70e0c 100644
> --- a/lib/cryptodev/rte_cryptodev_core.h
> +++ b/lib/cryptodev/rte_cryptodev_core.h
> @@ -25,6 +25,25 @@ typedef uint16_t (*enqueue_pkt_burst_t)(void *qp,
>  		struct rte_crypto_op **ops,	uint16_t nb_ops);
>  /**< Enqueue packets for processing on queue pair of a device. */
> 
> +typedef uint16_t (*rte_crypto_dequeue_burst_t)(uint8_t dev_id, uint8_t qp_id,
> +					      struct rte_crypto_op **ops,
> +					      uint16_t nb_ops);
> +/**< @internal Dequeue processed packets from queue pair of a device. */
> +typedef uint16_t (*rte_crypto_enqueue_burst_t)(uint8_t dev_id, uint8_t qp_id,
> +					      struct rte_crypto_op **ops,
> +					      uint16_t nb_ops);
> +/**< @internal Enqueue packets for processing on queue pair of a device. */
> +
> +struct rte_cryptodev_api {
> +	rte_crypto_enqueue_burst_t enqueue_burst;
> +	/**< PMD enqueue burst function. */
> +	rte_crypto_dequeue_burst_t dequeue_burst;
> +	/**< PMD dequeue burst function. */
> +	uintptr_t reserved[6];
> +} __rte_cache_aligned;
> +
> +extern struct rte_cryptodev_api *rte_cryptodev_api;
> +
>  /**
>   * @internal
>   * The data part, with no function pointers, associated with each device.
> diff --git a/lib/cryptodev/version.map b/lib/cryptodev/version.map
> index 2fdf70002d..050089ae55 100644
> --- a/lib/cryptodev/version.map
> +++ b/lib/cryptodev/version.map
> @@ -57,6 +57,9 @@ DPDK_22 {
>  	rte_cryptodev_sym_session_init;
>  	rte_cryptodevs;
> 
> +	#added in 21.11
> +	rte_cryptodev_api;
> +
>  	local: *;
>  };
> 
> @@ -114,6 +117,7 @@ INTERNAL {
>  	global:
> 
>  	rte_cryptodev_allocate_driver;
> +	rte_cryptodev_api_reset;
>  	rte_cryptodev_pmd_allocate;
>  	rte_cryptodev_pmd_callback_process;
>  	rte_cryptodev_pmd_create;
> --
> 2.25.1

[dpdk-dev] [PATCH v2 0/5] cryptodev: hide internal structures

[Akhil Goyal]

Structures rte_cryptodev and rte_cryptodev_data are not
supposed to be directly used by the application. These
are made public as they are used by inline datapath
public APIs.
This patchset, creates a new rte_cryptodev_core.h file
which helps in defining a data structure to hold datapath
APIs in a flat array based on the device identifier which
is filled by the PMD.

Similar series for ethdev and eventdev are also floated on ML.
https://patchwork.dpdk.org/project/dpdk/list/?series=19428
https://patchwork.dpdk.org/project/dpdk/list/?series=19405

changes in v2: align with the latest versions of above series.

Akhil Goyal (5):
  cryptodev: separate out internal structures
  cryptodev: allocate max space for internal qp array
  cryptodev: move inline APIs into separate structure
  cryptodev: update fast path APIs to use new flat array
  cryptodev: move device specific structures

 drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c    |   1 -
 drivers/crypto/ccp/ccp_dev.h                  |   2 +-
 drivers/crypto/cnxk/cn10k_ipsec.c             |   2 +-
 drivers/crypto/cnxk/cn9k_ipsec.c              |   2 +-
 .../crypto/cnxk/cnxk_cryptodev_capabilities.c |   2 +-
 drivers/crypto/cnxk/cnxk_cryptodev_sec.c      |   2 +-
 drivers/crypto/nitrox/nitrox_sym_reqmgr.c     |   2 +-
 drivers/crypto/octeontx/otx_cryptodev.c       |   1 -
 .../crypto/octeontx/otx_cryptodev_hw_access.c |   2 +-
 .../crypto/octeontx/otx_cryptodev_hw_access.h |   2 +-
 drivers/crypto/octeontx/otx_cryptodev_ops.h   |   2 +-
 .../crypto/octeontx2/otx2_cryptodev_mbox.c    |   2 +-
 drivers/crypto/scheduler/scheduler_failover.c |   2 +-
 .../crypto/scheduler/scheduler_multicore.c    |   2 +-
 .../scheduler/scheduler_pkt_size_distr.c      |   2 +-
 .../crypto/scheduler/scheduler_roundrobin.c   |   2 +-
 drivers/event/cnxk/cnxk_eventdev.h            |   2 +-
 drivers/event/dpaa/dpaa_eventdev.c            |   2 +-
 drivers/event/dpaa2/dpaa2_eventdev.c          |   2 +-
 drivers/event/octeontx/ssovf_evdev.c          |   2 +-
 .../event/octeontx2/otx2_evdev_crypto_adptr.c |   2 +-
 lib/cryptodev/cryptodev_pmd.c                 |  51 +++
 lib/cryptodev/cryptodev_pmd.h                 |  82 +++-
 lib/cryptodev/meson.build                     |   4 +-
 lib/cryptodev/rte_cryptodev.c                 |  50 ++-
 lib/cryptodev/rte_cryptodev.h                 | 367 +++++++-----------
 lib/cryptodev/rte_cryptodev_core.h            |  62 +++
 lib/cryptodev/version.map                     |   7 +-
 28 files changed, 398 insertions(+), 265 deletions(-)
 create mode 100644 lib/cryptodev/rte_cryptodev_core.h

-- 
2.25.1

[dpdk-dev] [PATCH v2 1/5] cryptodev: separate out internal structures

[Akhil Goyal]

A new header file rte_cryptodev_core.h is added and all
internal data structures which need not be exposed directly to
application are moved to this file. These structures are mostly
used by drivers, but they need to be in the public header file
as they are accessed by datapath inline functions for
performance reasons.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
---
 lib/cryptodev/cryptodev_pmd.h      |   6 -
 lib/cryptodev/meson.build          |   4 +-
 lib/cryptodev/rte_cryptodev.h      | 360 ++++++++++++-----------------
 lib/cryptodev/rte_cryptodev_core.h | 100 ++++++++
 4 files changed, 245 insertions(+), 225 deletions(-)
 create mode 100644 lib/cryptodev/rte_cryptodev_core.h

diff --git a/lib/cryptodev/cryptodev_pmd.h b/lib/cryptodev/cryptodev_pmd.h
index 8cc9051e09..36606dd10b 100644
--- a/lib/cryptodev/cryptodev_pmd.h
+++ b/lib/cryptodev/cryptodev_pmd.h
@@ -92,12 +92,6 @@ __rte_internal
 struct rte_cryptodev *
 rte_cryptodev_pmd_get_named_dev(const char *name);
 
-/**
- * The pool of rte_cryptodev structures.
- */
-extern struct rte_cryptodev *rte_cryptodevs;
-
-
 /**
  * Definitions of all functions exported by a driver through the
  * the generic structure of type *crypto_dev_ops* supplied in the
diff --git a/lib/cryptodev/meson.build b/lib/cryptodev/meson.build
index 51371c3aa2..289b66ab76 100644
--- a/lib/cryptodev/meson.build
+++ b/lib/cryptodev/meson.build
@@ -14,7 +14,9 @@ headers = files(
         'rte_crypto_sym.h',
         'rte_crypto_asym.h',
 )
-
+indirect_headers += files(
+        'rte_cryptodev_core.h',
+)
 driver_sdk_headers += files(
         'cryptodev_pmd.h',
 )
diff --git a/lib/cryptodev/rte_cryptodev.h b/lib/cryptodev/rte_cryptodev.h
index cdd7168fba..ce0dca72be 100644
--- a/lib/cryptodev/rte_cryptodev.h
+++ b/lib/cryptodev/rte_cryptodev.h
@@ -867,17 +867,6 @@ rte_cryptodev_callback_unregister(uint8_t dev_id,
 		enum rte_cryptodev_event_type event,
 		rte_cryptodev_cb_fn cb_fn, void *cb_arg);
 
-typedef uint16_t (*dequeue_pkt_burst_t)(void *qp,
-		struct rte_crypto_op **ops,	uint16_t nb_ops);
-/**< Dequeue processed packets from queue pair of a device. */
-
-typedef uint16_t (*enqueue_pkt_burst_t)(void *qp,
-		struct rte_crypto_op **ops,	uint16_t nb_ops);
-/**< Enqueue packets for processing on queue pair of a device. */
-
-
-
-
 struct rte_cryptodev_callback;
 
 /** Structure to keep track of registered callbacks */
@@ -907,216 +896,9 @@ struct rte_cryptodev_cb_rcu {
 	/**< RCU QSBR variable per queue pair */
 };
 
-/** The data structure associated with each crypto device. */
-struct rte_cryptodev {
-	dequeue_pkt_burst_t dequeue_burst;
-	/**< Pointer to PMD receive function. */
-	enqueue_pkt_burst_t enqueue_burst;
-	/**< Pointer to PMD transmit function. */
-
-	struct rte_cryptodev_data *data;
-	/**< Pointer to device data */
-	struct rte_cryptodev_ops *dev_ops;
-	/**< Functions exported by PMD */
-	uint64_t feature_flags;
-	/**< Feature flags exposes HW/SW features for the given device */
-	struct rte_device *device;
-	/**< Backing device */
-
-	uint8_t driver_id;
-	/**< Crypto driver identifier*/
-
-	struct rte_cryptodev_cb_list link_intr_cbs;
-	/**< User application callback for interrupts if present */
-
-	void *security_ctx;
-	/**< Context for security ops */
-
-	__extension__
-	uint8_t attached : 1;
-	/**< Flag indicating the device is attached */
-
-	struct rte_cryptodev_cb_rcu *enq_cbs;
-	/**< User application callback for pre enqueue processing */
-
-	struct rte_cryptodev_cb_rcu *deq_cbs;
-	/**< User application callback for post dequeue processing */
-} __rte_cache_aligned;
-
 void *
 rte_cryptodev_get_sec_ctx(uint8_t dev_id);
 
-/**
- *
- * The data part, with no function pointers, associated with each device.
- *
- * This structure is safe to place in shared memory to be common among
- * different processes in a multi-process configuration.
- */
-struct rte_cryptodev_data {
-	uint8_t dev_id;
-	/**< Device ID for this instance */
-	uint8_t socket_id;
-	/**< Socket ID where memory is allocated */
-	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
-	/**< Unique identifier name */
-
-	__extension__
-	uint8_t dev_started : 1;
-	/**< Device state: STARTED(1)/STOPPED(0) */
-
-	struct rte_mempool *session_pool;
-	/**< Session memory pool */
-	void **queue_pairs;
-	/**< Array of pointers to queue pairs. */
-	uint16_t nb_queue_pairs;
-	/**< Number of device queue pairs. */
-
-	void *dev_private;
-	/**< PMD-specific private data */
-} __rte_cache_aligned;
-
-extern struct rte_cryptodev *rte_cryptodevs;
-/**
- *
- * Dequeue a burst of processed crypto operations from a queue on the crypto
- * device. The dequeued operation are stored in *rte_crypto_op* structures
- * whose pointers are supplied in the *ops* array.
- *
- * The rte_cryptodev_dequeue_burst() function returns the number of ops
- * actually dequeued, which is the number of *rte_crypto_op* data structures
- * effectively supplied into the *ops* array.
- *
- * A return value equal to *nb_ops* indicates that the queue contained
- * at least *nb_ops* operations, and this is likely to signify that other
- * processed operations remain in the devices output queue. Applications
- * implementing a "retrieve as many processed operations as possible" policy
- * can check this specific case and keep invoking the
- * rte_cryptodev_dequeue_burst() function until a value less than
- * *nb_ops* is returned.
- *
- * The rte_cryptodev_dequeue_burst() function does not provide any error
- * notification to avoid the corresponding overhead.
- *
- * @param	dev_id		The symmetric crypto device identifier
- * @param	qp_id		The index of the queue pair from which to
- *				retrieve processed packets. The value must be
- *				in the range [0, nb_queue_pair - 1] previously
- *				supplied to rte_cryptodev_configure().
- * @param	ops		The address of an array of pointers to
- *				*rte_crypto_op* structures that must be
- *				large enough to store *nb_ops* pointers in it.
- * @param	nb_ops		The maximum number of operations to dequeue.
- *
- * @return
- *   - The number of operations actually dequeued, which is the number
- *   of pointers to *rte_crypto_op* structures effectively supplied to the
- *   *ops* array.
- */
-static inline uint16_t
-rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
-		struct rte_crypto_op **ops, uint16_t nb_ops)
-{
-	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
-
-	rte_cryptodev_trace_dequeue_burst(dev_id, qp_id, (void **)ops, nb_ops);
-	nb_ops = (*dev->dequeue_burst)
-			(dev->data->queue_pairs[qp_id], ops, nb_ops);
-#ifdef RTE_CRYPTO_CALLBACKS
-	if (unlikely(dev->deq_cbs != NULL)) {
-		struct rte_cryptodev_cb_rcu *list;
-		struct rte_cryptodev_cb *cb;
-
-		/* __ATOMIC_RELEASE memory order was used when the
-		 * call back was inserted into the list.
-		 * Since there is a clear dependency between loading
-		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
-		 * not required.
-		 */
-		list = &dev->deq_cbs[qp_id];
-		rte_rcu_qsbr_thread_online(list->qsbr, 0);
-		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
-
-		while (cb != NULL) {
-			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
-					cb->arg);
-			cb = cb->next;
-		};
-
-		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
-	}
-#endif
-	return nb_ops;
-}
-
-/**
- * Enqueue a burst of operations for processing on a crypto device.
- *
- * The rte_cryptodev_enqueue_burst() function is invoked to place
- * crypto operations on the queue *qp_id* of the device designated by
- * its *dev_id*.
- *
- * The *nb_ops* parameter is the number of operations to process which are
- * supplied in the *ops* array of *rte_crypto_op* structures.
- *
- * The rte_cryptodev_enqueue_burst() function returns the number of
- * operations it actually enqueued for processing. A return value equal to
- * *nb_ops* means that all packets have been enqueued.
- *
- * @param	dev_id		The identifier of the device.
- * @param	qp_id		The index of the queue pair which packets are
- *				to be enqueued for processing. The value
- *				must be in the range [0, nb_queue_pairs - 1]
- *				previously supplied to
- *				 *rte_cryptodev_configure*.
- * @param	ops		The address of an array of *nb_ops* pointers
- *				to *rte_crypto_op* structures which contain
- *				the crypto operations to be processed.
- * @param	nb_ops		The number of operations to process.
- *
- * @return
- * The number of operations actually enqueued on the crypto device. The return
- * value can be less than the value of the *nb_ops* parameter when the
- * crypto devices queue is full or if invalid parameters are specified in
- * a *rte_crypto_op*.
- */
-static inline uint16_t
-rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
-		struct rte_crypto_op **ops, uint16_t nb_ops)
-{
-	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
-
-#ifdef RTE_CRYPTO_CALLBACKS
-	if (unlikely(dev->enq_cbs != NULL)) {
-		struct rte_cryptodev_cb_rcu *list;
-		struct rte_cryptodev_cb *cb;
-
-		/* __ATOMIC_RELEASE memory order was used when the
-		 * call back was inserted into the list.
-		 * Since there is a clear dependency between loading
-		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
-		 * not required.
-		 */
-		list = &dev->enq_cbs[qp_id];
-		rte_rcu_qsbr_thread_online(list->qsbr, 0);
-		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
-
-		while (cb != NULL) {
-			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
-					cb->arg);
-			cb = cb->next;
-		};
-
-		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
-	}
-#endif
-
-	rte_cryptodev_trace_enqueue_burst(dev_id, qp_id, (void **)ops, nb_ops);
-	return (*dev->enqueue_burst)(
-			dev->data->queue_pairs[qp_id], ops, nb_ops);
-}
-
-
 /** Cryptodev symmetric crypto session
  * Each session is derived from a fixed xform chain. Therefore each session
  * has a fixed algo, key, op-type, digest_len etc.
@@ -2009,6 +1791,148 @@ int rte_cryptodev_remove_deq_callback(uint8_t dev_id,
 				      uint16_t qp_id,
 				      struct rte_cryptodev_cb *cb);
 
+#include <rte_cryptodev_core.h>
+/**
+ *
+ * Dequeue a burst of processed crypto operations from a queue on the crypto
+ * device. The dequeued operation are stored in *rte_crypto_op* structures
+ * whose pointers are supplied in the *ops* array.
+ *
+ * The rte_cryptodev_dequeue_burst() function returns the number of ops
+ * actually dequeued, which is the number of *rte_crypto_op* data structures
+ * effectively supplied into the *ops* array.
+ *
+ * A return value equal to *nb_ops* indicates that the queue contained
+ * at least *nb_ops* operations, and this is likely to signify that other
+ * processed operations remain in the devices output queue. Applications
+ * implementing a "retrieve as many processed operations as possible" policy
+ * can check this specific case and keep invoking the
+ * rte_cryptodev_dequeue_burst() function until a value less than
+ * *nb_ops* is returned.
+ *
+ * The rte_cryptodev_dequeue_burst() function does not provide any error
+ * notification to avoid the corresponding overhead.
+ *
+ * @param	dev_id		The symmetric crypto device identifier
+ * @param	qp_id		The index of the queue pair from which to
+ *				retrieve processed packets. The value must be
+ *				in the range [0, nb_queue_pair - 1] previously
+ *				supplied to rte_cryptodev_configure().
+ * @param	ops		The address of an array of pointers to
+ *				*rte_crypto_op* structures that must be
+ *				large enough to store *nb_ops* pointers in it.
+ * @param	nb_ops		The maximum number of operations to dequeue.
+ *
+ * @return
+ *   - The number of operations actually dequeued, which is the number
+ *   of pointers to *rte_crypto_op* structures effectively supplied to the
+ *   *ops* array.
+ */
+static inline uint16_t
+rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_crypto_op **ops, uint16_t nb_ops)
+{
+	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
+
+	rte_cryptodev_trace_dequeue_burst(dev_id, qp_id, (void **)ops, nb_ops);
+	nb_ops = (*dev->dequeue_burst)
+			(dev->data->queue_pairs[qp_id], ops, nb_ops);
+#ifdef RTE_CRYPTO_CALLBACKS
+	if (unlikely(dev->deq_cbs != NULL)) {
+		struct rte_cryptodev_cb_rcu *list;
+		struct rte_cryptodev_cb *cb;
+
+		/* __ATOMIC_RELEASE memory order was used when the
+		 * call back was inserted into the list.
+		 * Since there is a clear dependency between loading
+		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
+		 * not required.
+		 */
+		list = &dev->deq_cbs[qp_id];
+		rte_rcu_qsbr_thread_online(list->qsbr, 0);
+		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
+
+		while (cb != NULL) {
+			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
+					cb->arg);
+			cb = cb->next;
+		};
+
+		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
+	}
+#endif
+	return nb_ops;
+}
+
+/**
+ * Enqueue a burst of operations for processing on a crypto device.
+ *
+ * The rte_cryptodev_enqueue_burst() function is invoked to place
+ * crypto operations on the queue *qp_id* of the device designated by
+ * its *dev_id*.
+ *
+ * The *nb_ops* parameter is the number of operations to process which are
+ * supplied in the *ops* array of *rte_crypto_op* structures.
+ *
+ * The rte_cryptodev_enqueue_burst() function returns the number of
+ * operations it actually enqueued for processing. A return value equal to
+ * *nb_ops* means that all packets have been enqueued.
+ *
+ * @param	dev_id		The identifier of the device.
+ * @param	qp_id		The index of the queue pair which packets are
+ *				to be enqueued for processing. The value
+ *				must be in the range [0, nb_queue_pairs - 1]
+ *				previously supplied to
+ *				 *rte_cryptodev_configure*.
+ * @param	ops		The address of an array of *nb_ops* pointers
+ *				to *rte_crypto_op* structures which contain
+ *				the crypto operations to be processed.
+ * @param	nb_ops		The number of operations to process.
+ *
+ * @return
+ * The number of operations actually enqueued on the crypto device. The return
+ * value can be less than the value of the *nb_ops* parameter when the
+ * crypto devices queue is full or if invalid parameters are specified in
+ * a *rte_crypto_op*.
+ */
+static inline uint16_t
+rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_crypto_op **ops, uint16_t nb_ops)
+{
+	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
+
+#ifdef RTE_CRYPTO_CALLBACKS
+	if (unlikely(dev->enq_cbs != NULL)) {
+		struct rte_cryptodev_cb_rcu *list;
+		struct rte_cryptodev_cb *cb;
+
+		/* __ATOMIC_RELEASE memory order was used when the
+		 * call back was inserted into the list.
+		 * Since there is a clear dependency between loading
+		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
+		 * not required.
+		 */
+		list = &dev->enq_cbs[qp_id];
+		rte_rcu_qsbr_thread_online(list->qsbr, 0);
+		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
+
+		while (cb != NULL) {
+			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
+					cb->arg);
+			cb = cb->next;
+		};
+
+		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
+	}
+#endif
+
+	rte_cryptodev_trace_enqueue_burst(dev_id, qp_id, (void **)ops, nb_ops);
+	return (*dev->enqueue_burst)(
+			dev->data->queue_pairs[qp_id], ops, nb_ops);
+}
+
+
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/cryptodev/rte_cryptodev_core.h b/lib/cryptodev/rte_cryptodev_core.h
new file mode 100644
index 0000000000..1633e55889
--- /dev/null
+++ b/lib/cryptodev/rte_cryptodev_core.h
@@ -0,0 +1,100 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2021 Marvell.
+ */
+
+#ifndef _RTE_CRYPTODEV_CORE_H_
+#define _RTE_CRYPTODEV_CORE_H_
+
+/**
+ * @file
+ *
+ * RTE Crypto Device internal header.
+ *
+ * This header contains internal data types. But they are still part of the
+ * public API because they are used by inline functions in the published API.
+ *
+ * Applications should not use these directly.
+ *
+ */
+
+typedef uint16_t (*dequeue_pkt_burst_t)(void *qp,
+		struct rte_crypto_op **ops,	uint16_t nb_ops);
+/**< Dequeue processed packets from queue pair of a device. */
+
+typedef uint16_t (*enqueue_pkt_burst_t)(void *qp,
+		struct rte_crypto_op **ops,	uint16_t nb_ops);
+/**< Enqueue packets for processing on queue pair of a device. */
+
+/**
+ * @internal
+ * The data part, with no function pointers, associated with each device.
+ *
+ * This structure is safe to place in shared memory to be common among
+ * different processes in a multi-process configuration.
+ */
+struct rte_cryptodev_data {
+	uint8_t dev_id;
+	/**< Device ID for this instance */
+	uint8_t socket_id;
+	/**< Socket ID where memory is allocated */
+	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
+	/**< Unique identifier name */
+
+	__extension__
+	uint8_t dev_started : 1;
+	/**< Device state: STARTED(1)/STOPPED(0) */
+
+	struct rte_mempool *session_pool;
+	/**< Session memory pool */
+	void **queue_pairs;
+	/**< Array of pointers to queue pairs. */
+	uint16_t nb_queue_pairs;
+	/**< Number of device queue pairs. */
+
+	void *dev_private;
+	/**< PMD-specific private data */
+} __rte_cache_aligned;
+
+
+/** @internal The data structure associated with each crypto device. */
+struct rte_cryptodev {
+	dequeue_pkt_burst_t dequeue_burst;
+	/**< Pointer to PMD receive function. */
+	enqueue_pkt_burst_t enqueue_burst;
+	/**< Pointer to PMD transmit function. */
+
+	struct rte_cryptodev_data *data;
+	/**< Pointer to device data */
+	struct rte_cryptodev_ops *dev_ops;
+	/**< Functions exported by PMD */
+	uint64_t feature_flags;
+	/**< Feature flags exposes HW/SW features for the given device */
+	struct rte_device *device;
+	/**< Backing device */
+
+	uint8_t driver_id;
+	/**< Crypto driver identifier*/
+
+	struct rte_cryptodev_cb_list link_intr_cbs;
+	/**< User application callback for interrupts if present */
+
+	void *security_ctx;
+	/**< Context for security ops */
+
+	__extension__
+	uint8_t attached : 1;
+	/**< Flag indicating the device is attached */
+
+	struct rte_cryptodev_cb_rcu *enq_cbs;
+	/**< User application callback for pre enqueue processing */
+
+	struct rte_cryptodev_cb_rcu *deq_cbs;
+	/**< User application callback for post dequeue processing */
+} __rte_cache_aligned;
+
+/**
+ * The pool of rte_cryptodev structures.
+ */
+extern struct rte_cryptodev *rte_cryptodevs;
+
+#endif /* _RTE_CRYPTODEV_CORE_H_ */
-- 
2.25.1

[dpdk-dev] [PATCH v2 2/5] cryptodev: allocate max space for internal qp array

[Akhil Goyal]

At queue_pair config stage, allocate memory for maximum
number of queue pair pointers that a device can support.

This will allow fast path APIs(enqueue_burst/dequeue_burst) to
refer pointer to internal QP data without checking for currently
configured QPs.
This is required to hide the rte_cryptodev and rte_cryptodev_data
structure from user.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
---
 lib/cryptodev/rte_cryptodev.c | 21 +++------------------
 1 file changed, 3 insertions(+), 18 deletions(-)

diff --git a/lib/cryptodev/rte_cryptodev.c b/lib/cryptodev/rte_cryptodev.c
index b913c434c5..eb86e629aa 100644
--- a/lib/cryptodev/rte_cryptodev.c
+++ b/lib/cryptodev/rte_cryptodev.c
@@ -978,7 +978,8 @@ rte_cryptodev_queue_pairs_config(struct rte_cryptodev *dev, uint16_t nb_qpairs,
 	if (dev->data->queue_pairs == NULL) { /* first time configuration */
 		dev->data->queue_pairs = rte_zmalloc_socket(
 				"cryptodev->queue_pairs",
-				sizeof(dev->data->queue_pairs[0]) * nb_qpairs,
+				sizeof(dev->data->queue_pairs[0]) *
+				dev_info.max_nb_queue_pairs,
 				RTE_CACHE_LINE_SIZE, socket_id);
 
 		if (dev->data->queue_pairs == NULL) {
@@ -1001,25 +1002,9 @@ rte_cryptodev_queue_pairs_config(struct rte_cryptodev *dev, uint16_t nb_qpairs,
 			ret = (*dev->dev_ops->queue_pair_release)(dev, i);
 			if (ret < 0)
 				return ret;
+			qp[i] = NULL;
 		}
 
-		qp = rte_realloc(qp, sizeof(qp[0]) * nb_qpairs,
-				RTE_CACHE_LINE_SIZE);
-		if (qp == NULL) {
-			CDEV_LOG_ERR("failed to realloc qp meta data,"
-						" nb_queues %u", nb_qpairs);
-			return -(ENOMEM);
-		}
-
-		if (nb_qpairs > old_nb_queues) {
-			uint16_t new_qs = nb_qpairs - old_nb_queues;
-
-			memset(qp + old_nb_queues, 0,
-				sizeof(qp[0]) * new_qs);
-		}
-
-		dev->data->queue_pairs = qp;
-
 	}
 	dev->data->nb_queue_pairs = nb_qpairs;
 	return 0;
-- 
2.25.1

[dpdk-dev] [PATCH v2 3/5] cryptodev: move inline APIs into separate structure

[Akhil Goyal]

Move fastpath inline function pointers from rte_cryptodev into a
separate structure accessed via a flat array.
The intension is to make rte_cryptodev and related structures private
to avoid future API/ABI breakages.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
---
 lib/cryptodev/cryptodev_pmd.c      | 51 ++++++++++++++++++++++++++++++
 lib/cryptodev/cryptodev_pmd.h      | 11 +++++++
 lib/cryptodev/rte_cryptodev.c      | 29 +++++++++++++++++
 lib/cryptodev/rte_cryptodev_core.h | 29 +++++++++++++++++
 lib/cryptodev/version.map          |  5 +++
 5 files changed, 125 insertions(+)

diff --git a/lib/cryptodev/cryptodev_pmd.c b/lib/cryptodev/cryptodev_pmd.c
index 44a70ecb35..4646708045 100644
--- a/lib/cryptodev/cryptodev_pmd.c
+++ b/lib/cryptodev/cryptodev_pmd.c
@@ -4,6 +4,7 @@
 
 #include <sys/queue.h>
 
+#include <rte_errno.h>
 #include <rte_string_fns.h>
 #include <rte_malloc.h>
 
@@ -160,3 +161,53 @@ rte_cryptodev_pmd_destroy(struct rte_cryptodev *cryptodev)
 
 	return 0;
 }
+
+static uint16_t
+dummy_crypto_enqueue_burst(__rte_unused void *qp,
+			   __rte_unused struct rte_crypto_op **ops,
+			   __rte_unused uint16_t nb_ops)
+{
+	CDEV_LOG_ERR(
+		"crypto enqueue burst requested for unconfigured device");
+	rte_errno = ENOTSUP;
+	return 0;
+}
+
+static uint16_t
+dummy_crypto_dequeue_burst(__rte_unused void *qp,
+			   __rte_unused struct rte_crypto_op **ops,
+			   __rte_unused uint16_t nb_ops)
+{
+	CDEV_LOG_ERR(
+		"crypto dequeue burst requested for unconfigured device");
+	rte_errno = ENOTSUP;
+	return 0;
+}
+
+void
+cryptodev_fp_ops_reset(struct rte_crypto_fp_ops *fp_ops)
+{
+	static void *dummy_data[RTE_MAX_QUEUES_PER_PORT];
+	static const struct rte_crypto_fp_ops dummy = {
+		.enqueue_burst = dummy_crypto_enqueue_burst,
+		.dequeue_burst = dummy_crypto_dequeue_burst,
+		.qp = {
+			.data = dummy_data,
+			.enq_cb = dummy_data,
+			.deq_cb = dummy_data,
+		},
+	};
+
+	*fp_ops = dummy;
+}
+
+void
+cryptodev_fp_ops_set(struct rte_crypto_fp_ops *fp_ops,
+		     const struct rte_cryptodev *dev)
+{
+	fp_ops->enqueue_burst = dev->enqueue_burst;
+	fp_ops->dequeue_burst = dev->dequeue_burst;
+	fp_ops->qp.data = dev->data->queue_pairs;
+	fp_ops->qp.enq_cb = (void **)(uintptr_t)dev->enq_cbs;
+	fp_ops->qp.deq_cb = (void **)(uintptr_t)dev->deq_cbs;
+}
diff --git a/lib/cryptodev/cryptodev_pmd.h b/lib/cryptodev/cryptodev_pmd.h
index 36606dd10b..a71edbb991 100644
--- a/lib/cryptodev/cryptodev_pmd.h
+++ b/lib/cryptodev/cryptodev_pmd.h
@@ -516,6 +516,17 @@ RTE_INIT(init_ ##driver_id)\
 	driver_id = rte_cryptodev_allocate_driver(&crypto_drv, &(drv));\
 }
 
+/* Reset crypto device fastpath APIs to dummy values. */
+__rte_internal
+void
+cryptodev_fp_ops_reset(struct rte_crypto_fp_ops *fp_ops);
+
+/* Setup crypto device fastpath APIs. */
+__rte_internal
+void
+cryptodev_fp_ops_set(struct rte_crypto_fp_ops *fp_ops,
+		     const struct rte_cryptodev *dev);
+
 static inline void *
 get_sym_session_private_data(const struct rte_cryptodev_sym_session *sess,
 		uint8_t driver_id) {
diff --git a/lib/cryptodev/rte_cryptodev.c b/lib/cryptodev/rte_cryptodev.c
index eb86e629aa..2378892d40 100644
--- a/lib/cryptodev/rte_cryptodev.c
+++ b/lib/cryptodev/rte_cryptodev.c
@@ -53,6 +53,9 @@ static struct rte_cryptodev_global cryptodev_globals = {
 		.nb_devs		= 0
 };
 
+/* Public fastpath APIs. */
+struct rte_crypto_fp_ops rte_crypto_fp_ops[RTE_CRYPTO_MAX_DEVS];
+
 /* spinlock for crypto device callbacks */
 static rte_spinlock_t rte_cryptodev_cb_lock = RTE_SPINLOCK_INITIALIZER;
 
@@ -903,6 +906,16 @@ rte_cryptodev_pmd_allocate(const char *name, int socket_id)
 		cryptodev_globals.nb_devs++;
 	}
 
+	/*
+	 * for secondary process, at that point we expect device
+	 * to be already 'usable', so shared data and all function
+	 * pointers for fast-path devops have to be setup properly
+	 * inside rte_cryptodev.
+	 */
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
+		cryptodev_fp_ops_set(rte_crypto_fp_ops +
+				cryptodev->data->dev_id, cryptodev);
+
 	return cryptodev;
 }
 
@@ -917,6 +930,8 @@ rte_cryptodev_pmd_release_device(struct rte_cryptodev *cryptodev)
 
 	dev_id = cryptodev->data->dev_id;
 
+	cryptodev_fp_ops_reset(rte_crypto_fp_ops + dev_id);
+
 	/* Close device only if device operations have been set */
 	if (cryptodev->dev_ops) {
 		ret = rte_cryptodev_close(dev_id);
@@ -1080,6 +1095,9 @@ rte_cryptodev_start(uint8_t dev_id)
 	}
 
 	diag = (*dev->dev_ops->dev_start)(dev);
+	/* expose selection of PMD fast-path functions */
+	cryptodev_fp_ops_set(rte_crypto_fp_ops + dev_id, dev);
+
 	rte_cryptodev_trace_start(dev_id, diag);
 	if (diag == 0)
 		dev->data->dev_started = 1;
@@ -1109,6 +1127,9 @@ rte_cryptodev_stop(uint8_t dev_id)
 		return;
 	}
 
+	/* point fast-path functions to dummy ones */
+	cryptodev_fp_ops_reset(rte_crypto_fp_ops + dev_id);
+
 	(*dev->dev_ops->dev_stop)(dev);
 	rte_cryptodev_trace_stop(dev_id);
 	dev->data->dev_started = 0;
@@ -2411,3 +2432,11 @@ rte_cryptodev_allocate_driver(struct cryptodev_driver *crypto_drv,
 
 	return nb_drivers++;
 }
+
+RTE_INIT(cryptodev_init_fp_ops)
+{
+	uint32_t i;
+
+	for (i = 0; i != RTE_DIM(rte_crypto_fp_ops); i++)
+		cryptodev_fp_ops_reset(rte_crypto_fp_ops + i);
+}
diff --git a/lib/cryptodev/rte_cryptodev_core.h b/lib/cryptodev/rte_cryptodev_core.h
index 1633e55889..bac5f8d984 100644
--- a/lib/cryptodev/rte_cryptodev_core.h
+++ b/lib/cryptodev/rte_cryptodev_core.h
@@ -25,6 +25,35 @@ typedef uint16_t (*enqueue_pkt_burst_t)(void *qp,
 		struct rte_crypto_op **ops,	uint16_t nb_ops);
 /**< Enqueue packets for processing on queue pair of a device. */
 
+/**
+ * @internal
+ * Structure used to hold opaque pointers to internal ethdev Rx/Tx
+ * queues data.
+ * The main purpose to expose these pointers at all - allow compiler
+ * to fetch this data for fast-path cryptodev inline functions in advance.
+ */
+struct rte_cryptodev_qpdata {
+	/** points to array of internal queue pair data pointers. */
+	void **data;
+	/** points to array of enqueue callback data pointers */
+	void **enq_cb;
+	/** points to array of dequeue callback data pointers */
+	void **deq_cb;
+};
+
+struct rte_crypto_fp_ops {
+	/** PMD enqueue burst function. */
+	enqueue_pkt_burst_t enqueue_burst;
+	/** PMD dequeue burst function. */
+	dequeue_pkt_burst_t dequeue_burst;
+	/** Internal queue pair data pointers. */
+	struct rte_cryptodev_qpdata qp;
+	/** Reserved for future ops. */
+	uintptr_t reserved[4];
+} __rte_cache_aligned;
+
+extern struct rte_crypto_fp_ops rte_crypto_fp_ops[RTE_CRYPTO_MAX_DEVS];
+
 /**
  * @internal
  * The data part, with no function pointers, associated with each device.
diff --git a/lib/cryptodev/version.map b/lib/cryptodev/version.map
index 43cf937e40..ed62ced221 100644
--- a/lib/cryptodev/version.map
+++ b/lib/cryptodev/version.map
@@ -45,6 +45,9 @@ DPDK_22 {
 	rte_cryptodev_sym_session_init;
 	rte_cryptodevs;
 
+	#added in 21.11
+	rte_crypto_fp_ops;
+
 	local: *;
 };
 
@@ -109,6 +112,8 @@ EXPERIMENTAL {
 INTERNAL {
 	global:
 
+	cryptodev_fp_ops_reset;
+	cryptodev_fp_ops_set;
 	rte_cryptodev_allocate_driver;
 	rte_cryptodev_pmd_allocate;
 	rte_cryptodev_pmd_callback_process;
-- 
2.25.1

[dpdk-dev] [PATCH v2 4/5] cryptodev: update fast path APIs to use new flat array

[Akhil Goyal]

Rework fast-path cryptodev functions to use rte_crypto_fp_ops[].
While it is an API/ABI breakage, this change is intended to be
transparent for both users (no changes in user app is required) and
PMD developers (no changes in PMD is required).

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
---
 lib/cryptodev/rte_cryptodev.h | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)

diff --git a/lib/cryptodev/rte_cryptodev.h b/lib/cryptodev/rte_cryptodev.h
index ce0dca72be..739ad529e5 100644
--- a/lib/cryptodev/rte_cryptodev.h
+++ b/lib/cryptodev/rte_cryptodev.h
@@ -1832,13 +1832,18 @@ static inline uint16_t
 rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
 		struct rte_crypto_op **ops, uint16_t nb_ops)
 {
-	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
+	struct rte_crypto_fp_ops *fp_ops;
+	void *qp;
 
 	rte_cryptodev_trace_dequeue_burst(dev_id, qp_id, (void **)ops, nb_ops);
-	nb_ops = (*dev->dequeue_burst)
-			(dev->data->queue_pairs[qp_id], ops, nb_ops);
+
+	fp_ops = &rte_crypto_fp_ops[dev_id];
+	qp = fp_ops->qp.data[qp_id];
+
+	nb_ops = fp_ops->dequeue_burst(qp, ops, nb_ops);
+
 #ifdef RTE_CRYPTO_CALLBACKS
-	if (unlikely(dev->deq_cbs != NULL)) {
+	if (unlikely(fp_ops->qp.deq_cb != NULL)) {
 		struct rte_cryptodev_cb_rcu *list;
 		struct rte_cryptodev_cb *cb;
 
@@ -1848,7 +1853,7 @@ rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
 		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
 		 * not required.
 		 */
-		list = &dev->deq_cbs[qp_id];
+		list = (struct rte_cryptodev_cb_rcu *)&fp_ops->qp.deq_cb[qp_id];
 		rte_rcu_qsbr_thread_online(list->qsbr, 0);
 		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
 
@@ -1899,10 +1904,13 @@ static inline uint16_t
 rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
 		struct rte_crypto_op **ops, uint16_t nb_ops)
 {
-	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
+	struct rte_crypto_fp_ops *fp_ops;
+	void *qp;
 
+	fp_ops = &rte_crypto_fp_ops[dev_id];
+	qp = fp_ops->qp.data[qp_id];
 #ifdef RTE_CRYPTO_CALLBACKS
-	if (unlikely(dev->enq_cbs != NULL)) {
+	if (unlikely(fp_ops->qp.enq_cb != NULL)) {
 		struct rte_cryptodev_cb_rcu *list;
 		struct rte_cryptodev_cb *cb;
 
@@ -1912,7 +1920,7 @@ rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
 		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
 		 * not required.
 		 */
-		list = &dev->enq_cbs[qp_id];
+		list = (struct rte_cryptodev_cb_rcu *)&fp_ops->qp.enq_cb[qp_id];
 		rte_rcu_qsbr_thread_online(list->qsbr, 0);
 		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
 
@@ -1927,8 +1935,7 @@ rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
 #endif
 
 	rte_cryptodev_trace_enqueue_burst(dev_id, qp_id, (void **)ops, nb_ops);
-	return (*dev->enqueue_burst)(
-			dev->data->queue_pairs[qp_id], ops, nb_ops);
+	return fp_ops->enqueue_burst(qp, ops, nb_ops);
 }
 
 
-- 
2.25.1

[dpdk-dev] [PATCH v2 5/5] cryptodev: move device specific structures

[Akhil Goyal]

The device specific structures - rte_cryptodev
and rte_cryptodev_data are moved to cryptodev_pmd.h
to hide it from the applications.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
---
 drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c    |  1 -
 drivers/crypto/ccp/ccp_dev.h                  |  2 +-
 drivers/crypto/cnxk/cn10k_ipsec.c             |  2 +-
 drivers/crypto/cnxk/cn9k_ipsec.c              |  2 +-
 .../crypto/cnxk/cnxk_cryptodev_capabilities.c |  2 +-
 drivers/crypto/cnxk/cnxk_cryptodev_sec.c      |  2 +-
 drivers/crypto/nitrox/nitrox_sym_reqmgr.c     |  2 +-
 drivers/crypto/octeontx/otx_cryptodev.c       |  1 -
 .../crypto/octeontx/otx_cryptodev_hw_access.c |  2 +-
 .../crypto/octeontx/otx_cryptodev_hw_access.h |  2 +-
 drivers/crypto/octeontx/otx_cryptodev_ops.h   |  2 +-
 .../crypto/octeontx2/otx2_cryptodev_mbox.c    |  2 +-
 drivers/crypto/scheduler/scheduler_failover.c |  2 +-
 .../crypto/scheduler/scheduler_multicore.c    |  2 +-
 .../scheduler/scheduler_pkt_size_distr.c      |  2 +-
 .../crypto/scheduler/scheduler_roundrobin.c   |  2 +-
 drivers/event/cnxk/cnxk_eventdev.h            |  2 +-
 drivers/event/dpaa/dpaa_eventdev.c            |  2 +-
 drivers/event/dpaa2/dpaa2_eventdev.c          |  2 +-
 drivers/event/octeontx/ssovf_evdev.c          |  2 +-
 .../event/octeontx2/otx2_evdev_crypto_adptr.c |  2 +-
 lib/cryptodev/cryptodev_pmd.h                 | 65 ++++++++++++++++++
 lib/cryptodev/rte_cryptodev_core.h            | 67 -------------------
 lib/cryptodev/version.map                     |  2 +-
 24 files changed, 85 insertions(+), 89 deletions(-)

diff --git a/drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c b/drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c
index 60963a8208..bbf310166e 100644
--- a/drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c
+++ b/drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c
@@ -6,7 +6,6 @@
 
 #include <rte_common.h>
 #include <rte_hexdump.h>
-#include <rte_cryptodev.h>
 #include <cryptodev_pmd.h>
 #include <rte_bus_vdev.h>
 #include <rte_malloc.h>
diff --git a/drivers/crypto/ccp/ccp_dev.h b/drivers/crypto/ccp/ccp_dev.h
index ca5145c278..85c8fc47a2 100644
--- a/drivers/crypto/ccp/ccp_dev.h
+++ b/drivers/crypto/ccp/ccp_dev.h
@@ -17,7 +17,7 @@
 #include <rte_pci.h>
 #include <rte_spinlock.h>
 #include <rte_crypto_sym.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 
 /**< CCP sspecific */
 #define MAX_HW_QUEUES                   5
diff --git a/drivers/crypto/cnxk/cn10k_ipsec.c b/drivers/crypto/cnxk/cn10k_ipsec.c
index defc792aa8..27df1dcd64 100644
--- a/drivers/crypto/cnxk/cn10k_ipsec.c
+++ b/drivers/crypto/cnxk/cn10k_ipsec.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_malloc.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_esp.h>
 #include <rte_ip.h>
 #include <rte_security.h>
diff --git a/drivers/crypto/cnxk/cn9k_ipsec.c b/drivers/crypto/cnxk/cn9k_ipsec.c
index 9ca4d20c62..53fb793654 100644
--- a/drivers/crypto/cnxk/cn9k_ipsec.c
+++ b/drivers/crypto/cnxk/cn9k_ipsec.c
@@ -2,7 +2,7 @@
  * Copyright(C) 2021 Marvell.
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_ip.h>
 #include <rte_security.h>
 #include <rte_security_driver.h>
diff --git a/drivers/crypto/cnxk/cnxk_cryptodev_capabilities.c b/drivers/crypto/cnxk/cnxk_cryptodev_capabilities.c
index a227e6981c..a53b489a04 100644
--- a/drivers/crypto/cnxk/cnxk_cryptodev_capabilities.c
+++ b/drivers/crypto/cnxk/cnxk_cryptodev_capabilities.c
@@ -2,7 +2,7 @@
  * Copyright(C) 2021 Marvell.
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_security.h>
 
 #include "roc_api.h"
diff --git a/drivers/crypto/cnxk/cnxk_cryptodev_sec.c b/drivers/crypto/cnxk/cnxk_cryptodev_sec.c
index 8d04d4b575..2021d5c77e 100644
--- a/drivers/crypto/cnxk/cnxk_cryptodev_sec.c
+++ b/drivers/crypto/cnxk/cnxk_cryptodev_sec.c
@@ -2,7 +2,7 @@
  * Copyright(C) 2021 Marvell.
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_malloc.h>
 #include <rte_security.h>
 #include <rte_security_driver.h>
diff --git a/drivers/crypto/nitrox/nitrox_sym_reqmgr.c b/drivers/crypto/nitrox/nitrox_sym_reqmgr.c
index fe3ca25a0c..9edb0cc00f 100644
--- a/drivers/crypto/nitrox/nitrox_sym_reqmgr.c
+++ b/drivers/crypto/nitrox/nitrox_sym_reqmgr.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_crypto.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_cycles.h>
 #include <rte_errno.h>
 
diff --git a/drivers/crypto/octeontx/otx_cryptodev.c b/drivers/crypto/octeontx/otx_cryptodev.c
index c294f86d79..d09a17bb80 100644
--- a/drivers/crypto/octeontx/otx_cryptodev.c
+++ b/drivers/crypto/octeontx/otx_cryptodev.c
@@ -4,7 +4,6 @@
 
 #include <rte_bus_pci.h>
 #include <rte_common.h>
-#include <rte_cryptodev.h>
 #include <cryptodev_pmd.h>
 #include <rte_log.h>
 #include <rte_pci.h>
diff --git a/drivers/crypto/octeontx/otx_cryptodev_hw_access.c b/drivers/crypto/octeontx/otx_cryptodev_hw_access.c
index 7b89a62d81..20b288334a 100644
--- a/drivers/crypto/octeontx/otx_cryptodev_hw_access.c
+++ b/drivers/crypto/octeontx/otx_cryptodev_hw_access.c
@@ -7,7 +7,7 @@
 
 #include <rte_branch_prediction.h>
 #include <rte_common.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_errno.h>
 #include <rte_mempool.h>
 #include <rte_memzone.h>
diff --git a/drivers/crypto/octeontx/otx_cryptodev_hw_access.h b/drivers/crypto/octeontx/otx_cryptodev_hw_access.h
index 7c6b1e45b4..e48805fb09 100644
--- a/drivers/crypto/octeontx/otx_cryptodev_hw_access.h
+++ b/drivers/crypto/octeontx/otx_cryptodev_hw_access.h
@@ -7,7 +7,7 @@
 #include <stdbool.h>
 
 #include <rte_branch_prediction.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_cycles.h>
 #include <rte_io.h>
 #include <rte_memory.h>
diff --git a/drivers/crypto/octeontx/otx_cryptodev_ops.h b/drivers/crypto/octeontx/otx_cryptodev_ops.h
index f234f16970..83b82ea059 100644
--- a/drivers/crypto/octeontx/otx_cryptodev_ops.h
+++ b/drivers/crypto/octeontx/otx_cryptodev_ops.h
@@ -5,7 +5,7 @@
 #ifndef _OTX_CRYPTODEV_OPS_H_
 #define _OTX_CRYPTODEV_OPS_H_
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 
 #define OTX_CPT_MIN_HEADROOM_REQ	(24)
 #define OTX_CPT_MIN_TAILROOM_REQ	(8)
diff --git a/drivers/crypto/octeontx2/otx2_cryptodev_mbox.c b/drivers/crypto/octeontx2/otx2_cryptodev_mbox.c
index 1a8edae7eb..f9e7b0b474 100644
--- a/drivers/crypto/octeontx2/otx2_cryptodev_mbox.c
+++ b/drivers/crypto/octeontx2/otx2_cryptodev_mbox.c
@@ -1,7 +1,7 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright (C) 2019 Marvell International Ltd.
  */
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_ethdev.h>
 
 #include "otx2_cryptodev.h"
diff --git a/drivers/crypto/scheduler/scheduler_failover.c b/drivers/crypto/scheduler/scheduler_failover.c
index 844312dd1b..5023577ef8 100644
--- a/drivers/crypto/scheduler/scheduler_failover.c
+++ b/drivers/crypto/scheduler/scheduler_failover.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2017 Intel Corporation
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_malloc.h>
 
 #include "rte_cryptodev_scheduler_operations.h"
diff --git a/drivers/crypto/scheduler/scheduler_multicore.c b/drivers/crypto/scheduler/scheduler_multicore.c
index 1e2e8dbf9f..900ab4049d 100644
--- a/drivers/crypto/scheduler/scheduler_multicore.c
+++ b/drivers/crypto/scheduler/scheduler_multicore.c
@@ -3,7 +3,7 @@
  */
 #include <unistd.h>
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_malloc.h>
 
 #include "rte_cryptodev_scheduler_operations.h"
diff --git a/drivers/crypto/scheduler/scheduler_pkt_size_distr.c b/drivers/crypto/scheduler/scheduler_pkt_size_distr.c
index 57e330a744..933a5c6978 100644
--- a/drivers/crypto/scheduler/scheduler_pkt_size_distr.c
+++ b/drivers/crypto/scheduler/scheduler_pkt_size_distr.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2017 Intel Corporation
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_malloc.h>
 
 #include "rte_cryptodev_scheduler_operations.h"
diff --git a/drivers/crypto/scheduler/scheduler_roundrobin.c b/drivers/crypto/scheduler/scheduler_roundrobin.c
index bc4a632106..ace2dec2ec 100644
--- a/drivers/crypto/scheduler/scheduler_roundrobin.c
+++ b/drivers/crypto/scheduler/scheduler_roundrobin.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2017 Intel Corporation
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_malloc.h>
 
 #include "rte_cryptodev_scheduler_operations.h"
diff --git a/drivers/event/cnxk/cnxk_eventdev.h b/drivers/event/cnxk/cnxk_eventdev.h
index 8a5c737e4b..b57004c0dc 100644
--- a/drivers/event/cnxk/cnxk_eventdev.h
+++ b/drivers/event/cnxk/cnxk_eventdev.h
@@ -7,7 +7,7 @@
 
 #include <string.h>
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_devargs.h>
 #include <rte_ethdev.h>
 #include <rte_event_eth_rx_adapter.h>
diff --git a/drivers/event/dpaa/dpaa_eventdev.c b/drivers/event/dpaa/dpaa_eventdev.c
index ec74160325..1d7ddfe1d1 100644
--- a/drivers/event/dpaa/dpaa_eventdev.c
+++ b/drivers/event/dpaa/dpaa_eventdev.c
@@ -28,7 +28,7 @@
 #include <rte_ethdev.h>
 #include <rte_event_eth_rx_adapter.h>
 #include <rte_event_eth_tx_adapter.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_dpaa_bus.h>
 #include <rte_dpaa_logs.h>
 #include <rte_cycles.h>
diff --git a/drivers/event/dpaa2/dpaa2_eventdev.c b/drivers/event/dpaa2/dpaa2_eventdev.c
index 5ccf22f77f..e03afb2958 100644
--- a/drivers/event/dpaa2/dpaa2_eventdev.c
+++ b/drivers/event/dpaa2/dpaa2_eventdev.c
@@ -25,7 +25,7 @@
 #include <rte_pci.h>
 #include <rte_bus_vdev.h>
 #include <ethdev_driver.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_event_eth_rx_adapter.h>
 #include <rte_event_eth_tx_adapter.h>
 
diff --git a/drivers/event/octeontx/ssovf_evdev.c b/drivers/event/octeontx/ssovf_evdev.c
index b93f6ec8c6..9846fce34b 100644
--- a/drivers/event/octeontx/ssovf_evdev.c
+++ b/drivers/event/octeontx/ssovf_evdev.c
@@ -5,7 +5,7 @@
 #include <inttypes.h>
 
 #include <rte_common.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_debug.h>
 #include <rte_dev.h>
 #include <rte_eal.h>
diff --git a/drivers/event/octeontx2/otx2_evdev_crypto_adptr.c b/drivers/event/octeontx2/otx2_evdev_crypto_adptr.c
index d9a002625c..d59d6c53f6 100644
--- a/drivers/event/octeontx2/otx2_evdev_crypto_adptr.c
+++ b/drivers/event/octeontx2/otx2_evdev_crypto_adptr.c
@@ -2,7 +2,7 @@
  * Copyright (C) 2020-2021 Marvell.
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_eventdev.h>
 
 #include "otx2_cryptodev.h"
diff --git a/lib/cryptodev/cryptodev_pmd.h b/lib/cryptodev/cryptodev_pmd.h
index a71edbb991..0d1bef0e67 100644
--- a/lib/cryptodev/cryptodev_pmd.h
+++ b/lib/cryptodev/cryptodev_pmd.h
@@ -52,6 +52,71 @@ struct rte_cryptodev_pmd_init_params {
 	unsigned int max_nb_queue_pairs;
 };
 
+/**
+ * @internal
+ * The data part, with no function pointers, associated with each device.
+ *
+ * This structure is safe to place in shared memory to be common among
+ * different processes in a multi-process configuration.
+ */
+struct rte_cryptodev_data {
+	/** Device ID for this instance */
+	uint8_t dev_id;
+	/** Socket ID where memory is allocated */
+	uint8_t socket_id;
+	/** Unique identifier name */
+	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
+
+	__extension__
+	/** Device state: STARTED(1)/STOPPED(0) */
+	uint8_t dev_started : 1;
+
+	/** Session memory pool */
+	struct rte_mempool *session_pool;
+	/** Array of pointers to queue pairs. */
+	void **queue_pairs;
+	/** Number of device queue pairs. */
+	uint16_t nb_queue_pairs;
+
+	/** PMD-specific private data */
+	void *dev_private;
+} __rte_cache_aligned;
+
+/** @internal The data structure associated with each crypto device. */
+struct rte_cryptodev {
+	/** Pointer to PMD dequeue function. */
+	dequeue_pkt_burst_t dequeue_burst;
+	/** Pointer to PMD enqueue function. */
+	enqueue_pkt_burst_t enqueue_burst;
+
+	/** Pointer to device data */
+	struct rte_cryptodev_data *data;
+	/** Functions exported by PMD */
+	struct rte_cryptodev_ops *dev_ops;
+	/** Feature flags exposes HW/SW features for the given device */
+	uint64_t feature_flags;
+	/** Backing device */
+	struct rte_device *device;
+
+	/** Crypto driver identifier*/
+	uint8_t driver_id;
+
+	/** User application callback for interrupts if present */
+	struct rte_cryptodev_cb_list link_intr_cbs;
+
+	/** Context for security ops */
+	void *security_ctx;
+
+	__extension__
+	/** Flag indicating the device is attached */
+	uint8_t attached : 1;
+
+	/** User application callback for pre enqueue processing */
+	struct rte_cryptodev_cb_rcu *enq_cbs;
+	/** User application callback for post dequeue processing */
+	struct rte_cryptodev_cb_rcu *deq_cbs;
+} __rte_cache_aligned;
+
 /** Global structure used for maintaining state of allocated crypto devices */
 struct rte_cryptodev_global {
 	struct rte_cryptodev *devs;	/**< Device information array */
diff --git a/lib/cryptodev/rte_cryptodev_core.h b/lib/cryptodev/rte_cryptodev_core.h
index bac5f8d984..47d3985c25 100644
--- a/lib/cryptodev/rte_cryptodev_core.h
+++ b/lib/cryptodev/rte_cryptodev_core.h
@@ -54,73 +54,6 @@ struct rte_crypto_fp_ops {
 
 extern struct rte_crypto_fp_ops rte_crypto_fp_ops[RTE_CRYPTO_MAX_DEVS];
 
-/**
- * @internal
- * The data part, with no function pointers, associated with each device.
- *
- * This structure is safe to place in shared memory to be common among
- * different processes in a multi-process configuration.
- */
-struct rte_cryptodev_data {
-	uint8_t dev_id;
-	/**< Device ID for this instance */
-	uint8_t socket_id;
-	/**< Socket ID where memory is allocated */
-	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
-	/**< Unique identifier name */
-
-	__extension__
-	uint8_t dev_started : 1;
-	/**< Device state: STARTED(1)/STOPPED(0) */
-
-	struct rte_mempool *session_pool;
-	/**< Session memory pool */
-	void **queue_pairs;
-	/**< Array of pointers to queue pairs. */
-	uint16_t nb_queue_pairs;
-	/**< Number of device queue pairs. */
-
-	void *dev_private;
-	/**< PMD-specific private data */
-} __rte_cache_aligned;
-
-
-/** @internal The data structure associated with each crypto device. */
-struct rte_cryptodev {
-	dequeue_pkt_burst_t dequeue_burst;
-	/**< Pointer to PMD receive function. */
-	enqueue_pkt_burst_t enqueue_burst;
-	/**< Pointer to PMD transmit function. */
-
-	struct rte_cryptodev_data *data;
-	/**< Pointer to device data */
-	struct rte_cryptodev_ops *dev_ops;
-	/**< Functions exported by PMD */
-	uint64_t feature_flags;
-	/**< Feature flags exposes HW/SW features for the given device */
-	struct rte_device *device;
-	/**< Backing device */
-
-	uint8_t driver_id;
-	/**< Crypto driver identifier*/
-
-	struct rte_cryptodev_cb_list link_intr_cbs;
-	/**< User application callback for interrupts if present */
-
-	void *security_ctx;
-	/**< Context for security ops */
-
-	__extension__
-	uint8_t attached : 1;
-	/**< Flag indicating the device is attached */
-
-	struct rte_cryptodev_cb_rcu *enq_cbs;
-	/**< User application callback for pre enqueue processing */
-
-	struct rte_cryptodev_cb_rcu *deq_cbs;
-	/**< User application callback for post dequeue processing */
-} __rte_cache_aligned;
-
 /**
  * The pool of rte_cryptodev structures.
  */
diff --git a/lib/cryptodev/version.map b/lib/cryptodev/version.map
index ed62ced221..cd56ad52e2 100644
--- a/lib/cryptodev/version.map
+++ b/lib/cryptodev/version.map
@@ -43,7 +43,6 @@ DPDK_22 {
 	rte_cryptodev_sym_session_create;
 	rte_cryptodev_sym_session_free;
 	rte_cryptodev_sym_session_init;
-	rte_cryptodevs;
 
 	#added in 21.11
 	rte_crypto_fp_ops;
@@ -124,4 +123,5 @@ INTERNAL {
 	rte_cryptodev_pmd_get_named_dev;
 	rte_cryptodev_pmd_parse_input_args;
 	rte_cryptodev_pmd_release_device;
+	rte_cryptodevs;
 };
-- 
2.25.1

Re: [dpdk-dev] [PATCH v2 3/5] cryptodev: move inline APIs into separate structure

[Zhang, Roy Fan]

Hi Akhil,

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Monday, October 11, 2021 1:43 PM
> To: dev@dpdk.org
> Cc: thomas@monjalon.net; david.marchand@redhat.com;
> hemant.agrawal@nxp.com; anoobj@marvell.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> Doherty, Declan <declan.doherty@intel.com>; matan@nvidia.com;
> g.singh@nxp.com; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
> jianjay.zhou@huawei.com; asomalap@amd.com; ruifeng.wang@arm.com;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>; Nicolau, Radu
> <radu.nicolau@intel.com>; ajit.khaparde@broadcom.com;
> rnagadheeraj@marvell.com; adwivedi@marvell.com; Power, Ciara
> <ciara.power@intel.com>; Akhil Goyal <gakhil@marvell.com>
> Subject: [PATCH v2 3/5] cryptodev: move inline APIs into separate structure
> 
> Move fastpath inline function pointers from rte_cryptodev into a
> separate structure accessed via a flat array.
> The intension is to make rte_cryptodev and related structures private
> to avoid future API/ABI breakages.
> 
> Signed-off-by: Akhil Goyal <gakhil@marvell.com>
> ---
>  lib/cryptodev/cryptodev_pmd.c      | 51
> ++++++++++++++++++++++++++++++
>  lib/cryptodev/cryptodev_pmd.h      | 11 +++++++
>  lib/cryptodev/rte_cryptodev.c      | 29 +++++++++++++++++
>  lib/cryptodev/rte_cryptodev_core.h | 29 +++++++++++++++++
>  lib/cryptodev/version.map          |  5 +++
>  5 files changed, 125 insertions(+)
> 
> diff --git a/lib/cryptodev/cryptodev_pmd.c
> b/lib/cryptodev/cryptodev_pmd.c
> index 44a70ecb35..4646708045 100644
> --- a/lib/cryptodev/cryptodev_pmd.c
> +++ b/lib/cryptodev/cryptodev_pmd.c
> @@ -4,6 +4,7 @@
> 
>  #include <sys/queue.h>
> 
> +#include <rte_errno.h>
>  #include <rte_string_fns.h>
>  #include <rte_malloc.h>
> 
> @@ -160,3 +161,53 @@ rte_cryptodev_pmd_destroy(struct rte_cryptodev
> *cryptodev)
> 

When a device is removed - aka when rte_pci_remove() is called 
cryptodev_fp_ops_reset() will never be called. This may expose a problem.
Looks like cryptodev_fp_ops_reset() needs to be called here too.

>  	return 0;
... 

Re: [dpdk-dev] [PATCH v2 1/5] cryptodev: separate out internal structures

[Zhang, Roy Fan]

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Monday, October 11, 2021 1:43 PM
> To: dev@dpdk.org
> Cc: thomas@monjalon.net; david.marchand@redhat.com;
> hemant.agrawal@nxp.com; anoobj@marvell.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> Doherty, Declan <declan.doherty@intel.com>; matan@nvidia.com;
> g.singh@nxp.com; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
> jianjay.zhou@huawei.com; asomalap@amd.com; ruifeng.wang@arm.com;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>; Nicolau, Radu
> <radu.nicolau@intel.com>; ajit.khaparde@broadcom.com;
> rnagadheeraj@marvell.com; adwivedi@marvell.com; Power, Ciara
> <ciara.power@intel.com>; Akhil Goyal <gakhil@marvell.com>
> Subject: [PATCH v2 1/5] cryptodev: separate out internal structures
> 
> A new header file rte_cryptodev_core.h is added and all
> internal data structures which need not be exposed directly to
> application are moved to this file. These structures are mostly
> used by drivers, but they need to be in the public header file
> as they are accessed by datapath inline functions for
> performance reasons.
> 
> Signed-off-by: Akhil Goyal <gakhil@marvell.com>
> ---

Acked-by: Fan Zhang <roy.fan.zhang@intel.com>

Re: [dpdk-dev] [PATCH v2 2/5] cryptodev: allocate max space for internal qp array

[Zhang, Roy Fan]

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Monday, October 11, 2021 1:43 PM
> To: dev@dpdk.org
> Cc: thomas@monjalon.net; david.marchand@redhat.com;
> hemant.agrawal@nxp.com; anoobj@marvell.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> Doherty, Declan <declan.doherty@intel.com>; matan@nvidia.com;
> g.singh@nxp.com; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
> jianjay.zhou@huawei.com; asomalap@amd.com; ruifeng.wang@arm.com;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>; Nicolau, Radu
> <radu.nicolau@intel.com>; ajit.khaparde@broadcom.com;
> rnagadheeraj@marvell.com; adwivedi@marvell.com; Power, Ciara
> <ciara.power@intel.com>; Akhil Goyal <gakhil@marvell.com>
> Subject: [PATCH v2 2/5] cryptodev: allocate max space for internal qp array
> 
> At queue_pair config stage, allocate memory for maximum
> number of queue pair pointers that a device can support.
> 
> This will allow fast path APIs(enqueue_burst/dequeue_burst) to
> refer pointer to internal QP data without checking for currently
> configured QPs.
> This is required to hide the rte_cryptodev and rte_cryptodev_data
> structure from user.
> 
> Signed-off-by: Akhil Goyal <gakhil@marvell.com>
> ---
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>

Re: [dpdk-dev] [PATCH v2 4/5] cryptodev: update fast path APIs to use new flat array

[Zhang, Roy Fan]

Hi Akhil,

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Monday, October 11, 2021 1:43 PM
> To: dev@dpdk.org
> Cc: thomas@monjalon.net; david.marchand@redhat.com;
> hemant.agrawal@nxp.com; anoobj@marvell.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> Doherty, Declan <declan.doherty@intel.com>; matan@nvidia.com;
> g.singh@nxp.com; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
> jianjay.zhou@huawei.com; asomalap@amd.com; ruifeng.wang@arm.com;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>; Nicolau, Radu
> <radu.nicolau@intel.com>; ajit.khaparde@broadcom.com;
> rnagadheeraj@marvell.com; adwivedi@marvell.com; Power, Ciara
> <ciara.power@intel.com>; Akhil Goyal <gakhil@marvell.com>
> Subject: [PATCH v2 4/5] cryptodev: update fast path APIs to use new flat
> array
> 
> Rework fast-path cryptodev functions to use rte_crypto_fp_ops[].
> While it is an API/ABI breakage, this change is intended to be
> transparent for both users (no changes in user app is required) and
> PMD developers (no changes in PMD is required).
> 
> Signed-off-by: Akhil Goyal <gakhil@marvell.com>
> ---
>  lib/cryptodev/rte_cryptodev.h | 27 +++++++++++++++++----------
>  1 file changed, 17 insertions(+), 10 deletions(-)
> 
> diff --git a/lib/cryptodev/rte_cryptodev.h b/lib/cryptodev/rte_cryptodev.h
> index ce0dca72be..739ad529e5 100644
> --- a/lib/cryptodev/rte_cryptodev.h
> +++ b/lib/cryptodev/rte_cryptodev.h
> @@ -1832,13 +1832,18 @@ static inline uint16_t
>  rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
>  		struct rte_crypto_op **ops, uint16_t nb_ops)
>  {
> -	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
> +	struct rte_crypto_fp_ops *fp_ops;

We may need to use const for fp_ops since we only call the function pointers in it.
 
> +	void *qp;
> 
>  	rte_cryptodev_trace_dequeue_burst(dev_id, qp_id, (void **)ops,
> nb_ops);
> -	nb_ops = (*dev->dequeue_burst)
> -			(dev->data->queue_pairs[qp_id], ops, nb_ops);
> +
> +	fp_ops = &rte_crypto_fp_ops[dev_id];
> +	qp = fp_ops->qp.data[qp_id];
> +
> +	nb_ops = fp_ops->dequeue_burst(qp, ops, nb_ops);
> +
>  #ifdef RTE_CRYPTO_CALLBACKS
> -	if (unlikely(dev->deq_cbs != NULL)) {
> +	if (unlikely(fp_ops->qp.deq_cb != NULL)) {
>  		struct rte_cryptodev_cb_rcu *list;
>  		struct rte_cryptodev_cb *cb;
> 
> @@ -1848,7 +1853,7 @@ rte_cryptodev_dequeue_burst(uint8_t dev_id,
> uint16_t qp_id,
>  		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory
> order is
>  		 * not required.
>  		 */
> -		list = &dev->deq_cbs[qp_id];
> +		list = (struct rte_cryptodev_cb_rcu *)&fp_ops-
> >qp.deq_cb[qp_id];
>  		rte_rcu_qsbr_thread_online(list->qsbr, 0);
>  		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
> 
> @@ -1899,10 +1904,13 @@ static inline uint16_t
>  rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
>  		struct rte_crypto_op **ops, uint16_t nb_ops)
>  {
> -	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
> +	struct rte_crypto_fp_ops *fp_ops;

Same as above

> +	void *qp;
> 
> +	fp_ops = &rte_crypto_fp_ops[dev_id];
> +	qp = fp_ops->qp.data[qp_id];
>  #ifdef RTE_CRYPTO_CALLBACKS
> -	if (unlikely(dev->enq_cbs != NULL)) {
> +	if (unlikely(fp_ops->qp.enq_cb != NULL)) {
>  		struct rte_cryptodev_cb_rcu *list;
>  		struct rte_cryptodev_cb *cb;
> 
> @@ -1912,7 +1920,7 @@ rte_cryptodev_enqueue_burst(uint8_t dev_id,
> uint16_t qp_id,
>  		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory
> order is
>  		 * not required.
>  		 */
> -		list = &dev->enq_cbs[qp_id];
> +		list = (struct rte_cryptodev_cb_rcu *)&fp_ops-
> >qp.enq_cb[qp_id];
>  		rte_rcu_qsbr_thread_online(list->qsbr, 0);
>  		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
> 
> @@ -1927,8 +1935,7 @@ rte_cryptodev_enqueue_burst(uint8_t dev_id,
> uint16_t qp_id,
>  #endif
> 
>  	rte_cryptodev_trace_enqueue_burst(dev_id, qp_id, (void **)ops,
> nb_ops);
> -	return (*dev->enqueue_burst)(
> -			dev->data->queue_pairs[qp_id], ops, nb_ops);
> +	return fp_ops->enqueue_burst(qp, ops, nb_ops);
>  }
> 
> 
> --
> 2.25.1

Other than the minor comments above
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>

Re: [dpdk-dev] [PATCH v2 5/5] cryptodev: move device specific structures

[Zhang, Roy Fan]

Hi Akhil,

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Monday, October 11, 2021 1:43 PM
> To: dev@dpdk.org
> Cc: thomas@monjalon.net; david.marchand@redhat.com;
> hemant.agrawal@nxp.com; anoobj@marvell.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> Doherty, Declan <declan.doherty@intel.com>; matan@nvidia.com;
> g.singh@nxp.com; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
> jianjay.zhou@huawei.com; asomalap@amd.com; ruifeng.wang@arm.com;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>; Nicolau, Radu
> <radu.nicolau@intel.com>; ajit.khaparde@broadcom.com;
> rnagadheeraj@marvell.com; adwivedi@marvell.com; Power, Ciara
> <ciara.power@intel.com>; Akhil Goyal <gakhil@marvell.com>
> Subject: [PATCH v2 5/5] cryptodev: move device specific structures
> 
> The device specific structures - rte_cryptodev
> and rte_cryptodev_data are moved to cryptodev_pmd.h
> to hide it from the applications.
> 
> Signed-off-by: Akhil Goyal <gakhil@marvell.com>
> ---
...
> diff --git a/lib/cryptodev/cryptodev_pmd.h
> b/lib/cryptodev/cryptodev_pmd.h
> index a71edbb991..0d1bef0e67 100644
> --- a/lib/cryptodev/cryptodev_pmd.h
> +++ b/lib/cryptodev/cryptodev_pmd.h
> @@ -52,6 +52,71 @@ struct rte_cryptodev_pmd_init_params {
>  	unsigned int max_nb_queue_pairs;
>  };
> 
> +/**
> + * @internal
> + * The data part, with no function pointers, associated with each device.
> + *
> + * This structure is safe to place in shared memory to be common among
> + * different processes in a multi-process configuration.
> + */
> +struct rte_cryptodev_data {
> +	/** Device ID for this instance */
> +	uint8_t dev_id;
> +	/** Socket ID where memory is allocated */
> +	uint8_t socket_id;
> +	/** Unique identifier name */
> +	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
> +
> +	__extension__
> +	/** Device state: STARTED(1)/STOPPED(0) */
> +	uint8_t dev_started : 1;
> +
> +	/** Session memory pool */
> +	struct rte_mempool *session_pool;

Looks like we never used this pool ever - shall we take this chance to remove it?

> +	/** Array of pointers to queue pairs. */
> +	void **queue_pairs;
> +	/** Number of device queue pairs. */
> +	uint16_t nb_queue_pairs;

queue_pairs are likely the only item we use rte_cryptodev_data in the fast-path.
Also take this chance to move it to the beginning of the structure?

> +
> +	/** PMD-specific private data */
> +	void *dev_private;
> +} __rte_cache_aligned;
> +

Not really comments but some suggestions - could be done in the future as the
structure is moved to internal in this patchset.
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>

Re: [dpdk-dev] [PATCH v2 0/5] cryptodev: hide internal structures

[Zhang, Roy Fan]

Hi Akhil,

The approach looks great but we may have to check if it works in multi-process
environment - since all enqueue/dequeue handlers are set by primary process
the secondary process may not recognize the fp_ops data.

We will run a quick test to see if it is true.

Regards,
Fan

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Monday, October 11, 2021 1:43 PM
> To: dev@dpdk.org
> Cc: thomas@monjalon.net; david.marchand@redhat.com;
> hemant.agrawal@nxp.com; anoobj@marvell.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> Doherty, Declan <declan.doherty@intel.com>; matan@nvidia.com;
> g.singh@nxp.com; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
> jianjay.zhou@huawei.com; asomalap@amd.com; ruifeng.wang@arm.com;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>; Nicolau, Radu
> <radu.nicolau@intel.com>; ajit.khaparde@broadcom.com;
> rnagadheeraj@marvell.com; adwivedi@marvell.com; Power, Ciara
> <ciara.power@intel.com>; Akhil Goyal <gakhil@marvell.com>
> Subject: [PATCH v2 0/5] cryptodev: hide internal structures
> 
> Structures rte_cryptodev and rte_cryptodev_data are not
> supposed to be directly used by the application. These
> are made public as they are used by inline datapath
> public APIs.
> This patchset, creates a new rte_cryptodev_core.h file
> which helps in defining a data structure to hold datapath
> APIs in a flat array based on the device identifier which
> is filled by the PMD.
> 
> Similar series for ethdev and eventdev are also floated on ML.
> https://patchwork.dpdk.org/project/dpdk/list/?series=19428
> https://patchwork.dpdk.org/project/dpdk/list/?series=19405
> 
> changes in v2: align with the latest versions of above series.
> 
> Akhil Goyal (5):
>   cryptodev: separate out internal structures
>   cryptodev: allocate max space for internal qp array
>   cryptodev: move inline APIs into separate structure
>   cryptodev: update fast path APIs to use new flat array
>   cryptodev: move device specific structures
> 
>  drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c    |   1 -
>  drivers/crypto/ccp/ccp_dev.h                  |   2 +-
>  drivers/crypto/cnxk/cn10k_ipsec.c             |   2 +-
>  drivers/crypto/cnxk/cn9k_ipsec.c              |   2 +-
>  .../crypto/cnxk/cnxk_cryptodev_capabilities.c |   2 +-
>  drivers/crypto/cnxk/cnxk_cryptodev_sec.c      |   2 +-
>  drivers/crypto/nitrox/nitrox_sym_reqmgr.c     |   2 +-
>  drivers/crypto/octeontx/otx_cryptodev.c       |   1 -
>  .../crypto/octeontx/otx_cryptodev_hw_access.c |   2 +-
>  .../crypto/octeontx/otx_cryptodev_hw_access.h |   2 +-
>  drivers/crypto/octeontx/otx_cryptodev_ops.h   |   2 +-
>  .../crypto/octeontx2/otx2_cryptodev_mbox.c    |   2 +-
>  drivers/crypto/scheduler/scheduler_failover.c |   2 +-
>  .../crypto/scheduler/scheduler_multicore.c    |   2 +-
>  .../scheduler/scheduler_pkt_size_distr.c      |   2 +-
>  .../crypto/scheduler/scheduler_roundrobin.c   |   2 +-
>  drivers/event/cnxk/cnxk_eventdev.h            |   2 +-
>  drivers/event/dpaa/dpaa_eventdev.c            |   2 +-
>  drivers/event/dpaa2/dpaa2_eventdev.c          |   2 +-
>  drivers/event/octeontx/ssovf_evdev.c          |   2 +-
>  .../event/octeontx2/otx2_evdev_crypto_adptr.c |   2 +-
>  lib/cryptodev/cryptodev_pmd.c                 |  51 +++
>  lib/cryptodev/cryptodev_pmd.h                 |  82 +++-
>  lib/cryptodev/meson.build                     |   4 +-
>  lib/cryptodev/rte_cryptodev.c                 |  50 ++-
>  lib/cryptodev/rte_cryptodev.h                 | 367 +++++++-----------
>  lib/cryptodev/rte_cryptodev_core.h            |  62 +++
>  lib/cryptodev/version.map                     |   7 +-
>  28 files changed, 398 insertions(+), 265 deletions(-)
>  create mode 100644 lib/cryptodev/rte_cryptodev_core.h
> 
> --
> 2.25.1

Re: [dpdk-dev] [PATCH v2 0/5] cryptodev: hide internal structures

[Ji, Kai]

Hi Akhil,

Just ran a quick mutli process test against the patch set, unfortunately it failed on the secondary process enqueue or dequeue.

USER1: Configuring vector 0, using session 0
USER1: Start enqueuing packets on dev 0 qp 0
USER1: Start dequeuing packets on dev 0 qp 0
USER1: Enqueuing - Dequeueing -Segmentation fault (core dumped)

It will happen on any PMD type.

Regards

Kai 

  
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Zhang, Roy Fan
> Sent: Monday, October 11, 2021 5:03 PM
> To: Akhil Goyal <gakhil@marvell.com>; dev@dpdk.org
> Cc: thomas@monjalon.net; david.marchand@redhat.com;
> hemant.agrawal@nxp.com; anoobj@marvell.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> Doherty, Declan <declan.doherty@intel.com>; matan@nvidia.com;
> g.singh@nxp.com; jianjay.zhou@huawei.com; asomalap@amd.com;
> ruifeng.wang@arm.com; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; Nicolau, Radu <radu.nicolau@intel.com>;
> ajit.khaparde@broadcom.com; rnagadheeraj@marvell.com;
> adwivedi@marvell.com; Power, Ciara <ciara.power@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v2 0/5] cryptodev: hide internal structures
> 
> Hi Akhil,
> 
> The approach looks great but we may have to check if it works in multi-
> process environment - since all enqueue/dequeue handlers are set by
> primary process the secondary process may not recognize the fp_ops data.
> 
> We will run a quick test to see if it is true.
> 
> Regards,
> Fan
> 
> > -----Original Message-----
> > From: Akhil Goyal <gakhil@marvell.com>
> > Sent: Monday, October 11, 2021 1:43 PM
> > To: dev@dpdk.org
> > Cc: thomas@monjalon.net; david.marchand@redhat.com;
> > hemant.agrawal@nxp.com; anoobj@marvell.com; De Lara Guarch, Pablo
> > <pablo.de.lara.guarch@intel.com>; Trahe, Fiona
> > <fiona.trahe@intel.com>; Doherty, Declan <declan.doherty@intel.com>;
> > matan@nvidia.com; g.singh@nxp.com; Zhang, Roy Fan
> > <roy.fan.zhang@intel.com>; jianjay.zhou@huawei.com;
> asomalap@amd.com;
> > ruifeng.wang@arm.com; Ananyev, Konstantin
> > <konstantin.ananyev@intel.com>; Nicolau, Radu
> > <radu.nicolau@intel.com>; ajit.khaparde@broadcom.com;
> > rnagadheeraj@marvell.com; adwivedi@marvell.com; Power, Ciara
> > <ciara.power@intel.com>; Akhil Goyal <gakhil@marvell.com>
> > Subject: [PATCH v2 0/5] cryptodev: hide internal structures
> >
> > Structures rte_cryptodev and rte_cryptodev_data are not supposed to be
> > directly used by the application. These are made public as they are
> > used by inline datapath public APIs.
> > This patchset, creates a new rte_cryptodev_core.h file which helps in
> > defining a data structure to hold datapath APIs in a flat array based
> > on the device identifier which is filled by the PMD.
> >
> > Similar series for ethdev and eventdev are also floated on ML.
> > https://patchwork.dpdk.org/project/dpdk/list/?series=19428
> > https://patchwork.dpdk.org/project/dpdk/list/?series=19405
> >
> > changes in v2: align with the latest versions of above series.
> >
> > Akhil Goyal (5):
> >   cryptodev: separate out internal structures
> >   cryptodev: allocate max space for internal qp array
> >   cryptodev: move inline APIs into separate structure
> >   cryptodev: update fast path APIs to use new flat array
> >   cryptodev: move device specific structures
> >
> >  drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c    |   1 -
> >  drivers/crypto/ccp/ccp_dev.h                  |   2 +-
> >  drivers/crypto/cnxk/cn10k_ipsec.c             |   2 +-
> >  drivers/crypto/cnxk/cn9k_ipsec.c              |   2 +-
> >  .../crypto/cnxk/cnxk_cryptodev_capabilities.c |   2 +-
> >  drivers/crypto/cnxk/cnxk_cryptodev_sec.c      |   2 +-
> >  drivers/crypto/nitrox/nitrox_sym_reqmgr.c     |   2 +-
> >  drivers/crypto/octeontx/otx_cryptodev.c       |   1 -
> >  .../crypto/octeontx/otx_cryptodev_hw_access.c |   2 +-
> >  .../crypto/octeontx/otx_cryptodev_hw_access.h |   2 +-
> >  drivers/crypto/octeontx/otx_cryptodev_ops.h   |   2 +-
> >  .../crypto/octeontx2/otx2_cryptodev_mbox.c    |   2 +-
> >  drivers/crypto/scheduler/scheduler_failover.c |   2 +-
> >  .../crypto/scheduler/scheduler_multicore.c    |   2 +-
> >  .../scheduler/scheduler_pkt_size_distr.c      |   2 +-
> >  .../crypto/scheduler/scheduler_roundrobin.c   |   2 +-
> >  drivers/event/cnxk/cnxk_eventdev.h            |   2 +-
> >  drivers/event/dpaa/dpaa_eventdev.c            |   2 +-
> >  drivers/event/dpaa2/dpaa2_eventdev.c          |   2 +-
> >  drivers/event/octeontx/ssovf_evdev.c          |   2 +-
> >  .../event/octeontx2/otx2_evdev_crypto_adptr.c |   2 +-
> >  lib/cryptodev/cryptodev_pmd.c                 |  51 +++
> >  lib/cryptodev/cryptodev_pmd.h                 |  82 +++-
> >  lib/cryptodev/meson.build                     |   4 +-
> >  lib/cryptodev/rte_cryptodev.c                 |  50 ++-
> >  lib/cryptodev/rte_cryptodev.h                 | 367 +++++++-----------
> >  lib/cryptodev/rte_cryptodev_core.h            |  62 +++
> >  lib/cryptodev/version.map                     |   7 +-
> >  28 files changed, 398 insertions(+), 265 deletions(-)  create mode
> > 100644 lib/cryptodev/rte_cryptodev_core.h
> >
> > --
> > 2.25.1

Re: [dpdk-dev] [PATCH v2 0/5] cryptodev: hide internal structures

[Zhang, Roy Fan]

Hi Kai and Akhil,

Thanks Kai for finding the problem. 

To resolve the seg fault for multi-process and keep the structure private,
Instead  of creating the fp_ops we may ditch making
rte_cryptodev_enqueue_burst() and  rte_cryptodev_dequeue_burst() 
inline, and keep everything else as it was? To me these inlines looks not
very useful on performance wise anyway.

Regards,
Fan

> -----Original Message-----
> From: Ji, Kai <kai.ji@intel.com>
> Sent: Monday, October 11, 2021 6:07 PM
> To: Zhang, Roy Fan <roy.fan.zhang@intel.com>; Akhil Goyal
> <gakhil@marvell.com>; dev@dpdk.org
> Cc: thomas@monjalon.net; david.marchand@redhat.com;
> hemant.agrawal@nxp.com; anoobj@marvell.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> Doherty, Declan <declan.doherty@intel.com>; matan@nvidia.com;
> g.singh@nxp.com; jianjay.zhou@huawei.com; asomalap@amd.com;
> ruifeng.wang@arm.com; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; Nicolau, Radu <radu.nicolau@intel.com>;
> ajit.khaparde@broadcom.com; rnagadheeraj@marvell.com;
> adwivedi@marvell.com; Power, Ciara <ciara.power@intel.com>
> Subject: RE: [PATCH v2 0/5] cryptodev: hide internal structures
> 
> Hi Akhil,
> 
> Just ran a quick mutli process test against the patch set, unfortunately it
> failed on the secondary process enqueue or dequeue.
> 
> USER1: Configuring vector 0, using session 0
> USER1: Start enqueuing packets on dev 0 qp 0
> USER1: Start dequeuing packets on dev 0 qp 0
> USER1: Enqueuing - Dequeueing -Segmentation fault (core dumped)
> 
> It will happen on any PMD type.
> 
> Regards
> 
> Kai
> 
> 
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Zhang, Roy Fan
> > Sent: Monday, October 11, 2021 5:03 PM
> > To: Akhil Goyal <gakhil@marvell.com>; dev@dpdk.org
> > Cc: thomas@monjalon.net; david.marchand@redhat.com;
> > hemant.agrawal@nxp.com; anoobj@marvell.com; De Lara Guarch, Pablo
> > <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> > Doherty, Declan <declan.doherty@intel.com>; matan@nvidia.com;
> > g.singh@nxp.com; jianjay.zhou@huawei.com; asomalap@amd.com;
> > ruifeng.wang@arm.com; Ananyev, Konstantin
> > <konstantin.ananyev@intel.com>; Nicolau, Radu
> <radu.nicolau@intel.com>;
> > ajit.khaparde@broadcom.com; rnagadheeraj@marvell.com;
> > adwivedi@marvell.com; Power, Ciara <ciara.power@intel.com>
> > Subject: Re: [dpdk-dev] [PATCH v2 0/5] cryptodev: hide internal structures
> >
> > Hi Akhil,
> >
> > The approach looks great but we may have to check if it works in multi-
> > process environment - since all enqueue/dequeue handlers are set by
> > primary process the secondary process may not recognize the fp_ops data.
> >
> > We will run a quick test to see if it is true.
> >
> > Regards,
> > Fan
> >
> > > -----Original Message-----
> > > From: Akhil Goyal <gakhil@marvell.com>
> > > Sent: Monday, October 11, 2021 1:43 PM
> > > To: dev@dpdk.org
> > > Cc: thomas@monjalon.net; david.marchand@redhat.com;
> > > hemant.agrawal@nxp.com; anoobj@marvell.com; De Lara Guarch, Pablo
> > > <pablo.de.lara.guarch@intel.com>; Trahe, Fiona
> > > <fiona.trahe@intel.com>; Doherty, Declan <declan.doherty@intel.com>;
> > > matan@nvidia.com; g.singh@nxp.com; Zhang, Roy Fan
> > > <roy.fan.zhang@intel.com>; jianjay.zhou@huawei.com;
> > asomalap@amd.com;
> > > ruifeng.wang@arm.com; Ananyev, Konstantin
> > > <konstantin.ananyev@intel.com>; Nicolau, Radu
> > > <radu.nicolau@intel.com>; ajit.khaparde@broadcom.com;
> > > rnagadheeraj@marvell.com; adwivedi@marvell.com; Power, Ciara
> > > <ciara.power@intel.com>; Akhil Goyal <gakhil@marvell.com>
> > > Subject: [PATCH v2 0/5] cryptodev: hide internal structures
> > >
> > > Structures rte_cryptodev and rte_cryptodev_data are not supposed to
> be
> > > directly used by the application. These are made public as they are
> > > used by inline datapath public APIs.
> > > This patchset, creates a new rte_cryptodev_core.h file which helps in
> > > defining a data structure to hold datapath APIs in a flat array based
> > > on the device identifier which is filled by the PMD.
> > >
> > > Similar series for ethdev and eventdev are also floated on ML.
> > > https://patchwork.dpdk.org/project/dpdk/list/?series=19428
> > > https://patchwork.dpdk.org/project/dpdk/list/?series=19405
> > >
> > > changes in v2: align with the latest versions of above series.
> > >
> > > Akhil Goyal (5):
> > >   cryptodev: separate out internal structures
> > >   cryptodev: allocate max space for internal qp array
> > >   cryptodev: move inline APIs into separate structure
> > >   cryptodev: update fast path APIs to use new flat array
> > >   cryptodev: move device specific structures
> > >
> > >  drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c    |   1 -
> > >  drivers/crypto/ccp/ccp_dev.h                  |   2 +-
> > >  drivers/crypto/cnxk/cn10k_ipsec.c             |   2 +-
> > >  drivers/crypto/cnxk/cn9k_ipsec.c              |   2 +-
> > >  .../crypto/cnxk/cnxk_cryptodev_capabilities.c |   2 +-
> > >  drivers/crypto/cnxk/cnxk_cryptodev_sec.c      |   2 +-
> > >  drivers/crypto/nitrox/nitrox_sym_reqmgr.c     |   2 +-
> > >  drivers/crypto/octeontx/otx_cryptodev.c       |   1 -
> > >  .../crypto/octeontx/otx_cryptodev_hw_access.c |   2 +-
> > >  .../crypto/octeontx/otx_cryptodev_hw_access.h |   2 +-
> > >  drivers/crypto/octeontx/otx_cryptodev_ops.h   |   2 +-
> > >  .../crypto/octeontx2/otx2_cryptodev_mbox.c    |   2 +-
> > >  drivers/crypto/scheduler/scheduler_failover.c |   2 +-
> > >  .../crypto/scheduler/scheduler_multicore.c    |   2 +-
> > >  .../scheduler/scheduler_pkt_size_distr.c      |   2 +-
> > >  .../crypto/scheduler/scheduler_roundrobin.c   |   2 +-
> > >  drivers/event/cnxk/cnxk_eventdev.h            |   2 +-
> > >  drivers/event/dpaa/dpaa_eventdev.c            |   2 +-
> > >  drivers/event/dpaa2/dpaa2_eventdev.c          |   2 +-
> > >  drivers/event/octeontx/ssovf_evdev.c          |   2 +-
> > >  .../event/octeontx2/otx2_evdev_crypto_adptr.c |   2 +-
> > >  lib/cryptodev/cryptodev_pmd.c                 |  51 +++
> > >  lib/cryptodev/cryptodev_pmd.h                 |  82 +++-
> > >  lib/cryptodev/meson.build                     |   4 +-
> > >  lib/cryptodev/rte_cryptodev.c                 |  50 ++-
> > >  lib/cryptodev/rte_cryptodev.h                 | 367 +++++++-----------
> > >  lib/cryptodev/rte_cryptodev_core.h            |  62 +++
> > >  lib/cryptodev/version.map                     |   7 +-
> > >  28 files changed, 398 insertions(+), 265 deletions(-)  create mode
> > > 100644 lib/cryptodev/rte_cryptodev_core.h
> > >
> > > --
> > > 2.25.1

Re: [dpdk-dev] [PATCH v2 0/5] cryptodev: hide internal structures

[Ananyev, Konstantin]

Hi Akhil,

> Structures rte_cryptodev and rte_cryptodev_data are not
> supposed to be directly used by the application. These
> are made public as they are used by inline datapath
> public APIs.
> This patchset, creates a new rte_cryptodev_core.h file
> which helps in defining a data structure to hold datapath
> APIs in a flat array based on the device identifier which
> is filled by the PMD.
> 
> Similar series for ethdev and eventdev are also floated on ML.
> https://patchwork.dpdk.org/project/dpdk/list/?series=19428
> https://patchwork.dpdk.org/project/dpdk/list/?series=19405
> 
> changes in v2: align with the latest versions of above series.

Just to let you know this patch set causes to seg-fault ipsec-secgw:
examples/ipsec-secgw/test/run_test.sh -46m
...
[23695833.390785] dpdk-ipsec-secg[2491066]: segfault at 0 ip 0000564325730963 sp 00007fffb9111d00 error 4 in dpdk-ipsec-secgw[564324df0000+134d000]
[23695833.390791] Code: 28 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 00 4c 8d 04 cd 00 00 00 00 49 89 ce 4c 89 e7 4a 8d 34 00 48 8b 46 08 48 89 74 24 18 <48> 8b 08 48 89 88 80 00 00 00 f0 83 44 24 80 00 4c 8b 2e 4d 85 ed

So far, I didn't dig into it any further.
Will have a closer look at Monday.
  
Konstantin

> 
> Akhil Goyal (5):
>   cryptodev: separate out internal structures
>   cryptodev: allocate max space for internal qp array
>   cryptodev: move inline APIs into separate structure
>   cryptodev: update fast path APIs to use new flat array
>   cryptodev: move device specific structures
> 
>  drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c    |   1 -
>  drivers/crypto/ccp/ccp_dev.h                  |   2 +-
>  drivers/crypto/cnxk/cn10k_ipsec.c             |   2 +-
>  drivers/crypto/cnxk/cn9k_ipsec.c              |   2 +-
>  .../crypto/cnxk/cnxk_cryptodev_capabilities.c |   2 +-
>  drivers/crypto/cnxk/cnxk_cryptodev_sec.c      |   2 +-
>  drivers/crypto/nitrox/nitrox_sym_reqmgr.c     |   2 +-
>  drivers/crypto/octeontx/otx_cryptodev.c       |   1 -
>  .../crypto/octeontx/otx_cryptodev_hw_access.c |   2 +-
>  .../crypto/octeontx/otx_cryptodev_hw_access.h |   2 +-
>  drivers/crypto/octeontx/otx_cryptodev_ops.h   |   2 +-
>  .../crypto/octeontx2/otx2_cryptodev_mbox.c    |   2 +-
>  drivers/crypto/scheduler/scheduler_failover.c |   2 +-
>  .../crypto/scheduler/scheduler_multicore.c    |   2 +-
>  .../scheduler/scheduler_pkt_size_distr.c      |   2 +-
>  .../crypto/scheduler/scheduler_roundrobin.c   |   2 +-
>  drivers/event/cnxk/cnxk_eventdev.h            |   2 +-
>  drivers/event/dpaa/dpaa_eventdev.c            |   2 +-
>  drivers/event/dpaa2/dpaa2_eventdev.c          |   2 +-
>  drivers/event/octeontx/ssovf_evdev.c          |   2 +-
>  .../event/octeontx2/otx2_evdev_crypto_adptr.c |   2 +-
>  lib/cryptodev/cryptodev_pmd.c                 |  51 +++
>  lib/cryptodev/cryptodev_pmd.h                 |  82 +++-
>  lib/cryptodev/meson.build                     |   4 +-
>  lib/cryptodev/rte_cryptodev.c                 |  50 ++-
>  lib/cryptodev/rte_cryptodev.h                 | 367 +++++++-----------
>  lib/cryptodev/rte_cryptodev_core.h            |  62 +++
>  lib/cryptodev/version.map                     |   7 +-
>  28 files changed, 398 insertions(+), 265 deletions(-)
>  create mode 100644 lib/cryptodev/rte_cryptodev_core.h
> 
> --
> 2.25.1

Re: [dpdk-dev] [PATCH v2 0/5] cryptodev: hide internal structures

[Akhil Goyal]

> 
> Just to let you know this patch set causes to seg-fault ipsec-secgw:
> examples/ipsec-secgw/test/run_test.sh -46m
> ...
> [23695833.390785] dpdk-ipsec-secg[2491066]: segfault at 0 ip
> 0000564325730963 sp 00007fffb9111d00 error 4 in dpdk-ipsec-
> secgw[564324df0000+134d000]
> [23695833.390791] Code: 28 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 00 4c 8d 04
> cd 00 00 00 00 49 89 ce 4c 89 e7 4a 8d 34 00 48 8b 46 08 48 89 74 24 18 <48>
> 8b 08 48 89 88 80 00 00 00 f0 83 44 24 80 00 4c 8b 2e 4d 85 ed
> 
> So far, I didn't dig into it any further.
> Will have a closer look at Monday.
> 
Thanks for the update, planning to send a next version to fix the multi process issue
Reported by Fan, will look into the ipsec-secgw also over weekend.

Re: [dpdk-dev] [PATCH v2 3/5] cryptodev: move inline APIs into separate structure

[Akhil Goyal]

> Hi Akhil,
> 
> > Move fastpath inline function pointers from rte_cryptodev into a
> > separate structure accessed via a flat array.
> > The intension is to make rte_cryptodev and related structures private
> > to avoid future API/ABI breakages.
> >
> > Signed-off-by: Akhil Goyal <gakhil@marvell.com>
> > ---
> >  lib/cryptodev/cryptodev_pmd.c      | 51
> > ++++++++++++++++++++++++++++++
> >  lib/cryptodev/cryptodev_pmd.h      | 11 +++++++
> >  lib/cryptodev/rte_cryptodev.c      | 29 +++++++++++++++++
> >  lib/cryptodev/rte_cryptodev_core.h | 29 +++++++++++++++++
> >  lib/cryptodev/version.map          |  5 +++
> >  5 files changed, 125 insertions(+)
> >
> > diff --git a/lib/cryptodev/cryptodev_pmd.c
> > b/lib/cryptodev/cryptodev_pmd.c
> > index 44a70ecb35..4646708045 100644
> > --- a/lib/cryptodev/cryptodev_pmd.c
> > +++ b/lib/cryptodev/cryptodev_pmd.c
> > @@ -4,6 +4,7 @@
> >
> >  #include <sys/queue.h>
> >
> > +#include <rte_errno.h>
> >  #include <rte_string_fns.h>
> >  #include <rte_malloc.h>
> >
> > @@ -160,3 +161,53 @@ rte_cryptodev_pmd_destroy(struct rte_cryptodev
> > *cryptodev)
> >
> 
> When a device is removed - aka when rte_pci_remove() is called
> cryptodev_fp_ops_reset() will never be called. This may expose a problem.
> Looks like cryptodev_fp_ops_reset() needs to be called here too.
> 
rte_cryptodev_pmd_destroy internally calls rte_cryptodev_pmd_release_device
and reset is called in that.

Re: [dpdk-dev] [PATCH v2 5/5] cryptodev: move device specific structures

[Akhil Goyal]

> > +struct rte_cryptodev_data {
> > +	/** Device ID for this instance */
> > +	uint8_t dev_id;
> > +	/** Socket ID where memory is allocated */
> > +	uint8_t socket_id;
> > +	/** Unique identifier name */
> > +	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
> > +
> > +	__extension__
> > +	/** Device state: STARTED(1)/STOPPED(0) */
> > +	uint8_t dev_started : 1;
> > +
> > +	/** Session memory pool */
> > +	struct rte_mempool *session_pool;
> 
> Looks like we never used this pool ever - shall we take this chance to remove
> it?

It is getting used in qp release which checks if all sessions are cleared before releasing the QP.

> 
> > +	/** Array of pointers to queue pairs. */
> > +	void **queue_pairs;
> > +	/** Number of device queue pairs. */
> > +	uint16_t nb_queue_pairs;
> 
> queue_pairs are likely the only item we use rte_cryptodev_data in the fast-
> path.
> Also take this chance to move it to the beginning of the structure?

As per latest design, rte_cryptodev_data will not be used in fast path.
In control path, all data path params are set in the fp_array.
Hence no need to change.


> 
> > +
> > +	/** PMD-specific private data */
> > +	void *dev_private;
> > +} __rte_cache_aligned;
> > +
> 
> Not really comments but some suggestions - could be done in the future as
> the
> structure is moved to internal in this patchset.
> Acked-by: Fan Zhang <roy.fan.zhang@intel.com>

[dpdk-dev] [PATCH v3 0/7] cryptodev: hide internal structures

[Akhil Goyal]

Structures rte_cryptodev and rte_cryptodev_data are not
supposed to be directly used by the application. These
are made public as they are used by inline datapath
public APIs.
This patchset, creates a new rte_cryptodev_core.h file
which helps in defining a data structure to hold datapath
APIs in a flat array based on the device identifier which
is filled by the PMD.

Similar series for ethdev and eventdev are also floated on ML.
https://patchwork.dpdk.org/project/dpdk/list/?series=19428
https://patchwork.dpdk.org/project/dpdk/list/?series=19405


changes in
v3: fixed multiprocess and enq-deq callback handling.
v2: align with the latest versions of above series.


Akhil Goyal (7):
  cryptodev: separate out internal structures
  cryptodev: allocate max space for internal qp array
  cryptodev: move inline APIs into separate structure
  cryptodev: add PMD device probe finish API
  drivers/crypto: invoke probing finish function
  cryptodev: update fast path APIs to use new flat array
  cryptodev: move device specific structures

 drivers/crypto/armv8/rte_armv8_pmd.c          |   2 +
 drivers/crypto/bcmfs/bcmfs_sym_pmd.c          |   2 +
 drivers/crypto/caam_jr/caam_jr.c              |   2 +
 drivers/crypto/ccp/ccp_dev.h                  |   2 +-
 drivers/crypto/ccp/rte_ccp_pmd.c              |   2 +
 drivers/crypto/cnxk/cn10k_cryptodev.c         |   2 +
 drivers/crypto/cnxk/cn10k_ipsec.c             |   2 +-
 drivers/crypto/cnxk/cn9k_cryptodev.c          |   2 +
 drivers/crypto/cnxk/cn9k_ipsec.c              |   2 +-
 .../crypto/cnxk/cnxk_cryptodev_capabilities.c |   2 +-
 drivers/crypto/cnxk/cnxk_cryptodev_sec.c      |   2 +-
 drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c   |   4 +-
 drivers/crypto/dpaa_sec/dpaa_sec.c            |   4 +-
 drivers/crypto/ipsec_mb/ipsec_mb_private.c    |   2 +
 drivers/crypto/mlx5/mlx5_crypto.c             |   3 +
 drivers/crypto/mvsam/rte_mrvl_pmd.c           |   2 +
 drivers/crypto/nitrox/nitrox_sym.c            |   3 +
 drivers/crypto/nitrox/nitrox_sym_reqmgr.c     |   2 +-
 drivers/crypto/null/null_crypto_pmd.c         |   2 +
 drivers/crypto/octeontx/otx_cryptodev.c       |   5 +-
 .../crypto/octeontx/otx_cryptodev_hw_access.c |   2 +-
 .../crypto/octeontx/otx_cryptodev_hw_access.h |   2 +-
 drivers/crypto/octeontx/otx_cryptodev_ops.h   |   2 +-
 drivers/crypto/octeontx2/otx2_cryptodev.c     |   2 +
 .../crypto/octeontx2/otx2_cryptodev_mbox.c    |   2 +-
 drivers/crypto/openssl/rte_openssl_pmd.c      |   2 +
 drivers/crypto/qat/qat_asym_pmd.c             |   3 +
 drivers/crypto/qat/qat_sym_pmd.c              |   2 +
 drivers/crypto/scheduler/scheduler_failover.c |   2 +-
 .../crypto/scheduler/scheduler_multicore.c    |   2 +-
 .../scheduler/scheduler_pkt_size_distr.c      |   2 +-
 drivers/crypto/scheduler/scheduler_pmd.c      |   2 +
 .../crypto/scheduler/scheduler_roundrobin.c   |   2 +-
 drivers/crypto/virtio/virtio_cryptodev.c      |   2 +
 drivers/event/cnxk/cnxk_eventdev.h            |   2 +-
 drivers/event/dpaa/dpaa_eventdev.c            |   2 +-
 drivers/event/dpaa2/dpaa2_eventdev.c          |   2 +-
 drivers/event/octeontx/ssovf_evdev.c          |   2 +-
 .../event/octeontx2/otx2_evdev_crypto_adptr.c |   2 +-
 lib/cryptodev/cryptodev_pmd.c                 |  69 +++-
 lib/cryptodev/cryptodev_pmd.h                 |  95 ++++-
 lib/cryptodev/meson.build                     |   4 +-
 lib/cryptodev/rte_cryptodev.c                 |  40 +-
 lib/cryptodev/rte_cryptodev.h                 | 367 +++++++-----------
 lib/cryptodev/rte_cryptodev_core.h            |  62 +++
 lib/cryptodev/version.map                     |   8 +-
 46 files changed, 465 insertions(+), 268 deletions(-)
 create mode 100644 lib/cryptodev/rte_cryptodev_core.h

-- 
2.25.1

[dpdk-dev] [PATCH v3 1/7] cryptodev: separate out internal structures

[Akhil Goyal]

A new header file rte_cryptodev_core.h is added and all
internal data structures which need not be exposed directly to
application are moved to this file. These structures are mostly
used by drivers, but they need to be in the public header file
as they are accessed by datapath inline functions for
performance reasons.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
Tested-by: Rebecca Troy <rebecca.troy@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
---
 lib/cryptodev/cryptodev_pmd.h      |   6 -
 lib/cryptodev/meson.build          |   4 +-
 lib/cryptodev/rte_cryptodev.h      | 360 ++++++++++++-----------------
 lib/cryptodev/rte_cryptodev_core.h | 100 ++++++++
 4 files changed, 245 insertions(+), 225 deletions(-)
 create mode 100644 lib/cryptodev/rte_cryptodev_core.h

diff --git a/lib/cryptodev/cryptodev_pmd.h b/lib/cryptodev/cryptodev_pmd.h
index 8cc9051e09..36606dd10b 100644
--- a/lib/cryptodev/cryptodev_pmd.h
+++ b/lib/cryptodev/cryptodev_pmd.h
@@ -92,12 +92,6 @@ __rte_internal
 struct rte_cryptodev *
 rte_cryptodev_pmd_get_named_dev(const char *name);
 
-/**
- * The pool of rte_cryptodev structures.
- */
-extern struct rte_cryptodev *rte_cryptodevs;
-
-
 /**
  * Definitions of all functions exported by a driver through the
  * the generic structure of type *crypto_dev_ops* supplied in the
diff --git a/lib/cryptodev/meson.build b/lib/cryptodev/meson.build
index 51371c3aa2..289b66ab76 100644
--- a/lib/cryptodev/meson.build
+++ b/lib/cryptodev/meson.build
@@ -14,7 +14,9 @@ headers = files(
         'rte_crypto_sym.h',
         'rte_crypto_asym.h',
 )
-
+indirect_headers += files(
+        'rte_cryptodev_core.h',
+)
 driver_sdk_headers += files(
         'cryptodev_pmd.h',
 )
diff --git a/lib/cryptodev/rte_cryptodev.h b/lib/cryptodev/rte_cryptodev.h
index cdd7168fba..ce0dca72be 100644
--- a/lib/cryptodev/rte_cryptodev.h
+++ b/lib/cryptodev/rte_cryptodev.h
@@ -867,17 +867,6 @@ rte_cryptodev_callback_unregister(uint8_t dev_id,
 		enum rte_cryptodev_event_type event,
 		rte_cryptodev_cb_fn cb_fn, void *cb_arg);
 
-typedef uint16_t (*dequeue_pkt_burst_t)(void *qp,
-		struct rte_crypto_op **ops,	uint16_t nb_ops);
-/**< Dequeue processed packets from queue pair of a device. */
-
-typedef uint16_t (*enqueue_pkt_burst_t)(void *qp,
-		struct rte_crypto_op **ops,	uint16_t nb_ops);
-/**< Enqueue packets for processing on queue pair of a device. */
-
-
-
-
 struct rte_cryptodev_callback;
 
 /** Structure to keep track of registered callbacks */
@@ -907,216 +896,9 @@ struct rte_cryptodev_cb_rcu {
 	/**< RCU QSBR variable per queue pair */
 };
 
-/** The data structure associated with each crypto device. */
-struct rte_cryptodev {
-	dequeue_pkt_burst_t dequeue_burst;
-	/**< Pointer to PMD receive function. */
-	enqueue_pkt_burst_t enqueue_burst;
-	/**< Pointer to PMD transmit function. */
-
-	struct rte_cryptodev_data *data;
-	/**< Pointer to device data */
-	struct rte_cryptodev_ops *dev_ops;
-	/**< Functions exported by PMD */
-	uint64_t feature_flags;
-	/**< Feature flags exposes HW/SW features for the given device */
-	struct rte_device *device;
-	/**< Backing device */
-
-	uint8_t driver_id;
-	/**< Crypto driver identifier*/
-
-	struct rte_cryptodev_cb_list link_intr_cbs;
-	/**< User application callback for interrupts if present */
-
-	void *security_ctx;
-	/**< Context for security ops */
-
-	__extension__
-	uint8_t attached : 1;
-	/**< Flag indicating the device is attached */
-
-	struct rte_cryptodev_cb_rcu *enq_cbs;
-	/**< User application callback for pre enqueue processing */
-
-	struct rte_cryptodev_cb_rcu *deq_cbs;
-	/**< User application callback for post dequeue processing */
-} __rte_cache_aligned;
-
 void *
 rte_cryptodev_get_sec_ctx(uint8_t dev_id);
 
-/**
- *
- * The data part, with no function pointers, associated with each device.
- *
- * This structure is safe to place in shared memory to be common among
- * different processes in a multi-process configuration.
- */
-struct rte_cryptodev_data {
-	uint8_t dev_id;
-	/**< Device ID for this instance */
-	uint8_t socket_id;
-	/**< Socket ID where memory is allocated */
-	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
-	/**< Unique identifier name */
-
-	__extension__
-	uint8_t dev_started : 1;
-	/**< Device state: STARTED(1)/STOPPED(0) */
-
-	struct rte_mempool *session_pool;
-	/**< Session memory pool */
-	void **queue_pairs;
-	/**< Array of pointers to queue pairs. */
-	uint16_t nb_queue_pairs;
-	/**< Number of device queue pairs. */
-
-	void *dev_private;
-	/**< PMD-specific private data */
-} __rte_cache_aligned;
-
-extern struct rte_cryptodev *rte_cryptodevs;
-/**
- *
- * Dequeue a burst of processed crypto operations from a queue on the crypto
- * device. The dequeued operation are stored in *rte_crypto_op* structures
- * whose pointers are supplied in the *ops* array.
- *
- * The rte_cryptodev_dequeue_burst() function returns the number of ops
- * actually dequeued, which is the number of *rte_crypto_op* data structures
- * effectively supplied into the *ops* array.
- *
- * A return value equal to *nb_ops* indicates that the queue contained
- * at least *nb_ops* operations, and this is likely to signify that other
- * processed operations remain in the devices output queue. Applications
- * implementing a "retrieve as many processed operations as possible" policy
- * can check this specific case and keep invoking the
- * rte_cryptodev_dequeue_burst() function until a value less than
- * *nb_ops* is returned.
- *
- * The rte_cryptodev_dequeue_burst() function does not provide any error
- * notification to avoid the corresponding overhead.
- *
- * @param	dev_id		The symmetric crypto device identifier
- * @param	qp_id		The index of the queue pair from which to
- *				retrieve processed packets. The value must be
- *				in the range [0, nb_queue_pair - 1] previously
- *				supplied to rte_cryptodev_configure().
- * @param	ops		The address of an array of pointers to
- *				*rte_crypto_op* structures that must be
- *				large enough to store *nb_ops* pointers in it.
- * @param	nb_ops		The maximum number of operations to dequeue.
- *
- * @return
- *   - The number of operations actually dequeued, which is the number
- *   of pointers to *rte_crypto_op* structures effectively supplied to the
- *   *ops* array.
- */
-static inline uint16_t
-rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
-		struct rte_crypto_op **ops, uint16_t nb_ops)
-{
-	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
-
-	rte_cryptodev_trace_dequeue_burst(dev_id, qp_id, (void **)ops, nb_ops);
-	nb_ops = (*dev->dequeue_burst)
-			(dev->data->queue_pairs[qp_id], ops, nb_ops);
-#ifdef RTE_CRYPTO_CALLBACKS
-	if (unlikely(dev->deq_cbs != NULL)) {
-		struct rte_cryptodev_cb_rcu *list;
-		struct rte_cryptodev_cb *cb;
-
-		/* __ATOMIC_RELEASE memory order was used when the
-		 * call back was inserted into the list.
-		 * Since there is a clear dependency between loading
-		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
-		 * not required.
-		 */
-		list = &dev->deq_cbs[qp_id];
-		rte_rcu_qsbr_thread_online(list->qsbr, 0);
-		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
-
-		while (cb != NULL) {
-			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
-					cb->arg);
-			cb = cb->next;
-		};
-
-		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
-	}
-#endif
-	return nb_ops;
-}
-
-/**
- * Enqueue a burst of operations for processing on a crypto device.
- *
- * The rte_cryptodev_enqueue_burst() function is invoked to place
- * crypto operations on the queue *qp_id* of the device designated by
- * its *dev_id*.
- *
- * The *nb_ops* parameter is the number of operations to process which are
- * supplied in the *ops* array of *rte_crypto_op* structures.
- *
- * The rte_cryptodev_enqueue_burst() function returns the number of
- * operations it actually enqueued for processing. A return value equal to
- * *nb_ops* means that all packets have been enqueued.
- *
- * @param	dev_id		The identifier of the device.
- * @param	qp_id		The index of the queue pair which packets are
- *				to be enqueued for processing. The value
- *				must be in the range [0, nb_queue_pairs - 1]
- *				previously supplied to
- *				 *rte_cryptodev_configure*.
- * @param	ops		The address of an array of *nb_ops* pointers
- *				to *rte_crypto_op* structures which contain
- *				the crypto operations to be processed.
- * @param	nb_ops		The number of operations to process.
- *
- * @return
- * The number of operations actually enqueued on the crypto device. The return
- * value can be less than the value of the *nb_ops* parameter when the
- * crypto devices queue is full or if invalid parameters are specified in
- * a *rte_crypto_op*.
- */
-static inline uint16_t
-rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
-		struct rte_crypto_op **ops, uint16_t nb_ops)
-{
-	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
-
-#ifdef RTE_CRYPTO_CALLBACKS
-	if (unlikely(dev->enq_cbs != NULL)) {
-		struct rte_cryptodev_cb_rcu *list;
-		struct rte_cryptodev_cb *cb;
-
-		/* __ATOMIC_RELEASE memory order was used when the
-		 * call back was inserted into the list.
-		 * Since there is a clear dependency between loading
-		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
-		 * not required.
-		 */
-		list = &dev->enq_cbs[qp_id];
-		rte_rcu_qsbr_thread_online(list->qsbr, 0);
-		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
-
-		while (cb != NULL) {
-			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
-					cb->arg);
-			cb = cb->next;
-		};
-
-		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
-	}
-#endif
-
-	rte_cryptodev_trace_enqueue_burst(dev_id, qp_id, (void **)ops, nb_ops);
-	return (*dev->enqueue_burst)(
-			dev->data->queue_pairs[qp_id], ops, nb_ops);
-}
-
-
 /** Cryptodev symmetric crypto session
  * Each session is derived from a fixed xform chain. Therefore each session
  * has a fixed algo, key, op-type, digest_len etc.
@@ -2009,6 +1791,148 @@ int rte_cryptodev_remove_deq_callback(uint8_t dev_id,
 				      uint16_t qp_id,
 				      struct rte_cryptodev_cb *cb);
 
+#include <rte_cryptodev_core.h>
+/**
+ *
+ * Dequeue a burst of processed crypto operations from a queue on the crypto
+ * device. The dequeued operation are stored in *rte_crypto_op* structures
+ * whose pointers are supplied in the *ops* array.
+ *
+ * The rte_cryptodev_dequeue_burst() function returns the number of ops
+ * actually dequeued, which is the number of *rte_crypto_op* data structures
+ * effectively supplied into the *ops* array.
+ *
+ * A return value equal to *nb_ops* indicates that the queue contained
+ * at least *nb_ops* operations, and this is likely to signify that other
+ * processed operations remain in the devices output queue. Applications
+ * implementing a "retrieve as many processed operations as possible" policy
+ * can check this specific case and keep invoking the
+ * rte_cryptodev_dequeue_burst() function until a value less than
+ * *nb_ops* is returned.
+ *
+ * The rte_cryptodev_dequeue_burst() function does not provide any error
+ * notification to avoid the corresponding overhead.
+ *
+ * @param	dev_id		The symmetric crypto device identifier
+ * @param	qp_id		The index of the queue pair from which to
+ *				retrieve processed packets. The value must be
+ *				in the range [0, nb_queue_pair - 1] previously
+ *				supplied to rte_cryptodev_configure().
+ * @param	ops		The address of an array of pointers to
+ *				*rte_crypto_op* structures that must be
+ *				large enough to store *nb_ops* pointers in it.
+ * @param	nb_ops		The maximum number of operations to dequeue.
+ *
+ * @return
+ *   - The number of operations actually dequeued, which is the number
+ *   of pointers to *rte_crypto_op* structures effectively supplied to the
+ *   *ops* array.
+ */
+static inline uint16_t
+rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_crypto_op **ops, uint16_t nb_ops)
+{
+	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
+
+	rte_cryptodev_trace_dequeue_burst(dev_id, qp_id, (void **)ops, nb_ops);
+	nb_ops = (*dev->dequeue_burst)
+			(dev->data->queue_pairs[qp_id], ops, nb_ops);
+#ifdef RTE_CRYPTO_CALLBACKS
+	if (unlikely(dev->deq_cbs != NULL)) {
+		struct rte_cryptodev_cb_rcu *list;
+		struct rte_cryptodev_cb *cb;
+
+		/* __ATOMIC_RELEASE memory order was used when the
+		 * call back was inserted into the list.
+		 * Since there is a clear dependency between loading
+		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
+		 * not required.
+		 */
+		list = &dev->deq_cbs[qp_id];
+		rte_rcu_qsbr_thread_online(list->qsbr, 0);
+		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
+
+		while (cb != NULL) {
+			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
+					cb->arg);
+			cb = cb->next;
+		};
+
+		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
+	}
+#endif
+	return nb_ops;
+}
+
+/**
+ * Enqueue a burst of operations for processing on a crypto device.
+ *
+ * The rte_cryptodev_enqueue_burst() function is invoked to place
+ * crypto operations on the queue *qp_id* of the device designated by
+ * its *dev_id*.
+ *
+ * The *nb_ops* parameter is the number of operations to process which are
+ * supplied in the *ops* array of *rte_crypto_op* structures.
+ *
+ * The rte_cryptodev_enqueue_burst() function returns the number of
+ * operations it actually enqueued for processing. A return value equal to
+ * *nb_ops* means that all packets have been enqueued.
+ *
+ * @param	dev_id		The identifier of the device.
+ * @param	qp_id		The index of the queue pair which packets are
+ *				to be enqueued for processing. The value
+ *				must be in the range [0, nb_queue_pairs - 1]
+ *				previously supplied to
+ *				 *rte_cryptodev_configure*.
+ * @param	ops		The address of an array of *nb_ops* pointers
+ *				to *rte_crypto_op* structures which contain
+ *				the crypto operations to be processed.
+ * @param	nb_ops		The number of operations to process.
+ *
+ * @return
+ * The number of operations actually enqueued on the crypto device. The return
+ * value can be less than the value of the *nb_ops* parameter when the
+ * crypto devices queue is full or if invalid parameters are specified in
+ * a *rte_crypto_op*.
+ */
+static inline uint16_t
+rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_crypto_op **ops, uint16_t nb_ops)
+{
+	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
+
+#ifdef RTE_CRYPTO_CALLBACKS
+	if (unlikely(dev->enq_cbs != NULL)) {
+		struct rte_cryptodev_cb_rcu *list;
+		struct rte_cryptodev_cb *cb;
+
+		/* __ATOMIC_RELEASE memory order was used when the
+		 * call back was inserted into the list.
+		 * Since there is a clear dependency between loading
+		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
+		 * not required.
+		 */
+		list = &dev->enq_cbs[qp_id];
+		rte_rcu_qsbr_thread_online(list->qsbr, 0);
+		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
+
+		while (cb != NULL) {
+			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
+					cb->arg);
+			cb = cb->next;
+		};
+
+		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
+	}
+#endif
+
+	rte_cryptodev_trace_enqueue_burst(dev_id, qp_id, (void **)ops, nb_ops);
+	return (*dev->enqueue_burst)(
+			dev->data->queue_pairs[qp_id], ops, nb_ops);
+}
+
+
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/cryptodev/rte_cryptodev_core.h b/lib/cryptodev/rte_cryptodev_core.h
new file mode 100644
index 0000000000..1633e55889
--- /dev/null
+++ b/lib/cryptodev/rte_cryptodev_core.h
@@ -0,0 +1,100 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2021 Marvell.
+ */
+
+#ifndef _RTE_CRYPTODEV_CORE_H_
+#define _RTE_CRYPTODEV_CORE_H_
+
+/**
+ * @file
+ *
+ * RTE Crypto Device internal header.
+ *
+ * This header contains internal data types. But they are still part of the
+ * public API because they are used by inline functions in the published API.
+ *
+ * Applications should not use these directly.
+ *
+ */
+
+typedef uint16_t (*dequeue_pkt_burst_t)(void *qp,
+		struct rte_crypto_op **ops,	uint16_t nb_ops);
+/**< Dequeue processed packets from queue pair of a device. */
+
+typedef uint16_t (*enqueue_pkt_burst_t)(void *qp,
+		struct rte_crypto_op **ops,	uint16_t nb_ops);
+/**< Enqueue packets for processing on queue pair of a device. */
+
+/**
+ * @internal
+ * The data part, with no function pointers, associated with each device.
+ *
+ * This structure is safe to place in shared memory to be common among
+ * different processes in a multi-process configuration.
+ */
+struct rte_cryptodev_data {
+	uint8_t dev_id;
+	/**< Device ID for this instance */
+	uint8_t socket_id;
+	/**< Socket ID where memory is allocated */
+	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
+	/**< Unique identifier name */
+
+	__extension__
+	uint8_t dev_started : 1;
+	/**< Device state: STARTED(1)/STOPPED(0) */
+
+	struct rte_mempool *session_pool;
+	/**< Session memory pool */
+	void **queue_pairs;
+	/**< Array of pointers to queue pairs. */
+	uint16_t nb_queue_pairs;
+	/**< Number of device queue pairs. */
+
+	void *dev_private;
+	/**< PMD-specific private data */
+} __rte_cache_aligned;
+
+
+/** @internal The data structure associated with each crypto device. */
+struct rte_cryptodev {
+	dequeue_pkt_burst_t dequeue_burst;
+	/**< Pointer to PMD receive function. */
+	enqueue_pkt_burst_t enqueue_burst;
+	/**< Pointer to PMD transmit function. */
+
+	struct rte_cryptodev_data *data;
+	/**< Pointer to device data */
+	struct rte_cryptodev_ops *dev_ops;
+	/**< Functions exported by PMD */
+	uint64_t feature_flags;
+	/**< Feature flags exposes HW/SW features for the given device */
+	struct rte_device *device;
+	/**< Backing device */
+
+	uint8_t driver_id;
+	/**< Crypto driver identifier*/
+
+	struct rte_cryptodev_cb_list link_intr_cbs;
+	/**< User application callback for interrupts if present */
+
+	void *security_ctx;
+	/**< Context for security ops */
+
+	__extension__
+	uint8_t attached : 1;
+	/**< Flag indicating the device is attached */
+
+	struct rte_cryptodev_cb_rcu *enq_cbs;
+	/**< User application callback for pre enqueue processing */
+
+	struct rte_cryptodev_cb_rcu *deq_cbs;
+	/**< User application callback for post dequeue processing */
+} __rte_cache_aligned;
+
+/**
+ * The pool of rte_cryptodev structures.
+ */
+extern struct rte_cryptodev *rte_cryptodevs;
+
+#endif /* _RTE_CRYPTODEV_CORE_H_ */
-- 
2.25.1

[dpdk-dev] [PATCH v3 2/7] cryptodev: allocate max space for internal qp array

[Akhil Goyal]

At queue_pair config stage, allocate memory for maximum
number of queue pair pointers that a device can support.

This will allow fast path APIs(enqueue_burst/dequeue_burst) to
refer pointer to internal QP data without checking for currently
configured QPs.
This is required to hide the rte_cryptodev and rte_cryptodev_data
structure from user.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
---
 lib/cryptodev/rte_cryptodev.c | 21 +++------------------
 1 file changed, 3 insertions(+), 18 deletions(-)

diff --git a/lib/cryptodev/rte_cryptodev.c b/lib/cryptodev/rte_cryptodev.c
index b913c434c5..eb86e629aa 100644
--- a/lib/cryptodev/rte_cryptodev.c
+++ b/lib/cryptodev/rte_cryptodev.c
@@ -978,7 +978,8 @@ rte_cryptodev_queue_pairs_config(struct rte_cryptodev *dev, uint16_t nb_qpairs,
 	if (dev->data->queue_pairs == NULL) { /* first time configuration */
 		dev->data->queue_pairs = rte_zmalloc_socket(
 				"cryptodev->queue_pairs",
-				sizeof(dev->data->queue_pairs[0]) * nb_qpairs,
+				sizeof(dev->data->queue_pairs[0]) *
+				dev_info.max_nb_queue_pairs,
 				RTE_CACHE_LINE_SIZE, socket_id);
 
 		if (dev->data->queue_pairs == NULL) {
@@ -1001,25 +1002,9 @@ rte_cryptodev_queue_pairs_config(struct rte_cryptodev *dev, uint16_t nb_qpairs,
 			ret = (*dev->dev_ops->queue_pair_release)(dev, i);
 			if (ret < 0)
 				return ret;
+			qp[i] = NULL;
 		}
 
-		qp = rte_realloc(qp, sizeof(qp[0]) * nb_qpairs,
-				RTE_CACHE_LINE_SIZE);
-		if (qp == NULL) {
-			CDEV_LOG_ERR("failed to realloc qp meta data,"
-						" nb_queues %u", nb_qpairs);
-			return -(ENOMEM);
-		}
-
-		if (nb_qpairs > old_nb_queues) {
-			uint16_t new_qs = nb_qpairs - old_nb_queues;
-
-			memset(qp + old_nb_queues, 0,
-				sizeof(qp[0]) * new_qs);
-		}
-
-		dev->data->queue_pairs = qp;
-
 	}
 	dev->data->nb_queue_pairs = nb_qpairs;
 	return 0;
-- 
2.25.1

[dpdk-dev] [PATCH v3 3/7] cryptodev: move inline APIs into separate structure

[Akhil Goyal]

Move fastpath inline function pointers from rte_cryptodev into a
separate structure accessed via a flat array.
The intension is to make rte_cryptodev and related structures private
to avoid future API/ABI breakages.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
Tested-by: Rebecca Troy <rebecca.troy@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
---
 lib/cryptodev/cryptodev_pmd.c      | 53 +++++++++++++++++++++++++++++-
 lib/cryptodev/cryptodev_pmd.h      | 11 +++++++
 lib/cryptodev/rte_cryptodev.c      | 19 +++++++++++
 lib/cryptodev/rte_cryptodev_core.h | 29 ++++++++++++++++
 lib/cryptodev/version.map          |  5 +++
 5 files changed, 116 insertions(+), 1 deletion(-)

diff --git a/lib/cryptodev/cryptodev_pmd.c b/lib/cryptodev/cryptodev_pmd.c
index 44a70ecb35..fd74543682 100644
--- a/lib/cryptodev/cryptodev_pmd.c
+++ b/lib/cryptodev/cryptodev_pmd.c
@@ -3,7 +3,7 @@
  */
 
 #include <sys/queue.h>
-
+#include <rte_errno.h>
 #include <rte_string_fns.h>
 #include <rte_malloc.h>
 
@@ -160,3 +160,54 @@ rte_cryptodev_pmd_destroy(struct rte_cryptodev *cryptodev)
 
 	return 0;
 }
+
+static uint16_t
+dummy_crypto_enqueue_burst(__rte_unused void *qp,
+			   __rte_unused struct rte_crypto_op **ops,
+			   __rte_unused uint16_t nb_ops)
+{
+	CDEV_LOG_ERR(
+		"crypto enqueue burst requested for unconfigured device");
+	rte_errno = ENOTSUP;
+	return 0;
+}
+
+static uint16_t
+dummy_crypto_dequeue_burst(__rte_unused void *qp,
+			   __rte_unused struct rte_crypto_op **ops,
+			   __rte_unused uint16_t nb_ops)
+{
+	CDEV_LOG_ERR(
+		"crypto dequeue burst requested for unconfigured device");
+	rte_errno = ENOTSUP;
+	return 0;
+}
+
+void
+cryptodev_fp_ops_reset(struct rte_crypto_fp_ops *fp_ops)
+{
+	static struct rte_cryptodev_cb_rcu dummy_cb[RTE_MAX_QUEUES_PER_PORT];
+	static void *dummy_data[RTE_MAX_QUEUES_PER_PORT];
+	static const struct rte_crypto_fp_ops dummy = {
+		.enqueue_burst = dummy_crypto_enqueue_burst,
+		.dequeue_burst = dummy_crypto_dequeue_burst,
+		.qp = {
+			.data = dummy_data,
+			.enq_cb = dummy_cb,
+			.deq_cb = dummy_cb,
+		},
+	};
+
+	*fp_ops = dummy;
+}
+
+void
+cryptodev_fp_ops_set(struct rte_crypto_fp_ops *fp_ops,
+		     const struct rte_cryptodev *dev)
+{
+	fp_ops->enqueue_burst = dev->enqueue_burst;
+	fp_ops->dequeue_burst = dev->dequeue_burst;
+	fp_ops->qp.data = dev->data->queue_pairs;
+	fp_ops->qp.enq_cb = dev->enq_cbs;
+	fp_ops->qp.deq_cb = dev->deq_cbs;
+}
diff --git a/lib/cryptodev/cryptodev_pmd.h b/lib/cryptodev/cryptodev_pmd.h
index 36606dd10b..a71edbb991 100644
--- a/lib/cryptodev/cryptodev_pmd.h
+++ b/lib/cryptodev/cryptodev_pmd.h
@@ -516,6 +516,17 @@ RTE_INIT(init_ ##driver_id)\
 	driver_id = rte_cryptodev_allocate_driver(&crypto_drv, &(drv));\
 }
 
+/* Reset crypto device fastpath APIs to dummy values. */
+__rte_internal
+void
+cryptodev_fp_ops_reset(struct rte_crypto_fp_ops *fp_ops);
+
+/* Setup crypto device fastpath APIs. */
+__rte_internal
+void
+cryptodev_fp_ops_set(struct rte_crypto_fp_ops *fp_ops,
+		     const struct rte_cryptodev *dev);
+
 static inline void *
 get_sym_session_private_data(const struct rte_cryptodev_sym_session *sess,
 		uint8_t driver_id) {
diff --git a/lib/cryptodev/rte_cryptodev.c b/lib/cryptodev/rte_cryptodev.c
index eb86e629aa..305e013ebb 100644
--- a/lib/cryptodev/rte_cryptodev.c
+++ b/lib/cryptodev/rte_cryptodev.c
@@ -53,6 +53,9 @@ static struct rte_cryptodev_global cryptodev_globals = {
 		.nb_devs		= 0
 };
 
+/* Public fastpath APIs. */
+struct rte_crypto_fp_ops rte_crypto_fp_ops[RTE_CRYPTO_MAX_DEVS];
+
 /* spinlock for crypto device callbacks */
 static rte_spinlock_t rte_cryptodev_cb_lock = RTE_SPINLOCK_INITIALIZER;
 
@@ -917,6 +920,8 @@ rte_cryptodev_pmd_release_device(struct rte_cryptodev *cryptodev)
 
 	dev_id = cryptodev->data->dev_id;
 
+	cryptodev_fp_ops_reset(rte_crypto_fp_ops + dev_id);
+
 	/* Close device only if device operations have been set */
 	if (cryptodev->dev_ops) {
 		ret = rte_cryptodev_close(dev_id);
@@ -1080,6 +1085,9 @@ rte_cryptodev_start(uint8_t dev_id)
 	}
 
 	diag = (*dev->dev_ops->dev_start)(dev);
+	/* expose selection of PMD fast-path functions */
+	cryptodev_fp_ops_set(rte_crypto_fp_ops + dev_id, dev);
+
 	rte_cryptodev_trace_start(dev_id, diag);
 	if (diag == 0)
 		dev->data->dev_started = 1;
@@ -1109,6 +1117,9 @@ rte_cryptodev_stop(uint8_t dev_id)
 		return;
 	}
 
+	/* point fast-path functions to dummy ones */
+	cryptodev_fp_ops_reset(rte_crypto_fp_ops + dev_id);
+
 	(*dev->dev_ops->dev_stop)(dev);
 	rte_cryptodev_trace_stop(dev_id);
 	dev->data->dev_started = 0;
@@ -2411,3 +2422,11 @@ rte_cryptodev_allocate_driver(struct cryptodev_driver *crypto_drv,
 
 	return nb_drivers++;
 }
+
+RTE_INIT(cryptodev_init_fp_ops)
+{
+	uint32_t i;
+
+	for (i = 0; i != RTE_DIM(rte_crypto_fp_ops); i++)
+		cryptodev_fp_ops_reset(rte_crypto_fp_ops + i);
+}
diff --git a/lib/cryptodev/rte_cryptodev_core.h b/lib/cryptodev/rte_cryptodev_core.h
index 1633e55889..e9e9a44b3c 100644
--- a/lib/cryptodev/rte_cryptodev_core.h
+++ b/lib/cryptodev/rte_cryptodev_core.h
@@ -25,6 +25,35 @@ typedef uint16_t (*enqueue_pkt_burst_t)(void *qp,
 		struct rte_crypto_op **ops,	uint16_t nb_ops);
 /**< Enqueue packets for processing on queue pair of a device. */
 
+/**
+ * @internal
+ * Structure used to hold opaque pointers to internal ethdev Rx/Tx
+ * queues data.
+ * The main purpose to expose these pointers at all - allow compiler
+ * to fetch this data for fast-path cryptodev inline functions in advance.
+ */
+struct rte_cryptodev_qpdata {
+	/** points to array of internal queue pair data pointers. */
+	void **data;
+	/** points to array of enqueue callback data pointers */
+	struct rte_cryptodev_cb_rcu *enq_cb;
+	/** points to array of dequeue callback data pointers */
+	struct rte_cryptodev_cb_rcu *deq_cb;
+};
+
+struct rte_crypto_fp_ops {
+	/** PMD enqueue burst function. */
+	enqueue_pkt_burst_t enqueue_burst;
+	/** PMD dequeue burst function. */
+	dequeue_pkt_burst_t dequeue_burst;
+	/** Internal queue pair data pointers. */
+	struct rte_cryptodev_qpdata qp;
+	/** Reserved for future ops. */
+	uintptr_t reserved[4];
+} __rte_cache_aligned;
+
+extern struct rte_crypto_fp_ops rte_crypto_fp_ops[RTE_CRYPTO_MAX_DEVS];
+
 /**
  * @internal
  * The data part, with no function pointers, associated with each device.
diff --git a/lib/cryptodev/version.map b/lib/cryptodev/version.map
index 43cf937e40..ed62ced221 100644
--- a/lib/cryptodev/version.map
+++ b/lib/cryptodev/version.map
@@ -45,6 +45,9 @@ DPDK_22 {
 	rte_cryptodev_sym_session_init;
 	rte_cryptodevs;
 
+	#added in 21.11
+	rte_crypto_fp_ops;
+
 	local: *;
 };
 
@@ -109,6 +112,8 @@ EXPERIMENTAL {
 INTERNAL {
 	global:
 
+	cryptodev_fp_ops_reset;
+	cryptodev_fp_ops_set;
 	rte_cryptodev_allocate_driver;
 	rte_cryptodev_pmd_allocate;
 	rte_cryptodev_pmd_callback_process;
-- 
2.25.1

[dpdk-dev] [PATCH v3 4/7] cryptodev: add PMD device probe finish API

[Akhil Goyal]

Added a rte_cryptodev_pmd_probing_finish API which
need to be called by the PMD after the device is initialized
completely. This will set the fast path function pointers
in the flat array for secondary process. For primary process,
these are set in rte_cryptodev_start.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
---
 lib/cryptodev/cryptodev_pmd.c | 16 ++++++++++++++++
 lib/cryptodev/cryptodev_pmd.h | 13 +++++++++++++
 lib/cryptodev/version.map     |  1 +
 3 files changed, 30 insertions(+)

diff --git a/lib/cryptodev/cryptodev_pmd.c b/lib/cryptodev/cryptodev_pmd.c
index fd74543682..739a0b3f34 100644
--- a/lib/cryptodev/cryptodev_pmd.c
+++ b/lib/cryptodev/cryptodev_pmd.c
@@ -161,6 +161,22 @@ rte_cryptodev_pmd_destroy(struct rte_cryptodev *cryptodev)
 	return 0;
 }
 
+void
+rte_cryptodev_pmd_probing_finish(struct rte_cryptodev *cryptodev)
+{
+	if (cryptodev == NULL)
+		return;
+	/*
+	 * for secondary process, at that point we expect device
+	 * to be already 'usable', so shared data and all function
+	 * pointers for fast-path devops have to be setup properly
+	 * inside rte_cryptodev.
+	 */
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
+		cryptodev_fp_ops_set(rte_crypto_fp_ops +
+				cryptodev->data->dev_id, cryptodev);
+}
+
 static uint16_t
 dummy_crypto_enqueue_burst(__rte_unused void *qp,
 			   __rte_unused struct rte_crypto_op **ops,
diff --git a/lib/cryptodev/cryptodev_pmd.h b/lib/cryptodev/cryptodev_pmd.h
index a71edbb991..9bb1e47ae4 100644
--- a/lib/cryptodev/cryptodev_pmd.h
+++ b/lib/cryptodev/cryptodev_pmd.h
@@ -509,6 +509,19 @@ __rte_internal
 uint8_t rte_cryptodev_allocate_driver(struct cryptodev_driver *crypto_drv,
 		const struct rte_driver *drv);
 
+/**
+ * @internal
+ * This is the last step of device probing. It must be called after a
+ * cryptodev is allocated and initialized successfully.
+ *
+ * @param	dev	Pointer to cryptodev struct
+ *
+ * @return
+ *  void
+ */
+__rte_internal
+void
+rte_cryptodev_pmd_probing_finish(struct rte_cryptodev *dev);
 
 #define RTE_PMD_REGISTER_CRYPTO_DRIVER(crypto_drv, drv, driver_id)\
 RTE_INIT(init_ ##driver_id)\
diff --git a/lib/cryptodev/version.map b/lib/cryptodev/version.map
index ed62ced221..157dac521d 100644
--- a/lib/cryptodev/version.map
+++ b/lib/cryptodev/version.map
@@ -123,5 +123,6 @@ INTERNAL {
 	rte_cryptodev_pmd_get_dev;
 	rte_cryptodev_pmd_get_named_dev;
 	rte_cryptodev_pmd_parse_input_args;
+	rte_cryptodev_pmd_probing_finish;
 	rte_cryptodev_pmd_release_device;
 };
-- 
2.25.1

[dpdk-dev] [PATCH v3 5/7] drivers/crypto: invoke probing finish function

[Akhil Goyal]

Invoke event_dev_probing_finish() function at the end of probing,
this function sets the function pointers in the fp_ops flat array
in case of secondary process.
For primary process, fp_ops is updated in rte_cryptodev_start().

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
---
 drivers/crypto/armv8/rte_armv8_pmd.c        | 2 ++
 drivers/crypto/bcmfs/bcmfs_sym_pmd.c        | 2 ++
 drivers/crypto/caam_jr/caam_jr.c            | 2 ++
 drivers/crypto/ccp/rte_ccp_pmd.c            | 2 ++
 drivers/crypto/cnxk/cn10k_cryptodev.c       | 2 ++
 drivers/crypto/cnxk/cn9k_cryptodev.c        | 2 ++
 drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c | 4 +++-
 drivers/crypto/dpaa_sec/dpaa_sec.c          | 4 +++-
 drivers/crypto/ipsec_mb/ipsec_mb_private.c  | 2 ++
 drivers/crypto/mlx5/mlx5_crypto.c           | 3 +++
 drivers/crypto/mvsam/rte_mrvl_pmd.c         | 2 ++
 drivers/crypto/nitrox/nitrox_sym.c          | 3 +++
 drivers/crypto/null/null_crypto_pmd.c       | 2 ++
 drivers/crypto/octeontx/otx_cryptodev.c     | 4 +++-
 drivers/crypto/octeontx2/otx2_cryptodev.c   | 2 ++
 drivers/crypto/openssl/rte_openssl_pmd.c    | 2 ++
 drivers/crypto/qat/qat_asym_pmd.c           | 3 +++
 drivers/crypto/qat/qat_sym_pmd.c            | 2 ++
 drivers/crypto/scheduler/scheduler_pmd.c    | 2 ++
 drivers/crypto/virtio/virtio_cryptodev.c    | 2 ++
 20 files changed, 46 insertions(+), 3 deletions(-)

diff --git a/drivers/crypto/armv8/rte_armv8_pmd.c b/drivers/crypto/armv8/rte_armv8_pmd.c
index 36a1a9bb4f..32127a874c 100644
--- a/drivers/crypto/armv8/rte_armv8_pmd.c
+++ b/drivers/crypto/armv8/rte_armv8_pmd.c
@@ -802,6 +802,8 @@ cryptodev_armv8_crypto_create(const char *name,
 
 	internals->max_nb_qpairs = init_params->max_nb_queue_pairs;
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	return 0;
 
 init_error:
diff --git a/drivers/crypto/bcmfs/bcmfs_sym_pmd.c b/drivers/crypto/bcmfs/bcmfs_sym_pmd.c
index d1dd22823e..bc7fb67218 100644
--- a/drivers/crypto/bcmfs/bcmfs_sym_pmd.c
+++ b/drivers/crypto/bcmfs/bcmfs_sym_pmd.c
@@ -394,6 +394,8 @@ bcmfs_sym_dev_create(struct bcmfs_device *fsdev)
 	internals->sym_dev_id = cryptodev->data->dev_id;
 	internals->fsdev_capabilities = bcmfs_sym_get_capabilities();
 
+	rte_cryptodev_pmd_probing_finish(cryptodev);
+
 	BCMFS_LOG(DEBUG, "Created bcmfs-sym device %s as cryptodev instance %d",
 		  cryptodev->data->name, internals->sym_dev_id);
 	return 0;
diff --git a/drivers/crypto/caam_jr/caam_jr.c b/drivers/crypto/caam_jr/caam_jr.c
index 258750afe7..8c56610ac8 100644
--- a/drivers/crypto/caam_jr/caam_jr.c
+++ b/drivers/crypto/caam_jr/caam_jr.c
@@ -2368,6 +2368,8 @@ caam_jr_dev_init(const char *name,
 	security_instance->sess_cnt = 0;
 	dev->security_ctx = security_instance;
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	RTE_LOG(INFO, PMD, "%s cryptodev init\n", dev->data->name);
 
 	return 0;
diff --git a/drivers/crypto/ccp/rte_ccp_pmd.c b/drivers/crypto/ccp/rte_ccp_pmd.c
index a54d81de46..a35a8cd775 100644
--- a/drivers/crypto/ccp/rte_ccp_pmd.c
+++ b/drivers/crypto/ccp/rte_ccp_pmd.c
@@ -271,6 +271,8 @@ cryptodev_ccp_create(const char *name,
 	internals->auth_opt = init_params->auth_opt;
 	internals->crypto_num_dev = cryptodev_cnt;
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	return 0;
 
 init_error:
diff --git a/drivers/crypto/cnxk/cn10k_cryptodev.c b/drivers/crypto/cnxk/cn10k_cryptodev.c
index 869d322d9b..1e0a7b91c9 100644
--- a/drivers/crypto/cnxk/cn10k_cryptodev.c
+++ b/drivers/crypto/cnxk/cn10k_cryptodev.c
@@ -102,6 +102,8 @@ cn10k_cpt_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	cn10k_cpt_set_enqdeq_fns(dev);
 	cn10k_sec_ops_override();
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	return 0;
 
 dev_fini:
diff --git a/drivers/crypto/cnxk/cn9k_cryptodev.c b/drivers/crypto/cnxk/cn9k_cryptodev.c
index 54df06eec0..54cd77e575 100644
--- a/drivers/crypto/cnxk/cn9k_cryptodev.c
+++ b/drivers/crypto/cnxk/cn9k_cryptodev.c
@@ -102,6 +102,8 @@ cn9k_cpt_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	cn9k_cpt_set_enqdeq_fns(dev);
 	cn9k_sec_ops_override();
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	return 0;
 
 dev_fini:
diff --git a/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c b/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c
index 4eb3615250..cb2ad435bf 100644
--- a/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c
+++ b/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c
@@ -4032,8 +4032,10 @@ cryptodev_dpaa2_sec_probe(struct rte_dpaa2_driver *dpaa2_drv __rte_unused,
 
 	/* Invoke PMD device initialization function */
 	retval = dpaa2_sec_dev_init(cryptodev);
-	if (retval == 0)
+	if (retval == 0) {
+		rte_cryptodev_pmd_probing_finish(cryptodev);
 		return 0;
+	}
 
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		rte_free(cryptodev->data->dev_private);
diff --git a/drivers/crypto/dpaa_sec/dpaa_sec.c b/drivers/crypto/dpaa_sec/dpaa_sec.c
index c7ef1c7b0f..454b9c4785 100644
--- a/drivers/crypto/dpaa_sec/dpaa_sec.c
+++ b/drivers/crypto/dpaa_sec/dpaa_sec.c
@@ -3594,8 +3594,10 @@ cryptodev_dpaa_sec_probe(struct rte_dpaa_driver *dpaa_drv __rte_unused,
 
 	/* Invoke PMD device initialization function */
 	retval = dpaa_sec_dev_init(cryptodev);
-	if (retval == 0)
+	if (retval == 0) {
+		rte_cryptodev_pmd_probing_finish(cryptodev);
 		return 0;
+	}
 
 	retval = -ENXIO;
 out:
diff --git a/drivers/crypto/ipsec_mb/ipsec_mb_private.c b/drivers/crypto/ipsec_mb/ipsec_mb_private.c
index d896fa20b4..aab42c360c 100644
--- a/drivers/crypto/ipsec_mb/ipsec_mb_private.c
+++ b/drivers/crypto/ipsec_mb/ipsec_mb_private.c
@@ -134,6 +134,8 @@ ipsec_mb_create(struct rte_vdev_device *vdev,
 		break;
 	}
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	IPSEC_MB_LOG(INFO, "IPSec Multi-buffer library version used: %s\n",
 		     imb_get_version_str());
 
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 6a2f8b6ac6..14b6783e13 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -1061,6 +1061,9 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
 	pthread_mutex_lock(&priv_list_lock);
 	TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
+
+	rte_cryptodev_pmd_probing_finish(crypto_dev);
+
 	return 0;
 }
 
diff --git a/drivers/crypto/mvsam/rte_mrvl_pmd.c b/drivers/crypto/mvsam/rte_mrvl_pmd.c
index a72642a772..04efd9aaa8 100644
--- a/drivers/crypto/mvsam/rte_mrvl_pmd.c
+++ b/drivers/crypto/mvsam/rte_mrvl_pmd.c
@@ -1114,6 +1114,8 @@ cryptodev_mrvl_crypto_create(const char *name,
 	if (ret)
 		goto init_error;
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	return 0;
 
 init_error:
diff --git a/drivers/crypto/nitrox/nitrox_sym.c b/drivers/crypto/nitrox/nitrox_sym.c
index f8b7edcd69..cb5393d2f1 100644
--- a/drivers/crypto/nitrox/nitrox_sym.c
+++ b/drivers/crypto/nitrox/nitrox_sym.c
@@ -783,6 +783,9 @@ nitrox_sym_pmd_create(struct nitrox_device *ndev)
 	ndev->sym_dev = cdev->data->dev_private;
 	ndev->sym_dev->cdev = cdev;
 	ndev->sym_dev->ndev = ndev;
+
+	rte_cryptodev_pmd_probing_finish(cdev);
+
 	NITROX_LOG(DEBUG, "Created cryptodev '%s', dev_id %d, drv_id %d\n",
 		   cdev->data->name, cdev->data->dev_id, nitrox_sym_drv_id);
 	return 0;
diff --git a/drivers/crypto/null/null_crypto_pmd.c b/drivers/crypto/null/null_crypto_pmd.c
index f9935d52cc..9ecb434fd0 100644
--- a/drivers/crypto/null/null_crypto_pmd.c
+++ b/drivers/crypto/null/null_crypto_pmd.c
@@ -184,6 +184,8 @@ cryptodev_null_create(const char *name,
 
 	internals->max_nb_qpairs = init_params->max_nb_queue_pairs;
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	return 0;
 }
 
diff --git a/drivers/crypto/octeontx/otx_cryptodev.c b/drivers/crypto/octeontx/otx_cryptodev.c
index c294f86d79..05b78329d6 100644
--- a/drivers/crypto/octeontx/otx_cryptodev.c
+++ b/drivers/crypto/octeontx/otx_cryptodev.c
@@ -56,8 +56,10 @@ otx_cpt_pci_probe(struct rte_pci_driver *pci_drv,
 
 	/* Invoke PMD device initialization function */
 	retval = otx_cpt_dev_create(cryptodev);
-	if (retval == 0)
+	if (retval == 0) {
+		rte_cryptodev_pmd_probing_finish(cryptodev);
 		return 0;
+	}
 
 	CPT_LOG_ERR("[DRV %s]: Failed to create device "
 			"(vendor_id: 0x%x device_id: 0x%x",
diff --git a/drivers/crypto/octeontx2/otx2_cryptodev.c b/drivers/crypto/octeontx2/otx2_cryptodev.c
index 85b1f00263..fc7ad05366 100644
--- a/drivers/crypto/octeontx2/otx2_cryptodev.c
+++ b/drivers/crypto/octeontx2/otx2_cryptodev.c
@@ -135,6 +135,8 @@ otx2_cpt_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
 		otx2_cpt_set_enqdeq_fns(dev);
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	return 0;
 
 otx2_dev_fini:
diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c
index 13c6ea8724..5794ed8159 100644
--- a/drivers/crypto/openssl/rte_openssl_pmd.c
+++ b/drivers/crypto/openssl/rte_openssl_pmd.c
@@ -2213,6 +2213,8 @@ cryptodev_openssl_create(const char *name,
 
 	internals->max_nb_qpairs = init_params->max_nb_queue_pairs;
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	return 0;
 
 init_error:
diff --git a/drivers/crypto/qat/qat_asym_pmd.c b/drivers/crypto/qat/qat_asym_pmd.c
index e91bb0d317..0944d27a4d 100644
--- a/drivers/crypto/qat/qat_asym_pmd.c
+++ b/drivers/crypto/qat/qat_asym_pmd.c
@@ -343,6 +343,9 @@ qat_asym_dev_create(struct qat_pci_device *qat_pci_dev,
 	}
 
 	qat_pci_dev->asym_dev = internals;
+
+	rte_cryptodev_pmd_probing_finish(cryptodev);
+
 	QAT_LOG(DEBUG, "Created QAT ASYM device %s as cryptodev instance %d",
 			cryptodev->data->name, internals->asym_dev_id);
 	return 0;
diff --git a/drivers/crypto/qat/qat_sym_pmd.c b/drivers/crypto/qat/qat_sym_pmd.c
index efda921c05..d4f087733f 100644
--- a/drivers/crypto/qat/qat_sym_pmd.c
+++ b/drivers/crypto/qat/qat_sym_pmd.c
@@ -500,6 +500,8 @@ qat_sym_dev_create(struct qat_pci_device *qat_pci_dev,
 	QAT_LOG(DEBUG, "Created QAT SYM device %s as cryptodev instance %d",
 			cryptodev->data->name, internals->sym_dev_id);
 
+	rte_cryptodev_pmd_probing_finish(cryptodev);
+
 	return 0;
 
 error:
diff --git a/drivers/crypto/scheduler/scheduler_pmd.c b/drivers/crypto/scheduler/scheduler_pmd.c
index 560c26af50..dd198080bf 100644
--- a/drivers/crypto/scheduler/scheduler_pmd.c
+++ b/drivers/crypto/scheduler/scheduler_pmd.c
@@ -229,6 +229,8 @@ cryptodev_scheduler_create(const char *name,
 		return -ENOMEM;
 	}
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	return 0;
 }
 
diff --git a/drivers/crypto/virtio/virtio_cryptodev.c b/drivers/crypto/virtio/virtio_cryptodev.c
index 8faa39df4a..ed64866758 100644
--- a/drivers/crypto/virtio/virtio_cryptodev.c
+++ b/drivers/crypto/virtio/virtio_cryptodev.c
@@ -754,6 +754,8 @@ crypto_virtio_create(const char *name, struct rte_pci_device *pci_dev,
 			VIRTIO_CRYPTO_PMD_GUEST_FEATURES) < 0)
 		return -1;
 
+	rte_cryptodev_pmd_probing_finish(cryptodev);
+
 	return 0;
 }
 
-- 
2.25.1

[dpdk-dev] [PATCH v3 6/7] cryptodev: update fast path APIs to use new flat array

[Akhil Goyal]

Rework fast-path cryptodev functions to use rte_crypto_fp_ops[].
While it is an API/ABI breakage, this change is intended to be
transparent for both users (no changes in user app is required) and
PMD developers (no changes in PMD is required).

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
---
 lib/cryptodev/rte_cryptodev.h | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)

diff --git a/lib/cryptodev/rte_cryptodev.h b/lib/cryptodev/rte_cryptodev.h
index ce0dca72be..56e3868ada 100644
--- a/lib/cryptodev/rte_cryptodev.h
+++ b/lib/cryptodev/rte_cryptodev.h
@@ -1832,13 +1832,18 @@ static inline uint16_t
 rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
 		struct rte_crypto_op **ops, uint16_t nb_ops)
 {
-	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
+	const struct rte_crypto_fp_ops *fp_ops;
+	void *qp;
 
 	rte_cryptodev_trace_dequeue_burst(dev_id, qp_id, (void **)ops, nb_ops);
-	nb_ops = (*dev->dequeue_burst)
-			(dev->data->queue_pairs[qp_id], ops, nb_ops);
+
+	fp_ops = &rte_crypto_fp_ops[dev_id];
+	qp = fp_ops->qp.data[qp_id];
+
+	nb_ops = fp_ops->dequeue_burst(qp, ops, nb_ops);
+
 #ifdef RTE_CRYPTO_CALLBACKS
-	if (unlikely(dev->deq_cbs != NULL)) {
+	if (unlikely(fp_ops->qp.deq_cb != NULL)) {
 		struct rte_cryptodev_cb_rcu *list;
 		struct rte_cryptodev_cb *cb;
 
@@ -1848,7 +1853,7 @@ rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
 		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
 		 * not required.
 		 */
-		list = &dev->deq_cbs[qp_id];
+		list = &fp_ops->qp.deq_cb[qp_id];
 		rte_rcu_qsbr_thread_online(list->qsbr, 0);
 		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
 
@@ -1899,10 +1904,13 @@ static inline uint16_t
 rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
 		struct rte_crypto_op **ops, uint16_t nb_ops)
 {
-	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
+	const struct rte_crypto_fp_ops *fp_ops;
+	void *qp;
 
+	fp_ops = &rte_crypto_fp_ops[dev_id];
+	qp = fp_ops->qp.data[qp_id];
 #ifdef RTE_CRYPTO_CALLBACKS
-	if (unlikely(dev->enq_cbs != NULL)) {
+	if (unlikely(fp_ops->qp.enq_cb != NULL)) {
 		struct rte_cryptodev_cb_rcu *list;
 		struct rte_cryptodev_cb *cb;
 
@@ -1912,7 +1920,7 @@ rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
 		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
 		 * not required.
 		 */
-		list = &dev->enq_cbs[qp_id];
+		list = &fp_ops->qp.enq_cb[qp_id];
 		rte_rcu_qsbr_thread_online(list->qsbr, 0);
 		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
 
@@ -1927,8 +1935,7 @@ rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
 #endif
 
 	rte_cryptodev_trace_enqueue_burst(dev_id, qp_id, (void **)ops, nb_ops);
-	return (*dev->enqueue_burst)(
-			dev->data->queue_pairs[qp_id], ops, nb_ops);
+	return fp_ops->enqueue_burst(qp, ops, nb_ops);
 }
 
 
-- 
2.25.1

[dpdk-dev] [PATCH v3 7/7] cryptodev: move device specific structures

[Akhil Goyal]

The device specific structures - rte_cryptodev
and rte_cryptodev_data are moved to cryptodev_pmd.h
to hide it from the applications.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
Tested-by: Rebecca Troy <rebecca.troy@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
---
 drivers/crypto/ccp/ccp_dev.h                  |  2 +-
 drivers/crypto/cnxk/cn10k_ipsec.c             |  2 +-
 drivers/crypto/cnxk/cn9k_ipsec.c              |  2 +-
 .../crypto/cnxk/cnxk_cryptodev_capabilities.c |  2 +-
 drivers/crypto/cnxk/cnxk_cryptodev_sec.c      |  2 +-
 drivers/crypto/nitrox/nitrox_sym_reqmgr.c     |  2 +-
 drivers/crypto/octeontx/otx_cryptodev.c       |  1 -
 .../crypto/octeontx/otx_cryptodev_hw_access.c |  2 +-
 .../crypto/octeontx/otx_cryptodev_hw_access.h |  2 +-
 drivers/crypto/octeontx/otx_cryptodev_ops.h   |  2 +-
 .../crypto/octeontx2/otx2_cryptodev_mbox.c    |  2 +-
 drivers/crypto/scheduler/scheduler_failover.c |  2 +-
 .../crypto/scheduler/scheduler_multicore.c    |  2 +-
 .../scheduler/scheduler_pkt_size_distr.c      |  2 +-
 .../crypto/scheduler/scheduler_roundrobin.c   |  2 +-
 drivers/event/cnxk/cnxk_eventdev.h            |  2 +-
 drivers/event/dpaa/dpaa_eventdev.c            |  2 +-
 drivers/event/dpaa2/dpaa2_eventdev.c          |  2 +-
 drivers/event/octeontx/ssovf_evdev.c          |  2 +-
 .../event/octeontx2/otx2_evdev_crypto_adptr.c |  2 +-
 lib/cryptodev/cryptodev_pmd.h                 | 65 ++++++++++++++++++
 lib/cryptodev/rte_cryptodev_core.h            | 67 -------------------
 lib/cryptodev/version.map                     |  2 +-
 23 files changed, 85 insertions(+), 88 deletions(-)

diff --git a/drivers/crypto/ccp/ccp_dev.h b/drivers/crypto/ccp/ccp_dev.h
index ca5145c278..85c8fc47a2 100644
--- a/drivers/crypto/ccp/ccp_dev.h
+++ b/drivers/crypto/ccp/ccp_dev.h
@@ -17,7 +17,7 @@
 #include <rte_pci.h>
 #include <rte_spinlock.h>
 #include <rte_crypto_sym.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 
 /**< CCP sspecific */
 #define MAX_HW_QUEUES                   5
diff --git a/drivers/crypto/cnxk/cn10k_ipsec.c b/drivers/crypto/cnxk/cn10k_ipsec.c
index defc792aa8..27df1dcd64 100644
--- a/drivers/crypto/cnxk/cn10k_ipsec.c
+++ b/drivers/crypto/cnxk/cn10k_ipsec.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_malloc.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_esp.h>
 #include <rte_ip.h>
 #include <rte_security.h>
diff --git a/drivers/crypto/cnxk/cn9k_ipsec.c b/drivers/crypto/cnxk/cn9k_ipsec.c
index 9ca4d20c62..53fb793654 100644
--- a/drivers/crypto/cnxk/cn9k_ipsec.c
+++ b/drivers/crypto/cnxk/cn9k_ipsec.c
@@ -2,7 +2,7 @@
  * Copyright(C) 2021 Marvell.
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_ip.h>
 #include <rte_security.h>
 #include <rte_security_driver.h>
diff --git a/drivers/crypto/cnxk/cnxk_cryptodev_capabilities.c b/drivers/crypto/cnxk/cnxk_cryptodev_capabilities.c
index a227e6981c..a53b489a04 100644
--- a/drivers/crypto/cnxk/cnxk_cryptodev_capabilities.c
+++ b/drivers/crypto/cnxk/cnxk_cryptodev_capabilities.c
@@ -2,7 +2,7 @@
  * Copyright(C) 2021 Marvell.
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_security.h>
 
 #include "roc_api.h"
diff --git a/drivers/crypto/cnxk/cnxk_cryptodev_sec.c b/drivers/crypto/cnxk/cnxk_cryptodev_sec.c
index 8d04d4b575..2021d5c77e 100644
--- a/drivers/crypto/cnxk/cnxk_cryptodev_sec.c
+++ b/drivers/crypto/cnxk/cnxk_cryptodev_sec.c
@@ -2,7 +2,7 @@
  * Copyright(C) 2021 Marvell.
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_malloc.h>
 #include <rte_security.h>
 #include <rte_security_driver.h>
diff --git a/drivers/crypto/nitrox/nitrox_sym_reqmgr.c b/drivers/crypto/nitrox/nitrox_sym_reqmgr.c
index fe3ca25a0c..9edb0cc00f 100644
--- a/drivers/crypto/nitrox/nitrox_sym_reqmgr.c
+++ b/drivers/crypto/nitrox/nitrox_sym_reqmgr.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_crypto.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_cycles.h>
 #include <rte_errno.h>
 
diff --git a/drivers/crypto/octeontx/otx_cryptodev.c b/drivers/crypto/octeontx/otx_cryptodev.c
index 05b78329d6..337d06aab8 100644
--- a/drivers/crypto/octeontx/otx_cryptodev.c
+++ b/drivers/crypto/octeontx/otx_cryptodev.c
@@ -4,7 +4,6 @@
 
 #include <rte_bus_pci.h>
 #include <rte_common.h>
-#include <rte_cryptodev.h>
 #include <cryptodev_pmd.h>
 #include <rte_log.h>
 #include <rte_pci.h>
diff --git a/drivers/crypto/octeontx/otx_cryptodev_hw_access.c b/drivers/crypto/octeontx/otx_cryptodev_hw_access.c
index 7b89a62d81..20b288334a 100644
--- a/drivers/crypto/octeontx/otx_cryptodev_hw_access.c
+++ b/drivers/crypto/octeontx/otx_cryptodev_hw_access.c
@@ -7,7 +7,7 @@
 
 #include <rte_branch_prediction.h>
 #include <rte_common.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_errno.h>
 #include <rte_mempool.h>
 #include <rte_memzone.h>
diff --git a/drivers/crypto/octeontx/otx_cryptodev_hw_access.h b/drivers/crypto/octeontx/otx_cryptodev_hw_access.h
index 7c6b1e45b4..e48805fb09 100644
--- a/drivers/crypto/octeontx/otx_cryptodev_hw_access.h
+++ b/drivers/crypto/octeontx/otx_cryptodev_hw_access.h
@@ -7,7 +7,7 @@
 #include <stdbool.h>
 
 #include <rte_branch_prediction.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_cycles.h>
 #include <rte_io.h>
 #include <rte_memory.h>
diff --git a/drivers/crypto/octeontx/otx_cryptodev_ops.h b/drivers/crypto/octeontx/otx_cryptodev_ops.h
index f234f16970..83b82ea059 100644
--- a/drivers/crypto/octeontx/otx_cryptodev_ops.h
+++ b/drivers/crypto/octeontx/otx_cryptodev_ops.h
@@ -5,7 +5,7 @@
 #ifndef _OTX_CRYPTODEV_OPS_H_
 #define _OTX_CRYPTODEV_OPS_H_
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 
 #define OTX_CPT_MIN_HEADROOM_REQ	(24)
 #define OTX_CPT_MIN_TAILROOM_REQ	(8)
diff --git a/drivers/crypto/octeontx2/otx2_cryptodev_mbox.c b/drivers/crypto/octeontx2/otx2_cryptodev_mbox.c
index 1a8edae7eb..f9e7b0b474 100644
--- a/drivers/crypto/octeontx2/otx2_cryptodev_mbox.c
+++ b/drivers/crypto/octeontx2/otx2_cryptodev_mbox.c
@@ -1,7 +1,7 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright (C) 2019 Marvell International Ltd.
  */
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_ethdev.h>
 
 #include "otx2_cryptodev.h"
diff --git a/drivers/crypto/scheduler/scheduler_failover.c b/drivers/crypto/scheduler/scheduler_failover.c
index 844312dd1b..5023577ef8 100644
--- a/drivers/crypto/scheduler/scheduler_failover.c
+++ b/drivers/crypto/scheduler/scheduler_failover.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2017 Intel Corporation
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_malloc.h>
 
 #include "rte_cryptodev_scheduler_operations.h"
diff --git a/drivers/crypto/scheduler/scheduler_multicore.c b/drivers/crypto/scheduler/scheduler_multicore.c
index 1e2e8dbf9f..900ab4049d 100644
--- a/drivers/crypto/scheduler/scheduler_multicore.c
+++ b/drivers/crypto/scheduler/scheduler_multicore.c
@@ -3,7 +3,7 @@
  */
 #include <unistd.h>
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_malloc.h>
 
 #include "rte_cryptodev_scheduler_operations.h"
diff --git a/drivers/crypto/scheduler/scheduler_pkt_size_distr.c b/drivers/crypto/scheduler/scheduler_pkt_size_distr.c
index 57e330a744..933a5c6978 100644
--- a/drivers/crypto/scheduler/scheduler_pkt_size_distr.c
+++ b/drivers/crypto/scheduler/scheduler_pkt_size_distr.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2017 Intel Corporation
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_malloc.h>
 
 #include "rte_cryptodev_scheduler_operations.h"
diff --git a/drivers/crypto/scheduler/scheduler_roundrobin.c b/drivers/crypto/scheduler/scheduler_roundrobin.c
index bc4a632106..ace2dec2ec 100644
--- a/drivers/crypto/scheduler/scheduler_roundrobin.c
+++ b/drivers/crypto/scheduler/scheduler_roundrobin.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2017 Intel Corporation
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_malloc.h>
 
 #include "rte_cryptodev_scheduler_operations.h"
diff --git a/drivers/event/cnxk/cnxk_eventdev.h b/drivers/event/cnxk/cnxk_eventdev.h
index 8a5c737e4b..b57004c0dc 100644
--- a/drivers/event/cnxk/cnxk_eventdev.h
+++ b/drivers/event/cnxk/cnxk_eventdev.h
@@ -7,7 +7,7 @@
 
 #include <string.h>
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_devargs.h>
 #include <rte_ethdev.h>
 #include <rte_event_eth_rx_adapter.h>
diff --git a/drivers/event/dpaa/dpaa_eventdev.c b/drivers/event/dpaa/dpaa_eventdev.c
index ec74160325..1d7ddfe1d1 100644
--- a/drivers/event/dpaa/dpaa_eventdev.c
+++ b/drivers/event/dpaa/dpaa_eventdev.c
@@ -28,7 +28,7 @@
 #include <rte_ethdev.h>
 #include <rte_event_eth_rx_adapter.h>
 #include <rte_event_eth_tx_adapter.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_dpaa_bus.h>
 #include <rte_dpaa_logs.h>
 #include <rte_cycles.h>
diff --git a/drivers/event/dpaa2/dpaa2_eventdev.c b/drivers/event/dpaa2/dpaa2_eventdev.c
index 5ccf22f77f..e03afb2958 100644
--- a/drivers/event/dpaa2/dpaa2_eventdev.c
+++ b/drivers/event/dpaa2/dpaa2_eventdev.c
@@ -25,7 +25,7 @@
 #include <rte_pci.h>
 #include <rte_bus_vdev.h>
 #include <ethdev_driver.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_event_eth_rx_adapter.h>
 #include <rte_event_eth_tx_adapter.h>
 
diff --git a/drivers/event/octeontx/ssovf_evdev.c b/drivers/event/octeontx/ssovf_evdev.c
index b93f6ec8c6..9846fce34b 100644
--- a/drivers/event/octeontx/ssovf_evdev.c
+++ b/drivers/event/octeontx/ssovf_evdev.c
@@ -5,7 +5,7 @@
 #include <inttypes.h>
 
 #include <rte_common.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_debug.h>
 #include <rte_dev.h>
 #include <rte_eal.h>
diff --git a/drivers/event/octeontx2/otx2_evdev_crypto_adptr.c b/drivers/event/octeontx2/otx2_evdev_crypto_adptr.c
index d9a002625c..d59d6c53f6 100644
--- a/drivers/event/octeontx2/otx2_evdev_crypto_adptr.c
+++ b/drivers/event/octeontx2/otx2_evdev_crypto_adptr.c
@@ -2,7 +2,7 @@
  * Copyright (C) 2020-2021 Marvell.
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_eventdev.h>
 
 #include "otx2_cryptodev.h"
diff --git a/lib/cryptodev/cryptodev_pmd.h b/lib/cryptodev/cryptodev_pmd.h
index 9bb1e47ae4..89bf2af399 100644
--- a/lib/cryptodev/cryptodev_pmd.h
+++ b/lib/cryptodev/cryptodev_pmd.h
@@ -52,6 +52,71 @@ struct rte_cryptodev_pmd_init_params {
 	unsigned int max_nb_queue_pairs;
 };
 
+/**
+ * @internal
+ * The data part, with no function pointers, associated with each device.
+ *
+ * This structure is safe to place in shared memory to be common among
+ * different processes in a multi-process configuration.
+ */
+struct rte_cryptodev_data {
+	/** Device ID for this instance */
+	uint8_t dev_id;
+	/** Socket ID where memory is allocated */
+	uint8_t socket_id;
+	/** Unique identifier name */
+	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
+
+	__extension__
+	/** Device state: STARTED(1)/STOPPED(0) */
+	uint8_t dev_started : 1;
+
+	/** Session memory pool */
+	struct rte_mempool *session_pool;
+	/** Array of pointers to queue pairs. */
+	void **queue_pairs;
+	/** Number of device queue pairs. */
+	uint16_t nb_queue_pairs;
+
+	/** PMD-specific private data */
+	void *dev_private;
+} __rte_cache_aligned;
+
+/** @internal The data structure associated with each crypto device. */
+struct rte_cryptodev {
+	/** Pointer to PMD dequeue function. */
+	dequeue_pkt_burst_t dequeue_burst;
+	/** Pointer to PMD enqueue function. */
+	enqueue_pkt_burst_t enqueue_burst;
+
+	/** Pointer to device data */
+	struct rte_cryptodev_data *data;
+	/** Functions exported by PMD */
+	struct rte_cryptodev_ops *dev_ops;
+	/** Feature flags exposes HW/SW features for the given device */
+	uint64_t feature_flags;
+	/** Backing device */
+	struct rte_device *device;
+
+	/** Crypto driver identifier*/
+	uint8_t driver_id;
+
+	/** User application callback for interrupts if present */
+	struct rte_cryptodev_cb_list link_intr_cbs;
+
+	/** Context for security ops */
+	void *security_ctx;
+
+	__extension__
+	/** Flag indicating the device is attached */
+	uint8_t attached : 1;
+
+	/** User application callback for pre enqueue processing */
+	struct rte_cryptodev_cb_rcu *enq_cbs;
+	/** User application callback for post dequeue processing */
+	struct rte_cryptodev_cb_rcu *deq_cbs;
+} __rte_cache_aligned;
+
 /** Global structure used for maintaining state of allocated crypto devices */
 struct rte_cryptodev_global {
 	struct rte_cryptodev *devs;	/**< Device information array */
diff --git a/lib/cryptodev/rte_cryptodev_core.h b/lib/cryptodev/rte_cryptodev_core.h
index e9e9a44b3c..c13e214439 100644
--- a/lib/cryptodev/rte_cryptodev_core.h
+++ b/lib/cryptodev/rte_cryptodev_core.h
@@ -54,73 +54,6 @@ struct rte_crypto_fp_ops {
 
 extern struct rte_crypto_fp_ops rte_crypto_fp_ops[RTE_CRYPTO_MAX_DEVS];
 
-/**
- * @internal
- * The data part, with no function pointers, associated with each device.
- *
- * This structure is safe to place in shared memory to be common among
- * different processes in a multi-process configuration.
- */
-struct rte_cryptodev_data {
-	uint8_t dev_id;
-	/**< Device ID for this instance */
-	uint8_t socket_id;
-	/**< Socket ID where memory is allocated */
-	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
-	/**< Unique identifier name */
-
-	__extension__
-	uint8_t dev_started : 1;
-	/**< Device state: STARTED(1)/STOPPED(0) */
-
-	struct rte_mempool *session_pool;
-	/**< Session memory pool */
-	void **queue_pairs;
-	/**< Array of pointers to queue pairs. */
-	uint16_t nb_queue_pairs;
-	/**< Number of device queue pairs. */
-
-	void *dev_private;
-	/**< PMD-specific private data */
-} __rte_cache_aligned;
-
-
-/** @internal The data structure associated with each crypto device. */
-struct rte_cryptodev {
-	dequeue_pkt_burst_t dequeue_burst;
-	/**< Pointer to PMD receive function. */
-	enqueue_pkt_burst_t enqueue_burst;
-	/**< Pointer to PMD transmit function. */
-
-	struct rte_cryptodev_data *data;
-	/**< Pointer to device data */
-	struct rte_cryptodev_ops *dev_ops;
-	/**< Functions exported by PMD */
-	uint64_t feature_flags;
-	/**< Feature flags exposes HW/SW features for the given device */
-	struct rte_device *device;
-	/**< Backing device */
-
-	uint8_t driver_id;
-	/**< Crypto driver identifier*/
-
-	struct rte_cryptodev_cb_list link_intr_cbs;
-	/**< User application callback for interrupts if present */
-
-	void *security_ctx;
-	/**< Context for security ops */
-
-	__extension__
-	uint8_t attached : 1;
-	/**< Flag indicating the device is attached */
-
-	struct rte_cryptodev_cb_rcu *enq_cbs;
-	/**< User application callback for pre enqueue processing */
-
-	struct rte_cryptodev_cb_rcu *deq_cbs;
-	/**< User application callback for post dequeue processing */
-} __rte_cache_aligned;
-
 /**
  * The pool of rte_cryptodev structures.
  */
diff --git a/lib/cryptodev/version.map b/lib/cryptodev/version.map
index 157dac521d..b55b4b8e7e 100644
--- a/lib/cryptodev/version.map
+++ b/lib/cryptodev/version.map
@@ -43,7 +43,6 @@ DPDK_22 {
 	rte_cryptodev_sym_session_create;
 	rte_cryptodev_sym_session_free;
 	rte_cryptodev_sym_session_init;
-	rte_cryptodevs;
 
 	#added in 21.11
 	rte_crypto_fp_ops;
@@ -125,4 +124,5 @@ INTERNAL {
 	rte_cryptodev_pmd_parse_input_args;
 	rte_cryptodev_pmd_probing_finish;
 	rte_cryptodev_pmd_release_device;
+	rte_cryptodevs;
 };
-- 
2.25.1

Re: [dpdk-dev] [PATCH v2 0/5] cryptodev: hide internal structures

[Ananyev, Konstantin]

> >
> > Just to let you know this patch set causes to seg-fault ipsec-secgw:
> > examples/ipsec-secgw/test/run_test.sh -46m
> > ...
> > [23695833.390785] dpdk-ipsec-secg[2491066]: segfault at 0 ip
> > 0000564325730963 sp 00007fffb9111d00 error 4 in dpdk-ipsec-
> > secgw[564324df0000+134d000]
> > [23695833.390791] Code: 28 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 00 4c 8d 04
> > cd 00 00 00 00 49 89 ce 4c 89 e7 4a 8d 34 00 48 8b 46 08 48 89 74 24 18 <48>
> > 8b 08 48 89 88 80 00 00 00 f0 83 44 24 80 00 4c 8b 2e 4d 85 ed
> >
> > So far, I didn't dig into it any further.
> > Will have a closer look at Monday.
> >
> Thanks for the update, planning to send a next version to fix the multi process issue
> Reported by Fan, will look into the ipsec-secgw also over weekend.

As FYI: don't see such problem with v3.
In fact, ipsec-secgw functional tests passed with v3 on my box.

Re: [dpdk-dev] [PATCH v3 3/7] cryptodev: move inline APIs into separate structure

[Ananyev, Konstantin]

> diff --git a/lib/cryptodev/rte_cryptodev_core.h b/lib/cryptodev/rte_cryptodev_core.h
> index 1633e55889..e9e9a44b3c 100644
> --- a/lib/cryptodev/rte_cryptodev_core.h
> +++ b/lib/cryptodev/rte_cryptodev_core.h
> @@ -25,6 +25,35 @@ typedef uint16_t (*enqueue_pkt_burst_t)(void *qp,
>  		struct rte_crypto_op **ops,	uint16_t nb_ops);
>  /**< Enqueue packets for processing on queue pair of a device. */
> 
> +/**
> + * @internal
> + * Structure used to hold opaque pointers to internal ethdev Rx/Tx
> + * queues data.
> + * The main purpose to expose these pointers at all - allow compiler
> + * to fetch this data for fast-path cryptodev inline functions in advance.
> + */
> +struct rte_cryptodev_qpdata {
> +	/** points to array of internal queue pair data pointers. */
> +	void **data;
> +	/** points to array of enqueue callback data pointers */
> +	struct rte_cryptodev_cb_rcu *enq_cb;
> +	/** points to array of dequeue callback data pointers */
> +	struct rte_cryptodev_cb_rcu *deq_cb;
> +};
> +
> +struct rte_crypto_fp_ops {
> +	/** PMD enqueue burst function. */
> +	enqueue_pkt_burst_t enqueue_burst;
> +	/** PMD dequeue burst function. */
> +	dequeue_pkt_burst_t dequeue_burst;
> +	/** Internal queue pair data pointers. */
> +	struct rte_cryptodev_qpdata qp;
> +	/** Reserved for future ops. */
> +	uintptr_t reserved[4];

I think it has to be uintptr_t reserved[3];
2 function pointers + 3 data pointers +3 reserved pointers.
Otherwise it will occupy extra 64B line.

> +} __rte_cache_aligned;
> +
> +extern struct rte_crypto_fp_ops rte_crypto_fp_ops[RTE_CRYPTO_MAX_DEVS];
> +
>  /**
>   * @internal
>   * The data part, with no function pointers, associated with each device.
> diff --git a/lib/cryptodev/version.map b/lib/cryptodev/version.map
> index 43cf937e40..ed62ced221 100644
> --- a/lib/cryptodev/version.map
> +++ b/lib/cryptodev/version.map
> @@ -45,6 +45,9 @@ DPDK_22 {
>  	rte_cryptodev_sym_session_init;
>  	rte_cryptodevs;
> 
> +	#added in 21.11
> +	rte_crypto_fp_ops;
> +
>  	local: *;
>  };
> 
> @@ -109,6 +112,8 @@ EXPERIMENTAL {
>  INTERNAL {
>  	global:
> 
> +	cryptodev_fp_ops_reset;
> +	cryptodev_fp_ops_set;
>  	rte_cryptodev_allocate_driver;
>  	rte_cryptodev_pmd_allocate;
>  	rte_cryptodev_pmd_callback_process;
> --
> 2.25.1

Re: [dpdk-dev] [PATCH v3 3/7] cryptodev: move inline APIs into separate structure

[Akhil Goyal]

> > diff --git a/lib/cryptodev/rte_cryptodev_core.h
> b/lib/cryptodev/rte_cryptodev_core.h
> > index 1633e55889..e9e9a44b3c 100644
> > --- a/lib/cryptodev/rte_cryptodev_core.h
> > +++ b/lib/cryptodev/rte_cryptodev_core.h
> > @@ -25,6 +25,35 @@ typedef uint16_t (*enqueue_pkt_burst_t)(void *qp,
> >  		struct rte_crypto_op **ops,	uint16_t nb_ops);
> >  /**< Enqueue packets for processing on queue pair of a device. */
> >
> > +/**
> > + * @internal
> > + * Structure used to hold opaque pointers to internal ethdev Rx/Tx
> > + * queues data.
> > + * The main purpose to expose these pointers at all - allow compiler
> > + * to fetch this data for fast-path cryptodev inline functions in advance.
> > + */
> > +struct rte_cryptodev_qpdata {
> > +	/** points to array of internal queue pair data pointers. */
> > +	void **data;
> > +	/** points to array of enqueue callback data pointers */
> > +	struct rte_cryptodev_cb_rcu *enq_cb;
> > +	/** points to array of dequeue callback data pointers */
> > +	struct rte_cryptodev_cb_rcu *deq_cb;
> > +};
> > +
> > +struct rte_crypto_fp_ops {
> > +	/** PMD enqueue burst function. */
> > +	enqueue_pkt_burst_t enqueue_burst;
> > +	/** PMD dequeue burst function. */
> > +	dequeue_pkt_burst_t dequeue_burst;
> > +	/** Internal queue pair data pointers. */
> > +	struct rte_cryptodev_qpdata qp;
> > +	/** Reserved for future ops. */
> > +	uintptr_t reserved[4];
> 
> I think it has to be uintptr_t reserved[3];
> 2 function pointers + 3 data pointers +3 reserved pointers.
> Otherwise it will occupy extra 64B line.

Yep, I missed decrementing it. Thanks. Will update in v4.
Please review others also and ack if no comments.
We need to close all the patches(including the control path)
by tomorrow EOD.

Re: [dpdk-dev] [PATCH v3 6/7] cryptodev: update fast path APIs to use new flat array

[Ananyev, Konstantin]

> 
> Rework fast-path cryptodev functions to use rte_crypto_fp_ops[].
> While it is an API/ABI breakage, this change is intended to be
> transparent for both users (no changes in user app is required) and
> PMD developers (no changes in PMD is required).
> 
> Signed-off-by: Akhil Goyal <gakhil@marvell.com>
> Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
> ---
>  lib/cryptodev/rte_cryptodev.h | 27 +++++++++++++++++----------
>  1 file changed, 17 insertions(+), 10 deletions(-)
> 
> diff --git a/lib/cryptodev/rte_cryptodev.h b/lib/cryptodev/rte_cryptodev.h
> index ce0dca72be..56e3868ada 100644
> --- a/lib/cryptodev/rte_cryptodev.h
> +++ b/lib/cryptodev/rte_cryptodev.h
> @@ -1832,13 +1832,18 @@ static inline uint16_t
>  rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
>  		struct rte_crypto_op **ops, uint16_t nb_ops)
>  {
> -	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
> +	const struct rte_crypto_fp_ops *fp_ops;
> +	void *qp;
> 
>  	rte_cryptodev_trace_dequeue_burst(dev_id, qp_id, (void **)ops, nb_ops);
> -	nb_ops = (*dev->dequeue_burst)
> -			(dev->data->queue_pairs[qp_id], ops, nb_ops);
> +
> +	fp_ops = &rte_crypto_fp_ops[dev_id];
> +	qp = fp_ops->qp.data[qp_id];
> +
> +	nb_ops = fp_ops->dequeue_burst(qp, ops, nb_ops);
> +
>  #ifdef RTE_CRYPTO_CALLBACKS
> -	if (unlikely(dev->deq_cbs != NULL)) {
> +	if (unlikely(fp_ops->qp.deq_cb != NULL)) {
>  		struct rte_cryptodev_cb_rcu *list;
>  		struct rte_cryptodev_cb *cb;

As I ca see you decided to keep call-back related data-structs as public API.
I wonder that's to avoid extra changes with CB related code?
Or performance reasons?
Or probably something else?

> 
> @@ -1848,7 +1853,7 @@ rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
>  		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
>  		 * not required.
>  		 */
> -		list = &dev->deq_cbs[qp_id];
> +		list = &fp_ops->qp.deq_cb[qp_id];
>  		rte_rcu_qsbr_thread_online(list->qsbr, 0);
>  		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
> 
> @@ -1899,10 +1904,13 @@ static inline uint16_t
>  rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
>  		struct rte_crypto_op **ops, uint16_t nb_ops)
>  {
> -	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
> +	const struct rte_crypto_fp_ops *fp_ops;
> +	void *qp;
> 
> +	fp_ops = &rte_crypto_fp_ops[dev_id];
> +	qp = fp_ops->qp.data[qp_id];
>  #ifdef RTE_CRYPTO_CALLBACKS
> -	if (unlikely(dev->enq_cbs != NULL)) {
> +	if (unlikely(fp_ops->qp.enq_cb != NULL)) {
>  		struct rte_cryptodev_cb_rcu *list;
>  		struct rte_cryptodev_cb *cb;
> 
> @@ -1912,7 +1920,7 @@ rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
>  		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
>  		 * not required.
>  		 */
> -		list = &dev->enq_cbs[qp_id];
> +		list = &fp_ops->qp.enq_cb[qp_id];
>  		rte_rcu_qsbr_thread_online(list->qsbr, 0);
>  		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
> 
> @@ -1927,8 +1935,7 @@ rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
>  #endif
> 
>  	rte_cryptodev_trace_enqueue_burst(dev_id, qp_id, (void **)ops, nb_ops);
> -	return (*dev->enqueue_burst)(
> -			dev->data->queue_pairs[qp_id], ops, nb_ops);
> +	return fp_ops->enqueue_burst(qp, ops, nb_ops);
>  }
> 
> 
> --
> 2.25.1

Re: [dpdk-dev] [PATCH v3 6/7] cryptodev: update fast path APIs to use new flat array

[Akhil Goyal]

> > @@ -1832,13 +1832,18 @@ static inline uint16_t
> >  rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
> >  		struct rte_crypto_op **ops, uint16_t nb_ops)
> >  {
> > -	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
> > +	const struct rte_crypto_fp_ops *fp_ops;
> > +	void *qp;
> >
> >  	rte_cryptodev_trace_dequeue_burst(dev_id, qp_id, (void **)ops,
> nb_ops);
> > -	nb_ops = (*dev->dequeue_burst)
> > -			(dev->data->queue_pairs[qp_id], ops, nb_ops);
> > +
> > +	fp_ops = &rte_crypto_fp_ops[dev_id];
> > +	qp = fp_ops->qp.data[qp_id];
> > +
> > +	nb_ops = fp_ops->dequeue_burst(qp, ops, nb_ops);
> > +
> >  #ifdef RTE_CRYPTO_CALLBACKS
> > -	if (unlikely(dev->deq_cbs != NULL)) {
> > +	if (unlikely(fp_ops->qp.deq_cb != NULL)) {
> >  		struct rte_cryptodev_cb_rcu *list;
> >  		struct rte_cryptodev_cb *cb;
> 
> As I ca see you decided to keep call-back related data-structs as public API.
> I wonder that's to avoid extra changes with CB related code?
> Or performance reasons?
> Or probably something else?
I just wanted to avoid extra changes and it did not look that important at this point
Compared to other patches.
I would have done the changes if I had some more time.

Re: [dpdk-dev] [PATCH v3 6/7] cryptodev: update fast path APIs to use new flat array

[Ananyev, Konstantin]

 
> > > @@ -1832,13 +1832,18 @@ static inline uint16_t
> > >  rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
> > >  		struct rte_crypto_op **ops, uint16_t nb_ops)
> > >  {
> > > -	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
> > > +	const struct rte_crypto_fp_ops *fp_ops;
> > > +	void *qp;
> > >
> > >  	rte_cryptodev_trace_dequeue_burst(dev_id, qp_id, (void **)ops,
> > nb_ops);
> > > -	nb_ops = (*dev->dequeue_burst)
> > > -			(dev->data->queue_pairs[qp_id], ops, nb_ops);
> > > +
> > > +	fp_ops = &rte_crypto_fp_ops[dev_id];
> > > +	qp = fp_ops->qp.data[qp_id];
> > > +
> > > +	nb_ops = fp_ops->dequeue_burst(qp, ops, nb_ops);
> > > +
> > >  #ifdef RTE_CRYPTO_CALLBACKS
> > > -	if (unlikely(dev->deq_cbs != NULL)) {
> > > +	if (unlikely(fp_ops->qp.deq_cb != NULL)) {
> > >  		struct rte_cryptodev_cb_rcu *list;
> > >  		struct rte_cryptodev_cb *cb;
> >
> > As I ca see you decided to keep call-back related data-structs as public API.
> > I wonder that's to avoid extra changes with CB related code?
> > Or performance reasons?
> > Or probably something else?
> I just wanted to avoid extra changes and it did not look that important at this point
> Compared to other patches.
> I would have done the changes if I had some more time.

Understood, thanks for explanation.

Re: [dpdk-dev] [PATCH v3 3/7] cryptodev: move inline APIs into separate structure

[Ananyev, Konstantin]

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Tuesday, October 19, 2021 12:50 PM
> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
> Cc: thomas@monjalon.net; david.marchand@redhat.com; hemant.agrawal@nxp.com; Anoob Joseph <anoobj@marvell.com>; De Lara
> Guarch, Pablo <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>; Doherty, Declan <declan.doherty@intel.com>;
> matan@nvidia.com; g.singh@nxp.com; Zhang, Roy Fan <roy.fan.zhang@intel.com>; jianjay.zhou@huawei.com; asomalap@amd.com;
> ruifeng.wang@arm.com; Nicolau, Radu <radu.nicolau@intel.com>; ajit.khaparde@broadcom.com; Nagadheeraj Rottela
> <rnagadheeraj@marvell.com>; Ankur Dwivedi <adwivedi@marvell.com>; Power, Ciara <ciara.power@intel.com>; Troy, Rebecca
> <rebecca.troy@intel.com>
> Subject: RE: [PATCH v3 3/7] cryptodev: move inline APIs into separate structure
> 
> > > diff --git a/lib/cryptodev/rte_cryptodev_core.h
> > b/lib/cryptodev/rte_cryptodev_core.h
> > > index 1633e55889..e9e9a44b3c 100644
> > > --- a/lib/cryptodev/rte_cryptodev_core.h
> > > +++ b/lib/cryptodev/rte_cryptodev_core.h
> > > @@ -25,6 +25,35 @@ typedef uint16_t (*enqueue_pkt_burst_t)(void *qp,
> > >  		struct rte_crypto_op **ops,	uint16_t nb_ops);
> > >  /**< Enqueue packets for processing on queue pair of a device. */
> > >
> > > +/**
> > > + * @internal
> > > + * Structure used to hold opaque pointers to internal ethdev Rx/Tx
> > > + * queues data.
> > > + * The main purpose to expose these pointers at all - allow compiler
> > > + * to fetch this data for fast-path cryptodev inline functions in advance.
> > > + */
> > > +struct rte_cryptodev_qpdata {
> > > +	/** points to array of internal queue pair data pointers. */
> > > +	void **data;
> > > +	/** points to array of enqueue callback data pointers */
> > > +	struct rte_cryptodev_cb_rcu *enq_cb;
> > > +	/** points to array of dequeue callback data pointers */
> > > +	struct rte_cryptodev_cb_rcu *deq_cb;
> > > +};
> > > +
> > > +struct rte_crypto_fp_ops {
> > > +	/** PMD enqueue burst function. */
> > > +	enqueue_pkt_burst_t enqueue_burst;
> > > +	/** PMD dequeue burst function. */
> > > +	dequeue_pkt_burst_t dequeue_burst;
> > > +	/** Internal queue pair data pointers. */
> > > +	struct rte_cryptodev_qpdata qp;
> > > +	/** Reserved for future ops. */
> > > +	uintptr_t reserved[4];
> >
> > I think it has to be uintptr_t reserved[3];
> > 2 function pointers + 3 data pointers +3 reserved pointers.
> > Otherwise it will occupy extra 64B line.
> 
> Yep, I missed decrementing it. Thanks. Will update in v4.

Ok.

> Please review others also and ack if no comments.

I looked through other patches in this set.
All looks good to me.
With the nit above fixed:
Series Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>

> We need to close all the patches(including the control path)
> by tomorrow EOD.

Re: [dpdk-dev] [PATCH v3 3/7] cryptodev: move inline APIs into separate structure

[Zhang, Roy Fan]

Apart from the scheduler PMD changes required mentioned by Ciara,
re-acking this patch as all doubts are cleared on our end.
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Monday, October 18, 2021 3:42 PM
> To: dev@dpdk.org
> Cc: thomas@monjalon.net; david.marchand@redhat.com;
> hemant.agrawal@nxp.com; anoobj@marvell.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> Doherty, Declan <declan.doherty@intel.com>; matan@nvidia.com;
> g.singh@nxp.com; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
> jianjay.zhou@huawei.com; asomalap@amd.com; ruifeng.wang@arm.com;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>; Nicolau, Radu
> <radu.nicolau@intel.com>; ajit.khaparde@broadcom.com;
> rnagadheeraj@marvell.com; adwivedi@marvell.com; Power, Ciara
> <ciara.power@intel.com>; Akhil Goyal <gakhil@marvell.com>; Troy, Rebecca
> <rebecca.troy@intel.com>
> Subject: [PATCH v3 3/7] cryptodev: move inline APIs into separate structure
> 
> Move fastpath inline function pointers from rte_cryptodev into a
> separate structure accessed via a flat array.
> The intension is to make rte_cryptodev and related structures private
> to avoid future API/ABI breakages.
> 
> Signed-off-by: Akhil Goyal <gakhil@marvell.com>
> Tested-by: Rebecca Troy <rebecca.troy@intel.com>
> Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
> ---
>  lib/cryptodev/cryptodev_pmd.c      | 53
> +++++++++++++++++++++++++++++-
>  lib/cryptodev/cryptodev_pmd.h      | 11 +++++++
>  lib/cryptodev/rte_cryptodev.c      | 19 +++++++++++
>  lib/cryptodev/rte_cryptodev_core.h | 29 ++++++++++++++++
>  lib/cryptodev/version.map          |  5 +++
>  5 files changed, 116 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/cryptodev/cryptodev_pmd.c
> b/lib/cryptodev/cryptodev_pmd.c
> index 44a70ecb35..fd74543682 100644
> --- a/lib/cryptodev/cryptodev_pmd.c
> +++ b/lib/cryptodev/cryptodev_pmd.c
> @@ -3,7 +3,7 @@
>   */
> 
>  #include <sys/queue.h>
> -
> +#include <rte_errno.h>
>  #include <rte_string_fns.h>
>  #include <rte_malloc.h>
> 
> @@ -160,3 +160,54 @@ rte_cryptodev_pmd_destroy(struct rte_cryptodev
> *cryptodev)
> 
>  	return 0;
>  }
> +
> +static uint16_t
> +dummy_crypto_enqueue_burst(__rte_unused void *qp,
> +			   __rte_unused struct rte_crypto_op **ops,
> +			   __rte_unused uint16_t nb_ops)
> +{
> +	CDEV_LOG_ERR(
> +		"crypto enqueue burst requested for unconfigured device");
> +	rte_errno = ENOTSUP;
> +	return 0;
> +}
> +
> +static uint16_t
> +dummy_crypto_dequeue_burst(__rte_unused void *qp,
> +			   __rte_unused struct rte_crypto_op **ops,
> +			   __rte_unused uint16_t nb_ops)
> +{
> +	CDEV_LOG_ERR(
> +		"crypto dequeue burst requested for unconfigured device");
> +	rte_errno = ENOTSUP;
> +	return 0;
> +}
> +
> +void
> +cryptodev_fp_ops_reset(struct rte_crypto_fp_ops *fp_ops)
> +{
> +	static struct rte_cryptodev_cb_rcu
> dummy_cb[RTE_MAX_QUEUES_PER_PORT];
> +	static void *dummy_data[RTE_MAX_QUEUES_PER_PORT];
> +	static const struct rte_crypto_fp_ops dummy = {
> +		.enqueue_burst = dummy_crypto_enqueue_burst,
> +		.dequeue_burst = dummy_crypto_dequeue_burst,
> +		.qp = {
> +			.data = dummy_data,
> +			.enq_cb = dummy_cb,
> +			.deq_cb = dummy_cb,
> +		},
> +	};
> +
> +	*fp_ops = dummy;
> +}
> +
> +void
> +cryptodev_fp_ops_set(struct rte_crypto_fp_ops *fp_ops,
> +		     const struct rte_cryptodev *dev)
> +{
> +	fp_ops->enqueue_burst = dev->enqueue_burst;
> +	fp_ops->dequeue_burst = dev->dequeue_burst;
> +	fp_ops->qp.data = dev->data->queue_pairs;
> +	fp_ops->qp.enq_cb = dev->enq_cbs;
> +	fp_ops->qp.deq_cb = dev->deq_cbs;
> +}
> diff --git a/lib/cryptodev/cryptodev_pmd.h
> b/lib/cryptodev/cryptodev_pmd.h
> index 36606dd10b..a71edbb991 100644
> --- a/lib/cryptodev/cryptodev_pmd.h
> +++ b/lib/cryptodev/cryptodev_pmd.h
> @@ -516,6 +516,17 @@ RTE_INIT(init_ ##driver_id)\
>  	driver_id = rte_cryptodev_allocate_driver(&crypto_drv, &(drv));\
>  }
> 
> +/* Reset crypto device fastpath APIs to dummy values. */
> +__rte_internal
> +void
> +cryptodev_fp_ops_reset(struct rte_crypto_fp_ops *fp_ops);
> +
> +/* Setup crypto device fastpath APIs. */
> +__rte_internal
> +void
> +cryptodev_fp_ops_set(struct rte_crypto_fp_ops *fp_ops,
> +		     const struct rte_cryptodev *dev);
> +
>  static inline void *
>  get_sym_session_private_data(const struct rte_cryptodev_sym_session
> *sess,
>  		uint8_t driver_id) {
> diff --git a/lib/cryptodev/rte_cryptodev.c b/lib/cryptodev/rte_cryptodev.c
> index eb86e629aa..305e013ebb 100644
> --- a/lib/cryptodev/rte_cryptodev.c
> +++ b/lib/cryptodev/rte_cryptodev.c
> @@ -53,6 +53,9 @@ static struct rte_cryptodev_global cryptodev_globals = {
>  		.nb_devs		= 0
>  };
> 
> +/* Public fastpath APIs. */
> +struct rte_crypto_fp_ops rte_crypto_fp_ops[RTE_CRYPTO_MAX_DEVS];
> +
>  /* spinlock for crypto device callbacks */
>  static rte_spinlock_t rte_cryptodev_cb_lock = RTE_SPINLOCK_INITIALIZER;
> 
> @@ -917,6 +920,8 @@ rte_cryptodev_pmd_release_device(struct
> rte_cryptodev *cryptodev)
> 
>  	dev_id = cryptodev->data->dev_id;
> 
> +	cryptodev_fp_ops_reset(rte_crypto_fp_ops + dev_id);
> +
>  	/* Close device only if device operations have been set */
>  	if (cryptodev->dev_ops) {
>  		ret = rte_cryptodev_close(dev_id);
> @@ -1080,6 +1085,9 @@ rte_cryptodev_start(uint8_t dev_id)
>  	}
> 
>  	diag = (*dev->dev_ops->dev_start)(dev);
> +	/* expose selection of PMD fast-path functions */
> +	cryptodev_fp_ops_set(rte_crypto_fp_ops + dev_id, dev);
> +
>  	rte_cryptodev_trace_start(dev_id, diag);
>  	if (diag == 0)
>  		dev->data->dev_started = 1;
> @@ -1109,6 +1117,9 @@ rte_cryptodev_stop(uint8_t dev_id)
>  		return;
>  	}
> 
> +	/* point fast-path functions to dummy ones */
> +	cryptodev_fp_ops_reset(rte_crypto_fp_ops + dev_id);
> +
>  	(*dev->dev_ops->dev_stop)(dev);
>  	rte_cryptodev_trace_stop(dev_id);
>  	dev->data->dev_started = 0;
> @@ -2411,3 +2422,11 @@ rte_cryptodev_allocate_driver(struct
> cryptodev_driver *crypto_drv,
> 
>  	return nb_drivers++;
>  }
> +
> +RTE_INIT(cryptodev_init_fp_ops)
> +{
> +	uint32_t i;
> +
> +	for (i = 0; i != RTE_DIM(rte_crypto_fp_ops); i++)
> +		cryptodev_fp_ops_reset(rte_crypto_fp_ops + i);
> +}
> diff --git a/lib/cryptodev/rte_cryptodev_core.h
> b/lib/cryptodev/rte_cryptodev_core.h
> index 1633e55889..e9e9a44b3c 100644
> --- a/lib/cryptodev/rte_cryptodev_core.h
> +++ b/lib/cryptodev/rte_cryptodev_core.h
> @@ -25,6 +25,35 @@ typedef uint16_t (*enqueue_pkt_burst_t)(void *qp,
>  		struct rte_crypto_op **ops,	uint16_t nb_ops);
>  /**< Enqueue packets for processing on queue pair of a device. */
> 
> +/**
> + * @internal
> + * Structure used to hold opaque pointers to internal ethdev Rx/Tx
> + * queues data.
> + * The main purpose to expose these pointers at all - allow compiler
> + * to fetch this data for fast-path cryptodev inline functions in advance.
> + */
> +struct rte_cryptodev_qpdata {
> +	/** points to array of internal queue pair data pointers. */
> +	void **data;
> +	/** points to array of enqueue callback data pointers */
> +	struct rte_cryptodev_cb_rcu *enq_cb;
> +	/** points to array of dequeue callback data pointers */
> +	struct rte_cryptodev_cb_rcu *deq_cb;
> +};
> +
> +struct rte_crypto_fp_ops {
> +	/** PMD enqueue burst function. */
> +	enqueue_pkt_burst_t enqueue_burst;
> +	/** PMD dequeue burst function. */
> +	dequeue_pkt_burst_t dequeue_burst;
> +	/** Internal queue pair data pointers. */
> +	struct rte_cryptodev_qpdata qp;
> +	/** Reserved for future ops. */
> +	uintptr_t reserved[4];
> +} __rte_cache_aligned;
> +
> +extern struct rte_crypto_fp_ops
> rte_crypto_fp_ops[RTE_CRYPTO_MAX_DEVS];
> +
>  /**
>   * @internal
>   * The data part, with no function pointers, associated with each device.
> diff --git a/lib/cryptodev/version.map b/lib/cryptodev/version.map
> index 43cf937e40..ed62ced221 100644
> --- a/lib/cryptodev/version.map
> +++ b/lib/cryptodev/version.map
> @@ -45,6 +45,9 @@ DPDK_22 {
>  	rte_cryptodev_sym_session_init;
>  	rte_cryptodevs;
> 
> +	#added in 21.11
> +	rte_crypto_fp_ops;
> +
>  	local: *;
>  };
> 
> @@ -109,6 +112,8 @@ EXPERIMENTAL {
>  INTERNAL {
>  	global:
> 
> +	cryptodev_fp_ops_reset;
> +	cryptodev_fp_ops_set;
>  	rte_cryptodev_allocate_driver;
>  	rte_cryptodev_pmd_allocate;
>  	rte_cryptodev_pmd_callback_process;
> --
> 2.25.1

Re: [dpdk-dev] [PATCH v3 4/7] cryptodev: add PMD device probe finish API

[Zhang, Roy Fan]

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Monday, October 18, 2021 3:42 PM
> To: dev@dpdk.org
> Cc: thomas@monjalon.net; david.marchand@redhat.com;
> hemant.agrawal@nxp.com; anoobj@marvell.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> Doherty, Declan <declan.doherty@intel.com>; matan@nvidia.com;
> g.singh@nxp.com; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
> jianjay.zhou@huawei.com; asomalap@amd.com; ruifeng.wang@arm.com;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>; Nicolau, Radu
> <radu.nicolau@intel.com>; ajit.khaparde@broadcom.com;
> rnagadheeraj@marvell.com; adwivedi@marvell.com; Power, Ciara
> <ciara.power@intel.com>; Akhil Goyal <gakhil@marvell.com>
> Subject: [PATCH v3 4/7] cryptodev: add PMD device probe finish API
> 
> Added a rte_cryptodev_pmd_probing_finish API which
> need to be called by the PMD after the device is initialized
> completely. This will set the fast path function pointers
> in the flat array for secondary process. For primary process,
> these are set in rte_cryptodev_start.
> 
> Signed-off-by: Akhil Goyal <gakhil@marvell.com>

Acked-by: Fan Zhang <roy.fan.zhang@intel.com>

Re: [dpdk-dev] [PATCH v3 5/7] drivers/crypto: invoke probing finish function

[Zhang, Roy Fan]

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Monday, October 18, 2021 3:42 PM
> To: dev@dpdk.org
> Cc: thomas@monjalon.net; david.marchand@redhat.com;
> hemant.agrawal@nxp.com; anoobj@marvell.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> Doherty, Declan <declan.doherty@intel.com>; matan@nvidia.com;
> g.singh@nxp.com; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
> jianjay.zhou@huawei.com; asomalap@amd.com; ruifeng.wang@arm.com;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>; Nicolau, Radu
> <radu.nicolau@intel.com>; ajit.khaparde@broadcom.com;
> rnagadheeraj@marvell.com; adwivedi@marvell.com; Power, Ciara
> <ciara.power@intel.com>; Akhil Goyal <gakhil@marvell.com>
> Subject: [PATCH v3 5/7] drivers/crypto: invoke probing finish function
> 
> Invoke event_dev_probing_finish() function at the end of probing,
> this function sets the function pointers in the fp_ops flat array
> in case of secondary process.
> For primary process, fp_ops is updated in rte_cryptodev_start().
> 
> Signed-off-by: Akhil Goyal <gakhil@marvell.com>
> ---
>  drivers/crypto/armv8/rte_armv8_pmd.c        | 2 ++
>  drivers/crypto/bcmfs/bcmfs_sym_pmd.c        | 2 ++
>  drivers/crypto/caam_jr/caam_jr.c            | 2 ++
>  drivers/crypto/ccp/rte_ccp_pmd.c            | 2 ++
>  drivers/crypto/cnxk/cn10k_cryptodev.c       | 2 ++
>  drivers/crypto/cnxk/cn9k_cryptodev.c        | 2 ++
>  drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c | 4 +++-
>  drivers/crypto/dpaa_sec/dpaa_sec.c          | 4 +++-
>  drivers/crypto/ipsec_mb/ipsec_mb_private.c  | 2 ++
>  drivers/crypto/mlx5/mlx5_crypto.c           | 3 +++
>  drivers/crypto/mvsam/rte_mrvl_pmd.c         | 2 ++
>  drivers/crypto/nitrox/nitrox_sym.c          | 3 +++
>  drivers/crypto/null/null_crypto_pmd.c       | 2 ++
>  drivers/crypto/octeontx/otx_cryptodev.c     | 4 +++-
>  drivers/crypto/octeontx2/otx2_cryptodev.c   | 2 ++
>  drivers/crypto/openssl/rte_openssl_pmd.c    | 2 ++
>  drivers/crypto/qat/qat_asym_pmd.c           | 3 +++
>  drivers/crypto/qat/qat_sym_pmd.c            | 2 ++
>  drivers/crypto/scheduler/scheduler_pmd.c    | 2 ++
>  drivers/crypto/virtio/virtio_cryptodev.c    | 2 ++
>  20 files changed, 46 insertions(+), 3 deletions(-)
Scheduler PMD needs to be updated to fix a problem.
Other than that - if addressed in V4,
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>

Re: [dpdk-dev] [PATCH v3 5/7] drivers/crypto: invoke probing finish function

[Matan Azrad]

HI

From: Akhil Goyal
> Invoke event_dev_probing_finish() function at the end of probing, this
> function sets the function pointers in the fp_ops flat array in case of
> secondary process.
> For primary process, fp_ops is updated in rte_cryptodev_start().
> 
> Signed-off-by: Akhil Goyal <gakhil@marvell.com>

Acked-by: Matan Azrad <matan@nvidia.com>

Re: [dpdk-dev] [PATCH v3 0/7] cryptodev: hide internal structures

[Power, Ciara]

Hi Akhil,


>-----Original Message-----
>From: Akhil Goyal <gakhil@marvell.com>
>Sent: Monday 18 October 2021 15:42
>To: dev@dpdk.org
>Cc: thomas@monjalon.net; david.marchand@redhat.com;
>hemant.agrawal@nxp.com; anoobj@marvell.com; De Lara Guarch, Pablo
><pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
>Doherty, Declan <declan.doherty@intel.com>; matan@nvidia.com;
>g.singh@nxp.com; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
>jianjay.zhou@huawei.com; asomalap@amd.com; ruifeng.wang@arm.com;
>Ananyev, Konstantin <konstantin.ananyev@intel.com>; Nicolau, Radu
><radu.nicolau@intel.com>; ajit.khaparde@broadcom.com;
>rnagadheeraj@marvell.com; adwivedi@marvell.com; Power, Ciara
><ciara.power@intel.com>; Akhil Goyal <gakhil@marvell.com>
>Subject: [PATCH v3 0/7] cryptodev: hide internal structures
>
>Structures rte_cryptodev and rte_cryptodev_data are not supposed to be
>directly used by the application. These are made public as they are used by
>inline datapath public APIs.
>This patchset, creates a new rte_cryptodev_core.h file which helps in defining
>a data structure to hold datapath APIs in a flat array based on the device
>identifier which is filled by the PMD.
>
>Similar series for ethdev and eventdev are also floated on ML.
>https://patchwork.dpdk.org/project/dpdk/list/?series=19428
>https://patchwork.dpdk.org/project/dpdk/list/?series=19405
>
>
<snip>

With this patchset I see a seg fault with the cryptodev_scheduler_autotest.
It is due to the worker PMDs using the direct device start functions rather than rte_cryptodev_start(), so fp_ops never get set.

I believe this fix is needed:

diff --git a/drivers/crypto/scheduler/scheduler_pmd_ops.c b/drivers/crypto/scheduler/scheduler_pmd_ops.c
index 465b88ade8..d6f8d3ab78 100644
--- a/drivers/crypto/scheduler/scheduler_pmd_ops.c
+++ b/drivers/crypto/scheduler/scheduler_pmd_ops.c
@@ -181,10 +181,8 @@ scheduler_pmd_start(struct rte_cryptodev *dev)
        /* start all workers */
        for (i = 0; i < sched_ctx->nb_workers; i++) {
                uint8_t worker_dev_id = sched_ctx->workers[i].dev_id;
-               struct rte_cryptodev *worker_dev =
-                               rte_cryptodev_pmd_get_dev(worker_dev_id);
 
-               ret = (*worker_dev->dev_ops->dev_start)(worker_dev);
+               ret = rte_cryptodev_start(worker_dev_id);
                if (ret < 0) {
                        CR_SCHED_LOG(ERR, "Failed to start worker dev %u",
                                        worker_dev_id);
@@ -208,10 +206,8 @@ scheduler_pmd_stop(struct rte_cryptodev *dev)
        /* stop all workers first */
        for (i = 0; i < sched_ctx->nb_workers; i++) {
                uint8_t worker_dev_id = sched_ctx->workers[i].dev_id;
-               struct rte_cryptodev *worker_dev =
-                               rte_cryptodev_pmd_get_dev(worker_dev_id);
 
-               (*worker_dev->dev_ops->dev_stop)(worker_dev);
+               rte_cryptodev_stop(worker_dev_id);
        }
 
        if (*sched_ctx->ops.scheduler_stop)


Thanks,
Ciara

[dpdk-dev] [PATCH v4 0/8] cryptodev: hide internal structures

[Akhil Goyal]

Structures rte_cryptodev and rte_cryptodev_data are not
supposed to be directly used by the application. These
are made public as they are used by inline datapath
public APIs.
This patchset, creates a new rte_cryptodev_core.h file
which helps in defining a data structure to hold datapath
APIs in a flat array based on the device identifier which
is filled by the PMD.

Similar series for ethdev and eventdev are also floated on ML.
https://patchwork.dpdk.org/project/dpdk/list/?series=19428
https://patchwork.dpdk.org/project/dpdk/list/?series=19405


changes in
v4:
- fixed number of reserved pointers.(Konstantin)
- added release notes
- added a patch for scheduler issue(Ciara).
v3: fixed multiprocess and enq-deq callback handling.
v2: align with the latest versions of above series.


Akhil Goyal (8):
  cryptodev: separate out internal structures
  cryptodev: allocate max space for internal qp array
  cryptodev: move inline APIs into separate structure
  crypto/scheduler: use proper API for device start/stop
  cryptodev: add PMD device probe finish API
  drivers/crypto: invoke probing finish function
  cryptodev: update fast path APIs to use new flat array
  cryptodev: move device specific structures

 doc/guides/rel_notes/release_21_11.rst        |   6 +
 drivers/crypto/armv8/rte_armv8_pmd.c          |   2 +
 drivers/crypto/bcmfs/bcmfs_sym_pmd.c          |   2 +
 drivers/crypto/caam_jr/caam_jr.c              |   2 +
 drivers/crypto/ccp/ccp_dev.h                  |   2 +-
 drivers/crypto/ccp/rte_ccp_pmd.c              |   2 +
 drivers/crypto/cnxk/cn10k_cryptodev.c         |   2 +
 drivers/crypto/cnxk/cn10k_ipsec.c             |   2 +-
 drivers/crypto/cnxk/cn9k_cryptodev.c          |   2 +
 drivers/crypto/cnxk/cn9k_ipsec.c              |   2 +-
 .../crypto/cnxk/cnxk_cryptodev_capabilities.c |   2 +-
 drivers/crypto/cnxk/cnxk_cryptodev_sec.c      |   2 +-
 drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c   |   4 +-
 drivers/crypto/dpaa_sec/dpaa_sec.c            |   4 +-
 drivers/crypto/ipsec_mb/ipsec_mb_private.c    |   2 +
 drivers/crypto/mlx5/mlx5_crypto.c             |   3 +
 drivers/crypto/mvsam/rte_mrvl_pmd.c           |   2 +
 drivers/crypto/nitrox/nitrox_sym.c            |   3 +
 drivers/crypto/nitrox/nitrox_sym_reqmgr.c     |   2 +-
 drivers/crypto/null/null_crypto_pmd.c         |   2 +
 drivers/crypto/octeontx/otx_cryptodev.c       |   5 +-
 .../crypto/octeontx/otx_cryptodev_hw_access.c |   2 +-
 .../crypto/octeontx/otx_cryptodev_hw_access.h |   2 +-
 drivers/crypto/octeontx/otx_cryptodev_ops.h   |   2 +-
 drivers/crypto/octeontx2/otx2_cryptodev.c     |   2 +
 .../crypto/octeontx2/otx2_cryptodev_mbox.c    |   2 +-
 drivers/crypto/openssl/rte_openssl_pmd.c      |   2 +
 drivers/crypto/qat/qat_asym_pmd.c             |   3 +
 drivers/crypto/qat/qat_sym_pmd.c              |   2 +
 drivers/crypto/scheduler/scheduler_failover.c |   2 +-
 .../crypto/scheduler/scheduler_multicore.c    |   2 +-
 .../scheduler/scheduler_pkt_size_distr.c      |   2 +-
 drivers/crypto/scheduler/scheduler_pmd.c      |   2 +
 drivers/crypto/scheduler/scheduler_pmd_ops.c  |   9 +-
 .../crypto/scheduler/scheduler_roundrobin.c   |   2 +-
 drivers/crypto/virtio/virtio_cryptodev.c      |   2 +
 drivers/event/cnxk/cnxk_eventdev.h            |   2 +-
 drivers/event/dpaa/dpaa_eventdev.c            |   2 +-
 drivers/event/dpaa2/dpaa2_eventdev.c          |   2 +-
 drivers/event/octeontx/ssovf_evdev.c          |   2 +-
 .../event/octeontx2/otx2_evdev_crypto_adptr.c |   2 +-
 lib/cryptodev/cryptodev_pmd.c                 |  69 +++-
 lib/cryptodev/cryptodev_pmd.h                 |  95 ++++-
 lib/cryptodev/meson.build                     |   4 +-
 lib/cryptodev/rte_cryptodev.c                 |  40 +-
 lib/cryptodev/rte_cryptodev.h                 | 367 +++++++-----------
 lib/cryptodev/rte_cryptodev_core.h            |  62 +++
 lib/cryptodev/version.map                     |   8 +-
 48 files changed, 473 insertions(+), 275 deletions(-)
 create mode 100644 lib/cryptodev/rte_cryptodev_core.h

-- 
2.25.1

[dpdk-dev] [PATCH v4 1/8] cryptodev: separate out internal structures

[Akhil Goyal]

A new header file rte_cryptodev_core.h is added and all
internal data structures which need not be exposed directly to
application are moved to this file. These structures are mostly
used by drivers, but they need to be in the public header file
as they are accessed by datapath inline functions for
performance reasons.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
Tested-by: Rebecca Troy <rebecca.troy@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 lib/cryptodev/cryptodev_pmd.h      |   6 -
 lib/cryptodev/meson.build          |   4 +-
 lib/cryptodev/rte_cryptodev.h      | 360 ++++++++++++-----------------
 lib/cryptodev/rte_cryptodev_core.h | 100 ++++++++
 4 files changed, 245 insertions(+), 225 deletions(-)
 create mode 100644 lib/cryptodev/rte_cryptodev_core.h

diff --git a/lib/cryptodev/cryptodev_pmd.h b/lib/cryptodev/cryptodev_pmd.h
index 8cc9051e09..36606dd10b 100644
--- a/lib/cryptodev/cryptodev_pmd.h
+++ b/lib/cryptodev/cryptodev_pmd.h
@@ -92,12 +92,6 @@ __rte_internal
 struct rte_cryptodev *
 rte_cryptodev_pmd_get_named_dev(const char *name);
 
-/**
- * The pool of rte_cryptodev structures.
- */
-extern struct rte_cryptodev *rte_cryptodevs;
-
-
 /**
  * Definitions of all functions exported by a driver through the
  * the generic structure of type *crypto_dev_ops* supplied in the
diff --git a/lib/cryptodev/meson.build b/lib/cryptodev/meson.build
index 51371c3aa2..289b66ab76 100644
--- a/lib/cryptodev/meson.build
+++ b/lib/cryptodev/meson.build
@@ -14,7 +14,9 @@ headers = files(
         'rte_crypto_sym.h',
         'rte_crypto_asym.h',
 )
-
+indirect_headers += files(
+        'rte_cryptodev_core.h',
+)
 driver_sdk_headers += files(
         'cryptodev_pmd.h',
 )
diff --git a/lib/cryptodev/rte_cryptodev.h b/lib/cryptodev/rte_cryptodev.h
index cdd7168fba..ce0dca72be 100644
--- a/lib/cryptodev/rte_cryptodev.h
+++ b/lib/cryptodev/rte_cryptodev.h
@@ -867,17 +867,6 @@ rte_cryptodev_callback_unregister(uint8_t dev_id,
 		enum rte_cryptodev_event_type event,
 		rte_cryptodev_cb_fn cb_fn, void *cb_arg);
 
-typedef uint16_t (*dequeue_pkt_burst_t)(void *qp,
-		struct rte_crypto_op **ops,	uint16_t nb_ops);
-/**< Dequeue processed packets from queue pair of a device. */
-
-typedef uint16_t (*enqueue_pkt_burst_t)(void *qp,
-		struct rte_crypto_op **ops,	uint16_t nb_ops);
-/**< Enqueue packets for processing on queue pair of a device. */
-
-
-
-
 struct rte_cryptodev_callback;
 
 /** Structure to keep track of registered callbacks */
@@ -907,216 +896,9 @@ struct rte_cryptodev_cb_rcu {
 	/**< RCU QSBR variable per queue pair */
 };
 
-/** The data structure associated with each crypto device. */
-struct rte_cryptodev {
-	dequeue_pkt_burst_t dequeue_burst;
-	/**< Pointer to PMD receive function. */
-	enqueue_pkt_burst_t enqueue_burst;
-	/**< Pointer to PMD transmit function. */
-
-	struct rte_cryptodev_data *data;
-	/**< Pointer to device data */
-	struct rte_cryptodev_ops *dev_ops;
-	/**< Functions exported by PMD */
-	uint64_t feature_flags;
-	/**< Feature flags exposes HW/SW features for the given device */
-	struct rte_device *device;
-	/**< Backing device */
-
-	uint8_t driver_id;
-	/**< Crypto driver identifier*/
-
-	struct rte_cryptodev_cb_list link_intr_cbs;
-	/**< User application callback for interrupts if present */
-
-	void *security_ctx;
-	/**< Context for security ops */
-
-	__extension__
-	uint8_t attached : 1;
-	/**< Flag indicating the device is attached */
-
-	struct rte_cryptodev_cb_rcu *enq_cbs;
-	/**< User application callback for pre enqueue processing */
-
-	struct rte_cryptodev_cb_rcu *deq_cbs;
-	/**< User application callback for post dequeue processing */
-} __rte_cache_aligned;
-
 void *
 rte_cryptodev_get_sec_ctx(uint8_t dev_id);
 
-/**
- *
- * The data part, with no function pointers, associated with each device.
- *
- * This structure is safe to place in shared memory to be common among
- * different processes in a multi-process configuration.
- */
-struct rte_cryptodev_data {
-	uint8_t dev_id;
-	/**< Device ID for this instance */
-	uint8_t socket_id;
-	/**< Socket ID where memory is allocated */
-	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
-	/**< Unique identifier name */
-
-	__extension__
-	uint8_t dev_started : 1;
-	/**< Device state: STARTED(1)/STOPPED(0) */
-
-	struct rte_mempool *session_pool;
-	/**< Session memory pool */
-	void **queue_pairs;
-	/**< Array of pointers to queue pairs. */
-	uint16_t nb_queue_pairs;
-	/**< Number of device queue pairs. */
-
-	void *dev_private;
-	/**< PMD-specific private data */
-} __rte_cache_aligned;
-
-extern struct rte_cryptodev *rte_cryptodevs;
-/**
- *
- * Dequeue a burst of processed crypto operations from a queue on the crypto
- * device. The dequeued operation are stored in *rte_crypto_op* structures
- * whose pointers are supplied in the *ops* array.
- *
- * The rte_cryptodev_dequeue_burst() function returns the number of ops
- * actually dequeued, which is the number of *rte_crypto_op* data structures
- * effectively supplied into the *ops* array.
- *
- * A return value equal to *nb_ops* indicates that the queue contained
- * at least *nb_ops* operations, and this is likely to signify that other
- * processed operations remain in the devices output queue. Applications
- * implementing a "retrieve as many processed operations as possible" policy
- * can check this specific case and keep invoking the
- * rte_cryptodev_dequeue_burst() function until a value less than
- * *nb_ops* is returned.
- *
- * The rte_cryptodev_dequeue_burst() function does not provide any error
- * notification to avoid the corresponding overhead.
- *
- * @param	dev_id		The symmetric crypto device identifier
- * @param	qp_id		The index of the queue pair from which to
- *				retrieve processed packets. The value must be
- *				in the range [0, nb_queue_pair - 1] previously
- *				supplied to rte_cryptodev_configure().
- * @param	ops		The address of an array of pointers to
- *				*rte_crypto_op* structures that must be
- *				large enough to store *nb_ops* pointers in it.
- * @param	nb_ops		The maximum number of operations to dequeue.
- *
- * @return
- *   - The number of operations actually dequeued, which is the number
- *   of pointers to *rte_crypto_op* structures effectively supplied to the
- *   *ops* array.
- */
-static inline uint16_t
-rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
-		struct rte_crypto_op **ops, uint16_t nb_ops)
-{
-	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
-
-	rte_cryptodev_trace_dequeue_burst(dev_id, qp_id, (void **)ops, nb_ops);
-	nb_ops = (*dev->dequeue_burst)
-			(dev->data->queue_pairs[qp_id], ops, nb_ops);
-#ifdef RTE_CRYPTO_CALLBACKS
-	if (unlikely(dev->deq_cbs != NULL)) {
-		struct rte_cryptodev_cb_rcu *list;
-		struct rte_cryptodev_cb *cb;
-
-		/* __ATOMIC_RELEASE memory order was used when the
-		 * call back was inserted into the list.
-		 * Since there is a clear dependency between loading
-		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
-		 * not required.
-		 */
-		list = &dev->deq_cbs[qp_id];
-		rte_rcu_qsbr_thread_online(list->qsbr, 0);
-		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
-
-		while (cb != NULL) {
-			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
-					cb->arg);
-			cb = cb->next;
-		};
-
-		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
-	}
-#endif
-	return nb_ops;
-}
-
-/**
- * Enqueue a burst of operations for processing on a crypto device.
- *
- * The rte_cryptodev_enqueue_burst() function is invoked to place
- * crypto operations on the queue *qp_id* of the device designated by
- * its *dev_id*.
- *
- * The *nb_ops* parameter is the number of operations to process which are
- * supplied in the *ops* array of *rte_crypto_op* structures.
- *
- * The rte_cryptodev_enqueue_burst() function returns the number of
- * operations it actually enqueued for processing. A return value equal to
- * *nb_ops* means that all packets have been enqueued.
- *
- * @param	dev_id		The identifier of the device.
- * @param	qp_id		The index of the queue pair which packets are
- *				to be enqueued for processing. The value
- *				must be in the range [0, nb_queue_pairs - 1]
- *				previously supplied to
- *				 *rte_cryptodev_configure*.
- * @param	ops		The address of an array of *nb_ops* pointers
- *				to *rte_crypto_op* structures which contain
- *				the crypto operations to be processed.
- * @param	nb_ops		The number of operations to process.
- *
- * @return
- * The number of operations actually enqueued on the crypto device. The return
- * value can be less than the value of the *nb_ops* parameter when the
- * crypto devices queue is full or if invalid parameters are specified in
- * a *rte_crypto_op*.
- */
-static inline uint16_t
-rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
-		struct rte_crypto_op **ops, uint16_t nb_ops)
-{
-	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
-
-#ifdef RTE_CRYPTO_CALLBACKS
-	if (unlikely(dev->enq_cbs != NULL)) {
-		struct rte_cryptodev_cb_rcu *list;
-		struct rte_cryptodev_cb *cb;
-
-		/* __ATOMIC_RELEASE memory order was used when the
-		 * call back was inserted into the list.
-		 * Since there is a clear dependency between loading
-		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
-		 * not required.
-		 */
-		list = &dev->enq_cbs[qp_id];
-		rte_rcu_qsbr_thread_online(list->qsbr, 0);
-		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
-
-		while (cb != NULL) {
-			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
-					cb->arg);
-			cb = cb->next;
-		};
-
-		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
-	}
-#endif
-
-	rte_cryptodev_trace_enqueue_burst(dev_id, qp_id, (void **)ops, nb_ops);
-	return (*dev->enqueue_burst)(
-			dev->data->queue_pairs[qp_id], ops, nb_ops);
-}
-
-
 /** Cryptodev symmetric crypto session
  * Each session is derived from a fixed xform chain. Therefore each session
  * has a fixed algo, key, op-type, digest_len etc.
@@ -2009,6 +1791,148 @@ int rte_cryptodev_remove_deq_callback(uint8_t dev_id,
 				      uint16_t qp_id,
 				      struct rte_cryptodev_cb *cb);
 
+#include <rte_cryptodev_core.h>
+/**
+ *
+ * Dequeue a burst of processed crypto operations from a queue on the crypto
+ * device. The dequeued operation are stored in *rte_crypto_op* structures
+ * whose pointers are supplied in the *ops* array.
+ *
+ * The rte_cryptodev_dequeue_burst() function returns the number of ops
+ * actually dequeued, which is the number of *rte_crypto_op* data structures
+ * effectively supplied into the *ops* array.
+ *
+ * A return value equal to *nb_ops* indicates that the queue contained
+ * at least *nb_ops* operations, and this is likely to signify that other
+ * processed operations remain in the devices output queue. Applications
+ * implementing a "retrieve as many processed operations as possible" policy
+ * can check this specific case and keep invoking the
+ * rte_cryptodev_dequeue_burst() function until a value less than
+ * *nb_ops* is returned.
+ *
+ * The rte_cryptodev_dequeue_burst() function does not provide any error
+ * notification to avoid the corresponding overhead.
+ *
+ * @param	dev_id		The symmetric crypto device identifier
+ * @param	qp_id		The index of the queue pair from which to
+ *				retrieve processed packets. The value must be
+ *				in the range [0, nb_queue_pair - 1] previously
+ *				supplied to rte_cryptodev_configure().
+ * @param	ops		The address of an array of pointers to
+ *				*rte_crypto_op* structures that must be
+ *				large enough to store *nb_ops* pointers in it.
+ * @param	nb_ops		The maximum number of operations to dequeue.
+ *
+ * @return
+ *   - The number of operations actually dequeued, which is the number
+ *   of pointers to *rte_crypto_op* structures effectively supplied to the
+ *   *ops* array.
+ */
+static inline uint16_t
+rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_crypto_op **ops, uint16_t nb_ops)
+{
+	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
+
+	rte_cryptodev_trace_dequeue_burst(dev_id, qp_id, (void **)ops, nb_ops);
+	nb_ops = (*dev->dequeue_burst)
+			(dev->data->queue_pairs[qp_id], ops, nb_ops);
+#ifdef RTE_CRYPTO_CALLBACKS
+	if (unlikely(dev->deq_cbs != NULL)) {
+		struct rte_cryptodev_cb_rcu *list;
+		struct rte_cryptodev_cb *cb;
+
+		/* __ATOMIC_RELEASE memory order was used when the
+		 * call back was inserted into the list.
+		 * Since there is a clear dependency between loading
+		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
+		 * not required.
+		 */
+		list = &dev->deq_cbs[qp_id];
+		rte_rcu_qsbr_thread_online(list->qsbr, 0);
+		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
+
+		while (cb != NULL) {
+			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
+					cb->arg);
+			cb = cb->next;
+		};
+
+		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
+	}
+#endif
+	return nb_ops;
+}
+
+/**
+ * Enqueue a burst of operations for processing on a crypto device.
+ *
+ * The rte_cryptodev_enqueue_burst() function is invoked to place
+ * crypto operations on the queue *qp_id* of the device designated by
+ * its *dev_id*.
+ *
+ * The *nb_ops* parameter is the number of operations to process which are
+ * supplied in the *ops* array of *rte_crypto_op* structures.
+ *
+ * The rte_cryptodev_enqueue_burst() function returns the number of
+ * operations it actually enqueued for processing. A return value equal to
+ * *nb_ops* means that all packets have been enqueued.
+ *
+ * @param	dev_id		The identifier of the device.
+ * @param	qp_id		The index of the queue pair which packets are
+ *				to be enqueued for processing. The value
+ *				must be in the range [0, nb_queue_pairs - 1]
+ *				previously supplied to
+ *				 *rte_cryptodev_configure*.
+ * @param	ops		The address of an array of *nb_ops* pointers
+ *				to *rte_crypto_op* structures which contain
+ *				the crypto operations to be processed.
+ * @param	nb_ops		The number of operations to process.
+ *
+ * @return
+ * The number of operations actually enqueued on the crypto device. The return
+ * value can be less than the value of the *nb_ops* parameter when the
+ * crypto devices queue is full or if invalid parameters are specified in
+ * a *rte_crypto_op*.
+ */
+static inline uint16_t
+rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_crypto_op **ops, uint16_t nb_ops)
+{
+	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
+
+#ifdef RTE_CRYPTO_CALLBACKS
+	if (unlikely(dev->enq_cbs != NULL)) {
+		struct rte_cryptodev_cb_rcu *list;
+		struct rte_cryptodev_cb *cb;
+
+		/* __ATOMIC_RELEASE memory order was used when the
+		 * call back was inserted into the list.
+		 * Since there is a clear dependency between loading
+		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
+		 * not required.
+		 */
+		list = &dev->enq_cbs[qp_id];
+		rte_rcu_qsbr_thread_online(list->qsbr, 0);
+		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
+
+		while (cb != NULL) {
+			nb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,
+					cb->arg);
+			cb = cb->next;
+		};
+
+		rte_rcu_qsbr_thread_offline(list->qsbr, 0);
+	}
+#endif
+
+	rte_cryptodev_trace_enqueue_burst(dev_id, qp_id, (void **)ops, nb_ops);
+	return (*dev->enqueue_burst)(
+			dev->data->queue_pairs[qp_id], ops, nb_ops);
+}
+
+
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/cryptodev/rte_cryptodev_core.h b/lib/cryptodev/rte_cryptodev_core.h
new file mode 100644
index 0000000000..1633e55889
--- /dev/null
+++ b/lib/cryptodev/rte_cryptodev_core.h
@@ -0,0 +1,100 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2021 Marvell.
+ */
+
+#ifndef _RTE_CRYPTODEV_CORE_H_
+#define _RTE_CRYPTODEV_CORE_H_
+
+/**
+ * @file
+ *
+ * RTE Crypto Device internal header.
+ *
+ * This header contains internal data types. But they are still part of the
+ * public API because they are used by inline functions in the published API.
+ *
+ * Applications should not use these directly.
+ *
+ */
+
+typedef uint16_t (*dequeue_pkt_burst_t)(void *qp,
+		struct rte_crypto_op **ops,	uint16_t nb_ops);
+/**< Dequeue processed packets from queue pair of a device. */
+
+typedef uint16_t (*enqueue_pkt_burst_t)(void *qp,
+		struct rte_crypto_op **ops,	uint16_t nb_ops);
+/**< Enqueue packets for processing on queue pair of a device. */
+
+/**
+ * @internal
+ * The data part, with no function pointers, associated with each device.
+ *
+ * This structure is safe to place in shared memory to be common among
+ * different processes in a multi-process configuration.
+ */
+struct rte_cryptodev_data {
+	uint8_t dev_id;
+	/**< Device ID for this instance */
+	uint8_t socket_id;
+	/**< Socket ID where memory is allocated */
+	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
+	/**< Unique identifier name */
+
+	__extension__
+	uint8_t dev_started : 1;
+	/**< Device state: STARTED(1)/STOPPED(0) */
+
+	struct rte_mempool *session_pool;
+	/**< Session memory pool */
+	void **queue_pairs;
+	/**< Array of pointers to queue pairs. */
+	uint16_t nb_queue_pairs;
+	/**< Number of device queue pairs. */
+
+	void *dev_private;
+	/**< PMD-specific private data */
+} __rte_cache_aligned;
+
+
+/** @internal The data structure associated with each crypto device. */
+struct rte_cryptodev {
+	dequeue_pkt_burst_t dequeue_burst;
+	/**< Pointer to PMD receive function. */
+	enqueue_pkt_burst_t enqueue_burst;
+	/**< Pointer to PMD transmit function. */
+
+	struct rte_cryptodev_data *data;
+	/**< Pointer to device data */
+	struct rte_cryptodev_ops *dev_ops;
+	/**< Functions exported by PMD */
+	uint64_t feature_flags;
+	/**< Feature flags exposes HW/SW features for the given device */
+	struct rte_device *device;
+	/**< Backing device */
+
+	uint8_t driver_id;
+	/**< Crypto driver identifier*/
+
+	struct rte_cryptodev_cb_list link_intr_cbs;
+	/**< User application callback for interrupts if present */
+
+	void *security_ctx;
+	/**< Context for security ops */
+
+	__extension__
+	uint8_t attached : 1;
+	/**< Flag indicating the device is attached */
+
+	struct rte_cryptodev_cb_rcu *enq_cbs;
+	/**< User application callback for pre enqueue processing */
+
+	struct rte_cryptodev_cb_rcu *deq_cbs;
+	/**< User application callback for post dequeue processing */
+} __rte_cache_aligned;
+
+/**
+ * The pool of rte_cryptodev structures.
+ */
+extern struct rte_cryptodev *rte_cryptodevs;
+
+#endif /* _RTE_CRYPTODEV_CORE_H_ */
-- 
2.25.1

[dpdk-dev] [PATCH v4 2/8] cryptodev: allocate max space for internal qp array

[Akhil Goyal]

At queue_pair config stage, allocate memory for maximum
number of queue pair pointers that a device can support.

This will allow fast path APIs(enqueue_burst/dequeue_burst) to
refer pointer to internal QP data without checking for currently
configured QPs.
This is required to hide the rte_cryptodev and rte_cryptodev_data
structure from user.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 lib/cryptodev/rte_cryptodev.c | 21 +++------------------
 1 file changed, 3 insertions(+), 18 deletions(-)

diff --git a/lib/cryptodev/rte_cryptodev.c b/lib/cryptodev/rte_cryptodev.c
index b913c434c5..eb86e629aa 100644
--- a/lib/cryptodev/rte_cryptodev.c
+++ b/lib/cryptodev/rte_cryptodev.c
@@ -978,7 +978,8 @@ rte_cryptodev_queue_pairs_config(struct rte_cryptodev *dev, uint16_t nb_qpairs,
 	if (dev->data->queue_pairs == NULL) { /* first time configuration */
 		dev->data->queue_pairs = rte_zmalloc_socket(
 				"cryptodev->queue_pairs",
-				sizeof(dev->data->queue_pairs[0]) * nb_qpairs,
+				sizeof(dev->data->queue_pairs[0]) *
+				dev_info.max_nb_queue_pairs,
 				RTE_CACHE_LINE_SIZE, socket_id);
 
 		if (dev->data->queue_pairs == NULL) {
@@ -1001,25 +1002,9 @@ rte_cryptodev_queue_pairs_config(struct rte_cryptodev *dev, uint16_t nb_qpairs,
 			ret = (*dev->dev_ops->queue_pair_release)(dev, i);
 			if (ret < 0)
 				return ret;
+			qp[i] = NULL;
 		}
 
-		qp = rte_realloc(qp, sizeof(qp[0]) * nb_qpairs,
-				RTE_CACHE_LINE_SIZE);
-		if (qp == NULL) {
-			CDEV_LOG_ERR("failed to realloc qp meta data,"
-						" nb_queues %u", nb_qpairs);
-			return -(ENOMEM);
-		}
-
-		if (nb_qpairs > old_nb_queues) {
-			uint16_t new_qs = nb_qpairs - old_nb_queues;
-
-			memset(qp + old_nb_queues, 0,
-				sizeof(qp[0]) * new_qs);
-		}
-
-		dev->data->queue_pairs = qp;
-
 	}
 	dev->data->nb_queue_pairs = nb_qpairs;
 	return 0;
-- 
2.25.1

[dpdk-dev] [PATCH v4 3/8] cryptodev: move inline APIs into separate structure

[Akhil Goyal]

Move fastpath inline function pointers from rte_cryptodev into a
separate structure accessed via a flat array.
The intension is to make rte_cryptodev and related structures private
to avoid future API/ABI breakages.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
Tested-by: Rebecca Troy <rebecca.troy@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 lib/cryptodev/cryptodev_pmd.c      | 53 +++++++++++++++++++++++++++++-
 lib/cryptodev/cryptodev_pmd.h      | 11 +++++++
 lib/cryptodev/rte_cryptodev.c      | 19 +++++++++++
 lib/cryptodev/rte_cryptodev_core.h | 29 ++++++++++++++++
 lib/cryptodev/version.map          |  5 +++
 5 files changed, 116 insertions(+), 1 deletion(-)

diff --git a/lib/cryptodev/cryptodev_pmd.c b/lib/cryptodev/cryptodev_pmd.c
index 44a70ecb35..fd74543682 100644
--- a/lib/cryptodev/cryptodev_pmd.c
+++ b/lib/cryptodev/cryptodev_pmd.c
@@ -3,7 +3,7 @@
  */
 
 #include <sys/queue.h>
-
+#include <rte_errno.h>
 #include <rte_string_fns.h>
 #include <rte_malloc.h>
 
@@ -160,3 +160,54 @@ rte_cryptodev_pmd_destroy(struct rte_cryptodev *cryptodev)
 
 	return 0;
 }
+
+static uint16_t
+dummy_crypto_enqueue_burst(__rte_unused void *qp,
+			   __rte_unused struct rte_crypto_op **ops,
+			   __rte_unused uint16_t nb_ops)
+{
+	CDEV_LOG_ERR(
+		"crypto enqueue burst requested for unconfigured device");
+	rte_errno = ENOTSUP;
+	return 0;
+}
+
+static uint16_t
+dummy_crypto_dequeue_burst(__rte_unused void *qp,
+			   __rte_unused struct rte_crypto_op **ops,
+			   __rte_unused uint16_t nb_ops)
+{
+	CDEV_LOG_ERR(
+		"crypto dequeue burst requested for unconfigured device");
+	rte_errno = ENOTSUP;
+	return 0;
+}
+
+void
+cryptodev_fp_ops_reset(struct rte_crypto_fp_ops *fp_ops)
+{
+	static struct rte_cryptodev_cb_rcu dummy_cb[RTE_MAX_QUEUES_PER_PORT];
+	static void *dummy_data[RTE_MAX_QUEUES_PER_PORT];
+	static const struct rte_crypto_fp_ops dummy = {
+		.enqueue_burst = dummy_crypto_enqueue_burst,
+		.dequeue_burst = dummy_crypto_dequeue_burst,
+		.qp = {
+			.data = dummy_data,
+			.enq_cb = dummy_cb,
+			.deq_cb = dummy_cb,
+		},
+	};
+
+	*fp_ops = dummy;
+}
+
+void
+cryptodev_fp_ops_set(struct rte_crypto_fp_ops *fp_ops,
+		     const struct rte_cryptodev *dev)
+{
+	fp_ops->enqueue_burst = dev->enqueue_burst;
+	fp_ops->dequeue_burst = dev->dequeue_burst;
+	fp_ops->qp.data = dev->data->queue_pairs;
+	fp_ops->qp.enq_cb = dev->enq_cbs;
+	fp_ops->qp.deq_cb = dev->deq_cbs;
+}
diff --git a/lib/cryptodev/cryptodev_pmd.h b/lib/cryptodev/cryptodev_pmd.h
index 36606dd10b..a71edbb991 100644
--- a/lib/cryptodev/cryptodev_pmd.h
+++ b/lib/cryptodev/cryptodev_pmd.h
@@ -516,6 +516,17 @@ RTE_INIT(init_ ##driver_id)\
 	driver_id = rte_cryptodev_allocate_driver(&crypto_drv, &(drv));\
 }
 
+/* Reset crypto device fastpath APIs to dummy values. */
+__rte_internal
+void
+cryptodev_fp_ops_reset(struct rte_crypto_fp_ops *fp_ops);
+
+/* Setup crypto device fastpath APIs. */
+__rte_internal
+void
+cryptodev_fp_ops_set(struct rte_crypto_fp_ops *fp_ops,
+		     const struct rte_cryptodev *dev);
+
 static inline void *
 get_sym_session_private_data(const struct rte_cryptodev_sym_session *sess,
 		uint8_t driver_id) {
diff --git a/lib/cryptodev/rte_cryptodev.c b/lib/cryptodev/rte_cryptodev.c
index eb86e629aa..305e013ebb 100644
--- a/lib/cryptodev/rte_cryptodev.c
+++ b/lib/cryptodev/rte_cryptodev.c
@@ -53,6 +53,9 @@ static struct rte_cryptodev_global cryptodev_globals = {
 		.nb_devs		= 0
 };
 
+/* Public fastpath APIs. */
+struct rte_crypto_fp_ops rte_crypto_fp_ops[RTE_CRYPTO_MAX_DEVS];
+
 /* spinlock for crypto device callbacks */
 static rte_spinlock_t rte_cryptodev_cb_lock = RTE_SPINLOCK_INITIALIZER;
 
@@ -917,6 +920,8 @@ rte_cryptodev_pmd_release_device(struct rte_cryptodev *cryptodev)
 
 	dev_id = cryptodev->data->dev_id;
 
+	cryptodev_fp_ops_reset(rte_crypto_fp_ops + dev_id);
+
 	/* Close device only if device operations have been set */
 	if (cryptodev->dev_ops) {
 		ret = rte_cryptodev_close(dev_id);
@@ -1080,6 +1085,9 @@ rte_cryptodev_start(uint8_t dev_id)
 	}
 
 	diag = (*dev->dev_ops->dev_start)(dev);
+	/* expose selection of PMD fast-path functions */
+	cryptodev_fp_ops_set(rte_crypto_fp_ops + dev_id, dev);
+
 	rte_cryptodev_trace_start(dev_id, diag);
 	if (diag == 0)
 		dev->data->dev_started = 1;
@@ -1109,6 +1117,9 @@ rte_cryptodev_stop(uint8_t dev_id)
 		return;
 	}
 
+	/* point fast-path functions to dummy ones */
+	cryptodev_fp_ops_reset(rte_crypto_fp_ops + dev_id);
+
 	(*dev->dev_ops->dev_stop)(dev);
 	rte_cryptodev_trace_stop(dev_id);
 	dev->data->dev_started = 0;
@@ -2411,3 +2422,11 @@ rte_cryptodev_allocate_driver(struct cryptodev_driver *crypto_drv,
 
 	return nb_drivers++;
 }
+
+RTE_INIT(cryptodev_init_fp_ops)
+{
+	uint32_t i;
+
+	for (i = 0; i != RTE_DIM(rte_crypto_fp_ops); i++)
+		cryptodev_fp_ops_reset(rte_crypto_fp_ops + i);
+}
diff --git a/lib/cryptodev/rte_cryptodev_core.h b/lib/cryptodev/rte_cryptodev_core.h
index 1633e55889..2bb9a228c1 100644
--- a/lib/cryptodev/rte_cryptodev_core.h
+++ b/lib/cryptodev/rte_cryptodev_core.h
@@ -25,6 +25,35 @@ typedef uint16_t (*enqueue_pkt_burst_t)(void *qp,
 		struct rte_crypto_op **ops,	uint16_t nb_ops);
 /**< Enqueue packets for processing on queue pair of a device. */
 
+/**
+ * @internal
+ * Structure used to hold opaque pointers to internal ethdev Rx/Tx
+ * queues data.
+ * The main purpose to expose these pointers at all - allow compiler
+ * to fetch this data for fast-path cryptodev inline functions in advance.
+ */
+struct rte_cryptodev_qpdata {
+	/** points to array of internal queue pair data pointers. */
+	void **data;
+	/** points to array of enqueue callback data pointers */
+	struct rte_cryptodev_cb_rcu *enq_cb;
+	/** points to array of dequeue callback data pointers */
+	struct rte_cryptodev_cb_rcu *deq_cb;
+};
+
+struct rte_crypto_fp_ops {
+	/** PMD enqueue burst function. */
+	enqueue_pkt_burst_t enqueue_burst;
+	/** PMD dequeue burst function. */
+	dequeue_pkt_burst_t dequeue_burst;
+	/** Internal queue pair data pointers. */
+	struct rte_cryptodev_qpdata qp;
+	/** Reserved for future ops. */
+	uintptr_t reserved[3];
+} __rte_cache_aligned;
+
+extern struct rte_crypto_fp_ops rte_crypto_fp_ops[RTE_CRYPTO_MAX_DEVS];
+
 /**
  * @internal
  * The data part, with no function pointers, associated with each device.
diff --git a/lib/cryptodev/version.map b/lib/cryptodev/version.map
index 43cf937e40..ed62ced221 100644
--- a/lib/cryptodev/version.map
+++ b/lib/cryptodev/version.map
@@ -45,6 +45,9 @@ DPDK_22 {
 	rte_cryptodev_sym_session_init;
 	rte_cryptodevs;
 
+	#added in 21.11
+	rte_crypto_fp_ops;
+
 	local: *;
 };
 
@@ -109,6 +112,8 @@ EXPERIMENTAL {
 INTERNAL {
 	global:
 
+	cryptodev_fp_ops_reset;
+	cryptodev_fp_ops_set;
 	rte_cryptodev_allocate_driver;
 	rte_cryptodev_pmd_allocate;
 	rte_cryptodev_pmd_callback_process;
-- 
2.25.1

[dpdk-dev] [PATCH v4 4/8] crypto/scheduler: use proper API for device start/stop

[Akhil Goyal]

The worker PMDs were using direct device start/stop
functions rather than rte_cryptodev_start(),
so rte_crypto_fp_ops never get set. This patch calls
the rte_cryptodev_start and stop APIs which start and
stop devices properly and fp_ops get set.

Reported-by: Ciara Power <ciara.power@intel.com>
Signed-off-by: Akhil Goyal <gakhil@marvell.com>
---
 drivers/crypto/scheduler/scheduler_pmd_ops.c | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)

diff --git a/drivers/crypto/scheduler/scheduler_pmd_ops.c b/drivers/crypto/scheduler/scheduler_pmd_ops.c
index 465b88ade8..f61640c9d6 100644
--- a/drivers/crypto/scheduler/scheduler_pmd_ops.c
+++ b/drivers/crypto/scheduler/scheduler_pmd_ops.c
@@ -181,10 +181,7 @@ scheduler_pmd_start(struct rte_cryptodev *dev)
 	/* start all workers */
 	for (i = 0; i < sched_ctx->nb_workers; i++) {
 		uint8_t worker_dev_id = sched_ctx->workers[i].dev_id;
-		struct rte_cryptodev *worker_dev =
-				rte_cryptodev_pmd_get_dev(worker_dev_id);
-
-		ret = (*worker_dev->dev_ops->dev_start)(worker_dev);
+		ret = rte_cryptodev_start(worker_dev_id);
 		if (ret < 0) {
 			CR_SCHED_LOG(ERR, "Failed to start worker dev %u",
 					worker_dev_id);
@@ -208,10 +205,8 @@ scheduler_pmd_stop(struct rte_cryptodev *dev)
 	/* stop all workers first */
 	for (i = 0; i < sched_ctx->nb_workers; i++) {
 		uint8_t worker_dev_id = sched_ctx->workers[i].dev_id;
-		struct rte_cryptodev *worker_dev =
-				rte_cryptodev_pmd_get_dev(worker_dev_id);
 
-		(*worker_dev->dev_ops->dev_stop)(worker_dev);
+		rte_cryptodev_stop(worker_dev_id);
 	}
 
 	if (*sched_ctx->ops.scheduler_stop)
-- 
2.25.1

[dpdk-dev] [PATCH v4 5/8] cryptodev: add PMD device probe finish API

[Akhil Goyal]

Added a rte_cryptodev_pmd_probing_finish API which
need to be called by the PMD after the device is initialized
completely. This will set the fast path function pointers
in the flat array for secondary process. For primary process,
these are set in rte_cryptodev_start.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
---
 lib/cryptodev/cryptodev_pmd.c | 16 ++++++++++++++++
 lib/cryptodev/cryptodev_pmd.h | 13 +++++++++++++
 lib/cryptodev/version.map     |  1 +
 3 files changed, 30 insertions(+)

diff --git a/lib/cryptodev/cryptodev_pmd.c b/lib/cryptodev/cryptodev_pmd.c
index fd74543682..739a0b3f34 100644
--- a/lib/cryptodev/cryptodev_pmd.c
+++ b/lib/cryptodev/cryptodev_pmd.c
@@ -161,6 +161,22 @@ rte_cryptodev_pmd_destroy(struct rte_cryptodev *cryptodev)
 	return 0;
 }
 
+void
+rte_cryptodev_pmd_probing_finish(struct rte_cryptodev *cryptodev)
+{
+	if (cryptodev == NULL)
+		return;
+	/*
+	 * for secondary process, at that point we expect device
+	 * to be already 'usable', so shared data and all function
+	 * pointers for fast-path devops have to be setup properly
+	 * inside rte_cryptodev.
+	 */
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
+		cryptodev_fp_ops_set(rte_crypto_fp_ops +
+				cryptodev->data->dev_id, cryptodev);
+}
+
 static uint16_t
 dummy_crypto_enqueue_burst(__rte_unused void *qp,
 			   __rte_unused struct rte_crypto_op **ops,
diff --git a/lib/cryptodev/cryptodev_pmd.h b/lib/cryptodev/cryptodev_pmd.h
index a71edbb991..9bb1e47ae4 100644
--- a/lib/cryptodev/cryptodev_pmd.h
+++ b/lib/cryptodev/cryptodev_pmd.h
@@ -509,6 +509,19 @@ __rte_internal
 uint8_t rte_cryptodev_allocate_driver(struct cryptodev_driver *crypto_drv,
 		const struct rte_driver *drv);
 
+/**
+ * @internal
+ * This is the last step of device probing. It must be called after a
+ * cryptodev is allocated and initialized successfully.
+ *
+ * @param	dev	Pointer to cryptodev struct
+ *
+ * @return
+ *  void
+ */
+__rte_internal
+void
+rte_cryptodev_pmd_probing_finish(struct rte_cryptodev *dev);
 
 #define RTE_PMD_REGISTER_CRYPTO_DRIVER(crypto_drv, drv, driver_id)\
 RTE_INIT(init_ ##driver_id)\
diff --git a/lib/cryptodev/version.map b/lib/cryptodev/version.map
index ed62ced221..157dac521d 100644
--- a/lib/cryptodev/version.map
+++ b/lib/cryptodev/version.map
@@ -123,5 +123,6 @@ INTERNAL {
 	rte_cryptodev_pmd_get_dev;
 	rte_cryptodev_pmd_get_named_dev;
 	rte_cryptodev_pmd_parse_input_args;
+	rte_cryptodev_pmd_probing_finish;
 	rte_cryptodev_pmd_release_device;
 };
-- 
2.25.1

[dpdk-dev] [PATCH v4 6/8] drivers/crypto: invoke probing finish function

[Akhil Goyal]

Invoke event_dev_probing_finish() function at the end of probing,
this function sets the function pointers in the fp_ops flat array
in case of secondary process.
For primary process, fp_ops is updated in rte_cryptodev_start().

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/armv8/rte_armv8_pmd.c        | 2 ++
 drivers/crypto/bcmfs/bcmfs_sym_pmd.c        | 2 ++
 drivers/crypto/caam_jr/caam_jr.c            | 2 ++
 drivers/crypto/ccp/rte_ccp_pmd.c            | 2 ++
 drivers/crypto/cnxk/cn10k_cryptodev.c       | 2 ++
 drivers/crypto/cnxk/cn9k_cryptodev.c        | 2 ++
 drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c | 4 +++-
 drivers/crypto/dpaa_sec/dpaa_sec.c          | 4 +++-
 drivers/crypto/ipsec_mb/ipsec_mb_private.c  | 2 ++
 drivers/crypto/mlx5/mlx5_crypto.c           | 3 +++
 drivers/crypto/mvsam/rte_mrvl_pmd.c         | 2 ++
 drivers/crypto/nitrox/nitrox_sym.c          | 3 +++
 drivers/crypto/null/null_crypto_pmd.c       | 2 ++
 drivers/crypto/octeontx/otx_cryptodev.c     | 4 +++-
 drivers/crypto/octeontx2/otx2_cryptodev.c   | 2 ++
 drivers/crypto/openssl/rte_openssl_pmd.c    | 2 ++
 drivers/crypto/qat/qat_asym_pmd.c           | 3 +++
 drivers/crypto/qat/qat_sym_pmd.c            | 2 ++
 drivers/crypto/scheduler/scheduler_pmd.c    | 2 ++
 drivers/crypto/virtio/virtio_cryptodev.c    | 2 ++
 20 files changed, 46 insertions(+), 3 deletions(-)

diff --git a/drivers/crypto/armv8/rte_armv8_pmd.c b/drivers/crypto/armv8/rte_armv8_pmd.c
index 36a1a9bb4f..32127a874c 100644
--- a/drivers/crypto/armv8/rte_armv8_pmd.c
+++ b/drivers/crypto/armv8/rte_armv8_pmd.c
@@ -802,6 +802,8 @@ cryptodev_armv8_crypto_create(const char *name,
 
 	internals->max_nb_qpairs = init_params->max_nb_queue_pairs;
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	return 0;
 
 init_error:
diff --git a/drivers/crypto/bcmfs/bcmfs_sym_pmd.c b/drivers/crypto/bcmfs/bcmfs_sym_pmd.c
index d1dd22823e..bc7fb67218 100644
--- a/drivers/crypto/bcmfs/bcmfs_sym_pmd.c
+++ b/drivers/crypto/bcmfs/bcmfs_sym_pmd.c
@@ -394,6 +394,8 @@ bcmfs_sym_dev_create(struct bcmfs_device *fsdev)
 	internals->sym_dev_id = cryptodev->data->dev_id;
 	internals->fsdev_capabilities = bcmfs_sym_get_capabilities();
 
+	rte_cryptodev_pmd_probing_finish(cryptodev);
+
 	BCMFS_LOG(DEBUG, "Created bcmfs-sym device %s as cryptodev instance %d",
 		  cryptodev->data->name, internals->sym_dev_id);
 	return 0;
diff --git a/drivers/crypto/caam_jr/caam_jr.c b/drivers/crypto/caam_jr/caam_jr.c
index 258750afe7..8c56610ac8 100644
--- a/drivers/crypto/caam_jr/caam_jr.c
+++ b/drivers/crypto/caam_jr/caam_jr.c
@@ -2368,6 +2368,8 @@ caam_jr_dev_init(const char *name,
 	security_instance->sess_cnt = 0;
 	dev->security_ctx = security_instance;
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	RTE_LOG(INFO, PMD, "%s cryptodev init\n", dev->data->name);
 
 	return 0;
diff --git a/drivers/crypto/ccp/rte_ccp_pmd.c b/drivers/crypto/ccp/rte_ccp_pmd.c
index a54d81de46..a35a8cd775 100644
--- a/drivers/crypto/ccp/rte_ccp_pmd.c
+++ b/drivers/crypto/ccp/rte_ccp_pmd.c
@@ -271,6 +271,8 @@ cryptodev_ccp_create(const char *name,
 	internals->auth_opt = init_params->auth_opt;
 	internals->crypto_num_dev = cryptodev_cnt;
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	return 0;
 
 init_error:
diff --git a/drivers/crypto/cnxk/cn10k_cryptodev.c b/drivers/crypto/cnxk/cn10k_cryptodev.c
index 869d322d9b..1e0a7b91c9 100644
--- a/drivers/crypto/cnxk/cn10k_cryptodev.c
+++ b/drivers/crypto/cnxk/cn10k_cryptodev.c
@@ -102,6 +102,8 @@ cn10k_cpt_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	cn10k_cpt_set_enqdeq_fns(dev);
 	cn10k_sec_ops_override();
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	return 0;
 
 dev_fini:
diff --git a/drivers/crypto/cnxk/cn9k_cryptodev.c b/drivers/crypto/cnxk/cn9k_cryptodev.c
index 54df06eec0..54cd77e575 100644
--- a/drivers/crypto/cnxk/cn9k_cryptodev.c
+++ b/drivers/crypto/cnxk/cn9k_cryptodev.c
@@ -102,6 +102,8 @@ cn9k_cpt_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	cn9k_cpt_set_enqdeq_fns(dev);
 	cn9k_sec_ops_override();
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	return 0;
 
 dev_fini:
diff --git a/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c b/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c
index 4eb3615250..cb2ad435bf 100644
--- a/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c
+++ b/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c
@@ -4032,8 +4032,10 @@ cryptodev_dpaa2_sec_probe(struct rte_dpaa2_driver *dpaa2_drv __rte_unused,
 
 	/* Invoke PMD device initialization function */
 	retval = dpaa2_sec_dev_init(cryptodev);
-	if (retval == 0)
+	if (retval == 0) {
+		rte_cryptodev_pmd_probing_finish(cryptodev);
 		return 0;
+	}
 
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		rte_free(cryptodev->data->dev_private);
diff --git a/drivers/crypto/dpaa_sec/dpaa_sec.c b/drivers/crypto/dpaa_sec/dpaa_sec.c
index c7ef1c7b0f..454b9c4785 100644
--- a/drivers/crypto/dpaa_sec/dpaa_sec.c
+++ b/drivers/crypto/dpaa_sec/dpaa_sec.c
@@ -3594,8 +3594,10 @@ cryptodev_dpaa_sec_probe(struct rte_dpaa_driver *dpaa_drv __rte_unused,
 
 	/* Invoke PMD device initialization function */
 	retval = dpaa_sec_dev_init(cryptodev);
-	if (retval == 0)
+	if (retval == 0) {
+		rte_cryptodev_pmd_probing_finish(cryptodev);
 		return 0;
+	}
 
 	retval = -ENXIO;
 out:
diff --git a/drivers/crypto/ipsec_mb/ipsec_mb_private.c b/drivers/crypto/ipsec_mb/ipsec_mb_private.c
index d896fa20b4..aab42c360c 100644
--- a/drivers/crypto/ipsec_mb/ipsec_mb_private.c
+++ b/drivers/crypto/ipsec_mb/ipsec_mb_private.c
@@ -134,6 +134,8 @@ ipsec_mb_create(struct rte_vdev_device *vdev,
 		break;
 	}
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	IPSEC_MB_LOG(INFO, "IPSec Multi-buffer library version used: %s\n",
 		     imb_get_version_str());
 
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 6a2f8b6ac6..14b6783e13 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -1061,6 +1061,9 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
 	pthread_mutex_lock(&priv_list_lock);
 	TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
+
+	rte_cryptodev_pmd_probing_finish(crypto_dev);
+
 	return 0;
 }
 
diff --git a/drivers/crypto/mvsam/rte_mrvl_pmd.c b/drivers/crypto/mvsam/rte_mrvl_pmd.c
index a72642a772..04efd9aaa8 100644
--- a/drivers/crypto/mvsam/rte_mrvl_pmd.c
+++ b/drivers/crypto/mvsam/rte_mrvl_pmd.c
@@ -1114,6 +1114,8 @@ cryptodev_mrvl_crypto_create(const char *name,
 	if (ret)
 		goto init_error;
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	return 0;
 
 init_error:
diff --git a/drivers/crypto/nitrox/nitrox_sym.c b/drivers/crypto/nitrox/nitrox_sym.c
index f8b7edcd69..cb5393d2f1 100644
--- a/drivers/crypto/nitrox/nitrox_sym.c
+++ b/drivers/crypto/nitrox/nitrox_sym.c
@@ -783,6 +783,9 @@ nitrox_sym_pmd_create(struct nitrox_device *ndev)
 	ndev->sym_dev = cdev->data->dev_private;
 	ndev->sym_dev->cdev = cdev;
 	ndev->sym_dev->ndev = ndev;
+
+	rte_cryptodev_pmd_probing_finish(cdev);
+
 	NITROX_LOG(DEBUG, "Created cryptodev '%s', dev_id %d, drv_id %d\n",
 		   cdev->data->name, cdev->data->dev_id, nitrox_sym_drv_id);
 	return 0;
diff --git a/drivers/crypto/null/null_crypto_pmd.c b/drivers/crypto/null/null_crypto_pmd.c
index f9935d52cc..9ecb434fd0 100644
--- a/drivers/crypto/null/null_crypto_pmd.c
+++ b/drivers/crypto/null/null_crypto_pmd.c
@@ -184,6 +184,8 @@ cryptodev_null_create(const char *name,
 
 	internals->max_nb_qpairs = init_params->max_nb_queue_pairs;
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	return 0;
 }
 
diff --git a/drivers/crypto/octeontx/otx_cryptodev.c b/drivers/crypto/octeontx/otx_cryptodev.c
index c294f86d79..05b78329d6 100644
--- a/drivers/crypto/octeontx/otx_cryptodev.c
+++ b/drivers/crypto/octeontx/otx_cryptodev.c
@@ -56,8 +56,10 @@ otx_cpt_pci_probe(struct rte_pci_driver *pci_drv,
 
 	/* Invoke PMD device initialization function */
 	retval = otx_cpt_dev_create(cryptodev);
-	if (retval == 0)
+	if (retval == 0) {
+		rte_cryptodev_pmd_probing_finish(cryptodev);
 		return 0;
+	}
 
 	CPT_LOG_ERR("[DRV %s]: Failed to create device "
 			"(vendor_id: 0x%x device_id: 0x%x",
diff --git a/drivers/crypto/octeontx2/otx2_cryptodev.c b/drivers/crypto/octeontx2/otx2_cryptodev.c
index 85b1f00263..fc7ad05366 100644
--- a/drivers/crypto/octeontx2/otx2_cryptodev.c
+++ b/drivers/crypto/octeontx2/otx2_cryptodev.c
@@ -135,6 +135,8 @@ otx2_cpt_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
 		otx2_cpt_set_enqdeq_fns(dev);
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	return 0;
 
 otx2_dev_fini:
diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c
index 13c6ea8724..5794ed8159 100644
--- a/drivers/crypto/openssl/rte_openssl_pmd.c
+++ b/drivers/crypto/openssl/rte_openssl_pmd.c
@@ -2213,6 +2213,8 @@ cryptodev_openssl_create(const char *name,
 
 	internals->max_nb_qpairs = init_params->max_nb_queue_pairs;
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	return 0;
 
 init_error:
diff --git a/drivers/crypto/qat/qat_asym_pmd.c b/drivers/crypto/qat/qat_asym_pmd.c
index e91bb0d317..0944d27a4d 100644
--- a/drivers/crypto/qat/qat_asym_pmd.c
+++ b/drivers/crypto/qat/qat_asym_pmd.c
@@ -343,6 +343,9 @@ qat_asym_dev_create(struct qat_pci_device *qat_pci_dev,
 	}
 
 	qat_pci_dev->asym_dev = internals;
+
+	rte_cryptodev_pmd_probing_finish(cryptodev);
+
 	QAT_LOG(DEBUG, "Created QAT ASYM device %s as cryptodev instance %d",
 			cryptodev->data->name, internals->asym_dev_id);
 	return 0;
diff --git a/drivers/crypto/qat/qat_sym_pmd.c b/drivers/crypto/qat/qat_sym_pmd.c
index efda921c05..d4f087733f 100644
--- a/drivers/crypto/qat/qat_sym_pmd.c
+++ b/drivers/crypto/qat/qat_sym_pmd.c
@@ -500,6 +500,8 @@ qat_sym_dev_create(struct qat_pci_device *qat_pci_dev,
 	QAT_LOG(DEBUG, "Created QAT SYM device %s as cryptodev instance %d",
 			cryptodev->data->name, internals->sym_dev_id);
 
+	rte_cryptodev_pmd_probing_finish(cryptodev);
+
 	return 0;
 
 error:
diff --git a/drivers/crypto/scheduler/scheduler_pmd.c b/drivers/crypto/scheduler/scheduler_pmd.c
index 560c26af50..dd198080bf 100644
--- a/drivers/crypto/scheduler/scheduler_pmd.c
+++ b/drivers/crypto/scheduler/scheduler_pmd.c
@@ -229,6 +229,8 @@ cryptodev_scheduler_create(const char *name,
 		return -ENOMEM;
 	}
 
+	rte_cryptodev_pmd_probing_finish(dev);
+
 	return 0;
 }
 
diff --git a/drivers/crypto/virtio/virtio_cryptodev.c b/drivers/crypto/virtio/virtio_cryptodev.c
index 8faa39df4a..ed64866758 100644
--- a/drivers/crypto/virtio/virtio_cryptodev.c
+++ b/drivers/crypto/virtio/virtio_cryptodev.c
@@ -754,6 +754,8 @@ crypto_virtio_create(const char *name, struct rte_pci_device *pci_dev,
 			VIRTIO_CRYPTO_PMD_GUEST_FEATURES) < 0)
 		return -1;
 
+	rte_cryptodev_pmd_probing_finish(cryptodev);
+
 	return 0;
 }
 
-- 
2.25.1

[dpdk-dev] [PATCH v4 7/8] cryptodev: update fast path APIs to use new flat array

[Akhil Goyal]

Rework fast-path cryptodev functions to use rte_crypto_fp_ops[].
While it is an API/ABI breakage, this change is intended to be
transparent for both users (no changes in user app is required) and
PMD developers (no changes in PMD is required).

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 lib/cryptodev/rte_cryptodev.h | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)

diff --git a/lib/cryptodev/rte_cryptodev.h b/lib/cryptodev/rte_cryptodev.h
index ce0dca72be..56e3868ada 100644
--- a/lib/cryptodev/rte_cryptodev.h
+++ b/lib/cryptodev/rte_cryptodev.h
@@ -1832,13 +1832,18 @@ static inline uint16_t
 rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
 		struct rte_crypto_op **ops, uint16_t nb_ops)
 {
-	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
+	const struct rte_crypto_fp_ops *fp_ops;
+	void *qp;
 
 	rte_cryptodev_trace_dequeue_burst(dev_id, qp_id, (void **)ops, nb_ops);
-	nb_ops = (*dev->dequeue_burst)
-			(dev->data->queue_pairs[qp_id], ops, nb_ops);
+
+	fp_ops = &rte_crypto_fp_ops[dev_id];
+	qp = fp_ops->qp.data[qp_id];
+
+	nb_ops = fp_ops->dequeue_burst(qp, ops, nb_ops);
+
 #ifdef RTE_CRYPTO_CALLBACKS
-	if (unlikely(dev->deq_cbs != NULL)) {
+	if (unlikely(fp_ops->qp.deq_cb != NULL)) {
 		struct rte_cryptodev_cb_rcu *list;
 		struct rte_cryptodev_cb *cb;
 
@@ -1848,7 +1853,7 @@ rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
 		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
 		 * not required.
 		 */
-		list = &dev->deq_cbs[qp_id];
+		list = &fp_ops->qp.deq_cb[qp_id];
 		rte_rcu_qsbr_thread_online(list->qsbr, 0);
 		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
 
@@ -1899,10 +1904,13 @@ static inline uint16_t
 rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
 		struct rte_crypto_op **ops, uint16_t nb_ops)
 {
-	struct rte_cryptodev *dev = &rte_cryptodevs[dev_id];
+	const struct rte_crypto_fp_ops *fp_ops;
+	void *qp;
 
+	fp_ops = &rte_crypto_fp_ops[dev_id];
+	qp = fp_ops->qp.data[qp_id];
 #ifdef RTE_CRYPTO_CALLBACKS
-	if (unlikely(dev->enq_cbs != NULL)) {
+	if (unlikely(fp_ops->qp.enq_cb != NULL)) {
 		struct rte_cryptodev_cb_rcu *list;
 		struct rte_cryptodev_cb *cb;
 
@@ -1912,7 +1920,7 @@ rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
 		 * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
 		 * not required.
 		 */
-		list = &dev->enq_cbs[qp_id];
+		list = &fp_ops->qp.enq_cb[qp_id];
 		rte_rcu_qsbr_thread_online(list->qsbr, 0);
 		cb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);
 
@@ -1927,8 +1935,7 @@ rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
 #endif
 
 	rte_cryptodev_trace_enqueue_burst(dev_id, qp_id, (void **)ops, nb_ops);
-	return (*dev->enqueue_burst)(
-			dev->data->queue_pairs[qp_id], ops, nb_ops);
+	return fp_ops->enqueue_burst(qp, ops, nb_ops);
 }
 
 
-- 
2.25.1

[dpdk-dev] [PATCH v4 8/8] cryptodev: move device specific structures

[Akhil Goyal]

The device specific structures - rte_cryptodev
and rte_cryptodev_data are moved to cryptodev_pmd.h
to hide it from the applications.

Signed-off-by: Akhil Goyal <gakhil@marvell.com>
Tested-by: Rebecca Troy <rebecca.troy@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 doc/guides/rel_notes/release_21_11.rst        |  6 ++
 drivers/crypto/ccp/ccp_dev.h                  |  2 +-
 drivers/crypto/cnxk/cn10k_ipsec.c             |  2 +-
 drivers/crypto/cnxk/cn9k_ipsec.c              |  2 +-
 .../crypto/cnxk/cnxk_cryptodev_capabilities.c |  2 +-
 drivers/crypto/cnxk/cnxk_cryptodev_sec.c      |  2 +-
 drivers/crypto/nitrox/nitrox_sym_reqmgr.c     |  2 +-
 drivers/crypto/octeontx/otx_cryptodev.c       |  1 -
 .../crypto/octeontx/otx_cryptodev_hw_access.c |  2 +-
 .../crypto/octeontx/otx_cryptodev_hw_access.h |  2 +-
 drivers/crypto/octeontx/otx_cryptodev_ops.h   |  2 +-
 .../crypto/octeontx2/otx2_cryptodev_mbox.c    |  2 +-
 drivers/crypto/scheduler/scheduler_failover.c |  2 +-
 .../crypto/scheduler/scheduler_multicore.c    |  2 +-
 .../scheduler/scheduler_pkt_size_distr.c      |  2 +-
 .../crypto/scheduler/scheduler_roundrobin.c   |  2 +-
 drivers/event/cnxk/cnxk_eventdev.h            |  2 +-
 drivers/event/dpaa/dpaa_eventdev.c            |  2 +-
 drivers/event/dpaa2/dpaa2_eventdev.c          |  2 +-
 drivers/event/octeontx/ssovf_evdev.c          |  2 +-
 .../event/octeontx2/otx2_evdev_crypto_adptr.c |  2 +-
 lib/cryptodev/cryptodev_pmd.h                 | 65 ++++++++++++++++++
 lib/cryptodev/rte_cryptodev_core.h            | 67 -------------------
 lib/cryptodev/version.map                     |  2 +-
 24 files changed, 91 insertions(+), 88 deletions(-)

diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst
index faa9164546..23bc854d16 100644
--- a/doc/guides/rel_notes/release_21_11.rst
+++ b/doc/guides/rel_notes/release_21_11.rst
@@ -328,6 +328,12 @@ ABI Changes
    Also, make sure to start the actual text at the margin.
    =======================================================
 
+* cryptodev: Made ``rte_cryptodev``, ``rte_cryptodev_data`` private
+  structures internal to DPDK. ``rte_cryptodevs`` can't be accessed directly
+  by user any more. While it is an ABI breakage, this change is intended
+  to be transparent for both users (no changes in user app is required) and
+  PMD developers (no changes in PMD is required).
+
 * security: ``rte_security_set_pkt_metadata`` and ``rte_security_get_userdata``
   routines used by inline outbound and inline inbound security processing were
   made inline and enhanced to do simple 64-bit set/get for PMDs that do not
diff --git a/drivers/crypto/ccp/ccp_dev.h b/drivers/crypto/ccp/ccp_dev.h
index ca5145c278..85c8fc47a2 100644
--- a/drivers/crypto/ccp/ccp_dev.h
+++ b/drivers/crypto/ccp/ccp_dev.h
@@ -17,7 +17,7 @@
 #include <rte_pci.h>
 #include <rte_spinlock.h>
 #include <rte_crypto_sym.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 
 /**< CCP sspecific */
 #define MAX_HW_QUEUES                   5
diff --git a/drivers/crypto/cnxk/cn10k_ipsec.c b/drivers/crypto/cnxk/cn10k_ipsec.c
index defc792aa8..27df1dcd64 100644
--- a/drivers/crypto/cnxk/cn10k_ipsec.c
+++ b/drivers/crypto/cnxk/cn10k_ipsec.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_malloc.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_esp.h>
 #include <rte_ip.h>
 #include <rte_security.h>
diff --git a/drivers/crypto/cnxk/cn9k_ipsec.c b/drivers/crypto/cnxk/cn9k_ipsec.c
index 9ca4d20c62..53fb793654 100644
--- a/drivers/crypto/cnxk/cn9k_ipsec.c
+++ b/drivers/crypto/cnxk/cn9k_ipsec.c
@@ -2,7 +2,7 @@
  * Copyright(C) 2021 Marvell.
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_ip.h>
 #include <rte_security.h>
 #include <rte_security_driver.h>
diff --git a/drivers/crypto/cnxk/cnxk_cryptodev_capabilities.c b/drivers/crypto/cnxk/cnxk_cryptodev_capabilities.c
index a227e6981c..a53b489a04 100644
--- a/drivers/crypto/cnxk/cnxk_cryptodev_capabilities.c
+++ b/drivers/crypto/cnxk/cnxk_cryptodev_capabilities.c
@@ -2,7 +2,7 @@
  * Copyright(C) 2021 Marvell.
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_security.h>
 
 #include "roc_api.h"
diff --git a/drivers/crypto/cnxk/cnxk_cryptodev_sec.c b/drivers/crypto/cnxk/cnxk_cryptodev_sec.c
index 8d04d4b575..2021d5c77e 100644
--- a/drivers/crypto/cnxk/cnxk_cryptodev_sec.c
+++ b/drivers/crypto/cnxk/cnxk_cryptodev_sec.c
@@ -2,7 +2,7 @@
  * Copyright(C) 2021 Marvell.
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_malloc.h>
 #include <rte_security.h>
 #include <rte_security_driver.h>
diff --git a/drivers/crypto/nitrox/nitrox_sym_reqmgr.c b/drivers/crypto/nitrox/nitrox_sym_reqmgr.c
index fe3ca25a0c..9edb0cc00f 100644
--- a/drivers/crypto/nitrox/nitrox_sym_reqmgr.c
+++ b/drivers/crypto/nitrox/nitrox_sym_reqmgr.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_crypto.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_cycles.h>
 #include <rte_errno.h>
 
diff --git a/drivers/crypto/octeontx/otx_cryptodev.c b/drivers/crypto/octeontx/otx_cryptodev.c
index 05b78329d6..337d06aab8 100644
--- a/drivers/crypto/octeontx/otx_cryptodev.c
+++ b/drivers/crypto/octeontx/otx_cryptodev.c
@@ -4,7 +4,6 @@
 
 #include <rte_bus_pci.h>
 #include <rte_common.h>
-#include <rte_cryptodev.h>
 #include <cryptodev_pmd.h>
 #include <rte_log.h>
 #include <rte_pci.h>
diff --git a/drivers/crypto/octeontx/otx_cryptodev_hw_access.c b/drivers/crypto/octeontx/otx_cryptodev_hw_access.c
index 7b89a62d81..20b288334a 100644
--- a/drivers/crypto/octeontx/otx_cryptodev_hw_access.c
+++ b/drivers/crypto/octeontx/otx_cryptodev_hw_access.c
@@ -7,7 +7,7 @@
 
 #include <rte_branch_prediction.h>
 #include <rte_common.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_errno.h>
 #include <rte_mempool.h>
 #include <rte_memzone.h>
diff --git a/drivers/crypto/octeontx/otx_cryptodev_hw_access.h b/drivers/crypto/octeontx/otx_cryptodev_hw_access.h
index 7c6b1e45b4..e48805fb09 100644
--- a/drivers/crypto/octeontx/otx_cryptodev_hw_access.h
+++ b/drivers/crypto/octeontx/otx_cryptodev_hw_access.h
@@ -7,7 +7,7 @@
 #include <stdbool.h>
 
 #include <rte_branch_prediction.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_cycles.h>
 #include <rte_io.h>
 #include <rte_memory.h>
diff --git a/drivers/crypto/octeontx/otx_cryptodev_ops.h b/drivers/crypto/octeontx/otx_cryptodev_ops.h
index f234f16970..83b82ea059 100644
--- a/drivers/crypto/octeontx/otx_cryptodev_ops.h
+++ b/drivers/crypto/octeontx/otx_cryptodev_ops.h
@@ -5,7 +5,7 @@
 #ifndef _OTX_CRYPTODEV_OPS_H_
 #define _OTX_CRYPTODEV_OPS_H_
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 
 #define OTX_CPT_MIN_HEADROOM_REQ	(24)
 #define OTX_CPT_MIN_TAILROOM_REQ	(8)
diff --git a/drivers/crypto/octeontx2/otx2_cryptodev_mbox.c b/drivers/crypto/octeontx2/otx2_cryptodev_mbox.c
index 1a8edae7eb..f9e7b0b474 100644
--- a/drivers/crypto/octeontx2/otx2_cryptodev_mbox.c
+++ b/drivers/crypto/octeontx2/otx2_cryptodev_mbox.c
@@ -1,7 +1,7 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright (C) 2019 Marvell International Ltd.
  */
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_ethdev.h>
 
 #include "otx2_cryptodev.h"
diff --git a/drivers/crypto/scheduler/scheduler_failover.c b/drivers/crypto/scheduler/scheduler_failover.c
index 844312dd1b..5023577ef8 100644
--- a/drivers/crypto/scheduler/scheduler_failover.c
+++ b/drivers/crypto/scheduler/scheduler_failover.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2017 Intel Corporation
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_malloc.h>
 
 #include "rte_cryptodev_scheduler_operations.h"
diff --git a/drivers/crypto/scheduler/scheduler_multicore.c b/drivers/crypto/scheduler/scheduler_multicore.c
index 1e2e8dbf9f..900ab4049d 100644
--- a/drivers/crypto/scheduler/scheduler_multicore.c
+++ b/drivers/crypto/scheduler/scheduler_multicore.c
@@ -3,7 +3,7 @@
  */
 #include <unistd.h>
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_malloc.h>
 
 #include "rte_cryptodev_scheduler_operations.h"
diff --git a/drivers/crypto/scheduler/scheduler_pkt_size_distr.c b/drivers/crypto/scheduler/scheduler_pkt_size_distr.c
index 57e330a744..933a5c6978 100644
--- a/drivers/crypto/scheduler/scheduler_pkt_size_distr.c
+++ b/drivers/crypto/scheduler/scheduler_pkt_size_distr.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2017 Intel Corporation
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_malloc.h>
 
 #include "rte_cryptodev_scheduler_operations.h"
diff --git a/drivers/crypto/scheduler/scheduler_roundrobin.c b/drivers/crypto/scheduler/scheduler_roundrobin.c
index bc4a632106..ace2dec2ec 100644
--- a/drivers/crypto/scheduler/scheduler_roundrobin.c
+++ b/drivers/crypto/scheduler/scheduler_roundrobin.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2017 Intel Corporation
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_malloc.h>
 
 #include "rte_cryptodev_scheduler_operations.h"
diff --git a/drivers/event/cnxk/cnxk_eventdev.h b/drivers/event/cnxk/cnxk_eventdev.h
index 8a5c737e4b..b57004c0dc 100644
--- a/drivers/event/cnxk/cnxk_eventdev.h
+++ b/drivers/event/cnxk/cnxk_eventdev.h
@@ -7,7 +7,7 @@
 
 #include <string.h>
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_devargs.h>
 #include <rte_ethdev.h>
 #include <rte_event_eth_rx_adapter.h>
diff --git a/drivers/event/dpaa/dpaa_eventdev.c b/drivers/event/dpaa/dpaa_eventdev.c
index ec74160325..1d7ddfe1d1 100644
--- a/drivers/event/dpaa/dpaa_eventdev.c
+++ b/drivers/event/dpaa/dpaa_eventdev.c
@@ -28,7 +28,7 @@
 #include <rte_ethdev.h>
 #include <rte_event_eth_rx_adapter.h>
 #include <rte_event_eth_tx_adapter.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_dpaa_bus.h>
 #include <rte_dpaa_logs.h>
 #include <rte_cycles.h>
diff --git a/drivers/event/dpaa2/dpaa2_eventdev.c b/drivers/event/dpaa2/dpaa2_eventdev.c
index 5ccf22f77f..e03afb2958 100644
--- a/drivers/event/dpaa2/dpaa2_eventdev.c
+++ b/drivers/event/dpaa2/dpaa2_eventdev.c
@@ -25,7 +25,7 @@
 #include <rte_pci.h>
 #include <rte_bus_vdev.h>
 #include <ethdev_driver.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_event_eth_rx_adapter.h>
 #include <rte_event_eth_tx_adapter.h>
 
diff --git a/drivers/event/octeontx/ssovf_evdev.c b/drivers/event/octeontx/ssovf_evdev.c
index b93f6ec8c6..9846fce34b 100644
--- a/drivers/event/octeontx/ssovf_evdev.c
+++ b/drivers/event/octeontx/ssovf_evdev.c
@@ -5,7 +5,7 @@
 #include <inttypes.h>
 
 #include <rte_common.h>
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_debug.h>
 #include <rte_dev.h>
 #include <rte_eal.h>
diff --git a/drivers/event/octeontx2/otx2_evdev_crypto_adptr.c b/drivers/event/octeontx2/otx2_evdev_crypto_adptr.c
index d9a002625c..d59d6c53f6 100644
--- a/drivers/event/octeontx2/otx2_evdev_crypto_adptr.c
+++ b/drivers/event/octeontx2/otx2_evdev_crypto_adptr.c
@@ -2,7 +2,7 @@
  * Copyright (C) 2020-2021 Marvell.
  */
 
-#include <rte_cryptodev.h>
+#include <cryptodev_pmd.h>
 #include <rte_eventdev.h>
 
 #include "otx2_cryptodev.h"
diff --git a/lib/cryptodev/cryptodev_pmd.h b/lib/cryptodev/cryptodev_pmd.h
index 9bb1e47ae4..89bf2af399 100644
--- a/lib/cryptodev/cryptodev_pmd.h
+++ b/lib/cryptodev/cryptodev_pmd.h
@@ -52,6 +52,71 @@ struct rte_cryptodev_pmd_init_params {
 	unsigned int max_nb_queue_pairs;
 };
 
+/**
+ * @internal
+ * The data part, with no function pointers, associated with each device.
+ *
+ * This structure is safe to place in shared memory to be common among
+ * different processes in a multi-process configuration.
+ */
+struct rte_cryptodev_data {
+	/** Device ID for this instance */
+	uint8_t dev_id;
+	/** Socket ID where memory is allocated */
+	uint8_t socket_id;
+	/** Unique identifier name */
+	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
+
+	__extension__
+	/** Device state: STARTED(1)/STOPPED(0) */
+	uint8_t dev_started : 1;
+
+	/** Session memory pool */
+	struct rte_mempool *session_pool;
+	/** Array of pointers to queue pairs. */
+	void **queue_pairs;
+	/** Number of device queue pairs. */
+	uint16_t nb_queue_pairs;
+
+	/** PMD-specific private data */
+	void *dev_private;
+} __rte_cache_aligned;
+
+/** @internal The data structure associated with each crypto device. */
+struct rte_cryptodev {
+	/** Pointer to PMD dequeue function. */
+	dequeue_pkt_burst_t dequeue_burst;
+	/** Pointer to PMD enqueue function. */
+	enqueue_pkt_burst_t enqueue_burst;
+
+	/** Pointer to device data */
+	struct rte_cryptodev_data *data;
+	/** Functions exported by PMD */
+	struct rte_cryptodev_ops *dev_ops;
+	/** Feature flags exposes HW/SW features for the given device */
+	uint64_t feature_flags;
+	/** Backing device */
+	struct rte_device *device;
+
+	/** Crypto driver identifier*/
+	uint8_t driver_id;
+
+	/** User application callback for interrupts if present */
+	struct rte_cryptodev_cb_list link_intr_cbs;
+
+	/** Context for security ops */
+	void *security_ctx;
+
+	__extension__
+	/** Flag indicating the device is attached */
+	uint8_t attached : 1;
+
+	/** User application callback for pre enqueue processing */
+	struct rte_cryptodev_cb_rcu *enq_cbs;
+	/** User application callback for post dequeue processing */
+	struct rte_cryptodev_cb_rcu *deq_cbs;
+} __rte_cache_aligned;
+
 /** Global structure used for maintaining state of allocated crypto devices */
 struct rte_cryptodev_global {
 	struct rte_cryptodev *devs;	/**< Device information array */
diff --git a/lib/cryptodev/rte_cryptodev_core.h b/lib/cryptodev/rte_cryptodev_core.h
index 2bb9a228c1..16832f645d 100644
--- a/lib/cryptodev/rte_cryptodev_core.h
+++ b/lib/cryptodev/rte_cryptodev_core.h
@@ -54,73 +54,6 @@ struct rte_crypto_fp_ops {
 
 extern struct rte_crypto_fp_ops rte_crypto_fp_ops[RTE_CRYPTO_MAX_DEVS];
 
-/**
- * @internal
- * The data part, with no function pointers, associated with each device.
- *
- * This structure is safe to place in shared memory to be common among
- * different processes in a multi-process configuration.
- */
-struct rte_cryptodev_data {
-	uint8_t dev_id;
-	/**< Device ID for this instance */
-	uint8_t socket_id;
-	/**< Socket ID where memory is allocated */
-	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
-	/**< Unique identifier name */
-
-	__extension__
-	uint8_t dev_started : 1;
-	/**< Device state: STARTED(1)/STOPPED(0) */
-
-	struct rte_mempool *session_pool;
-	/**< Session memory pool */
-	void **queue_pairs;
-	/**< Array of pointers to queue pairs. */
-	uint16_t nb_queue_pairs;
-	/**< Number of device queue pairs. */
-
-	void *dev_private;
-	/**< PMD-specific private data */
-} __rte_cache_aligned;
-
-
-/** @internal The data structure associated with each crypto device. */
-struct rte_cryptodev {
-	dequeue_pkt_burst_t dequeue_burst;
-	/**< Pointer to PMD receive function. */
-	enqueue_pkt_burst_t enqueue_burst;
-	/**< Pointer to PMD transmit function. */
-
-	struct rte_cryptodev_data *data;
-	/**< Pointer to device data */
-	struct rte_cryptodev_ops *dev_ops;
-	/**< Functions exported by PMD */
-	uint64_t feature_flags;
-	/**< Feature flags exposes HW/SW features for the given device */
-	struct rte_device *device;
-	/**< Backing device */
-
-	uint8_t driver_id;
-	/**< Crypto driver identifier*/
-
-	struct rte_cryptodev_cb_list link_intr_cbs;
-	/**< User application callback for interrupts if present */
-
-	void *security_ctx;
-	/**< Context for security ops */
-
-	__extension__
-	uint8_t attached : 1;
-	/**< Flag indicating the device is attached */
-
-	struct rte_cryptodev_cb_rcu *enq_cbs;
-	/**< User application callback for pre enqueue processing */
-
-	struct rte_cryptodev_cb_rcu *deq_cbs;
-	/**< User application callback for post dequeue processing */
-} __rte_cache_aligned;
-
 /**
  * The pool of rte_cryptodev structures.
  */
diff --git a/lib/cryptodev/version.map b/lib/cryptodev/version.map
index 157dac521d..b55b4b8e7e 100644
--- a/lib/cryptodev/version.map
+++ b/lib/cryptodev/version.map
@@ -43,7 +43,6 @@ DPDK_22 {
 	rte_cryptodev_sym_session_create;
 	rte_cryptodev_sym_session_free;
 	rte_cryptodev_sym_session_init;
-	rte_cryptodevs;
 
 	#added in 21.11
 	rte_crypto_fp_ops;
@@ -125,4 +124,5 @@ INTERNAL {
 	rte_cryptodev_pmd_parse_input_args;
 	rte_cryptodev_pmd_probing_finish;
 	rte_cryptodev_pmd_release_device;
+	rte_cryptodevs;
 };
-- 
2.25.1

Re: [dpdk-dev] [PATCH v4 4/8] crypto/scheduler: use proper API for device start/stop

[Zhang, Roy Fan]

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Wednesday, October 20, 2021 12:28 PM
> To: dev@dpdk.org
> Cc: thomas@monjalon.net; david.marchand@redhat.com;
> hemant.agrawal@nxp.com; anoobj@marvell.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> Doherty, Declan <declan.doherty@intel.com>; matan@nvidia.com;
> g.singh@nxp.com; Zhang, Roy Fan <roy.fan.zhang@intel.com>;
> jianjay.zhou@huawei.com; asomalap@amd.com; ruifeng.wang@arm.com;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>; Nicolau, Radu
> <radu.nicolau@intel.com>; ajit.khaparde@broadcom.com;
> rnagadheeraj@marvell.com; adwivedi@marvell.com; Power, Ciara
> <ciara.power@intel.com>; Akhil Goyal <gakhil@marvell.com>
> Subject: [PATCH v4 4/8] crypto/scheduler: use proper API for device
> start/stop
> 
> The worker PMDs were using direct device start/stop
> functions rather than rte_cryptodev_start(),
> so rte_crypto_fp_ops never get set. This patch calls
> the rte_cryptodev_start and stop APIs which start and
> stop devices properly and fp_ops get set.
> 
> Reported-by: Ciara Power <ciara.power@intel.com>
> Signed-off-by: Akhil Goyal <gakhil@marvell.com>
> ---
>  drivers/crypto/scheduler/scheduler_pmd_ops.c | 9 ++-------
>  1 file changed, 2 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/crypto/scheduler/scheduler_pmd_ops.c
> b/drivers/crypto/scheduler/scheduler_pmd_ops.c
> index 465b88ade8..f61640c9d6 100644
> --- a/drivers/crypto/scheduler/scheduler_pmd_ops.c
> +++ b/drivers/crypto/scheduler/scheduler_pmd_ops.c
> @@ -181,10 +181,7 @@ scheduler_pmd_start(struct rte_cryptodev *dev)
>  	/* start all workers */
>  	for (i = 0; i < sched_ctx->nb_workers; i++) {
>  		uint8_t worker_dev_id = sched_ctx->workers[i].dev_id;
> -		struct rte_cryptodev *worker_dev =
> -
> 	rte_cryptodev_pmd_get_dev(worker_dev_id);
> -
> -		ret = (*worker_dev->dev_ops->dev_start)(worker_dev);
> +		ret = rte_cryptodev_start(worker_dev_id);
>  		if (ret < 0) {
>  			CR_SCHED_LOG(ERR, "Failed to start worker dev %u",
>  					worker_dev_id);
> @@ -208,10 +205,8 @@ scheduler_pmd_stop(struct rte_cryptodev *dev)
>  	/* stop all workers first */
>  	for (i = 0; i < sched_ctx->nb_workers; i++) {
>  		uint8_t worker_dev_id = sched_ctx->workers[i].dev_id;
> -		struct rte_cryptodev *worker_dev =
> -
> 	rte_cryptodev_pmd_get_dev(worker_dev_id);
> 
> -		(*worker_dev->dev_ops->dev_stop)(worker_dev);
> +		rte_cryptodev_stop(worker_dev_id);
>  	}
> 
>  	if (*sched_ctx->ops.scheduler_stop)
> --
> 2.25.1
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>

Re: [dpdk-dev] [PATCH v4 4/8] crypto/scheduler: use proper API for device start/stop

[Ananyev, Konstantin]

> > The worker PMDs were using direct device start/stop
> > functions rather than rte_cryptodev_start(),
> > so rte_crypto_fp_ops never get set. This patch calls
> > the rte_cryptodev_start and stop APIs which start and
> > stop devices properly and fp_ops get set.
> >
> > Reported-by: Ciara Power <ciara.power@intel.com>
> > Signed-off-by: Akhil Goyal <gakhil@marvell.com>
> > ---
> >  drivers/crypto/scheduler/scheduler_pmd_ops.c | 9 ++-------
> >  1 file changed, 2 insertions(+), 7 deletions(-)
> >
> > diff --git a/drivers/crypto/scheduler/scheduler_pmd_ops.c
> > b/drivers/crypto/scheduler/scheduler_pmd_ops.c
> > index 465b88ade8..f61640c9d6 100644
> > --- a/drivers/crypto/scheduler/scheduler_pmd_ops.c
> > +++ b/drivers/crypto/scheduler/scheduler_pmd_ops.c
> > @@ -181,10 +181,7 @@ scheduler_pmd_start(struct rte_cryptodev *dev)
> >  	/* start all workers */
> >  	for (i = 0; i < sched_ctx->nb_workers; i++) {
> >  		uint8_t worker_dev_id = sched_ctx->workers[i].dev_id;
> > -		struct rte_cryptodev *worker_dev =
> > -
> > 	rte_cryptodev_pmd_get_dev(worker_dev_id);
> > -
> > -		ret = (*worker_dev->dev_ops->dev_start)(worker_dev);
> > +		ret = rte_cryptodev_start(worker_dev_id);
> >  		if (ret < 0) {
> >  			CR_SCHED_LOG(ERR, "Failed to start worker dev %u",
> >  					worker_dev_id);
> > @@ -208,10 +205,8 @@ scheduler_pmd_stop(struct rte_cryptodev *dev)
> >  	/* stop all workers first */
> >  	for (i = 0; i < sched_ctx->nb_workers; i++) {
> >  		uint8_t worker_dev_id = sched_ctx->workers[i].dev_id;
> > -		struct rte_cryptodev *worker_dev =
> > -
> > 	rte_cryptodev_pmd_get_dev(worker_dev_id);
> >
> > -		(*worker_dev->dev_ops->dev_stop)(worker_dev);
> > +		rte_cryptodev_stop(worker_dev_id);
> >  	}
> >
> >  	if (*sched_ctx->ops.scheduler_stop)
> > --
> > 2.25.1
> Acked-by: Fan Zhang <roy.fan.zhang@intel.com>

Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>

Re: [dpdk-dev] [PATCH v4 0/8] cryptodev: hide internal structures

[Akhil Goyal]

> Subject: [PATCH v4 0/8] cryptodev: hide internal structures
> 
> Structures rte_cryptodev and rte_cryptodev_data are not
> supposed to be directly used by the application. These
> are made public as they are used by inline datapath
> public APIs.
> This patchset, creates a new rte_cryptodev_core.h file
> which helps in defining a data structure to hold datapath
> APIs in a flat array based on the device identifier which
> is filled by the PMD.
> 
> Similar series for ethdev and eventdev are also floated on ML.
> https://patchwork.dpdk.org/project/dpdk/list/?series=19428
> https://patchwork.dpdk.org/project/dpdk/list/?series=19405
> 
> 
> changes in
> v4:
> - fixed number of reserved pointers.(Konstantin)
> - added release notes
> - added a patch for scheduler issue(Ciara).
> v3: fixed multiprocess and enq-deq callback handling.
> v2: align with the latest versions of above series.

Series Applied to dpdk-next-crypto