DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] [RFC PATCH 0/5] Support Intel IPSec MB v0.53 in DPDK 18.11
@ 2020-03-05 15:34 Pablo de Lara
  2020-03-05 15:34 ` [dpdk-dev] [RFC PATCH 1/5] crypto/zuc: use IPSec MB library v0.53 Pablo de Lara
                   ` (5 more replies)
  0 siblings, 6 replies; 8+ messages in thread
From: Pablo de Lara @ 2020-03-05 15:34 UTC (permalink / raw)
  To: dev; +Cc: Pablo de Lara

This patchset adds support to the following crypto PMDs to use
Intel IPSec MB v0.53, in DPDK v18.11:
- AESNI MB PMD: had support up to v0.52, extending to v0.53
- AESNI GCM PMD: had support up to v0.52, extending to v0.53
- SNOW3G PMD: linking now to IPSec MB v0.53, instead of libsso
- ZUC PMD: linking now to IPSec MB v0.53, instead of libsso
- KASUMI PMD: linking now to IPSec MB v0.53, instead of libsso

Pablo de Lara (5):
  crypto/zuc: use IPSec MB library v0.53
  crypto/snow3g: use IPSec MB library v0.53
  crypto/kasumi: use IPSec MB library v0.53
  crypto/aesni_mb: support IPSec MB library v0.53
  crypto/aesni_gcm: support IPSec MB library v0.53

 devtools/test-build.sh                        |  14 +-
 doc/guides/cryptodevs/kasumi.rst              |  62 +--
 doc/guides/cryptodevs/snow3g.rst              |  58 ++-
 doc/guides/cryptodevs/zuc.rst                 |  52 +-
 drivers/crypto/aesni_gcm/aesni_gcm_ops.h      |  65 +--
 drivers/crypto/aesni_gcm/aesni_gcm_pmd.c      | 130 +++--
 drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c  |   4 +-
 .../crypto/aesni_gcm/aesni_gcm_pmd_private.h  |   4 +
 drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c    | 476 ++++++++++++------
 .../crypto/aesni_mb/rte_aesni_mb_pmd_ops.c    | 205 +++++---
 .../aesni_mb/rte_aesni_mb_pmd_private.h       |  30 +-
 drivers/crypto/kasumi/Makefile                |  26 +-
 drivers/crypto/kasumi/meson.build             |  11 +-
 drivers/crypto/kasumi/rte_kasumi_pmd.c        |  79 +--
 drivers/crypto/kasumi/rte_kasumi_pmd_ops.c    |   8 +-
 .../crypto/kasumi/rte_kasumi_pmd_private.h    |  12 +-
 drivers/crypto/snow3g/Makefile                |  29 +-
 drivers/crypto/snow3g/meson.build             |  21 +
 drivers/crypto/snow3g/rte_snow3g_pmd.c        |  79 +--
 drivers/crypto/snow3g/rte_snow3g_pmd_ops.c    |   8 +-
 .../crypto/snow3g/rte_snow3g_pmd_private.h    |  12 +-
 drivers/crypto/zuc/Makefile                   |  28 +-
 drivers/crypto/zuc/meson.build                |  13 +-
 drivers/crypto/zuc/rte_zuc_pmd.c              |  58 ++-
 drivers/crypto/zuc/rte_zuc_pmd_ops.c          |   2 +
 drivers/crypto/zuc/rte_zuc_pmd_private.h      |   6 +-
 mk/rte.app.mk                                 |   6 +-
 27 files changed, 972 insertions(+), 526 deletions(-)
 create mode 100644 drivers/crypto/snow3g/meson.build

-- 
2.24.1


^ permalink raw reply	[flat|nested] 8+ messages in thread

* [dpdk-dev] [RFC PATCH 1/5] crypto/zuc: use IPSec MB library v0.53
  2020-03-05 15:34 [dpdk-dev] [RFC PATCH 0/5] Support Intel IPSec MB v0.53 in DPDK 18.11 Pablo de Lara
@ 2020-03-05 15:34 ` Pablo de Lara
  2020-03-05 15:34 ` [dpdk-dev] [RFC PATCH 2/5] crypto/snow3g: " Pablo de Lara
                   ` (4 subsequent siblings)
  5 siblings, 0 replies; 8+ messages in thread
From: Pablo de Lara @ 2020-03-05 15:34 UTC (permalink / raw)
  To: dev; +Cc: Pablo de Lara

Link against Intel IPSec Multi-buffer library, which
added support for ZUC-EEA3 and ZUC-EIA3 from version v0.53,
moving from libSSO ZUC library.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
---
 devtools/test-build.sh                   |  6 +--
 doc/guides/cryptodevs/zuc.rst            | 52 ++++++++++++---------
 drivers/crypto/zuc/Makefile              | 28 +++++++-----
 drivers/crypto/zuc/meson.build           | 13 +++++-
 drivers/crypto/zuc/rte_zuc_pmd.c         | 58 +++++++++++++++++-------
 drivers/crypto/zuc/rte_zuc_pmd_ops.c     |  2 +
 drivers/crypto/zuc/rte_zuc_pmd_private.h |  6 ++-
 mk/rte.app.mk                            |  2 +-
 8 files changed, 110 insertions(+), 57 deletions(-)

diff --git a/devtools/test-build.sh b/devtools/test-build.sh
index 42f4ad003..911f77e01 100755
--- a/devtools/test-build.sh
+++ b/devtools/test-build.sh
@@ -25,7 +25,6 @@ default_path=$PATH
 # - LIBMUSDK_PATH
 # - LIBSSO_SNOW3G_PATH
 # - LIBSSO_KASUMI_PATH
-# - LIBSSO_ZUC_PATH
 . $(dirname $(readlink -e $0))/load-devel-config
 
 print_usage () {
@@ -111,7 +110,6 @@ reset_env ()
 	unset LIBMUSDK_PATH
 	unset LIBSSO_SNOW3G_PATH
 	unset LIBSSO_KASUMI_PATH
-	unset LIBSSO_ZUC_PATH
 	unset PQOS_INSTALL_PATH
 }
 
@@ -165,12 +163,12 @@ config () # <directory> <target> <options>
 		sed -ri       's,(PMD_AESNI_MB=)n,\1y,' $1/.config
 		test "$DPDK_DEP_IPSEC_MB" != y || \
 		sed -ri      's,(PMD_AESNI_GCM=)n,\1y,' $1/.config
+		test "$DPDK_DEP_IPSEC_MB" != y || \
+		sed -ri            's,(PMD_ZUC=)n,\1y,' $1/.config
 		test -z "$LIBSSO_SNOW3G_PATH" || \
 		sed -ri         's,(PMD_SNOW3G=)n,\1y,' $1/.config
 		test -z "$LIBSSO_KASUMI_PATH" || \
 		sed -ri         's,(PMD_KASUMI=)n,\1y,' $1/.config
-		test -z "$LIBSSO_ZUC_PATH" || \
-		sed -ri            's,(PMD_ZUC=)n,\1y,' $1/.config
 		test "$DPDK_DEP_SSL" != y || \
 		sed -ri            's,(PMD_CCP=)n,\1y,' $1/.config
 		test "$DPDK_DEP_SSL" != y || \
diff --git a/doc/guides/cryptodevs/zuc.rst b/doc/guides/cryptodevs/zuc.rst
index e38989968..d4001b1f4 100644
--- a/doc/guides/cryptodevs/zuc.rst
+++ b/doc/guides/cryptodevs/zuc.rst
@@ -1,12 +1,12 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
-    Copyright(c) 2016 Intel Corporation.
+    Copyright(c) 2016-2019 Intel Corporation.
 
 ZUC Crypto Poll Mode Driver
 ===========================
 
-The ZUC PMD (**librte_pmd_zuc**) provides poll mode crypto driver
-support for utilizing Intel Libsso library, which implements F8 and F9 functions
-for ZUC EEA3 cipher and EIA3 hash algorithms.
+The ZUC PMD (**librte_pmd_zuc**) provides poll mode crypto driver support for
+utilizing `Intel IPSec Multi-buffer library <https://github.com/01org/intel-ipsec-mb>`_
+which implements F8 and F9 functions for ZUC EEA3 cipher and EIA3 hash algorithms.
 
 Features
 --------
@@ -27,36 +27,46 @@ Limitations
 * Chained mbufs are not supported.
 * ZUC (EIA3) supported only if hash offset field is byte-aligned.
 * ZUC (EEA3) supported only if cipher length, cipher offset fields are byte-aligned.
-* ZUC PMD cannot be built as a shared library, due to limitations in
-  in the underlying library.
 
 
 Installation
 ------------
 
-To build DPDK with the ZUC_PMD the user is required to download
-the export controlled ``libsso_zuc`` library, by registering in
-`Intel Resource & Design Center <https://www.intel.com/content/www/us/en/design/resource-design-center.html>`_.
-Once approval has been granted, the user needs to search for
-*ZUC 128-EAA3 and 128-EIA3 3GPP cryptographic algorithms Software Library* to download the
-library or directly through this `link <https://cdrdv2.intel.com/v1/dl/getContent/575868>`_.
+To build DPDK with the ZUC_PMD the user is required to download the multi-buffer
+library from `here <https://github.com/01org/intel-ipsec-mb>`_
+and compile it on their user system before building DPDK.
+The latest version of the library supported by this PMD is v0.53, which
+can be downloaded from `<https://github.com/01org/intel-ipsec-mb/archive/v0.53.zip>`_.
+
 After downloading the library, the user needs to unpack and compile it
-on their system before building DPDK::
+on their system before building DPDK:
+
+.. code-block:: console
+
+    make
+    make install
+
+As a reference, the following table shows a mapping between the past DPDK versions
+and the external crypto libraries supported by them:
+
+.. _table_zuc_versions:
+
+.. table:: DPDK and external crypto library version compatibility
+
+   =============  ================================
+   DPDK version   Crypto library version
+   =============  ================================
+   16.11 - 19.11  LibSSO ZUC
+   20.02+         Multi-buffer library 0.53
+   =============  ================================
 
-   make
 
 Initialization
 --------------
 
 In order to enable this virtual crypto PMD, user must:
 
-* Export the environmental variable LIBSSO_ZUC_PATH with the path where
-  the library was extracted (zuc folder).
-
-* Export the environmental variable LD_LIBRARY_PATH with the path
-  where the built libsso library is (LIBSSO_ZUC_PATH/build).
-
-* Build the LIBSSO_ZUC library (explained in Installation section).
+* Build the multi buffer library (explained in Installation section).
 
 * Build DPDK as follows:
 
diff --git a/drivers/crypto/zuc/Makefile b/drivers/crypto/zuc/Makefile
index 68d84eebc..cc0d7943b 100644
--- a/drivers/crypto/zuc/Makefile
+++ b/drivers/crypto/zuc/Makefile
@@ -1,14 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2016 Intel Corporation
+# Copyright(c) 2016-2019 Intel Corporation
 
 include $(RTE_SDK)/mk/rte.vars.mk
 
-ifneq ($(MAKECMDGOALS),clean)
-ifeq ($(LIBSSO_ZUC_PATH),)
-$(error "Please define LIBSSO_ZUC_PATH environment variable")
-endif
-endif
-
 # library name
 LIB = librte_pmd_zuc.a
 
@@ -23,14 +17,26 @@ LIBABIVER := 1
 EXPORT_MAP := rte_pmd_zuc_version.map
 
 # external library dependencies
-CFLAGS += -I$(LIBSSO_ZUC_PATH)
-CFLAGS += -I$(LIBSSO_ZUC_PATH)/include
-CFLAGS += -I$(LIBSSO_ZUC_PATH)/build
-LDLIBS += -L$(LIBSSO_ZUC_PATH)/build -lsso_zuc
+LDLIBS += -lIPSec_MB
 LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring
 LDLIBS += -lrte_cryptodev
 LDLIBS += -lrte_bus_vdev
 
+IMB_HDR = $(shell echo '\#include <intel-ipsec-mb.h>' | \
+	$(CC) -E $(EXTRA_CFLAGS) - | grep 'intel-ipsec-mb.h' | \
+	head -n1 | cut -d'"' -f2)
+
+# Detect library version
+IMB_VERSION = $(shell grep -e "IMB_VERSION_STR" $(IMB_HDR) | cut -d'"' -f2)
+IMB_VERSION_NUM = $(shell grep -e "IMB_VERSION_NUM" $(IMB_HDR) | cut -d' ' -f3)
+
+ifeq ($(IMB_VERSION),)
+$(error "IPSec_MB version >= 0.53 is required")
+endif
+
+ifeq ($(shell expr $(IMB_VERSION_NUM) \< 0x3400), 1)
+$(error "IPSec_MB version >= 0.53 is required")
+endif
 # library source files
 SRCS-$(CONFIG_RTE_LIBRTE_PMD_ZUC) += rte_zuc_pmd.c
 SRCS-$(CONFIG_RTE_LIBRTE_PMD_ZUC) += rte_zuc_pmd_ops.c
diff --git a/drivers/crypto/zuc/meson.build b/drivers/crypto/zuc/meson.build
index b8ca7107e..f0fcd8246 100644
--- a/drivers/crypto/zuc/meson.build
+++ b/drivers/crypto/zuc/meson.build
@@ -1,11 +1,20 @@
 # SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2018 Intel Corporation
+# Copyright(c) 2018-2020 Intel Corporation
 
-lib = cc.find_library('libsso_zuc', required: false)
+IMB_required_ver = '0.53.0'
+lib = cc.find_library('IPSec_MB', required: false)
 if not lib.found()
 	build = false
 else
 	ext_deps += lib
+	# version comes with quotes, so we split based on " and take the middle
+	imb_ver = cc.get_define('IMB_VERSION_STR',
+		prefix : '#include<intel-ipsec-mb.h>').split('"')[1]
+
+	if (imb_ver == '') or (imb_ver.version_compare('<' + IMB_required_ver))
+		build = false
+	endif
+
 endif
 
 sources = files('rte_zuc_pmd.c', 'rte_zuc_pmd_ops.c')
diff --git a/drivers/crypto/zuc/rte_zuc_pmd.c b/drivers/crypto/zuc/rte_zuc_pmd.c
index 313f4590b..c880eea7c 100644
--- a/drivers/crypto/zuc/rte_zuc_pmd.c
+++ b/drivers/crypto/zuc/rte_zuc_pmd.c
@@ -11,7 +11,7 @@
 #include <rte_cpuflags.h>
 
 #include "rte_zuc_pmd_private.h"
-#define ZUC_MAX_BURST 4
+#define ZUC_MAX_BURST 16
 #define BYTE_LEN 8
 
 static uint8_t cryptodev_driver_id;
@@ -169,16 +169,17 @@ zuc_get_session(struct zuc_qp *qp, struct rte_crypto_op *op)
 
 /** Encrypt/decrypt mbufs. */
 static uint8_t
-process_zuc_cipher_op(struct rte_crypto_op **ops,
+process_zuc_cipher_op(struct zuc_qp *qp, struct rte_crypto_op **ops,
 		struct zuc_session **sessions,
 		uint8_t num_ops)
 {
 	unsigned i;
 	uint8_t processed_ops = 0;
-	uint8_t *src[ZUC_MAX_BURST], *dst[ZUC_MAX_BURST];
-	uint8_t *iv[ZUC_MAX_BURST];
+	const void *src[ZUC_MAX_BURST];
+	void *dst[ZUC_MAX_BURST];
+	const void *iv[ZUC_MAX_BURST];
 	uint32_t num_bytes[ZUC_MAX_BURST];
-	uint8_t *cipher_keys[ZUC_MAX_BURST];
+	const void *cipher_keys[ZUC_MAX_BURST];
 	struct zuc_session *sess;
 
 	for (i = 0; i < num_ops; i++) {
@@ -221,7 +222,8 @@ process_zuc_cipher_op(struct rte_crypto_op **ops,
 		processed_ops++;
 	}
 
-	sso_zuc_eea3_n_buffer(cipher_keys, iv, src, dst,
+	IMB_ZUC_EEA3_N_BUFFER(qp->mb_mgr, (const void **)cipher_keys,
+			(const void **)iv, (const void **)src, (void **)dst,
 			num_bytes, processed_ops);
 
 	return processed_ops;
@@ -261,7 +263,7 @@ process_zuc_hash_op(struct zuc_qp *qp, struct rte_crypto_op **ops,
 		if (sess->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) {
 			dst = (uint32_t *)qp->temp_digest;
 
-			sso_zuc_eia3_1_buffer(sess->pKey_hash,
+			IMB_ZUC_EIA3_1_BUFFER(qp->mb_mgr, sess->pKey_hash,
 					iv, src,
 					length_in_bits,	dst);
 			/* Verify digest. */
@@ -271,7 +273,7 @@ process_zuc_hash_op(struct zuc_qp *qp, struct rte_crypto_op **ops,
 		} else  {
 			dst = (uint32_t *)ops[i]->sym->auth.digest.data;
 
-			sso_zuc_eia3_1_buffer(sess->pKey_hash,
+			IMB_ZUC_EIA3_1_BUFFER(qp->mb_mgr, sess->pKey_hash,
 					iv, src,
 					length_in_bits, dst);
 		}
@@ -293,7 +295,7 @@ process_ops(struct rte_crypto_op **ops, enum zuc_operation op_type,
 
 	switch (op_type) {
 	case ZUC_OP_ONLY_CIPHER:
-		processed_ops = process_zuc_cipher_op(ops,
+		processed_ops = process_zuc_cipher_op(qp, ops,
 				sessions, num_ops);
 		break;
 	case ZUC_OP_ONLY_AUTH:
@@ -301,14 +303,14 @@ process_ops(struct rte_crypto_op **ops, enum zuc_operation op_type,
 				num_ops);
 		break;
 	case ZUC_OP_CIPHER_AUTH:
-		processed_ops = process_zuc_cipher_op(ops, sessions,
+		processed_ops = process_zuc_cipher_op(qp, ops, sessions,
 				num_ops);
 		process_zuc_hash_op(qp, ops, sessions, processed_ops);
 		break;
 	case ZUC_OP_AUTH_CIPHER:
 		processed_ops = process_zuc_hash_op(qp, ops, sessions,
 				num_ops);
-		process_zuc_cipher_op(ops, sessions, processed_ops);
+		process_zuc_cipher_op(qp, ops, sessions, processed_ops);
 		break;
 	default:
 		/* Operation not supported. */
@@ -455,8 +457,7 @@ cryptodev_zuc_create(const char *name,
 {
 	struct rte_cryptodev *dev;
 	struct zuc_private *internals;
-	uint64_t cpu_flags = RTE_CRYPTODEV_FF_CPU_SSE;
-
+	MB_MGR *mb_mgr;
 
 	dev = rte_cryptodev_pmd_create(name, &vdev->device, init_params);
 	if (dev == NULL) {
@@ -464,6 +465,27 @@ cryptodev_zuc_create(const char *name,
 		goto init_error;
 	}
 
+	dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
+			RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING;
+
+	mb_mgr = alloc_mb_mgr(0);
+	if (mb_mgr == NULL)
+		return -ENOMEM;
+
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F)) {
+		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX512;
+		init_mb_mgr_avx512(mb_mgr);
+	} else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2)) {
+		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX2;
+		init_mb_mgr_avx2(mb_mgr);
+	} else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX)) {
+		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX;
+		init_mb_mgr_avx(mb_mgr);
+	} else {
+		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_SSE;
+		init_mb_mgr_sse(mb_mgr);
+	}
+
 	dev->driver_id = cryptodev_driver_id;
 	dev->dev_ops = rte_zuc_pmd_ops;
 
@@ -471,11 +493,8 @@ cryptodev_zuc_create(const char *name,
 	dev->dequeue_burst = zuc_pmd_dequeue_burst;
 	dev->enqueue_burst = zuc_pmd_enqueue_burst;
 
-	dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
-			RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING |
-			cpu_flags;
-
 	internals = dev->data->dev_private;
+	internals->mb_mgr = mb_mgr;
 
 	internals->max_nb_queue_pairs = init_params->max_nb_queue_pairs;
 
@@ -516,6 +535,7 @@ cryptodev_zuc_remove(struct rte_vdev_device *vdev)
 
 	struct rte_cryptodev *cryptodev;
 	const char *name;
+	struct zuc_private *internals;
 
 	name = rte_vdev_device_name(vdev);
 	if (name == NULL)
@@ -525,6 +545,10 @@ cryptodev_zuc_remove(struct rte_vdev_device *vdev)
 	if (cryptodev == NULL)
 		return -ENODEV;
 
+	internals = cryptodev->data->dev_private;
+
+	free_mb_mgr(internals->mb_mgr);
+
 	return rte_cryptodev_pmd_destroy(cryptodev);
 }
 
diff --git a/drivers/crypto/zuc/rte_zuc_pmd_ops.c b/drivers/crypto/zuc/rte_zuc_pmd_ops.c
index 6da396542..14a831867 100644
--- a/drivers/crypto/zuc/rte_zuc_pmd_ops.c
+++ b/drivers/crypto/zuc/rte_zuc_pmd_ops.c
@@ -196,6 +196,7 @@ zuc_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 		int socket_id, struct rte_mempool *session_pool)
 {
 	struct zuc_qp *qp = NULL;
+	struct zuc_private *internals = dev->data->dev_private;
 
 	/* Free memory prior to re-allocation if needed. */
 	if (dev->data->queue_pairs[qp_id] != NULL)
@@ -218,6 +219,7 @@ zuc_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	if (qp->processed_ops == NULL)
 		goto qp_setup_cleanup;
 
+	qp->mb_mgr = internals->mb_mgr;
 	qp->sess_mp = session_pool;
 
 	memset(&qp->qp_stats, 0, sizeof(qp->qp_stats));
diff --git a/drivers/crypto/zuc/rte_zuc_pmd_private.h b/drivers/crypto/zuc/rte_zuc_pmd_private.h
index 5e5906ddb..e049df377 100644
--- a/drivers/crypto/zuc/rte_zuc_pmd_private.h
+++ b/drivers/crypto/zuc/rte_zuc_pmd_private.h
@@ -5,7 +5,7 @@
 #ifndef _RTE_ZUC_PMD_PRIVATE_H_
 #define _RTE_ZUC_PMD_PRIVATE_H_
 
-#include <sso_zuc.h>
+#include <intel-ipsec-mb.h>
 
 #define CRYPTODEV_NAME_ZUC_PMD		crypto_zuc
 /**< KASUMI PMD device name */
@@ -24,6 +24,8 @@ int zuc_logtype_driver;
 struct zuc_private {
 	unsigned max_nb_queue_pairs;
 	/**< Max number of queue pairs supported by device */
+	MB_MGR *mb_mgr;
+	/**< Multi-buffer instance */
 };
 
 /** ZUC buffer queue pair */
@@ -43,6 +45,8 @@ struct zuc_qp {
 	 * by the driver when verifying a digest provided
 	 * by the user (using authentication verify operation)
 	 */
+	MB_MGR *mb_mgr;
+	/**< Multi-buffer instance */
 } __rte_cache_aligned;
 
 enum zuc_operation {
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 5699d979d..7c04ee490 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -233,7 +233,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_SNOW3G)      += -L$(LIBSSO_SNOW3G_PATH)/build -l
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_KASUMI)      += -lrte_pmd_kasumi
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_KASUMI)      += -L$(LIBSSO_KASUMI_PATH)/build -lsso_kasumi
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_ZUC)         += -lrte_pmd_zuc
-_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_ZUC)         += -L$(LIBSSO_ZUC_PATH)/build -lsso_zuc
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_ZUC)         += -lIPSec_MB
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_ARMV8_CRYPTO)    += -lrte_pmd_armv8
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_ARMV8_CRYPTO)    += -L$(ARMV8_CRYPTO_LIB_PATH) -larmv8_crypto
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_MVSAM_CRYPTO) += -L$(LIBMUSDK_PATH)/lib -lrte_pmd_mvsam_crypto -lmusdk
-- 
2.24.1


^ permalink raw reply	[flat|nested] 8+ messages in thread

* [dpdk-dev] [RFC PATCH 2/5] crypto/snow3g: use IPSec MB library v0.53
  2020-03-05 15:34 [dpdk-dev] [RFC PATCH 0/5] Support Intel IPSec MB v0.53 in DPDK 18.11 Pablo de Lara
  2020-03-05 15:34 ` [dpdk-dev] [RFC PATCH 1/5] crypto/zuc: use IPSec MB library v0.53 Pablo de Lara
@ 2020-03-05 15:34 ` Pablo de Lara
  2020-03-05 15:34 ` [dpdk-dev] [RFC PATCH 3/5] crypto/kasumi: " Pablo de Lara
                   ` (3 subsequent siblings)
  5 siblings, 0 replies; 8+ messages in thread
From: Pablo de Lara @ 2020-03-05 15:34 UTC (permalink / raw)
  To: dev; +Cc: Pablo de Lara

Link against Intel IPSec Multi-buffer library, which
added support for SNOW3G-UEA2 and SNOW3G-UIA2 from version v0.53,
moving from libSSO SNOW3G library.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
---
 devtools/test-build.sh                        |  4 +-
 doc/guides/cryptodevs/snow3g.rst              | 58 ++++++++------
 drivers/crypto/snow3g/Makefile                | 29 ++++---
 drivers/crypto/snow3g/meson.build             | 21 +++++
 drivers/crypto/snow3g/rte_snow3g_pmd.c        | 79 ++++++++++++-------
 drivers/crypto/snow3g/rte_snow3g_pmd_ops.c    |  8 +-
 .../crypto/snow3g/rte_snow3g_pmd_private.h    | 12 ++-
 mk/rte.app.mk                                 |  2 +-
 8 files changed, 139 insertions(+), 74 deletions(-)
 create mode 100644 drivers/crypto/snow3g/meson.build

diff --git a/devtools/test-build.sh b/devtools/test-build.sh
index 911f77e01..0c62c3950 100755
--- a/devtools/test-build.sh
+++ b/devtools/test-build.sh
@@ -23,7 +23,6 @@ default_path=$PATH
 # - DPDK_NOTIFY (notify-send)
 # - FLEXRAN_SDK
 # - LIBMUSDK_PATH
-# - LIBSSO_SNOW3G_PATH
 # - LIBSSO_KASUMI_PATH
 . $(dirname $(readlink -e $0))/load-devel-config
 
@@ -108,7 +107,6 @@ reset_env ()
 	unset ARMV8_CRYPTO_LIB_PATH
 	unset FLEXRAN_SDK
 	unset LIBMUSDK_PATH
-	unset LIBSSO_SNOW3G_PATH
 	unset LIBSSO_KASUMI_PATH
 	unset PQOS_INSTALL_PATH
 }
@@ -165,7 +163,7 @@ config () # <directory> <target> <options>
 		sed -ri      's,(PMD_AESNI_GCM=)n,\1y,' $1/.config
 		test "$DPDK_DEP_IPSEC_MB" != y || \
 		sed -ri            's,(PMD_ZUC=)n,\1y,' $1/.config
-		test -z "$LIBSSO_SNOW3G_PATH" || \
+		test "$DPDK_DEP_IPSEC_MB" != y || \
 		sed -ri         's,(PMD_SNOW3G=)n,\1y,' $1/.config
 		test -z "$LIBSSO_KASUMI_PATH" || \
 		sed -ri         's,(PMD_KASUMI=)n,\1y,' $1/.config
diff --git a/doc/guides/cryptodevs/snow3g.rst b/doc/guides/cryptodevs/snow3g.rst
index 7cba712c1..d45bcad67 100644
--- a/doc/guides/cryptodevs/snow3g.rst
+++ b/doc/guides/cryptodevs/snow3g.rst
@@ -1,12 +1,12 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
-    Copyright(c) 2016 Intel Corporation.
+    Copyright(c) 2016-2019 Intel Corporation.
 
 SNOW 3G Crypto Poll Mode Driver
 ===============================
 
-The SNOW 3G PMD (**librte_pmd_snow3g**) provides poll mode crypto driver
-support for utilizing Intel Libsso library, which implements F8 and F9 functions
-for SNOW 3G UEA2 cipher and UIA2 hash algorithms.
+The SNOW3G PMD (**librte_snow3g_zuc**) provides poll mode crypto driver support for
+utilizing `Intel IPSec Multi-buffer library <https://github.com/01org/intel-ipsec-mb>`_
+which implements F8 and F8 functions for SNOW 3G UEA2 cipher and UIA2 hash algorithms.
 
 Features
 --------
@@ -32,26 +32,33 @@ Limitations
 Installation
 ------------
 
-To build DPDK with the SNOW3G_PMD the user is required to download
-the export controlled ``libsso_snow3g`` library, by registering in
-`Intel Resource & Design Center <https://www.intel.com/content/www/us/en/design/resource-design-center.html>`_.
-Once approval has been granted, the user needs to search for
-*Snow3G F8 F9 3GPP cryptographic algorithms Software Library* to download the
-library or directly through this `link <https://cdrdv2.intel.com/v1/dl/getContent/575867>`_.
+To build DPDK with the SNOW3G_PMD the user is required to download the multi-buffer
+library from `here <https://github.com/01org/intel-ipsec-mb>`_
+and compile it on their user system before building DPDK.
+The latest version of the library supported by this PMD is v0.53, which
+can be downloaded from `<https://github.com/01org/intel-ipsec-mb/archive/v0.53.zip>`_.
+
 After downloading the library, the user needs to unpack and compile it
-on their system before building DPDK::
+on their system before building DPDK:
+
+.. code-block:: console
+
+    make
+    make install
 
-   make snow3G
+As a reference, the following table shows a mapping between the past DPDK versions
+and the external crypto libraries supported by them:
 
-**Note**: When encrypting with SNOW3G UEA2, by default the library
-encrypts blocks of 4 bytes, regardless the number of bytes to
-be encrypted provided (which leads to a possible buffer overflow).
-To avoid this situation, it is necessary not to pass
-3GPP_SAFE_BUFFERS as a compilation flag.
-For this, in the Makefile of the library, make sure that this flag
-is commented out.::
+.. _table_zuc_versions:
 
-  #EXTRA_CFLAGS  += -D_3GPP_SAFE_BUFFERS
+.. table:: DPDK and external crypto library version compatibility
+
+   =============  ================================
+   DPDK version   Crypto library version
+   =============  ================================
+   16.04 - 19.11  LibSSO SNOW3G
+   20.02+         Multi-buffer library 0.53
+   =============  ================================
 
 
 Initialization
@@ -59,12 +66,15 @@ Initialization
 
 In order to enable this virtual crypto PMD, user must:
 
-* Export the environmental variable LIBSSO_SNOW3G_PATH with the path where
-  the library was extracted (snow3g folder).
+* Build the multi buffer library (explained in Installation section).
+
+* Build DPDK as follows:
 
-* Build the LIBSSO_SNOW3G library (explained in Installation section).
+.. code-block:: console
 
-* Set CONFIG_RTE_LIBRTE_PMD_SNOW3G=y in config/common_base.
+	make config T=x86_64-native-linux-gcc
+	sed -i 's,\(CONFIG_RTE_LIBRTE_PMD_SNOW3G\)=n,\1=y,' build/.config
+	make
 
 To use the PMD in an application, user must:
 
diff --git a/drivers/crypto/snow3g/Makefile b/drivers/crypto/snow3g/Makefile
index ee5027d0c..9c2f59d36 100644
--- a/drivers/crypto/snow3g/Makefile
+++ b/drivers/crypto/snow3g/Makefile
@@ -1,14 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2016 Intel Corporation
+# Copyright(c) 2016-2019 Intel Corporation
 
 include $(RTE_SDK)/mk/rte.vars.mk
 
-ifneq ($(MAKECMDGOALS),clean)
-ifeq ($(LIBSSO_SNOW3G_PATH),)
-$(error "Please define LIBSSO_SNOW3G_PATH environment variable")
-endif
-endif
-
 # library name
 LIB = librte_pmd_snow3g.a
 
@@ -23,14 +17,27 @@ LIBABIVER := 1
 EXPORT_MAP := rte_pmd_snow3g_version.map
 
 # external library dependencies
-CFLAGS += -I$(LIBSSO_SNOW3G_PATH)
-CFLAGS += -I$(LIBSSO_SNOW3G_PATH)/include
-CFLAGS += -I$(LIBSSO_SNOW3G_PATH)/build
-LDLIBS += -L$(LIBSSO_SNOW3G_PATH)/build -lsso_snow3g
+LDLIBS += -lIPSec_MB
 LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring
 LDLIBS += -lrte_cryptodev
 LDLIBS += -lrte_bus_vdev
 
+IMB_HDR = $(shell echo '\#include <intel-ipsec-mb.h>' | \
+	$(CC) -E $(EXTRA_CFLAGS) - | grep 'intel-ipsec-mb.h' | \
+	head -n1 | cut -d'"' -f2)
+
+# Detect library version
+IMB_VERSION = $(shell grep -e "IMB_VERSION_STR" $(IMB_HDR) | cut -d'"' -f2)
+IMB_VERSION_NUM = $(shell grep -e "IMB_VERSION_NUM" $(IMB_HDR) | cut -d' ' -f3)
+
+ifeq ($(IMB_VERSION),)
+$(error "IPSec_MB version >= 0.53 is required")
+endif
+
+ifeq ($(shell expr $(IMB_VERSION_NUM) \< 0x3400), 1)
+$(error "IPSec_MB version >= 0.53 is required")
+endif
+
 # library source files
 SRCS-$(CONFIG_RTE_LIBRTE_PMD_SNOW3G) += rte_snow3g_pmd.c
 SRCS-$(CONFIG_RTE_LIBRTE_PMD_SNOW3G) += rte_snow3g_pmd_ops.c
diff --git a/drivers/crypto/snow3g/meson.build b/drivers/crypto/snow3g/meson.build
new file mode 100644
index 000000000..e04138e92
--- /dev/null
+++ b/drivers/crypto/snow3g/meson.build
@@ -0,0 +1,21 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018-2020 Intel Corporation
+
+IMB_required_ver = '0.53.0'
+lib = cc.find_library('IPSec_MB', required: false)
+if not lib.found()
+	build = false
+else
+	ext_deps += lib
+	# version comes with quotes, so we split based on " and take the middle
+	imb_ver = cc.get_define('IMB_VERSION_STR',
+		prefix : '#include<intel-ipsec-mb.h>').split('"')[1]
+
+	if (imb_ver == '') or (imb_ver.version_compare('<' + IMB_required_ver))
+		build = false
+	endif
+
+endif
+
+sources = files('rte_snow3g_pmd.c', 'rte_snow3g_pmd_ops.c')
+deps += ['bus_vdev']
diff --git a/drivers/crypto/snow3g/rte_snow3g_pmd.c b/drivers/crypto/snow3g/rte_snow3g_pmd.c
index a17536b77..5abf7e090 100644
--- a/drivers/crypto/snow3g/rte_snow3g_pmd.c
+++ b/drivers/crypto/snow3g/rte_snow3g_pmd.c
@@ -53,7 +53,7 @@ snow3g_get_mode(const struct rte_crypto_sym_xform *xform)
 
 /** Parse crypto xform chain and set private session parameters. */
 int
-snow3g_set_session_parameters(struct snow3g_session *sess,
+snow3g_set_session_parameters(MB_MGR *mgr, struct snow3g_session *sess,
 		const struct rte_crypto_sym_xform *xform)
 {
 	const struct rte_crypto_sym_xform *auth_xform = NULL;
@@ -95,8 +95,8 @@ snow3g_set_session_parameters(struct snow3g_session *sess,
 		sess->cipher_iv_offset = cipher_xform->cipher.iv.offset;
 
 		/* Initialize key */
-		sso_snow3g_init_key_sched(cipher_xform->cipher.key.data,
-				&sess->pKeySched_cipher);
+		IMB_SNOW3G_INIT_KEY_SCHED(mgr, cipher_xform->cipher.key.data,
+					&sess->pKeySched_cipher);
 	}
 
 	if (auth_xform) {
@@ -118,11 +118,10 @@ snow3g_set_session_parameters(struct snow3g_session *sess,
 		sess->auth_iv_offset = auth_xform->auth.iv.offset;
 
 		/* Initialize key */
-		sso_snow3g_init_key_sched(auth_xform->auth.key.data,
-				&sess->pKeySched_hash);
+		IMB_SNOW3G_INIT_KEY_SCHED(mgr, auth_xform->auth.key.data,
+					&sess->pKeySched_hash);
 	}
 
-
 	sess->op = mode;
 
 	return 0;
@@ -152,7 +151,7 @@ snow3g_get_session(struct snow3g_qp *qp, struct rte_crypto_op *op)
 
 		sess = (struct snow3g_session *)_sess_private_data;
 
-		if (unlikely(snow3g_set_session_parameters(sess,
+		if (unlikely(snow3g_set_session_parameters(qp->mgr, sess,
 				op->sym->xform) != 0)) {
 			rte_mempool_put(qp->sess_mp, _sess);
 			rte_mempool_put(qp->sess_mp, _sess_private_data);
@@ -172,14 +171,15 @@ snow3g_get_session(struct snow3g_qp *qp, struct rte_crypto_op *op)
 
 /** Encrypt/decrypt mbufs with same cipher key. */
 static uint8_t
-process_snow3g_cipher_op(struct rte_crypto_op **ops,
+process_snow3g_cipher_op(struct snow3g_qp *qp, struct rte_crypto_op **ops,
 		struct snow3g_session *session,
 		uint8_t num_ops)
 {
 	unsigned i;
 	uint8_t processed_ops = 0;
-	uint8_t *src[SNOW3G_MAX_BURST], *dst[SNOW3G_MAX_BURST];
-	uint8_t *iv[SNOW3G_MAX_BURST];
+	const void *src[SNOW3G_MAX_BURST];
+	void *dst[SNOW3G_MAX_BURST];
+	const void *iv[SNOW3G_MAX_BURST];
 	uint32_t num_bytes[SNOW3G_MAX_BURST];
 
 	for (i = 0; i < num_ops; i++) {
@@ -197,15 +197,16 @@ process_snow3g_cipher_op(struct rte_crypto_op **ops,
 		processed_ops++;
 	}
 
-	sso_snow3g_f8_n_buffer(&session->pKeySched_cipher, iv, src, dst,
-			num_bytes, processed_ops);
+	IMB_SNOW3G_F8_N_BUFFER(qp->mgr, &session->pKeySched_cipher, iv,
+			src, dst, num_bytes, processed_ops);
 
 	return processed_ops;
 }
 
 /** Encrypt/decrypt mbuf (bit level function). */
 static uint8_t
-process_snow3g_cipher_op_bit(struct rte_crypto_op *op,
+process_snow3g_cipher_op_bit(struct snow3g_qp *qp,
+		struct rte_crypto_op *op,
 		struct snow3g_session *session)
 {
 	uint8_t *src, *dst;
@@ -224,7 +225,7 @@ process_snow3g_cipher_op_bit(struct rte_crypto_op *op,
 				session->cipher_iv_offset);
 	length_in_bits = op->sym->cipher.data.length;
 
-	sso_snow3g_f8_1_buffer_bit(&session->pKeySched_cipher, iv,
+	IMB_SNOW3G_F8_1_BUFFER_BIT(qp->mgr, &session->pKeySched_cipher, iv,
 			src, dst, length_in_bits, offset_in_bits);
 
 	return 1;
@@ -260,9 +261,9 @@ process_snow3g_hash_op(struct snow3g_qp *qp, struct rte_crypto_op **ops,
 		if (session->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) {
 			dst = qp->temp_digest;
 
-			sso_snow3g_f9_1_buffer(&session->pKeySched_hash,
-					iv, src,
-					length_in_bits,	dst);
+			IMB_SNOW3G_F9_1_BUFFER(qp->mgr,
+					&session->pKeySched_hash,
+					iv, src, length_in_bits, dst);
 			/* Verify digest. */
 			if (memcmp(dst, ops[i]->sym->auth.digest.data,
 					SNOW3G_DIGEST_LENGTH) != 0)
@@ -270,9 +271,9 @@ process_snow3g_hash_op(struct snow3g_qp *qp, struct rte_crypto_op **ops,
 		} else  {
 			dst = ops[i]->sym->auth.digest.data;
 
-			sso_snow3g_f9_1_buffer(&session->pKeySched_hash,
-					iv, src,
-					length_in_bits, dst);
+			IMB_SNOW3G_F9_1_BUFFER(qp->mgr,
+					&session->pKeySched_hash,
+					iv, src, length_in_bits, dst);
 		}
 		processed_ops++;
 	}
@@ -306,7 +307,7 @@ process_ops(struct rte_crypto_op **ops, struct snow3g_session *session,
 
 	switch (session->op) {
 	case SNOW3G_OP_ONLY_CIPHER:
-		processed_ops = process_snow3g_cipher_op(ops,
+		processed_ops = process_snow3g_cipher_op(qp, ops,
 				session, num_ops);
 		break;
 	case SNOW3G_OP_ONLY_AUTH:
@@ -314,14 +315,14 @@ process_ops(struct rte_crypto_op **ops, struct snow3g_session *session,
 				num_ops);
 		break;
 	case SNOW3G_OP_CIPHER_AUTH:
-		processed_ops = process_snow3g_cipher_op(ops, session,
+		processed_ops = process_snow3g_cipher_op(qp, ops, session,
 				num_ops);
 		process_snow3g_hash_op(qp, ops, session, processed_ops);
 		break;
 	case SNOW3G_OP_AUTH_CIPHER:
 		processed_ops = process_snow3g_hash_op(qp, ops, session,
 				num_ops);
-		process_snow3g_cipher_op(ops, session, processed_ops);
+		process_snow3g_cipher_op(qp, ops, session, processed_ops);
 		break;
 	default:
 		/* Operation not supported. */
@@ -363,21 +364,21 @@ process_op_bit(struct rte_crypto_op *op, struct snow3g_session *session,
 
 	switch (session->op) {
 	case SNOW3G_OP_ONLY_CIPHER:
-		processed_op = process_snow3g_cipher_op_bit(op,
+		processed_op = process_snow3g_cipher_op_bit(qp, op,
 				session);
 		break;
 	case SNOW3G_OP_ONLY_AUTH:
 		processed_op = process_snow3g_hash_op(qp, &op, session, 1);
 		break;
 	case SNOW3G_OP_CIPHER_AUTH:
-		processed_op = process_snow3g_cipher_op_bit(op, session);
+		processed_op = process_snow3g_cipher_op_bit(qp, op, session);
 		if (processed_op == 1)
 			process_snow3g_hash_op(qp, &op, session, 1);
 		break;
 	case SNOW3G_OP_AUTH_CIPHER:
 		processed_op = process_snow3g_hash_op(qp, &op, session, 1);
 		if (processed_op == 1)
-			process_snow3g_cipher_op_bit(op, session);
+			process_snow3g_cipher_op_bit(qp, op, session);
 		break;
 	default:
 		/* Operation not supported. */
@@ -533,7 +534,7 @@ cryptodev_snow3g_create(const char *name,
 {
 	struct rte_cryptodev *dev;
 	struct snow3g_private *internals;
-	uint64_t cpu_flags = RTE_CRYPTODEV_FF_CPU_SSE;
+	MB_MGR *mgr;
 
 	dev = rte_cryptodev_pmd_create(name, &vdev->device, init_params);
 	if (dev == NULL) {
@@ -549,10 +550,25 @@ cryptodev_snow3g_create(const char *name,
 	dev->enqueue_burst = snow3g_pmd_enqueue_burst;
 
 	dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
-			RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING |
-			cpu_flags;
+			RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING;
+
+	mgr = alloc_mb_mgr(0);
+	if (mgr == NULL)
+		return -ENOMEM;
+
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2)) {
+		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX2;
+		init_mb_mgr_avx2(mgr);
+	} else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX)) {
+		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX;
+		init_mb_mgr_avx(mgr);
+	} else {
+		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_SSE;
+		init_mb_mgr_sse(mgr);
+	}
 
 	internals = dev->data->dev_private;
+	internals->mgr = mgr;
 
 	internals->max_nb_queue_pairs = init_params->max_nb_queue_pairs;
 
@@ -592,6 +608,7 @@ cryptodev_snow3g_remove(struct rte_vdev_device *vdev)
 {
 	struct rte_cryptodev *cryptodev;
 	const char *name;
+	struct snow3g_private *internals;
 
 	name = rte_vdev_device_name(vdev);
 	if (name == NULL)
@@ -601,6 +618,10 @@ cryptodev_snow3g_remove(struct rte_vdev_device *vdev)
 	if (cryptodev == NULL)
 		return -ENODEV;
 
+	internals = cryptodev->data->dev_private;
+
+	free_mb_mgr(internals->mgr);
+
 	return rte_cryptodev_pmd_destroy(cryptodev);
 }
 
diff --git a/drivers/crypto/snow3g/rte_snow3g_pmd_ops.c b/drivers/crypto/snow3g/rte_snow3g_pmd_ops.c
index cfbc9522a..414029b5a 100644
--- a/drivers/crypto/snow3g/rte_snow3g_pmd_ops.c
+++ b/drivers/crypto/snow3g/rte_snow3g_pmd_ops.c
@@ -196,6 +196,7 @@ snow3g_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 		int socket_id, struct rte_mempool *session_pool)
 {
 	struct snow3g_qp *qp = NULL;
+	struct snow3g_private *internals = dev->data->dev_private;
 
 	/* Free memory prior to re-allocation if needed. */
 	if (dev->data->queue_pairs[qp_id] != NULL)
@@ -218,6 +219,7 @@ snow3g_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	if (qp->processed_ops == NULL)
 		goto qp_setup_cleanup;
 
+	qp->mgr = internals->mgr;
 	qp->sess_mp = session_pool;
 
 	memset(&qp->qp_stats, 0, sizeof(qp->qp_stats));
@@ -247,13 +249,14 @@ snow3g_pmd_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
 
 /** Configure a SNOW 3G session from a crypto xform chain */
 static int
-snow3g_pmd_sym_session_configure(struct rte_cryptodev *dev __rte_unused,
+snow3g_pmd_sym_session_configure(struct rte_cryptodev *dev,
 		struct rte_crypto_sym_xform *xform,
 		struct rte_cryptodev_sym_session *sess,
 		struct rte_mempool *mempool)
 {
 	void *sess_private_data;
 	int ret;
+	struct snow3g_private *internals = dev->data->dev_private;
 
 	if (unlikely(sess == NULL)) {
 		SNOW3G_LOG(ERR, "invalid session struct");
@@ -266,7 +269,8 @@ snow3g_pmd_sym_session_configure(struct rte_cryptodev *dev __rte_unused,
 		return -ENOMEM;
 	}
 
-	ret = snow3g_set_session_parameters(sess_private_data, xform);
+	ret = snow3g_set_session_parameters(internals->mgr,
+					sess_private_data, xform);
 	if (ret != 0) {
 		SNOW3G_LOG(ERR, "failed configure session parameters");
 
diff --git a/drivers/crypto/snow3g/rte_snow3g_pmd_private.h b/drivers/crypto/snow3g/rte_snow3g_pmd_private.h
index b7807b621..670140144 100644
--- a/drivers/crypto/snow3g/rte_snow3g_pmd_private.h
+++ b/drivers/crypto/snow3g/rte_snow3g_pmd_private.h
@@ -5,7 +5,7 @@
 #ifndef _RTE_SNOW3G_PMD_PRIVATE_H_
 #define _RTE_SNOW3G_PMD_PRIVATE_H_
 
-#include <sso_snow3g.h>
+#include <intel-ipsec-mb.h>
 
 #define CRYPTODEV_NAME_SNOW3G_PMD	crypto_snow3g
 /**< SNOW 3G PMD device name */
@@ -24,6 +24,8 @@ int snow3g_logtype_driver;
 struct snow3g_private {
 	unsigned max_nb_queue_pairs;
 	/**< Max number of queue pairs supported by device */
+	MB_MGR *mgr;
+	/**< Multi-buffer instance */
 };
 
 /** SNOW 3G buffer queue pair */
@@ -43,6 +45,8 @@ struct snow3g_qp {
 	 * by the driver when verifying a digest provided
 	 * by the user (using authentication verify operation)
 	 */
+	MB_MGR *mgr;
+	/**< Multi-buffer instance */
 } __rte_cache_aligned;
 
 enum snow3g_operation {
@@ -57,15 +61,15 @@ enum snow3g_operation {
 struct snow3g_session {
 	enum snow3g_operation op;
 	enum rte_crypto_auth_operation auth_op;
-	sso_snow3g_key_schedule_t pKeySched_cipher;
-	sso_snow3g_key_schedule_t pKeySched_hash;
+	snow3g_key_schedule_t pKeySched_cipher;
+	snow3g_key_schedule_t pKeySched_hash;
 	uint16_t cipher_iv_offset;
 	uint16_t auth_iv_offset;
 } __rte_cache_aligned;
 
 
 extern int
-snow3g_set_session_parameters(struct snow3g_session *sess,
+snow3g_set_session_parameters(MB_MGR *mgr, struct snow3g_session *sess,
 		const struct rte_crypto_sym_xform *xform);
 
 
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 7c04ee490..b33603ef5 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -229,7 +229,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_PMD_QAT),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_QAT_SYM)     += -lrte_pmd_qat -lcrypto
 endif # CONFIG_RTE_LIBRTE_PMD_QAT
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_SNOW3G)      += -lrte_pmd_snow3g
-_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_SNOW3G)      += -L$(LIBSSO_SNOW3G_PATH)/build -lsso_snow3g
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_SNOW3G)      += -lIPSec_MB
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_KASUMI)      += -lrte_pmd_kasumi
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_KASUMI)      += -L$(LIBSSO_KASUMI_PATH)/build -lsso_kasumi
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_ZUC)         += -lrte_pmd_zuc
-- 
2.24.1


^ permalink raw reply	[flat|nested] 8+ messages in thread

* [dpdk-dev] [RFC PATCH 3/5] crypto/kasumi: use IPSec MB library v0.53
  2020-03-05 15:34 [dpdk-dev] [RFC PATCH 0/5] Support Intel IPSec MB v0.53 in DPDK 18.11 Pablo de Lara
  2020-03-05 15:34 ` [dpdk-dev] [RFC PATCH 1/5] crypto/zuc: use IPSec MB library v0.53 Pablo de Lara
  2020-03-05 15:34 ` [dpdk-dev] [RFC PATCH 2/5] crypto/snow3g: " Pablo de Lara
@ 2020-03-05 15:34 ` Pablo de Lara
  2020-03-05 15:34 ` [dpdk-dev] [RFC PATCH 4/5] crypto/aesni_mb: support " Pablo de Lara
                   ` (2 subsequent siblings)
  5 siblings, 0 replies; 8+ messages in thread
From: Pablo de Lara @ 2020-03-05 15:34 UTC (permalink / raw)
  To: dev; +Cc: Pablo de Lara

Link against Intel IPSec Multi-buffer library, which
added support for KASUMI-F8 and KASUMI-F9 from version v0.53,
moving from libSSO KASUMI library.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
---
 devtools/test-build.sh                        |  4 +-
 doc/guides/cryptodevs/kasumi.rst              | 62 ++++++++-------
 drivers/crypto/kasumi/Makefile                | 26 +++---
 drivers/crypto/kasumi/meson.build             | 11 ++-
 drivers/crypto/kasumi/rte_kasumi_pmd.c        | 79 +++++++++++--------
 drivers/crypto/kasumi/rte_kasumi_pmd_ops.c    |  8 +-
 .../crypto/kasumi/rte_kasumi_pmd_private.h    | 12 ++-
 mk/rte.app.mk                                 |  2 +-
 8 files changed, 120 insertions(+), 84 deletions(-)

diff --git a/devtools/test-build.sh b/devtools/test-build.sh
index 0c62c3950..73b4bdd1a 100755
--- a/devtools/test-build.sh
+++ b/devtools/test-build.sh
@@ -23,7 +23,6 @@ default_path=$PATH
 # - DPDK_NOTIFY (notify-send)
 # - FLEXRAN_SDK
 # - LIBMUSDK_PATH
-# - LIBSSO_KASUMI_PATH
 . $(dirname $(readlink -e $0))/load-devel-config
 
 print_usage () {
@@ -107,7 +106,6 @@ reset_env ()
 	unset ARMV8_CRYPTO_LIB_PATH
 	unset FLEXRAN_SDK
 	unset LIBMUSDK_PATH
-	unset LIBSSO_KASUMI_PATH
 	unset PQOS_INSTALL_PATH
 }
 
@@ -165,7 +163,7 @@ config () # <directory> <target> <options>
 		sed -ri            's,(PMD_ZUC=)n,\1y,' $1/.config
 		test "$DPDK_DEP_IPSEC_MB" != y || \
 		sed -ri         's,(PMD_SNOW3G=)n,\1y,' $1/.config
-		test -z "$LIBSSO_KASUMI_PATH" || \
+		test "$DPDK_DEP_IPSEC_MB" != y || \
 		sed -ri         's,(PMD_KASUMI=)n,\1y,' $1/.config
 		test "$DPDK_DEP_SSL" != y || \
 		sed -ri            's,(PMD_CCP=)n,\1y,' $1/.config
diff --git a/doc/guides/cryptodevs/kasumi.rst b/doc/guides/cryptodevs/kasumi.rst
index 2265eee4e..6c86fe264 100644
--- a/doc/guides/cryptodevs/kasumi.rst
+++ b/doc/guides/cryptodevs/kasumi.rst
@@ -1,12 +1,12 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
-    Copyright(c) 2016 Intel Corporation.
+    Copyright(c) 2016-2019 Intel Corporation.
 
 KASUMI Crypto Poll Mode Driver
 ===============================
 
-The KASUMI PMD (**librte_pmd_kasumi**) provides poll mode crypto driver
-support for utilizing Intel Libsso library, which implements F8 and F9 functions
-for KASUMI UEA1 cipher and UIA1 hash algorithms.
+The KASUMI PMD (**librte_pmd_kasumi**) provides poll mode crypto driver support for
+utilizing `Intel IPSec Multi-buffer library <https://github.com/01org/intel-ipsec-mb>`_
+which implements F8 and F9 functions for KASUMI UEA1 cipher and UIA1 hash algorithms.
 
 Features
 --------
@@ -33,33 +33,33 @@ Limitations
 Installation
 ------------
 
-To build DPDK with the KASUMI_PMD the user is required to download
-the export controlled ``libsso_kasumi`` library, by registering in
-`Intel Resource & Design Center <https://www.intel.com/content/www/us/en/design/resource-design-center.html>`_.
-Once approval has been granted, the user needs to search for
-*Kasumi F8 F9 3GPP cryptographic algorithms Software Library* to download the
-library or directly through this `link <https://cdrdv2.intel.com/v1/dl/getContent/575866>`_.
+To build DPDK with the KASUMI_PMD the user is required to download the multi-buffer
+library from `here <https://github.com/01org/intel-ipsec-mb>`_
+and compile it on their user system before building DPDK.
+The latest version of the library supported by this PMD is v0.53, which
+can be downloaded from `<https://github.com/01org/intel-ipsec-mb/archive/v0.53.zip>`_.
+
 After downloading the library, the user needs to unpack and compile it
-on their system before building DPDK::
+on their system before building DPDK:
+
+.. code-block:: console
 
-   make
+    make
+    make install
 
-**Note**: When encrypting with KASUMI F8, by default the library
-encrypts full blocks of 8 bytes, regardless the number of bytes to
-be encrypted provided (which leads to a possible buffer overflow).
-To avoid this situation, it is necessary not to pass
-3GPP_SAFE_BUFFERS as a compilation flag.
-Also, this is required when using chained operations
-(cipher-then-auth/auth-then-cipher).
-For this, in the Makefile of the library, make sure that this flag
-is commented out::
+As a reference, the following table shows a mapping between the past DPDK versions
+and the external crypto libraries supported by them:
 
-  #EXTRA_CFLAGS  += -D_3GPP_SAFE_BUFFERS
+.. _table_kasumi_versions:
 
-**Note**: To build the PMD as a shared library, the libsso_kasumi
-library must be built as follows::
+.. table:: DPDK and external crypto library version compatibility
 
-  make KASUMI_CFLAGS=-DKASUMI_C
+   =============  ================================
+   DPDK version   Crypto library version
+   =============  ================================
+   16.11 - 19.11  LibSSO KASUMI
+   20.02+         Multi-buffer library 0.53
+   =============  ================================
 
 
 Initialization
@@ -67,12 +67,16 @@ Initialization
 
 In order to enable this virtual crypto PMD, user must:
 
-* Export the environmental variable LIBSSO_KASUMI_PATH with the path where
-  the library was extracted (kasumi folder).
+* Build the multi buffer library (explained in Installation section).
+
+* Build DPDK as follows:
+
+.. code-block:: console
 
-* Build the LIBSSO library (explained in Installation section).
+	make config T=x86_64-native-linux-gcc
+	sed -i 's,\(CONFIG_RTE_LIBRTE_PMD_KASUMI\)=n,\1=y,' build/.config
+	make
 
-* Set CONFIG_RTE_LIBRTE_PMD_KASUMI=y in config/common_base.
 
 To use the PMD in an application, user must:
 
diff --git a/drivers/crypto/kasumi/Makefile b/drivers/crypto/kasumi/Makefile
index cafe94986..51f31d0aa 100644
--- a/drivers/crypto/kasumi/Makefile
+++ b/drivers/crypto/kasumi/Makefile
@@ -3,12 +3,6 @@
 
 include $(RTE_SDK)/mk/rte.vars.mk
 
-ifneq ($(MAKECMDGOALS),clean)
-ifeq ($(LIBSSO_KASUMI_PATH),)
-$(error "Please define LIBSSO_KASUMI_PATH environment variable")
-endif
-endif
-
 # library name
 LIB = librte_pmd_kasumi.a
 
@@ -23,14 +17,26 @@ LIBABIVER := 1
 EXPORT_MAP := rte_pmd_kasumi_version.map
 
 # external library dependencies
-CFLAGS += -I$(LIBSSO_KASUMI_PATH)
-CFLAGS += -I$(LIBSSO_KASUMI_PATH)/include
-CFLAGS += -I$(LIBSSO_KASUMI_PATH)/build
-LDLIBS += -L$(LIBSSO_KASUMI_PATH)/build -lsso_kasumi
+LDLIBS += -lIPSec_MB
 LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring
 LDLIBS += -lrte_cryptodev
 LDLIBS += -lrte_bus_vdev
 
+IMB_HDR = $(shell echo '\#include <intel-ipsec-mb.h>' | \
+	$(CC) -E $(EXTRA_CFLAGS) - | grep 'intel-ipsec-mb.h' | \
+	head -n1 | cut -d'"' -f2)
+
+# Detect library version
+IMB_VERSION = $(shell grep -e "IMB_VERSION_STR" $(IMB_HDR) | cut -d'"' -f2)
+IMB_VERSION_NUM = $(shell grep -e "IMB_VERSION_NUM" $(IMB_HDR) | cut -d' ' -f3)
+
+ifeq ($(IMB_VERSION),)
+$(error "IPSec_MB version >= 0.53 is required")
+endif
+
+ifeq ($(shell expr $(IMB_VERSION_NUM) \< 0x3400), 1)
+$(error "IPSec_MB version >= 0.53 is required")
+endif
 # library source files
 SRCS-$(CONFIG_RTE_LIBRTE_PMD_KASUMI) += rte_kasumi_pmd.c
 SRCS-$(CONFIG_RTE_LIBRTE_PMD_KASUMI) += rte_kasumi_pmd_ops.c
diff --git a/drivers/crypto/kasumi/meson.build b/drivers/crypto/kasumi/meson.build
index a09b0e251..bda9a4052 100644
--- a/drivers/crypto/kasumi/meson.build
+++ b/drivers/crypto/kasumi/meson.build
@@ -1,11 +1,20 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2018 Intel Corporation
 
-lib = cc.find_library('libsso_kasumi', required: false)
+IMB_required_ver = '0.53.0'
+lib = cc.find_library('IPSec_MB', required: false)
 if not lib.found()
 	build = false
 else
 	ext_deps += lib
+	# version comes with quotes, so we split based on " and take the middle
+	imb_ver = cc.get_define('IMB_VERSION_STR',
+		prefix : '#include<intel-ipsec-mb.h>').split('"')[1]
+
+	if (imb_ver == '') or (imb_ver.version_compare('<' + IMB_required_ver))
+		build = false
+	endif
+
 endif
 
 sources = files('rte_kasumi_pmd.c', 'rte_kasumi_pmd_ops.c')
diff --git a/drivers/crypto/kasumi/rte_kasumi_pmd.c b/drivers/crypto/kasumi/rte_kasumi_pmd.c
index 239a1cf44..037abf710 100644
--- a/drivers/crypto/kasumi/rte_kasumi_pmd.c
+++ b/drivers/crypto/kasumi/rte_kasumi_pmd.c
@@ -54,7 +54,7 @@ kasumi_get_mode(const struct rte_crypto_sym_xform *xform)
 
 /** Parse crypto xform chain and set private session parameters. */
 int
-kasumi_set_session_parameters(struct kasumi_session *sess,
+kasumi_set_session_parameters(MB_MGR *mgr, struct kasumi_session *sess,
 		const struct rte_crypto_sym_xform *xform)
 {
 	const struct rte_crypto_sym_xform *auth_xform = NULL;
@@ -97,7 +97,7 @@ kasumi_set_session_parameters(struct kasumi_session *sess,
 		}
 
 		/* Initialize key */
-		sso_kasumi_init_f8_key_sched(cipher_xform->cipher.key.data,
+		IMB_KASUMI_INIT_F8_KEY_SCHED(mgr, cipher_xform->cipher.key.data,
 				&sess->pKeySched_cipher);
 	}
 
@@ -116,7 +116,7 @@ kasumi_set_session_parameters(struct kasumi_session *sess,
 		sess->auth_op = auth_xform->auth.op;
 
 		/* Initialize key */
-		sso_kasumi_init_f9_key_sched(auth_xform->auth.key.data,
+		IMB_KASUMI_INIT_F9_KEY_SCHED(mgr, auth_xform->auth.key.data,
 				&sess->pKeySched_hash);
 	}
 
@@ -150,7 +150,7 @@ kasumi_get_session(struct kasumi_qp *qp, struct rte_crypto_op *op)
 
 		sess = (struct kasumi_session *)_sess_private_data;
 
-		if (unlikely(kasumi_set_session_parameters(sess,
+		if (unlikely(kasumi_set_session_parameters(qp->mgr, sess,
 				op->sym->xform) != 0)) {
 			rte_mempool_put(qp->sess_mp, _sess);
 			rte_mempool_put(qp->sess_mp, _sess_private_data);
@@ -169,13 +169,13 @@ kasumi_get_session(struct kasumi_qp *qp, struct rte_crypto_op *op)
 
 /** Encrypt/decrypt mbufs with same cipher key. */
 static uint8_t
-process_kasumi_cipher_op(struct rte_crypto_op **ops,
-		struct kasumi_session *session,
-		uint8_t num_ops)
+process_kasumi_cipher_op(struct kasumi_qp *qp, struct rte_crypto_op **ops,
+		struct kasumi_session *session, uint8_t num_ops)
 {
 	unsigned i;
 	uint8_t processed_ops = 0;
-	uint8_t *src[num_ops], *dst[num_ops];
+	const void *src[num_ops];
+	void *dst[num_ops];
 	uint8_t *iv_ptr;
 	uint64_t iv[num_ops];
 	uint32_t num_bytes[num_ops];
@@ -197,7 +197,7 @@ process_kasumi_cipher_op(struct rte_crypto_op **ops,
 	}
 
 	if (processed_ops != 0)
-		sso_kasumi_f8_n_buffer(&session->pKeySched_cipher, iv,
+		IMB_KASUMI_F8_N_BUFFER(qp->mgr, &session->pKeySched_cipher, iv,
 			src, dst, num_bytes, processed_ops);
 
 	return processed_ops;
@@ -205,7 +205,7 @@ process_kasumi_cipher_op(struct rte_crypto_op **ops,
 
 /** Encrypt/decrypt mbuf (bit level function). */
 static uint8_t
-process_kasumi_cipher_op_bit(struct rte_crypto_op *op,
+process_kasumi_cipher_op_bit(struct kasumi_qp *qp, struct rte_crypto_op *op,
 		struct kasumi_session *session)
 {
 	uint8_t *src, *dst;
@@ -215,18 +215,16 @@ process_kasumi_cipher_op_bit(struct rte_crypto_op *op,
 
 	offset_in_bits = op->sym->cipher.data.offset;
 	src = rte_pktmbuf_mtod(op->sym->m_src, uint8_t *);
-	if (op->sym->m_dst == NULL) {
-		op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
-		KASUMI_LOG(ERR, "bit-level in-place not supported");
-		return 0;
-	}
-	dst = rte_pktmbuf_mtod(op->sym->m_dst, uint8_t *);
+	if (op->sym->m_dst == NULL)
+		dst = src;
+	else
+		dst = rte_pktmbuf_mtod(op->sym->m_dst, uint8_t *);
 	iv_ptr = rte_crypto_op_ctod_offset(op, uint8_t *,
 			session->cipher_iv_offset);
 	iv = *((uint64_t *)(iv_ptr));
 	length_in_bits = op->sym->cipher.data.length;
 
-	sso_kasumi_f8_1_buffer_bit(&session->pKeySched_cipher, iv,
+	IMB_KASUMI_F8_1_BUFFER_BIT(qp->mgr, &session->pKeySched_cipher, iv,
 			src, dst, length_in_bits, offset_in_bits);
 
 	return 1;
@@ -261,7 +259,8 @@ process_kasumi_hash_op(struct kasumi_qp *qp, struct rte_crypto_op **ops,
 
 		if (session->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) {
 			dst = qp->temp_digest;
-			sso_kasumi_f9_1_buffer(&session->pKeySched_hash, src,
+			IMB_KASUMI_F9_1_BUFFER(qp->mgr,
+					&session->pKeySched_hash, src,
 					num_bytes, dst);
 
 			/* Verify digest. */
@@ -271,7 +270,8 @@ process_kasumi_hash_op(struct kasumi_qp *qp, struct rte_crypto_op **ops,
 		} else  {
 			dst = ops[i]->sym->auth.digest.data;
 
-			sso_kasumi_f9_1_buffer(&session->pKeySched_hash, src,
+			IMB_KASUMI_F9_1_BUFFER(qp->mgr,
+					&session->pKeySched_hash, src,
 					num_bytes, dst);
 		}
 		processed_ops++;
@@ -291,7 +291,7 @@ process_ops(struct rte_crypto_op **ops, struct kasumi_session *session,
 
 	switch (session->op) {
 	case KASUMI_OP_ONLY_CIPHER:
-		processed_ops = process_kasumi_cipher_op(ops,
+		processed_ops = process_kasumi_cipher_op(qp, ops,
 				session, num_ops);
 		break;
 	case KASUMI_OP_ONLY_AUTH:
@@ -299,14 +299,14 @@ process_ops(struct rte_crypto_op **ops, struct kasumi_session *session,
 				num_ops);
 		break;
 	case KASUMI_OP_CIPHER_AUTH:
-		processed_ops = process_kasumi_cipher_op(ops, session,
+		processed_ops = process_kasumi_cipher_op(qp, ops, session,
 				num_ops);
 		process_kasumi_hash_op(qp, ops, session, processed_ops);
 		break;
 	case KASUMI_OP_AUTH_CIPHER:
 		processed_ops = process_kasumi_hash_op(qp, ops, session,
 				num_ops);
-		process_kasumi_cipher_op(ops, session, processed_ops);
+		process_kasumi_cipher_op(qp, ops, session, processed_ops);
 		break;
 	default:
 		/* Operation not supported. */
@@ -348,21 +348,21 @@ process_op_bit(struct rte_crypto_op *op, struct kasumi_session *session,
 
 	switch (session->op) {
 	case KASUMI_OP_ONLY_CIPHER:
-		processed_op = process_kasumi_cipher_op_bit(op,
+		processed_op = process_kasumi_cipher_op_bit(qp, op,
 				session);
 		break;
 	case KASUMI_OP_ONLY_AUTH:
 		processed_op = process_kasumi_hash_op(qp, &op, session, 1);
 		break;
 	case KASUMI_OP_CIPHER_AUTH:
-		processed_op = process_kasumi_cipher_op_bit(op, session);
+		processed_op = process_kasumi_cipher_op_bit(qp, op, session);
 		if (processed_op == 1)
 			process_kasumi_hash_op(qp, &op, session, 1);
 		break;
 	case KASUMI_OP_AUTH_CIPHER:
 		processed_op = process_kasumi_hash_op(qp, &op, session, 1);
 		if (processed_op == 1)
-			process_kasumi_cipher_op_bit(op, session);
+			process_kasumi_cipher_op_bit(qp, op, session);
 		break;
 	default:
 		/* Operation not supported. */
@@ -531,7 +531,7 @@ cryptodev_kasumi_create(const char *name,
 {
 	struct rte_cryptodev *dev;
 	struct kasumi_private *internals;
-	uint64_t cpu_flags = 0;
+	MB_MGR *mgr;
 
 	dev = rte_cryptodev_pmd_create(name, &vdev->device, init_params);
 	if (dev == NULL) {
@@ -539,12 +539,6 @@ cryptodev_kasumi_create(const char *name,
 		goto init_error;
 	}
 
-	/* Check CPU for supported vector instruction set */
-	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX))
-		cpu_flags |= RTE_CRYPTODEV_FF_CPU_AVX;
-	else
-		cpu_flags |= RTE_CRYPTODEV_FF_CPU_SSE;
-
 	dev->driver_id = cryptodev_driver_id;
 	dev->dev_ops = rte_kasumi_pmd_ops;
 
@@ -553,12 +547,24 @@ cryptodev_kasumi_create(const char *name,
 	dev->enqueue_burst = kasumi_pmd_enqueue_burst;
 
 	dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
-			RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING |
-			cpu_flags;
+			RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING;
+
+	mgr = alloc_mb_mgr(0);
+	if (mgr == NULL)
+		return -ENOMEM;
+
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX)) {
+		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX;
+		init_mb_mgr_avx(mgr);
+	} else {
+		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_SSE;
+		init_mb_mgr_sse(mgr);
+	}
 
 	internals = dev->data->dev_private;
 
 	internals->max_nb_queue_pairs = init_params->max_nb_queue_pairs;
+	internals->mgr = mgr;
 
 	return 0;
 init_error:
@@ -596,6 +602,7 @@ cryptodev_kasumi_remove(struct rte_vdev_device *vdev)
 {
 	struct rte_cryptodev *cryptodev;
 	const char *name;
+	struct kasumi_private *internals;
 
 	name = rte_vdev_device_name(vdev);
 	if (name == NULL)
@@ -605,6 +612,10 @@ cryptodev_kasumi_remove(struct rte_vdev_device *vdev)
 	if (cryptodev == NULL)
 		return -ENODEV;
 
+	internals = cryptodev->data->dev_private;
+
+	free_mb_mgr(internals->mgr);
+
 	return rte_cryptodev_pmd_destroy(cryptodev);
 }
 
diff --git a/drivers/crypto/kasumi/rte_kasumi_pmd_ops.c b/drivers/crypto/kasumi/rte_kasumi_pmd_ops.c
index 9e4bf1b52..2f30115b9 100644
--- a/drivers/crypto/kasumi/rte_kasumi_pmd_ops.c
+++ b/drivers/crypto/kasumi/rte_kasumi_pmd_ops.c
@@ -195,6 +195,7 @@ kasumi_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 		int socket_id, struct rte_mempool *session_pool)
 {
 	struct kasumi_qp *qp = NULL;
+	struct kasumi_private *internals = dev->data->dev_private;
 
 	/* Free memory prior to re-allocation if needed. */
 	if (dev->data->queue_pairs[qp_id] != NULL)
@@ -217,6 +218,7 @@ kasumi_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	if (qp->processed_ops == NULL)
 		goto qp_setup_cleanup;
 
+	qp->mgr = internals->mgr;
 	qp->sess_mp = session_pool;
 
 	memset(&qp->qp_stats, 0, sizeof(qp->qp_stats));
@@ -245,13 +247,14 @@ kasumi_pmd_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
 
 /** Configure a KASUMI session from a crypto xform chain */
 static int
-kasumi_pmd_sym_session_configure(struct rte_cryptodev *dev __rte_unused,
+kasumi_pmd_sym_session_configure(struct rte_cryptodev *dev,
 		struct rte_crypto_sym_xform *xform,
 		struct rte_cryptodev_sym_session *sess,
 		struct rte_mempool *mempool)
 {
 	void *sess_private_data;
 	int ret;
+	struct kasumi_private *internals = dev->data->dev_private;
 
 	if (unlikely(sess == NULL)) {
 		KASUMI_LOG(ERR, "invalid session struct");
@@ -264,7 +267,8 @@ kasumi_pmd_sym_session_configure(struct rte_cryptodev *dev __rte_unused,
 		return -ENOMEM;
 	}
 
-	ret = kasumi_set_session_parameters(sess_private_data, xform);
+	ret = kasumi_set_session_parameters(internals->mgr,
+					sess_private_data, xform);
 	if (ret != 0) {
 		KASUMI_LOG(ERR, "failed configure session parameters");
 
diff --git a/drivers/crypto/kasumi/rte_kasumi_pmd_private.h b/drivers/crypto/kasumi/rte_kasumi_pmd_private.h
index 488777ca8..3db52c03a 100644
--- a/drivers/crypto/kasumi/rte_kasumi_pmd_private.h
+++ b/drivers/crypto/kasumi/rte_kasumi_pmd_private.h
@@ -5,7 +5,7 @@
 #ifndef _RTE_KASUMI_PMD_PRIVATE_H_
 #define _RTE_KASUMI_PMD_PRIVATE_H_
 
-#include <sso_kasumi.h>
+#include <intel-ipsec-mb.h>
 
 #define CRYPTODEV_NAME_KASUMI_PMD	crypto_kasumi
 /**< KASUMI PMD device name */
@@ -24,6 +24,8 @@ int kasumi_logtype_driver;
 struct kasumi_private {
 	unsigned max_nb_queue_pairs;
 	/**< Max number of queue pairs supported by device */
+	MB_MGR *mgr;
+	/**< Multi-buffer instance */
 };
 
 /** KASUMI buffer queue pair */
@@ -43,6 +45,8 @@ struct kasumi_qp {
 	 * by the driver when verifying a digest provided
 	 * by the user (using authentication verify operation)
 	 */
+	MB_MGR *mgr;
+	/**< Multi-buffer instance */
 } __rte_cache_aligned;
 
 enum kasumi_operation {
@@ -56,8 +60,8 @@ enum kasumi_operation {
 /** KASUMI private session structure */
 struct kasumi_session {
 	/* Keys have to be 16-byte aligned */
-	sso_kasumi_key_sched_t pKeySched_cipher;
-	sso_kasumi_key_sched_t pKeySched_hash;
+	kasumi_key_sched_t pKeySched_cipher;
+	kasumi_key_sched_t pKeySched_hash;
 	enum kasumi_operation op;
 	enum rte_crypto_auth_operation auth_op;
 	uint16_t cipher_iv_offset;
@@ -65,7 +69,7 @@ struct kasumi_session {
 
 
 int
-kasumi_set_session_parameters(struct kasumi_session *sess,
+kasumi_set_session_parameters(MB_MGR *mgr, struct kasumi_session *sess,
 		const struct rte_crypto_sym_xform *xform);
 
 
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index b33603ef5..0ec48482c 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -231,7 +231,7 @@ endif # CONFIG_RTE_LIBRTE_PMD_QAT
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_SNOW3G)      += -lrte_pmd_snow3g
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_SNOW3G)      += -lIPSec_MB
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_KASUMI)      += -lrte_pmd_kasumi
-_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_KASUMI)      += -L$(LIBSSO_KASUMI_PATH)/build -lsso_kasumi
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_KASUMI)      += -lIPSec_MB
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_ZUC)         += -lrte_pmd_zuc
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_ZUC)         += -lIPSec_MB
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_ARMV8_CRYPTO)    += -lrte_pmd_armv8
-- 
2.24.1


^ permalink raw reply	[flat|nested] 8+ messages in thread

* [dpdk-dev] [RFC PATCH 4/5] crypto/aesni_mb: support IPSec MB library v0.53
  2020-03-05 15:34 [dpdk-dev] [RFC PATCH 0/5] Support Intel IPSec MB v0.53 in DPDK 18.11 Pablo de Lara
                   ` (2 preceding siblings ...)
  2020-03-05 15:34 ` [dpdk-dev] [RFC PATCH 3/5] crypto/kasumi: " Pablo de Lara
@ 2020-03-05 15:34 ` Pablo de Lara
  2020-03-05 15:34 ` [dpdk-dev] [RFC PATCH 5/5] crypto/aesni_gcm: " Pablo de Lara
  2020-03-19 14:32 ` [dpdk-dev] [RFC PATCH 0/5] Support Intel IPSec MB v0.53 in DPDK 18.11 Kevin Traynor
  5 siblings, 0 replies; 8+ messages in thread
From: Pablo de Lara @ 2020-03-05 15:34 UTC (permalink / raw)
  To: dev; +Cc: Pablo de Lara

Add support for underlying IPSec Multi-buffer library v0.53.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
---
 drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c    | 476 ++++++++++++------
 .../crypto/aesni_mb/rte_aesni_mb_pmd_ops.c    | 205 +++++---
 .../aesni_mb/rte_aesni_mb_pmd_private.h       |  30 +-
 3 files changed, 489 insertions(+), 222 deletions(-)

diff --git a/drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c b/drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c
index 83250e32c..9dfa89f71 100644
--- a/drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c
+++ b/drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c
@@ -35,7 +35,7 @@ typedef void (*aes_keyexp_t)(const void *key, void *enc_exp_keys, void *dec_exp_
 static void
 calculate_auth_precomputes(hash_one_block_t one_block_hash,
 		uint8_t *ipad, uint8_t *opad,
-		uint8_t *hkey, uint16_t hkey_len,
+		const uint8_t *hkey, uint16_t hkey_len,
 		uint16_t blocksize)
 {
 	unsigned i, length;
@@ -85,6 +85,25 @@ aesni_mb_get_chain_order(const struct rte_crypto_sym_xform *xform)
 			return AESNI_MB_OP_HASH_CIPHER;
 	}
 
+#if IMB_VERSION_NUM > IMB_VERSION(0, 52, 0)
+	if (xform->type == RTE_CRYPTO_SYM_XFORM_AEAD) {
+		if (xform->aead.op == RTE_CRYPTO_AEAD_OP_ENCRYPT) {
+			/*
+			 * CCM requires to hash first and cipher later
+			 * when encrypting
+			 */
+			if (xform->aead.algo == RTE_CRYPTO_AEAD_AES_CCM)
+				return AESNI_MB_OP_AEAD_HASH_CIPHER;
+			else
+				return AESNI_MB_OP_AEAD_CIPHER_HASH;
+		} else {
+			if (xform->aead.algo == RTE_CRYPTO_AEAD_AES_CCM)
+				return AESNI_MB_OP_AEAD_CIPHER_HASH;
+			else
+				return AESNI_MB_OP_AEAD_HASH_CIPHER;
+		}
+	}
+#else
 	if (xform->type == RTE_CRYPTO_SYM_XFORM_AEAD) {
 		if (xform->aead.algo == RTE_CRYPTO_AEAD_AES_CCM ||
 				xform->aead.algo == RTE_CRYPTO_AEAD_AES_GCM) {
@@ -94,19 +113,21 @@ aesni_mb_get_chain_order(const struct rte_crypto_sym_xform *xform)
 				return AESNI_MB_OP_AEAD_HASH_CIPHER;
 		}
 	}
+#endif
 
 	return AESNI_MB_OP_NOT_SUPPORTED;
 }
 
 /** Set session authentication parameters */
 static int
-aesni_mb_set_session_auth_parameters(const struct aesni_mb_op_fns *mb_ops,
+aesni_mb_set_session_auth_parameters(const MB_MGR *mb_mgr,
 		struct aesni_mb_session *sess,
 		const struct rte_crypto_sym_xform *xform)
 {
-	hash_one_block_t hash_oneblock_fn;
+	hash_one_block_t hash_oneblock_fn = NULL;
 	unsigned int key_larger_block_size = 0;
 	uint8_t hashed_key[HMAC_MAX_BLOCK_SIZE] = { 0 };
+	uint32_t auth_precompute = 1;
 
 	if (xform == NULL) {
 		sess->auth.algo = NULL_HASH;
@@ -135,13 +156,16 @@ aesni_mb_set_session_auth_parameters(const struct aesni_mb_op_fns *mb_ops,
 			return -EINVAL;
 		}
 		sess->auth.gen_digest_len = sess->auth.req_digest_len;
-		(*mb_ops->aux.keyexp.aes_xcbc)(xform->auth.key.data,
+
+		IMB_AES_XCBC_KEYEXP(mb_mgr, xform->auth.key.data,
 				sess->auth.xcbc.k1_expanded,
 				sess->auth.xcbc.k2, sess->auth.xcbc.k3);
 		return 0;
 	}
 
 	if (xform->auth.algo == RTE_CRYPTO_AUTH_AES_CMAC) {
+		uint32_t dust[4*15];
+
 		sess->auth.algo = AES_CMAC;
 
 		uint16_t cmac_digest_len = get_digest_byte_length(AES_CMAC);
@@ -150,102 +174,144 @@ aesni_mb_set_session_auth_parameters(const struct aesni_mb_op_fns *mb_ops,
 			AESNI_MB_LOG(ERR, "Invalid digest size\n");
 			return -EINVAL;
 		}
+		if (sess->auth.req_digest_len < 4)
+			sess->auth.gen_digest_len = cmac_digest_len;
+		else
+			sess->auth.gen_digest_len = sess->auth.req_digest_len;
+		IMB_AES_KEYEXP_128(mb_mgr, xform->auth.key.data,
+				sess->auth.cmac.expkey, dust);
+		IMB_AES_CMAC_SUBKEY_GEN_128(mb_mgr, sess->auth.cmac.expkey,
+				sess->auth.cmac.skey1, sess->auth.cmac.skey2);
+		return 0;
+	}
+
+	if (xform->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) {
+		if (xform->auth.op == RTE_CRYPTO_AUTH_OP_GENERATE) {
+			sess->cipher.direction = ENCRYPT;
+			sess->chain_order = CIPHER_HASH;
+		} else
+			sess->cipher.direction = DECRYPT;
+
+		sess->auth.algo = AES_GMAC;
 		/*
-		 * Multi-buffer lib supports digest sizes from 4 to 16 bytes
-		 * in version 0.50 and sizes of 12 and 16 bytes,
-		 * in version 0.49.
+		 * Multi-buffer lib supports 8, 12 and 16 bytes of digest.
 		 * If size requested is different, generate the full digest
 		 * (16 bytes) in a temporary location and then memcpy
 		 * the requested number of bytes.
 		 */
-#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0)
-		if (sess->auth.req_digest_len < 4)
-#else
-		uint16_t cmac_trunc_digest_len =
-				get_truncated_digest_byte_length(AES_CMAC);
-		if (sess->auth.req_digest_len != cmac_digest_len &&
-				sess->auth.req_digest_len != cmac_trunc_digest_len)
-#endif
-			sess->auth.gen_digest_len = cmac_digest_len;
-		else
+		if (sess->auth.req_digest_len != 16 &&
+				sess->auth.req_digest_len != 12 &&
+				sess->auth.req_digest_len != 8) {
+			sess->auth.gen_digest_len = 16;
+		} else {
 			sess->auth.gen_digest_len = sess->auth.req_digest_len;
-		(*mb_ops->aux.keyexp.aes_cmac_expkey)(xform->auth.key.data,
-				sess->auth.cmac.expkey);
+		}
+		sess->iv.length = xform->auth.iv.length;
+		sess->iv.offset = xform->auth.iv.offset;
+
+		switch (xform->auth.key.length) {
+		case AES_128_BYTES:
+			IMB_AES128_GCM_PRE(mb_mgr, xform->auth.key.data,
+				&sess->cipher.gcm_key);
+			sess->cipher.key_length_in_bytes = AES_128_BYTES;
+			break;
+		case AES_192_BYTES:
+			IMB_AES192_GCM_PRE(mb_mgr, xform->auth.key.data,
+				&sess->cipher.gcm_key);
+			sess->cipher.key_length_in_bytes = AES_192_BYTES;
+			break;
+		case AES_256_BYTES:
+			IMB_AES256_GCM_PRE(mb_mgr, xform->auth.key.data,
+				&sess->cipher.gcm_key);
+			sess->cipher.key_length_in_bytes = AES_256_BYTES;
+			break;
+		default:
+			RTE_LOG(ERR, PMD, "failed to parse test type\n");
+			return -EINVAL;
+		}
 
-		(*mb_ops->aux.keyexp.aes_cmac_subkey)(sess->auth.cmac.expkey,
-				sess->auth.cmac.skey1, sess->auth.cmac.skey2);
 		return 0;
 	}
 
 	switch (xform->auth.algo) {
 	case RTE_CRYPTO_AUTH_MD5_HMAC:
 		sess->auth.algo = MD5;
-		hash_oneblock_fn = mb_ops->aux.one_block.md5;
+		hash_oneblock_fn = mb_mgr->md5_one_block;
 		break;
 	case RTE_CRYPTO_AUTH_SHA1_HMAC:
 		sess->auth.algo = SHA1;
-		hash_oneblock_fn = mb_ops->aux.one_block.sha1;
-#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0)
+		hash_oneblock_fn = mb_mgr->sha1_one_block;
 		if (xform->auth.key.length > get_auth_algo_blocksize(SHA1)) {
-			mb_ops->aux.multi_block.sha1(
+			IMB_SHA1(mb_mgr,
 				xform->auth.key.data,
 				xform->auth.key.length,
 				hashed_key);
 			key_larger_block_size = 1;
 		}
-#endif
+		break;
+	case RTE_CRYPTO_AUTH_SHA1:
+		sess->auth.algo = PLAIN_SHA1;
+		auth_precompute = 0;
 		break;
 	case RTE_CRYPTO_AUTH_SHA224_HMAC:
 		sess->auth.algo = SHA_224;
-		hash_oneblock_fn = mb_ops->aux.one_block.sha224;
-#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0)
+		hash_oneblock_fn = mb_mgr->sha224_one_block;
 		if (xform->auth.key.length > get_auth_algo_blocksize(SHA_224)) {
-			mb_ops->aux.multi_block.sha224(
+			IMB_SHA224(mb_mgr,
 				xform->auth.key.data,
 				xform->auth.key.length,
 				hashed_key);
 			key_larger_block_size = 1;
 		}
-#endif
+		break;
+	case RTE_CRYPTO_AUTH_SHA224:
+		sess->auth.algo = PLAIN_SHA_224;
+		auth_precompute = 0;
 		break;
 	case RTE_CRYPTO_AUTH_SHA256_HMAC:
 		sess->auth.algo = SHA_256;
-		hash_oneblock_fn = mb_ops->aux.one_block.sha256;
-#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0)
+		hash_oneblock_fn = mb_mgr->sha256_one_block;
 		if (xform->auth.key.length > get_auth_algo_blocksize(SHA_256)) {
-			mb_ops->aux.multi_block.sha256(
+			IMB_SHA256(mb_mgr,
 				xform->auth.key.data,
 				xform->auth.key.length,
 				hashed_key);
 			key_larger_block_size = 1;
 		}
-#endif
+		break;
+	case RTE_CRYPTO_AUTH_SHA256:
+		sess->auth.algo = PLAIN_SHA_256;
+		auth_precompute = 0;
 		break;
 	case RTE_CRYPTO_AUTH_SHA384_HMAC:
 		sess->auth.algo = SHA_384;
-		hash_oneblock_fn = mb_ops->aux.one_block.sha384;
-#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0)
+		hash_oneblock_fn = mb_mgr->sha384_one_block;
 		if (xform->auth.key.length > get_auth_algo_blocksize(SHA_384)) {
-			mb_ops->aux.multi_block.sha384(
+			IMB_SHA384(mb_mgr,
 				xform->auth.key.data,
 				xform->auth.key.length,
 				hashed_key);
 			key_larger_block_size = 1;
 		}
-#endif
+		break;
+	case RTE_CRYPTO_AUTH_SHA384:
+		sess->auth.algo = PLAIN_SHA_384;
+		auth_precompute = 0;
 		break;
 	case RTE_CRYPTO_AUTH_SHA512_HMAC:
 		sess->auth.algo = SHA_512;
-		hash_oneblock_fn = mb_ops->aux.one_block.sha512;
-#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0)
+		hash_oneblock_fn = mb_mgr->sha512_one_block;
 		if (xform->auth.key.length > get_auth_algo_blocksize(SHA_512)) {
-			mb_ops->aux.multi_block.sha512(
+			IMB_SHA512(mb_mgr,
 				xform->auth.key.data,
 				xform->auth.key.length,
 				hashed_key);
 			key_larger_block_size = 1;
 		}
-#endif
+		break;
+	case RTE_CRYPTO_AUTH_SHA512:
+		sess->auth.algo = PLAIN_SHA_512;
+		auth_precompute = 0;
 		break;
 	default:
 		AESNI_MB_LOG(ERR, "Unsupported authentication algorithm selection");
@@ -256,12 +322,8 @@ aesni_mb_set_session_auth_parameters(const struct aesni_mb_op_fns *mb_ops,
 	uint16_t full_digest_size =
 			get_digest_byte_length(sess->auth.algo);
 
-#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0)
 	if (sess->auth.req_digest_len > full_digest_size ||
 			sess->auth.req_digest_len == 0) {
-#else
-	if (sess->auth.req_digest_len != trunc_digest_size) {
-#endif
 		AESNI_MB_LOG(ERR, "Invalid digest size\n");
 		return -EINVAL;
 	}
@@ -272,6 +334,10 @@ aesni_mb_set_session_auth_parameters(const struct aesni_mb_op_fns *mb_ops,
 	else
 		sess->auth.gen_digest_len = sess->auth.req_digest_len;
 
+	/* Plain SHA does not require precompute key */
+	if (auth_precompute == 0)
+		return 0;
+
 	/* Calculate Authentication precomputes */
 	if (key_larger_block_size) {
 		calculate_auth_precomputes(hash_oneblock_fn,
@@ -292,13 +358,12 @@ aesni_mb_set_session_auth_parameters(const struct aesni_mb_op_fns *mb_ops,
 
 /** Set session cipher parameters */
 static int
-aesni_mb_set_session_cipher_parameters(const struct aesni_mb_op_fns *mb_ops,
+aesni_mb_set_session_cipher_parameters(const MB_MGR *mb_mgr,
 		struct aesni_mb_session *sess,
 		const struct rte_crypto_sym_xform *xform)
 {
 	uint8_t is_aes = 0;
 	uint8_t is_3DES = 0;
-	aes_keyexp_t aes_keyexp_fn;
 
 	if (xform == NULL) {
 		sess->cipher.mode = NULL_CIPHER;
@@ -361,26 +426,26 @@ aesni_mb_set_session_cipher_parameters(const struct aesni_mb_op_fns *mb_ops,
 		switch (xform->cipher.key.length) {
 		case AES_128_BYTES:
 			sess->cipher.key_length_in_bytes = AES_128_BYTES;
-			aes_keyexp_fn = mb_ops->aux.keyexp.aes128;
+			IMB_AES_KEYEXP_128(mb_mgr, xform->cipher.key.data,
+					sess->cipher.expanded_aes_keys.encode,
+					sess->cipher.expanded_aes_keys.decode);
 			break;
 		case AES_192_BYTES:
 			sess->cipher.key_length_in_bytes = AES_192_BYTES;
-			aes_keyexp_fn = mb_ops->aux.keyexp.aes192;
+			IMB_AES_KEYEXP_192(mb_mgr, xform->cipher.key.data,
+					sess->cipher.expanded_aes_keys.encode,
+					sess->cipher.expanded_aes_keys.decode);
 			break;
 		case AES_256_BYTES:
 			sess->cipher.key_length_in_bytes = AES_256_BYTES;
-			aes_keyexp_fn = mb_ops->aux.keyexp.aes256;
+			IMB_AES_KEYEXP_256(mb_mgr, xform->cipher.key.data,
+					sess->cipher.expanded_aes_keys.encode,
+					sess->cipher.expanded_aes_keys.decode);
 			break;
 		default:
 			AESNI_MB_LOG(ERR, "Invalid cipher key length");
 			return -EINVAL;
 		}
-
-		/* Expanded cipher keys */
-		(*aes_keyexp_fn)(xform->cipher.key.data,
-				sess->cipher.expanded_aes_keys.encode,
-				sess->cipher.expanded_aes_keys.decode);
-
 	} else if (is_3DES) {
 		uint64_t *keys[3] = {sess->cipher.exp_3des_keys.key[0],
 				sess->cipher.exp_3des_keys.key[1],
@@ -388,9 +453,12 @@ aesni_mb_set_session_cipher_parameters(const struct aesni_mb_op_fns *mb_ops,
 
 		switch (xform->cipher.key.length) {
 		case  24:
-			des_key_schedule(keys[0], xform->cipher.key.data);
-			des_key_schedule(keys[1], xform->cipher.key.data+8);
-			des_key_schedule(keys[2], xform->cipher.key.data+16);
+			IMB_DES_KEYSCHED(mb_mgr, keys[0],
+					xform->cipher.key.data);
+			IMB_DES_KEYSCHED(mb_mgr, keys[1],
+					xform->cipher.key.data + 8);
+			IMB_DES_KEYSCHED(mb_mgr, keys[2],
+					xform->cipher.key.data + 16);
 
 			/* Initialize keys - 24 bytes: [K1-K2-K3] */
 			sess->cipher.exp_3des_keys.ks_ptr[0] = keys[0];
@@ -398,8 +466,10 @@ aesni_mb_set_session_cipher_parameters(const struct aesni_mb_op_fns *mb_ops,
 			sess->cipher.exp_3des_keys.ks_ptr[2] = keys[2];
 			break;
 		case 16:
-			des_key_schedule(keys[0], xform->cipher.key.data);
-			des_key_schedule(keys[1], xform->cipher.key.data+8);
+			IMB_DES_KEYSCHED(mb_mgr, keys[0],
+					xform->cipher.key.data);
+			IMB_DES_KEYSCHED(mb_mgr, keys[1],
+					xform->cipher.key.data + 8);
 
 			/* Initialize keys - 16 bytes: [K1=K1,K2=K2,K3=K1] */
 			sess->cipher.exp_3des_keys.ks_ptr[0] = keys[0];
@@ -407,7 +477,8 @@ aesni_mb_set_session_cipher_parameters(const struct aesni_mb_op_fns *mb_ops,
 			sess->cipher.exp_3des_keys.ks_ptr[2] = keys[0];
 			break;
 		case 8:
-			des_key_schedule(keys[0], xform->cipher.key.data);
+			IMB_DES_KEYSCHED(mb_mgr, keys[0],
+					xform->cipher.key.data);
 
 			/* Initialize keys - 8 bytes: [K1 = K2 = K3] */
 			sess->cipher.exp_3des_keys.ks_ptr[0] = keys[0];
@@ -419,11 +490,7 @@ aesni_mb_set_session_cipher_parameters(const struct aesni_mb_op_fns *mb_ops,
 			return -EINVAL;
 		}
 
-#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0)
 		sess->cipher.key_length_in_bytes = 24;
-#else
-		sess->cipher.key_length_in_bytes = 8;
-#endif
 	} else {
 		if (xform->cipher.key.length != 8) {
 			AESNI_MB_LOG(ERR, "Invalid cipher key length");
@@ -431,9 +498,11 @@ aesni_mb_set_session_cipher_parameters(const struct aesni_mb_op_fns *mb_ops,
 		}
 		sess->cipher.key_length_in_bytes = 8;
 
-		des_key_schedule((uint64_t *)sess->cipher.expanded_aes_keys.encode,
+		IMB_DES_KEYSCHED(mb_mgr,
+			(uint64_t *)sess->cipher.expanded_aes_keys.encode,
 				xform->cipher.key.data);
-		des_key_schedule((uint64_t *)sess->cipher.expanded_aes_keys.decode,
+		IMB_DES_KEYSCHED(mb_mgr,
+			(uint64_t *)sess->cipher.expanded_aes_keys.decode,
 				xform->cipher.key.data);
 	}
 
@@ -441,15 +510,10 @@ aesni_mb_set_session_cipher_parameters(const struct aesni_mb_op_fns *mb_ops,
 }
 
 static int
-aesni_mb_set_session_aead_parameters(const struct aesni_mb_op_fns *mb_ops,
+aesni_mb_set_session_aead_parameters(const MB_MGR *mb_mgr,
 		struct aesni_mb_session *sess,
 		const struct rte_crypto_sym_xform *xform)
 {
-	union {
-		aes_keyexp_t aes_keyexp_fn;
-		aes_gcm_keyexp_t aes_gcm_keyexp_fn;
-	} keyexp;
-
 	switch (xform->aead.op) {
 	case RTE_CRYPTO_AEAD_OP_ENCRYPT:
 		sess->cipher.direction = ENCRYPT;
@@ -473,17 +537,15 @@ aesni_mb_set_session_aead_parameters(const struct aesni_mb_op_fns *mb_ops,
 		switch (xform->aead.key.length) {
 		case AES_128_BYTES:
 			sess->cipher.key_length_in_bytes = AES_128_BYTES;
-			keyexp.aes_keyexp_fn = mb_ops->aux.keyexp.aes128;
+			IMB_AES_KEYEXP_128(mb_mgr, xform->aead.key.data,
+					sess->cipher.expanded_aes_keys.encode,
+					sess->cipher.expanded_aes_keys.decode);
 			break;
 		default:
 			AESNI_MB_LOG(ERR, "Invalid cipher key length");
 			return -EINVAL;
 		}
 
-		/* Expanded cipher keys */
-		(*keyexp.aes_keyexp_fn)(xform->aead.key.data,
-				sess->cipher.expanded_aes_keys.encode,
-				sess->cipher.expanded_aes_keys.decode);
 		break;
 
 	case RTE_CRYPTO_AEAD_AES_GCM:
@@ -493,26 +555,24 @@ aesni_mb_set_session_aead_parameters(const struct aesni_mb_op_fns *mb_ops,
 		switch (xform->aead.key.length) {
 		case AES_128_BYTES:
 			sess->cipher.key_length_in_bytes = AES_128_BYTES;
-			keyexp.aes_gcm_keyexp_fn =
-					mb_ops->aux.keyexp.aes_gcm_128;
+			IMB_AES128_GCM_PRE(mb_mgr, xform->aead.key.data,
+				&sess->cipher.gcm_key);
 			break;
 		case AES_192_BYTES:
 			sess->cipher.key_length_in_bytes = AES_192_BYTES;
-			keyexp.aes_gcm_keyexp_fn =
-					mb_ops->aux.keyexp.aes_gcm_192;
+			IMB_AES192_GCM_PRE(mb_mgr, xform->aead.key.data,
+				&sess->cipher.gcm_key);
 			break;
 		case AES_256_BYTES:
 			sess->cipher.key_length_in_bytes = AES_256_BYTES;
-			keyexp.aes_gcm_keyexp_fn =
-					mb_ops->aux.keyexp.aes_gcm_256;
+			IMB_AES256_GCM_PRE(mb_mgr, xform->aead.key.data,
+				&sess->cipher.gcm_key);
 			break;
 		default:
 			AESNI_MB_LOG(ERR, "Invalid cipher key length");
 			return -EINVAL;
 		}
 
-		(keyexp.aes_gcm_keyexp_fn)(xform->aead.key.data,
-				&sess->cipher.gcm_key);
 		break;
 
 	default:
@@ -539,7 +599,7 @@ aesni_mb_set_session_aead_parameters(const struct aesni_mb_op_fns *mb_ops,
 
 /** Parse crypto xform chain and set private session parameters */
 int
-aesni_mb_set_session_parameters(const struct aesni_mb_op_fns *mb_ops,
+aesni_mb_set_session_parameters(const MB_MGR *mb_mgr,
 		struct aesni_mb_session *sess,
 		const struct rte_crypto_sym_xform *xform)
 {
@@ -598,13 +658,13 @@ aesni_mb_set_session_parameters(const struct aesni_mb_op_fns *mb_ops,
 	/* Default IV length = 0 */
 	sess->iv.length = 0;
 
-	ret = aesni_mb_set_session_auth_parameters(mb_ops, sess, auth_xform);
+	ret = aesni_mb_set_session_auth_parameters(mb_mgr, sess, auth_xform);
 	if (ret != 0) {
 		AESNI_MB_LOG(ERR, "Invalid/unsupported authentication parameters");
 		return ret;
 	}
 
-	ret = aesni_mb_set_session_cipher_parameters(mb_ops, sess,
+	ret = aesni_mb_set_session_cipher_parameters(mb_mgr, sess,
 			cipher_xform);
 	if (ret != 0) {
 		AESNI_MB_LOG(ERR, "Invalid/unsupported cipher parameters");
@@ -612,7 +672,7 @@ aesni_mb_set_session_parameters(const struct aesni_mb_op_fns *mb_ops,
 	}
 
 	if (aead_xform) {
-		ret = aesni_mb_set_session_aead_parameters(mb_ops, sess,
+		ret = aesni_mb_set_session_aead_parameters(mb_mgr, sess,
 				aead_xform);
 		if (ret != 0) {
 			AESNI_MB_LOG(ERR, "Invalid/unsupported aead parameters");
@@ -673,7 +733,7 @@ get_session(struct aesni_mb_qp *qp, struct rte_crypto_op *op)
 
 		sess = (struct aesni_mb_session *)_sess_private_data;
 
-		if (unlikely(aesni_mb_set_session_parameters(qp->op_fns,
+		if (unlikely(aesni_mb_set_session_parameters(qp->mb_mgr,
 				sess, op->sym->xform) != 0)) {
 			rte_mempool_put(qp->sess_mp, _sess);
 			rte_mempool_put(qp->sess_mp, _sess_private_data);
@@ -690,6 +750,56 @@ get_session(struct aesni_mb_qp *qp, struct rte_crypto_op *op)
 	return sess;
 }
 
+static inline uint64_t
+auth_start_offset(struct rte_crypto_op *op, struct aesni_mb_session *session,
+		uint32_t oop)
+{
+	struct rte_mbuf *m_src, *m_dst;
+	uint8_t *p_src, *p_dst;
+	uintptr_t u_src, u_dst;
+	uint32_t cipher_end, auth_end;
+
+	/* Only cipher then hash needs special calculation. */
+	if (!oop || session->chain_order != CIPHER_HASH)
+		return op->sym->auth.data.offset;
+
+	m_src = op->sym->m_src;
+	m_dst = op->sym->m_dst;
+
+	p_src = rte_pktmbuf_mtod(m_src, uint8_t *);
+	p_dst = rte_pktmbuf_mtod(m_dst, uint8_t *);
+	u_src = (uintptr_t)p_src;
+	u_dst = (uintptr_t)p_dst + op->sym->auth.data.offset;
+
+	/**
+	 * Copy the content between cipher offset and auth offset for generating
+	 * correct digest.
+	 */
+	if (op->sym->cipher.data.offset > op->sym->auth.data.offset)
+		memcpy(p_dst + op->sym->auth.data.offset,
+				p_src + op->sym->auth.data.offset,
+				op->sym->cipher.data.offset -
+				op->sym->auth.data.offset);
+
+	/**
+	 * Copy the content between (cipher offset + length) and (auth offset +
+	 * length) for generating correct digest
+	 */
+	cipher_end = op->sym->cipher.data.offset + op->sym->cipher.data.length;
+	auth_end = op->sym->auth.data.offset + op->sym->auth.data.length;
+	if (cipher_end < auth_end)
+		memcpy(p_dst + cipher_end, p_src + cipher_end,
+				auth_end - cipher_end);
+
+	/**
+	 * Since intel-ipsec-mb only supports positive values,
+	 * we need to deduct the correct offset between src and dst.
+	 */
+
+	return u_src < u_dst ? (u_dst - u_src) :
+			(UINT64_MAX - u_src + u_dst + 1);
+}
+
 /**
  * Process a crypto operation and complete a JOB_AES_HMAC job structure for
  * submission to the multi buffer library for processing.
@@ -708,7 +818,7 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp,
 {
 	struct rte_mbuf *m_src = op->sym->m_src, *m_dst;
 	struct aesni_mb_session *session;
-	uint16_t m_offset = 0;
+	uint32_t m_offset, oop;
 
 	session = get_session(qp, op);
 	if (session == NULL) {
@@ -760,8 +870,16 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp,
 		break;
 
 	case AES_GMAC:
-		job->u.GCM.aad = op->sym->aead.aad.data;
-		job->u.GCM.aad_len_in_bytes = session->aead.aad_len;
+		if (session->cipher.mode == GCM) {
+			job->u.GCM.aad = op->sym->aead.aad.data;
+			job->u.GCM.aad_len_in_bytes = session->aead.aad_len;
+		} else {
+			/* For GMAC */
+			job->u.GCM.aad = rte_pktmbuf_mtod_offset(m_src,
+					uint8_t *, op->sym->auth.data.offset);
+			job->u.GCM.aad_len_in_bytes = op->sym->auth.data.length;
+			job->cipher_mode = GCM;
+		}
 		job->aes_enc_key_expanded = &session->cipher.gcm_key;
 		job->aes_dec_key_expanded = &session->cipher.gcm_key;
 		break;
@@ -783,37 +901,34 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp,
 		}
 	}
 
-	/* Mutable crypto operation parameters */
-	if (op->sym->m_dst) {
-		m_src = m_dst = op->sym->m_dst;
-
-		/* append space for output data to mbuf */
-		char *odata = rte_pktmbuf_append(m_dst,
-				rte_pktmbuf_data_len(op->sym->m_src));
-		if (odata == NULL) {
-			AESNI_MB_LOG(ERR, "failed to allocate space in destination "
-					"mbuf for source data");
-			op->status = RTE_CRYPTO_OP_STATUS_ERROR;
-			return -1;
-		}
-
-		memcpy(odata, rte_pktmbuf_mtod(op->sym->m_src, void*),
-				rte_pktmbuf_data_len(op->sym->m_src));
-	} else {
+	if (!op->sym->m_dst) {
+		/* in-place operation */
 		m_dst = m_src;
-		if (job->hash_alg == AES_CCM || job->hash_alg == AES_GMAC)
-			m_offset = op->sym->aead.data.offset;
-		else
-			m_offset = op->sym->cipher.data.offset;
+		oop = 0;
+	} else if (op->sym->m_dst == op->sym->m_src) {
+		/* in-place operation */
+		m_dst = m_src;
+		oop = 0;
+	} else {
+		/* out-of-place operation */
+		m_dst = op->sym->m_dst;
+		oop = 1;
 	}
 
+	if (job->hash_alg == AES_CCM || (job->hash_alg == AES_GMAC &&
+			session->cipher.mode == GCM))
+		m_offset = op->sym->aead.data.offset;
+	else
+		m_offset = op->sym->cipher.data.offset;
+
 	/* Set digest output location */
 	if (job->hash_alg != NULL_HASH &&
 			session->auth.operation == RTE_CRYPTO_AUTH_OP_VERIFY) {
 		job->auth_tag_output = qp->temp_digests[*digest_idx];
 		*digest_idx = (*digest_idx + 1) % MAX_JOBS;
 	} else {
-		if (job->hash_alg == AES_CCM || job->hash_alg == AES_GMAC)
+		if (job->hash_alg == AES_CCM || (job->hash_alg == AES_GMAC &&
+				session->cipher.mode == GCM))
 			job->auth_tag_output = op->sym->aead.digest.data;
 		else
 			job->auth_tag_output = op->sym->auth.digest.data;
@@ -834,7 +949,7 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp,
 	/* Set IV parameters */
 	job->iv_len_in_bytes = session->iv.length;
 
-	/* Data  Parameter */
+	/* Data Parameters */
 	job->src = rte_pktmbuf_mtod(m_src, uint8_t *);
 	job->dst = rte_pktmbuf_mtod_offset(m_dst, uint8_t *, m_offset);
 
@@ -851,11 +966,24 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp,
 		break;
 
 	case AES_GMAC:
-		job->cipher_start_src_offset_in_bytes =
-				op->sym->aead.data.offset;
-		job->hash_start_src_offset_in_bytes = op->sym->aead.data.offset;
-		job->msg_len_to_cipher_in_bytes = op->sym->aead.data.length;
-		job->msg_len_to_hash_in_bytes = job->msg_len_to_cipher_in_bytes;
+		if (session->cipher.mode == GCM) {
+			job->cipher_start_src_offset_in_bytes =
+					op->sym->aead.data.offset;
+			job->hash_start_src_offset_in_bytes =
+					op->sym->aead.data.offset;
+			job->msg_len_to_cipher_in_bytes =
+					op->sym->aead.data.length;
+			job->msg_len_to_hash_in_bytes =
+					op->sym->aead.data.length;
+		} else {
+			job->cipher_start_src_offset_in_bytes =
+					op->sym->auth.data.offset;
+			job->hash_start_src_offset_in_bytes =
+					op->sym->auth.data.offset;
+			job->msg_len_to_cipher_in_bytes = 0;
+			job->msg_len_to_hash_in_bytes = 0;
+		}
+
 		job->iv = rte_crypto_op_ctod_offset(op, uint8_t *,
 				session->iv.offset);
 		break;
@@ -865,7 +993,8 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp,
 				op->sym->cipher.data.offset;
 		job->msg_len_to_cipher_in_bytes = op->sym->cipher.data.length;
 
-		job->hash_start_src_offset_in_bytes = op->sym->auth.data.offset;
+		job->hash_start_src_offset_in_bytes = auth_start_offset(op,
+				session, oop);
 		job->msg_len_to_hash_in_bytes = op->sym->auth.data.length;
 
 		job->iv = rte_crypto_op_ctod_offset(op, uint8_t *,
@@ -879,26 +1008,18 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp,
 }
 
 static inline void
-verify_digest(JOB_AES_HMAC *job, struct rte_crypto_op *op,
-		struct aesni_mb_session *sess)
+verify_digest(JOB_AES_HMAC *job, void *digest, uint16_t len, uint8_t *status)
 {
 	/* Verify digest if required */
-	if (job->hash_alg == AES_CCM || job->hash_alg == AES_GMAC) {
-		if (memcmp(job->auth_tag_output, op->sym->aead.digest.data,
-				sess->auth.req_digest_len) != 0)
-			op->status = RTE_CRYPTO_OP_STATUS_AUTH_FAILED;
-	} else {
-		if (memcmp(job->auth_tag_output, op->sym->auth.digest.data,
-				sess->auth.req_digest_len) != 0)
-			op->status = RTE_CRYPTO_OP_STATUS_AUTH_FAILED;
-	}
+	if (memcmp(job->auth_tag_output, digest, len) != 0)
+		*status = RTE_CRYPTO_OP_STATUS_AUTH_FAILED;
 }
 
 static inline void
 generate_digest(JOB_AES_HMAC *job, struct rte_crypto_op *op,
 		struct aesni_mb_session *sess)
 {
-	/* No extra copy neeed */
+	/* No extra copy needed */
 	if (likely(sess->auth.req_digest_len == sess->auth.gen_digest_len))
 		return;
 
@@ -933,13 +1054,24 @@ post_process_mb_job(struct aesni_mb_qp *qp, JOB_AES_HMAC *job)
 		case STS_COMPLETED:
 			op->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
 
-			if (job->hash_alg != NULL_HASH) {
-				if (sess->auth.operation ==
-						RTE_CRYPTO_AUTH_OP_VERIFY)
-					verify_digest(job, op, sess);
+			if (job->hash_alg == NULL_HASH)
+				break;
+
+			if (sess->auth.operation == RTE_CRYPTO_AUTH_OP_VERIFY) {
+				if (job->hash_alg == AES_CCM ||
+					(job->hash_alg == AES_GMAC &&
+						sess->cipher.mode == GCM))
+					verify_digest(job,
+						op->sym->aead.digest.data,
+						sess->auth.req_digest_len,
+						&op->status);
 				else
-					generate_digest(job, op, sess);
-			}
+					verify_digest(job,
+						op->sym->auth.digest.data,
+						sess->auth.req_digest_len,
+						&op->status);
+			} else
+				generate_digest(job, op, sess);
 			break;
 		default:
 			op->status = RTE_CRYPTO_OP_STATUS_ERROR;
@@ -989,7 +1121,7 @@ handle_completed_jobs(struct aesni_mb_qp *qp, JOB_AES_HMAC *job,
 		if (processed_jobs == nb_ops)
 			break;
 
-		job = (*qp->op_fns->job.get_completed_job)(qp->mb_mgr);
+		job = IMB_GET_COMPLETED_JOB(qp->mb_mgr);
 	}
 
 	return processed_jobs;
@@ -1002,7 +1134,7 @@ flush_mb_mgr(struct aesni_mb_qp *qp, struct rte_crypto_op **ops,
 	int processed_ops = 0;
 
 	/* Flush the remaining jobs */
-	JOB_AES_HMAC *job = (*qp->op_fns->job.flush_job)(qp->mb_mgr);
+	JOB_AES_HMAC *job = IMB_FLUSH_JOB(qp->mb_mgr);
 
 	if (job)
 		processed_ops += handle_completed_jobs(qp, job,
@@ -1042,7 +1174,7 @@ aesni_mb_pmd_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	uint8_t digest_idx = qp->digest_idx;
 	do {
 		/* Get next free mb job struct from mb manager */
-		job = (*qp->op_fns->job.get_next)(qp->mb_mgr);
+		job = IMB_GET_NEXT_JOB(qp->mb_mgr);
 		if (unlikely(job == NULL)) {
 			/* if no free mb job structs we need to flush mb_mgr */
 			processed_jobs += flush_mb_mgr(qp,
@@ -1052,7 +1184,7 @@ aesni_mb_pmd_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
 			if (nb_ops == processed_jobs)
 				break;
 
-			job = (*qp->op_fns->job.get_next)(qp->mb_mgr);
+			job = IMB_GET_NEXT_JOB(qp->mb_mgr);
 		}
 
 		/*
@@ -1072,8 +1204,11 @@ aesni_mb_pmd_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
 		}
 
 		/* Submit job to multi-buffer for processing */
-		job = (*qp->op_fns->job.submit)(qp->mb_mgr);
-
+#ifdef RTE_LIBRTE_PMD_AESNI_MB_DEBUG
+		job = IMB_SUBMIT_JOB(qp->mb_mgr);
+#else
+		job = IMB_SUBMIT_JOB_NOCHECK(qp->mb_mgr);
+#endif
 		/*
 		 * If submit returns a processed job then handle it,
 		 * before submitting subsequent jobs
@@ -1105,12 +1240,7 @@ cryptodev_aesni_mb_create(const char *name,
 	struct rte_cryptodev *dev;
 	struct aesni_mb_private *internals;
 	enum aesni_mb_vector_mode vector_mode;
-
-	/* Check CPU for support for AES instruction set */
-	if (!rte_cpu_get_flag_enabled(RTE_CPUFLAG_AES)) {
-		AESNI_MB_LOG(ERR, "AES instructions not supported by CPU");
-		return -EFAULT;
-	}
+	MB_MGR *mb_mgr;
 
 	dev = rte_cryptodev_pmd_create(name, &vdev->device, init_params);
 	if (dev == NULL) {
@@ -1137,23 +1267,38 @@ cryptodev_aesni_mb_create(const char *name,
 
 	dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
 			RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING |
-			RTE_CRYPTODEV_FF_CPU_AESNI;
+			RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT;
+
+	/* Check CPU for support for AES instruction set */
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AES))
+		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AESNI;
+	else
+		AESNI_MB_LOG(WARNING, "AES instructions not supported by CPU");
+
+	mb_mgr = alloc_mb_mgr(0);
+	if (mb_mgr == NULL)
+		return -ENOMEM;
 
 	switch (vector_mode) {
 	case RTE_AESNI_MB_SSE:
 		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_SSE;
+		init_mb_mgr_sse(mb_mgr);
 		break;
 	case RTE_AESNI_MB_AVX:
 		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX;
+		init_mb_mgr_avx(mb_mgr);
 		break;
 	case RTE_AESNI_MB_AVX2:
 		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX2;
+		init_mb_mgr_avx2(mb_mgr);
 		break;
 	case RTE_AESNI_MB_AVX512:
 		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX512;
+		init_mb_mgr_avx512(mb_mgr);
 		break;
 	default:
-		break;
+		AESNI_MB_LOG(ERR, "Unsupported vector mode %u\n", vector_mode);
+		goto error_exit;
 	}
 
 	/* Set vector instructions mode supported */
@@ -1161,15 +1306,19 @@ cryptodev_aesni_mb_create(const char *name,
 
 	internals->vector_mode = vector_mode;
 	internals->max_nb_queue_pairs = init_params->max_nb_queue_pairs;
+	internals->mb_mgr = mb_mgr;
 
-#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0)
 	AESNI_MB_LOG(INFO, "IPSec Multi-buffer library version used: %s\n",
 			imb_get_version_str());
-#else
-	AESNI_MB_LOG(INFO, "IPSec Multi-buffer library version used: 0.49.0\n");
-#endif
 
 	return 0;
+error_exit:
+	if (mb_mgr)
+		free_mb_mgr(mb_mgr);
+
+	rte_cryptodev_pmd_destroy(dev);
+
+	return -1;
 }
 
 static int
@@ -1204,6 +1353,7 @@ static int
 cryptodev_aesni_mb_remove(struct rte_vdev_device *vdev)
 {
 	struct rte_cryptodev *cryptodev;
+	struct aesni_mb_private *internals;
 	const char *name;
 
 	name = rte_vdev_device_name(vdev);
@@ -1214,6 +1364,10 @@ cryptodev_aesni_mb_remove(struct rte_vdev_device *vdev)
 	if (cryptodev == NULL)
 		return -ENODEV;
 
+	internals = cryptodev->data->dev_private;
+
+	free_mb_mgr(internals->mb_mgr);
+
 	return rte_cryptodev_pmd_destroy(cryptodev);
 }
 
diff --git a/drivers/crypto/aesni_mb/rte_aesni_mb_pmd_ops.c b/drivers/crypto/aesni_mb/rte_aesni_mb_pmd_ops.c
index f3eff2685..e58dfa33b 100644
--- a/drivers/crypto/aesni_mb/rte_aesni_mb_pmd_ops.c
+++ b/drivers/crypto/aesni_mb/rte_aesni_mb_pmd_ops.c
@@ -25,15 +25,9 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = {
 					.increment = 1
 				},
 				.digest_size = {
-#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0)
 					.min = 1,
 					.max = 16,
 					.increment = 1
-#else
-					.min = 12,
-					.max = 12,
-					.increment = 0
-#endif
 				},
 				.iv_size = { 0 }
 			}, }
@@ -48,23 +42,34 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = {
 				.block_size = 64,
 				.key_size = {
 					.min = 1,
-#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0)
 					.max = 65535,
-#else
-					.max = 64,
-#endif
 					.increment = 1
 				},
 				.digest_size = {
-#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0)
 					.min = 1,
 					.max = 20,
 					.increment = 1
-#else
-					.min = 12,
-					.max = 12,
+				},
+				.iv_size = { 0 }
+			}, }
+		}, }
+	},
+	{	/* SHA1 */
+		.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
+		{.sym = {
+			.xform_type = RTE_CRYPTO_SYM_XFORM_AUTH,
+			{.auth = {
+				.algo = RTE_CRYPTO_AUTH_SHA1,
+				.block_size = 64,
+				.key_size = {
+					.min = 0,
+					.max = 0,
 					.increment = 0
-#endif
+				},
+				.digest_size = {
+					.min = 1,
+					.max = 20,
+					.increment = 1
 				},
 				.iv_size = { 0 }
 			}, }
@@ -79,23 +84,34 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = {
 				.block_size = 64,
 				.key_size = {
 					.min = 1,
-#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0)
 					.max = 65535,
-#else
-					.max = 64,
-#endif
 					.increment = 1
 				},
 				.digest_size = {
-#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0)
 					.min = 1,
 					.max = 28,
 					.increment = 1
-#else
-					.min = 14,
-					.max = 14,
+				},
+				.iv_size = { 0 }
+			}, }
+		}, }
+	},
+	{	/* SHA224 */
+		.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
+		{.sym = {
+			.xform_type = RTE_CRYPTO_SYM_XFORM_AUTH,
+			{.auth = {
+				.algo = RTE_CRYPTO_AUTH_SHA224,
+				.block_size = 64,
+				.key_size = {
+					.min = 0,
+					.max = 0,
 					.increment = 0
-#endif
+				},
+				.digest_size = {
+					.min = 1,
+					.max = 28,
+					.increment = 1
 				},
 				.iv_size = { 0 }
 			}, }
@@ -110,23 +126,34 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = {
 				.block_size = 64,
 				.key_size = {
 					.min = 1,
-#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0)
 					.max = 65535,
-#else
-					.max = 64,
-#endif
 					.increment = 1
 				},
 				.digest_size = {
-#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0)
 					.min = 1,
 					.max = 32,
 					.increment = 1
-#else
-					.min = 16,
-					.max = 16,
+				},
+				.iv_size = { 0 }
+			}, }
+		}, }
+	},
+	{	/* SHA256 */
+		.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
+		{.sym = {
+			.xform_type = RTE_CRYPTO_SYM_XFORM_AUTH,
+			{.auth = {
+				.algo = RTE_CRYPTO_AUTH_SHA256,
+				.block_size = 64,
+				.key_size = {
+					.min = 0,
+					.max = 0,
 					.increment = 0
-#endif
+				},
+				.digest_size = {
+					.min = 1,
+					.max = 32,
+					.increment = 1
 				},
 				.iv_size = { 0 }
 			}, }
@@ -141,23 +168,34 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = {
 				.block_size = 128,
 				.key_size = {
 					.min = 1,
-#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0)
 					.max = 65535,
-#else
-					.max = 128,
-#endif
 					.increment = 1
 				},
 				.digest_size = {
-#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0)
 					.min = 1,
 					.max = 48,
 					.increment = 1
-#else
-					.min = 24,
-					.max = 24,
+				},
+				.iv_size = { 0 }
+			}, }
+		}, }
+	},
+	{	/* SHA384 */
+		.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
+		{.sym = {
+			.xform_type = RTE_CRYPTO_SYM_XFORM_AUTH,
+			{.auth = {
+				.algo = RTE_CRYPTO_AUTH_SHA384,
+				.block_size = 128,
+				.key_size = {
+					.min = 0,
+					.max = 0,
 					.increment = 0
-#endif
+				},
+				.digest_size = {
+					.min = 1,
+					.max = 48,
+					.increment = 1
 				},
 				.iv_size = { 0 }
 			}, }
@@ -172,23 +210,34 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = {
 				.block_size = 128,
 				.key_size = {
 					.min = 1,
-#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0)
 					.max = 65535,
-#else
-					.max = 128,
-#endif
 					.increment = 1
 				},
 				.digest_size = {
-#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0)
 					.min = 1,
 					.max = 64,
 					.increment = 1
-#else
-					.min = 32,
-					.max = 32,
+				},
+				.iv_size = { 0 }
+			}, }
+		}, }
+	},
+	{	/* SHA512  */
+		.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
+		{.sym = {
+			.xform_type = RTE_CRYPTO_SYM_XFORM_AUTH,
+			{.auth = {
+				.algo = RTE_CRYPTO_AUTH_SHA512,
+				.block_size = 128,
+				.key_size = {
+					.min = 0,
+					.max = 0,
 					.increment = 0
-#endif
+				},
+				.digest_size = {
+					.min = 1,
+					.max = 64,
+					.increment = 1
 				},
 				.iv_size = { 0 }
 			}, }
@@ -416,6 +465,31 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = {
 			}, }
 		}, }
 	},
+	{	/* AES GMAC (AUTH) */
+		.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
+		{.sym = {
+			.xform_type = RTE_CRYPTO_SYM_XFORM_AUTH,
+			{.auth = {
+				.algo = RTE_CRYPTO_AUTH_AES_GMAC,
+				.block_size = 16,
+				.key_size = {
+					.min = 16,
+					.max = 32,
+					.increment = 8
+				},
+				.digest_size = {
+					.min = 8,
+					.max = 16,
+					.increment = 4
+				},
+				.iv_size = {
+					.min = 12,
+					.max = 12,
+					.increment = 0
+				}
+			}, }
+		}, }
+	},
 	RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST()
 };
 
@@ -595,7 +669,28 @@ aesni_mb_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 		goto qp_setup_cleanup;
 	}
 
-	qp->op_fns = &job_ops[internals->vector_mode];
+	switch (internals->vector_mode) {
+	case RTE_AESNI_MB_SSE:
+		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_SSE;
+		init_mb_mgr_sse(qp->mb_mgr);
+		break;
+	case RTE_AESNI_MB_AVX:
+		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX;
+		init_mb_mgr_avx(qp->mb_mgr);
+		break;
+	case RTE_AESNI_MB_AVX2:
+		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX2;
+		init_mb_mgr_avx2(qp->mb_mgr);
+		break;
+	case RTE_AESNI_MB_AVX512:
+		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX512;
+		init_mb_mgr_avx512(qp->mb_mgr);
+		break;
+	default:
+		AESNI_MB_LOG(ERR, "Unsupported vector mode %u\n",
+				internals->vector_mode);
+		goto qp_setup_cleanup;
+	}
 
 	qp->ingress_queue = aesni_mb_pmd_qp_create_processed_ops_ring(qp,
 			qp_conf->nb_descriptors, socket_id);
@@ -613,13 +708,11 @@ aesni_mb_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	snprintf(mp_name, RTE_MEMPOOL_NAMESIZE,
 				"digest_mp_%u_%u", dev->data->dev_id, qp_id);
 
-	/* Initialise multi-buffer manager */
-	(*qp->op_fns->job.init_mgr)(qp->mb_mgr);
 	return 0;
 
 qp_setup_cleanup:
 	if (qp) {
-		if (qp->mb_mgr == NULL)
+		if (qp->mb_mgr)
 			free_mb_mgr(qp->mb_mgr);
 		rte_free(qp);
 	}
@@ -663,7 +756,7 @@ aesni_mb_pmd_sym_session_configure(struct rte_cryptodev *dev,
 		return -ENOMEM;
 	}
 
-	ret = aesni_mb_set_session_parameters(&job_ops[internals->vector_mode],
+	ret = aesni_mb_set_session_parameters(internals->mb_mgr,
 			sess_private_data, xform);
 	if (ret != 0) {
 		AESNI_MB_LOG(ERR, "failed configure session parameters");
diff --git a/drivers/crypto/aesni_mb/rte_aesni_mb_pmd_private.h b/drivers/crypto/aesni_mb/rte_aesni_mb_pmd_private.h
index d8021cdaa..6a5df942c 100644
--- a/drivers/crypto/aesni_mb/rte_aesni_mb_pmd_private.h
+++ b/drivers/crypto/aesni_mb/rte_aesni_mb_pmd_private.h
@@ -25,6 +25,7 @@ int aesni_mb_logtype_driver;
 /* Maximum length for digest */
 #define DIGEST_LENGTH_MAX 64
 static const unsigned auth_blocksize[] = {
+		[NULL_HASH]	= 0,
 		[MD5]		= 64,
 		[SHA1]		= 64,
 		[SHA_224]	= 64,
@@ -33,6 +34,13 @@ static const unsigned auth_blocksize[] = {
 		[SHA_512]	= 128,
 		[AES_XCBC]	= 16,
 		[AES_CCM]	= 16,
+		[AES_CMAC]	= 16,
+		[AES_GMAC]	= 16,
+		[PLAIN_SHA1]	= 64,
+		[PLAIN_SHA_224]	= 64,
+		[PLAIN_SHA_256]	= 64,
+		[PLAIN_SHA_384]	= 128,
+		[PLAIN_SHA_512]	= 128
 };
 
 /**
@@ -57,7 +65,13 @@ static const unsigned auth_truncated_digest_byte_lengths[] = {
 		[AES_XCBC]	= 12,
 		[AES_CMAC]	= 12,
 		[AES_CCM]	= 8,
-		[NULL_HASH]	= 0
+		[NULL_HASH]	= 0,
+		[AES_GMAC]	= 16,
+		[PLAIN_SHA1]	= 20,
+		[PLAIN_SHA_224]	= 28,
+		[PLAIN_SHA_256]	= 32,
+		[PLAIN_SHA_384]	= 48,
+		[PLAIN_SHA_512]	= 64
 };
 
 /**
@@ -82,8 +96,14 @@ static const unsigned auth_digest_byte_lengths[] = {
 		[SHA_512]	= 64,
 		[AES_XCBC]	= 16,
 		[AES_CMAC]	= 16,
+		[AES_CCM]	= 16,
 		[AES_GMAC]	= 12,
-		[NULL_HASH]		= 0
+		[NULL_HASH]	= 0,
+		[PLAIN_SHA1]	= 20,
+		[PLAIN_SHA_224]	= 28,
+		[PLAIN_SHA_256]	= 32,
+		[PLAIN_SHA_384]	= 48,
+		[PLAIN_SHA_512]	= 64
 };
 
 /**
@@ -115,6 +135,8 @@ struct aesni_mb_private {
 	/**< CPU vector instruction set mode */
 	unsigned max_nb_queue_pairs;
 	/**< Max number of queue pairs supported by device */
+	MB_MGR *mb_mgr;
+	/**< Multi-buffer instance */
 };
 
 /** AESNI Multi buffer queue pair */
@@ -123,8 +145,6 @@ struct aesni_mb_qp {
 	/**< Queue Pair Identifier */
 	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
 	/**< Unique Queue Pair Name */
-	const struct aesni_mb_op_fns *op_fns;
-	/**< Vector mode dependent pointer table of the multi-buffer APIs */
 	MB_MGR *mb_mgr;
 	/**< Multi-buffer instance */
 	struct rte_ring *ingress_queue;
@@ -238,7 +258,7 @@ struct aesni_mb_session {
  *
  */
 extern int
-aesni_mb_set_session_parameters(const struct aesni_mb_op_fns *mb_ops,
+aesni_mb_set_session_parameters(const MB_MGR *mb_mgr,
 		struct aesni_mb_session *sess,
 		const struct rte_crypto_sym_xform *xform);
 
-- 
2.24.1


^ permalink raw reply	[flat|nested] 8+ messages in thread

* [dpdk-dev] [RFC PATCH 5/5] crypto/aesni_gcm: support IPSec MB library v0.53
  2020-03-05 15:34 [dpdk-dev] [RFC PATCH 0/5] Support Intel IPSec MB v0.53 in DPDK 18.11 Pablo de Lara
                   ` (3 preceding siblings ...)
  2020-03-05 15:34 ` [dpdk-dev] [RFC PATCH 4/5] crypto/aesni_mb: support " Pablo de Lara
@ 2020-03-05 15:34 ` Pablo de Lara
  2020-03-19 14:32 ` [dpdk-dev] [RFC PATCH 0/5] Support Intel IPSec MB v0.53 in DPDK 18.11 Kevin Traynor
  5 siblings, 0 replies; 8+ messages in thread
From: Pablo de Lara @ 2020-03-05 15:34 UTC (permalink / raw)
  To: dev; +Cc: Pablo de Lara

Add support for underlying Intel IPSec Multi-buffer library v0.53.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
---
 drivers/crypto/aesni_gcm/aesni_gcm_ops.h      |  65 ++-------
 drivers/crypto/aesni_gcm/aesni_gcm_pmd.c      | 130 +++++++++++++-----
 drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c  |   4 +-
 .../crypto/aesni_gcm/aesni_gcm_pmd_private.h  |   4 +
 4 files changed, 114 insertions(+), 89 deletions(-)

diff --git a/drivers/crypto/aesni_gcm/aesni_gcm_ops.h b/drivers/crypto/aesni_gcm/aesni_gcm_ops.h
index 450616698..b2cc4002e 100644
--- a/drivers/crypto/aesni_gcm/aesni_gcm_ops.h
+++ b/drivers/crypto/aesni_gcm/aesni_gcm_ops.h
@@ -17,14 +17,15 @@ enum aesni_gcm_vector_mode {
 	RTE_AESNI_GCM_SSE,
 	RTE_AESNI_GCM_AVX,
 	RTE_AESNI_GCM_AVX2,
+	RTE_AESNI_GCM_AVX512,
 	RTE_AESNI_GCM_VECTOR_NUM
 };
 
 enum aesni_gcm_key {
-	AESNI_GCM_KEY_128,
-	AESNI_GCM_KEY_192,
-	AESNI_GCM_KEY_256,
-	AESNI_GCM_KEY_NUM
+	GCM_KEY_128 = 0,
+	GCM_KEY_192,
+	GCM_KEY_256,
+	GCM_KEY_NUM
 };
 
 
@@ -34,7 +35,7 @@ typedef void (*aesni_gcm_t)(const struct gcm_key_data *gcm_key_data,
 		const uint8_t *aad, uint64_t aad_len,
 		uint8_t *auth_tag, uint64_t auth_tag_len);
 
-typedef void (*aesni_gcm_precomp_t)(const void *key, struct gcm_key_data *gcm_data);
+typedef void (*aesni_gcm_pre_t)(const void *key, struct gcm_key_data *gcm_data);
 
 typedef void (*aesni_gcm_init_t)(const struct gcm_key_data *gcm_key_data,
 		struct gcm_context_data *gcm_ctx_data,
@@ -57,60 +58,12 @@ typedef void (*aesni_gcm_finalize_t)(const struct gcm_key_data *gcm_key_data,
 struct aesni_gcm_ops {
 	aesni_gcm_t enc;        /**< GCM encode function pointer */
 	aesni_gcm_t dec;        /**< GCM decode function pointer */
-	aesni_gcm_precomp_t precomp;    /**< GCM pre-compute */
+	aesni_gcm_pre_t pre;    /**< GCM pre-compute */
 	aesni_gcm_init_t init;
 	aesni_gcm_update_t update_enc;
 	aesni_gcm_update_t update_dec;
-	aesni_gcm_finalize_t finalize;
+	aesni_gcm_finalize_t finalize_enc;
+	aesni_gcm_finalize_t finalize_dec;
 };
 
-#define AES_GCM_FN(keylen, arch) \
-aes_gcm_enc_##keylen##_##arch,\
-aes_gcm_dec_##keylen##_##arch,\
-aes_gcm_pre_##keylen##_##arch,\
-aes_gcm_init_##keylen##_##arch,\
-aes_gcm_enc_##keylen##_update_##arch,\
-aes_gcm_dec_##keylen##_update_##arch,\
-aes_gcm_enc_##keylen##_finalize_##arch,
-
-static const struct aesni_gcm_ops gcm_ops[RTE_AESNI_GCM_VECTOR_NUM][AESNI_GCM_KEY_NUM] = {
-	[RTE_AESNI_GCM_NOT_SUPPORTED] = {
-		[AESNI_GCM_KEY_128] = {NULL},
-		[AESNI_GCM_KEY_192] = {NULL},
-		[AESNI_GCM_KEY_256] = {NULL}
-	},
-	[RTE_AESNI_GCM_SSE] = {
-		[AESNI_GCM_KEY_128] = {
-			AES_GCM_FN(128, sse)
-		},
-		[AESNI_GCM_KEY_192] = {
-			AES_GCM_FN(192, sse)
-		},
-		[AESNI_GCM_KEY_256] = {
-			AES_GCM_FN(256, sse)
-		}
-	},
-	[RTE_AESNI_GCM_AVX] = {
-		[AESNI_GCM_KEY_128] = {
-			AES_GCM_FN(128, avx_gen2)
-		},
-		[AESNI_GCM_KEY_192] = {
-			AES_GCM_FN(192, avx_gen2)
-		},
-		[AESNI_GCM_KEY_256] = {
-			AES_GCM_FN(256, avx_gen2)
-		}
-	},
-	[RTE_AESNI_GCM_AVX2] = {
-		[AESNI_GCM_KEY_128] = {
-			AES_GCM_FN(128, avx_gen4)
-		},
-		[AESNI_GCM_KEY_192] = {
-			AES_GCM_FN(192, avx_gen4)
-		},
-		[AESNI_GCM_KEY_256] = {
-			AES_GCM_FN(256, avx_gen4)
-		}
-	}
-};
 #endif /* _AESNI_GCM_OPS_H_ */
diff --git a/drivers/crypto/aesni_gcm/aesni_gcm_pmd.c b/drivers/crypto/aesni_gcm/aesni_gcm_pmd.c
index ebdf7c35a..2bda5a560 100644
--- a/drivers/crypto/aesni_gcm/aesni_gcm_pmd.c
+++ b/drivers/crypto/aesni_gcm/aesni_gcm_pmd.c
@@ -24,7 +24,7 @@ aesni_gcm_set_session_parameters(const struct aesni_gcm_ops *gcm_ops,
 	const struct rte_crypto_sym_xform *auth_xform;
 	const struct rte_crypto_sym_xform *aead_xform;
 	uint8_t key_length;
-	uint8_t *key;
+	const uint8_t *key;
 
 	/* AES-GMAC */
 	if (xform->type == RTE_CRYPTO_SYM_XFORM_AUTH) {
@@ -89,20 +89,20 @@ aesni_gcm_set_session_parameters(const struct aesni_gcm_ops *gcm_ops,
 	/* Check key length and calculate GCM pre-compute. */
 	switch (key_length) {
 	case 16:
-		sess->key = AESNI_GCM_KEY_128;
+		sess->key = GCM_KEY_128;
 		break;
 	case 24:
-		sess->key = AESNI_GCM_KEY_192;
+		sess->key = GCM_KEY_192;
 		break;
 	case 32:
-		sess->key = AESNI_GCM_KEY_256;
+		sess->key = GCM_KEY_256;
 		break;
 	default:
 		AESNI_GCM_LOG(ERR, "Invalid key length");
 		return -EINVAL;
 	}
 
-	gcm_ops[sess->key].precomp(key, &sess->gdata_key);
+	gcm_ops[sess->key].pre(key, &sess->gdata_key);
 
 	/* Digest check */
 	if (sess->req_digest_length > 16) {
@@ -195,6 +195,7 @@ process_gcm_crypto_op(struct aesni_gcm_qp *qp, struct rte_crypto_op *op,
 	uint32_t offset, data_offset, data_length;
 	uint32_t part_len, total_len, data_len;
 	uint8_t *tag;
+	unsigned int oop = 0;
 
 	if (session->op == AESNI_GCM_OP_AUTHENTICATED_ENCRYPTION ||
 			session->op == AESNI_GCM_OP_AUTHENTICATED_DECRYPTION) {
@@ -216,27 +217,30 @@ process_gcm_crypto_op(struct aesni_gcm_qp *qp, struct rte_crypto_op *op,
 		RTE_ASSERT(m_src != NULL);
 	}
 
+	src = rte_pktmbuf_mtod_offset(m_src, uint8_t *, offset);
+
 	data_len = m_src->data_len - offset;
 	part_len = (data_len < data_length) ? data_len :
 			data_length;
 
-	/* Destination buffer is required when segmented source buffer */
-	RTE_ASSERT((part_len == data_length) ||
-			((part_len != data_length) &&
-					(sym_op->m_dst != NULL)));
-	/* Segmented destination buffer is not supported */
 	RTE_ASSERT((sym_op->m_dst == NULL) ||
 			((sym_op->m_dst != NULL) &&
 					rte_pktmbuf_is_contiguous(sym_op->m_dst)));
 
-
-	dst = sym_op->m_dst ?
-			rte_pktmbuf_mtod_offset(sym_op->m_dst, uint8_t *,
-					data_offset) :
-			rte_pktmbuf_mtod_offset(sym_op->m_src, uint8_t *,
+	/* In-place */
+	if (sym_op->m_dst == NULL || (sym_op->m_dst == sym_op->m_src))
+		dst = src;
+	/* Out-of-place */
+	else {
+		oop = 1;
+		/*
+		 * Segmented destination buffer is not supported if operation is
+		 * Out-of-place
+		 */
+		RTE_ASSERT(rte_pktmbuf_is_contiguous(sym_op->m_dst));
+		dst = rte_pktmbuf_mtod_offset(sym_op->m_dst, uint8_t *,
 					data_offset);
-
-	src = rte_pktmbuf_mtod_offset(m_src, uint8_t *, offset);
+	}
 
 	iv_ptr = rte_crypto_op_ctod_offset(op, uint8_t *,
 				session->iv.offset);
@@ -254,12 +258,15 @@ process_gcm_crypto_op(struct aesni_gcm_qp *qp, struct rte_crypto_op *op,
 		total_len = data_length - part_len;
 
 		while (total_len) {
-			dst += part_len;
 			m_src = m_src->next;
 
 			RTE_ASSERT(m_src != NULL);
 
 			src = rte_pktmbuf_mtod(m_src, uint8_t *);
+			if (oop)
+				dst += part_len;
+			else
+				dst = src;
 			part_len = (m_src->data_len < total_len) ?
 					m_src->data_len : total_len;
 
@@ -274,7 +281,7 @@ process_gcm_crypto_op(struct aesni_gcm_qp *qp, struct rte_crypto_op *op,
 		else
 			tag = sym_op->aead.digest.data;
 
-		qp->ops[session->key].finalize(&session->gdata_key,
+		qp->ops[session->key].finalize_enc(&session->gdata_key,
 				&qp->gdata_ctx,
 				tag,
 				session->gen_digest_length);
@@ -291,12 +298,15 @@ process_gcm_crypto_op(struct aesni_gcm_qp *qp, struct rte_crypto_op *op,
 		total_len = data_length - part_len;
 
 		while (total_len) {
-			dst += part_len;
 			m_src = m_src->next;
 
 			RTE_ASSERT(m_src != NULL);
 
 			src = rte_pktmbuf_mtod(m_src, uint8_t *);
+			if (oop)
+				dst += part_len;
+			else
+				dst = src;
 			part_len = (m_src->data_len < total_len) ?
 					m_src->data_len : total_len;
 
@@ -308,7 +318,7 @@ process_gcm_crypto_op(struct aesni_gcm_qp *qp, struct rte_crypto_op *op,
 		}
 
 		tag = qp->temp_digest;
-		qp->ops[session->key].finalize(&session->gdata_key,
+		qp->ops[session->key].finalize_dec(&session->gdata_key,
 				&qp->gdata_ctx,
 				tag,
 				session->gen_digest_length);
@@ -322,7 +332,7 @@ process_gcm_crypto_op(struct aesni_gcm_qp *qp, struct rte_crypto_op *op,
 			tag = qp->temp_digest;
 		else
 			tag = sym_op->auth.digest.data;
-		qp->ops[session->key].finalize(&session->gdata_key,
+		qp->ops[session->key].finalize_enc(&session->gdata_key,
 				&qp->gdata_ctx,
 				tag,
 				session->gen_digest_length);
@@ -338,7 +348,7 @@ process_gcm_crypto_op(struct aesni_gcm_qp *qp, struct rte_crypto_op *op,
 		 * the bytes passed.
 		 */
 		tag = qp->temp_digest;
-		qp->ops[session->key].finalize(&session->gdata_key,
+		qp->ops[session->key].finalize_enc(&session->gdata_key,
 				&qp->gdata_ctx,
 				tag,
 				session->gen_digest_length);
@@ -487,12 +497,8 @@ aesni_gcm_create(const char *name,
 	struct rte_cryptodev *dev;
 	struct aesni_gcm_private *internals;
 	enum aesni_gcm_vector_mode vector_mode;
+	MB_MGR *mb_mgr;
 
-	/* Check CPU for support for AES instruction set */
-	if (!rte_cpu_get_flag_enabled(RTE_CPUFLAG_AES)) {
-		AESNI_GCM_LOG(ERR, "AES instructions not supported by CPU");
-		return -EFAULT;
-	}
 	dev = rte_cryptodev_pmd_create(name, &vdev->device, init_params);
 	if (dev == NULL) {
 		AESNI_GCM_LOG(ERR, "driver %s: create failed",
@@ -501,7 +507,9 @@ aesni_gcm_create(const char *name,
 	}
 
 	/* Check CPU for supported vector instruction set */
-	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2))
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F))
+		vector_mode = RTE_AESNI_GCM_AVX512;
+	else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2))
 		vector_mode = RTE_AESNI_GCM_AVX2;
 	else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX))
 		vector_mode = RTE_AESNI_GCM_AVX;
@@ -517,27 +525,74 @@ aesni_gcm_create(const char *name,
 
 	dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
 			RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING |
-			RTE_CRYPTODEV_FF_CPU_AESNI |
+			RTE_CRYPTODEV_FF_IN_PLACE_SGL |
 			RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT |
 			RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT;
 
+	/* Check CPU for support for AES instruction set */
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AES))
+		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AESNI;
+	else
+		AESNI_GCM_LOG(WARNING, "AES instructions not supported by CPU");
+
+	mb_mgr = alloc_mb_mgr(0);
+	if (mb_mgr == NULL)
+		return -ENOMEM;
+
 	switch (vector_mode) {
 	case RTE_AESNI_GCM_SSE:
 		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_SSE;
+		init_mb_mgr_sse(mb_mgr);
 		break;
 	case RTE_AESNI_GCM_AVX:
 		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX;
+		init_mb_mgr_avx(mb_mgr);
 		break;
 	case RTE_AESNI_GCM_AVX2:
 		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX2;
+		init_mb_mgr_avx2(mb_mgr);
 		break;
-	default:
+	case RTE_AESNI_GCM_AVX512:
+		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX2;
+		init_mb_mgr_avx512(mb_mgr);
 		break;
+	default:
+		AESNI_GCM_LOG(ERR, "Unsupported vector mode %u\n", vector_mode);
+		goto error_exit;
 	}
 
 	internals = dev->data->dev_private;
 
 	internals->vector_mode = vector_mode;
+	internals->mb_mgr = mb_mgr;
+
+	/* Set arch independent function pointers, based on key size */
+	internals->ops[GCM_KEY_128].enc = mb_mgr->gcm128_enc;
+	internals->ops[GCM_KEY_128].dec = mb_mgr->gcm128_dec;
+	internals->ops[GCM_KEY_128].pre = mb_mgr->gcm128_pre;
+	internals->ops[GCM_KEY_128].init = mb_mgr->gcm128_init;
+	internals->ops[GCM_KEY_128].update_enc = mb_mgr->gcm128_enc_update;
+	internals->ops[GCM_KEY_128].update_dec = mb_mgr->gcm128_dec_update;
+	internals->ops[GCM_KEY_128].finalize_enc = mb_mgr->gcm128_enc_finalize;
+	internals->ops[GCM_KEY_128].finalize_dec = mb_mgr->gcm128_dec_finalize;
+
+	internals->ops[GCM_KEY_192].enc = mb_mgr->gcm192_enc;
+	internals->ops[GCM_KEY_192].dec = mb_mgr->gcm192_dec;
+	internals->ops[GCM_KEY_192].pre = mb_mgr->gcm192_pre;
+	internals->ops[GCM_KEY_192].init = mb_mgr->gcm192_init;
+	internals->ops[GCM_KEY_192].update_enc = mb_mgr->gcm192_enc_update;
+	internals->ops[GCM_KEY_192].update_dec = mb_mgr->gcm192_dec_update;
+	internals->ops[GCM_KEY_192].finalize_enc = mb_mgr->gcm192_enc_finalize;
+	internals->ops[GCM_KEY_192].finalize_dec = mb_mgr->gcm192_dec_finalize;
+
+	internals->ops[GCM_KEY_256].enc = mb_mgr->gcm256_enc;
+	internals->ops[GCM_KEY_256].dec = mb_mgr->gcm256_dec;
+	internals->ops[GCM_KEY_256].pre = mb_mgr->gcm256_pre;
+	internals->ops[GCM_KEY_256].init = mb_mgr->gcm256_init;
+	internals->ops[GCM_KEY_256].update_enc = mb_mgr->gcm256_enc_update;
+	internals->ops[GCM_KEY_256].update_dec = mb_mgr->gcm256_dec_update;
+	internals->ops[GCM_KEY_256].finalize_enc = mb_mgr->gcm256_enc_finalize;
+	internals->ops[GCM_KEY_256].finalize_dec = mb_mgr->gcm256_dec_finalize;
 
 	internals->max_nb_queue_pairs = init_params->max_nb_queue_pairs;
 
@@ -549,6 +604,14 @@ aesni_gcm_create(const char *name,
 #endif
 
 	return 0;
+
+error_exit:
+	if (mb_mgr)
+		free_mb_mgr(mb_mgr);
+
+	rte_cryptodev_pmd_destroy(dev);
+
+	return -1;
 }
 
 static int
@@ -576,6 +639,7 @@ static int
 aesni_gcm_remove(struct rte_vdev_device *vdev)
 {
 	struct rte_cryptodev *cryptodev;
+	struct aesni_gcm_private *internals;
 	const char *name;
 
 	name = rte_vdev_device_name(vdev);
@@ -586,6 +650,10 @@ aesni_gcm_remove(struct rte_vdev_device *vdev)
 	if (cryptodev == NULL)
 		return -ENODEV;
 
+	internals = cryptodev->data->dev_private;
+
+	free_mb_mgr(internals->mb_mgr);
+
 	return rte_cryptodev_pmd_destroy(cryptodev);
 }
 
diff --git a/drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c b/drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c
index c343a393f..f599fc3f7 100644
--- a/drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c
+++ b/drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c
@@ -222,7 +222,7 @@ aesni_gcm_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	if (aesni_gcm_pmd_qp_set_unique_name(dev, qp))
 		goto qp_setup_cleanup;
 
-	qp->ops = (const struct aesni_gcm_ops *)gcm_ops[internals->vector_mode];
+	qp->ops = (const struct aesni_gcm_ops *)internals->ops;
 
 	qp->processed_pkts = aesni_gcm_pmd_qp_create_processed_pkts_ring(qp,
 			qp_conf->nb_descriptors, socket_id);
@@ -277,7 +277,7 @@ aesni_gcm_pmd_sym_session_configure(struct rte_cryptodev *dev __rte_unused,
 				"Couldn't get object from session mempool");
 		return -ENOMEM;
 	}
-	ret = aesni_gcm_set_session_parameters(gcm_ops[internals->vector_mode],
+	ret = aesni_gcm_set_session_parameters(internals->ops,
 				sess_private_data, xform);
 	if (ret != 0) {
 		AESNI_GCM_LOG(ERR, "failed configure session parameters");
diff --git a/drivers/crypto/aesni_gcm/aesni_gcm_pmd_private.h b/drivers/crypto/aesni_gcm/aesni_gcm_pmd_private.h
index 92b041354..fd43df3cf 100644
--- a/drivers/crypto/aesni_gcm/aesni_gcm_pmd_private.h
+++ b/drivers/crypto/aesni_gcm/aesni_gcm_pmd_private.h
@@ -35,6 +35,10 @@ struct aesni_gcm_private {
 	/**< Vector mode */
 	unsigned max_nb_queue_pairs;
 	/**< Max number of queue pairs supported by device */
+	MB_MGR *mb_mgr;
+	/**< Multi-buffer instance */
+	struct aesni_gcm_ops ops[GCM_KEY_NUM];
+	/**< Function pointer table of the gcm APIs */
 };
 
 struct aesni_gcm_qp {
-- 
2.24.1


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 0/5] Support Intel IPSec MB v0.53 in DPDK 18.11
  2020-03-05 15:34 [dpdk-dev] [RFC PATCH 0/5] Support Intel IPSec MB v0.53 in DPDK 18.11 Pablo de Lara
                   ` (4 preceding siblings ...)
  2020-03-05 15:34 ` [dpdk-dev] [RFC PATCH 5/5] crypto/aesni_gcm: " Pablo de Lara
@ 2020-03-19 14:32 ` Kevin Traynor
  2020-03-20 15:14   ` De Lara Guarch, Pablo
  5 siblings, 1 reply; 8+ messages in thread
From: Kevin Traynor @ 2020-03-19 14:32 UTC (permalink / raw)
  To: Pablo de Lara, stable; +Cc: Luca Boccassi, dev

+stable@dpdk.org

Hi Pablo,

Sorry, but I'm not comfortable with these patches for 18.11.

On 05/03/2020 15:34, Pablo de Lara wrote:
> This patchset adds support to the following crypto PMDs to use
> Intel IPSec MB v0.53, in DPDK v18.11:
> - AESNI MB PMD: had support up to v0.52, extending to v0.53
> - AESNI GCM PMD: had support up to v0.52, extending to v0.53

For the AES ones, it looks like it is removing support for <0.50? I'm
also not clear if it's changing the default or not. The patches are
very intrusive too. My concern is that it might break backwards
compatibility and introduce regressions.

> - SNOW3G PMD: linking now to IPSec MB v0.53, instead of libsso
> - ZUC PMD: linking now to IPSec MB v0.53, instead of libsso
> - KASUMI PMD: linking now to IPSec MB v0.53, instead of libsso
> 

Aren't these the ones we discussed offline? If so, Luca and I both
commented that this will break build for existing users and is not a
backwards compatible change that could be put on stable branches.

> Pablo de Lara (5):
>   crypto/zuc: use IPSec MB library v0.53
>   crypto/snow3g: use IPSec MB library v0.53
>   crypto/kasumi: use IPSec MB library v0.53
>   crypto/aesni_mb: support IPSec MB library v0.53
>   crypto/aesni_gcm: support IPSec MB library v0.53
> 
>  devtools/test-build.sh                        |  14 +-
>  doc/guides/cryptodevs/kasumi.rst              |  62 +--
>  doc/guides/cryptodevs/snow3g.rst              |  58 ++-
>  doc/guides/cryptodevs/zuc.rst                 |  52 +-
>  drivers/crypto/aesni_gcm/aesni_gcm_ops.h      |  65 +--
>  drivers/crypto/aesni_gcm/aesni_gcm_pmd.c      | 130 +++--
>  drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c  |   4 +-
>  .../crypto/aesni_gcm/aesni_gcm_pmd_private.h  |   4 +
>  drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c    | 476 ++++++++++++------
>  .../crypto/aesni_mb/rte_aesni_mb_pmd_ops.c    | 205 +++++---
>  .../aesni_mb/rte_aesni_mb_pmd_private.h       |  30 +-
>  drivers/crypto/kasumi/Makefile                |  26 +-
>  drivers/crypto/kasumi/meson.build             |  11 +-
>  drivers/crypto/kasumi/rte_kasumi_pmd.c        |  79 +--
>  drivers/crypto/kasumi/rte_kasumi_pmd_ops.c    |   8 +-
>  .../crypto/kasumi/rte_kasumi_pmd_private.h    |  12 +-
>  drivers/crypto/snow3g/Makefile                |  29 +-
>  drivers/crypto/snow3g/meson.build             |  21 +
>  drivers/crypto/snow3g/rte_snow3g_pmd.c        |  79 +--
>  drivers/crypto/snow3g/rte_snow3g_pmd_ops.c    |   8 +-
>  .../crypto/snow3g/rte_snow3g_pmd_private.h    |  12 +-
>  drivers/crypto/zuc/Makefile                   |  28 +-
>  drivers/crypto/zuc/meson.build                |  13 +-
>  drivers/crypto/zuc/rte_zuc_pmd.c              |  58 ++-
>  drivers/crypto/zuc/rte_zuc_pmd_ops.c          |   2 +
>  drivers/crypto/zuc/rte_zuc_pmd_private.h      |   6 +-
>  mk/rte.app.mk                                 |   6 +-
>  27 files changed, 972 insertions(+), 526 deletions(-)

                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This is a *lot* of code churn

>  create mode 100644 drivers/crypto/snow3g/meson.build
> 

thanks,
Kevin.


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 0/5] Support Intel IPSec MB v0.53 in DPDK 18.11
  2020-03-19 14:32 ` [dpdk-dev] [RFC PATCH 0/5] Support Intel IPSec MB v0.53 in DPDK 18.11 Kevin Traynor
@ 2020-03-20 15:14   ` De Lara Guarch, Pablo
  0 siblings, 0 replies; 8+ messages in thread
From: De Lara Guarch, Pablo @ 2020-03-20 15:14 UTC (permalink / raw)
  To: Kevin Traynor, stable; +Cc: Luca Boccassi, dev

Hi Kevin,

> -----Original Message-----
> From: Kevin Traynor <ktraynor@redhat.com>
> Sent: Thursday, March 19, 2020 2:32 PM
> To: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; stable@dpdk.org
> Cc: Luca Boccassi <bluca@debian.org>; dev@dpdk.org
> Subject: Re: [dpdk-dev] [RFC PATCH 0/5] Support Intel IPSec MB v0.53 in DPDK
> 18.11
> 
> +stable@dpdk.org
> 
> Hi Pablo,
> 
> Sorry, but I'm not comfortable with these patches for 18.11.
> 
> On 05/03/2020 15:34, Pablo de Lara wrote:
> > This patchset adds support to the following crypto PMDs to use Intel
> > IPSec MB v0.53, in DPDK v18.11:
> > - AESNI MB PMD: had support up to v0.52, extending to v0.53
> > - AESNI GCM PMD: had support up to v0.52, extending to v0.53
> 
> For the AES ones, it looks like it is removing support for <0.50? I'm also not clear
> if it's changing the default or not. The patches are very intrusive too. My
> concern is that it might break backwards compatibility and introduce
> regressions.
> 
> > - SNOW3G PMD: linking now to IPSec MB v0.53, instead of libsso
> > - ZUC PMD: linking now to IPSec MB v0.53, instead of libsso
> > - KASUMI PMD: linking now to IPSec MB v0.53, instead of libsso
> >
> 
> Aren't these the ones we discussed offline? If so, Luca and I both commented
> that this will break build for existing users and is not a backwards compatible
> change that could be put on stable branches.

No problem, we can park these patches and possibly explore in the future a way to introduce them into 18.11
(probably not, knowing that 18.11 support will be dropped in a few months).
Having these patches available publicly is OK for us, even if they don't get merged.

Thanks,
Pablo

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2020-03-20 15:15 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-03-05 15:34 [dpdk-dev] [RFC PATCH 0/5] Support Intel IPSec MB v0.53 in DPDK 18.11 Pablo de Lara
2020-03-05 15:34 ` [dpdk-dev] [RFC PATCH 1/5] crypto/zuc: use IPSec MB library v0.53 Pablo de Lara
2020-03-05 15:34 ` [dpdk-dev] [RFC PATCH 2/5] crypto/snow3g: " Pablo de Lara
2020-03-05 15:34 ` [dpdk-dev] [RFC PATCH 3/5] crypto/kasumi: " Pablo de Lara
2020-03-05 15:34 ` [dpdk-dev] [RFC PATCH 4/5] crypto/aesni_mb: support " Pablo de Lara
2020-03-05 15:34 ` [dpdk-dev] [RFC PATCH 5/5] crypto/aesni_gcm: " Pablo de Lara
2020-03-19 14:32 ` [dpdk-dev] [RFC PATCH 0/5] Support Intel IPSec MB v0.53 in DPDK 18.11 Kevin Traynor
2020-03-20 15:14   ` De Lara Guarch, Pablo

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).