* [dpdk-dev] [PATCH 00/15] Add support for fourth generation of Intel QuickAssist Technology devices
@ 2021-05-31 14:10 Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 01/15] common/qat: rework qp per service function Arek Kusztal
` (14 more replies)
0 siblings, 15 replies; 16+ messages in thread
From: Arek Kusztal @ 2021-05-31 14:10 UTC (permalink / raw)
To: dev; +Cc: gakhil, fiona.trahe, roy.fan.zhang, Arek Kusztal
This patchset adds support for fourth generation (GEN4) of Intel QuickAssist Technology (QAT) devices.
Symmetric crypto PMD is enabled with following algorithms:
* AES-CBC
* AES-CMAC
* AES-XCBC MAC
* NULL (auth, cipher)
* SHA1-HMAC
* SHA2-HMAC (224, 256, 384, 512)
* Chacha20-Poly1305
* AES-CCM
* AES-GCM
Other services (compression, asymmetric crypto) are not added with this patchset.
Adam Dybkowski (2):
crypto/qat: enable RAW API on QAT GEN1-3 only
test/crypto: check if RAW API is supported
Arek Kusztal (12):
common/qat: rework qp per service function
crypto/qat: add fourth generation qat devices support
crypto/qat: enable gen4 legacy algorithms
crypto/qat: add fourth generation ucs slice type, add ctr mode
crypto/qat: rename content descriptor functions
crypto/qat: add legacy gcm and ccm
crypto/qat: rework init common header function
crypto/qat: add aes gcm in ucs spc mode
crypto/qat: add chacha-poly in ucs spc mode
crypto/qat: add gmac in legacy mode on fourth generation
common/qat: add pf2vf communication in qat
common/qat: reset ring pairs before setting pmd
Fan Zhang (1):
crypto/qat: update raw dp api
app/test/test_cryptodev.c | 34 +-
doc/guides/cryptodevs/qat.rst | 10 +-
doc/guides/rel_notes/release_21_08.rst | 6 +
drivers/common/qat/meson.build | 1 +
drivers/common/qat/qat_adf/adf_pf2vf_msg.h | 154 ++++++
.../adf_transport_access_macros_gen4.h | 52 ++
.../adf_transport_access_macros_gen4vf.h | 48 ++
drivers/common/qat/qat_adf/icp_qat_fw_la.h | 28 ++
drivers/common/qat/qat_adf/icp_qat_hw.h | 10 +
drivers/common/qat/qat_common.h | 3 +-
drivers/common/qat/qat_device.c | 69 +++
drivers/common/qat/qat_device.h | 17 +
drivers/common/qat/qat_pf2vf.c | 80 +++
drivers/common/qat/qat_pf2vf.h | 19 +
drivers/common/qat/qat_qp.c | 246 ++++++---
drivers/common/qat/qat_qp.h | 31 +-
drivers/compress/qat/qat_comp_pmd.c | 16 +-
drivers/crypto/qat/qat_asym_pmd.c | 16 +-
drivers/crypto/qat/qat_sym.c | 57 ++-
drivers/crypto/qat/qat_sym_capabilities.h | 472 ++++++++++++++++++
drivers/crypto/qat/qat_sym_hw_dp.c | 419 ++++++++--------
drivers/crypto/qat/qat_sym_pmd.c | 53 +-
drivers/crypto/qat/qat_sym_session.c | 334 ++++++++-----
drivers/crypto/qat/qat_sym_session.h | 31 +-
24 files changed, 1731 insertions(+), 475 deletions(-)
create mode 100644 drivers/common/qat/qat_adf/adf_pf2vf_msg.h
create mode 100644 drivers/common/qat/qat_adf/adf_transport_access_macros_gen4.h
create mode 100644 drivers/common/qat/qat_adf/adf_transport_access_macros_gen4vf.h
create mode 100644 drivers/common/qat/qat_pf2vf.c
create mode 100644 drivers/common/qat/qat_pf2vf.h
--
2.25.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [dpdk-dev] [PATCH 01/15] common/qat: rework qp per service function
2021-05-31 14:10 [dpdk-dev] [PATCH 00/15] Add support for fourth generation of Intel QuickAssist Technology devices Arek Kusztal
@ 2021-05-31 14:10 ` Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 02/15] crypto/qat: add fourth generation qat devices support Arek Kusztal
` (13 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Arek Kusztal @ 2021-05-31 14:10 UTC (permalink / raw)
To: dev; +Cc: gakhil, fiona.trahe, roy.fan.zhang, Arek Kusztal
Different generations of Intel QuickAssist Technology devices may
differ in approach to allocate queues. Queue pair number function
therefore needs to be more generic.
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
---
drivers/common/qat/qat_qp.c | 15 ++++++++++-----
drivers/common/qat/qat_qp.h | 2 +-
drivers/compress/qat/qat_comp_pmd.c | 9 ++++-----
drivers/crypto/qat/qat_asym_pmd.c | 9 ++++-----
drivers/crypto/qat/qat_sym_pmd.c | 9 ++++-----
5 files changed, 23 insertions(+), 21 deletions(-)
diff --git a/drivers/common/qat/qat_qp.c b/drivers/common/qat/qat_qp.c
index 4a8078541c..aa64d2e168 100644
--- a/drivers/common/qat/qat_qp.c
+++ b/drivers/common/qat/qat_qp.c
@@ -145,14 +145,19 @@ static void adf_queue_arb_disable(struct qat_queue *txq, void *base_addr,
rte_spinlock_t *lock);
-int qat_qps_per_service(const struct qat_qp_hw_data *qp_hw_data,
+int qat_qps_per_service(struct qat_pci_device *qat_dev,
enum qat_service_type service)
{
- int i, count;
-
- for (i = 0, count = 0; i < ADF_MAX_QPS_ON_ANY_SERVICE; i++)
- if (qp_hw_data[i].service_type == service)
+ int i = 0, count = 0, max_ops_per_srv = 0;
+ const struct qat_qp_hw_data*
+ sym_hw_qps = qat_gen_config[qat_dev->qat_dev_gen]
+ .qp_hw_data[service];
+
+ max_ops_per_srv = ADF_MAX_QPS_ON_ANY_SERVICE;
+ for (; i < max_ops_per_srv; i++)
+ if (sym_hw_qps[i].service_type == service)
count++;
+
return count;
}
diff --git a/drivers/common/qat/qat_qp.h b/drivers/common/qat/qat_qp.h
index 74f7e7daee..d353e8552b 100644
--- a/drivers/common/qat/qat_qp.h
+++ b/drivers/common/qat/qat_qp.h
@@ -98,7 +98,7 @@ qat_qp_setup(struct qat_pci_device *qat_dev,
struct qat_qp_config *qat_qp_conf);
int
-qat_qps_per_service(const struct qat_qp_hw_data *qp_hw_data,
+qat_qps_per_service(struct qat_pci_device *qat_dev,
enum qat_service_type service);
int
diff --git a/drivers/compress/qat/qat_comp_pmd.c b/drivers/compress/qat/qat_comp_pmd.c
index 8de41f6b6e..6eb1ae3a21 100644
--- a/drivers/compress/qat/qat_comp_pmd.c
+++ b/drivers/compress/qat/qat_comp_pmd.c
@@ -106,6 +106,7 @@ qat_comp_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
struct qat_qp **qp_addr =
(struct qat_qp **)&(dev->data->queue_pairs[qp_id]);
struct qat_comp_dev_private *qat_private = dev->data->dev_private;
+ struct qat_pci_device *qat_dev = qat_private->qat_dev;
const struct qat_qp_hw_data *comp_hw_qps =
qat_gen_config[qat_private->qat_dev->qat_dev_gen]
.qp_hw_data[QAT_SERVICE_COMPRESSION];
@@ -117,7 +118,7 @@ qat_comp_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
if (ret < 0)
return ret;
}
- if (qp_id >= qat_qps_per_service(comp_hw_qps,
+ if (qp_id >= qat_qps_per_service(qat_dev,
QAT_SERVICE_COMPRESSION)) {
QAT_LOG(ERR, "qp_id %u invalid for this device", qp_id);
return -EINVAL;
@@ -592,13 +593,11 @@ qat_comp_dev_info_get(struct rte_compressdev *dev,
struct rte_compressdev_info *info)
{
struct qat_comp_dev_private *comp_dev = dev->data->dev_private;
- const struct qat_qp_hw_data *comp_hw_qps =
- qat_gen_config[comp_dev->qat_dev->qat_dev_gen]
- .qp_hw_data[QAT_SERVICE_COMPRESSION];
+ struct qat_pci_device *qat_dev = comp_dev->qat_dev;
if (info != NULL) {
info->max_nb_queue_pairs =
- qat_qps_per_service(comp_hw_qps,
+ qat_qps_per_service(qat_dev,
QAT_SERVICE_COMPRESSION);
info->feature_flags = dev->feature_flags;
info->capabilities = comp_dev->qat_dev_capabilities;
diff --git a/drivers/crypto/qat/qat_asym_pmd.c b/drivers/crypto/qat/qat_asym_pmd.c
index a2c8aca2c1..f0c8ed1bcf 100644
--- a/drivers/crypto/qat/qat_asym_pmd.c
+++ b/drivers/crypto/qat/qat_asym_pmd.c
@@ -54,12 +54,10 @@ static void qat_asym_dev_info_get(struct rte_cryptodev *dev,
struct rte_cryptodev_info *info)
{
struct qat_asym_dev_private *internals = dev->data->dev_private;
- const struct qat_qp_hw_data *asym_hw_qps =
- qat_gen_config[internals->qat_dev->qat_dev_gen]
- .qp_hw_data[QAT_SERVICE_ASYMMETRIC];
+ struct qat_pci_device *qat_dev = internals->qat_dev;
if (info != NULL) {
- info->max_nb_queue_pairs = qat_qps_per_service(asym_hw_qps,
+ info->max_nb_queue_pairs = qat_qps_per_service(qat_dev,
QAT_SERVICE_ASYMMETRIC);
info->feature_flags = dev->feature_flags;
info->capabilities = internals->qat_dev_capabilities;
@@ -128,6 +126,7 @@ static int qat_asym_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id,
struct qat_qp **qp_addr =
(struct qat_qp **)&(dev->data->queue_pairs[qp_id]);
struct qat_asym_dev_private *qat_private = dev->data->dev_private;
+ struct qat_pci_device *qat_dev = qat_private->qat_dev;
const struct qat_qp_hw_data *asym_hw_qps =
qat_gen_config[qat_private->qat_dev->qat_dev_gen]
.qp_hw_data[QAT_SERVICE_ASYMMETRIC];
@@ -139,7 +138,7 @@ static int qat_asym_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id,
if (ret < 0)
return ret;
}
- if (qp_id >= qat_qps_per_service(asym_hw_qps, QAT_SERVICE_ASYMMETRIC)) {
+ if (qp_id >= qat_qps_per_service(qat_dev, QAT_SERVICE_ASYMMETRIC)) {
QAT_LOG(ERR, "qp_id %u invalid for this device", qp_id);
return -EINVAL;
}
diff --git a/drivers/crypto/qat/qat_sym_pmd.c b/drivers/crypto/qat/qat_sym_pmd.c
index b9601c6c3a..549345b6fa 100644
--- a/drivers/crypto/qat/qat_sym_pmd.c
+++ b/drivers/crypto/qat/qat_sym_pmd.c
@@ -90,13 +90,11 @@ static void qat_sym_dev_info_get(struct rte_cryptodev *dev,
struct rte_cryptodev_info *info)
{
struct qat_sym_dev_private *internals = dev->data->dev_private;
- const struct qat_qp_hw_data *sym_hw_qps =
- qat_gen_config[internals->qat_dev->qat_dev_gen]
- .qp_hw_data[QAT_SERVICE_SYMMETRIC];
+ struct qat_pci_device *qat_dev = internals->qat_dev;
if (info != NULL) {
info->max_nb_queue_pairs =
- qat_qps_per_service(sym_hw_qps, QAT_SERVICE_SYMMETRIC);
+ qat_qps_per_service(qat_dev, QAT_SERVICE_SYMMETRIC);
info->feature_flags = dev->feature_flags;
info->capabilities = internals->qat_dev_capabilities;
info->driver_id = qat_sym_driver_id;
@@ -164,6 +162,7 @@ static int qat_sym_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id,
struct qat_qp **qp_addr =
(struct qat_qp **)&(dev->data->queue_pairs[qp_id]);
struct qat_sym_dev_private *qat_private = dev->data->dev_private;
+ struct qat_pci_device *qat_dev = qat_private->qat_dev;
const struct qat_qp_hw_data *sym_hw_qps =
qat_gen_config[qat_private->qat_dev->qat_dev_gen]
.qp_hw_data[QAT_SERVICE_SYMMETRIC];
@@ -175,7 +174,7 @@ static int qat_sym_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id,
if (ret < 0)
return ret;
}
- if (qp_id >= qat_qps_per_service(sym_hw_qps, QAT_SERVICE_SYMMETRIC)) {
+ if (qp_id >= qat_qps_per_service(qat_dev, QAT_SERVICE_SYMMETRIC)) {
QAT_LOG(ERR, "qp_id %u invalid for this device", qp_id);
return -EINVAL;
}
--
2.25.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [dpdk-dev] [PATCH 02/15] crypto/qat: add fourth generation qat devices support
2021-05-31 14:10 [dpdk-dev] [PATCH 00/15] Add support for fourth generation of Intel QuickAssist Technology devices Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 01/15] common/qat: rework qp per service function Arek Kusztal
@ 2021-05-31 14:10 ` Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 03/15] crypto/qat: enable gen4 legacy algorithms Arek Kusztal
` (12 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Arek Kusztal @ 2021-05-31 14:10 UTC (permalink / raw)
To: dev; +Cc: gakhil, fiona.trahe, roy.fan.zhang, Arek Kusztal
This commit adds support for generation 4 (GEN4) of
Intel QuickAssist (QAT) Technology devices.
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
---
doc/guides/cryptodevs/qat.rst | 10 +-
doc/guides/rel_notes/release_21_08.rst | 6 +
.../adf_transport_access_macros_gen4.h | 52 ++++
.../adf_transport_access_macros_gen4vf.h | 48 ++++
drivers/common/qat/qat_common.h | 3 +-
drivers/common/qat/qat_device.c | 22 ++
drivers/common/qat/qat_device.h | 5 +
drivers/common/qat/qat_qp.c | 243 +++++++++++++-----
drivers/common/qat/qat_qp.h | 29 ++-
drivers/compress/qat/qat_comp_pmd.c | 7 +-
drivers/crypto/qat/qat_asym_pmd.c | 7 +-
drivers/crypto/qat/qat_sym_pmd.c | 33 ++-
drivers/crypto/qat/qat_sym_session.c | 1 +
13 files changed, 388 insertions(+), 78 deletions(-)
create mode 100644 drivers/common/qat/qat_adf/adf_transport_access_macros_gen4.h
create mode 100644 drivers/common/qat/qat_adf/adf_transport_access_macros_gen4vf.h
diff --git a/doc/guides/cryptodevs/qat.rst b/doc/guides/cryptodevs/qat.rst
index 96f5ab6afe..960e15d10c 100644
--- a/doc/guides/cryptodevs/qat.rst
+++ b/doc/guides/cryptodevs/qat.rst
@@ -25,6 +25,7 @@ poll mode crypto driver support for the following hardware accelerator devices:
* ``Intel QuickAssist Technology 200xx``
* ``Intel QuickAssist Technology D15xx``
* ``Intel QuickAssist Technology C4xxx``
+* ``Intel QuickAssist Technology 4xxx``
Features
@@ -94,15 +95,16 @@ All the usual chains are supported and also some mixed chains:
+==================+===========+=============+==========+==========+
| NULL CIPHER | Y | 2&3 | 2&3 | Y |
+------------------+-----------+-------------+----------+----------+
- | SNOW3G UEA2 | 2&3 | Y | 2&3 | 2&3 |
+ | SNOW3G UEA2 | 2&3 | 1&2&3 | 2&3 | 2&3 |
+------------------+-----------+-------------+----------+----------+
| ZUC EEA3 | 2&3 | 2&3 | 2&3 | 2&3 |
+------------------+-----------+-------------+----------+----------+
- | AES CTR | Y | 2&3 | 2&3 | Y |
+ | AES CTR | 1&2&3 | 2&3 | 2&3 | Y |
+------------------+-----------+-------------+----------+----------+
* The combinations marked as "Y" are supported on all QAT hardware versions.
-* The combinations marked as "2&3" are supported on GEN2/GEN3 QAT hardware only.
+* The combinations marked as "2&3" are supported on GEN2 and GEN3 QAT hardware only.
+* The combinations marked as "1&2&3" are supported on GEN1, GEN2 and GEN3 QAT hardware only.
Limitations
@@ -373,6 +375,8 @@ to see the full table)
+-----+-----+-----+-----+----------+---------------+---------------+------------+--------+------+--------+--------+
| Yes | No | No | 3 | C4xxx | p | qat_c4xxx | c4xxx | 18a0 | 1 | 18a1 | 128 |
+-----+-----+-----+-----+----------+---------------+---------------+------------+--------+------+--------+--------+
+ | Yes | No | No | 4 | 4xxx | 5.11.0+ | qat_4xxx | 4xxx | 4940 | 4 | 4941 | 16 |
+ +-----+-----+-----+-----+----------+---------------+---------------+------------+--------+------+--------+--------+
* Note: Symmetric mixed crypto algorithms feature on Gen 2 works only with 01.org driver version 4.9.0+
diff --git a/doc/guides/rel_notes/release_21_08.rst b/doc/guides/rel_notes/release_21_08.rst
index a6ecfdf3ce..69ef43acf6 100644
--- a/doc/guides/rel_notes/release_21_08.rst
+++ b/doc/guides/rel_notes/release_21_08.rst
@@ -55,6 +55,12 @@ New Features
Also, make sure to start the actual text at the margin.
=======================================================
+* **Updated Intel QuickAssist PMD.**
+
+ Added fourth generation of QuickAssist Technology devices support.
+ Only symmetric crypto has been currently enabled, compression and asymmetric
+ crypto PMD will fail to create.
+
Removed Items
-------------
diff --git a/drivers/common/qat/qat_adf/adf_transport_access_macros_gen4.h b/drivers/common/qat/qat_adf/adf_transport_access_macros_gen4.h
new file mode 100644
index 0000000000..3ab873db5e
--- /dev/null
+++ b/drivers/common/qat/qat_adf/adf_transport_access_macros_gen4.h
@@ -0,0 +1,52 @@
+/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+ * Copyright(c) 2021 Intel Corporation
+ */
+
+#ifndef ADF_TRANSPORT_ACCESS_MACROS_GEN4_H
+#define ADF_TRANSPORT_ACCESS_MACROS_GEN4_H
+
+#include "adf_transport_access_macros.h"
+
+#define ADF_RINGS_PER_INT_SRCSEL_GEN4 2
+#define ADF_BANK_INT_SRC_SEL_MASK_GEN4 0x44UL
+#define ADF_BANK_INT_FLAG_CLEAR_MASK_GEN4 0x3
+#define ADF_RING_BUNDLE_SIZE_GEN4 0x2000
+#define ADF_RING_CSR_ADDR_OFFSET_GEN4 0x100000
+#define ADF_RING_CSR_RING_CONFIG_GEN4 0x1000
+#define ADF_RING_CSR_RING_LBASE_GEN4 0x1040
+#define ADF_RING_CSR_RING_UBASE_GEN4 0x1080
+
+#define BUILD_RING_BASE_ADDR_GEN4(addr, size) \
+ ((((addr) >> 6) & (0xFFFFFFFFFFFFFFFFULL << (size))) << 6)
+
+#define WRITE_CSR_RING_BASE_GEN4(csr_base_addr, bank, ring, value) \
+do { \
+ uint32_t l_base = 0, u_base = 0; \
+ l_base = (uint32_t)(value & 0xFFFFFFFF); \
+ u_base = (uint32_t)((value & 0xFFFFFFFF00000000ULL) >> 32); \
+ ADF_CSR_WR(csr_base_addr + ADF_RING_CSR_ADDR_OFFSET_GEN4, \
+ (ADF_RING_BUNDLE_SIZE_GEN4 * bank) + \
+ ADF_RING_CSR_RING_LBASE_GEN4 + (ring << 2), \
+ l_base); \
+ ADF_CSR_WR(csr_base_addr + ADF_RING_CSR_ADDR_OFFSET_GEN4, \
+ (ADF_RING_BUNDLE_SIZE_GEN4 * bank) + \
+ ADF_RING_CSR_RING_UBASE_GEN4 + (ring << 2), \
+ u_base); \
+} while (0)
+
+#define WRITE_CSR_RING_CONFIG_GEN4(csr_base_addr, bank, ring, value) \
+ ADF_CSR_WR(csr_base_addr + ADF_RING_CSR_ADDR_OFFSET_GEN4, \
+ (ADF_RING_BUNDLE_SIZE_GEN4 * bank) + \
+ ADF_RING_CSR_RING_CONFIG_GEN4 + (ring << 2), value)
+
+#define WRITE_CSR_RING_TAIL_GEN4(csr_base_addr, bank, ring, value) \
+ ADF_CSR_WR((u8 *)(csr_base_addr) + ADF_RING_CSR_ADDR_OFFSET_GEN4, \
+ (ADF_RING_BUNDLE_SIZE_GEN4 * (bank)) + \
+ ADF_RING_CSR_RING_TAIL + ((ring) << 2), value)
+
+#define WRITE_CSR_RING_HEAD_GEN4(csr_base_addr, bank, ring, value) \
+ ADF_CSR_WR((u8 *)(csr_base_addr) + ADF_RING_CSR_ADDR_OFFSET_GEN4, \
+ (ADF_RING_BUNDLE_SIZE_GEN4 * (bank)) + \
+ ADF_RING_CSR_RING_HEAD + ((ring) << 2), value)
+
+#endif
diff --git a/drivers/common/qat/qat_adf/adf_transport_access_macros_gen4vf.h b/drivers/common/qat/qat_adf/adf_transport_access_macros_gen4vf.h
new file mode 100644
index 0000000000..37e113c443
--- /dev/null
+++ b/drivers/common/qat/qat_adf/adf_transport_access_macros_gen4vf.h
@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+ * Copyright(c) 2021 Intel Corporation
+ */
+
+#ifndef ADF_TRANSPORT_ACCESS_MACROS_GEN4VF_H
+#define ADF_TRANSPORT_ACCESS_MACROS_GEN4VF_H
+
+#include "adf_transport_access_macros.h"
+#include "adf_transport_access_macros_gen4.h"
+
+#define ADF_RING_CSR_ADDR_OFFSET_GEN4VF 0x0
+
+#define WRITE_CSR_RING_BASE_GEN4VF(csr_base_addr, bank, ring, value) \
+do { \
+ uint32_t l_base = 0, u_base = 0; \
+ l_base = (uint32_t)(value & 0xFFFFFFFF); \
+ u_base = (uint32_t)((value & 0xFFFFFFFF00000000ULL) >> 32); \
+ ADF_CSR_WR(csr_base_addr + ADF_RING_CSR_ADDR_OFFSET_GEN4VF, \
+ (ADF_RING_BUNDLE_SIZE_GEN4 * bank) + \
+ ADF_RING_CSR_RING_LBASE_GEN4 + (ring << 2), \
+ l_base); \
+ ADF_CSR_WR(csr_base_addr + ADF_RING_CSR_ADDR_OFFSET_GEN4VF, \
+ (ADF_RING_BUNDLE_SIZE_GEN4 * bank) + \
+ ADF_RING_CSR_RING_UBASE_GEN4 + (ring << 2), \
+ u_base); \
+} while (0)
+
+#define WRITE_CSR_RING_CONFIG_GEN4VF(csr_base_addr, bank, ring, value) \
+ ADF_CSR_WR(csr_base_addr + ADF_RING_CSR_ADDR_OFFSET_GEN4VF, \
+ (ADF_RING_BUNDLE_SIZE_GEN4 * bank) + \
+ ADF_RING_CSR_RING_CONFIG_GEN4 + (ring << 2), value)
+
+#define WRITE_CSR_RING_TAIL_GEN4VF(csr_base_addr, bank, ring, value) \
+ ADF_CSR_WR((csr_base_addr) + ADF_RING_CSR_ADDR_OFFSET_GEN4VF, \
+ (ADF_RING_BUNDLE_SIZE_GEN4 * (bank)) + \
+ ADF_RING_CSR_RING_TAIL + ((ring) << 2), (value))
+
+#define WRITE_CSR_RING_HEAD_GEN4VF(csr_base_addr, bank, ring, value) \
+ ADF_CSR_WR((csr_base_addr) + ADF_RING_CSR_ADDR_OFFSET_GEN4VF, \
+ (ADF_RING_BUNDLE_SIZE_GEN4 * (bank)) + \
+ ADF_RING_CSR_RING_HEAD + ((ring) << 2), (value))
+
+#define WRITE_CSR_RING_SRV_ARB_EN_GEN4VF(csr_base_addr, bank, value) \
+ ADF_CSR_WR((csr_base_addr) + ADF_RING_CSR_ADDR_OFFSET_GEN4VF, \
+ (ADF_RING_BUNDLE_SIZE_GEN4 * (bank)) + \
+ ADF_RING_CSR_RING_SRV_ARB_EN, (value))
+
+#endif
diff --git a/drivers/common/qat/qat_common.h b/drivers/common/qat/qat_common.h
index cf840fea9b..845c8d99ab 100644
--- a/drivers/common/qat/qat_common.h
+++ b/drivers/common/qat/qat_common.h
@@ -18,7 +18,8 @@
enum qat_device_gen {
QAT_GEN1 = 1,
QAT_GEN2,
- QAT_GEN3
+ QAT_GEN3,
+ QAT_GEN4
};
enum qat_service_type {
diff --git a/drivers/common/qat/qat_device.c b/drivers/common/qat/qat_device.c
index 9fa142b5e5..932d7110f7 100644
--- a/drivers/common/qat/qat_device.c
+++ b/drivers/common/qat/qat_device.c
@@ -30,6 +30,11 @@ struct qat_gen_hw_data qat_gen_config[] = {
.qp_hw_data = qat_gen3_qps,
.comp_num_im_bufs_required = QAT_NUM_INTERM_BUFS_GEN3
},
+ [QAT_GEN4] = {
+ .dev_gen = QAT_GEN4,
+ .qp_hw_data = NULL,
+ .comp_num_im_bufs_required = QAT_NUM_INTERM_BUFS_GEN3
+ },
};
/* per-process array of device data */
@@ -59,6 +64,9 @@ static const struct rte_pci_id pci_id_qat_map[] = {
{
RTE_PCI_DEVICE(0x8086, 0x18a1),
},
+ {
+ RTE_PCI_DEVICE(0x8086, 0x4941),
+ },
{.device_id = 0},
};
@@ -232,6 +240,9 @@ qat_pci_device_allocate(struct rte_pci_device *pci_dev,
case 0x18a1:
qat_dev->qat_dev_gen = QAT_GEN3;
break;
+ case 0x4941:
+ qat_dev->qat_dev_gen = QAT_GEN4;
+ break;
default:
QAT_LOG(ERR, "Invalid dev_id, can't determine generation");
rte_memzone_free(qat_pci_devs[qat_dev->qat_dev_id].mz);
@@ -241,6 +252,17 @@ qat_pci_device_allocate(struct rte_pci_device *pci_dev,
if (devargs && devargs->drv_str)
qat_dev_parse_cmd(devargs->drv_str, qat_dev_cmd_param);
+ if (qat_dev->qat_dev_gen >= QAT_GEN4) {
+ int ret = qat_read_qp_config(qat_dev, qat_dev->qat_dev_gen);
+
+ if (ret) {
+ QAT_LOG(ERR,
+ "Cannot acquire ring configuration for QAT_%d",
+ qat_dev_id);
+ return NULL;
+ }
+ }
+
rte_spinlock_init(&qat_dev->arb_csr_lock);
qat_nb_pci_devices++;
diff --git a/drivers/common/qat/qat_device.h b/drivers/common/qat/qat_device.h
index 9c6a3ca4e6..b987fd1309 100644
--- a/drivers/common/qat/qat_device.h
+++ b/drivers/common/qat/qat_device.h
@@ -105,6 +105,11 @@ struct qat_pci_device {
/* Data relating to compression service */
struct qat_comp_dev_private *comp_dev;
/**< link back to compressdev private data */
+ union {
+ struct qat_qp_hw_data qp_gen4_data[QAT_GEN4_BUNDLE_NUM]
+ [QAT_GEN4_QPS_PER_BUNDLE_NUM];
+ };
+ /**< Data of ring configuration on gen4 */
};
struct qat_gen_hw_data {
diff --git a/drivers/common/qat/qat_qp.c b/drivers/common/qat/qat_qp.c
index aa64d2e168..8be59779f9 100644
--- a/drivers/common/qat/qat_qp.c
+++ b/drivers/common/qat/qat_qp.c
@@ -19,6 +19,7 @@
#include "qat_asym.h"
#include "qat_comp.h"
#include "adf_transport_access_macros.h"
+#include "adf_transport_access_macros_gen4vf.h"
#define QAT_CQ_MAX_DEQ_RETRIES 10
@@ -138,25 +139,33 @@ static int qat_queue_create(struct qat_pci_device *qat_dev,
struct qat_queue *queue, struct qat_qp_config *, uint8_t dir);
static int adf_verify_queue_size(uint32_t msg_size, uint32_t msg_num,
uint32_t *queue_size_for_csr);
-static void adf_configure_queues(struct qat_qp *queue);
-static void adf_queue_arb_enable(struct qat_queue *txq, void *base_addr,
- rte_spinlock_t *lock);
-static void adf_queue_arb_disable(struct qat_queue *txq, void *base_addr,
- rte_spinlock_t *lock);
-
+static void adf_configure_queues(struct qat_qp *queue,
+ enum qat_device_gen qat_dev_gen);
+static void adf_queue_arb_enable(enum qat_device_gen qat_dev_gen,
+ struct qat_queue *txq, void *base_addr, rte_spinlock_t *lock);
+static void adf_queue_arb_disable(enum qat_device_gen qat_dev_gen,
+ struct qat_queue *txq, void *base_addr, rte_spinlock_t *lock);
int qat_qps_per_service(struct qat_pci_device *qat_dev,
enum qat_service_type service)
{
int i = 0, count = 0, max_ops_per_srv = 0;
- const struct qat_qp_hw_data*
- sym_hw_qps = qat_gen_config[qat_dev->qat_dev_gen]
- .qp_hw_data[service];
- max_ops_per_srv = ADF_MAX_QPS_ON_ANY_SERVICE;
- for (; i < max_ops_per_srv; i++)
- if (sym_hw_qps[i].service_type == service)
- count++;
+ if (qat_dev->qat_dev_gen == QAT_GEN4) {
+ max_ops_per_srv = QAT_GEN4_BUNDLE_NUM;
+ for (i = 0, count = 0; i < max_ops_per_srv; i++)
+ if (qat_dev->qp_gen4_data[i][0].service_type == service)
+ count++;
+ } else {
+ const struct qat_qp_hw_data *sym_hw_qps =
+ qat_gen_config[qat_dev->qat_dev_gen]
+ .qp_hw_data[service];
+
+ max_ops_per_srv = ADF_MAX_QPS_ON_ANY_SERVICE;
+ for (i = 0, count = 0; i < max_ops_per_srv; i++)
+ if (sym_hw_qps[i].service_type == service)
+ count++;
+ }
return count;
}
@@ -195,12 +204,12 @@ int qat_qp_setup(struct qat_pci_device *qat_dev,
struct qat_qp **qp_addr,
uint16_t queue_pair_id,
struct qat_qp_config *qat_qp_conf)
-
{
struct qat_qp *qp;
struct rte_pci_device *pci_dev =
qat_pci_devs[qat_dev->qat_dev_id].pci_dev;
char op_cookie_pool_name[RTE_RING_NAMESIZE];
+ enum qat_device_gen qat_dev_gen = qat_dev->qat_dev_gen;
uint32_t i;
QAT_LOG(DEBUG, "Setup qp %u on qat pci device %d gen %d",
@@ -264,8 +273,8 @@ int qat_qp_setup(struct qat_pci_device *qat_dev,
goto create_err;
}
- adf_configure_queues(qp);
- adf_queue_arb_enable(&qp->tx_q, qp->mmap_bar_addr,
+ adf_configure_queues(qp, qat_dev_gen);
+ adf_queue_arb_enable(qat_dev_gen, &qp->tx_q, qp->mmap_bar_addr,
&qat_dev->arb_csr_lock);
snprintf(op_cookie_pool_name, RTE_RING_NAMESIZE,
@@ -314,7 +323,8 @@ int qat_qp_setup(struct qat_pci_device *qat_dev,
return -EFAULT;
}
-int qat_qp_release(struct qat_qp **qp_addr)
+
+int qat_qp_release(enum qat_device_gen qat_dev_gen, struct qat_qp **qp_addr)
{
struct qat_qp *qp = *qp_addr;
uint32_t i;
@@ -335,8 +345,8 @@ int qat_qp_release(struct qat_qp **qp_addr)
return -EAGAIN;
}
- adf_queue_arb_disable(&(qp->tx_q), qp->mmap_bar_addr,
- &qp->qat_dev->arb_csr_lock);
+ adf_queue_arb_disable(qat_dev_gen, &(qp->tx_q), qp->mmap_bar_addr,
+ &qp->qat_dev->arb_csr_lock);
for (i = 0; i < qp->nb_descriptors; i++)
rte_mempool_put(qp->op_cookie_pool, qp->op_cookies[i]);
@@ -386,6 +396,7 @@ qat_queue_create(struct qat_pci_device *qat_dev, struct qat_queue *queue,
const struct rte_memzone *qp_mz;
struct rte_pci_device *pci_dev =
qat_pci_devs[qat_dev->qat_dev_id].pci_dev;
+ enum qat_device_gen qat_dev_gen = qat_dev->qat_dev_gen;
int ret = 0;
uint16_t desc_size = (dir == ADF_RING_DIR_TX ?
qp_conf->hw->tx_msg_size : qp_conf->hw->rx_msg_size);
@@ -445,14 +456,19 @@ qat_queue_create(struct qat_pci_device *qat_dev, struct qat_queue *queue,
* Write an unused pattern to the queue memory.
*/
memset(queue->base_addr, 0x7F, queue_size_bytes);
-
- queue_base = BUILD_RING_BASE_ADDR(queue->base_phys_addr,
- queue->queue_size);
-
io_addr = pci_dev->mem_resource[0].addr;
- WRITE_CSR_RING_BASE(io_addr, queue->hw_bundle_number,
+ if (qat_dev_gen == QAT_GEN4) {
+ queue_base = BUILD_RING_BASE_ADDR_GEN4(queue->base_phys_addr,
+ queue->queue_size);
+ WRITE_CSR_RING_BASE_GEN4VF(io_addr, queue->hw_bundle_number,
+ queue->hw_queue_number, queue_base);
+ } else {
+ queue_base = BUILD_RING_BASE_ADDR(queue->base_phys_addr,
+ queue->queue_size);
+ WRITE_CSR_RING_BASE(io_addr, queue->hw_bundle_number,
queue->hw_queue_number, queue_base);
+ }
QAT_LOG(DEBUG, "RING: Name:%s, size in CSR: %u, in bytes %u,"
" nb msgs %u, msg_size %u, modulo mask %u",
@@ -468,6 +484,61 @@ qat_queue_create(struct qat_pci_device *qat_dev, struct qat_queue *queue,
return ret;
}
+int
+qat_select_valid_queue(struct qat_pci_device *qat_dev, int qp_id,
+ enum qat_service_type service_type)
+{
+ if (qat_dev->qat_dev_gen == QAT_GEN4) {
+ int i = 0, valid_qps = 0;
+
+ for (; i < QAT_GEN4_BUNDLE_NUM; i++) {
+ if (qat_dev->qp_gen4_data[i][0].service_type ==
+ service_type) {
+ if (valid_qps == qp_id)
+ return i;
+ ++valid_qps;
+ }
+ }
+ }
+ return -1;
+}
+
+int
+qat_read_qp_config(struct qat_pci_device *qat_dev,
+ enum qat_device_gen qat_dev_gen)
+{
+ if (qat_dev_gen == QAT_GEN4) {
+ /* Read default configuration,
+ * until some probe of it can be done
+ */
+ int i = 0;
+
+ for (; i < QAT_GEN4_BUNDLE_NUM; i++) {
+ struct qat_qp_hw_data *hw_data =
+ &qat_dev->qp_gen4_data[i][0];
+ enum qat_service_type service_type =
+ (QAT_GEN4_QP_DEFCON >> (8 * i)) & 0xFF;
+
+ memset(hw_data, 0, sizeof(*hw_data));
+ hw_data->service_type = service_type;
+ if (service_type == QAT_SERVICE_ASYMMETRIC) {
+ hw_data->tx_msg_size = 64;
+ hw_data->rx_msg_size = 32;
+ } else if (service_type == QAT_SERVICE_SYMMETRIC ||
+ service_type ==
+ QAT_SERVICE_COMPRESSION) {
+ hw_data->tx_msg_size = 128;
+ hw_data->rx_msg_size = 32;
+ }
+ hw_data->tx_ring_num = 0;
+ hw_data->rx_ring_num = 1;
+ hw_data->hw_bundle_num = i;
+ }
+ }
+ /* With default config will always return success */
+ return 0;
+}
+
static int qat_qp_check_queue_alignment(uint64_t phys_addr,
uint32_t queue_size_bytes)
{
@@ -491,54 +562,81 @@ static int adf_verify_queue_size(uint32_t msg_size, uint32_t msg_num,
return -EINVAL;
}
-static void adf_queue_arb_enable(struct qat_queue *txq, void *base_addr,
- rte_spinlock_t *lock)
+static void
+adf_queue_arb_enable(enum qat_device_gen qat_dev_gen, struct qat_queue *txq,
+ void *base_addr, rte_spinlock_t *lock)
{
- uint32_t arb_csr_offset = ADF_ARB_RINGSRVARBEN_OFFSET +
- (ADF_ARB_REG_SLOT *
- txq->hw_bundle_number);
- uint32_t value;
+ uint32_t arb_csr_offset = 0, value;
rte_spinlock_lock(lock);
- value = ADF_CSR_RD(base_addr, arb_csr_offset);
+ if (qat_dev_gen == QAT_GEN4) {
+ arb_csr_offset = ADF_ARB_RINGSRVARBEN_OFFSET +
+ (ADF_RING_BUNDLE_SIZE_GEN4 *
+ txq->hw_bundle_number);
+ value = ADF_CSR_RD(base_addr + ADF_RING_CSR_ADDR_OFFSET_GEN4VF,
+ arb_csr_offset);
+ } else {
+ arb_csr_offset = ADF_ARB_RINGSRVARBEN_OFFSET +
+ (ADF_ARB_REG_SLOT *
+ txq->hw_bundle_number);
+ value = ADF_CSR_RD(base_addr,
+ arb_csr_offset);
+ }
value |= (0x01 << txq->hw_queue_number);
ADF_CSR_WR(base_addr, arb_csr_offset, value);
rte_spinlock_unlock(lock);
}
-static void adf_queue_arb_disable(struct qat_queue *txq, void *base_addr,
- rte_spinlock_t *lock)
+static void adf_queue_arb_disable(enum qat_device_gen qat_dev_gen,
+ struct qat_queue *txq, void *base_addr, rte_spinlock_t *lock)
{
- uint32_t arb_csr_offset = ADF_ARB_RINGSRVARBEN_OFFSET +
- (ADF_ARB_REG_SLOT *
- txq->hw_bundle_number);
- uint32_t value;
+ uint32_t arb_csr_offset = 0, value;
rte_spinlock_lock(lock);
- value = ADF_CSR_RD(base_addr, arb_csr_offset);
+ if (qat_dev_gen == QAT_GEN4) {
+ arb_csr_offset = ADF_ARB_RINGSRVARBEN_OFFSET +
+ (ADF_RING_BUNDLE_SIZE_GEN4 *
+ txq->hw_bundle_number);
+ value = ADF_CSR_RD(base_addr + ADF_RING_CSR_ADDR_OFFSET_GEN4VF,
+ arb_csr_offset);
+ } else {
+ arb_csr_offset = ADF_ARB_RINGSRVARBEN_OFFSET +
+ (ADF_ARB_REG_SLOT *
+ txq->hw_bundle_number);
+ value = ADF_CSR_RD(base_addr,
+ arb_csr_offset);
+ }
value &= ~(0x01 << txq->hw_queue_number);
ADF_CSR_WR(base_addr, arb_csr_offset, value);
rte_spinlock_unlock(lock);
}
-static void adf_configure_queues(struct qat_qp *qp)
+static void adf_configure_queues(struct qat_qp *qp,
+ enum qat_device_gen qat_dev_gen)
{
- uint32_t queue_config;
- struct qat_queue *queue = &qp->tx_q;
-
- queue_config = BUILD_RING_CONFIG(queue->queue_size);
-
- WRITE_CSR_RING_CONFIG(qp->mmap_bar_addr, queue->hw_bundle_number,
- queue->hw_queue_number, queue_config);
-
- queue = &qp->rx_q;
- queue_config =
- BUILD_RESP_RING_CONFIG(queue->queue_size,
- ADF_RING_NEAR_WATERMARK_512,
- ADF_RING_NEAR_WATERMARK_0);
-
- WRITE_CSR_RING_CONFIG(qp->mmap_bar_addr, queue->hw_bundle_number,
- queue->hw_queue_number, queue_config);
+ uint32_t q_tx_config, q_resp_config;
+ struct qat_queue *q_tx = &qp->tx_q, *q_rx = &qp->rx_q;
+
+ q_tx_config = BUILD_RING_CONFIG(q_tx->queue_size);
+ q_resp_config = BUILD_RESP_RING_CONFIG(q_rx->queue_size,
+ ADF_RING_NEAR_WATERMARK_512,
+ ADF_RING_NEAR_WATERMARK_0);
+
+ if (qat_dev_gen == QAT_GEN4) {
+ WRITE_CSR_RING_CONFIG_GEN4VF(qp->mmap_bar_addr,
+ q_tx->hw_bundle_number, q_tx->hw_queue_number,
+ q_tx_config);
+ WRITE_CSR_RING_CONFIG_GEN4VF(qp->mmap_bar_addr,
+ q_rx->hw_bundle_number, q_rx->hw_queue_number,
+ q_resp_config);
+ } else {
+ WRITE_CSR_RING_CONFIG(qp->mmap_bar_addr,
+ q_tx->hw_bundle_number, q_tx->hw_queue_number,
+ q_tx_config);
+ WRITE_CSR_RING_CONFIG(qp->mmap_bar_addr,
+ q_rx->hw_bundle_number, q_rx->hw_queue_number,
+ q_resp_config);
+ }
}
static inline uint32_t adf_modulo(uint32_t data, uint32_t modulo_mask)
@@ -547,14 +645,21 @@ static inline uint32_t adf_modulo(uint32_t data, uint32_t modulo_mask)
}
static inline void
-txq_write_tail(struct qat_qp *qp, struct qat_queue *q) {
- WRITE_CSR_RING_TAIL(qp->mmap_bar_addr, q->hw_bundle_number,
+txq_write_tail(enum qat_device_gen qat_dev_gen,
+ struct qat_qp *qp, struct qat_queue *q) {
+
+ if (qat_dev_gen == QAT_GEN4) {
+ WRITE_CSR_RING_TAIL_GEN4VF(qp->mmap_bar_addr,
+ q->hw_bundle_number, q->hw_queue_number, q->tail);
+ } else {
+ WRITE_CSR_RING_TAIL(qp->mmap_bar_addr, q->hw_bundle_number,
q->hw_queue_number, q->tail);
- q->csr_tail = q->tail;
+ }
}
static inline
-void rxq_free_desc(struct qat_qp *qp, struct qat_queue *q)
+void rxq_free_desc(enum qat_device_gen qat_dev_gen, struct qat_qp *qp,
+ struct qat_queue *q)
{
uint32_t old_head, new_head;
uint32_t max_head;
@@ -576,8 +681,14 @@ void rxq_free_desc(struct qat_qp *qp, struct qat_queue *q)
q->csr_head = new_head;
/* write current head to CSR */
- WRITE_CSR_RING_HEAD(qp->mmap_bar_addr, q->hw_bundle_number,
- q->hw_queue_number, new_head);
+ if (qat_dev_gen == QAT_GEN4) {
+ WRITE_CSR_RING_HEAD_GEN4VF(qp->mmap_bar_addr,
+ q->hw_bundle_number, q->hw_queue_number, new_head);
+ } else {
+ WRITE_CSR_RING_HEAD(qp->mmap_bar_addr, q->hw_bundle_number,
+ q->hw_queue_number, new_head);
+ }
+
}
uint16_t
@@ -670,7 +781,7 @@ qat_enqueue_op_burst(void *qp, void **ops, uint16_t nb_ops)
queue->tail = tail;
tmp_qp->enqueued += nb_ops_sent;
tmp_qp->stats.enqueued_count += nb_ops_sent;
- txq_write_tail(tmp_qp, queue);
+ txq_write_tail(tmp_qp->qat_dev_gen, tmp_qp, queue);
return nb_ops_sent;
}
@@ -843,7 +954,7 @@ qat_enqueue_comp_op_burst(void *qp, void **ops, uint16_t nb_ops)
queue->tail = tail;
tmp_qp->enqueued += total_descriptors_built;
tmp_qp->stats.enqueued_count += nb_ops_sent;
- txq_write_tail(tmp_qp, queue);
+ txq_write_tail(tmp_qp->qat_dev_gen, tmp_qp, queue);
return nb_ops_sent;
}
@@ -909,7 +1020,7 @@ qat_dequeue_op_burst(void *qp, void **ops, uint16_t nb_ops)
rx_queue->head = head;
if (rx_queue->nb_processed_responses > QAT_CSR_HEAD_WRITE_THRESH)
- rxq_free_desc(tmp_qp, rx_queue);
+ rxq_free_desc(tmp_qp->qat_dev_gen, tmp_qp, rx_queue);
QAT_DP_LOG(DEBUG, "Dequeue burst return: %u, QAT responses: %u",
op_resp_counter, fw_resp_counter);
@@ -951,7 +1062,7 @@ qat_cq_dequeue_response(struct qat_qp *qp, void *out_data)
queue->head = adf_modulo(queue->head + queue->msg_size,
queue->modulo_mask);
- rxq_free_desc(qp, queue);
+ rxq_free_desc(qp->qat_dev_gen, qp, queue);
}
return result;
@@ -986,7 +1097,7 @@ qat_cq_get_fw_version(struct qat_qp *qp)
memcpy(base_addr + queue->tail, &null_msg, sizeof(null_msg));
queue->tail = adf_modulo(queue->tail + queue->msg_size,
queue->modulo_mask);
- txq_write_tail(qp, queue);
+ txq_write_tail(qp->qat_dev_gen, qp, queue);
/* receive a response */
if (qat_cq_dequeue_response(qp, &response)) {
diff --git a/drivers/common/qat/qat_qp.h b/drivers/common/qat/qat_qp.h
index d353e8552b..3d9a757349 100644
--- a/drivers/common/qat/qat_qp.h
+++ b/drivers/common/qat/qat_qp.h
@@ -14,6 +14,16 @@ struct qat_pci_device;
#define QAT_QP_MIN_INFL_THRESHOLD 256
+/* Default qp configuration for GEN4 devices */
+#define QAT_GEN4_QP_DEFCON (QAT_SERVICE_SYMMETRIC | \
+ QAT_SERVICE_SYMMETRIC << 8 | \
+ QAT_SERVICE_SYMMETRIC << 16 | \
+ QAT_SERVICE_SYMMETRIC << 24)
+
+/* QAT GEN 4 specific macros */
+#define QAT_GEN4_BUNDLE_NUM 4
+#define QAT_GEN4_QPS_PER_BUNDLE_NUM 1
+
/**
* Structure with data needed for creation of queue pair.
*/
@@ -26,6 +36,15 @@ struct qat_qp_hw_data {
uint16_t rx_msg_size;
};
+/**
+ * Structure with data needed for creation of queue pair on gen4.
+ */
+struct qat_qp_gen4_data {
+ struct qat_qp_hw_data qat_qp_hw_data;
+ uint8_t reserved;
+ uint8_t valid;
+};
+
/**
* Structure with data needed for creation of queue pair.
*/
@@ -90,7 +109,7 @@ uint16_t
qat_dequeue_op_burst(void *qp, void **ops, uint16_t nb_ops);
int
-qat_qp_release(struct qat_qp **qp_addr);
+qat_qp_release(enum qat_device_gen qat_dev_gen, struct qat_qp **qp_addr);
int
qat_qp_setup(struct qat_pci_device *qat_dev,
@@ -110,4 +129,12 @@ qat_comp_process_response(void **op __rte_unused, uint8_t *resp __rte_unused,
void *op_cookie __rte_unused,
uint64_t *dequeue_err_count __rte_unused);
+int
+qat_select_valid_queue(struct qat_pci_device *qat_dev, int qp_id,
+ enum qat_service_type service_type);
+
+int
+qat_read_qp_config(struct qat_pci_device *qat_dev,
+ enum qat_device_gen qat_dev_gen);
+
#endif /* _QAT_QP_H_ */
diff --git a/drivers/compress/qat/qat_comp_pmd.c b/drivers/compress/qat/qat_comp_pmd.c
index 6eb1ae3a21..cfdcb6b3d1 100644
--- a/drivers/compress/qat/qat_comp_pmd.c
+++ b/drivers/compress/qat/qat_comp_pmd.c
@@ -74,6 +74,7 @@ qat_comp_qp_release(struct rte_compressdev *dev, uint16_t queue_pair_id)
struct qat_qp **qp_addr =
(struct qat_qp **)&(dev->data->queue_pairs[queue_pair_id]);
struct qat_qp *qp = (struct qat_qp *)*qp_addr;
+ enum qat_device_gen qat_dev_gen = qat_private->qat_dev->qat_dev_gen;
uint32_t i;
QAT_LOG(DEBUG, "Release comp qp %u on device %d",
@@ -90,7 +91,7 @@ qat_comp_qp_release(struct rte_compressdev *dev, uint16_t queue_pair_id)
rte_free(cookie->qat_sgl_dst_d);
}
- return qat_qp_release((struct qat_qp **)
+ return qat_qp_release(qat_dev_gen, (struct qat_qp **)
&(dev->data->queue_pairs[queue_pair_id]));
}
@@ -710,6 +711,10 @@ qat_comp_dev_create(struct qat_pci_device *qat_pci_dev,
const struct rte_compressdev_capabilities *capabilities;
uint64_t capa_size;
+ if (qat_pci_dev->qat_dev_gen == QAT_GEN4) {
+ QAT_LOG(ERR, "Compression PMD not supported on QAT 4xxx");
+ return 0;
+ }
snprintf(name, RTE_COMPRESSDEV_NAME_MAX_LEN, "%s_%s",
qat_pci_dev->name, "comp");
QAT_LOG(DEBUG, "Creating QAT COMP device %s", name);
diff --git a/drivers/crypto/qat/qat_asym_pmd.c b/drivers/crypto/qat/qat_asym_pmd.c
index f0c8ed1bcf..56ccca36d1 100644
--- a/drivers/crypto/qat/qat_asym_pmd.c
+++ b/drivers/crypto/qat/qat_asym_pmd.c
@@ -103,6 +103,7 @@ static int qat_asym_qp_release(struct rte_cryptodev *dev,
uint16_t queue_pair_id)
{
struct qat_asym_dev_private *qat_private = dev->data->dev_private;
+ enum qat_device_gen qat_dev_gen = qat_private->qat_dev->qat_dev_gen;
QAT_LOG(DEBUG, "Release asym qp %u on device %d",
queue_pair_id, dev->data->dev_id);
@@ -110,7 +111,7 @@ static int qat_asym_qp_release(struct rte_cryptodev *dev,
qat_private->qat_dev->qps_in_use[QAT_SERVICE_ASYMMETRIC][queue_pair_id]
= NULL;
- return qat_qp_release((struct qat_qp **)
+ return qat_qp_release(qat_dev_gen, (struct qat_qp **)
&(dev->data->queue_pairs[queue_pair_id]));
}
@@ -250,6 +251,10 @@ qat_asym_dev_create(struct qat_pci_device *qat_pci_dev,
struct rte_cryptodev *cryptodev;
struct qat_asym_dev_private *internals;
+ if (qat_pci_dev->qat_dev_gen == QAT_GEN4) {
+ QAT_LOG(ERR, "Asymmetric crypto PMD not supported on QAT 4xxx");
+ return 0;
+ }
snprintf(name, RTE_CRYPTODEV_NAME_MAX_LEN, "%s_%s",
qat_pci_dev->name, "asym");
QAT_LOG(DEBUG, "Creating QAT ASYM device %s\n", name);
diff --git a/drivers/crypto/qat/qat_sym_pmd.c b/drivers/crypto/qat/qat_sym_pmd.c
index 549345b6fa..e15722ad66 100644
--- a/drivers/crypto/qat/qat_sym_pmd.c
+++ b/drivers/crypto/qat/qat_sym_pmd.c
@@ -139,6 +139,7 @@ static void qat_sym_stats_reset(struct rte_cryptodev *dev)
static int qat_sym_qp_release(struct rte_cryptodev *dev, uint16_t queue_pair_id)
{
struct qat_sym_dev_private *qat_private = dev->data->dev_private;
+ enum qat_device_gen qat_dev_gen = qat_private->qat_dev->qat_dev_gen;
QAT_LOG(DEBUG, "Release sym qp %u on device %d",
queue_pair_id, dev->data->dev_id);
@@ -146,7 +147,7 @@ static int qat_sym_qp_release(struct rte_cryptodev *dev, uint16_t queue_pair_id)
qat_private->qat_dev->qps_in_use[QAT_SERVICE_SYMMETRIC][queue_pair_id]
= NULL;
- return qat_qp_release((struct qat_qp **)
+ return qat_qp_release(qat_dev_gen, (struct qat_qp **)
&(dev->data->queue_pairs[queue_pair_id]));
}
@@ -158,15 +159,33 @@ static int qat_sym_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id,
int ret = 0;
uint32_t i;
struct qat_qp_config qat_qp_conf;
+ const struct qat_qp_hw_data *sym_hw_qps = NULL;
+ const struct qat_qp_hw_data *qp_hw_data = NULL;
struct qat_qp **qp_addr =
(struct qat_qp **)&(dev->data->queue_pairs[qp_id]);
struct qat_sym_dev_private *qat_private = dev->data->dev_private;
struct qat_pci_device *qat_dev = qat_private->qat_dev;
- const struct qat_qp_hw_data *sym_hw_qps =
- qat_gen_config[qat_private->qat_dev->qat_dev_gen]
- .qp_hw_data[QAT_SERVICE_SYMMETRIC];
- const struct qat_qp_hw_data *qp_hw_data = sym_hw_qps + qp_id;
+
+ if (qat_dev->qat_dev_gen == QAT_GEN4) {
+ int ring_pair =
+ qat_select_valid_queue(qat_dev, qp_id,
+ QAT_SERVICE_SYMMETRIC);
+ sym_hw_qps =
+ &qat_dev->qp_gen4_data[0][0];
+ qp_hw_data =
+ &qat_dev->qp_gen4_data[ring_pair][0];
+ if (ring_pair < 0) {
+ QAT_LOG(ERR,
+ "qp_id %u invalid for this device, no enough services allocated for GEN4 device",
+ qp_id);
+ return -EINVAL;
+ }
+ } else {
+ sym_hw_qps = qat_gen_config[qat_dev->qat_dev_gen]
+ .qp_hw_data[QAT_SERVICE_SYMMETRIC];
+ qp_hw_data = sym_hw_qps + qp_id;
+ }
/* If qp is already in use free ring memory and qp metadata. */
if (*qp_addr != NULL) {
@@ -430,6 +449,10 @@ qat_sym_dev_create(struct qat_pci_device *qat_pci_dev,
capabilities = qat_gen3_sym_capabilities;
capa_size = sizeof(qat_gen3_sym_capabilities);
break;
+ case QAT_GEN4:
+ capabilities = NULL;
+ capa_size = 0;
+ break;
default:
QAT_LOG(DEBUG,
"QAT gen %d capabilities unknown",
diff --git a/drivers/crypto/qat/qat_sym_session.c b/drivers/crypto/qat/qat_sym_session.c
index 231b1640da..506ffddd20 100644
--- a/drivers/crypto/qat/qat_sym_session.c
+++ b/drivers/crypto/qat/qat_sym_session.c
@@ -550,6 +550,7 @@ qat_sym_session_set_parameters(struct rte_cryptodev *dev,
return -EINVAL;
}
+ memset(session, 0, sizeof(*session));
/* Set context descriptor physical address */
session->cd_paddr = session_paddr +
offsetof(struct qat_sym_session, cd);
--
2.25.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [dpdk-dev] [PATCH 03/15] crypto/qat: enable gen4 legacy algorithms
2021-05-31 14:10 [dpdk-dev] [PATCH 00/15] Add support for fourth generation of Intel QuickAssist Technology devices Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 01/15] common/qat: rework qp per service function Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 02/15] crypto/qat: add fourth generation qat devices support Arek Kusztal
@ 2021-05-31 14:10 ` Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 04/15] crypto/qat: add fourth generation ucs slice type, add ctr mode Arek Kusztal
` (11 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Arek Kusztal @ 2021-05-31 14:10 UTC (permalink / raw)
To: dev; +Cc: gakhil, fiona.trahe, roy.fan.zhang, Arek Kusztal
This commit enables algorithms labeled as 'legacy'
on QAT generation 4 devices.
Following algorithms were enabled:
* AES-CBC
* AES-CMAC
* AES-XCBC MAC
* NULL (auth, cipher)
* SHA1-HMAC
* SHA2-HMAC (224, 256, 384, 512)
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
---
drivers/crypto/qat/qat_sym_capabilities.h | 337 ++++++++++++++++++++++
drivers/crypto/qat/qat_sym_pmd.c | 9 +-
2 files changed, 344 insertions(+), 2 deletions(-)
diff --git a/drivers/crypto/qat/qat_sym_capabilities.h b/drivers/crypto/qat/qat_sym_capabilities.h
index f7cab2f471..21c817bccc 100644
--- a/drivers/crypto/qat/qat_sym_capabilities.h
+++ b/drivers/crypto/qat/qat_sym_capabilities.h
@@ -731,6 +731,343 @@
}, } \
}
+#define QAT_BASE_GEN4_SYM_CAPABILITIES \
+ { /* AES CBC */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER, \
+ {.cipher = { \
+ .algo = RTE_CRYPTO_CIPHER_AES_CBC, \
+ .block_size = 16, \
+ .key_size = { \
+ .min = 16, \
+ .max = 32, \
+ .increment = 8 \
+ }, \
+ .iv_size = { \
+ .min = 16, \
+ .max = 16, \
+ .increment = 0 \
+ } \
+ }, } \
+ }, } \
+ }, \
+ { /* SHA1 HMAC */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, \
+ {.auth = { \
+ .algo = RTE_CRYPTO_AUTH_SHA1_HMAC, \
+ .block_size = 64, \
+ .key_size = { \
+ .min = 1, \
+ .max = 64, \
+ .increment = 1 \
+ }, \
+ .digest_size = { \
+ .min = 1, \
+ .max = 20, \
+ .increment = 1 \
+ }, \
+ .iv_size = { 0 } \
+ }, } \
+ }, } \
+ }, \
+ { /* SHA224 HMAC */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, \
+ {.auth = { \
+ .algo = RTE_CRYPTO_AUTH_SHA224_HMAC, \
+ .block_size = 64, \
+ .key_size = { \
+ .min = 1, \
+ .max = 64, \
+ .increment = 1 \
+ }, \
+ .digest_size = { \
+ .min = 1, \
+ .max = 28, \
+ .increment = 1 \
+ }, \
+ .iv_size = { 0 } \
+ }, } \
+ }, } \
+ }, \
+ { /* SHA256 HMAC */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, \
+ {.auth = { \
+ .algo = RTE_CRYPTO_AUTH_SHA256_HMAC, \
+ .block_size = 64, \
+ .key_size = { \
+ .min = 1, \
+ .max = 64, \
+ .increment = 1 \
+ }, \
+ .digest_size = { \
+ .min = 1, \
+ .max = 32, \
+ .increment = 1 \
+ }, \
+ .iv_size = { 0 } \
+ }, } \
+ }, } \
+ }, \
+ { /* SHA384 HMAC */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, \
+ {.auth = { \
+ .algo = RTE_CRYPTO_AUTH_SHA384_HMAC, \
+ .block_size = 128, \
+ .key_size = { \
+ .min = 1, \
+ .max = 128, \
+ .increment = 1 \
+ }, \
+ .digest_size = { \
+ .min = 1, \
+ .max = 48, \
+ .increment = 1 \
+ }, \
+ .iv_size = { 0 } \
+ }, } \
+ }, } \
+ }, \
+ { /* SHA512 HMAC */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, \
+ {.auth = { \
+ .algo = RTE_CRYPTO_AUTH_SHA512_HMAC, \
+ .block_size = 128, \
+ .key_size = { \
+ .min = 1, \
+ .max = 128, \
+ .increment = 1 \
+ }, \
+ .digest_size = { \
+ .min = 1, \
+ .max = 64, \
+ .increment = 1 \
+ }, \
+ .iv_size = { 0 } \
+ }, } \
+ }, } \
+ }, \
+ { /* AES XCBC MAC */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, \
+ {.auth = { \
+ .algo = RTE_CRYPTO_AUTH_AES_XCBC_MAC, \
+ .block_size = 16, \
+ .key_size = { \
+ .min = 16, \
+ .max = 16, \
+ .increment = 0 \
+ }, \
+ .digest_size = { \
+ .min = 12, \
+ .max = 12, \
+ .increment = 0 \
+ }, \
+ .aad_size = { 0 }, \
+ .iv_size = { 0 } \
+ }, } \
+ }, } \
+ }, \
+ { /* AES CMAC */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, \
+ {.auth = { \
+ .algo = RTE_CRYPTO_AUTH_AES_CMAC, \
+ .block_size = 16, \
+ .key_size = { \
+ .min = 16, \
+ .max = 16, \
+ .increment = 0 \
+ }, \
+ .digest_size = { \
+ .min = 4, \
+ .max = 16, \
+ .increment = 4 \
+ } \
+ }, } \
+ }, } \
+ }, \
+ { /* AES DOCSIS BPI */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER, \
+ {.cipher = { \
+ .algo = RTE_CRYPTO_CIPHER_AES_DOCSISBPI,\
+ .block_size = 16, \
+ .key_size = { \
+ .min = 16, \
+ .max = 32, \
+ .increment = 16 \
+ }, \
+ .iv_size = { \
+ .min = 16, \
+ .max = 16, \
+ .increment = 0 \
+ } \
+ }, } \
+ }, } \
+ }, \
+ { /* NULL (AUTH) */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, \
+ {.auth = { \
+ .algo = RTE_CRYPTO_AUTH_NULL, \
+ .block_size = 1, \
+ .key_size = { \
+ .min = 0, \
+ .max = 0, \
+ .increment = 0 \
+ }, \
+ .digest_size = { \
+ .min = 0, \
+ .max = 0, \
+ .increment = 0 \
+ }, \
+ .iv_size = { 0 } \
+ }, }, \
+ }, }, \
+ }, \
+ { /* NULL (CIPHER) */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER, \
+ {.cipher = { \
+ .algo = RTE_CRYPTO_CIPHER_NULL, \
+ .block_size = 1, \
+ .key_size = { \
+ .min = 0, \
+ .max = 0, \
+ .increment = 0 \
+ }, \
+ .iv_size = { \
+ .min = 0, \
+ .max = 0, \
+ .increment = 0 \
+ } \
+ }, }, \
+ }, } \
+ }, \
+ { /* SHA1 */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, \
+ {.auth = { \
+ .algo = RTE_CRYPTO_AUTH_SHA1, \
+ .block_size = 64, \
+ .key_size = { \
+ .min = 0, \
+ .max = 0, \
+ .increment = 0 \
+ }, \
+ .digest_size = { \
+ .min = 1, \
+ .max = 20, \
+ .increment = 1 \
+ }, \
+ .iv_size = { 0 } \
+ }, } \
+ }, } \
+ }, \
+ { /* SHA224 */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, \
+ {.auth = { \
+ .algo = RTE_CRYPTO_AUTH_SHA224, \
+ .block_size = 64, \
+ .key_size = { \
+ .min = 0, \
+ .max = 0, \
+ .increment = 0 \
+ }, \
+ .digest_size = { \
+ .min = 1, \
+ .max = 28, \
+ .increment = 1 \
+ }, \
+ .iv_size = { 0 } \
+ }, } \
+ }, } \
+ }, \
+ { /* SHA256 */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, \
+ {.auth = { \
+ .algo = RTE_CRYPTO_AUTH_SHA256, \
+ .block_size = 64, \
+ .key_size = { \
+ .min = 0, \
+ .max = 0, \
+ .increment = 0 \
+ }, \
+ .digest_size = { \
+ .min = 1, \
+ .max = 32, \
+ .increment = 1 \
+ }, \
+ .iv_size = { 0 } \
+ }, } \
+ }, } \
+ }, \
+ { /* SHA384 */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, \
+ {.auth = { \
+ .algo = RTE_CRYPTO_AUTH_SHA384, \
+ .block_size = 128, \
+ .key_size = { \
+ .min = 0, \
+ .max = 0, \
+ .increment = 0 \
+ }, \
+ .digest_size = { \
+ .min = 1, \
+ .max = 48, \
+ .increment = 1 \
+ }, \
+ .iv_size = { 0 } \
+ }, } \
+ }, } \
+ }, \
+ { /* SHA512 */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, \
+ {.auth = { \
+ .algo = RTE_CRYPTO_AUTH_SHA512, \
+ .block_size = 128, \
+ .key_size = { \
+ .min = 0, \
+ .max = 0, \
+ .increment = 0 \
+ }, \
+ .digest_size = { \
+ .min = 1, \
+ .max = 64, \
+ .increment = 1 \
+ }, \
+ .iv_size = { 0 } \
+ }, } \
+ }, } \
+ } \
+
+
+
#ifdef RTE_LIB_SECURITY
#define QAT_SECURITY_SYM_CAPABILITIES \
{ /* AES DOCSIS BPI */ \
diff --git a/drivers/crypto/qat/qat_sym_pmd.c b/drivers/crypto/qat/qat_sym_pmd.c
index e15722ad66..0097ee210f 100644
--- a/drivers/crypto/qat/qat_sym_pmd.c
+++ b/drivers/crypto/qat/qat_sym_pmd.c
@@ -39,6 +39,11 @@ static const struct rte_cryptodev_capabilities qat_gen3_sym_capabilities[] = {
RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST()
};
+static const struct rte_cryptodev_capabilities qat_gen4_sym_capabilities[] = {
+ QAT_BASE_GEN4_SYM_CAPABILITIES,
+ RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST()
+};
+
#ifdef RTE_LIB_SECURITY
static const struct rte_cryptodev_capabilities
qat_security_sym_capabilities[] = {
@@ -450,8 +455,8 @@ qat_sym_dev_create(struct qat_pci_device *qat_pci_dev,
capa_size = sizeof(qat_gen3_sym_capabilities);
break;
case QAT_GEN4:
- capabilities = NULL;
- capa_size = 0;
+ capabilities = qat_gen4_sym_capabilities;
+ capa_size = sizeof(qat_gen4_sym_capabilities);
break;
default:
QAT_LOG(DEBUG,
--
2.25.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [dpdk-dev] [PATCH 04/15] crypto/qat: add fourth generation ucs slice type, add ctr mode
2021-05-31 14:10 [dpdk-dev] [PATCH 00/15] Add support for fourth generation of Intel QuickAssist Technology devices Arek Kusztal
` (2 preceding siblings ...)
2021-05-31 14:10 ` [dpdk-dev] [PATCH 03/15] crypto/qat: enable gen4 legacy algorithms Arek Kusztal
@ 2021-05-31 14:10 ` Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 05/15] crypto/qat: rename content descriptor functions Arek Kusztal
` (10 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Arek Kusztal @ 2021-05-31 14:10 UTC (permalink / raw)
To: dev; +Cc: gakhil, fiona.trahe, roy.fan.zhang, Arek Kusztal
This commit adds unified cipher slice to Intel QuickAssist
Technology PMD and enables AES-CTR algorithm.
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
---
drivers/common/qat/qat_adf/icp_qat_fw_la.h | 28 ++++++++++++++++++++++
drivers/common/qat/qat_adf/icp_qat_hw.h | 10 ++++++++
drivers/crypto/qat/qat_sym_capabilities.h | 20 ++++++++++++++++
drivers/crypto/qat/qat_sym_session.c | 27 ++++++++++++++++++++-
drivers/crypto/qat/qat_sym_session.h | 1 +
5 files changed, 85 insertions(+), 1 deletion(-)
diff --git a/drivers/common/qat/qat_adf/icp_qat_fw_la.h b/drivers/common/qat/qat_adf/icp_qat_fw_la.h
index 20eb145def..c4901eb869 100644
--- a/drivers/common/qat/qat_adf/icp_qat_fw_la.h
+++ b/drivers/common/qat/qat_adf/icp_qat_fw_la.h
@@ -371,4 +371,32 @@ struct icp_qat_fw_la_resp {
& ICP_QAT_FW_COMN_NEXT_ID_MASK) | \
((val) & ICP_QAT_FW_COMN_CURR_ID_MASK)) }
+#define ICP_QAT_FW_LA_USE_WIRELESS_SLICE_TYPE 2
+#define ICP_QAT_FW_LA_USE_UCS_SLICE_TYPE 1
+#define ICP_QAT_FW_LA_USE_LEGACY_SLICE_TYPE 0
+#define QAT_LA_SLICE_TYPE_BITPOS 14
+#define QAT_LA_SLICE_TYPE_MASK 0x3
+#define ICP_QAT_FW_LA_SLICE_TYPE_SET(flags, val) \
+ QAT_FIELD_SET(flags, val, QAT_LA_SLICE_TYPE_BITPOS, \
+ QAT_LA_SLICE_TYPE_MASK)
+
+struct icp_qat_fw_la_cipher_20_req_params {
+ uint32_t cipher_offset;
+ uint32_t cipher_length;
+ union {
+ uint32_t cipher_IV_array[ICP_QAT_FW_NUM_LONGWORDS_4];
+ struct {
+ uint64_t cipher_IV_ptr;
+ uint64_t resrvd1;
+ } s;
+
+ } u;
+ uint32_t spc_aad_offset;
+ uint32_t spc_aad_sz;
+ uint64_t spc_aad_addr;
+ uint64_t spc_auth_res_addr;
+ uint8_t reserved[3];
+ uint8_t spc_auth_res_sz;
+};
+
#endif
diff --git a/drivers/common/qat/qat_adf/icp_qat_hw.h b/drivers/common/qat/qat_adf/icp_qat_hw.h
index fdc0f191a2..b1e6a1fa15 100644
--- a/drivers/common/qat/qat_adf/icp_qat_hw.h
+++ b/drivers/common/qat/qat_adf/icp_qat_hw.h
@@ -342,6 +342,16 @@ struct icp_qat_hw_cipher_algo_blk {
uint8_t key[ICP_QAT_HW_CIPHER_MAX_KEY_SZ];
} __rte_cache_aligned;
+struct icp_qat_hw_ucs_cipher_config {
+ uint32_t val;
+ uint32_t reserved[3];
+};
+
+struct icp_qat_hw_cipher_algo_blk20 {
+ struct icp_qat_hw_ucs_cipher_config cipher_config;
+ uint8_t key[ICP_QAT_HW_CIPHER_MAX_KEY_SZ];
+} __rte_cache_aligned;
+
/* ========================================================================= */
/* COMPRESSION SLICE */
/* ========================================================================= */
diff --git a/drivers/crypto/qat/qat_sym_capabilities.h b/drivers/crypto/qat/qat_sym_capabilities.h
index 21c817bccc..aca528b991 100644
--- a/drivers/crypto/qat/qat_sym_capabilities.h
+++ b/drivers/crypto/qat/qat_sym_capabilities.h
@@ -1064,6 +1064,26 @@
.iv_size = { 0 } \
}, } \
}, } \
+ }, \
+ { /* AES CTR */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER, \
+ {.cipher = { \
+ .algo = RTE_CRYPTO_CIPHER_AES_CTR, \
+ .block_size = 16, \
+ .key_size = { \
+ .min = 16, \
+ .max = 32, \
+ .increment = 8 \
+ }, \
+ .iv_size = { \
+ .min = 16, \
+ .max = 16, \
+ .increment = 0 \
+ } \
+ }, } \
+ }, } \
} \
diff --git a/drivers/crypto/qat/qat_sym_session.c b/drivers/crypto/qat/qat_sym_session.c
index 506ffddd20..2c44b1f1aa 100644
--- a/drivers/crypto/qat/qat_sym_session.c
+++ b/drivers/crypto/qat/qat_sym_session.c
@@ -246,6 +246,8 @@ qat_sym_session_configure_cipher(struct rte_cryptodev *dev,
{
struct qat_sym_dev_private *internals = dev->data->dev_private;
struct rte_crypto_cipher_xform *cipher_xform = NULL;
+ enum qat_device_gen qat_dev_gen =
+ internals->qat_dev->qat_dev_gen;
int ret;
/* Get cipher xform from crypto xform chain */
@@ -272,6 +274,13 @@ qat_sym_session_configure_cipher(struct rte_cryptodev *dev,
goto error_out;
}
session->qat_mode = ICP_QAT_HW_CIPHER_CTR_MODE;
+ if (qat_dev_gen == QAT_GEN4) {
+ /* TODO: Filter WCP */
+ ICP_QAT_FW_LA_SLICE_TYPE_SET(
+ session->fw_req.comn_hdr.serv_specif_flags,
+ ICP_QAT_FW_LA_USE_UCS_SLICE_TYPE);
+ session->is_ucs = 1;
+ }
break;
case RTE_CRYPTO_CIPHER_SNOW3G_UEA2:
if (qat_sym_validate_snow3g_key(cipher_xform->key.length,
@@ -556,6 +565,7 @@ qat_sym_session_set_parameters(struct rte_cryptodev *dev,
offsetof(struct qat_sym_session, cd);
session->min_qat_dev_gen = QAT_GEN1;
+ session->is_ucs = 0;
/* Get requested QAT command id */
qat_cmd_id = qat_get_cmd_id(xform);
@@ -1518,6 +1528,7 @@ int qat_sym_session_aead_create_cd_cipher(struct qat_sym_session *cdesc,
uint32_t cipherkeylen)
{
struct icp_qat_hw_cipher_algo_blk *cipher;
+ struct icp_qat_hw_cipher_algo_blk20 *cipher20;
struct icp_qat_fw_la_bulk_req *req_tmpl = &cdesc->fw_req;
struct icp_qat_fw_comn_req_hdr_cd_pars *cd_pars = &req_tmpl->cd_pars;
struct icp_qat_fw_comn_req_hdr *header = &req_tmpl->comn_hdr;
@@ -1611,7 +1622,6 @@ int qat_sym_session_aead_create_cd_cipher(struct qat_sym_session *cdesc,
qat_proto_flag =
qat_get_crypto_proto_flag(header->serv_specif_flags);
}
- cipher_cd_ctrl->cipher_key_sz = total_key_size >> 3;
cipher_offset = cdesc->cd_cur_ptr-((uint8_t *)&cdesc->cd);
cipher_cd_ctrl->cipher_cfg_offset = cipher_offset >> 3;
@@ -1619,6 +1629,7 @@ int qat_sym_session_aead_create_cd_cipher(struct qat_sym_session *cdesc,
qat_sym_session_init_common_hdr(header, qat_proto_flag);
cipher = (struct icp_qat_hw_cipher_algo_blk *)cdesc->cd_cur_ptr;
+ cipher20 = (struct icp_qat_hw_cipher_algo_blk20 *)cdesc->cd_cur_ptr;
cipher->cipher_config.val =
ICP_QAT_HW_CIPHER_CONFIG_BUILD(cdesc->qat_mode,
cdesc->qat_cipher_alg, key_convert,
@@ -1638,6 +1649,19 @@ int qat_sym_session_aead_create_cd_cipher(struct qat_sym_session *cdesc,
cdesc->cd_cur_ptr += sizeof(struct icp_qat_hw_cipher_config) +
cipherkeylen + cipherkeylen;
+ } else if (cdesc->is_ucs) {
+ const uint8_t *final_key = cipherkey;
+
+ total_key_size = RTE_ALIGN_CEIL(cipherkeylen,
+ ICP_QAT_HW_AES_128_KEY_SZ);
+ cipher20->cipher_config.reserved[0] = 0;
+ cipher20->cipher_config.reserved[1] = 0;
+ cipher20->cipher_config.reserved[2] = 0;
+
+ rte_memcpy(cipher20->key, final_key, cipherkeylen);
+ cdesc->cd_cur_ptr +=
+ sizeof(struct icp_qat_hw_ucs_cipher_config) +
+ cipherkeylen;
} else {
memcpy(cipher->key, cipherkey, cipherkeylen);
cdesc->cd_cur_ptr += sizeof(struct icp_qat_hw_cipher_config) +
@@ -1664,6 +1688,7 @@ int qat_sym_session_aead_create_cd_cipher(struct qat_sym_session *cdesc,
}
cd_size = cdesc->cd_cur_ptr-(uint8_t *)&cdesc->cd;
cd_pars->u.s.content_desc_params_sz = RTE_ALIGN_CEIL(cd_size, 8) >> 3;
+ cipher_cd_ctrl->cipher_key_sz = total_key_size >> 3;
return 0;
}
diff --git a/drivers/crypto/qat/qat_sym_session.h b/drivers/crypto/qat/qat_sym_session.h
index 72eee06597..4450df6911 100644
--- a/drivers/crypto/qat/qat_sym_session.h
+++ b/drivers/crypto/qat/qat_sym_session.h
@@ -92,6 +92,7 @@ struct qat_sym_session {
uint8_t aes_cmac;
uint8_t is_single_pass;
uint8_t is_single_pass_gmac;
+ uint8_t is_ucs;
};
int
--
2.25.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [dpdk-dev] [PATCH 05/15] crypto/qat: rename content descriptor functions
2021-05-31 14:10 [dpdk-dev] [PATCH 00/15] Add support for fourth generation of Intel QuickAssist Technology devices Arek Kusztal
` (3 preceding siblings ...)
2021-05-31 14:10 ` [dpdk-dev] [PATCH 04/15] crypto/qat: add fourth generation ucs slice type, add ctr mode Arek Kusztal
@ 2021-05-31 14:10 ` Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 06/15] crypto/qat: add legacy gcm and ccm Arek Kusztal
` (9 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Arek Kusztal @ 2021-05-31 14:10 UTC (permalink / raw)
To: dev; +Cc: gakhil, fiona.trahe, roy.fan.zhang, Arek Kusztal
Content descriptor functions are incorrectly named,
having them with proper name will improve readability and
facilitate further work.
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
---
drivers/crypto/qat/qat_sym_session.c | 39 ++++++++++++++++++----------
drivers/crypto/qat/qat_sym_session.h | 13 ----------
2 files changed, 26 insertions(+), 26 deletions(-)
diff --git a/drivers/crypto/qat/qat_sym_session.c b/drivers/crypto/qat/qat_sym_session.c
index 2c44b1f1aa..56c85e8435 100644
--- a/drivers/crypto/qat/qat_sym_session.c
+++ b/drivers/crypto/qat/qat_sym_session.c
@@ -57,6 +57,19 @@ static const uint8_t sha512InitialState[] = {
0x2b, 0x3e, 0x6c, 0x1f, 0x1f, 0x83, 0xd9, 0xab, 0xfb, 0x41, 0xbd,
0x6b, 0x5b, 0xe0, 0xcd, 0x19, 0x13, 0x7e, 0x21, 0x79};
+static int
+qat_sym_cd_cipher_set(struct qat_sym_session *cd,
+ const uint8_t *enckey,
+ uint32_t enckeylen);
+
+static int
+qat_sym_cd_auth_set(struct qat_sym_session *cdesc,
+ const uint8_t *authkey,
+ uint32_t authkeylen,
+ uint32_t aad_length,
+ uint32_t digestsize,
+ unsigned int operation);
+
/** Frees a context previously created
* Depends on openssl libcrypto
*/
@@ -420,7 +433,7 @@ qat_sym_session_configure_cipher(struct rte_cryptodev *dev,
else
session->qat_dir = ICP_QAT_HW_CIPHER_DECRYPT;
- if (qat_sym_session_aead_create_cd_cipher(session,
+ if (qat_sym_cd_cipher_set(session,
cipher_xform->key.data,
cipher_xform->key.length)) {
ret = -EINVAL;
@@ -669,7 +682,7 @@ qat_sym_session_handle_single_pass(struct qat_sym_session *session,
}
session->cipher_iv.offset = aead_xform->iv.offset;
session->cipher_iv.length = aead_xform->iv.length;
- if (qat_sym_session_aead_create_cd_cipher(session,
+ if (qat_sym_cd_cipher_set(session,
aead_xform->key.data, aead_xform->key.length))
return -EINVAL;
session->aad_len = aead_xform->aad_length;
@@ -825,12 +838,12 @@ qat_sym_session_configure_auth(struct rte_cryptodev *dev,
* then authentication
*/
- if (qat_sym_session_aead_create_cd_cipher(session,
+ if (qat_sym_cd_cipher_set(session,
auth_xform->key.data,
auth_xform->key.length))
return -EINVAL;
- if (qat_sym_session_aead_create_cd_auth(session,
+ if (qat_sym_cd_auth_set(session,
key_data,
key_length,
0,
@@ -845,7 +858,7 @@ qat_sym_session_configure_auth(struct rte_cryptodev *dev,
* then cipher
*/
- if (qat_sym_session_aead_create_cd_auth(session,
+ if (qat_sym_cd_auth_set(session,
key_data,
key_length,
0,
@@ -853,7 +866,7 @@ qat_sym_session_configure_auth(struct rte_cryptodev *dev,
auth_xform->op))
return -EINVAL;
- if (qat_sym_session_aead_create_cd_cipher(session,
+ if (qat_sym_cd_cipher_set(session,
auth_xform->key.data,
auth_xform->key.length))
return -EINVAL;
@@ -861,7 +874,7 @@ qat_sym_session_configure_auth(struct rte_cryptodev *dev,
/* Restore to authentication only only */
session->qat_cmd = ICP_QAT_FW_LA_CMD_AUTH;
} else {
- if (qat_sym_session_aead_create_cd_auth(session,
+ if (qat_sym_cd_auth_set(session,
key_data,
key_length,
0,
@@ -948,12 +961,12 @@ qat_sym_session_configure_aead(struct rte_cryptodev *dev,
crypto_operation = aead_xform->algo == RTE_CRYPTO_AEAD_AES_GCM ?
RTE_CRYPTO_AUTH_OP_GENERATE : RTE_CRYPTO_AUTH_OP_VERIFY;
- if (qat_sym_session_aead_create_cd_cipher(session,
+ if (qat_sym_cd_cipher_set(session,
aead_xform->key.data,
aead_xform->key.length))
return -EINVAL;
- if (qat_sym_session_aead_create_cd_auth(session,
+ if (qat_sym_cd_auth_set(session,
aead_xform->key.data,
aead_xform->key.length,
aead_xform->aad_length,
@@ -970,7 +983,7 @@ qat_sym_session_configure_aead(struct rte_cryptodev *dev,
crypto_operation = aead_xform->algo == RTE_CRYPTO_AEAD_AES_GCM ?
RTE_CRYPTO_AUTH_OP_VERIFY : RTE_CRYPTO_AUTH_OP_GENERATE;
- if (qat_sym_session_aead_create_cd_auth(session,
+ if (qat_sym_cd_auth_set(session,
aead_xform->key.data,
aead_xform->key.length,
aead_xform->aad_length,
@@ -978,7 +991,7 @@ qat_sym_session_configure_aead(struct rte_cryptodev *dev,
crypto_operation))
return -EINVAL;
- if (qat_sym_session_aead_create_cd_cipher(session,
+ if (qat_sym_cd_cipher_set(session,
aead_xform->key.data,
aead_xform->key.length))
return -EINVAL;
@@ -1523,7 +1536,7 @@ qat_get_crypto_proto_flag(uint16_t flags)
return qat_proto_flag;
}
-int qat_sym_session_aead_create_cd_cipher(struct qat_sym_session *cdesc,
+int qat_sym_cd_cipher_set(struct qat_sym_session *cdesc,
const uint8_t *cipherkey,
uint32_t cipherkeylen)
{
@@ -1693,7 +1706,7 @@ int qat_sym_session_aead_create_cd_cipher(struct qat_sym_session *cdesc,
return 0;
}
-int qat_sym_session_aead_create_cd_auth(struct qat_sym_session *cdesc,
+int qat_sym_cd_auth_set(struct qat_sym_session *cdesc,
const uint8_t *authkey,
uint32_t authkeylen,
uint32_t aad_length,
diff --git a/drivers/crypto/qat/qat_sym_session.h b/drivers/crypto/qat/qat_sym_session.h
index 4450df6911..5d28e5a089 100644
--- a/drivers/crypto/qat/qat_sym_session.h
+++ b/drivers/crypto/qat/qat_sym_session.h
@@ -120,19 +120,6 @@ qat_sym_session_configure_auth(struct rte_cryptodev *dev,
struct rte_crypto_sym_xform *xform,
struct qat_sym_session *session);
-int
-qat_sym_session_aead_create_cd_cipher(struct qat_sym_session *cd,
- const uint8_t *enckey,
- uint32_t enckeylen);
-
-int
-qat_sym_session_aead_create_cd_auth(struct qat_sym_session *cdesc,
- const uint8_t *authkey,
- uint32_t authkeylen,
- uint32_t aad_length,
- uint32_t digestsize,
- unsigned int operation);
-
void
qat_sym_session_clear(struct rte_cryptodev *dev,
struct rte_cryptodev_sym_session *session);
--
2.25.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [dpdk-dev] [PATCH 06/15] crypto/qat: add legacy gcm and ccm
2021-05-31 14:10 [dpdk-dev] [PATCH 00/15] Add support for fourth generation of Intel QuickAssist Technology devices Arek Kusztal
` (4 preceding siblings ...)
2021-05-31 14:10 ` [dpdk-dev] [PATCH 05/15] crypto/qat: rename content descriptor functions Arek Kusztal
@ 2021-05-31 14:10 ` Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 07/15] crypto/qat: rework init common header function Arek Kusztal
` (8 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Arek Kusztal @ 2021-05-31 14:10 UTC (permalink / raw)
To: dev; +Cc: gakhil, fiona.trahe, roy.fan.zhang, Arek Kusztal
Add AES-GCM, AES-CCM algorithms in legacy mode.
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
---
drivers/crypto/qat/qat_sym_capabilities.h | 60 +++++++++++++++++++++++
drivers/crypto/qat/qat_sym_session.c | 27 +++++-----
drivers/crypto/qat/qat_sym_session.h | 3 +-
3 files changed, 78 insertions(+), 12 deletions(-)
diff --git a/drivers/crypto/qat/qat_sym_capabilities.h b/drivers/crypto/qat/qat_sym_capabilities.h
index aca528b991..fc8e667687 100644
--- a/drivers/crypto/qat/qat_sym_capabilities.h
+++ b/drivers/crypto/qat/qat_sym_capabilities.h
@@ -1084,6 +1084,66 @@
} \
}, } \
}, } \
+ }, \
+ { /* AES GCM */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_AEAD, \
+ {.aead = { \
+ .algo = RTE_CRYPTO_AEAD_AES_GCM, \
+ .block_size = 16, \
+ .key_size = { \
+ .min = 16, \
+ .max = 32, \
+ .increment = 8 \
+ }, \
+ .digest_size = { \
+ .min = 8, \
+ .max = 16, \
+ .increment = 4 \
+ }, \
+ .aad_size = { \
+ .min = 0, \
+ .max = 240, \
+ .increment = 1 \
+ }, \
+ .iv_size = { \
+ .min = 0, \
+ .max = 12, \
+ .increment = 12 \
+ }, \
+ }, } \
+ }, } \
+ }, \
+ { /* AES CCM */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_AEAD, \
+ {.aead = { \
+ .algo = RTE_CRYPTO_AEAD_AES_CCM, \
+ .block_size = 16, \
+ .key_size = { \
+ .min = 16, \
+ .max = 16, \
+ .increment = 0 \
+ }, \
+ .digest_size = { \
+ .min = 4, \
+ .max = 16, \
+ .increment = 2 \
+ }, \
+ .aad_size = { \
+ .min = 0, \
+ .max = 224, \
+ .increment = 1 \
+ }, \
+ .iv_size = { \
+ .min = 7, \
+ .max = 13, \
+ .increment = 1 \
+ }, \
+ }, } \
+ }, } \
} \
diff --git a/drivers/crypto/qat/qat_sym_session.c b/drivers/crypto/qat/qat_sym_session.c
index 56c85e8435..5140d61a9c 100644
--- a/drivers/crypto/qat/qat_sym_session.c
+++ b/drivers/crypto/qat/qat_sym_session.c
@@ -287,13 +287,8 @@ qat_sym_session_configure_cipher(struct rte_cryptodev *dev,
goto error_out;
}
session->qat_mode = ICP_QAT_HW_CIPHER_CTR_MODE;
- if (qat_dev_gen == QAT_GEN4) {
- /* TODO: Filter WCP */
- ICP_QAT_FW_LA_SLICE_TYPE_SET(
- session->fw_req.comn_hdr.serv_specif_flags,
- ICP_QAT_FW_LA_USE_UCS_SLICE_TYPE);
+ if (qat_dev_gen == QAT_GEN4)
session->is_ucs = 1;
- }
break;
case RTE_CRYPTO_CIPHER_SNOW3G_UEA2:
if (qat_sym_validate_snow3g_key(cipher_xform->key.length,
@@ -918,14 +913,15 @@ qat_sym_session_configure_aead(struct rte_cryptodev *dev,
}
session->qat_mode = ICP_QAT_HW_CIPHER_CTR_MODE;
session->qat_hash_alg = ICP_QAT_HW_AUTH_ALGO_GALOIS_128;
- if (qat_dev_gen > QAT_GEN2 && aead_xform->iv.length ==
+ if (qat_dev_gen == QAT_GEN3 && aead_xform->iv.length ==
QAT_AES_GCM_SPC_IV_SIZE) {
return qat_sym_session_handle_single_pass(session,
aead_xform);
}
if (session->cipher_iv.length == 0)
session->cipher_iv.length = AES_GCM_J0_LEN;
-
+ if (qat_dev_gen == QAT_GEN4)
+ session->is_ucs = 1;
break;
case RTE_CRYPTO_AEAD_AES_CCM:
if (qat_sym_validate_aes_key(aead_xform->key.length,
@@ -935,6 +931,8 @@ qat_sym_session_configure_aead(struct rte_cryptodev *dev,
}
session->qat_mode = ICP_QAT_HW_CIPHER_CTR_MODE;
session->qat_hash_alg = ICP_QAT_HW_AUTH_ALGO_AES_CBC_MAC;
+ if (qat_dev_gen == QAT_GEN4)
+ session->is_ucs = 1;
break;
case RTE_CRYPTO_AEAD_CHACHA20_POLY1305:
if (aead_xform->key.length != ICP_QAT_HW_CHACHAPOLY_KEY_SZ)
@@ -1469,7 +1467,8 @@ static int qat_sym_do_precomputes(enum icp_qat_hw_auth_algo hash_alg,
}
static void
-qat_sym_session_init_common_hdr(struct icp_qat_fw_comn_req_hdr *header,
+qat_sym_session_init_common_hdr(struct qat_sym_session *session,
+ struct icp_qat_fw_comn_req_hdr *header,
enum qat_sym_proto_flag proto_flags)
{
header->hdr_flags =
@@ -1510,6 +1509,12 @@ qat_sym_session_init_common_hdr(struct icp_qat_fw_comn_req_hdr *header,
ICP_QAT_FW_LA_NO_UPDATE_STATE);
ICP_QAT_FW_LA_DIGEST_IN_BUFFER_SET(header->serv_specif_flags,
ICP_QAT_FW_LA_NO_DIGEST_IN_BUFFER);
+
+ if (session->is_ucs) {
+ ICP_QAT_FW_LA_SLICE_TYPE_SET(
+ session->fw_req.comn_hdr.serv_specif_flags,
+ ICP_QAT_FW_LA_USE_UCS_SLICE_TYPE);
+ }
}
/*
@@ -1639,7 +1644,7 @@ int qat_sym_cd_cipher_set(struct qat_sym_session *cdesc,
cipher_cd_ctrl->cipher_cfg_offset = cipher_offset >> 3;
header->service_cmd_id = cdesc->qat_cmd;
- qat_sym_session_init_common_hdr(header, qat_proto_flag);
+ qat_sym_session_init_common_hdr(cdesc, header, qat_proto_flag);
cipher = (struct icp_qat_hw_cipher_algo_blk *)cdesc->cd_cur_ptr;
cipher20 = (struct icp_qat_hw_cipher_algo_blk20 *)cdesc->cd_cur_ptr;
@@ -2032,7 +2037,7 @@ int qat_sym_cd_auth_set(struct qat_sym_session *cdesc,
}
/* Request template setup */
- qat_sym_session_init_common_hdr(header, qat_proto_flag);
+ qat_sym_session_init_common_hdr(cdesc, header, qat_proto_flag);
header->service_cmd_id = cdesc->qat_cmd;
/* Auth CD config setup */
diff --git a/drivers/crypto/qat/qat_sym_session.h b/drivers/crypto/qat/qat_sym_session.h
index 5d28e5a089..e003a34f7f 100644
--- a/drivers/crypto/qat/qat_sym_session.h
+++ b/drivers/crypto/qat/qat_sym_session.h
@@ -128,7 +128,8 @@ unsigned int
qat_sym_session_get_private_size(struct rte_cryptodev *dev);
void
-qat_sym_sesssion_init_common_hdr(struct icp_qat_fw_comn_req_hdr *header,
+qat_sym_sesssion_init_common_hdr(struct qat_sym_session *session,
+ struct icp_qat_fw_comn_req_hdr *header,
enum qat_sym_proto_flag proto_flags);
int
qat_sym_validate_aes_key(int key_len, enum icp_qat_hw_cipher_algo *alg);
--
2.25.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [dpdk-dev] [PATCH 07/15] crypto/qat: rework init common header function
2021-05-31 14:10 [dpdk-dev] [PATCH 00/15] Add support for fourth generation of Intel QuickAssist Technology devices Arek Kusztal
` (5 preceding siblings ...)
2021-05-31 14:10 ` [dpdk-dev] [PATCH 06/15] crypto/qat: add legacy gcm and ccm Arek Kusztal
@ 2021-05-31 14:10 ` Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 08/15] crypto/qat: add aes gcm in ucs spc mode Arek Kusztal
` (7 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Arek Kusztal @ 2021-05-31 14:10 UTC (permalink / raw)
To: dev; +Cc: gakhil, fiona.trahe, roy.fan.zhang, Arek Kusztal
Rework init common header function for request
descriptor so it can be called only once.
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
---
drivers/crypto/qat/qat_sym.c | 25 +--
drivers/crypto/qat/qat_sym_session.c | 264 ++++++++++++++-------------
drivers/crypto/qat/qat_sym_session.h | 12 ++
3 files changed, 157 insertions(+), 144 deletions(-)
diff --git a/drivers/crypto/qat/qat_sym.c b/drivers/crypto/qat/qat_sym.c
index 9415ec7d32..eef4a886c5 100644
--- a/drivers/crypto/qat/qat_sym.c
+++ b/drivers/crypto/qat/qat_sym.c
@@ -289,8 +289,9 @@ qat_sym_build_request(void *in_op, uint8_t *out_msg,
auth_param = (void *)((uint8_t *)cipher_param +
ICP_QAT_FW_HASH_REQUEST_PARAMETERS_OFFSET);
- if (ctx->qat_cmd == ICP_QAT_FW_LA_CMD_HASH_CIPHER ||
- ctx->qat_cmd == ICP_QAT_FW_LA_CMD_CIPHER_HASH) {
+ if ((ctx->qat_cmd == ICP_QAT_FW_LA_CMD_HASH_CIPHER ||
+ ctx->qat_cmd == ICP_QAT_FW_LA_CMD_CIPHER_HASH) &&
+ !ctx->is_gmac) {
/* AES-GCM or AES-CCM */
if (ctx->qat_hash_alg == ICP_QAT_HW_AUTH_ALGO_GALOIS_128 ||
ctx->qat_hash_alg == ICP_QAT_HW_AUTH_ALGO_GALOIS_64 ||
@@ -303,7 +304,7 @@ qat_sym_build_request(void *in_op, uint8_t *out_msg,
do_auth = 1;
do_cipher = 1;
}
- } else if (ctx->qat_cmd == ICP_QAT_FW_LA_CMD_AUTH) {
+ } else if (ctx->qat_cmd == ICP_QAT_FW_LA_CMD_AUTH || ctx->is_gmac) {
do_auth = 1;
do_cipher = 0;
} else if (ctx->qat_cmd == ICP_QAT_FW_LA_CMD_CIPHER) {
@@ -383,15 +384,6 @@ qat_sym_build_request(void *in_op, uint8_t *out_msg,
auth_param->u1.aad_adr = 0;
auth_param->u2.aad_sz = 0;
- /*
- * If len(iv)==12B fw computes J0
- */
- if (ctx->auth_iv.length == 12) {
- ICP_QAT_FW_LA_GCM_IV_LEN_FLAG_SET(
- qat_req->comn_hdr.serv_specif_flags,
- ICP_QAT_FW_LA_GCM_IV_LEN_12_OCTETS);
-
- }
} else {
auth_ofs = op->sym->auth.data.offset;
auth_len = op->sym->auth.data.length;
@@ -416,14 +408,7 @@ qat_sym_build_request(void *in_op, uint8_t *out_msg,
ICP_QAT_HW_AUTH_ALGO_GALOIS_128 ||
ctx->qat_hash_alg ==
ICP_QAT_HW_AUTH_ALGO_GALOIS_64) {
- /*
- * If len(iv)==12B fw computes J0
- */
- if (ctx->cipher_iv.length == 12) {
- ICP_QAT_FW_LA_GCM_IV_LEN_FLAG_SET(
- qat_req->comn_hdr.serv_specif_flags,
- ICP_QAT_FW_LA_GCM_IV_LEN_12_OCTETS);
- }
+
set_cipher_iv(ctx->cipher_iv.length,
ctx->cipher_iv.offset,
cipher_param, op, qat_req);
diff --git a/drivers/crypto/qat/qat_sym_session.c b/drivers/crypto/qat/qat_sym_session.c
index 5140d61a9c..4b52ffd459 100644
--- a/drivers/crypto/qat/qat_sym_session.c
+++ b/drivers/crypto/qat/qat_sym_session.c
@@ -69,6 +69,15 @@ qat_sym_cd_auth_set(struct qat_sym_session *cdesc,
uint32_t aad_length,
uint32_t digestsize,
unsigned int operation);
+static void
+qat_sym_session_init_common_hdr(struct qat_sym_session *session);
+
+/* Req/cd init functions */
+
+static void qat_sym_session_finalize(struct qat_sym_session *session)
+{
+ qat_sym_session_init_common_hdr(session);
+}
/** Frees a context previously created
* Depends on openssl libcrypto
@@ -558,6 +567,7 @@ qat_sym_session_set_parameters(struct rte_cryptodev *dev,
enum qat_device_gen qat_dev_gen = internals->qat_dev->qat_dev_gen;
int ret;
int qat_cmd_id;
+ int handle_mixed = 0;
/* Verify the session physical address is known */
rte_iova_t session_paddr = rte_mempool_virt2iova(session);
@@ -573,6 +583,7 @@ qat_sym_session_set_parameters(struct rte_cryptodev *dev,
offsetof(struct qat_sym_session, cd);
session->min_qat_dev_gen = QAT_GEN1;
+ session->qat_proto_flag = QAT_CRYPTO_PROTO_FLAG_NONE;
session->is_ucs = 0;
/* Get requested QAT command id */
@@ -612,8 +623,7 @@ qat_sym_session_set_parameters(struct rte_cryptodev *dev,
xform, session);
if (ret < 0)
return ret;
- /* Special handling of mixed hash+cipher algorithms */
- qat_sym_session_handle_mixed(dev, session);
+ handle_mixed = 1;
}
break;
case ICP_QAT_FW_LA_CMD_HASH_CIPHER:
@@ -631,8 +641,7 @@ qat_sym_session_set_parameters(struct rte_cryptodev *dev,
xform, session);
if (ret < 0)
return ret;
- /* Special handling of mixed hash+cipher algorithms */
- qat_sym_session_handle_mixed(dev, session);
+ handle_mixed = 1;
}
break;
case ICP_QAT_FW_LA_CMD_TRNG_GET_RANDOM:
@@ -652,72 +661,41 @@ qat_sym_session_set_parameters(struct rte_cryptodev *dev,
session->qat_cmd);
return -ENOTSUP;
}
+ qat_sym_session_finalize(session);
+ if (handle_mixed) {
+ /* Special handling of mixed hash+cipher algorithms */
+ qat_sym_session_handle_mixed(dev, session);
+ }
return 0;
}
static int
qat_sym_session_handle_single_pass(struct qat_sym_session *session,
- struct rte_crypto_aead_xform *aead_xform)
+ const struct rte_crypto_aead_xform *aead_xform)
{
- struct icp_qat_fw_la_cipher_req_params *cipher_param =
- (void *) &session->fw_req.serv_specif_rqpars;
-
session->is_single_pass = 1;
+ session->is_auth = 1;
session->min_qat_dev_gen = QAT_GEN3;
session->qat_cmd = ICP_QAT_FW_LA_CMD_CIPHER;
+ /* Chacha-Poly is special case that use QAT CTR mode */
if (aead_xform->algo == RTE_CRYPTO_AEAD_AES_GCM) {
session->qat_mode = ICP_QAT_HW_CIPHER_AEAD_MODE;
- ICP_QAT_FW_LA_GCM_IV_LEN_FLAG_SET(
- session->fw_req.comn_hdr.serv_specif_flags,
- ICP_QAT_FW_LA_GCM_IV_LEN_12_OCTETS);
} else {
- /* Chacha-Poly is special case that use QAT CTR mode */
session->qat_mode = ICP_QAT_HW_CIPHER_CTR_MODE;
}
session->cipher_iv.offset = aead_xform->iv.offset;
session->cipher_iv.length = aead_xform->iv.length;
- if (qat_sym_cd_cipher_set(session,
- aead_xform->key.data, aead_xform->key.length))
- return -EINVAL;
session->aad_len = aead_xform->aad_length;
session->digest_length = aead_xform->digest_length;
+
if (aead_xform->op == RTE_CRYPTO_AEAD_OP_ENCRYPT) {
session->qat_dir = ICP_QAT_HW_CIPHER_ENCRYPT;
session->auth_op = ICP_QAT_HW_AUTH_GENERATE;
- ICP_QAT_FW_LA_RET_AUTH_SET(
- session->fw_req.comn_hdr.serv_specif_flags,
- ICP_QAT_FW_LA_RET_AUTH_RES);
} else {
session->qat_dir = ICP_QAT_HW_CIPHER_DECRYPT;
session->auth_op = ICP_QAT_HW_AUTH_VERIFY;
- ICP_QAT_FW_LA_CMP_AUTH_SET(
- session->fw_req.comn_hdr.serv_specif_flags,
- ICP_QAT_FW_LA_CMP_AUTH_RES);
}
- ICP_QAT_FW_LA_SINGLE_PASS_PROTO_FLAG_SET(
- session->fw_req.comn_hdr.serv_specif_flags,
- ICP_QAT_FW_LA_SINGLE_PASS_PROTO);
- ICP_QAT_FW_LA_PROTO_SET(
- session->fw_req.comn_hdr.serv_specif_flags,
- ICP_QAT_FW_LA_NO_PROTO);
- session->fw_req.comn_hdr.service_cmd_id =
- ICP_QAT_FW_LA_CMD_CIPHER;
- session->cd.cipher.cipher_config.val =
- ICP_QAT_HW_CIPHER_CONFIG_BUILD(
- ICP_QAT_HW_CIPHER_AEAD_MODE,
- session->qat_cipher_alg,
- ICP_QAT_HW_CIPHER_NO_CONVERT,
- session->qat_dir);
- QAT_FIELD_SET(session->cd.cipher.cipher_config.val,
- aead_xform->digest_length,
- QAT_CIPHER_AEAD_HASH_CMP_LEN_BITPOS,
- QAT_CIPHER_AEAD_HASH_CMP_LEN_MASK);
- session->cd.cipher.cipher_config.reserved =
- ICP_QAT_HW_CIPHER_CONFIG_BUILD_UPPER(
- aead_xform->aad_length);
- cipher_param->spc_aad_sz = aead_xform->aad_length;
- cipher_param->spc_auth_res_sz = aead_xform->digest_length;
return 0;
}
@@ -737,6 +715,7 @@ qat_sym_session_configure_auth(struct rte_cryptodev *dev,
session->auth_iv.offset = auth_xform->iv.offset;
session->auth_iv.length = auth_xform->iv.length;
session->auth_mode = ICP_QAT_HW_AUTH_MODE1;
+ session->is_auth = 1;
switch (auth_xform->algo) {
case RTE_CRYPTO_AUTH_SHA1:
@@ -791,6 +770,8 @@ qat_sym_session_configure_auth(struct rte_cryptodev *dev,
session->qat_hash_alg = ICP_QAT_HW_AUTH_ALGO_GALOIS_128;
if (session->auth_iv.length == 0)
session->auth_iv.length = AES_GCM_J0_LEN;
+ else
+ session->is_iv12B = 1;
break;
case RTE_CRYPTO_AUTH_SNOW3G_UIA2:
session->qat_hash_alg = ICP_QAT_HW_AUTH_ALGO_SNOW_3G_UIA2;
@@ -825,6 +806,7 @@ qat_sym_session_configure_auth(struct rte_cryptodev *dev,
}
if (auth_xform->algo == RTE_CRYPTO_AUTH_AES_GMAC) {
+ session->is_gmac = 1;
if (auth_xform->op == RTE_CRYPTO_AUTH_OP_GENERATE) {
session->qat_cmd = ICP_QAT_FW_LA_CMD_CIPHER_HASH;
session->qat_dir = ICP_QAT_HW_CIPHER_ENCRYPT;
@@ -832,7 +814,6 @@ qat_sym_session_configure_auth(struct rte_cryptodev *dev,
* It needs to create cipher desc content first,
* then authentication
*/
-
if (qat_sym_cd_cipher_set(session,
auth_xform->key.data,
auth_xform->key.length))
@@ -866,8 +847,6 @@ qat_sym_session_configure_auth(struct rte_cryptodev *dev,
auth_xform->key.length))
return -EINVAL;
}
- /* Restore to authentication only only */
- session->qat_cmd = ICP_QAT_FW_LA_CMD_AUTH;
} else {
if (qat_sym_cd_auth_set(session,
key_data,
@@ -902,6 +881,8 @@ qat_sym_session_configure_aead(struct rte_cryptodev *dev,
session->cipher_iv.length = xform->aead.iv.length;
session->auth_mode = ICP_QAT_HW_AUTH_MODE1;
+ session->is_auth = 1;
+ session->digest_length = aead_xform->digest_length;
session->is_single_pass = 0;
switch (aead_xform->algo) {
@@ -913,15 +894,19 @@ qat_sym_session_configure_aead(struct rte_cryptodev *dev,
}
session->qat_mode = ICP_QAT_HW_CIPHER_CTR_MODE;
session->qat_hash_alg = ICP_QAT_HW_AUTH_ALGO_GALOIS_128;
- if (qat_dev_gen == QAT_GEN3 && aead_xform->iv.length ==
- QAT_AES_GCM_SPC_IV_SIZE) {
- return qat_sym_session_handle_single_pass(session,
- aead_xform);
- }
- if (session->cipher_iv.length == 0)
- session->cipher_iv.length = AES_GCM_J0_LEN;
+
if (qat_dev_gen == QAT_GEN4)
session->is_ucs = 1;
+
+ if (session->cipher_iv.length == 0) {
+ session->cipher_iv.length = AES_GCM_J0_LEN;
+ break;
+ }
+ session->is_iv12B = 1;
+ if (qat_dev_gen == QAT_GEN3) {
+ qat_sym_session_handle_single_pass(session,
+ aead_xform);
+ }
break;
case RTE_CRYPTO_AEAD_AES_CCM:
if (qat_sym_validate_aes_key(aead_xform->key.length,
@@ -939,15 +924,20 @@ qat_sym_session_configure_aead(struct rte_cryptodev *dev,
return -EINVAL;
session->qat_cipher_alg =
ICP_QAT_HW_CIPHER_ALGO_CHACHA20_POLY1305;
- return qat_sym_session_handle_single_pass(session,
+ qat_sym_session_handle_single_pass(session,
aead_xform);
+ break;
default:
QAT_LOG(ERR, "Crypto: Undefined AEAD specified %u\n",
aead_xform->algo);
return -EINVAL;
}
- if ((aead_xform->op == RTE_CRYPTO_AEAD_OP_ENCRYPT &&
+ if (session->is_single_pass) {
+ if (qat_sym_cd_cipher_set(session,
+ aead_xform->key.data, aead_xform->key.length))
+ return -EINVAL;
+ } else if ((aead_xform->op == RTE_CRYPTO_AEAD_OP_ENCRYPT &&
aead_xform->algo == RTE_CRYPTO_AEAD_AES_GCM) ||
(aead_xform->op == RTE_CRYPTO_AEAD_OP_DECRYPT &&
aead_xform->algo == RTE_CRYPTO_AEAD_AES_CCM)) {
@@ -995,7 +985,6 @@ qat_sym_session_configure_aead(struct rte_cryptodev *dev,
return -EINVAL;
}
- session->digest_length = aead_xform->digest_length;
return 0;
}
@@ -1467,13 +1456,17 @@ static int qat_sym_do_precomputes(enum icp_qat_hw_auth_algo hash_alg,
}
static void
-qat_sym_session_init_common_hdr(struct qat_sym_session *session,
- struct icp_qat_fw_comn_req_hdr *header,
- enum qat_sym_proto_flag proto_flags)
+qat_sym_session_init_common_hdr(struct qat_sym_session *session)
{
+ struct icp_qat_fw_la_bulk_req *req_tmpl = &session->fw_req;
+ struct icp_qat_fw_comn_req_hdr *header = &req_tmpl->comn_hdr;
+ enum qat_sym_proto_flag proto_flags = session->qat_proto_flag;
+ uint32_t slice_flags = session->slice_types;
+
header->hdr_flags =
ICP_QAT_FW_COMN_HDR_FLAGS_BUILD(ICP_QAT_FW_COMN_REQ_FLAG_SET);
header->service_type = ICP_QAT_FW_COMN_REQ_CPM_FW_LA;
+ header->service_cmd_id = session->qat_cmd;
header->comn_req_flags =
ICP_QAT_FW_COMN_FLAGS_BUILD(QAT_COMN_CD_FLD_TYPE_64BIT_ADR,
QAT_COMN_PTR_TYPE_FLAT);
@@ -1505,40 +1498,47 @@ qat_sym_session_init_common_hdr(struct qat_sym_session *session,
break;
}
- ICP_QAT_FW_LA_UPDATE_STATE_SET(header->serv_specif_flags,
- ICP_QAT_FW_LA_NO_UPDATE_STATE);
- ICP_QAT_FW_LA_DIGEST_IN_BUFFER_SET(header->serv_specif_flags,
- ICP_QAT_FW_LA_NO_DIGEST_IN_BUFFER);
-
- if (session->is_ucs) {
+ /* More than one of the following flags can be set at once */
+ if (QAT_SESSION_IS_SLICE_SET(slice_flags, QAT_CRYPTO_SLICE_SPC)) {
+ ICP_QAT_FW_LA_SINGLE_PASS_PROTO_FLAG_SET(
+ header->serv_specif_flags,
+ ICP_QAT_FW_LA_SINGLE_PASS_PROTO);
+ }
+ if (QAT_SESSION_IS_SLICE_SET(slice_flags, QAT_CRYPTO_SLICE_UCS)) {
ICP_QAT_FW_LA_SLICE_TYPE_SET(
- session->fw_req.comn_hdr.serv_specif_flags,
- ICP_QAT_FW_LA_USE_UCS_SLICE_TYPE);
+ header->serv_specif_flags,
+ ICP_QAT_FW_LA_USE_UCS_SLICE_TYPE);
}
-}
-/*
- * Snow3G and ZUC should never use this function
- * and set its protocol flag in both cipher and auth part of content
- * descriptor building function
- */
-static enum qat_sym_proto_flag
-qat_get_crypto_proto_flag(uint16_t flags)
-{
- int proto = ICP_QAT_FW_LA_PROTO_GET(flags);
- enum qat_sym_proto_flag qat_proto_flag =
- QAT_CRYPTO_PROTO_FLAG_NONE;
+ if (session->is_auth) {
+ if (session->auth_op == ICP_QAT_HW_AUTH_VERIFY) {
+ ICP_QAT_FW_LA_RET_AUTH_SET(header->serv_specif_flags,
+ ICP_QAT_FW_LA_NO_RET_AUTH_RES);
+ ICP_QAT_FW_LA_CMP_AUTH_SET(header->serv_specif_flags,
+ ICP_QAT_FW_LA_CMP_AUTH_RES);
+ } else if (session->auth_op == ICP_QAT_HW_AUTH_GENERATE) {
+ ICP_QAT_FW_LA_RET_AUTH_SET(header->serv_specif_flags,
+ ICP_QAT_FW_LA_RET_AUTH_RES);
+ ICP_QAT_FW_LA_CMP_AUTH_SET(header->serv_specif_flags,
+ ICP_QAT_FW_LA_NO_CMP_AUTH_RES);
+ }
+ } else {
+ ICP_QAT_FW_LA_RET_AUTH_SET(header->serv_specif_flags,
+ ICP_QAT_FW_LA_NO_RET_AUTH_RES);
+ ICP_QAT_FW_LA_CMP_AUTH_SET(header->serv_specif_flags,
+ ICP_QAT_FW_LA_NO_CMP_AUTH_RES);
+ }
- switch (proto) {
- case ICP_QAT_FW_LA_GCM_PROTO:
- qat_proto_flag = QAT_CRYPTO_PROTO_FLAG_GCM;
- break;
- case ICP_QAT_FW_LA_CCM_PROTO:
- qat_proto_flag = QAT_CRYPTO_PROTO_FLAG_CCM;
- break;
+ if (session->is_iv12B) {
+ ICP_QAT_FW_LA_GCM_IV_LEN_FLAG_SET(
+ header->serv_specif_flags,
+ ICP_QAT_FW_LA_GCM_IV_LEN_12_OCTETS);
}
- return qat_proto_flag;
+ ICP_QAT_FW_LA_UPDATE_STATE_SET(header->serv_specif_flags,
+ ICP_QAT_FW_LA_NO_UPDATE_STATE);
+ ICP_QAT_FW_LA_DIGEST_IN_BUFFER_SET(header->serv_specif_flags,
+ ICP_QAT_FW_LA_NO_DIGEST_IN_BUFFER);
}
int qat_sym_cd_cipher_set(struct qat_sym_session *cdesc,
@@ -1554,8 +1554,12 @@ int qat_sym_cd_cipher_set(struct qat_sym_session *cdesc,
struct icp_qat_fw_cipher_cd_ctrl_hdr *cipher_cd_ctrl = ptr;
struct icp_qat_fw_auth_cd_ctrl_hdr *hash_cd_ctrl = ptr;
enum icp_qat_hw_cipher_convert key_convert;
- enum qat_sym_proto_flag qat_proto_flag =
- QAT_CRYPTO_PROTO_FLAG_NONE;
+ struct icp_qat_fw_la_cipher_20_req_params *req_ucs =
+ (struct icp_qat_fw_la_cipher_20_req_params *)
+ &cdesc->fw_req.serv_specif_rqpars;
+ struct icp_qat_fw_la_cipher_req_params *req_cipher =
+ (struct icp_qat_fw_la_cipher_req_params *)
+ &cdesc->fw_req.serv_specif_rqpars;
uint32_t total_key_size;
uint16_t cipher_offset, cd_size;
uint32_t wordIndex = 0;
@@ -1591,9 +1595,16 @@ int qat_sym_cd_cipher_set(struct qat_sym_session *cdesc,
if (cdesc->qat_mode == ICP_QAT_HW_CIPHER_CTR_MODE) {
/*
* CTR Streaming ciphers are a special case. Decrypt = encrypt
- * Overriding default values previously set
+ * Overriding default values previously set.
+ * Chacha20-Poly1305 is special case, CTR but single-pass
+ * so both direction need to be used.
*/
cdesc->qat_dir = ICP_QAT_HW_CIPHER_ENCRYPT;
+ if (cdesc->qat_cipher_alg ==
+ ICP_QAT_HW_CIPHER_ALGO_CHACHA20_POLY1305 &&
+ cdesc->auth_op == ICP_QAT_HW_AUTH_VERIFY) {
+ cdesc->qat_dir = ICP_QAT_HW_CIPHER_DECRYPT;
+ }
key_convert = ICP_QAT_HW_CIPHER_NO_CONVERT;
} else if (cdesc->qat_cipher_alg == ICP_QAT_HW_CIPHER_ALGO_SNOW_3G_UEA2
|| cdesc->qat_cipher_alg ==
@@ -1601,6 +1612,8 @@ int qat_sym_cd_cipher_set(struct qat_sym_session *cdesc,
key_convert = ICP_QAT_HW_CIPHER_KEY_CONVERT;
else if (cdesc->qat_dir == ICP_QAT_HW_CIPHER_ENCRYPT)
key_convert = ICP_QAT_HW_CIPHER_NO_CONVERT;
+ else if (cdesc->qat_mode == ICP_QAT_HW_CIPHER_AEAD_MODE)
+ key_convert = ICP_QAT_HW_CIPHER_NO_CONVERT;
else
key_convert = ICP_QAT_HW_CIPHER_KEY_CONVERT;
@@ -1609,7 +1622,7 @@ int qat_sym_cd_cipher_set(struct qat_sym_session *cdesc,
ICP_QAT_HW_SNOW_3G_UEA2_IV_SZ;
cipher_cd_ctrl->cipher_state_sz =
ICP_QAT_HW_SNOW_3G_UEA2_IV_SZ >> 3;
- qat_proto_flag = QAT_CRYPTO_PROTO_FLAG_SNOW3G;
+ cdesc->qat_proto_flag = QAT_CRYPTO_PROTO_FLAG_SNOW3G;
} else if (cdesc->qat_cipher_alg == ICP_QAT_HW_CIPHER_ALGO_KASUMI) {
total_key_size = ICP_QAT_HW_KASUMI_F8_KEY_SZ;
@@ -1619,33 +1632,24 @@ int qat_sym_cd_cipher_set(struct qat_sym_session *cdesc,
} else if (cdesc->qat_cipher_alg == ICP_QAT_HW_CIPHER_ALGO_3DES) {
total_key_size = ICP_QAT_HW_3DES_KEY_SZ;
cipher_cd_ctrl->cipher_state_sz = ICP_QAT_HW_3DES_BLK_SZ >> 3;
- qat_proto_flag =
- qat_get_crypto_proto_flag(header->serv_specif_flags);
} else if (cdesc->qat_cipher_alg == ICP_QAT_HW_CIPHER_ALGO_DES) {
total_key_size = ICP_QAT_HW_DES_KEY_SZ;
cipher_cd_ctrl->cipher_state_sz = ICP_QAT_HW_DES_BLK_SZ >> 3;
- qat_proto_flag =
- qat_get_crypto_proto_flag(header->serv_specif_flags);
} else if (cdesc->qat_cipher_alg ==
ICP_QAT_HW_CIPHER_ALGO_ZUC_3G_128_EEA3) {
total_key_size = ICP_QAT_HW_ZUC_3G_EEA3_KEY_SZ +
ICP_QAT_HW_ZUC_3G_EEA3_IV_SZ;
cipher_cd_ctrl->cipher_state_sz =
ICP_QAT_HW_ZUC_3G_EEA3_IV_SZ >> 3;
- qat_proto_flag = QAT_CRYPTO_PROTO_FLAG_ZUC;
+ cdesc->qat_proto_flag = QAT_CRYPTO_PROTO_FLAG_ZUC;
cdesc->min_qat_dev_gen = QAT_GEN2;
} else {
total_key_size = cipherkeylen;
cipher_cd_ctrl->cipher_state_sz = ICP_QAT_HW_AES_BLK_SZ >> 3;
- qat_proto_flag =
- qat_get_crypto_proto_flag(header->serv_specif_flags);
}
cipher_offset = cdesc->cd_cur_ptr-((uint8_t *)&cdesc->cd);
cipher_cd_ctrl->cipher_cfg_offset = cipher_offset >> 3;
- header->service_cmd_id = cdesc->qat_cmd;
- qat_sym_session_init_common_hdr(cdesc, header, qat_proto_flag);
-
cipher = (struct icp_qat_hw_cipher_algo_blk *)cdesc->cd_cur_ptr;
cipher20 = (struct icp_qat_hw_cipher_algo_blk20 *)cdesc->cd_cur_ptr;
cipher->cipher_config.val =
@@ -1670,6 +1674,7 @@ int qat_sym_cd_cipher_set(struct qat_sym_session *cdesc,
} else if (cdesc->is_ucs) {
const uint8_t *final_key = cipherkey;
+ cdesc->slice_types |= QAT_CRYPTO_SLICE_UCS;
total_key_size = RTE_ALIGN_CEIL(cipherkeylen,
ICP_QAT_HW_AES_128_KEY_SZ);
cipher20->cipher_config.reserved[0] = 0;
@@ -1686,6 +1691,18 @@ int qat_sym_cd_cipher_set(struct qat_sym_session *cdesc,
cipherkeylen;
}
+ if (cdesc->is_single_pass) {
+ QAT_FIELD_SET(cipher->cipher_config.val,
+ cdesc->digest_length,
+ QAT_CIPHER_AEAD_HASH_CMP_LEN_BITPOS,
+ QAT_CIPHER_AEAD_HASH_CMP_LEN_MASK);
+ /* UCS and SPC 1.8/2.0 share configuration of 2nd config word */
+ cdesc->cd.cipher.cipher_config.reserved =
+ ICP_QAT_HW_CIPHER_CONFIG_BUILD_UPPER(
+ cdesc->aad_len);
+ cdesc->slice_types |= QAT_CRYPTO_SLICE_SPC;
+ }
+
if (total_key_size > cipherkeylen) {
uint32_t padding_size = total_key_size-cipherkeylen;
if ((cdesc->qat_cipher_alg == ICP_QAT_HW_CIPHER_ALGO_3DES)
@@ -1704,6 +1721,20 @@ int qat_sym_cd_cipher_set(struct qat_sym_session *cdesc,
cdesc->cd_cur_ptr += padding_size;
}
+ if (cdesc->is_ucs) {
+ /*
+ * These values match in terms of position auth
+ * slice request fields
+ */
+ req_ucs->spc_auth_res_sz = cdesc->digest_length;
+ if (!cdesc->is_gmac) {
+ req_ucs->spc_aad_sz = cdesc->aad_len;
+ req_ucs->spc_aad_offset = 0;
+ }
+ } else if (cdesc->is_single_pass) {
+ req_cipher->spc_aad_sz = cdesc->aad_len;
+ req_cipher->spc_auth_res_sz = cdesc->digest_length;
+ }
cd_size = cdesc->cd_cur_ptr-(uint8_t *)&cdesc->cd;
cd_pars->u.s.content_desc_params_sz = RTE_ALIGN_CEIL(cd_size, 8) >> 3;
cipher_cd_ctrl->cipher_key_sz = total_key_size >> 3;
@@ -1722,7 +1753,6 @@ int qat_sym_cd_auth_set(struct qat_sym_session *cdesc,
struct icp_qat_hw_cipher_algo_blk *cipherconfig;
struct icp_qat_fw_la_bulk_req *req_tmpl = &cdesc->fw_req;
struct icp_qat_fw_comn_req_hdr_cd_pars *cd_pars = &req_tmpl->cd_pars;
- struct icp_qat_fw_comn_req_hdr *header = &req_tmpl->comn_hdr;
void *ptr = &req_tmpl->cd_ctrl;
struct icp_qat_fw_cipher_cd_ctrl_hdr *cipher_cd_ctrl = ptr;
struct icp_qat_fw_auth_cd_ctrl_hdr *hash_cd_ctrl = ptr;
@@ -1735,8 +1765,6 @@ int qat_sym_cd_auth_set(struct qat_sym_session *cdesc,
uint32_t *aad_len = NULL;
uint32_t wordIndex = 0;
uint32_t *pTempKey;
- enum qat_sym_proto_flag qat_proto_flag =
- QAT_CRYPTO_PROTO_FLAG_NONE;
if (cdesc->qat_cmd == ICP_QAT_FW_LA_CMD_AUTH) {
ICP_QAT_FW_COMN_CURR_ID_SET(hash_cd_ctrl,
@@ -1759,19 +1787,10 @@ int qat_sym_cd_auth_set(struct qat_sym_session *cdesc,
return -EFAULT;
}
- if (operation == RTE_CRYPTO_AUTH_OP_VERIFY) {
- ICP_QAT_FW_LA_RET_AUTH_SET(header->serv_specif_flags,
- ICP_QAT_FW_LA_NO_RET_AUTH_RES);
- ICP_QAT_FW_LA_CMP_AUTH_SET(header->serv_specif_flags,
- ICP_QAT_FW_LA_CMP_AUTH_RES);
+ if (operation == RTE_CRYPTO_AUTH_OP_VERIFY)
cdesc->auth_op = ICP_QAT_HW_AUTH_VERIFY;
- } else {
- ICP_QAT_FW_LA_RET_AUTH_SET(header->serv_specif_flags,
- ICP_QAT_FW_LA_RET_AUTH_RES);
- ICP_QAT_FW_LA_CMP_AUTH_SET(header->serv_specif_flags,
- ICP_QAT_FW_LA_NO_CMP_AUTH_RES);
+ else
cdesc->auth_op = ICP_QAT_HW_AUTH_GENERATE;
- }
/*
* Setup the inner hash config
@@ -1913,7 +1932,7 @@ int qat_sym_cd_auth_set(struct qat_sym_session *cdesc,
break;
case ICP_QAT_HW_AUTH_ALGO_GALOIS_128:
case ICP_QAT_HW_AUTH_ALGO_GALOIS_64:
- qat_proto_flag = QAT_CRYPTO_PROTO_FLAG_GCM;
+ cdesc->qat_proto_flag = QAT_CRYPTO_PROTO_FLAG_GCM;
state1_size = ICP_QAT_HW_GALOIS_128_STATE1_SZ;
if (qat_sym_do_precomputes(cdesc->qat_hash_alg, authkey,
authkeylen, cdesc->cd_cur_ptr + state1_size,
@@ -1936,7 +1955,7 @@ int qat_sym_cd_auth_set(struct qat_sym_session *cdesc,
cdesc->aad_len = aad_length;
break;
case ICP_QAT_HW_AUTH_ALGO_SNOW_3G_UIA2:
- qat_proto_flag = QAT_CRYPTO_PROTO_FLAG_SNOW3G;
+ cdesc->qat_proto_flag = QAT_CRYPTO_PROTO_FLAG_SNOW3G;
state1_size = qat_hash_get_state1_size(
ICP_QAT_HW_AUTH_ALGO_SNOW_3G_UIA2);
state2_size = ICP_QAT_HW_SNOW_3G_UIA2_STATE2_SZ;
@@ -1960,7 +1979,7 @@ int qat_sym_cd_auth_set(struct qat_sym_session *cdesc,
hash->auth_config.config =
ICP_QAT_HW_AUTH_CONFIG_BUILD(ICP_QAT_HW_AUTH_MODE0,
cdesc->qat_hash_alg, digestsize);
- qat_proto_flag = QAT_CRYPTO_PROTO_FLAG_ZUC;
+ cdesc->qat_proto_flag = QAT_CRYPTO_PROTO_FLAG_ZUC;
state1_size = qat_hash_get_state1_size(
ICP_QAT_HW_AUTH_ALGO_ZUC_3G_128_EIA3);
state2_size = ICP_QAT_HW_ZUC_3G_EIA3_STATE2_SZ;
@@ -1988,7 +2007,7 @@ int qat_sym_cd_auth_set(struct qat_sym_session *cdesc,
state2_size = ICP_QAT_HW_NULL_STATE2_SZ;
break;
case ICP_QAT_HW_AUTH_ALGO_AES_CBC_MAC:
- qat_proto_flag = QAT_CRYPTO_PROTO_FLAG_CCM;
+ cdesc->qat_proto_flag = QAT_CRYPTO_PROTO_FLAG_CCM;
state1_size = qat_hash_get_state1_size(
ICP_QAT_HW_AUTH_ALGO_AES_CBC_MAC);
state2_size = ICP_QAT_HW_AES_CBC_MAC_KEY_SZ +
@@ -2036,10 +2055,6 @@ int qat_sym_cd_auth_set(struct qat_sym_session *cdesc,
return -EFAULT;
}
- /* Request template setup */
- qat_sym_session_init_common_hdr(cdesc, header, qat_proto_flag);
- header->service_cmd_id = cdesc->qat_cmd;
-
/* Auth CD config setup */
hash_cd_ctrl->hash_cfg_offset = hash_offset >> 3;
hash_cd_ctrl->hash_flags = ICP_QAT_FW_AUTH_HDR_FLAG_NO_NESTED;
@@ -2248,6 +2263,7 @@ qat_sec_session_set_docsis_parameters(struct rte_cryptodev *dev,
ret = qat_sym_session_configure_cipher(dev, xform, session);
if (ret < 0)
return ret;
+ qat_sym_session_finalize(session);
return 0;
}
diff --git a/drivers/crypto/qat/qat_sym_session.h b/drivers/crypto/qat/qat_sym_session.h
index e003a34f7f..1568e09200 100644
--- a/drivers/crypto/qat/qat_sym_session.h
+++ b/drivers/crypto/qat/qat_sym_session.h
@@ -48,6 +48,13 @@
#define QAT_AES_CMAC_CONST_RB 0x87
+#define QAT_CRYPTO_SLICE_SPC 1
+#define QAT_CRYPTO_SLICE_UCS 2
+#define QAT_CRYPTO_SLICE_WCP 4
+
+#define QAT_SESSION_IS_SLICE_SET(flags, flag) \
+ (!!((flags) & (flag)))
+
enum qat_sym_proto_flag {
QAT_CRYPTO_PROTO_FLAG_NONE = 0,
QAT_CRYPTO_PROTO_FLAG_CCM = 1,
@@ -93,6 +100,11 @@ struct qat_sym_session {
uint8_t is_single_pass;
uint8_t is_single_pass_gmac;
uint8_t is_ucs;
+ uint8_t is_iv12B;
+ uint8_t is_gmac;
+ uint8_t is_auth;
+ uint32_t slice_types;
+ enum qat_sym_proto_flag qat_proto_flag;
};
int
--
2.25.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [dpdk-dev] [PATCH 08/15] crypto/qat: add aes gcm in ucs spc mode
2021-05-31 14:10 [dpdk-dev] [PATCH 00/15] Add support for fourth generation of Intel QuickAssist Technology devices Arek Kusztal
` (6 preceding siblings ...)
2021-05-31 14:10 ` [dpdk-dev] [PATCH 07/15] crypto/qat: rework init common header function Arek Kusztal
@ 2021-05-31 14:10 ` Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 09/15] crypto/qat: add chacha-poly " Arek Kusztal
` (6 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Arek Kusztal @ 2021-05-31 14:10 UTC (permalink / raw)
To: dev; +Cc: gakhil, fiona.trahe, roy.fan.zhang, Arek Kusztal
This commit adds AES-GCM algorithm that works
in UCS (Unified crypto slice) SPC(Single-Pass) mode.
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
---
drivers/crypto/qat/qat_sym.c | 32 ++++++++++++++++++++--------
drivers/crypto/qat/qat_sym_session.c | 9 ++++----
2 files changed, 27 insertions(+), 14 deletions(-)
diff --git a/drivers/crypto/qat/qat_sym.c b/drivers/crypto/qat/qat_sym.c
index eef4a886c5..00fc4d6b1a 100644
--- a/drivers/crypto/qat/qat_sym.c
+++ b/drivers/crypto/qat/qat_sym.c
@@ -217,6 +217,7 @@ qat_sym_build_request(void *in_op, uint8_t *out_msg,
int ret = 0;
struct qat_sym_session *ctx = NULL;
struct icp_qat_fw_la_cipher_req_params *cipher_param;
+ struct icp_qat_fw_la_cipher_20_req_params *cipher_param20;
struct icp_qat_fw_la_auth_req_params *auth_param;
register struct icp_qat_fw_la_bulk_req *qat_req;
uint8_t do_auth = 0, do_cipher = 0, do_aead = 0;
@@ -286,6 +287,7 @@ qat_sym_build_request(void *in_op, uint8_t *out_msg,
rte_mov128((uint8_t *)qat_req, (const uint8_t *)&(ctx->fw_req));
qat_req->comn_mid.opaque_data = (uint64_t)(uintptr_t)op;
cipher_param = (void *)&qat_req->serv_specif_rqpars;
+ cipher_param20 = (void *)&qat_req->serv_specif_rqpars;
auth_param = (void *)((uint8_t *)cipher_param +
ICP_QAT_FW_HASH_REQUEST_PARAMETERS_OFFSET);
@@ -563,13 +565,17 @@ qat_sym_build_request(void *in_op, uint8_t *out_msg,
cipher_param->cipher_length = 0;
}
- if (do_auth || do_aead) {
- auth_param->auth_off = (uint32_t)rte_pktmbuf_iova_offset(
+ if (!ctx->is_single_pass) {
+ /* Do not let to owerwrite spc_aad len */
+ if (do_auth || do_aead) {
+ auth_param->auth_off =
+ (uint32_t)rte_pktmbuf_iova_offset(
op->sym->m_src, auth_ofs) - src_buf_start;
- auth_param->auth_len = auth_len;
- } else {
- auth_param->auth_off = 0;
- auth_param->auth_len = 0;
+ auth_param->auth_len = auth_len;
+ } else {
+ auth_param->auth_off = 0;
+ auth_param->auth_len = 0;
+ }
}
qat_req->comn_mid.dst_length =
@@ -675,10 +681,18 @@ qat_sym_build_request(void *in_op, uint8_t *out_msg,
}
if (ctx->is_single_pass) {
- /* Handle Single-Pass GCM */
- cipher_param->spc_aad_addr = op->sym->aead.aad.phys_addr;
- cipher_param->spc_auth_res_addr =
+ if (ctx->is_ucs) {
+ /* GEN 4 */
+ cipher_param20->spc_aad_addr =
+ op->sym->aead.aad.phys_addr;
+ cipher_param20->spc_auth_res_addr =
op->sym->aead.digest.phys_addr;
+ } else {
+ cipher_param->spc_aad_addr =
+ op->sym->aead.aad.phys_addr;
+ cipher_param->spc_auth_res_addr =
+ op->sym->aead.digest.phys_addr;
+ }
} else if (ctx->is_single_pass_gmac &&
op->sym->auth.data.length <= QAT_AES_GMAC_SPC_MAX_SIZE) {
/* Handle Single-Pass AES-GMAC */
diff --git a/drivers/crypto/qat/qat_sym_session.c b/drivers/crypto/qat/qat_sym_session.c
index 4b52ffd459..7d66ca5172 100644
--- a/drivers/crypto/qat/qat_sym_session.c
+++ b/drivers/crypto/qat/qat_sym_session.c
@@ -897,16 +897,15 @@ qat_sym_session_configure_aead(struct rte_cryptodev *dev,
if (qat_dev_gen == QAT_GEN4)
session->is_ucs = 1;
-
if (session->cipher_iv.length == 0) {
session->cipher_iv.length = AES_GCM_J0_LEN;
break;
}
session->is_iv12B = 1;
- if (qat_dev_gen == QAT_GEN3) {
- qat_sym_session_handle_single_pass(session,
- aead_xform);
- }
+ if (qat_dev_gen < QAT_GEN3)
+ break;
+ qat_sym_session_handle_single_pass(session,
+ aead_xform);
break;
case RTE_CRYPTO_AEAD_AES_CCM:
if (qat_sym_validate_aes_key(aead_xform->key.length,
--
2.25.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [dpdk-dev] [PATCH 09/15] crypto/qat: add chacha-poly in ucs spc mode
2021-05-31 14:10 [dpdk-dev] [PATCH 00/15] Add support for fourth generation of Intel QuickAssist Technology devices Arek Kusztal
` (7 preceding siblings ...)
2021-05-31 14:10 ` [dpdk-dev] [PATCH 08/15] crypto/qat: add aes gcm in ucs spc mode Arek Kusztal
@ 2021-05-31 14:10 ` Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 10/15] crypto/qat: add gmac in legacy mode on fourth generation Arek Kusztal
` (5 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Arek Kusztal @ 2021-05-31 14:10 UTC (permalink / raw)
To: dev; +Cc: gakhil, fiona.trahe, roy.fan.zhang, Arek Kusztal
This commit adds Chacha20-Poly1305 aglorithm that works
in UCS (Unified crypto slice) SPC(Single-Pass) mode.
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
---
drivers/crypto/qat/qat_sym_capabilities.h | 32 ++++++++++++++++++++++-
drivers/crypto/qat/qat_sym_session.c | 2 ++
2 files changed, 33 insertions(+), 1 deletion(-)
diff --git a/drivers/crypto/qat/qat_sym_capabilities.h b/drivers/crypto/qat/qat_sym_capabilities.h
index fc8e667687..5c6e723466 100644
--- a/drivers/crypto/qat/qat_sym_capabilities.h
+++ b/drivers/crypto/qat/qat_sym_capabilities.h
@@ -1144,7 +1144,37 @@
}, \
}, } \
}, } \
- } \
+ }, \
+ { /* Chacha20-Poly1305 */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_AEAD, \
+ {.aead = { \
+ .algo = RTE_CRYPTO_AEAD_CHACHA20_POLY1305, \
+ .block_size = 64, \
+ .key_size = { \
+ .min = 32, \
+ .max = 32, \
+ .increment = 0 \
+ }, \
+ .digest_size = { \
+ .min = 16, \
+ .max = 16, \
+ .increment = 0 \
+ }, \
+ .aad_size = { \
+ .min = 0, \
+ .max = 240, \
+ .increment = 1 \
+ }, \
+ .iv_size = { \
+ .min = 12, \
+ .max = 12, \
+ .increment = 0 \
+ }, \
+ }, } \
+ }, } \
+ }
diff --git a/drivers/crypto/qat/qat_sym_session.c b/drivers/crypto/qat/qat_sym_session.c
index 7d66ca5172..221b950aeb 100644
--- a/drivers/crypto/qat/qat_sym_session.c
+++ b/drivers/crypto/qat/qat_sym_session.c
@@ -921,6 +921,8 @@ qat_sym_session_configure_aead(struct rte_cryptodev *dev,
case RTE_CRYPTO_AEAD_CHACHA20_POLY1305:
if (aead_xform->key.length != ICP_QAT_HW_CHACHAPOLY_KEY_SZ)
return -EINVAL;
+ if (qat_dev_gen == QAT_GEN4)
+ session->is_ucs = 1;
session->qat_cipher_alg =
ICP_QAT_HW_CIPHER_ALGO_CHACHA20_POLY1305;
qat_sym_session_handle_single_pass(session,
--
2.25.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [dpdk-dev] [PATCH 10/15] crypto/qat: add gmac in legacy mode on fourth generation
2021-05-31 14:10 [dpdk-dev] [PATCH 00/15] Add support for fourth generation of Intel QuickAssist Technology devices Arek Kusztal
` (8 preceding siblings ...)
2021-05-31 14:10 ` [dpdk-dev] [PATCH 09/15] crypto/qat: add chacha-poly " Arek Kusztal
@ 2021-05-31 14:10 ` Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 11/15] common/qat: add pf2vf communication in qat Arek Kusztal
` (4 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Arek Kusztal @ 2021-05-31 14:10 UTC (permalink / raw)
To: dev; +Cc: gakhil, fiona.trahe, roy.fan.zhang, Arek Kusztal
Add AES-GMAC algorithm in legacy mode to generation 4 devices.
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
---
drivers/crypto/qat/qat_sym_capabilities.h | 27 ++++++++++++++++++++++-
drivers/crypto/qat/qat_sym_session.c | 9 +++++++-
drivers/crypto/qat/qat_sym_session.h | 2 ++
3 files changed, 36 insertions(+), 2 deletions(-)
diff --git a/drivers/crypto/qat/qat_sym_capabilities.h b/drivers/crypto/qat/qat_sym_capabilities.h
index 5c6e723466..cfb176ca94 100644
--- a/drivers/crypto/qat/qat_sym_capabilities.h
+++ b/drivers/crypto/qat/qat_sym_capabilities.h
@@ -1174,7 +1174,32 @@
}, \
}, } \
}, } \
- }
+ }, \
+ { /* AES GMAC (AUTH) */ \
+ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \
+ {.sym = { \
+ .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, \
+ {.auth = { \
+ .algo = RTE_CRYPTO_AUTH_AES_GMAC, \
+ .block_size = 16, \
+ .key_size = { \
+ .min = 16, \
+ .max = 32, \
+ .increment = 8 \
+ }, \
+ .digest_size = { \
+ .min = 8, \
+ .max = 16, \
+ .increment = 4 \
+ }, \
+ .iv_size = { \
+ .min = 0, \
+ .max = 12, \
+ .increment = 12 \
+ } \
+ }, } \
+ }, } \
+ } \
diff --git a/drivers/crypto/qat/qat_sym_session.c b/drivers/crypto/qat/qat_sym_session.c
index 221b950aeb..c04b04da00 100644
--- a/drivers/crypto/qat/qat_sym_session.c
+++ b/drivers/crypto/qat/qat_sym_session.c
@@ -709,6 +709,8 @@ qat_sym_session_configure_auth(struct rte_cryptodev *dev,
struct qat_sym_dev_private *internals = dev->data->dev_private;
const uint8_t *key_data = auth_xform->key.data;
uint8_t key_length = auth_xform->key.length;
+ enum qat_device_gen qat_dev_gen =
+ internals->qat_dev->qat_dev_gen;
session->aes_cmac = 0;
session->auth_key_length = auth_xform->key.length;
@@ -716,6 +718,7 @@ qat_sym_session_configure_auth(struct rte_cryptodev *dev,
session->auth_iv.length = auth_xform->iv.length;
session->auth_mode = ICP_QAT_HW_AUTH_MODE1;
session->is_auth = 1;
+ session->digest_length = auth_xform->digest_length;
switch (auth_xform->algo) {
case RTE_CRYPTO_AUTH_SHA1:
@@ -772,6 +775,10 @@ qat_sym_session_configure_auth(struct rte_cryptodev *dev,
session->auth_iv.length = AES_GCM_J0_LEN;
else
session->is_iv12B = 1;
+ if (qat_dev_gen == QAT_GEN4) {
+ session->is_cnt_zero = 1;
+ session->is_ucs = 1;
+ }
break;
case RTE_CRYPTO_AUTH_SNOW3G_UIA2:
session->qat_hash_alg = ICP_QAT_HW_AUTH_ALGO_SNOW_3G_UIA2;
@@ -857,7 +864,6 @@ qat_sym_session_configure_auth(struct rte_cryptodev *dev,
return -EINVAL;
}
- session->digest_length = auth_xform->digest_length;
return 0;
}
@@ -1810,6 +1816,7 @@ int qat_sym_cd_auth_set(struct qat_sym_session *cdesc,
|| cdesc->qat_hash_alg == ICP_QAT_HW_AUTH_ALGO_AES_XCBC_MAC
|| cdesc->qat_hash_alg == ICP_QAT_HW_AUTH_ALGO_AES_CBC_MAC
|| cdesc->qat_hash_alg == ICP_QAT_HW_AUTH_ALGO_NULL
+ || cdesc->is_cnt_zero
)
hash->auth_counter.counter = 0;
else {
diff --git a/drivers/crypto/qat/qat_sym_session.h b/drivers/crypto/qat/qat_sym_session.h
index 1568e09200..33b236e49b 100644
--- a/drivers/crypto/qat/qat_sym_session.h
+++ b/drivers/crypto/qat/qat_sym_session.h
@@ -103,6 +103,8 @@ struct qat_sym_session {
uint8_t is_iv12B;
uint8_t is_gmac;
uint8_t is_auth;
+ uint8_t is_cnt_zero;
+ /* Some generations need different setup of counter */
uint32_t slice_types;
enum qat_sym_proto_flag qat_proto_flag;
};
--
2.25.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [dpdk-dev] [PATCH 11/15] common/qat: add pf2vf communication in qat
2021-05-31 14:10 [dpdk-dev] [PATCH 00/15] Add support for fourth generation of Intel QuickAssist Technology devices Arek Kusztal
` (9 preceding siblings ...)
2021-05-31 14:10 ` [dpdk-dev] [PATCH 10/15] crypto/qat: add gmac in legacy mode on fourth generation Arek Kusztal
@ 2021-05-31 14:10 ` Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 12/15] common/qat: reset ring pairs before setting pmd Arek Kusztal
` (3 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Arek Kusztal @ 2021-05-31 14:10 UTC (permalink / raw)
To: dev; +Cc: gakhil, fiona.trahe, roy.fan.zhang, Arek Kusztal
Add communication between physical device and virtual function
in Intel QucikAssist Technology PMD.
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
---
drivers/common/qat/meson.build | 1 +
drivers/common/qat/qat_adf/adf_pf2vf_msg.h | 154 +++++++++++++++++++++
drivers/common/qat/qat_device.c | 22 ++-
drivers/common/qat/qat_device.h | 12 ++
drivers/common/qat/qat_pf2vf.c | 80 +++++++++++
drivers/common/qat/qat_pf2vf.h | 19 +++
6 files changed, 287 insertions(+), 1 deletion(-)
create mode 100644 drivers/common/qat/qat_adf/adf_pf2vf_msg.h
create mode 100644 drivers/common/qat/qat_pf2vf.c
create mode 100644 drivers/common/qat/qat_pf2vf.h
diff --git a/drivers/common/qat/meson.build b/drivers/common/qat/meson.build
index 479a46f9f0..11ed37c910 100644
--- a/drivers/common/qat/meson.build
+++ b/drivers/common/qat/meson.build
@@ -49,6 +49,7 @@ sources += files(
'qat_qp.c',
'qat_device.c',
'qat_logs.c',
+ 'qat_pf2vf.c'
)
includes += include_directories(
'qat_adf',
diff --git a/drivers/common/qat/qat_adf/adf_pf2vf_msg.h b/drivers/common/qat/qat_adf/adf_pf2vf_msg.h
new file mode 100644
index 0000000000..4029b1c14a
--- /dev/null
+++ b/drivers/common/qat/qat_adf/adf_pf2vf_msg.h
@@ -0,0 +1,154 @@
+/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+ * Copyright(c) 2021 Intel Corporation
+ */
+#ifndef ADF_PF2VF_MSG_H_
+#define ADF_PF2VF_MSG_H_
+
+/* VF/PF compatibility version. */
+/* ADF_PFVF_COMPATIBILITY_EXT_CAP: Support for extended capabilities */
+#define ADF_PFVF_COMPATIBILITY_CAPABILITIES 2
+/* ADF_PFVF_COMPATIBILITY_FAST_ACK: In-use pattern cleared by receiver */
+#define ADF_PFVF_COMPATIBILITY_FAST_ACK 3
+#define ADF_PFVF_COMPATIBILITY_RING_TO_SVC_MAP 4
+#define ADF_PFVF_COMPATIBILITY_VERSION 4 /* PF<->VF compat */
+
+#define ADF_PFVF_INT 1
+#define ADF_PFVF_MSGORIGIN_SYSTEM 2
+#define ADF_PFVF_1X_MSGTYPE_SHIFT 2
+#define ADF_PFVF_1X_MSGTYPE_MASK 0xF
+#define ADF_PFVF_1X_MSGDATA_SHIFT 6
+#define ADF_PFVF_1X_MSGDATA_MASK 0x3FF
+#define ADF_PFVF_2X_MSGTYPE_SHIFT 2
+#define ADF_PFVF_2X_MSGTYPE_MASK 0x3F
+#define ADF_PFVF_2X_MSGDATA_SHIFT 8
+#define ADF_PFVF_2X_MSGDATA_MASK 0xFFFFFF
+
+#define ADF_PFVF_IN_USE 0x6AC2
+#define ADF_PFVF_IN_USE_MASK 0xFFFE
+#define ADF_PFVF_VF_MSG_SHIFT 16
+
+/* PF->VF messages */
+#define ADF_PF2VF_MSGTYPE_RESTARTING 0x01
+#define ADF_PF2VF_MSGTYPE_VERSION_RESP 0x02
+#define ADF_PF2VF_MSGTYPE_BLOCK_RESP 0x03
+#define ADF_PF2VF_MSGTYPE_FATAL_ERROR 0x04
+/* Do not use messages which start from 0x10 to 1.x as 1.x only use
+ * 4 bits as message types. Hence they are only applicable to 2.0
+ */
+#define ADF_PF2VF_MSGTYPE_RP_RESET_RESP 0x10
+
+/* PF->VF Version Response - ADF_PF2VF_MSGTYPE_VERSION_RESP */
+#define ADF_PF2VF_VERSION_RESP_VERS_MASK 0xFF
+#define ADF_PF2VF_VERSION_RESP_VERS_SHIFT 0
+#define ADF_PF2VF_VERSION_RESP_RESULT_MASK 0x03
+#define ADF_PF2VF_VERSION_RESP_RESULT_SHIFT 8
+#define ADF_PF2VF_MINORVERSION_SHIFT 0
+#define ADF_PF2VF_MAJORVERSION_SHIFT 4
+#define ADF_PF2VF_VF_COMPATIBLE 1
+#define ADF_PF2VF_VF_INCOMPATIBLE 2
+#define ADF_PF2VF_VF_COMPAT_UNKNOWN 3
+
+/* PF->VF Block Response Type - ADF_PF2VF_MSGTYPE_BLOCK_RESP */
+#define ADF_PF2VF_BLOCK_RESP_TYPE_DATA 0x0
+#define ADF_PF2VF_BLOCK_RESP_TYPE_CRC 0x1
+#define ADF_PF2VF_BLOCK_RESP_TYPE_ERROR 0x2
+#define ADF_PF2VF_BLOCK_RESP_TYPE_MASK 0x03
+#define ADF_PF2VF_BLOCK_RESP_TYPE_SHIFT 0
+#define ADF_PF2VF_BLOCK_RESP_DATA_MASK 0xFF
+#define ADF_PF2VF_BLOCK_RESP_DATA_SHIFT 2
+
+/*
+ * PF->VF Block Error Code - Returned in data field when the
+ * response type indicates an error
+ */
+#define ADF_PF2VF_INVALID_BLOCK_TYPE 0x0
+#define ADF_PF2VF_INVALID_BYTE_NUM_REQ 0x1
+#define ADF_PF2VF_PAYLOAD_TRUNCATED 0x2
+#define ADF_PF2VF_UNSPECIFIED_ERROR 0x3
+
+/* VF->PF messages */
+#define ADF_VF2PF_MSGTYPE_INIT 0x3
+#define ADF_VF2PF_MSGTYPE_SHUTDOWN 0x4
+#define ADF_VF2PF_MSGTYPE_VERSION_REQ 0x5
+#define ADF_VF2PF_MSGTYPE_COMPAT_VER_REQ 0x6
+#define ADF_VF2PF_MSGTYPE_GET_LARGE_BLOCK_REQ 0x7
+#define ADF_VF2PF_MSGTYPE_GET_MEDIUM_BLOCK_REQ 0x8
+#define ADF_VF2PF_MSGTYPE_GET_SMALL_BLOCK_REQ 0x9
+/* Do not use messages which start from 0x10 to 1.x as 1.x only use
+ * 4 bits as message types. Hence they are only applicable to 2.0
+ */
+#define ADF_VF2PF_MSGTYPE_RP_RESET 0x10
+
+/* VF->PF Block Request Type - ADF_VF2PF_MSGTYPE_GET_xxx_BLOCK_REQ */
+#define ADF_VF2PF_MIN_SMALL_MESSAGE_TYPE 0
+#define ADF_VF2PF_MAX_SMALL_MESSAGE_TYPE \
+ (ADF_VF2PF_MIN_SMALL_MESSAGE_TYPE + 15)
+#define ADF_VF2PF_MIN_MEDIUM_MESSAGE_TYPE \
+ (ADF_VF2PF_MAX_SMALL_MESSAGE_TYPE + 1)
+#define ADF_VF2PF_MAX_MEDIUM_MESSAGE_TYPE \
+ (ADF_VF2PF_MIN_MEDIUM_MESSAGE_TYPE + 7)
+#define ADF_VF2PF_MIN_LARGE_MESSAGE_TYPE \
+ (ADF_VF2PF_MAX_MEDIUM_MESSAGE_TYPE + 1)
+#define ADF_VF2PF_MAX_LARGE_MESSAGE_TYPE \
+ (ADF_VF2PF_MIN_LARGE_MESSAGE_TYPE + 3)
+#define ADF_VF2PF_SMALL_PAYLOAD_SIZE 30
+#define ADF_VF2PF_MEDIUM_PAYLOAD_SIZE 62
+#define ADF_VF2PF_LARGE_PAYLOAD_SIZE 126
+
+#define ADF_VF2PF_BLOCK_REQ_TYPE_SHIFT 0
+#define ADF_VF2PF_LARGE_BLOCK_REQ_TYPE_MASK 0x3
+#define ADF_VF2PF_MEDIUM_BLOCK_REQ_TYPE_MASK 0x7
+#define ADF_VF2PF_SMALL_BLOCK_REQ_TYPE_MASK 0xF
+
+#define ADF_VF2PF_LARGE_BLOCK_BYTE_NUM_SHIFT 2
+#define ADF_VF2PF_LARGE_BLOCK_BYTE_NUM_MASK 0x7F
+#define ADF_VF2PF_MEDIUM_BLOCK_BYTE_NUM_SHIFT 3
+#define ADF_VF2PF_MEDIUM_BLOCK_BYTE_NUM_MASK 0x3F
+#define ADF_VF2PF_SMALL_BLOCK_BYTE_NUM_SHIFT 4
+#define ADF_VF2PF_SMALL_BLOCK_BYTE_NUM_MASK 0x1F
+#define ADF_VF2PF_BLOCK_REQ_CRC_SHIFT 9
+
+/* PF-VF block message header bytes */
+#define ADF_VF2PF_BLOCK_VERSION_BYTE 0
+#define ADF_VF2PF_BLOCK_LEN_BYTE 1
+#define ADF_VF2PF_BLOCK_DATA 2
+
+/* Block message types
+ * 0..15 - 32 byte message
+ * 16..23 - 64 byte message
+ * 24..27 - 128 byte message
+ * 2 - Get Capability Request message
+ */
+#define ADF_VF2PF_BLOCK_MSG_CAP_SUMMARY 0x2
+#define ADF_VF2PF_BLOCK_MSG_GET_RING_TO_SVC_REQ 0x3
+
+/* VF->PF Compatible Version Request - ADF_VF2PF_MSGTYPE_VERSION_REQ */
+#define ADF_VF2PF_COMPAT_VER_SHIFT 0
+#define ADF_VF2PF_COMPAT_VER_MASK 0xFF
+
+/* How long to wait for far side to acknowledge receipt */
+#define ADF_IOV_MSG_ACK_DELAY_US 5
+#define ADF_IOV_MSG_ACK_MAX_RETRY (100 * 1000 / ADF_IOV_MSG_ACK_DELAY_US)
+/* If CSR is busy, how long to delay before retrying */
+#define ADF_IOV_MSG_RETRY_DELAY 5
+#define ADF_IOV_MSG_MAX_RETRIES 3
+/* How long to wait for a response from the other side */
+#define ADF_IOV_MSG_RESP_TIMEOUT 100
+/* How often to retry when there is no response */
+#define ADF_IOV_MSG_RESP_RETRIES 5
+
+#define ADF_IOV_RATELIMIT_INTERVAL 8
+#define ADF_IOV_RATELIMIT_BURST 130
+/* PF VF message byte shift */
+#define ADF_PFVF_DATA_SHIFT 8
+#define ADF_PFVF_DATA_MASK 0xFF
+
+/* CRC Calculation */
+#define ADF_CRC8_INIT_VALUE 0xFF
+
+/* Per device register offsets */
+/* GEN 4 */
+#define ADF_4XXXIOV_PF2VM_OFFSET 0x1008
+#define ADF_4XXXIOV_VM2PF_OFFSET 0x100C
+
+#endif /* ADF_IOV_MSG_H */
diff --git a/drivers/common/qat/qat_device.c b/drivers/common/qat/qat_device.c
index 932d7110f7..5ee441171e 100644
--- a/drivers/common/qat/qat_device.c
+++ b/drivers/common/qat/qat_device.c
@@ -10,6 +10,17 @@
#include "adf_transport_access_macros.h"
#include "qat_sym_pmd.h"
#include "qat_comp_pmd.h"
+#include "adf_pf2vf_msg.h"
+
+/* pv2vf data Gen 4*/
+struct qat_pf2vf_dev qat_pf2vf_gen4 = {
+ .pf2vf_offset = ADF_4XXXIOV_PF2VM_OFFSET,
+ .vf2pf_offset = ADF_4XXXIOV_VM2PF_OFFSET,
+ .pf2vf_type_shift = ADF_PFVF_2X_MSGTYPE_SHIFT,
+ .pf2vf_type_mask = ADF_PFVF_2X_MSGTYPE_MASK,
+ .pf2vf_data_shift = ADF_PFVF_2X_MSGDATA_SHIFT,
+ .pf2vf_data_mask = ADF_PFVF_2X_MSGDATA_MASK,
+};
/* Hardware device information per generation */
__extension__
@@ -33,7 +44,8 @@ struct qat_gen_hw_data qat_gen_config[] = {
[QAT_GEN4] = {
.dev_gen = QAT_GEN4,
.qp_hw_data = NULL,
- .comp_num_im_bufs_required = QAT_NUM_INTERM_BUFS_GEN3
+ .comp_num_im_bufs_required = QAT_NUM_INTERM_BUFS_GEN3,
+ .pf2vf_dev = &qat_pf2vf_gen4
},
};
@@ -249,6 +261,14 @@ qat_pci_device_allocate(struct rte_pci_device *pci_dev,
return NULL;
}
+ if (qat_dev->qat_dev_gen == QAT_GEN4) {
+ qat_dev->misc_bar_io_addr = pci_dev->mem_resource[2].addr;
+ if (qat_dev->misc_bar_io_addr == NULL) {
+ QAT_LOG(ERR, "QAT cannot get access to VF misc bar");
+ return NULL;
+ }
+ }
+
if (devargs && devargs->drv_str)
qat_dev_parse_cmd(devargs->drv_str, qat_dev_cmd_param);
diff --git a/drivers/common/qat/qat_device.h b/drivers/common/qat/qat_device.h
index b987fd1309..a4bd5b4304 100644
--- a/drivers/common/qat/qat_device.h
+++ b/drivers/common/qat/qat_device.h
@@ -110,12 +110,24 @@ struct qat_pci_device {
[QAT_GEN4_QPS_PER_BUNDLE_NUM];
};
/**< Data of ring configuration on gen4 */
+ void *misc_bar_io_addr;
+ /**< Address of misc bar */
};
struct qat_gen_hw_data {
enum qat_device_gen dev_gen;
const struct qat_qp_hw_data (*qp_hw_data)[ADF_MAX_QPS_ON_ANY_SERVICE];
enum qat_comp_num_im_buffers comp_num_im_bufs_required;
+ struct qat_pf2vf_dev *pf2vf_dev;
+};
+
+struct qat_pf2vf_dev {
+ uint32_t pf2vf_offset;
+ uint32_t vf2pf_offset;
+ int pf2vf_type_shift;
+ uint32_t pf2vf_type_mask;
+ int pf2vf_data_shift;
+ uint32_t pf2vf_data_mask;
};
extern struct qat_gen_hw_data qat_gen_config[];
diff --git a/drivers/common/qat/qat_pf2vf.c b/drivers/common/qat/qat_pf2vf.c
new file mode 100644
index 0000000000..d3346975f1
--- /dev/null
+++ b/drivers/common/qat/qat_pf2vf.c
@@ -0,0 +1,80 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2021 Intel Corporation
+ */
+
+#include "qat_pf2vf.h"
+#include "adf_pf2vf_msg.h"
+
+#include <rte_cycles.h>
+
+int qat_pf2vf_exch_msg(struct qat_pci_device *qat_dev,
+ struct qat_pf2vf_msg pf2vf_msg,
+ int len, uint8_t *ret)
+{
+ int i = 0;
+ struct qat_pf2vf_dev *qat_pf2vf =
+ qat_gen_config[qat_dev->qat_dev_gen].pf2vf_dev;
+ void *pmisc_bar_addr = qat_dev->misc_bar_io_addr;
+ uint32_t msg = 0, count = 0, val = 0;
+ uint32_t vf_csr_off = qat_pf2vf->vf2pf_offset;
+ uint32_t pf_csr_off = qat_pf2vf->pf2vf_offset;
+ int type_shift = qat_pf2vf->pf2vf_type_shift;
+ uint32_t type_mask = qat_pf2vf->pf2vf_type_mask;
+ int blck_hdr_shift = qat_pf2vf->pf2vf_data_shift;
+ int data_shift = blck_hdr_shift;
+
+ switch (pf2vf_msg.msg_type) {
+ case ADF_VF2PF_MSGTYPE_GET_SMALL_BLOCK_REQ:
+ data_shift += ADF_VF2PF_SMALL_BLOCK_BYTE_NUM_SHIFT;
+ break;
+ case ADF_VF2PF_MSGTYPE_GET_MEDIUM_BLOCK_REQ:
+ data_shift += ADF_VF2PF_MEDIUM_BLOCK_BYTE_NUM_SHIFT;
+ break;
+ case ADF_VF2PF_MSGTYPE_GET_LARGE_BLOCK_REQ:
+ data_shift += ADF_VF2PF_LARGE_BLOCK_BYTE_NUM_SHIFT;
+ break;
+ }
+
+ if ((pf2vf_msg.msg_type & type_mask) != pf2vf_msg.msg_type) {
+ QAT_LOG(ERR, "PF2VF message type 0x%X out of range\n",
+ pf2vf_msg.msg_type);
+ return -EINVAL;
+ }
+
+ for (; i < len; i++) {
+ count = 0;
+ if (len == 1) {
+ msg = (pf2vf_msg.msg_type << type_shift) |
+ (pf2vf_msg.msg_data << (data_shift));
+ } else
+ msg = (pf2vf_msg.msg_type << type_shift) |
+ (i << (data_shift));
+ if (pf2vf_msg.block_hdr > 0)
+ msg |= pf2vf_msg.block_hdr << blck_hdr_shift;
+ msg |= ADF_PFVF_INT | ADF_PFVF_MSGORIGIN_SYSTEM;
+
+ ADF_CSR_WR(pmisc_bar_addr, vf_csr_off, msg);
+ int us = 0;
+ /*
+ * Wait for confirmation from remote that it received
+ * the message
+ */
+ do {
+ rte_delay_us_sleep(5);
+ us += 5;
+ val = ADF_CSR_RD(pmisc_bar_addr, vf_csr_off);
+ } while ((val & ADF_PFVF_INT) &&
+ (++count < ADF_IOV_MSG_ACK_MAX_RETRY));
+
+ if (val & ADF_PFVF_INT) {
+ QAT_LOG(ERR, "ACK not received from remote\n");
+ return -EIO;
+ }
+
+ uint32_t pf_val = ADF_CSR_RD(pmisc_bar_addr, pf_csr_off);
+
+ *(ret + i) = (uint8_t)(pf_val >> (pf2vf_msg.block_hdr > 0 ?
+ 10 : 8) & 0xff);
+ }
+ return 0;
+}
diff --git a/drivers/common/qat/qat_pf2vf.h b/drivers/common/qat/qat_pf2vf.h
new file mode 100644
index 0000000000..df59277347
--- /dev/null
+++ b/drivers/common/qat/qat_pf2vf.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2021 Intel Corporation
+ */
+
+#include "qat_device.h"
+
+#ifndef QAT_PF2VF_H_
+#define QAT_PF2VF_H_
+
+struct qat_pf2vf_msg {
+ uint32_t msg_data;
+ int block_hdr;
+ uint16_t msg_type;
+};
+
+int qat_pf2vf_exch_msg(struct qat_pci_device *qat_dev,
+ struct qat_pf2vf_msg pf2vf_msg, int len, uint8_t *ret);
+
+#endif
--
2.25.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [dpdk-dev] [PATCH 12/15] common/qat: reset ring pairs before setting pmd
2021-05-31 14:10 [dpdk-dev] [PATCH 00/15] Add support for fourth generation of Intel QuickAssist Technology devices Arek Kusztal
` (10 preceding siblings ...)
2021-05-31 14:10 ` [dpdk-dev] [PATCH 11/15] common/qat: add pf2vf communication in qat Arek Kusztal
@ 2021-05-31 14:10 ` Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 13/15] crypto/qat: update raw dp api Arek Kusztal
` (2 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Arek Kusztal @ 2021-05-31 14:10 UTC (permalink / raw)
To: dev; +Cc: gakhil, fiona.trahe, roy.fan.zhang, Arek Kusztal
This commit resets ring pairs of particular vf before
setting PMD on fourth generation devices.
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
---
drivers/common/qat/qat_device.c | 27 +++++++++++++++++++++++++++
1 file changed, 27 insertions(+)
diff --git a/drivers/common/qat/qat_device.c b/drivers/common/qat/qat_device.c
index 5ee441171e..d6bf0f52db 100644
--- a/drivers/common/qat/qat_device.c
+++ b/drivers/common/qat/qat_device.c
@@ -11,6 +11,7 @@
#include "qat_sym_pmd.h"
#include "qat_comp_pmd.h"
#include "adf_pf2vf_msg.h"
+#include "qat_pf2vf.h"
/* pv2vf data Gen 4*/
struct qat_pf2vf_dev qat_pf2vf_gen4 = {
@@ -125,6 +126,27 @@ qat_get_qat_dev_from_pci_dev(struct rte_pci_device *pci_dev)
return qat_pci_get_named_dev(name);
}
+static int qat_gen4_reset_ring_pair(struct qat_pci_device *qat_pci_dev)
+{
+ int ret = 0, i;
+ uint8_t data[4];
+ struct qat_pf2vf_msg pf2vf_msg;
+
+ pf2vf_msg.msg_type = ADF_VF2PF_MSGTYPE_RP_RESET;
+ pf2vf_msg.block_hdr = -1;
+ for (i = 0; i < QAT_GEN4_BUNDLE_NUM; i++) {
+ pf2vf_msg.msg_data = i;
+ ret = qat_pf2vf_exch_msg(qat_pci_dev, pf2vf_msg, 1, data);
+ if (ret) {
+ QAT_LOG(ERR, "QAT error when reset bundle no %d",
+ i);
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
static void qat_dev_parse_cmd(const char *str, struct qat_dev_cmd_param
*qat_dev_cmd_param)
{
@@ -371,6 +393,11 @@ static int qat_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
if (qat_pci_dev == NULL)
return -ENODEV;
+ if (qat_pci_dev->qat_dev_gen == QAT_GEN4) {
+ if (qat_gen4_reset_ring_pair(qat_pci_dev))
+ return -ENODEV;
+ }
+
sym_ret = qat_sym_dev_create(qat_pci_dev, qat_dev_cmd_param);
if (sym_ret == 0) {
num_pmds_created++;
--
2.25.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [dpdk-dev] [PATCH 13/15] crypto/qat: update raw dp api
2021-05-31 14:10 [dpdk-dev] [PATCH 00/15] Add support for fourth generation of Intel QuickAssist Technology devices Arek Kusztal
` (11 preceding siblings ...)
2021-05-31 14:10 ` [dpdk-dev] [PATCH 12/15] common/qat: reset ring pairs before setting pmd Arek Kusztal
@ 2021-05-31 14:10 ` Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 14/15] crypto/qat: enable RAW API on QAT GEN1-3 only Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 15/15] test/crypto: check if RAW API is supported Arek Kusztal
14 siblings, 0 replies; 16+ messages in thread
From: Arek Kusztal @ 2021-05-31 14:10 UTC (permalink / raw)
To: dev; +Cc: gakhil, fiona.trahe, roy.fan.zhang
From: Fan Zhang <roy.fan.zhang@intel.com>
This commit updates the QAT raw data-path API to support the
changes made to device and sessions. The QAT RAW data-path API
now works on Generation 1-3 devices.
Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com>
---
drivers/crypto/qat/qat_sym_hw_dp.c | 419 +++++++++++++++--------------
1 file changed, 216 insertions(+), 203 deletions(-)
diff --git a/drivers/crypto/qat/qat_sym_hw_dp.c b/drivers/crypto/qat/qat_sym_hw_dp.c
index 2f64de44a1..4305579b54 100644
--- a/drivers/crypto/qat/qat_sym_hw_dp.c
+++ b/drivers/crypto/qat/qat_sym_hw_dp.c
@@ -101,204 +101,6 @@ qat_sym_dp_fill_vec_status(int32_t *sta, int status, uint32_t n)
#define QAT_SYM_DP_GET_MAX_ENQ(q, c, n) \
RTE_MIN((q->max_inflights - q->enqueued + q->dequeued - c), n)
-static __rte_always_inline void
-enqueue_one_aead_job(struct qat_sym_session *ctx,
- struct icp_qat_fw_la_bulk_req *req,
- struct rte_crypto_va_iova_ptr *iv,
- struct rte_crypto_va_iova_ptr *digest,
- struct rte_crypto_va_iova_ptr *aad,
- union rte_crypto_sym_ofs ofs, uint32_t data_len)
-{
- struct icp_qat_fw_la_cipher_req_params *cipher_param =
- (void *)&req->serv_specif_rqpars;
- struct icp_qat_fw_la_auth_req_params *auth_param =
- (void *)((uint8_t *)&req->serv_specif_rqpars +
- ICP_QAT_FW_HASH_REQUEST_PARAMETERS_OFFSET);
- uint8_t *aad_data;
- uint8_t aad_ccm_real_len;
- uint8_t aad_len_field_sz;
- uint32_t msg_len_be;
- rte_iova_t aad_iova = 0;
- uint8_t q;
-
- switch (ctx->qat_hash_alg) {
- case ICP_QAT_HW_AUTH_ALGO_GALOIS_128:
- case ICP_QAT_HW_AUTH_ALGO_GALOIS_64:
- ICP_QAT_FW_LA_GCM_IV_LEN_FLAG_SET(
- req->comn_hdr.serv_specif_flags,
- ICP_QAT_FW_LA_GCM_IV_LEN_12_OCTETS);
- rte_memcpy(cipher_param->u.cipher_IV_array, iv->va,
- ctx->cipher_iv.length);
- aad_iova = aad->iova;
- break;
- case ICP_QAT_HW_AUTH_ALGO_AES_CBC_MAC:
- aad_data = aad->va;
- aad_iova = aad->iova;
- aad_ccm_real_len = 0;
- aad_len_field_sz = 0;
- msg_len_be = rte_bswap32((uint32_t)data_len -
- ofs.ofs.cipher.head);
-
- if (ctx->aad_len > ICP_QAT_HW_CCM_AAD_DATA_OFFSET) {
- aad_len_field_sz = ICP_QAT_HW_CCM_AAD_LEN_INFO;
- aad_ccm_real_len = ctx->aad_len -
- ICP_QAT_HW_CCM_AAD_B0_LEN -
- ICP_QAT_HW_CCM_AAD_LEN_INFO;
- } else {
- aad_data = iv->va;
- aad_iova = iv->iova;
- }
-
- q = ICP_QAT_HW_CCM_NQ_CONST - ctx->cipher_iv.length;
- aad_data[0] = ICP_QAT_HW_CCM_BUILD_B0_FLAGS(
- aad_len_field_sz, ctx->digest_length, q);
- if (q > ICP_QAT_HW_CCM_MSG_LEN_MAX_FIELD_SIZE) {
- memcpy(aad_data + ctx->cipher_iv.length +
- ICP_QAT_HW_CCM_NONCE_OFFSET + (q -
- ICP_QAT_HW_CCM_MSG_LEN_MAX_FIELD_SIZE),
- (uint8_t *)&msg_len_be,
- ICP_QAT_HW_CCM_MSG_LEN_MAX_FIELD_SIZE);
- } else {
- memcpy(aad_data + ctx->cipher_iv.length +
- ICP_QAT_HW_CCM_NONCE_OFFSET,
- (uint8_t *)&msg_len_be +
- (ICP_QAT_HW_CCM_MSG_LEN_MAX_FIELD_SIZE
- - q), q);
- }
-
- if (aad_len_field_sz > 0) {
- *(uint16_t *)&aad_data[ICP_QAT_HW_CCM_AAD_B0_LEN] =
- rte_bswap16(aad_ccm_real_len);
-
- if ((aad_ccm_real_len + aad_len_field_sz)
- % ICP_QAT_HW_CCM_AAD_B0_LEN) {
- uint8_t pad_len = 0;
- uint8_t pad_idx = 0;
-
- pad_len = ICP_QAT_HW_CCM_AAD_B0_LEN -
- ((aad_ccm_real_len +
- aad_len_field_sz) %
- ICP_QAT_HW_CCM_AAD_B0_LEN);
- pad_idx = ICP_QAT_HW_CCM_AAD_B0_LEN +
- aad_ccm_real_len +
- aad_len_field_sz;
- memset(&aad_data[pad_idx], 0, pad_len);
- }
- }
-
- rte_memcpy(((uint8_t *)cipher_param->u.cipher_IV_array)
- + ICP_QAT_HW_CCM_NONCE_OFFSET,
- (uint8_t *)iv->va +
- ICP_QAT_HW_CCM_NONCE_OFFSET, ctx->cipher_iv.length);
- *(uint8_t *)&cipher_param->u.cipher_IV_array[0] =
- q - ICP_QAT_HW_CCM_NONCE_OFFSET;
-
- rte_memcpy((uint8_t *)aad->va +
- ICP_QAT_HW_CCM_NONCE_OFFSET,
- (uint8_t *)iv->va + ICP_QAT_HW_CCM_NONCE_OFFSET,
- ctx->cipher_iv.length);
- break;
- default:
- break;
- }
-
- cipher_param->cipher_offset = ofs.ofs.cipher.head;
- cipher_param->cipher_length = data_len - ofs.ofs.cipher.head -
- ofs.ofs.cipher.tail;
- auth_param->auth_off = ofs.ofs.cipher.head;
- auth_param->auth_len = cipher_param->cipher_length;
- auth_param->auth_res_addr = digest->iova;
- auth_param->u1.aad_adr = aad_iova;
-
- if (ctx->is_single_pass) {
- cipher_param->spc_aad_addr = aad_iova;
- cipher_param->spc_auth_res_addr = digest->iova;
- }
-}
-
-static __rte_always_inline int
-qat_sym_dp_enqueue_single_aead(void *qp_data, uint8_t *drv_ctx,
- struct rte_crypto_vec *data, uint16_t n_data_vecs,
- union rte_crypto_sym_ofs ofs,
- struct rte_crypto_va_iova_ptr *iv,
- struct rte_crypto_va_iova_ptr *digest,
- struct rte_crypto_va_iova_ptr *aad,
- void *user_data)
-{
- struct qat_qp *qp = qp_data;
- struct qat_sym_dp_ctx *dp_ctx = (void *)drv_ctx;
- struct qat_queue *tx_queue = &qp->tx_q;
- struct qat_sym_session *ctx = dp_ctx->session;
- struct icp_qat_fw_la_bulk_req *req;
- int32_t data_len;
- uint32_t tail = dp_ctx->tail;
-
- req = (struct icp_qat_fw_la_bulk_req *)(
- (uint8_t *)tx_queue->base_addr + tail);
- tail = (tail + tx_queue->msg_size) & tx_queue->modulo_mask;
- rte_mov128((uint8_t *)req, (const uint8_t *)&(ctx->fw_req));
- rte_prefetch0((uint8_t *)tx_queue->base_addr + tail);
- data_len = qat_sym_dp_parse_data_vec(qp, req, data, n_data_vecs);
- if (unlikely(data_len < 0))
- return -1;
- req->comn_mid.opaque_data = (uint64_t)(uintptr_t)user_data;
-
- enqueue_one_aead_job(ctx, req, iv, digest, aad, ofs,
- (uint32_t)data_len);
-
- dp_ctx->tail = tail;
- dp_ctx->cached_enqueue++;
-
- return 0;
-}
-
-static __rte_always_inline uint32_t
-qat_sym_dp_enqueue_aead_jobs(void *qp_data, uint8_t *drv_ctx,
- struct rte_crypto_sym_vec *vec, union rte_crypto_sym_ofs ofs,
- void *user_data[], int *status)
-{
- struct qat_qp *qp = qp_data;
- struct qat_sym_dp_ctx *dp_ctx = (void *)drv_ctx;
- struct qat_queue *tx_queue = &qp->tx_q;
- struct qat_sym_session *ctx = dp_ctx->session;
- uint32_t i, n;
- uint32_t tail;
- struct icp_qat_fw_la_bulk_req *req;
- int32_t data_len;
-
- n = QAT_SYM_DP_GET_MAX_ENQ(qp, dp_ctx->cached_enqueue, vec->num);
- if (unlikely(n == 0)) {
- qat_sym_dp_fill_vec_status(vec->status, -1, vec->num);
- *status = 0;
- return 0;
- }
-
- tail = dp_ctx->tail;
-
- for (i = 0; i < n; i++) {
- req = (struct icp_qat_fw_la_bulk_req *)(
- (uint8_t *)tx_queue->base_addr + tail);
- rte_mov128((uint8_t *)req, (const uint8_t *)&(ctx->fw_req));
-
- data_len = qat_sym_dp_parse_data_vec(qp, req, vec->sgl[i].vec,
- vec->sgl[i].num);
- if (unlikely(data_len < 0))
- break;
- req->comn_mid.opaque_data = (uint64_t)(uintptr_t)user_data[i];
- enqueue_one_aead_job(ctx, req, &vec->iv[i], &vec->digest[i],
- &vec->aad[i], ofs, (uint32_t)data_len);
- tail = (tail + tx_queue->msg_size) & tx_queue->modulo_mask;
- }
-
- if (unlikely(i < n))
- qat_sym_dp_fill_vec_status(vec->status + i, -1, n - i);
-
- dp_ctx->tail = tail;
- dp_ctx->cached_enqueue += i;
- *status = 0;
- return i;
-}
-
static __rte_always_inline void
enqueue_one_cipher_job(struct qat_sym_session *ctx,
struct icp_qat_fw_la_bulk_req *req,
@@ -704,6 +506,207 @@ qat_sym_dp_enqueue_chain_jobs(void *qp_data, uint8_t *drv_ctx,
return i;
}
+static __rte_always_inline void
+enqueue_one_aead_job(struct qat_sym_session *ctx,
+ struct icp_qat_fw_la_bulk_req *req,
+ struct rte_crypto_va_iova_ptr *iv,
+ struct rte_crypto_va_iova_ptr *digest,
+ struct rte_crypto_va_iova_ptr *aad,
+ union rte_crypto_sym_ofs ofs, uint32_t data_len)
+{
+ struct icp_qat_fw_la_cipher_req_params *cipher_param =
+ (void *)&req->serv_specif_rqpars;
+ struct icp_qat_fw_la_auth_req_params *auth_param =
+ (void *)((uint8_t *)&req->serv_specif_rqpars +
+ ICP_QAT_FW_HASH_REQUEST_PARAMETERS_OFFSET);
+ uint8_t *aad_data;
+ uint8_t aad_ccm_real_len;
+ uint8_t aad_len_field_sz;
+ uint32_t msg_len_be;
+ rte_iova_t aad_iova = 0;
+ uint8_t q;
+
+ /* CPM 1.7 uses single pass to treat AEAD as cipher operation */
+ if (ctx->is_single_pass) {
+ enqueue_one_cipher_job(ctx, req, iv, ofs, data_len);
+ cipher_param->spc_aad_addr = aad->iova;
+ cipher_param->spc_auth_res_addr = digest->iova;
+ return;
+ }
+
+ switch (ctx->qat_hash_alg) {
+ case ICP_QAT_HW_AUTH_ALGO_GALOIS_128:
+ case ICP_QAT_HW_AUTH_ALGO_GALOIS_64:
+ ICP_QAT_FW_LA_GCM_IV_LEN_FLAG_SET(
+ req->comn_hdr.serv_specif_flags,
+ ICP_QAT_FW_LA_GCM_IV_LEN_12_OCTETS);
+ rte_memcpy(cipher_param->u.cipher_IV_array, iv->va,
+ ctx->cipher_iv.length);
+ aad_iova = aad->iova;
+ break;
+ case ICP_QAT_HW_AUTH_ALGO_AES_CBC_MAC:
+ aad_data = aad->va;
+ aad_iova = aad->iova;
+ aad_ccm_real_len = 0;
+ aad_len_field_sz = 0;
+ msg_len_be = rte_bswap32((uint32_t)data_len -
+ ofs.ofs.cipher.head);
+
+ if (ctx->aad_len > ICP_QAT_HW_CCM_AAD_DATA_OFFSET) {
+ aad_len_field_sz = ICP_QAT_HW_CCM_AAD_LEN_INFO;
+ aad_ccm_real_len = ctx->aad_len -
+ ICP_QAT_HW_CCM_AAD_B0_LEN -
+ ICP_QAT_HW_CCM_AAD_LEN_INFO;
+ } else {
+ aad_data = iv->va;
+ aad_iova = iv->iova;
+ }
+
+ q = ICP_QAT_HW_CCM_NQ_CONST - ctx->cipher_iv.length;
+ aad_data[0] = ICP_QAT_HW_CCM_BUILD_B0_FLAGS(
+ aad_len_field_sz, ctx->digest_length, q);
+ if (q > ICP_QAT_HW_CCM_MSG_LEN_MAX_FIELD_SIZE) {
+ memcpy(aad_data + ctx->cipher_iv.length +
+ ICP_QAT_HW_CCM_NONCE_OFFSET + (q -
+ ICP_QAT_HW_CCM_MSG_LEN_MAX_FIELD_SIZE),
+ (uint8_t *)&msg_len_be,
+ ICP_QAT_HW_CCM_MSG_LEN_MAX_FIELD_SIZE);
+ } else {
+ memcpy(aad_data + ctx->cipher_iv.length +
+ ICP_QAT_HW_CCM_NONCE_OFFSET,
+ (uint8_t *)&msg_len_be +
+ (ICP_QAT_HW_CCM_MSG_LEN_MAX_FIELD_SIZE
+ - q), q);
+ }
+
+ if (aad_len_field_sz > 0) {
+ *(uint16_t *)&aad_data[ICP_QAT_HW_CCM_AAD_B0_LEN] =
+ rte_bswap16(aad_ccm_real_len);
+
+ if ((aad_ccm_real_len + aad_len_field_sz)
+ % ICP_QAT_HW_CCM_AAD_B0_LEN) {
+ uint8_t pad_len = 0;
+ uint8_t pad_idx = 0;
+
+ pad_len = ICP_QAT_HW_CCM_AAD_B0_LEN -
+ ((aad_ccm_real_len +
+ aad_len_field_sz) %
+ ICP_QAT_HW_CCM_AAD_B0_LEN);
+ pad_idx = ICP_QAT_HW_CCM_AAD_B0_LEN +
+ aad_ccm_real_len +
+ aad_len_field_sz;
+ memset(&aad_data[pad_idx], 0, pad_len);
+ }
+ }
+
+ rte_memcpy(((uint8_t *)cipher_param->u.cipher_IV_array)
+ + ICP_QAT_HW_CCM_NONCE_OFFSET,
+ (uint8_t *)iv->va +
+ ICP_QAT_HW_CCM_NONCE_OFFSET, ctx->cipher_iv.length);
+ *(uint8_t *)&cipher_param->u.cipher_IV_array[0] =
+ q - ICP_QAT_HW_CCM_NONCE_OFFSET;
+
+ rte_memcpy((uint8_t *)aad->va +
+ ICP_QAT_HW_CCM_NONCE_OFFSET,
+ (uint8_t *)iv->va + ICP_QAT_HW_CCM_NONCE_OFFSET,
+ ctx->cipher_iv.length);
+ break;
+ default:
+ break;
+ }
+
+ cipher_param->cipher_offset = ofs.ofs.cipher.head;
+ cipher_param->cipher_length = data_len - ofs.ofs.cipher.head -
+ ofs.ofs.cipher.tail;
+ auth_param->auth_off = ofs.ofs.cipher.head;
+ auth_param->auth_len = cipher_param->cipher_length;
+ auth_param->auth_res_addr = digest->iova;
+ auth_param->u1.aad_adr = aad_iova;
+}
+
+static __rte_always_inline int
+qat_sym_dp_enqueue_single_aead(void *qp_data, uint8_t *drv_ctx,
+ struct rte_crypto_vec *data, uint16_t n_data_vecs,
+ union rte_crypto_sym_ofs ofs,
+ struct rte_crypto_va_iova_ptr *iv,
+ struct rte_crypto_va_iova_ptr *digest,
+ struct rte_crypto_va_iova_ptr *aad,
+ void *user_data)
+{
+ struct qat_qp *qp = qp_data;
+ struct qat_sym_dp_ctx *dp_ctx = (void *)drv_ctx;
+ struct qat_queue *tx_queue = &qp->tx_q;
+ struct qat_sym_session *ctx = dp_ctx->session;
+ struct icp_qat_fw_la_bulk_req *req;
+ int32_t data_len;
+ uint32_t tail = dp_ctx->tail;
+
+ req = (struct icp_qat_fw_la_bulk_req *)(
+ (uint8_t *)tx_queue->base_addr + tail);
+ tail = (tail + tx_queue->msg_size) & tx_queue->modulo_mask;
+ rte_mov128((uint8_t *)req, (const uint8_t *)&(ctx->fw_req));
+ rte_prefetch0((uint8_t *)tx_queue->base_addr + tail);
+ data_len = qat_sym_dp_parse_data_vec(qp, req, data, n_data_vecs);
+ if (unlikely(data_len < 0))
+ return -1;
+ req->comn_mid.opaque_data = (uint64_t)(uintptr_t)user_data;
+
+ enqueue_one_aead_job(ctx, req, iv, digest, aad, ofs,
+ (uint32_t)data_len);
+
+ dp_ctx->tail = tail;
+ dp_ctx->cached_enqueue++;
+
+ return 0;
+}
+
+static __rte_always_inline uint32_t
+qat_sym_dp_enqueue_aead_jobs(void *qp_data, uint8_t *drv_ctx,
+ struct rte_crypto_sym_vec *vec, union rte_crypto_sym_ofs ofs,
+ void *user_data[], int *status)
+{
+ struct qat_qp *qp = qp_data;
+ struct qat_sym_dp_ctx *dp_ctx = (void *)drv_ctx;
+ struct qat_queue *tx_queue = &qp->tx_q;
+ struct qat_sym_session *ctx = dp_ctx->session;
+ uint32_t i, n;
+ uint32_t tail;
+ struct icp_qat_fw_la_bulk_req *req;
+ int32_t data_len;
+
+ n = QAT_SYM_DP_GET_MAX_ENQ(qp, dp_ctx->cached_enqueue, vec->num);
+ if (unlikely(n == 0)) {
+ qat_sym_dp_fill_vec_status(vec->status, -1, vec->num);
+ *status = 0;
+ return 0;
+ }
+
+ tail = dp_ctx->tail;
+
+ for (i = 0; i < n; i++) {
+ req = (struct icp_qat_fw_la_bulk_req *)(
+ (uint8_t *)tx_queue->base_addr + tail);
+ rte_mov128((uint8_t *)req, (const uint8_t *)&(ctx->fw_req));
+
+ data_len = qat_sym_dp_parse_data_vec(qp, req, vec->sgl[i].vec,
+ vec->sgl[i].num);
+ if (unlikely(data_len < 0))
+ break;
+ req->comn_mid.opaque_data = (uint64_t)(uintptr_t)user_data[i];
+ enqueue_one_aead_job(ctx, req, &vec->iv[i], &vec->digest[i],
+ &vec->aad[i], ofs, (uint32_t)data_len);
+ tail = (tail + tx_queue->msg_size) & tx_queue->modulo_mask;
+ }
+
+ if (unlikely(i < n))
+ qat_sym_dp_fill_vec_status(vec->status + i, -1, n - i);
+
+ dp_ctx->tail = tail;
+ dp_ctx->cached_enqueue += i;
+ *status = 0;
+ return i;
+}
+
static __rte_always_inline uint32_t
qat_sym_dp_dequeue_burst(void *qp_data, uint8_t *drv_ctx,
rte_cryptodev_raw_get_dequeue_count_t get_dequeue_count,
@@ -937,8 +940,9 @@ qat_sym_configure_dp_ctx(struct rte_cryptodev *dev, uint16_t qp_id,
raw_dp_ctx->dequeue = qat_sym_dp_dequeue;
raw_dp_ctx->dequeue_done = qat_sym_dp_update_head;
- if (ctx->qat_cmd == ICP_QAT_FW_LA_CMD_HASH_CIPHER ||
- ctx->qat_cmd == ICP_QAT_FW_LA_CMD_CIPHER_HASH) {
+ if ((ctx->qat_cmd == ICP_QAT_FW_LA_CMD_HASH_CIPHER ||
+ ctx->qat_cmd == ICP_QAT_FW_LA_CMD_CIPHER_HASH) &&
+ !ctx->is_gmac) {
/* AES-GCM or AES-CCM */
if (ctx->qat_hash_alg == ICP_QAT_HW_AUTH_ALGO_GALOIS_128 ||
ctx->qat_hash_alg == ICP_QAT_HW_AUTH_ALGO_GALOIS_64 ||
@@ -954,12 +958,21 @@ qat_sym_configure_dp_ctx(struct rte_cryptodev *dev, uint16_t qp_id,
qat_sym_dp_enqueue_chain_jobs;
raw_dp_ctx->enqueue = qat_sym_dp_enqueue_single_chain;
}
- } else if (ctx->qat_cmd == ICP_QAT_FW_LA_CMD_AUTH) {
+ } else if (ctx->qat_cmd == ICP_QAT_FW_LA_CMD_AUTH || ctx->is_gmac) {
raw_dp_ctx->enqueue_burst = qat_sym_dp_enqueue_auth_jobs;
raw_dp_ctx->enqueue = qat_sym_dp_enqueue_single_auth;
} else if (ctx->qat_cmd == ICP_QAT_FW_LA_CMD_CIPHER) {
- raw_dp_ctx->enqueue_burst = qat_sym_dp_enqueue_cipher_jobs;
- raw_dp_ctx->enqueue = qat_sym_dp_enqueue_single_cipher;
+ if (ctx->qat_mode == ICP_QAT_HW_CIPHER_AEAD_MODE ||
+ ctx->qat_cipher_alg ==
+ ICP_QAT_HW_CIPHER_ALGO_CHACHA20_POLY1305) {
+ raw_dp_ctx->enqueue_burst =
+ qat_sym_dp_enqueue_aead_jobs;
+ raw_dp_ctx->enqueue = qat_sym_dp_enqueue_single_aead;
+ } else {
+ raw_dp_ctx->enqueue_burst =
+ qat_sym_dp_enqueue_cipher_jobs;
+ raw_dp_ctx->enqueue = qat_sym_dp_enqueue_single_cipher;
+ }
} else
return -1;
--
2.25.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [dpdk-dev] [PATCH 14/15] crypto/qat: enable RAW API on QAT GEN1-3 only
2021-05-31 14:10 [dpdk-dev] [PATCH 00/15] Add support for fourth generation of Intel QuickAssist Technology devices Arek Kusztal
` (12 preceding siblings ...)
2021-05-31 14:10 ` [dpdk-dev] [PATCH 13/15] crypto/qat: update raw dp api Arek Kusztal
@ 2021-05-31 14:10 ` Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 15/15] test/crypto: check if RAW API is supported Arek Kusztal
14 siblings, 0 replies; 16+ messages in thread
From: Arek Kusztal @ 2021-05-31 14:10 UTC (permalink / raw)
To: dev; +Cc: gakhil, fiona.trahe, roy.fan.zhang, Adam Dybkowski
From: Adam Dybkowski <adamx.dybkowski@intel.com>
This patch enables RAW API in feature flags on QAT generations
1 to 3 only. Disables it for later generations.
Signed-off-by: Adam Dybkowski <adamx.dybkowski@intel.com>
---
drivers/crypto/qat/qat_sym_pmd.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/drivers/crypto/qat/qat_sym_pmd.c b/drivers/crypto/qat/qat_sym_pmd.c
index 0097ee210f..1c7b142511 100644
--- a/drivers/crypto/qat/qat_sym_pmd.c
+++ b/drivers/crypto/qat/qat_sym_pmd.c
@@ -409,8 +409,10 @@ qat_sym_dev_create(struct qat_pci_device *qat_pci_dev,
RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT |
RTE_CRYPTODEV_FF_OOP_LB_IN_SGL_OUT |
RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT |
- RTE_CRYPTODEV_FF_DIGEST_ENCRYPTED |
- RTE_CRYPTODEV_FF_SYM_RAW_DP;
+ RTE_CRYPTODEV_FF_DIGEST_ENCRYPTED;
+
+ if (qat_pci_dev->qat_dev_gen < QAT_GEN4)
+ cryptodev->feature_flags |= RTE_CRYPTODEV_FF_SYM_RAW_DP;
if (rte_eal_process_type() != RTE_PROC_PRIMARY)
return 0;
--
2.25.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [dpdk-dev] [PATCH 15/15] test/crypto: check if RAW API is supported
2021-05-31 14:10 [dpdk-dev] [PATCH 00/15] Add support for fourth generation of Intel QuickAssist Technology devices Arek Kusztal
` (13 preceding siblings ...)
2021-05-31 14:10 ` [dpdk-dev] [PATCH 14/15] crypto/qat: enable RAW API on QAT GEN1-3 only Arek Kusztal
@ 2021-05-31 14:10 ` Arek Kusztal
14 siblings, 0 replies; 16+ messages in thread
From: Arek Kusztal @ 2021-05-31 14:10 UTC (permalink / raw)
To: dev; +Cc: gakhil, fiona.trahe, roy.fan.zhang, Adam Dybkowski
From: Adam Dybkowski <adamx.dybkowski@intel.com>
This patch adds checking if RAW API is supported at the start
of the test command "cryptodev_qat_raw_api_autotest".
Signed-off-by: Adam Dybkowski <adamx.dybkowski@intel.com>
---
app/test/test_cryptodev.c | 34 +++++++++++++++++++++++++++++++++-
1 file changed, 33 insertions(+), 1 deletion(-)
diff --git a/app/test/test_cryptodev.c b/app/test/test_cryptodev.c
index 39db52b17a..64b6cc0db7 100644
--- a/app/test/test_cryptodev.c
+++ b/app/test/test_cryptodev.c
@@ -14769,7 +14769,39 @@ test_cryptodev_bcmfs(void)
static int
test_cryptodev_qat_raw_api(void /*argv __rte_unused, int argc __rte_unused*/)
{
- int ret;
+ static const char *pmd_name = RTE_STR(CRYPTODEV_NAME_QAT_SYM_PMD);
+ struct rte_cryptodev_info dev_info;
+ uint8_t i, nb_devs, found = 0;
+ int driver_id, ret;
+
+ driver_id = rte_cryptodev_driver_id_get(pmd_name);
+ if (driver_id == -1) {
+ RTE_LOG(WARNING, USER1, "%s PMD must be loaded.\n", pmd_name);
+ return TEST_SKIPPED;
+ }
+
+ nb_devs = rte_cryptodev_count();
+ if (nb_devs < 1) {
+ RTE_LOG(WARNING, USER1, "No crypto devices found?\n");
+ return TEST_SKIPPED;
+ }
+
+ for (i = 0; i < nb_devs; i++) {
+ rte_cryptodev_info_get(i, &dev_info);
+ if (dev_info.driver_id == driver_id) {
+ if (!(dev_info.feature_flags &
+ RTE_CRYPTODEV_FF_SYM_RAW_DP)) {
+ RTE_LOG(INFO, USER1, "RAW API not supported\n");
+ return TEST_SKIPPED;
+ }
+ found = 1;
+ break;
+ }
+ }
+ if (!found) {
+ RTE_LOG(INFO, USER1, "RAW API not supported\n");
+ return TEST_SKIPPED;
+ }
global_api_test_type = CRYPTODEV_RAW_API_TEST;
ret = run_cryptodev_testsuite(RTE_STR(CRYPTODEV_NAME_QAT_SYM_PMD));
--
2.25.1
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2021-05-31 14:12 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-31 14:10 [dpdk-dev] [PATCH 00/15] Add support for fourth generation of Intel QuickAssist Technology devices Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 01/15] common/qat: rework qp per service function Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 02/15] crypto/qat: add fourth generation qat devices support Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 03/15] crypto/qat: enable gen4 legacy algorithms Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 04/15] crypto/qat: add fourth generation ucs slice type, add ctr mode Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 05/15] crypto/qat: rename content descriptor functions Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 06/15] crypto/qat: add legacy gcm and ccm Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 07/15] crypto/qat: rework init common header function Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 08/15] crypto/qat: add aes gcm in ucs spc mode Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 09/15] crypto/qat: add chacha-poly " Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 10/15] crypto/qat: add gmac in legacy mode on fourth generation Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 11/15] common/qat: add pf2vf communication in qat Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 12/15] common/qat: reset ring pairs before setting pmd Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 13/15] crypto/qat: update raw dp api Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 14/15] crypto/qat: enable RAW API on QAT GEN1-3 only Arek Kusztal
2021-05-31 14:10 ` [dpdk-dev] [PATCH 15/15] test/crypto: check if RAW API is supported Arek Kusztal
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).