RE: [EXTERNAL] [PATCH v15 01/21] zsda: add zsdadev driver documents

[Akhil Goyal <gakhil@marvell.com>]

> Introduce ZTE Storage Data Accelerator(ZSDA) drivers
> which can help accelerate storage data process.
> 
> The official product documenttation web page is:
> https://enterprise.zte.com.cn/products.html?id=101
> 
> It is recommended to update MAINTAINERS in the
> first patch in the new PMD guidelines.
> https://patches.dpdk.org/project/dpdk/patch/20241006184254.53499-1-nandinipersad361@gmail.com/

This patch should also have driver skeleton as mentioned in above patch.
In your case, you have a crypto and a compress driver together in a single patchset which is dependent on common code.
You can split this into crypto and compress documentation and add it in first patch of where these drivers are initialized.
First you can have the common code and then compress driver followed by crypto driver.
Update MAINTAINERS wherever a new folder/file is added.

So in your case the patches would be like
1. config: add zsda device number
2. common/zsda: configure pci_driver for zsda -- 
	- This is your skeleton where probe and remove are there.
               - Update MAINTAINERS for common/zsda and .ini .rst files added in this patch
               - Here you can add common documentation
               - also add .ini and .rst files without adding specific features.
3. common/zsda: add logging macros
4. common/zsda: add common functions
5. common/zsda: add functions to operate hardware queue
6. common/zsda: configure queue operation
7. common/zsda: configure the enqueue datapath
8. common/zsda: configure the dequeue datapath
9. compress/zsda: add compression algo match ---- update MAINTAINERS for compress/zsda and update .rst for general features.
10. compress/zsda: add treatment of compressdev cqe and chksum  -> rephrase your title.
11. compress/zsda: add configuration of compressdev_ops  -> rephrase
12. compress/zsda: add functions of compressdev create and destroy
	- here your feature is complete for compression. You can update .ini file for the features supported.
	- also update release notes for compressdev.
	- update .rst file for testing/debugging.
13. crypto/zsda: introduce key expansion code ---- update MAINTAINERS for crypto/zsda
14. crypto/zsda: configure session  -- update .rst file for general features.
15. crypto/zsda: add sym algo match
16. crypto/zsda: configure wqe of cryptodev
17. crypto/zsda: add treatment of cryptodev cqe
18. crypto/zsda: add capabilities of sym
	- update features in .ini as per the capabilities.
19. crypto/zsda: add configuration of cryptodev_ops
20. crypto/zsda: add functions of cryptodev create and destroy
	- update release notes for crypto driver.
	- update .rst for testing/debugging

The .ini file features should be added in subsequent patches where the feature is added.
First patch is to make a skeleton of the driver and need not have complete documentation
Of something which is not yet there in that patch.

> 
> This patch may contain compilation errors.
> Because the patch is depended on the other patch.
> 
> Depends-on: series-cryptodev: add SM4-XTS algo and test cases

This patch is merged, so this can be removed.
Now, every patch should compile without any warnings.

> 
> Signed-off-by: Hanxiao Li <li.hanxiao@zte.com.cn>
> ---
>  MAINTAINERS                               |   7 +
>  doc/guides/compressdevs/features/zsda.ini |  15 ++
>  doc/guides/compressdevs/index.rst         |   1 +
>  doc/guides/compressdevs/zsda.rst          |  45 ++++
>  doc/guides/cryptodevs/features/zsda.ini   |  51 +++++
>  doc/guides/cryptodevs/index.rst           |   1 +
>  doc/guides/cryptodevs/zsda.rst            | 260 ++++++++++++++++++++++
>  doc/guides/rel_notes/release_24_11.rst    |   8 +
>  8 files changed, 388 insertions(+)
>  create mode 100644 doc/guides/compressdevs/features/zsda.ini
>  create mode 100644 doc/guides/compressdevs/zsda.rst
>  create mode 100644 doc/guides/cryptodevs/features/zsda.ini
>  create mode 100644 doc/guides/cryptodevs/zsda.rst
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index c5a703b5c0..ff4fc977a0 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1221,6 +1221,9 @@ F: drivers/crypto/virtio/
>  F: doc/guides/cryptodevs/virtio.rst
>  F: doc/guides/cryptodevs/features/virtio.ini
> 
> +ZTE Storage Data Accelerator(ZSDA)
> +M: Hanxiao Li <li.hanxiao@zte.com.cn>
> +F: drivers/crypto/zsda/


.ini and .rst file missing here.


> 
>  Compression Drivers
>  -------------------
> @@ -1268,6 +1271,10 @@ F: drivers/compress/zlib/
>  F: doc/guides/compressdevs/zlib.rst
>  F: doc/guides/compressdevs/features/zlib.ini
> 
> +ZTE Storage Data Accelerator(ZSDA)
> +M: Hanxiao Li <li.hanxiao@zte.com.cn>
> +F: drivers/common/zsda/
> +F: drivers/compress/zsda/
Doc files missing

> 
>  DMAdev Drivers
>  --------------
> diff --git a/doc/guides/compressdevs/features/zsda.ini
> b/doc/guides/compressdevs/features/zsda.ini
> new file mode 100644
> index 0000000000..3b087ea7f9
> --- /dev/null
> +++ b/doc/guides/compressdevs/features/zsda.ini
> @@ -0,0 +1,15 @@
> +;
> +; Refer to default.ini for the full list of available PMD features.
> +;
> +; Supported features of 'ZSDA' compression driver.
> +;
> +[Features]
> +HW Accelerated         = Y
> +OOP SGL In SGL Out     = Y
> +OOP SGL In LB  Out     = Y
> +OOP LB  In SGL Out     = Y
> +Deflate                = Y
> +Adler32                = Y
> +Crc32                  = Y
> +Fixed                  = Y
> +Dynamic                = Y

You should add all these features in .ini file in the same patch as the feature is added.

In this patch just add the file with just HW accelerated =Y.
Rest all can be added where the feature is supported.


> diff --git a/doc/guides/compressdevs/index.rst
> b/doc/guides/compressdevs/index.rst
> index 87ed4f72a4..bab226ffbc 100644
> --- a/doc/guides/compressdevs/index.rst
> +++ b/doc/guides/compressdevs/index.rst
> @@ -17,3 +17,4 @@ Compression Device Drivers
>      qat_comp
>      uadk
>      zlib
> +    zsda
> diff --git a/doc/guides/compressdevs/zsda.rst
> b/doc/guides/compressdevs/zsda.rst
> new file mode 100644
> index 0000000000..0140032a61
> --- /dev/null
> +++ b/doc/guides/compressdevs/zsda.rst
> @@ -0,0 +1,45 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright(c) 2024 ZTE Corporation.
> +
> +ZTE Storage Data Accelerator (ZSDA) Poll Mode Driver
> +=======================================================
> +
> +The ZSDA compression PMD provides poll mode compression & decompression
> driver
> +support for the following hardware accelerator devices:
> +
> +* ``ZTE Processing accelerators 1cf2``
> +
> +
> +Features
> +--------
> +
> +ZSDA compression PMD has support for:
> +
> +Compression/Decompression algorithm:
> +
> +    * DEFLATE - using Fixed and Dynamic Huffman encoding
> +
> +Checksum generation:
> +
> +    * CRC32, Adler32
> +
> +Huffman code type:
> +
> +* FIXED
> +* DYNAMIC
> +
> +
> +Limitations
> +-----------
> +
> +* Compressdev level 0, no compression, is not supported.
> +* No BSD support as BSD ZSDA kernel driver not available.
> +* Stateful is not supported.
> +
> +
> +Installation
> +------------
> +
> +The ZSDA compression PMD is built by default with a standard DPDK build.
> +
> +It depends on a ZSDA kernel driver, see :ref:`building_zsda`.
> diff --git a/doc/guides/cryptodevs/features/zsda.ini
> b/doc/guides/cryptodevs/features/zsda.ini
> new file mode 100644
> index 0000000000..4d97976815
> --- /dev/null
> +++ b/doc/guides/cryptodevs/features/zsda.ini
> @@ -0,0 +1,51 @@
> +;
> +; Supported features of the 'zsda' crypto driver.
> +;
> +; Refer to default.ini for the full list of available PMD features.
> +;
> +[Features]
> +Symmetric crypto       = Y
> +HW Accelerated         = Y
> +Protocol offload       = Y
> +In Place SGL           = Y
> +OOP SGL In SGL Out     = Y
> +OOP SGL In LB  Out     = Y
> +OOP LB  In SGL Out     = Y
> +OOP LB  In LB  Out     = Y
> +Digest encrypted       = Y
> +Sym raw data path API  = Y
> +
> +;
> +; Supported crypto algorithms of the 'zsda' crypto driver.
> +;
> +[Cipher]
> +AES XTS (128)  = Y
> +AES XTS (256)  = Y
> +SM4 XTS        = Y
> +;
> +; Supported authentication algorithms of the 'zsda' crypto driver.
> +;
> +[Auth]
> +SHA1         = Y
> +SHA224       = Y
> +SHA256       = Y
> +SHA384       = Y
> +SHA512       = Y
> +SM3          = Y

Same comment for crypto


> +
> +;
> +; Supported AEAD algorithms of the 'zsda' crypto driver.
> +;
> +[AEAD]
> +
> +
> +;
> +; Supported Asymmetric algorithms of the 'zsda' crypto driver.
> +;
> +[Asymmetric]
> +
> +;
> +; Supported Operating systems of the 'zsda' crypto driver.
> +;
> +[OS]
> +Linux = Y
> diff --git a/doc/guides/cryptodevs/index.rst b/doc/guides/cryptodevs/index.rst
> index 1e57a9fe86..be2620f185 100644
> --- a/doc/guides/cryptodevs/index.rst
> +++ b/doc/guides/cryptodevs/index.rst
> @@ -34,3 +34,4 @@ Crypto Device Drivers
>      uadk
>      virtio
>      zuc
> +    zsda
> diff --git a/doc/guides/cryptodevs/zsda.rst b/doc/guides/cryptodevs/zsda.rst
> new file mode 100644
> index 0000000000..31ba60dd0e
> --- /dev/null
> +++ b/doc/guides/cryptodevs/zsda.rst
> @@ -0,0 +1,260 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright(c) 2024 ZTE Corporation.
> +
> +ZTE Storage Data Accelerator (ZSDA) Poll Mode Driver
> +====================================================
> +
> +ZSDA documentation consists of three parts:
> +
> +* Details of the symmetric crypto services below.
> +* Details of the :doc:`compression service <../compressdevs/zsda>`
> +  in the compressdev drivers section.
> +* Details of building the common ZSDA infrastructure and the PMDs to support
> the
> +  above services. See :ref:`building_zsda` below.
> +
> +
> +Symmetric Crypto Service on ZSDA
> +--------------------------------
> +
> +The ZSDA symmetric crypto PMD provides poll mode crypto driver
> +support for the following hardware accelerator devices:
> +
> +* ``ZTE Processing accelerators 1cf2``
> +
> +Features
> +~~~~~~~~
> +
> +The ZSDA SYM PMD has support for:
> +
> +Cipher algorithms:
> +
> +* ``RTE_CRYPTO_CIPHER_AES_XTS``
> +* ``RTE_CRYPTO_CIPHER_SM4_XTS``
> +
> +Hash algorithms:
> +
> +* ``RTE_CRYPTO_AUTH_SHA1``
> +* ``RTE_CRYPTO_AUTH_SHA224``
> +* ``RTE_CRYPTO_AUTH_SHA256``
> +* ``RTE_CRYPTO_AUTH_SHA384``
> +* ``RTE_CRYPTO_AUTH_SHA512``
> +* ``RTE_CRYPTO_AUTH_SM3``
> +
> +Limitations
> +~~~~~~~~~~~
> +
> +* Only supports the session-oriented API implementation (session-less APIs are
> not supported).
> +* No BSD support as BSD ZSDA kernel driver not available.
> +
> +* Queue-pairs are thread-safe on Intel CPUs but Queues are not (that is, within
> a single
> +  queue-pair all enqueues to the TX queue must be done from one thread and all
> dequeues
> +  from the RX queue must be done from one thread, but enqueues and dequeues
> may be done
> +  in different threads.)
> +
> +
> +.. _building_zsda:
> +
> +Building PMDs on ZSDA
> +---------------------
> +
> +A ZSDA device can host multiple acceleration services:
> +
> +* symmetric cryptography
> +* data compression
> +
> +These services are provided to DPDK applications via PMDs which register to
> +implement the corresponding cryptodev and compressdev APIs. The PMDs use
> +common ZSDA driver code which manages the ZSDA PCI device.
> +
> +
> +Configuring and Building the DPDK ZSDA PMDs
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +
> +Further information on configuring, building and installing DPDK is described
> +:doc:`here <../linux_gsg/build_dpdk>`.
> +
> +.. _building_zsda_config:
> +
> +Build Configuration
> +~~~~~~~~~~~~~~~~~~~
> +These is the build configuration options affecting ZSDA, and its default values:
> +
> +.. code-block:: console
> +
> +    RTE_PMD_ZSDA_MAX_PCI_DEVICES=256
> +
> +ZSDA SYM PMD has an external dependency on libcrypto, so is not built by
> default.
> +
> +Ubuntu
> +
> +.. code-block:: console
> +
> +   apt install libssl-dev
> +
> +RHEL
> +
> +.. code-block:: console
> +
> +   dnf install openssl-devel
> +
> +The ZSDA compressdev PMD has no external dependencies, so is built by
> default.
> +
> +
> +Device and driver naming
> +~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +* The zsda cryptodev symmetric crypto driver name is "crypto_zsda".
> +* The zsda compressdev compress driver name is "compress_zsda".
> +
> +The "rte_cryptodev_devices_get()" returns the devices exposed by either of
> these drivers.
> +
> +* Each zsda sym crypto device has a unique name, in format
> +  "<pci bdf>", e.g. "0000:cc:00.3_zsda".
> +  This name can be passed to "rte_cryptodev_get_dev_id()" to get the device_id.
> +
> +.. Note::
> +
> +    The cryptodev driver name is passed to the dpdk-test-crypto-perf tool in the
> "-devtype" parameter.
> +
> +    The zsda crypto device name is in the format of the worker parameter passed
> to the crypto scheduler.
> +
> +* The zsda compressdev driver name is "compress_zsda".
> +  The rte_compressdev_devices_get() returns the devices exposed by this driver.
> +
> +* Each zsda compression device has a unique name, in format
> +  <pci bdf>, e.g. "0000:cc:00.3_zsda".
> +  This name can be passed to rte_compressdev_get_dev_id() to get the
> device_id.
> +
> +
> +Enable VFs
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Instructions for installation are below, but first an explanation of the
> +relationships between the PF/VF devices and the PMDs visible to
> +DPDK applications.
> +
> +Each ZSDA PF device exposes a number of VF devices. Each VF device can
> +enable one symmetric cryptodev PMD and/or one compressdev PMD.
> +
> +These ZSDA PMDs share the same underlying device and pci-mgmt code, but are
> +enumerated independently on their respective APIs and appear as independent
> +devices to applications.
> +.. Note::
> +
> +   Each VF can only be used by one DPDK process. It is not possible to share
> +   the same VF across multiple processes, even if these processes are using
> +   different acceleration services.
> +   Conversely one DPDK process can use one or more ZSDA VFs and can expose
> both
> +   cryptodev and compressdev instances on each of those VFs.
> +
> +
> +The examples below are based on the 1cf2 device, if you have a different device
> +use the corresponding values in the above table.
> +
> +In BIOS ensure that SRIOV is enabled and either:
> +
> +* Disable VT-d or
> +* Enable VT-d and set ``"intel_iommu=on iommu=pt"`` in the grub file.
> +
> +you need to expose the Virtual Functions (VFs) using the sysfs file system.
> +
> +First find the BDFs (Bus-Device-Function) of the physical functions (PFs) of
> +your device, e.g.::
> +
> +    lspci -d:8050
> +
> +You should see output similar to::
> +
> +
> +    cc:00.4 Processing accelerators: Device 1cf2:8050 (rev 01)
> +    ce:00.3 Processing accelerators: Device 1cf2:8050 (rev 01)
> +    d0:00.3 Processing accelerators: Device 1cf2:8050 (rev 01)
> +    d2:00.3 Processing accelerators: Device 1cf2:8050 (rev 01)
> +
> +Enable the VFs for each PF by echoing the number of VFs per PF to the pci
> driver::
> +
> +     echo 31 > /sys/bus/pci/device/0000:cc:00.4/sriov_numvfs
> +     echo 31 > /sys/bus/pci/device/0000:ce:00.3/sriov_numvfs
> +     echo 31 > /sys/bus/pci/device/0000:d0:00.3/sriov_numvfs
> +     echo 31 > /sys/bus/pci/device/0000:d2:00.3/sriov_numvfs
> +
> +Check that the VFs are available for use. For example ``lspci -d:8051`` should
> +list 124 VF devices available.
> +
> +To complete the installation follow the instructions in
> +`Binding the available VFs to the vfio-pci driver`_.
> +
> +.. Note::
> +
> +   If you see the following warning in ``/var/log/messages`` it can be ignored:
> +   ``IOMMU should be enabled for SR-IOV to work correctly``.
> +
> +Binding the available VFs to the vfio-pci driver
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Note:
> +
> +* Please note that due to security issues, the usage of older DPDK igb_uio
> +  driver is not recommended. This document shows how to use the more secure
> +  vfio-pci driver.
> +
> +Unbind the VFs from the stock driver so they can be bound to the vfio-pci driver.
> +
> +Bind to the vfio-pci driver
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +Load the vfio-pci driver, bind the VF PCI Device id to it using the
> +``dpdk-devbind.py`` script then use the ``--status`` option
> +to confirm the VF devices are now in use by vfio-pci kernel driver,
> +e.g. for the 1cf2 device::
> +
> +    cd to the top-level DPDK directory
> +    modprobe vfio-pci
> +    usertools/dpdk-devbind.py -b vfio-pci 0000:cc:01.4
> +    usertools/dpdk-devbind.py --status
> +
> +Use ``modprobe vfio-pci disable_denylist=1`` from kernel 5.9 onwards.
> +See note in the section `Binding the available VFs to the vfio-pci driver`_
> +above.
> +
> +Testing
> +~~~~~~~
> +
> +ZSDA SYM crypto PMD can be tested by running the test application::
> +
> +    cd ./<build_dir>/app/test
> +    ./dpdk-test -l1 -n1 -a <your zsda bdf>
> +    RTE>>cryptodev_zsda_autotest
> +
> +ZSDA compression PMD can be tested by running the test application::
> +
> +    cd ./<build_dir>/app/test
> +    ./dpdk-test -l1 -n1 -a <your zsda bdf>
> +    RTE>>compressdev_autotest

This should be added in a later patch where it can actually work.
At this patch zsda driver does not exist.

> +
> +
> +Debugging
> +~~~~~~~~~
> +
> +There are 2 sets of trace available via the dynamic logging feature:
> +
> +* pmd.zsda.dp exposes trace on the data-path.
> +* pmd.zsda.general exposes all other trace.
> +
> +pmd.zsda exposes both sets of traces.
> +They can be enabled using the log-level option (where 8=maximum log level) on
> +the process cmdline, e.g. using any of the following::
> +
> +    --log-level="pmd.zsda.general,8"
> +    --log-level="pmd.zsda.dp,8"
> +
> +.. Note::
> +
> +    The global RTE_LOG_DP_LEVEL overrides data-path trace so must be set to
> +    RTE_LOG_DEBUG to see all the trace. This variable is in config/rte_config.h
> +    for meson build.
> +    Also the dynamic global log level overrides both sets of trace, so e.g. no
> +    ZSDA trace would display in this case::
> +
> +    --log-level="pmd.zsda.general,8" --log-level="pmd.zsda,8"
> diff --git a/doc/guides/rel_notes/release_24_11.rst
> b/doc/guides/rel_notes/release_24_11.rst
> index 0ff70d9057..5ef8d82677 100644
> --- a/doc/guides/rel_notes/release_24_11.rst
> +++ b/doc/guides/rel_notes/release_24_11.rst
> @@ -24,6 +24,14 @@ DPDK Release 24.11
>  New Features
>  ------------
> 
> +* **Added a compressdev driver for the ZTE Storage Data Accelerator device.**
> +  Added the new ``zsda`` compressdev driver for the ZTE ZSDA device.
> +  See the :doc:`../compressdevs/zsda` compressdev guide for more details on
> this new driver.
> +
> +* **Added a cryptodev driver for the ZTE Storage Data Accelerator device.**
> +  Added the new ``zsda`` cryptodev driver for the ZTE ZSDA device.
> +  See the :doc:`../cryptodevs/zsda` cryptodev guide for more details on this new
> driver.
> +

Positioning of release notes is not correct. Please check other ones in the file.

>  .. This section should contain new features added in this release.
>     Sample format:
> 
> --
> 2.27.0