[dpdk-dev] [PATCH v3 16/16] docs/qat: refactor docs adding compression guide

[Fiona Trahe <fiona.trahe@intel.com>]

Extend QAT guide to cover crypto and compression and common
information, particularly about kernel driver dependency.
Update release note.
Update compression feature list for qat.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
---
 config/common_base                       |   2 +-
 doc/guides/compressdevs/features/qat.ini |  22 ++++
 doc/guides/compressdevs/index.rst        |   1 +
 doc/guides/compressdevs/qat_comp.rst     |  49 +++++++++
 doc/guides/cryptodevs/qat.rst            | 183 +++++++++++++++++++++----------
 doc/guides/rel_notes/release_18_08.rst   |   5 +
 6 files changed, 203 insertions(+), 59 deletions(-)
 create mode 100644 doc/guides/compressdevs/features/qat.ini
 create mode 100644 doc/guides/compressdevs/qat_comp.rst

diff --git a/config/common_base b/config/common_base
index 93c8857..402231d 100644
--- a/config/common_base
+++ b/config/common_base
@@ -482,7 +482,7 @@ CONFIG_RTE_LIBRTE_DPAA_MAX_CRYPTODEV=4
 CONFIG_RTE_DPAA_SEC_PMD_MAX_NB_SESSIONS=2048
 
 #
-# Compile PMD for QuickAssist based devices
+# Compile PMD for QuickAssist based devices - see docs for details
 #
 CONFIG_RTE_LIBRTE_PMD_QAT=y
 CONFIG_RTE_LIBRTE_PMD_QAT_SYM=n
diff --git a/doc/guides/compressdevs/features/qat.ini b/doc/guides/compressdevs/features/qat.ini
new file mode 100644
index 0000000..68f36c8
--- /dev/null
+++ b/doc/guides/compressdevs/features/qat.ini
@@ -0,0 +1,22 @@
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+; Supported features of 'QAT' compression driver.
+;
+[Features]
+HW Accelerated = Y
+CPU SSE        =
+CPU AVX        =
+CPU AVX2       =
+CPU AVX512     =
+CPU NEON       =
+Stateful       =
+By-Pass        =
+Chained mbufs  =
+Deflate        = Y
+LZS            =
+Adler32        = Y
+Crc32          = Y
+Adler32&Crc32  = Y
+Fixed          = Y
+Dynamic        =
diff --git a/doc/guides/compressdevs/index.rst b/doc/guides/compressdevs/index.rst
index bc59ce8..4228768 100644
--- a/doc/guides/compressdevs/index.rst
+++ b/doc/guides/compressdevs/index.rst
@@ -11,3 +11,4 @@ Compression Device Drivers
 
     overview
     isal
+    qat_comp
diff --git a/doc/guides/compressdevs/qat_comp.rst b/doc/guides/compressdevs/qat_comp.rst
new file mode 100644
index 0000000..167f816
--- /dev/null
+++ b/doc/guides/compressdevs/qat_comp.rst
@@ -0,0 +1,49 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2018 Intel Corporation.
+
+Intel(R) QuickAssist (QAT) Compression Poll Mode Driver
+=======================================================
+
+The QAT compression PMD provides poll mode compression & decompression driver
+support for the following hardware accelerator devices:
+
+* ``Intel QuickAssist Technology C62x``
+* ``Intel QuickAssist Technology C3xxx``
+
+
+Features
+--------
+
+QAT compression PMD has support for:
+
+Compression/Decompression algorithm:
+
+    * DEFLATE
+
+Huffman code type:
+
+    * FIXED
+
+Window size support:
+
+    * 32K
+
+Checksum generation:
+
+    * CRC32, Adler and combined checksum
+
+Limitations
+-----------
+
+* Chained mbufs are not yet supported, therefore max data size which can be passed to the PMD in a single mbuf is 64K - 1. If data is larger than this it will need to be split up and sent as multiple operations.
+
+* Compressdev level 0, no compression, is not supported.
+
+* Dynamic Huffman encoding is not yet supported.
+
+Installation
+------------
+
+The QAT compression PMD is built by default with a standard DPDK build.
+
+It depends on a QAT kernel driver, see :ref:`qat_kernel_installation`.
diff --git a/doc/guides/cryptodevs/qat.rst b/doc/guides/cryptodevs/qat.rst
index b899985..bdc58eb 100644
--- a/doc/guides/cryptodevs/qat.rst
+++ b/doc/guides/cryptodevs/qat.rst
@@ -68,12 +68,32 @@ Limitations
 * Queue pairs are not thread-safe (that is, within a single queue pair, RX and TX from different lcores is not supported).
 
 
-Installation
-------------
+Extra notes on KASUMI F9
+------------------------
+
+When using KASUMI F9 authentication algorithm, the input buffer must be
+constructed according to the 3GPP KASUMI specifications (section 4.4, page 13):
+`<http://cryptome.org/3gpp/35201-900.pdf>`_.
+Input buffer has to have COUNT (4 bytes), FRESH (4 bytes), MESSAGE and DIRECTION (1 bit)
+concatenated. After the DIRECTION bit, a single '1' bit is appended, followed by
+between 0 and 7 '0' bits, so that the total length of the buffer is multiple of 8 bits.
+Note that the actual message can be any length, specified in bits.
 
-To enable QAT in DPDK, follow the instructions for modifying the compile-time
+Once this buffer is passed this way, when creating the crypto operation,
+length of data to authenticate (op.sym.auth.data.length) must be the length
+of all the items described above, including the padding at the end.
+Also, offset of data to authenticate (op.sym.auth.data.offset)
+must be such that points at the start of the COUNT bytes.
+
+
+Building the DPDK QAT cryptodev PMD
+-----------------------------------
+
+
+To enable QAT crypto in DPDK, follow the instructions for modifying the compile-time
 configuration file as described `here <http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html>`_.
 
+
 Quick instructions are as follows:
 
 .. code-block:: console
@@ -81,29 +101,95 @@ Quick instructions are as follows:
 	cd to the top-level DPDK directory
 	make config T=x86_64-native-linuxapp-gcc
 	sed -i 's,\(CONFIG_RTE_LIBRTE_PMD_QAT\)=n,\1=y,' build/.config
+	sed -i 's,\(CONFIG_RTE_LIBRTE_PMD_QAT_SYM\)=n,\1=y,' build/.config
 	make
 
-To use the DPDK QAT PMD an SRIOV-enabled QAT kernel driver is required. The VF
-devices exposed by this driver will be used by the QAT PMD. The devices and
-available kernel drivers and device ids are :
+
+.. _qat_kernel_installation:
+
+Dependency on the QAT kernel driver
+-----------------------------------
+
+To use the QAT PMD an SRIOV-enabled QAT kernel driver is required. The VF
+devices created and initialised by this driver will be used by the QAT PMD.
+
+Instructions for installation are below, but first an explanation of the
+relationships between the PF/VF devices and the PMDs visible to
+DPDK applications.
+
+
+Acceleration services - cryptography and compression - are provided to DPDK
+applications via PMDs which register to implement the corresponding
+cryptodev and compressdev APIs.
+
+Each QuickAssist VF device can expose one cryptodev PMD and/or one compressdev PMD.
+These QAT PMDs share the same underlying device and pci-mgmt code, but are
+enumerated independently on their respective APIs and appear as independent
+devices to applications.
+
+.. Note::
+
+   Each VF can only be used by one DPDK process. It is not possible to share
+   the same VF across multiple processes, even if these processes are using
+   different acceleration services.
+
+   Conversely one DPDK process can use one or more QAT VFs and can expose both
+   cryptodev and compressdev instances on each of those VFs.
+
+
+
+Device and driver naming
+------------------------
+
+* The qat cryptodev driver name is "crypto_qat".
+  The rte_cryptodev_devices_get() returns the devices exposed by this driver.
+
+* Each qat crypto device has a unique name, in format
+  <pci bdf>_<service>, e.g. "0000:41:01.0_qat_sym".
+  This name can be passed to rte_cryptodev_get_dev_id() to get the device_id.
+
+.. Note::
+
+	The qat crypto driver name is passed to the dpdk-test-crypto-perf tool in the -devtype parameter.
+
+	The qat crypto device name is in the format of the slave parameter passed to the crypto scheduler.
+
+* The qat compressdev driver name is "comp_qat".
+  The rte_compressdev_devices_get() returns the devices exposed by this driver.
+
+* Each qat compression device has a unique name, in format
+  <pci bdf>_<service>, e.g. "0000:41:01.0_qat_comp".
+  This name can be passed to rte_compressdev_get_dev_id() to get the device_id.
+
+
+Available kernel drivers
+------------------------
+
+Kernel drivers for each device are listed in the following table. Scroll right
+to check that the driver and device supports the servic you require.
+
 
 .. _table_qat_pmds_drivers:
 
 .. table:: QAT device generations, devices and drivers
 
-   +-----+----------+--------+---------------+------------+--------+------+--------+--------+
-   | Gen | Device   | Driver | Kernel Module | Pci Driver | PF Did | #PFs | Vf Did | VFs/PF |
-   +=====+==========+========+===============+============+========+======+========+========+
-   | 1   | DH895xCC | 01.org | icp_qa_al     | n/a        | 435    | 1    | 443    | 32     |
-   +-----+----------+--------+---------------+------------+--------+------+--------+--------+
-   | 1   | DH895xCC | 4.4+   | qat_dh895xcc  | dh895xcc   | 435    | 1    | 443    | 32     |
-   +-----+----------+--------+---------------+------------+--------+------+--------+--------+
-   | 2   | C62x     | 4.5+   | qat_c62x      | c6xx       | 37c8   | 3    | 37c9   | 16     |
-   +-----+----------+--------+---------------+------------+--------+------+--------+--------+
-   | 2   | C3xxx    | 4.5+   | qat_c3xxx     | c3xxx      | 19e2   | 1    | 19e3   | 16     |
-   +-----+----------+--------+---------------+------------+--------+------+--------+--------+
-   | 2   | D15xx    | p      | qat_d15xx     | d15xx      | 6f54   | 1    | 6f55   | 16     |
-   +-----+----------+--------+---------------+------------+--------+------+--------+--------+
+   +-----+----------+---------------+---------------+------------+--------+------+--------+--------+-----------+-------------+
+   | Gen | Device   | Driver/ver    | Kernel Module | Pci Driver | PF Did | #PFs | VF Did | VFs/PF | cryptodev | compressdev |
+   +=====+==========+===============+===============+============+========+======+========+========+===========+=============+
+   | 1   | DH895xCC | linux/4.4+    | qat_dh895xcc  | dh895xcc   | 435    | 1    | 443    | 32     | Yes       | No          |
+   +-----+----------+---------------+---------------+------------+--------+------+--------+--------+-----------+-------------+
+   | "   | "        | 01.org/4.2.0+ | "             | "          | "      | "    | "      | "      | Yes       | No          |
+   +-----+----------+---------------+---------------+------------+--------+------+--------+--------+-----------+-------------+
+   | 2   | C62x     | linux/4.5+    | qat_c62x      | c6xx       | 37c8   | 3    | 37c9   | 16     | Yes       | No          |
+   +-----+----------+---------------+---------------+------------+--------+------+--------+--------+-----------+-------------+
+   | "   | "        | 01.org/4.2.0+ | "             | "          | "      | "    | "      | "      | Yes       | Yes         |
+   +-----+----------+---------------+---------------+------------+--------+------+--------+--------+-----------+-------------+
+   | 2   | C3xxx    | linux/4.5+    | qat_c3xxx     | c3xxx      | 19e2   | 1    | 19e3   | 16     | Yes       | No          |
+   +-----+----------+---------------+---------------+------------+--------+------+--------+--------+-----------+-------------+
+   | "   | "        | 01.org/4.2.0+ | "             | "          | "      | "    | "      | "      | Yes       | Yes         |
+   +-----+----------+---------------+---------------+------------+--------+------+--------+--------+-----------+-------------+
+   | 2   | D15xx    | p             | qat_d15xx     | d15xx      | 6f54   | 1    | 6f55   | 16     | Yes       | No          |
+   +-----+----------+---------------+---------------+------------+--------+------+--------+--------+-----------+-------------+
 
 
 The ``Driver`` column indicates either the Linux kernel version in which
@@ -196,9 +282,9 @@ Consult the *Getting Started Guide* at the same URL for further information.
 
 The steps below assume you are:
 
-* Building on a platform with one ``DH895xCC`` device.
-* Using package ``qatmux.l.2.3.0-34.tgz``.
-* On Fedora21 kernel ``3.17.4-301.fc21.x86_64``.
+* Building on a platform with one ``C62x`` device.
+* Using package ``qat1.7.l.4.2.0-000xx.tar.gz``.
+* On Fedora26 kernel ``4.11.11-300.fc26.x86_64``.
 
 In the BIOS ensure that SRIOV is enabled and VT-d is disabled.
 
@@ -206,21 +292,30 @@ Uninstall any existing QAT driver, for example by running:
 
 * ``./installer.sh uninstall`` in the directory where originally installed.
 
-* or ``rmmod qat_dh895xcc; rmmod intel_qat``.
 
 Build and install the SRIOV-enabled QAT driver::
 
     mkdir /QAT
     cd /QAT
 
-    # Copy qatmux.l.2.3.0-34.tgz to this location
-    tar zxof qatmux.l.2.3.0-34.tgz
+    # Copy the package to this location and unpack
+    tar zxof qat1.7.l.4.2.0-000xx.tar.gz
 
-    export ICP_WITHOUT_IOMMU=1
-    ./installer.sh install QAT1.6 host
+    ./configure --enable-icp-sriov=host
+    make install
+
+You can use ``cat /sys/kernel/debug/qat<your device type and bdf>/version/fw`` to confirm the driver is correctly installed and is using firmware version 4.2.0.
+You can use ``lspci -d:37c9`` to confirm the presence of the 16 VF devices available per ``C62x`` PF.
+
+Confirm the driver is correctly installed and is using firmware version 4.2.0::
+
+    cat /sys/kernel/debug/qat<your device type and bdf>/version/fw
+
+
+Confirm the presence of 48 VF devices - 16 per PF::
+
+    lspci -d:37c9
 
-You can use ``cat /proc/icp_dh895xcc_dev0/version`` to confirm the driver is correctly installed.
-You can use ``lspci -d:443`` to confirm the  of the 32 VF devices available per ``DH895xCC`` device.
 
 To complete the installation - follow instructions in `Binding the available VFs to the DPDK UIO driver`_.
 
@@ -261,6 +356,7 @@ To complete the installation - follow instructions in `Binding the available VFs
 
       sudo yum install zlib-devel
       sudo yum install openssl-devel
+      sudo yum install libudev-devel
 
 .. Note::
 
@@ -343,35 +439,6 @@ Another way to bind the VFs to the DPDK UIO driver is by using the
     ./usertools/dpdk-devbind.py -b igb_uio 0000:03:01.1
 
 
-Extra notes on KASUMI F9
-------------------------
-
-When using KASUMI F9 authentication algorithm, the input buffer must be
-constructed according to the 3GPP KASUMI specifications (section 4.4, page 13):
-`<http://cryptome.org/3gpp/35201-900.pdf>`_.
-Input buffer has to have COUNT (4 bytes), FRESH (4 bytes), MESSAGE and DIRECTION (1 bit)
-concatenated. After the DIRECTION bit, a single '1' bit is appended, followed by
-between 0 and 7 '0' bits, so that the total length of the buffer is multiple of 8 bits.
-Note that the actual message can be any length, specified in bits.
-
-Once this buffer is passed this way, when creating the crypto operation,
-length of data to authenticate (op.sym.auth.data.length) must be the length
-of all the items described above, including the padding at the end.
-Also, offset of data to authenticate (op.sym.auth.data.offset)
-must be such that points at the start of the COUNT bytes.
-
-Device and driver naming
-------------------------
-
-The qat crypto driver name is "crypto_qat".
-This name is passed to the dpdk-test-crypto-perf tool in the -devtype parameter.
-The rte_cryptodev_devices_get() can return the devices exposed by a driver.
-
-Each qat crypto device has a unique name, in format
-<pci bdf>_<service>, e.g. "0000:41:01.0_qat_sym".
-This name can be passed to rte_cryptodev_get_dev_id() to get the device_id.
-This is also the format of the slave parameter passed to the crypto scheduler.
-
 Debugging
 ----------------------------------------
 
diff --git a/doc/guides/rel_notes/release_18_08.rst b/doc/guides/rel_notes/release_18_08.rst
index bc01242..2230d3e 100644
--- a/doc/guides/rel_notes/release_18_08.rst
+++ b/doc/guides/rel_notes/release_18_08.rst
@@ -46,6 +46,11 @@ New Features
   Flow API support has been added to CXGBE Poll Mode Driver to offload
   flows to Chelsio T5/T6 NICs.
 
+* **Added a new compression PMD using Intel's QuickAssist (QAT) device family.**
+
+   Added the new ``QAT`` compression driver, for compression and decompression
+   operations in software. See the :doc:`../compressdevs/qat_comp` compression
+   driver guide for details on this new driver.
 
 API Changes
 -----------
-- 
2.7.4