Re: [dpdk-dev] [PATCH v2] examples: add multi process crypto application

["Kusztal, ArkadiuszX" <arkadiuszx.kusztal@intel.com>]

Hi Akhil,

> -----Original Message-----
> From: Akhil Goyal <akhil.goyal@nxp.com>
> Sent: Thursday, July 2, 2020 9:30 PM
> To: Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>; dev@dpdk.org;
> Trahe, Fiona <fiona.trahe@intel.com>; Burakov, Anatoly
> <anatoly.burakov@intel.com>
> Subject: RE: [PATCH v2] examples: add multi process crypto application
> 
> Hi Arek/Fiona,
> 
> > This patch adds example application that can test usage of cryptodev
> > in multi process environment.
> > More can be found in mp_crypto.rst in sample app guides.
> >
> > Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
> > ---
> 
> Shouldn't this be part of examples/multi_process?
> 
> >  doc/guides/sample_app_ug/index.rst     |    1 +
> >  doc/guides/sample_app_ug/mp_crypto.rst |  153 +++++
> >  examples/mp_crypto/Makefile            |   58 ++
> >  examples/mp_crypto/main.c              | 1109
> > ++++++++++++++++++++++++++++++++
> >  examples/mp_crypto/meson.build         |   18 +
> >  examples/mp_crypto/mp_crypto.c         |  139 ++++
> >  examples/mp_crypto/mp_crypto.h         |  224 +++++++
> >  examples/mp_crypto/mp_crypto_ipc.c     |   32 +
> >  examples/mp_crypto/mp_crypto_parser.c  |  511 +++++++++++++++
> > examples/mp_crypto/mp_crypto_parser.h  |  149 +++++
> > examples/mp_crypto/mp_crypto_vectors.c |  175 +++++
> >  examples/mp_crypto/mp_crypto_vectors.h |   66 ++
> >  12 files changed, 2635 insertions(+)
> 
> Can we split this patch in some logical way so that it can be reviewed easily?
> 
> >  create mode 100644 doc/guides/sample_app_ug/mp_crypto.rst
> >  create mode 100644 examples/mp_crypto/Makefile  create mode 100644
> > examples/mp_crypto/main.c  create mode 100644
> > examples/mp_crypto/meson.build  create mode 100644
> > examples/mp_crypto/mp_crypto.c  create mode 100644
> > examples/mp_crypto/mp_crypto.h  create mode 100644
> > examples/mp_crypto/mp_crypto_ipc.c
> >  create mode 100644 examples/mp_crypto/mp_crypto_parser.c
> >  create mode 100644 examples/mp_crypto/mp_crypto_parser.h
> >  create mode 100644 examples/mp_crypto/mp_crypto_vectors.c
> >  create mode 100644 examples/mp_crypto/mp_crypto_vectors.h
> >
> > diff --git a/doc/guides/sample_app_ug/index.rst
> > b/doc/guides/sample_app_ug/index.rst
> > index affa9c5..ff033e4 100644
> > --- a/doc/guides/sample_app_ug/index.rst
> > +++ b/doc/guides/sample_app_ug/index.rst
> > @@ -35,6 +35,7 @@ Sample Applications User Guides
> >      link_status_intr
> >      server_node_efd
> >      service_cores
> > +    mp_crypto
> >      multi_process
> >      qos_metering
> >      qos_scheduler
> > diff --git a/doc/guides/sample_app_ug/mp_crypto.rst
> > b/doc/guides/sample_app_ug/mp_crypto.rst
> > new file mode 100644
> > index 0000000..d3cb1d9
> > --- /dev/null
> > +++ b/doc/guides/sample_app_ug/mp_crypto.rst
> > @@ -0,0 +1,153 @@
> > +..  SPDX-License-Identifier: BSD-3-Clause
> > +    Copyright(c) 2020 Intel Corporation.
> > +
> > +.. _mp_crypto:
> > +
> > +Multi-process Crypto Sample Application
> > +=======================================
> > +
> > +The Multi-process Crypto application is a simple application that
> > +allows to run crypto related operations in a multiple process
> > +environment. It builds on the EAL primary/secondary process
> infrastructure.
> > +
> > +The application allows a user to configure devices, setup
> > +queue-pairs, create and init sessions and specify data-path flow
> > +(enqueue/dequeue) in different processes. The app can help to check
> > +if the PMD behaves correctly in scenarios like the following:
> > +
> > +* device is configured in primary process, queue-pairs are setup in
> > +secondary
> > process
> > +
> > +* queue pair is shared across processes, i.e. enqueue in one process
> > +and
> > dequeue in another
> > +
> > +
> Extra line
> 
> > +Compiling the Application
> > +-------------------------
> > +
> > +To compile the sample application see :doc:`compiling`.
> > +
> > +The application is located in the ``mp_crypto`` sub-directory.
> > +
> > +Running the Application
> > +-----------------------
> > +
> > +App binary: mp_crypto (in mp_crypto/build/app)
> > +
> > +For running PRIMARY or SECONDARY process standard EAL options apply:
> > +
> > +.. code-block:: console
> > +
> > +    ./mp_crypto --proc-type primary
> > +
> > +    ./mp_crypto --proc-type secondary
> > +
> > +.. Note::
> > +
> > +	The same set of BDFs must be passed to all processes.
> > +
> > +.. Note::
> > +	The same crypto devices must be created in all processes, e.g. in qat
> > +	case if asym and sym devices are enabled in the primary process,
> they
> > +	must be enabled in all secondary processes.
> > +
> > +General help can by checked by running:
> > +
> > +.. code-block:: console
> > +
> > +    ./mp_crypto -- -h
> > +
> > +The application has a number of command line options:
> > +
> > +.. code-block:: console
> > +
> > +    ./mp_crypto -- --devtype [dev-name]
> > +
> > +This option specifies which driver to use by its name (for example
> "crypto_qat").
> > +The same name must be passed to all processes.
> > +
> > +.. code-block:: console
> > +
> > +    ./mp_crypto -- --config_dev [devA, devB,]
> > +
> > +This option specifies the list of devices that should be configured
> > +by this
> > process,
> > +this results in a call to the ``rte_cryptodev_configure`` API. devX
> > +is a positive integer (including zero), the value is according to
> > +probe order (from the
> > smallest
> > +BDF number), not necessarily the cmdline order.
> > +
> > +Example command:
> > +
> > +.. code-block:: console
> > +
> > +    ./mp_crypto -w 03:01.2 -w 03:01.1 -w 03:01.3 --config-dev 0,2
> > +
> > +will configure devices 03:01.1 and 03:01.3.
> > +
> > +.. code-block:: console
> > +
> > +    ./mp_crypto -- --qp-config=[devA]:[qp_A, qp_B,];[devB]:[qp_A,
> > + qp_C];
> > +
> > +devX - positive integer (including zero), as in config_dev command
> > +
> > +qp_X - positive integer (including zero), specifies which queue pair
> > +should be
> > setup
> > +
> > +This command specifies which queue pairs should be setup, resulting
> > +in a call to ``rte_cryptodev_queue_pair_setup`` API.
> > +
> > +.. code-block:: console
> > +
> > +    ./mp_crypto -w 03:01.2 -w 03:01.1 -w 03:01.3 --qp-
> config="0:0,1;1:1;2:0,1;"
> > +
> > +This command will configure queue pairs 0 and 1 on device 0
> > +(03:01.1), queue
> > pair 1
> > +on device 1 (03:01.2), queue pairs 0 and 1 on device 2 (03:01.3). The
> > +device in
> > question
> > +should be configured before that, though not necessarily by the same
> process.
> > +
> > +.. code-block:: console
> > +
> > +    ./mp_crypto -- --enq=[devX]:[qpX]:[ops]:[vector_id]
> > +    ./mp_crypto -- --deq=[devX]:[qpX]:[ops]:[vector_id]
> > +
> > +devX - positive integer (including zero), as in config_dev command
> > +
> > +qp_X - positive integer (including zero), as in qp-config command
> qpX
> 
> > +
> > +ops - when positive integer - number of operations to
> > +enqueue/dequeue, when
> > 0 infinite loop
> > +
> > +vector_id - positive integer (including zero), vector_id used by this
> > +process
> What is vector_id? How is it meant to be used?
> 
> > +
> > +This commands will enqueue/dequeue "ops" number of packets to qp_X
> on
> > devX.
> > +Example usage:
> > +
> > +.. code-block:: console
> > +
> > +    ./mp_crypto -- --enq=2:0:0:0, --deq=2:0:0:0,
> > +
> > +Note. ',' comma character is necessary at the end due to some parser
> > shortcomings.
> > +
> > +To close the application when running in an infinite loop a signal
> > +handler is registered to catch interrupt signals i.e. ``ctrl-c``
> > +should be used. When used in primary process other processes will be
> > +notified about exiting intention and will close after collecting remaining
> packets (if dequeuing).
> > +
> > +Example commands
> > +----------------
> > +
> > +Use two different devices on 3 separate queues:
> > +
> > +.. code-block:: console
> > +
> > +    ./mp_crypto --proc-type primary -c 1 -w 03:01.1 -w 03:01.2 --
> > + --devtype
> > "crypto_qat" --config-dev 0,1   --qp-config="0:0,1;1:0,1;" --session-
> mask=0x3  --
> > enq=0:0:0:0, --deq=0:0:0:0,  --print-stats
> > +    ./mp_crypto --proc-type secondary -c 2 -w 03:01.1 -w 03:01.2 --
> > + --devtype
> > "crypto_qat"  --enq=0:1:0:0, --deq=0:1:0:0,  --print-stats
> > +    ./mp_crypto --proc-type secondary -c 4 -w 03:01.1 -w 03:01.2 --
> > + --devtype
> > "crypto_qat"  --enq=1:0:0:0, --deq=1:0:0:0,  --print-stats
> > +
> > +Use different processes to enqueue and dequeue to one queue pair:
> > +
> > +.. code-block:: console
> > +
> > +    ./mp_crypto --proc-type primary -c 1 -w 03:01.1 -- --devtype
> > + "crypto_qat" --
> > config-dev 0    --session-mask=0x3 --qp-config="0:1;"   --enq=0:1:0:0,   --
> print-
> > stats
> > +    ./mp_crypto --proc-type secondary -c 2 -w 03:01.1 -- --devtype
> "crypto_qat"
> > --deq=0:1:0:0,   --print-stats
> 
> What are the expected results out of this app. Can we have some logs in this
> doc?
[Arek] - yes we will add (soon we will send v3), example output would look something like that:

For following command (enqueue only in primary process, dequeue in one of secondary's)
./mp_crypto --proc-type primary -c 1 -w 03:01.1 -w 03:01.2 -w 03:02.7 --log-level 7 -- --devtype "crypto_qat" --config-dev 2    --session-mask=0x3 --qp-config="2:1;"   --enq=2:1:0:0,   --print-stats
---------------------------------------------------------------------
USER1: Run enqueue on device 2
USER1: Run enqueue on qp 1
USER1: - Setting driver 2 for this process
USER1: - Starting PRIMARY process
USER1: Number of processes = 1
USER1: Configure devices according to mask: 0x4
USER1: - Configure Device id 2
CRYPTODEV: elt_size 16 is expanded to 192

CRYPTODEV: elt_size 16 is expanded to 192

USER1: - Configuring queues:
CRYPTODEV: qp 1 on dev 2 is not initialised
USER1: Created qp 1 on dev 2
USER1: - Configured 1 queues.
USER1: Initialized session = 0
USER1: Initialized session = 1
USER1: Configuring vector 0, using session 0
USER1: Start enqueuing packets on dev 2 qp 1
CRYPTODEV: qp 1 on dev 2 is initialised
USER1: Enqueuing /^C
Interrupted, finalizing...USER1: STATS: Enqueued on dev 2 = 16337024
USER1: STATS: Enqueue err count on dev 2 = 0

Press 'q' to quit: q
USER1: Waiting for 1 out of 10 seconds
USER1: All secondary processes exited normally
---------------------------------------------------------------------

> 
> Can we kill the primary app first? Will the secondary get killed automatically
> or will throw error?
[Arek] - primary will inform all secondary processes to collect remaining packets (if possible) and finish. Primary will wait at most 10 seconds. SIGINT is registered so actually it will work correctly with ctrl-c.

> Can we open and kill secondary apps multiple times without killing the
> primary?
[Arek] - yes.
> 
> This multi process app is only taking care of  crypto queues while others are
> for NICs.
> Is it not worth to have crypto+NIC multi process app instead of this app?
[Arek] - initially main purpose was to check PMD behavior when:
1) configure cryptodev, sessions, queues and do enqueue/dequeue from another different processes
2) run enqueue/dequeue from different processes on the same queue pair.
If it can be done with one app I think it is ok.
> I believe most common usecases of crypto are with network traffic.
> Can we modify l2fwd-crypto for multi process?
> 
> > +
> > +Limitations
> > +-----------
> > +
> > +Software devices are not supported currently, but small changes in
> > +code suffice
> > to enable it.
> > +
> > +Only one crypto vector and session type is possible to chose right
> > +now and it is
> > AES-GCM test case.
> 
> Regards,
> Akhil