Re: [dpdk-dev] [PATCH v2 0/4] introduce multi-function processing support

["Trahe, Fiona" <fiona.trahe@intel.com>]

Hi David, Ferruh,

> -----Original Message-----
> From: Coyle, David <david.coyle@intel.com>
> Sent: Tuesday, April 7, 2020 12:28 PM
> To: Yigit, Ferruh <ferruh.yigit@intel.com>; dev@dpdk.org
> Cc: Doherty, Declan <declan.doherty@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>; De Lara
> Guarch, Pablo <pablo.de.lara.guarch@intel.com>; Ryan, Brendan <brendan.ryan@intel.com>;
> shreyansh.jain@nxp.com; hemant.agrawal@nxp.com; Akhil Goyal <akhil.goyal@nxp.com>
> Subject: RE: [dpdk-dev] [PATCH v2 0/4] introduce multi-function processing support
> 
> Hi Ferruh, see below
> 
> > >
> > > While DPDK's rte_cryptodev and rte_compressdev allow many
> > > cryptographic and compression algorithms to be chained together in one
> > > operation, there is no way to chain these with any error detection or
> > > checksum algorithms. And there is no way to chain crypto and
> > > compression algorithms together. The multi-function  interface will
> > > allow these chains to be created, and also allow any future type of
> > operation to be easily added.
> >
> > I was thinking if the cryptodev can be used instead but this paragraph already
> > seems explained it. But again can you please elaborate why rawdev is used?
> 
> [DC] There are a number of reasons the rawdev approach was ultimately chosen:
> 
> 1) As the paragraph above explains, our primary use-case was to chain a crypto operation with error
> detection algorithms such as CRC or BIP as this could leverage optimized multi-function
> implementations such as in the IPSec Multi-Buffer library and have a significant impact on
> performance of network access dataplane processing such as for vCMTS (DOCSIS MAC).
> However such error detection algorithms are not Crypto functions so some early advice we took was
> that it would not be suitable to add these to cryptodev.
> Also, with a view to the future, the multi-function rawdev approach allows crypto operations to be
> chained with compression operations.
> Again, neither cryptodev or compressdev allows this type chaining.
> 
> 2) An earlier version of multi-function suggested adding a new library called rte_accelerator, as
> described here http://mails.dpdk.org/archives/dev/2020-February/157045.html
> We received some comments on the dev mailing list that we should not add yet another acceleration
> library to DPDK.
> And we also subsequently felt that the rawdev approach is better - that rationale is described below.
> 
> rte_accelerator was also built on top of crypto and compress devices which already existed e.g.
> drivers/crypto/aesni_mb, drivers/crypto/qat and drivers/compress/qat .
> We subsequently realized that this was somewhat confusing when performing multi-function type
> operations. For example, for combined Crypto-Compression operations in the future, it would use
> either an existing crypto or compress device, but neither really made sense when the operations are
> combined.
> What was needed was a raw device which allowed an application to configure any type of device and
> it's queue pairs and send any type of operation to that device.
> 
> For both of these reasons, we decided to go down the rawdev route, with a multi-function interface
> which can be used by several raw device drivers.
> 
> 3) rawdev is the ideal place to try out a new approach like this to accessing devices.
> Adding it here allows potential consumers of this such as VNF solution providers to study and try out
> this approach, and take advantage of the multi-function operations already supported in the IPSec
> Multi-Buffer library such as Crypto-CRC and Crypto-CRC-BIP, all without DPDK committing to a new
> library upfront.
> We would hope that the multi-function rawdev approach will mature over time (through feedback
> from customers, new use-cases arising etc.), at which point it could be potentially be moved into the
> main DPDK library set.
> 
[Fiona] agree with above, in particular item (2). Just to expand a bit more on this: To do a crypto+compression op
one would only send one op to one device. That device could have been either a crypto device which also
implemented multi-fn, by adding compression, crc, etc to its capabilities OR a compression device which added crypto, crc, bip capabilities.
Both were confusing and both raised questions about whether one could still do "normal" ops on the device, e.g. whether
a normal crypto op could be interleaved on same qp as a multi-fn op. And how the capabilities would reflect what the device could do.
It seems better to me to have a multifn device, which does explicitly just multifn ops.

Building this on top of rawdev is a good fit in my opinion, for the following reasons:
 * avoids duplication of device APIs, as rawdev configure, start, stop, qp_setup, etc are all there already, also a nice set of stats APIs
 * no impact or dependency added to rawdev lib
 * avoids breakages on cryptodev or compressdev APIs
 * avoids code duplication where functionality is already in a lib, e.g. re-uses cryptodev and compressdev headers. This adds a dependency, but I think that's ok as multi-function inherently depends on these functions.
 * allows easy extension to add new functionality not currently available in any lib (CRC and BIP)
 * allows evolution - range of useful chains which may emerge is not yet clear.

I do have some concerns, but these are resolvable in my opinion.
    (A)    as there's no rawdev capability APIs and capabilities are essentially opaque to the rawdev API, the application uses explicit device naming to create or find a device that it knows will fulfil the multifunction APIs. I can see how this works for rawdevs which expect to have only one PMD that will fulfil the service, however I'd expect multi-fn to have at least 2 driver types, probably more eventually. To be extensible I'd suggest a naming convention for a class of devices. E.g. all devices and drivers that implement multi-fn should create a rawdev named mfn_xxx, e.g. mfn_aesni_mb, mfn_qat. The "mfn_" string should be defined in the mfn hdr. This would allow creation of apis like rte_multi_fn_count() which could find rawdevs which implement mfn_ without hardcoding specific driver names.
    (B)    version control of the multi-function APIs. Putting the multifn API into the drivers/raw/common directory gives a lot of freedom while it's experimental. But can it benefit from API/ABI breakage infrastructure once the experimental tag is removed? Is there any reason not to move the common files to a lib/librte_multi_fn API?
    (C)    xstat name strings should be moved from aesni_mb PMD to common and maybe use same naming convention, so appl can query same stats from any device, e.g. "mfn_successful_enqueues" could be implemented by all PMDs. If PMDs want to add driver-specific stats they can add their own without the mfn_, instead create their own unique stat name.
    (D)    The unit test code is not extensible - again probably as based on previous rawdevs where there's only 1 implementation. For mfn I'd suggest replacing test_rawdev_selftest_aesni_mb() with a test_rawdev_selftest_multi_function(), which finds and/or creates all the raw PMDs implementing the mfn API and runs a test on each. And move the test files from the drivers/raw/aesni_mb dir to app/test and make generic so can run against any device named mfn_xxx
    (E)    the main reason to piggyback onto crypto_perf_tool is to get the benefit of parsing and of all the crypto setup.  However this code has been inflated a lot, in part due to name diffs like rte_cryptodev_enqueue_burst() vs rte_multi_fn_enqueue_burst(). Maybe could be a lot slimmer with macros like ENQUEUE_BURST(dev, qp, void *op, burst_size) ? would mean a compile time decision to do either multifn OR cryptodev API calls, but I think that may work and simplify it.
    (F)    ok, this is a bit pedantic, (sorry David!) but should the aesni_mb rawdev be renamed aesni_mb_mfn throughout (files, fns, dev and driver name). I mean it's implementing the mfn type of rawdev. I'm thinking ahead to QAT - it can implement a sym device, an asym device, a compression device and in future a multi-fn device. I'd propose to name it qat_multifn in case there'll be some other kind of rawdev device it could also implement in future. So the name qat_raw wouldn't be so helpful. (we made that mistake with qat_crypto, which should probably have been qat_sym_crypto - in my opinion more specific names are better)

I have a few minor comment- I'll reply on specific patches.