Re: [dpdk-dev] [PATCH v4 0/8] vhost: intdroduce vhost user crypto backend

Re: [dpdk-dev] [PATCH v4 0/8] vhost: intdroduce vhost user crypto backend

[Harish Kumar Ambati]

Hi ,

I am trying to verify Virtio Crypto use case   on  DPDK 19.05 . While running DPDK test app "cryptodev_virtio_autotest " in the VM ,I could see control connection created on the vhost backend(vhost-crypto) but functionality does not work/test case fails.

Below is the configuration used in the setup.  I have enabled debug in virtio/crypto components. Could someone please help me here.

On debugging ,I found that  test_blockcipher_one_case() pauses to dequeue burst  . Also , I see virtio_crypto_init_device(): hw->max_dataqueues is 1  in the VM though in the backend it's initialized to 8.

                  /* Process crypto operation */
        if (rte_cryptodev_enqueue_burst(dev_id, 0, &op, 1) != 1) {
                snprintf(test_msg, BLOCKCIPHER_TEST_MSG_LEN,
                        "line %u FAILED: %s",
                        __LINE__, "Error sending packet for encryption");
                status = TEST_FAILED;
                goto error_exit;
        }

                printf("testsuite_setup func %s line %d \n", __func__, __LINE__);
        op = NULL;

        while (rte_cryptodev_dequeue_burst(dev_id, 0, &op, 1) == 0)
                rte_pause();



Qemu command:

./bins/bin/qemu-system-x86_64 -machine accel=kvm -cpu host -smp 2 -m 4G -drive file=os_compute.img,if=none,id=disk -device ide-hd,drive=disk,bootindex=0 -object memory-backend-file,id=mem,size=4G,mem-path=/dev/hugepages,share=on -mem-prealloc -numa node,memdev=mem -chardev socket,id=charcrypto0,path=/tmp/vm0_crypto0.sock -object cryptodev-vhost-user,id=cryptodev0,chardev=charcrypto0 -device virtio-crypto-pci,id=crypto0,cryptodev=cryptodev0  -device e1000,netdev=network0,mac=52:55:00:d1:55:01 -netdev tap,id=network0,ifname=tap0,script=no,downscript=no
VNC server running on 127.0.0.1:5901


On host :
root@linux:/home/user/harish/dpdk_qs/dpdk-19.05#<mailto:root@linux:/home/user/harish/dpdk_qs/dpdk-19.05> ./examples/vhost_crypto/build/vhost-crypto --socket-mem 2048,0 --legacy-mem  --vdev crypto_openssl  -l 1,2  -n 4  -- --config "(1,0,0)(1,0,1)(1,0,2)(2,0,1)(2,0,2)" --socket-file 1,/tmp/vm0_crypto0.sock  --socket-file=2,/tmp/vm0_crypto1.sock
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Debug dataplane logs available - lower performance
EAL: Probing VFIO support...
EAL: PCI device 0000:00:19.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:1502 net_e1000_em
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:107d net_e1000_em
CRYPTODEV: Creating cryptodev crypto_openssl

CRYPTODEV: Initialisation parameters - name: crypto_openssl,socket id: 0, max queue pairs: 8

VHOST_CONFIG: read message VHOST_USER_CRYPTO_CREATE_SESS
USER1: [Vhost-Crypto] vhost_crypto_create_sess() line 391: Session 1 created for vdev 0.
Vhost-Crypto func vhost_crypto_create_sess line 395




In the VM :


# modprobe uio_pci_generic
#  echo -n 0000:00:03.0 > /sys/bus/pci/drivers/virtio-pci/unbind
# echo "1af4 1054" > /sys/bus/pci/drivers/uio_pci_generic/new_id
#echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

root@user-Standard-PC-i440FX-PIIX-1996:/home/user/dpdk-19.05/app/test#<mailto:root@user-Standard-PC-i440FX-PIIX-1996:/home/user/dpdk-19.05/app/test> ./test
EAL: Detected 2 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Debug dataplane logs available - lower performance
EAL: Probing VFIO support...
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
EAL: PCI device 0000:00:03.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 1af4:1054 crypto_virtio
PMD: crypto_virtio_create():  >>
CRYPTODEV: Creating cryptodev 0000:00:03.0

CRYPTODEV: Initialisation parameters - name: 0000:00:03.0,socket id: 0, max queue pairs: 0
INIT: crypto_virtio_create(): dev 0 vendorID=0x1af4 deviceID=0x1054
INIT: virtio_read_caps(): [84] cfg type: 5, bar: 0, offset: 0000, len: 0
INIT: virtio_read_caps(): [70] cfg type: 2, bar: 4, offset: 3000, len: 4096
INIT: virtio_read_caps(): [60] cfg type: 4, bar: 4, offset: 2000, len: 4096
INIT: virtio_read_caps(): [50] cfg type: 3, bar: 4, offset: 1000, len: 4096
INIT: virtio_read_caps(): [40] cfg type: 1, bar: 4, offset: 0000, len: 4096
INIT: virtio_read_caps(): found modern virtio pci device.
INIT: virtio_read_caps(): common cfg mapped at: 0x1100800000
INIT: virtio_read_caps(): device cfg mapped at: 0x1100802000
INIT: virtio_read_caps(): isr cfg mapped at: 0x1100801000
INIT: virtio_read_caps(): notify base: 0x1100803000, notify off multiplier: 4
INIT: vtpci_cryptodev_init(): modern virtio pci detected.
PMD: virtio_crypto_init_device():  >>
PMD: virtio_negotiate_features():  >>
INIT: virtio_negotiate_features(): guest_features before negotiate = 100000000
INIT: virtio_negotiate_features(): host_features before negotiate = 130000000
INIT: virtio_negotiate_features(): features after negotiate = 100000000
INIT: virtio_crypto_init_device(): hw->max_dataqueues=1
EAL: PCI device 0000:00:04.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:100e net_e1000_em
APP: HPET is not enabled, using TSC as default timer
RTE>>cryptodev_virtio_autotest
+ ------------------------------------------------------- +
+ Test Suite : Crypto VIRTIO Unit Test Suite
PMD: virtio_crypto_dev_info_get():  >>
PMD: virtio_crypto_dev_info_get():  >>
PMD: virtio_crypto_sym_get_session_private_size():  >>
CRYPTODEV: elt_size 0 is expanded to 128

PMD: virtio_crypto_dev_info_get():  >>
PMD: virtio_crypto_dev_configure():  >>
PMD: virtio_crypto_init_device():  >>
PMD: virtio_negotiate_features():  >>
INIT: virtio_negotiate_features(): guest_features before negotiate = 100000000
INIT: virtio_negotiate_features(): host_features before negotiate = 130000000
INIT: virtio_negotiate_features(): features after negotiate = 100000000
INIT: virtio_crypto_init_device(): hw->max_dataqueues=1
PMD: virtio_crypto_ctrlq_setup():  >>
PMD: virtio_crypto_queue_setup():  >>
INIT: virtio_crypto_queue_setup(): setting up queue: 1
INIT: virtio_crypto_queue_setup(): vq_size: 64
INIT: virtio_crypto_queue_setup(): ctrlq vring_size: 4612, rounded_vring_size: 8192
INIT: virtio_crypto_queue_setup(): vq->vq_ring_mem(physical): 0x14469000
INIT: virtio_crypto_queue_setup(): vq->vq_ring_virt_mem: 0x100669000
PMD: virtio_crypto_vring_start():  >>
INIT: modern_setup_queue(): queue 1 addresses:
INIT: modern_setup_queue():      desc_addr: 14469000
INIT: modern_setup_queue():      aval_addr: 14469400
INIT: modern_setup_queue():      used_addr: 1446a000
INIT: modern_setup_queue():      notify addr: 0x1100803004 (notify offset: 1)
INIT: virtio_crypto_ctrlq_start(): VQ: - size=64; free=64; used=0; desc_head_idx=0; avail.idx=0; used_cons_idx=0; used.idx=0; avail.flags=0x1; used.flags=0x0
PMD: virtio_crypto_sym_get_session_private_size():  >>
PMD: virtio_crypto_qp_setup():  >>
PMD: virtio_crypto_queue_setup():  >>
INIT: virtio_crypto_queue_setup(): setting up queue: 0
INIT: virtio_crypto_queue_setup(): vq_size: 1024
INIT: virtio_crypto_queue_setup(): dataq vring_size: 28676, rounded_vring_size: 32768
INIT: virtio_crypto_queue_setup(): vq->vq_ring_mem(physical): 0x14425000
INIT: virtio_crypto_queue_setup(): vq->vq_ring_virt_mem: 0x100625000
PMD: virtio_crypto_dev_info_get():  >>
PMD: virtio_crypto_dev_configure():  >>
PMD: virtio_crypto_init_device():  >>
PMD: virtio_negotiate_features():  >>
INIT: virtio_negotiate_features(): guest_features before negotiate = 100000000
INIT: virtio_negotiate_features(): host_features before negotiate = 130000000
INIT: virtio_negotiate_features(): features after negotiate = 100000000
INIT: virtio_crypto_init_device(): hw->max_dataqueues=1
PMD: virtio_crypto_ctrlq_setup():  >>
PMD: virtio_crypto_queue_setup():  >>
INIT: virtio_crypto_queue_setup(): setting up queue: 1
INIT: virtio_crypto_queue_setup(): vq_size: 64
INIT: virtio_crypto_queue_setup(): ctrlq vring_size: 4612, rounded_vring_size: 8192
INIT: virtio_crypto_queue_setup(): vq->vq_ring_mem(physical): 0x14469000
INIT: virtio_crypto_queue_setup(): vq->vq_ring_virt_mem: 0x100669000
PMD: virtio_crypto_vring_start():  >>
INIT: modern_setup_queue(): queue 1 addresses:
INIT: modern_setup_queue():      desc_addr: 14469000
INIT: modern_setup_queue():      aval_addr: 14469400
INIT: modern_setup_queue():      used_addr: 1446a000
INIT: modern_setup_queue():      notify addr: 0x1100803004 (notify offset: 1)
INIT: virtio_crypto_ctrlq_start(): VQ: - size=64; free=64; used=0; desc_head_idx=0; avail.idx=0; used_cons_idx=0; used.idx=0; avail.flags=0x1; used.flags=0x0
PMD: virtio_crypto_sym_get_session_private_size():  >>
PMD: virtio_crypto_qp_setup():  >>
PMD: virtio_crypto_queue_setup():  >>
INIT: virtio_crypto_queue_setup(): setting up queue: 0
INIT: virtio_crypto_queue_setup(): vq_size: 1024
INIT: virtio_crypto_queue_setup(): dataq vring_size: 28676, rounded_vring_size: 32768
INIT: virtio_crypto_queue_setup(): vq->vq_ring_mem(physical): 0x14425000
INIT: virtio_crypto_queue_setup(): vq->vq_ring_virt_mem: 0x100625000
PMD: virtio_crypto_dev_stats_reset():  >>
PMD: virtio_crypto_dataq_start():  >>
PMD: virtio_crypto_vring_start():  >>
INIT: modern_setup_queue(): queue 0 addresses:
INIT: modern_setup_queue():      desc_addr: 14425000
INIT: modern_setup_queue():      aval_addr: 14429000
INIT: modern_setup_queue():      used_addr: 1442a000
INIT: modern_setup_queue():      notify addr: 0x1100803000 (notify offset: 0)
INIT: virtio_crypto_dataq_start(): VQ: - size=1024; free=1024; used=0; desc_head_idx=0; avail.idx=0; used_cons_idx=0; used.idx=0; avail.flags=0x1; used.flags=0x0
PMD: virtio_crypto_dev_info_get():  >>
PMD: virtio_crypto_sym_get_session_private_size():  >>
PMD: virtio_crypto_sym_configure_session():  >>
PMD: virtio_crypto_check_sym_session_paras():  >>
PMD: virtio_crypto_send_command():  >>
INIT: virtio_crypto_send_command(): vq->vq_desc_head_idx = 0, vq = 0x1006248c0
INIT: virtio_crypto_send_command(): vq->vq_queue_index = 1
INIT: virtio_crypto_send_command(): vq->vq_free_cnt=64
vq->vq_desc_head_idx=0


Another observation is when I tried to use   cryptodev-linux in the VM ,results are show as pass independent of the vhost-crypto backend on the host. Do we need to change  anything in the qemu for kernel crypto to work in the VM..



$ git clone https://github.com/cryptodev-linux/cryptodev-linux.git

$ cd cryptodev-linux

$ make; make install

$ cd tests

$ ./cipher -

requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver virtio_crypto_aes_cbc

AES Test passed

requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver virtio_crypto_aes_cbc

requested cipher CRYPTO_AES_CBC, got cbc(aes) with driver virtio_crypto_aes_cbc

Test passed


Regards,
Harish

[dpdk-dev] [PATCH v4 0/8] vhost: intdroduce vhost user crypto backend

[Fan Zhang]

This patchset adds crypto backend suppport to vhost library
including a proof-of-concept sample application. The implementation
follows the virtio-crypto specification and have been tested
with qemu 2.11.50 (with several patches applied, detailed later)
with Fedora 24 running in the frontend.

The vhost_crypto library acts as a "bridge" method that translate
the virtio-crypto crypto requests to DPDK crypto operations, so it
is purely software implementation. However it does require the user
to provide the DPDK Cryptodev ID so it knows how to handle the
virtio-crypto session creation and deletion mesages.

Currently the implementation supports AES-CBC-128 and HMAC-SHA1
cipher only/chaining modes and does not support sessionless mode
yet. The guest can use standard virtio-crypto driver to set up
session and sends encryption/decryption requests to backend. The
vhost-crypto sample application provided in this patchset will
do the actual crypto work.

The following steps are involved to enable vhost-crypto support.

In the host:
1. Download the qemu source code.

2. Recompile your qemu with vhost-crypto option enabled.

3. Apply this patchset to latest DPDK code and recompile DPDK.

4. Compile and run vhost-crypto sample application.

./examples/vhost_crypto/build/vhost-crypto -l 11,12 -w 0000:86:01.0 \
 --socket-mem 2048,2048

Where 0000:86:01.0 is the QAT PCI address. You may use AES-NI-MB if it is
not available. The sample application requires 2 lcores: 1 master and 1
worker. The application will create a UNIX socket file
/tmp/vhost_crypto1.socket.

5. Start your qemu application. Here is my command:

qemu/x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -cpu host \
-smp 2 -m 1G -hda ~/path-to-your/image.qcow \
-object memory-backend-file,id=mem,size=1G,mem-path=/dev/hugepages,share=on \
-mem-prealloc -numa node,memdev=mem -chardev \
socket,id=charcrypto0,path=/tmp/vhost_crypto1.socket \
-object cryptodev-vhost-user,id=cryptodev0,chardev=charcrypto0 \
-device virtio-crypto-pci,id=crypto0,cryptodev=cryptodev0

6. Once guest is booted. The Linux virtio_crypto kernel module is loaded by
default. You shall see the following logs in your demsg:

[   17.611044] virtio_crypto: loading out-of-tree module taints kernel.
[   17.611083] virtio_crypto: module verification failed: signature and/or ...
[   17.611723] virtio_crypto virtio0: max_queues: 1, max_cipher_key_len: ...
[   17.612156] virtio_crypto virtio0: will run requests pump with realtime ...
[   18.376100] virtio_crypto virtio0: Accelerator is ready

The virtio_crypto driver in the guest is now up and running.

7. The rest steps can be as same as the Testing section in
https://wiki.qemu.org/Features/VirtioCrypto

8. It is possible to use DPDK Virtio Crypto PMD
(https://dpdk.org/dev/patchwork/patch/36471/) in the guest to work with
this patchset to achieve optimal performance.

v4:
- Changed external vhost backend ops register API.
- Fixed a bug.

v3:
- Changed external vhost backend private data and message handling
- Added experimental tag to rte_vhost_crypto_set_zero_copy()

v2:
- Moved vhost_crypto_data_req data from crypto op to source mbuf.
- Removed ZERO-COPY flag from config option and make it run-timely changeable.
- Guest-polling mode possible.
- Simplified vring descriptor access procedure.
- Work with both LKCF and DPDK Virtio-Crypto PMD guest drivers.

Fan Zhang (8):
  lib/librte_vhost: add external backend support
  lib/librte_vhost: add virtio-crypto user message structure
  lib/librte_vhost: add session message handler
  lib/librte_vhost: add request handler
  lib/librte_vhost: add public function implementation
  lib/librte_vhost: update makefile
  examples/vhost_crypto: add vhost crypto sample application
  doc: update for vhost crypto support

 doc/guides/prog_guide/vhost_lib.rst       |   21 +
 doc/guides/rel_notes/release_18_05.rst    |   10 +
 doc/guides/sample_app_ug/index.rst        |    1 +
 doc/guides/sample_app_ug/vhost_crypto.rst |   84 ++
 examples/vhost_crypto/Makefile            |   32 +
 examples/vhost_crypto/main.c              |  541 ++++++++++++
 examples/vhost_crypto/meson.build         |   14 +
 lib/librte_vhost/Makefile                 |    6 +-
 lib/librte_vhost/meson.build              |    8 +-
 lib/librte_vhost/rte_vhost.h              |   64 +-
 lib/librte_vhost/rte_vhost_crypto.h       |  125 +++
 lib/librte_vhost/rte_vhost_version.map    |   11 +
 lib/librte_vhost/vhost.c                  |   17 +-
 lib/librte_vhost/vhost.h                  |    8 +-
 lib/librte_vhost/vhost_crypto.c           | 1285 +++++++++++++++++++++++++++++
 lib/librte_vhost/vhost_user.c             |   33 +-
 lib/librte_vhost/vhost_user.h             |   36 +-
 17 files changed, 2281 insertions(+), 15 deletions(-)
 create mode 100644 doc/guides/sample_app_ug/vhost_crypto.rst
 create mode 100644 examples/vhost_crypto/Makefile
 create mode 100644 examples/vhost_crypto/main.c
 create mode 100644 examples/vhost_crypto/meson.build
 create mode 100644 lib/librte_vhost/rte_vhost_crypto.h
 create mode 100644 lib/librte_vhost/vhost_crypto.c

-- 
2.7.4