[dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode)

[dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode)

[Chinmaya Dwibedy]

Hi All,



I have created two VM instances  (Fedora release 20 (Heisenbug) using
openstack. The kernel version of the VM : 3.12.9-301.fc20.x86_64. The
compute node (real system has CentOS 7.2, kernel version:
3.10.0-327.18.2.el7.x86_64)
has two Intel QAT devices.



[root@localhost ~(keystone_admin)]# lspci -vd:0435

83:00.0 Co-processor: Intel Corporation DH895XCC Series QAT

        Subsystem: Intel Corporation Device 35c5

        Physical Slot: 0-1

        Flags: bus master, fast devsel, latency 0, IRQ 35

        Memory at 387fffd00000 (64-bit, prefetchable) [size=512K]

        Memory at c8200000 (64-bit, non-prefetchable) [size=256K]

        Memory at c8240000 (64-bit, non-prefetchable) [size=256K]

        Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+

        Capabilities: [60] MSI-X: Enable+ Count=33 Masked-

        Capabilities: [6c] Power Management version 3

        Capabilities: [74] Express Endpoint, MSI 00

        Capabilities: [100] Advanced Error Reporting

        Capabilities: [138] Alternative Routing-ID Interpretation (ARI)

        Capabilities: [140] Single Root I/O Virtualization (SR-IOV)

        Kernel driver in use: vfio-pci



88:00.0 Co-processor: Intel Corporation DH895XCC Series QAT

        Subsystem: Intel Corporation Device 35c5

        Physical Slot: 0-2

        Flags: bus master, fast devsel, latency 0, IRQ 39

        Memory at 387fffe00000 (64-bit, prefetchable) [size=512K]

        Memory at c8000000 (64-bit, non-prefetchable) [size=256K]

        Memory at c8040000 (64-bit, non-prefetchable) [size=256K]

        Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+

        Capabilities: [60] MSI-X: Enable+ Count=33 Masked-

        Capabilities: [6c] Power Management version 3

        Capabilities: [74] Express Endpoint, MSI 00

        Capabilities: [100] Advanced Error Reporting

        Capabilities: [138] Alternative Routing-ID Interpretation (ARI)

        Capabilities: [140] Single Root I/O Virtualization (SR-IOV)

        Kernel driver in use: vfio-pci



[root@localhost ~(keystone_admin)]#



I have configured both the QAT devices to be pass-through devices.  So
that, each VM will have exclusive access to an Intel QAT card. On VMs, I
configured, build  and   installed latest Intel driver (qatmux.l.2.6.0-60)
(downloaded from
https://01.org/packet-processing/intel%C2%AE-quickassist-technology-drivers-and-patches).
Started
the driver and checked via #service qat_service status and found that, it
detects 1 acceleration device(s) in the system.


[root@vpn-c openssl-async]# service qat_service status

There is 1 acceleration device(s) in the system:

 icp_dev0 - type=dh895xcc, inst_id=0, node_id=0,  bdf=00:05:0, #accel=6,
#engines=12, state=up

[root@vpn-c openssl-async]#



Also the output lspci on VM as follows

[root@vpn-c openssl-async]# lspci -nn | grep 0435

00:05.0 Co-processor [0b40]: Intel Corporation Coleto Creek PCIe Endpoint
[8086:0435]

[root@vpn-c openssl-async]#



I have configured “intel_iommu=off” in boot arguments (VMs).


[root@vpn-c dpdk-2.2.0]# cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-3.12.9-301.fc20.x86_64
root=UUID=ab47cbc9-68ee-403c-96a5-184e68238e65 ro console=ttyS0,115200n8
intel_iommu=off

[root@vpn-c dpdk-2.2.0]#


I run the  ipsec-secgw sample application (
http://dpdk.org/browse/dpdk/tree/examples/ipsec-secgw) on VM with -cdev QAT
(e.g., #./examples/ipsec-secgw/build/ipsec-secgw  -l 0,1 -n 4  -- -p 0x3 -P
--config="(0,0,0), (1,0,1)" --cdev QAT --ep0) . I found that,
rte_cryptodev_count_devtype() returns zero (i.e., no QAT crypto device
found).


Can anyone please suggest what might be the issue and the way to resolve
this ?  Also kindly let me know  using QAT device (passed through mode) by
DPDK application is possible or not. Thank you in advance for your time and
help. please feel free to let me know if additional information is needed.


Here go the console logs.



EAL: Master lcore 0 is ready (tid=e110a940;cpuset=[0])

EAL: lcore 1 is ready (tid=2f0b7700;cpuset=[1])

EAL: PCI device 0000:00:03.0 on NUMA socket -1

EAL:   probe driver: 1af4:1000 rte_virtio_pmd

PMD: parse_sysfs_value(): parse_sysfs_value(): cannot open sysfs value
/sys/bus/pci/devices/0000:00:03.0/uio/uio1/portio/port0/size

PMD: virtio_resource_init_by_uio(): virtio_resource_init_by_uio(): cannot
parse size

PMD: virtio_resource_init_by_ioports(): PCI Port IO found start=0xc060 with
size=0x20

PMD: virtio_negotiate_features(): guest_features before negotiate = cf8020

PMD: virtio_negotiate_features(): host_features before negotiate = 799fffe3

PMD: virtio_negotiate_features(): features after negotiate = 8f8020

PMD: eth_virtio_dev_init(): PORT MAC: FA:16:3E:BB:8A:DC

PMD: eth_virtio_dev_init(): VIRTIO_NET_F_MQ is not supported

PMD: virtio_dev_cq_queue_setup():  >>

PMD: virtio_dev_queue_setup(): selecting queue: 2

PMD: virtio_dev_queue_setup(): vq_size: 64 nb_desc:0

PMD: virtio_dev_queue_setup(): vring_size: 4612, rounded_vring_size: 8192

PMD: virtio_dev_queue_setup(): vq->vq_ring_mem:      0x219143000

PMD: virtio_dev_queue_setup(): vq->vq_ring_virt_mem: 0x7fcf30f43000

PMD: eth_virtio_dev_init(): config->max_virtqueue_pairs=1

PMD: eth_virtio_dev_init(): config->status=1

PMD: eth_virtio_dev_init(): PORT MAC: FA:16:3E:BB:8A:DC

PMD: eth_virtio_dev_init(): hw->max_rx_queues=1   hw->max_tx_queues=1

PMD: eth_virtio_dev_init(): port 0 vendorID=0x1af4 deviceID=0x1000

PMD: virtio_dev_vring_start():  >>

EAL: PCI device 0000:00:06.0 on NUMA socket -1

EAL:   probe driver: 8086:10fb rte_ixgbe_pmd

EAL:   PCI memory mapped at 0x7fd0dee00000

EAL:   PCI memory mapped at 0x7fd0def00000

PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 15, SFP+: 5

PMD: eth_ixgbe_dev_init(): port 1 vendorID=0x8086 deviceID=0x10fb

Promiscuous mode selected

endpoint 0

Found 0 QAT devices

PANIC in cryptodevs_init():

Not enough QAT devices detected, need 2 (1 per core), found 0

5: [./examples/ipsec-secgw/build/ipsec-secgw() [0x42ebf5]]

4: [/lib64/libc.so.6(__libc_start_main+0xf5) [0x7fd0dfe49d65]]

3: [./examples/ipsec-secgw/build/ipsec-secgw(main+0xd75) [0x42d695]]

2: [./examples/ipsec-secgw/build/ipsec-secgw(__rte_panic+0xc9) [0x42733e]]

1: [./examples/ipsec-secgw/build/ipsec-secgw(rte_dump_stack+0x1a)
[0x49561a]]

./run.sh: line 5:  8202 Aborted                 (core dumped)
./examples/ipsec-secgw/build/ipsec-secgw -l 0,1 -n 4 -- -p 0x3 -P
--config="(0,0,0), (1,0,1)" --cdev QAT --ep0



Regards,

Chinmaya

Re: [dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode)

[Sergio Gonzalez Monroy]

On 15/06/2016 13:01, Chinmaya Dwibedy wrote:
> Also the output lspci on VM as follows
>
> [root@vpn-c openssl-async]# lspci -nn | grep 0435
>
> 00:05.0 Co-processor [0b40]: Intel Corporation Coleto Creek PCIe Endpoint
> [8086:0435]
>

This is the PF, not the VF.

> I run the  ipsec-secgw sample application (
> http://dpdk.org/browse/dpdk/tree/examples/ipsec-secgw) on VM with -cdev QAT
> (e.g., #./examples/ipsec-secgw/build/ipsec-secgw  -l 0,1 -n 4  -- -p 0x3 -P
> --config="(0,0,0), (1,0,1)" --cdev QAT --ep0) . I found that,
> rte_cryptodev_count_devtype() returns zero (i.e., no QAT crypto device
> found).
>
>
> Can anyone please suggest what might be the issue and the way to resolve
> this ?  Also kindly let me know  using QAT device (passed through mode) by
> DPDK application is possible or not. Thank you in advance for your time and
> help. please feel free to let me know if additional information is needed.
>

I reckon that your issue is that you are trying to use the PF (not 
supported by DPDK)
instead of the VF.

 From 
http://dpdk.readthedocs.io/en/v2.2.0/cryptodevs/qat.html#installation-using-01-org-qat-driver:

"You can use|cat/proc/icp_dh895xcc_dev0/version|to confirm the driver is 
correctly installed.
You can use|lspci-d:443|to confirm the bdf of the 32 VF devices are 
available per|DH895xCC|device."

You should pass-through one of the VFs.

Sergio

Re: [dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode)

[Chinmaya Dwibedy]

Hi All,


@Sergio:  Thank you for your valuable suggestion.


I passed through one of VFs and it detected the QAT device in VM. I just
sent one UDP datagram which has to be encapsulated using H/W crypto device
(i.e., Intel QAT). I run the ipsec-segw application with following
arguments.  ./build/ipsec-secgw  -l 0 -n 4  -- -p 0x3 -P
--config="(0,0,0),(1,0,0)" --cdev QAT --ep0. Found that, the
rte_crypto_enqueue_burst() function returns one. It means it could able to
suceesfully place packet on the queue “queue_id” of the QAT device
designated by its “dev_id”*.  But The rte_crypto_dequeue_burst() function
returns zero. It means it cannot dequeue the packet and that packet might
not have been processed by QAT device.


Please note that, I have tested the same application with AESNI device and
did not encounter any such issue. Furthermore the
rte_crypto_dequeue_burst() API does not provide any error notification.
Can anyone please suggest what might be the issue and is there any way to
debug further?

Here are the configuration options and console logs .



[root@vpn-s ipsec-secgw]# lspci -nn | grep 0443

00:06.0 Co-processor [0b40]: Intel Corporation Device [8086:0443]

[root@vpn-s ipsec-secgw]#


I have configured the followings in config file


CONFIG_RTE_LIBRTE_PMD_QAT=y

CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_INIT=y

CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_TX=y

CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_RX=y

CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_DRIVER=y



Console log while running ipsec-segw sample application



PMD: crypto_qat_dev_init():  >>

PMD: crypto_qat_dev_init(): Found crypto device at 00:06.0

Promiscuous mode selected

endpoint 0

Found 1 QAT devices

PMD: qat_dev_info_get():  >>

Initialising QAT device 0

CRYPTODEV: rte_cryptodev_queue_pairs_config() line 467: Setup 1 queues
pairs on device 0

PMD: qat_dev_info_get():  >>

PMD: qat_crypto_sym_session_init():  >>

PMD: qat_crypto_sym_session_init():  >>

PMD: qat_crypto_sym_session_init():  >>

PMD: qat_crypto_sym_session_init():  >>

PMD: qat_crypto_sym_session_init():  >>

PMD: qat_crypto_sym_session_init():  >>

PMD: qat_crypto_sym_session_init():  >>

PMD: qat_crypto_sym_session_init():  >>

PMD: qat_crypto_sym_session_init():  >>

PMD: qat_crypto_sym_session_init():  >>

PMD: qat_crypto_sym_session_init():  >>

PMD: qat_crypto_sym_session_init():  >>

PMD: qat_crypto_sym_session_init():  >>

PMD: qat_crypto_sym_session_init():  >>

CRYPTODEV: rte_crypto_session_pool_create() line 1031: cdev_0_sess_mp
mempool created!

PMD: qat_crypto_sym_qp_setup():  >>

PMD: qat_tx_queue_create():  >>

PMD: qat_tx_queue_create(): TX ring for 2048 msgs: qp_id 0, bundle 0, ring 2

PMD: qat_queue_create():  >>

PMD: queue_dma_zone_reserve():  >>

PMD: queue_dma_zone_reserve(): Allocate memzone for
rte_qat_pmd_qp_mem_0_0_2, size 262144 on socket 0

PMD: qat_qp_check_queue_alignment():  >>

PMD: adf_verify_queue_size():  >>

PMD: qat_queue_create(): RING size in CSR: 12, in bytes 262144, nb msgs
2048, msg_size 128, max_inflights 2047 modulo 18

PMD: qat_rx_queue_create():  >>

PMD: qat_rx_queue_create(): RX ring for 2048 msgs: qp id 0, bundle 0, ring
10

PMD: qat_queue_create():  >>

PMD: queue_dma_zone_reserve():  >>

PMD: queue_dma_zone_reserve(): Allocate memzone for
rte_qat_pmd_qp_mem_0_0_10, size 65536 on socket 0

PMD: qat_qp_check_queue_alignment():  >>

PMD: adf_verify_queue_size():  >>

PMD: qat_queue_create(): RING size in CSR: 10, in bytes 65536, nb msgs
2048, msg_size 32, max_inflights 2047 modulo 16

PMD: adf_configure_queues():  >>

PMD: adf_queue_arb_enable():  >>

PMD: qat_dev_info_get():  >>

Creating SA context with 64 maximum entries

Creating SA context with 64 maximum entries

PMD: qat_crypto_sym_configure_session():  >>

PMD: qat_alg_aead_session_create_content_desc():  >>

PMD: qat_alg_do_precomputes():  >>

PMD: partial_hash_compute():  >>

PMD: partial_hash_compute():  >>

PMD: qat_alg_init_common_hdr():  >>

PMD: qat_crypto_sym_configure_session():  >>

PMD: qat_alg_aead_session_create_content_desc():  >>

PMD: qat_alg_do_precomputes():  >>

PMD: partial_hash_compute():  >>

PMD: partial_hash_compute():  >>

PMD: qat_alg_init_common_hdr():  >>

PMD: qat_crypto_sym_configure_session():  >>

PMD: qat_alg_aead_session_create_content_desc():  >>

PMD: partial_hash_compute():  >>

PMD: qat_alg_init_common_hdr():  >>

Creating SP context with 1000 max rules

IPv4 sp_ipv4_in_0 entries [4]:

        1:0.0.0.0/0 30.1.1.0/24 0 : 65535 0 : 65535 0x0/0x0 0x1-0x1-0x5

        2:0.0.0.0/0 192.168.116.0/24 0 : 65535 0 : 65535 0x0/0x0 0x1-0x2-0x6

        3:0.0.0.0/0 192.168.117.0/24 0 : 65535 0 : 65535 0x0/0x0 0x1-0x3-0x7

        4:0.0.0.0/0 192.168.118.0/24 0 : 65535 0 : 65535 0x0/0x0 0x1-0x4-0x8

ACL: Gen phase for ACL "sp_ipv4_in_0":

runtime memory footprint on socket 0:

single nodes/bytes used: 4/32

quad nodes/vectors/bytes used: 5/19/152

DFA nodes/group64/bytes used: 1/4/4104

match nodes/bytes used: 4/512

total: 7008 bytes

max limit: 18446744073709551615 bytes

ACL: Build phase for ACL "sp_ipv4_in_0":

node limit for tree split: 2048

nodes created: 14

memory consumed: 8388615

ACL: trie 0: number of rules: 4, indexes: 2

acl context <sp_ipv4_in_0>@0x7faefcb88500

  socket_id=0

  alg=3

  max_rules=1000

  rule_size=96

  num_rules=4

  num_categories=1

  num_tries=1

Creating SP context with 1000 max rules

IPv4 sp_ipv4_out_0 entries [4]:

        1:0.0.0.0/0 30.1.1.0/24 0 : 65535 0 : 65535 0x0/0x0 0x1-0x1-0x5

        2:0.0.0.0/0 192.168.4.0/24 0 : 65535 0 : 65535 0x0/0x0 0x1-0x2-0x6

        3:0.0.0.0/0 192.168.5.0/24 0 : 65535 0 : 65535 0x0/0x0 0x1-0x3-0x7

        4:0.0.0.0/0 192.168.6.0/24 0 : 65535 0 : 65535 0x0/0x0 0x1-0x4-0x8

ACL: Gen phase for ACL "sp_ipv4_out_0":

runtime memory footprint on socket 0:

single nodes/bytes used: 4/32

quad nodes/vectors/bytes used: 5/19/152

DFA nodes/group64/bytes used: 1/4/4104

match nodes/bytes used: 4/512

total: 7008 bytes

max limit: 18446744073709551615 bytes

ACL: Build phase for ACL "sp_ipv4_out_0":

node limit for tree split: 2048

-------------------------------------



Port 0 Link Up - speed 10000 Mbps - full-duplex

Port 1 Link Up - speed 10000 Mbps - full-duplex

IPSEC: entering main loop on lcore 0

IPSEC:  -- lcoreid=0 portid=0 rxqueueid=0

IPSEC:  -- lcoreid=0 portid=1 rxqueueid=0

IPSEC_ESP: pktlen 110, esp_len 24, digest_len 12, payload_len 110,
pad_payload_len 112, block_sz 16, pad_len 2

IPSEC_ESP: pktlen 168

qat_req: at [0x7faefcb40000], len=128

00000000: 00 02 04 80 24 00 00 00 50 B4 42 B1 00 00 00 00 | ....$...P.B.....

00000010: 00 00 28 00 00 00 00 00 80 FA A5 75 AF 7F 00 00 | ..(........u....

00000020: 12 FC 25 AA 00 00 00 00 12 FC 25 AA 00 00 00 00 | ..%.......%.....

00000030: A8 00 00 00 A8 00 00 00 2C 00 00 00 70 00 00 00 | ........,...p...

00000040: 85 CD 08 06 74 43 34 53 29 A0 2E 16 B5 0D FC 6A | ....tC4S)......j

00000050: 14 00 00 00 88 00 00 00 00 00 00 00 00 00 00 00 | ................

00000060: AE FC 25 AA 00 00 00 00 00 00 00 0C 02 02 00 21 | ..%............!

00000070: 00 00 10 42 00 00 0C 0C 00 18 15 18 00 00 00 00 | ...B............

IPSEC: Cryptodev 0 queue 0: enqueued 1 packets (out of 1)

IPSEC: Cryptodev 0 queue 0: dequeued 0 packets (out of 32)



Regards,

Chinmaya

On Wed, Jun 15, 2016 at 6:05 PM, Sergio Gonzalez Monroy <
sergio.gonzalez.monroy@intel.com> wrote:

> On 15/06/2016 13:01, Chinmaya Dwibedy wrote:
>
>> Also the output lspci on VM as follows
>>
>> [root@vpn-c openssl-async]# lspci -nn | grep 0435
>>
>> 00:05.0 Co-processor [0b40]: Intel Corporation Coleto Creek PCIe Endpoint
>> [8086:0435]
>>
>>
> This is the PF, not the VF.
>
> I run the  ipsec-secgw sample application (
>> http://dpdk.org/browse/dpdk/tree/examples/ipsec-secgw) on VM with -cdev
>> QAT
>> (e.g., #./examples/ipsec-secgw/build/ipsec-secgw  -l 0,1 -n 4  -- -p 0x3
>> -P
>> --config="(0,0,0), (1,0,1)" --cdev QAT --ep0) . I found that,
>> rte_cryptodev_count_devtype() returns zero (i.e., no QAT crypto device
>> found).
>>
>>
>> Can anyone please suggest what might be the issue and the way to resolve
>> this ?  Also kindly let me know  using QAT device (passed through mode) by
>> DPDK application is possible or not. Thank you in advance for your time
>> and
>> help. please feel free to let me know if additional information is needed.
>>
>>
> I reckon that your issue is that you are trying to use the PF (not
> supported by DPDK)
> instead of the VF.
>
> From
> http://dpdk.readthedocs.io/en/v2.2.0/cryptodevs/qat.html#installation-using-01-org-qat-driver
> :
>
> "You can use|cat/proc/icp_dh895xcc_dev0/version|to confirm the driver is
> correctly installed.
> You can use|lspci-d:443|to confirm the bdf of the 32 VF devices are
> available per|DH895xCC|device."
>
> You should pass-through one of the VFs.
>
> Sergio
>

Re: [dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode)

[Sergio Gonzalez Monroy]

On 20/06/2016 08:33, Chinmaya Dwibedy wrote:
>
> Hi All,
>
>
> @Sergio:  Thank you for your valuable suggestion.
>
>
> I passed through one of VFs and it detected the QAT device in VM. I 
> just sent one UDP datagram which has to be encapsulated using H/W 
> crypto device (i.e., Intel QAT). I run the ipsec-segw application with 
> following arguments.  ./build/ipsec-secgw  -l 0 -n 4 -- -p 0x3 -P 
> --config="(0,0,0),(1,0,0)" --cdev QAT --ep0. Found that, the 
> rte_crypto_enqueue_burst() function returns one. It means it could 
> able to suceesfully place packet on the queue “queue_id” of the QAT 
> device designated by its “dev_id”*.  But The 
> rte_crypto_dequeue_burst() functionreturns zero. It means it cannot 
> dequeue the packet and that packet might not have been processed by 
> QAT device.
>
>

It's expected when using crypto HW offload that you might not dequeue 
the same amount of crypto ops you just previously enqueued, it's 
asynchronous.

> Please note that, I have tested the same application with AESNI device 
> and did not encounter any such issue. Furthermore the 
> rte_crypto_dequeue_burst() API does not provide any error 
> notification.  Can anyone please suggest what might be the issue and 
> is there any way to debug further?
>

Can you give a bit more details about what the issue is when using HW vs 
using SW crypto?

Are using the same config (SP/SA/RT) when using HW and SW?

Sergio

Re: [dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode)

[Chinmaya Dwibedy]

Hi Sergio,


Agreed.  We might not dequeue the same amount of crypto ops we just
previously enqueued, it's asynchronous. But in this case, I have sent just
one UDP packet. So there will be one crypto ops. Right?  Also I put a sleep
(50) after the rte_crypto_enqueue_burst() function in ipsec_processing()
(ipsec.c) , so as to allow more time  ( for QAT device) for processing.
Still getting the same result i.e., the rte_crypto_dequeue_burst () function
returns zero.


In case of S/W crypto device (i.e., AESNI),   the VM gets inbound UDP
packets on Port 1/eth1, encapsulates (after consulting its SPD) in an IPsec
ESP packet and sends to its peer through Port 0/eth0 interface.


Yes, the security policy, security association and Routing
entries/configurations are exactly same. Please feel free to let me know if
you need additional information.



Here goes my configuration file.



[root@vpn-s ipsec-secgw]# cat
../../config/defconfig_x86_64-native-linuxapp-gcc

#include "common_linuxapp"

CONFIG_RTE_MACHINE="native"

CONFIG_RTE_ARCH="x86_64"

CONFIG_RTE_ARCH_X86_64=y

CONFIG_RTE_ARCH_64=y



CONFIG_RTE_TOOLCHAIN="gcc"

CONFIG_RTE_TOOLCHAIN_GCC=y

CONFIG_RTE_LIBRTE_PMD_AESNI_MB=y

CONFIG_RTE_LIBRTE_CRYPTODEV_DEBUG=y

CONFIG_RTE_LIBRTE_VIRTIO_PMD=y

CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=y

CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=y

CONFIG_RTE_LIBRTE_PMD_QAT=y

CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_INIT=y

CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_TX=y

CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_RX=y

CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_DRIVER=y

[root@vpn-s ipsec-secgw]#



Regards,
Chinmaya

On Mon, Jun 20, 2016 at 1:36 PM, Sergio Gonzalez Monroy <
sergio.gonzalez.monroy@intel.com> wrote:

> On 20/06/2016 08:33, Chinmaya Dwibedy wrote:
>
>
> Hi All,
>
>
> @Sergio:  Thank you for your valuable suggestion.
>
>
> I passed through one of VFs and it detected the QAT device in VM. I just
> sent one UDP datagram which has to be encapsulated using H/W crypto device
> (i.e., Intel QAT). I run the ipsec-segw application with following
> arguments.  ./build/ipsec-secgw  -l 0 -n 4  -- -p 0x3 -P
> --config="(0,0,0),(1,0,0)" --cdev QAT --ep0. Found that, the
> rte_crypto_enqueue_burst() function returns one. It means it could able to
> suceesfully place packet on the queue “queue_id” of the QAT device
> designated by its “dev_id”*.  But The rte_crypto_dequeue_burst() function
> returns zero. It means it cannot dequeue the packet and that packet might
> not have been processed by QAT device.
>
>
>
> It's expected when using crypto HW offload that you might not dequeue the
> same amount of crypto ops you just previously enqueued, it's asynchronous.
>
> Please note that, I have tested the same application with AESNI device and
> did not encounter any such issue. Furthermore the
> rte_crypto_dequeue_burst() API does not provide any error notification.
> Can anyone please suggest what might be the issue and is there any way to
> debug further?
>
>
> Can you give a bit more details about what the issue is when using HW vs
> using SW crypto?
>
> Are using the same config (SP/SA/RT) when using HW and SW?
>
> Sergio
>
>

Re: [dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode)

[Sergio Gonzalez Monroy]

On 20/06/2016 10:08, Chinmaya Dwibedy wrote:
>
> Hi Sergio,
>
>
> Agreed.  We might not dequeue the same amount of crypto ops we just 
> previously enqueued, it's asynchronous. But in this case, I have sent 
> just one UDP packet. So there will be one crypto ops. Right?  Also I 
> put a sleep (50) after the rte_crypto_enqueue_burst() function in 
> ipsec_processing() (ipsec.c) , so as to allow more time  ( for QAT 
> device) for processing. Still getting the same result i.e., the 
> rte_crypto_dequeue_burst () functionreturns zero.
>
>
> In case of S/W crypto device (i.e., AESNI),   the VM gets inbound UDP 
> packets on Port 1/eth1, encapsulates (after consulting its SPD) in an 
> IPsec ESP packet and sends to its peer through Port 0/eth0 interface.
>
>
> Yes, the security policy, security association and Routing 
> entries/configurations are exactly same. Please feel free to let me 
> know if you need additional information.
>

Could you try to run 'app/test' application then run 
'cryptodev_qat_autotest' ? That is a functional test for cryptodev QAT PMD.

Sergio

Re: [dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode)

[Chinmaya Dwibedy]

Hi Sergio,


As suggested by you, run ‘app/test' application and then run
'cryptodev_qat_autotest' . Found the same issue i.e., It halts in
rte_cryptodev_dequeue_burst () (while loop) in process_crypto_request ()
function (app/test/test_cryptodev.c).

Debugged the qat_crypto_pkt_rx_burst() (in drivers/crypto/qat/qat_crypto.c)
and found that , the value of resp_msg is 0x7F7F7F7F (i.e.,
ADF_RING_EMPTY_SIG) . Thus it is not getting crypto packet (i.e., rte_mbuf)
from the RX queue.  But I find that, in qat_crypto_pkt_tx_burst(),
the  qat_alg_write_mbuf_entry() is being called. The
qat_alg_write_mbuf_entry () prints qat_req  using rte_hexdump () and
returns zero.


One question, the base_addr + tail of tx_q (in qat_crypto_pkt_tx_burst())
should be same as base_addr + head (in qat_crypto_pkt_rx_burst()). Right ?
Please feel free to correct me if I am wrong.


Also can you please provide some pointers for further debugging? Would it
be configuration issue? Note that, I am using VF in VM.


Thank you once again for your prompt response and great support.


Regards,

Chinmaya

On Mon, Jun 20, 2016 at 3:04 PM, Sergio Gonzalez Monroy <
sergio.gonzalez.monroy@intel.com> wrote:

> On 20/06/2016 10:08, Chinmaya Dwibedy wrote:
>
>>
>> Hi Sergio,
>>
>>
>> Agreed.  We might not dequeue the same amount of crypto ops we just
>> previously enqueued, it's asynchronous. But in this case, I have sent just
>> one UDP packet. So there will be one crypto ops. Right?  Also I put a sleep
>> (50) after the rte_crypto_enqueue_burst() function in ipsec_processing()
>> (ipsec.c) , so as to allow more time  ( for QAT device) for processing.
>> Still getting the same result i.e., the rte_crypto_dequeue_burst ()
>> functionreturns zero.
>>
>>
>> In case of S/W crypto device (i.e., AESNI),   the VM gets inbound UDP
>> packets on Port 1/eth1, encapsulates (after consulting its SPD) in an IPsec
>> ESP packet and sends to its peer through Port 0/eth0 interface.
>>
>>
>> Yes, the security policy, security association and Routing
>> entries/configurations are exactly same. Please feel free to let me know if
>> you need additional information.
>>
>>
> Could you try to run 'app/test' application then run
> 'cryptodev_qat_autotest' ? That is a functional test for cryptodev QAT PMD.
>
> Sergio
>
>

Re: [dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode)

[Sergio Gonzalez Monroy]

On 21/06/2016 08:26, Chinmaya Dwibedy wrote:
>
> Hi Sergio,
>
>
> As suggested by you, run ‘app/test' application and then run 
> 'cryptodev_qat_autotest' . Found the same issue i.e., It halts in 
> rte_cryptodev_dequeue_burst () (while loop) in process_crypto_request 
> () function (app/test/test_cryptodev.c).
>
> Debugged the qat_crypto_pkt_rx_burst() (in 
> drivers/crypto/qat/qat_crypto.c) and found that , the value of 
> resp_msg is 0x7F7F7F7F (i.e., ADF_RING_EMPTY_SIG) . Thus it is not 
> getting crypto packet (i.e., rte_mbuf) from the RX queue.  But I find 
> that, in qat_crypto_pkt_tx_burst(), the  qat_alg_write_mbuf_entry() is 
> being called. The qat_alg_write_mbuf_entry () prints qat_req  using 
> rte_hexdump () and returns zero.
>
>
> One question, the base_addr + tail of tx_q (in 
> qat_crypto_pkt_tx_burst()) should be same as base_addr + head (in 
> qat_crypto_pkt_rx_burst()). Right ? Please feel free to correct me if 
> I am wrong.
>
>
> Also can you please provide some pointers for further debugging? Would 
> it be configuration issue? Note that, I am using VF in VM.
>

In theory  you should be able to run ipsec-secgw sample app with QAT VF 
as long as the setup is right.
That the autotest fails would likely point to some possible issues with 
the setup.

Have you tried to run the autotest in the host?

Sergio

>
> Thank you once again for your prompt response and great support.
>
>
> Regards,
>
> Chinmaya
>
>
> On Mon, Jun 20, 2016 at 3:04 PM, Sergio Gonzalez Monroy 
> <sergio.gonzalez.monroy@intel.com 
> <mailto:sergio.gonzalez.monroy@intel.com>> wrote:
>
>     On 20/06/2016 10:08, Chinmaya Dwibedy wrote:
>
>
>         Hi Sergio,
>
>
>         Agreed.  We might not dequeue the same amount of crypto ops we
>         just previously enqueued, it's asynchronous. But in this case,
>         I have sent just one UDP packet. So there will be one crypto
>         ops. Right?  Also I put a sleep (50) after the
>         rte_crypto_enqueue_burst() function in ipsec_processing()
>         (ipsec.c) , so as to allow more time  ( for QAT device) for
>         processing. Still getting the same result i.e., the
>         rte_crypto_dequeue_burst () functionreturns zero.
>
>
>         In case of S/W crypto device (i.e., AESNI),   the VM gets
>         inbound UDP packets on Port 1/eth1, encapsulates (after
>         consulting its SPD) in an IPsec ESP packet and sends to its
>         peer through Port 0/eth0 interface.
>
>
>         Yes, the security policy, security association and Routing
>         entries/configurations are exactly same. Please feel free to
>         let me know if you need additional information.
>
>
>     Could you try to run 'app/test' application then run
>     'cryptodev_qat_autotest' ? That is a functional test for cryptodev
>     QAT PMD.
>
>     Sergio
>
>

Re: [dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode)

[Chinmaya Dwibedy]

Hi Sergio,


Thank your for clarification. I am using dpdk-2.2.0. Here goes our
configuration on VM. Can you please have a look into the same and let me
know if anything is wrong. Will try to run the autotest, ipsec-segw on host
as per your suggestion.


[root@vpn-s ~]# uname -r

3.12.9-301.fc20.x86_64

[root@vpn-s ~]#


[root@vpn-s ~]# cat /etc/redhat-release

Fedora release 20 (Heisenbug)

[root@vpn-s ~]#




[root@vpn-s ~]# cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-3.12.9-301.fc20.x86_64
root=UUID=ab47cbc9-68ee-403c-96a5-184e68238e65 ro console=ttyS0,115200n8
iommu=pt intel_iommu=on

[root@vpn-s ~]#


[root@vpn-s QAT]# grep -i "iommu.*enabled" /var/log/messages

Jun 22 07:31:17 vpn-s kernel: [    0.000000] Intel-IOMMU: enabled


[root@vpn-s QAT]#

[root@vpn-s dpdk-2.2.0]# lspci -nn | grep 0443

00:06.0 Co-processor [0b40]: Intel Corporation Device [8086:0443]

[root@vpn-s dpdk-2.2.0]#


Installed Intel qat driver installation on VM (downloaded from
https://01.org/packet-processing/intel%C2%AE-quickassist-technology-drivers-and-patches)
and followed the

https://01.org/sites/default/files/page/330750-005_qat_gsg.pdf guide.
Note, used the following installation options i.e., ./installer.sh install
QAT1.6 guest. After this , it detects QAT VF.


[root@vpn-s QAT]# service qat_service status

There is 1 acceleration device(s) in the system:

 icp_dev0 - type=dh895xcc, inst_id=0, node_id=0,  bdf=00:06:0, #accel=1,
#engines=1, state=up

[root@vpn-s QAT]#


[root@vpn-s QAT]# lsmod | grep qa

icp_qa_al_vf          813442  1

[root@vpn-s QAT]#



The below are DPDK configuration options in
config/defconfig_x86_64-native-linuxapp-gcc.


include "common_linuxapp"



CONFIG_RTE_MACHINE="native"

CONFIG_RTE_ARCH="x86_64"

CONFIG_RTE_ARCH_X86_64=y

CONFIG_RTE_ARCH_64=y

CONFIG_RTE_TOOLCHAIN="gcc"

CONFIG_RTE_TOOLCHAIN_GCC=y

CONFIG_RTE_LIBRTE_PMD_AESNI_MB=y

CONFIG_RTE_LIBRTE_CRYPTODEV_DEBUG=y

CONFIG_RTE_LIBRTE_VIRTIO_PMD=y

CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=y

CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=y

CONFIG_RTE_LIBRTE_PMD_QAT=y

CONFIG_RTE_QAT_PMD_MAX_NB_SESSIONS=128

CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_INIT=y

CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_TX=y

CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_RX=y

CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_DRIVER=y



The below is my setup.sh


export RTE_SDK=/opt/dpdk-2.2.0

export RTE_TARGET=x86_64-native-linuxapp-gcc

export RTE_ARCH=x86_64

export AESNI_MULTI_BUFFER_LIB_PATH=/opt/code

make config T=x86_64-native-linuxapp-gcc

sed -ri 's,(PMD_PCAP=).*,\1y,' build/.config

make install T=x86_64-native-linuxapp-gcc

mkdir -p /mnt/huge

mount -t hugetlbfs nodev /mnt/huge

echo 2048 >
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages

sudo modprobe uio_pci_generic

sudo modprobe uio

rmmod igb_uio

insmod
./x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.ko

echo -n "0000:00:06.0" > /sys/bus/pci/devices/0000\:00\:06.0/driver/unbind

echo "8086 0443" > /sys/bus/pci/drivers/igb_uio/new_id

lspci -vvd:443

./tools/dpdk_nic_bind.py --bind=uio_pci_generic eth1

./tools/dpdk_nic_bind.py --bind=uio_pci_generic 00:03.0

./tools/dpdk_nic_bind.py -s


Also modified the cryptodevs_init() ( examples/ipsec-secgw/ipsec-secgw.c)
to rte_cryptodev_start(cdev_id); after rte_cryptodev_queue_pair_setup().
The purpose of calling this to start the transmit and the receive units of
the QAT device. It returns success. But still getting the same issue i.e.
rte_cryptodev_dequeue_burst () does not return crypto packet. Note that,
when I call rte_cryptodev_queue_pair_setup () with 3rd argument as NULL (in
which case default configuration will be used) , it crashes. Is it a known
issue?





Regards,

Chinmaya

On Tue, Jun 21, 2016 at 9:10 PM, Sergio Gonzalez Monroy <
sergio.gonzalez.monroy@intel.com> wrote:

> On 21/06/2016 08:26, Chinmaya Dwibedy wrote:
>
>>
>> Hi Sergio,
>>
>>
>> As suggested by you, run ‘app/test' application and then run
>> 'cryptodev_qat_autotest' . Found the same issue i.e., It halts in
>> rte_cryptodev_dequeue_burst () (while loop) in process_crypto_request ()
>> function (app/test/test_cryptodev.c).
>>
>> Debugged the qat_crypto_pkt_rx_burst() (in
>> drivers/crypto/qat/qat_crypto.c) and found that , the value of resp_msg is
>> 0x7F7F7F7F (i.e., ADF_RING_EMPTY_SIG) . Thus it is not getting crypto
>> packet (i.e., rte_mbuf) from the RX queue.  But I find that, in
>> qat_crypto_pkt_tx_burst(), the  qat_alg_write_mbuf_entry() is being called.
>> The qat_alg_write_mbuf_entry () prints qat_req  using rte_hexdump () and
>> returns zero.
>>
>>
>> One question, the base_addr + tail of tx_q (in qat_crypto_pkt_tx_burst())
>> should be same as base_addr + head (in qat_crypto_pkt_rx_burst()). Right ?
>> Please feel free to correct me if I am wrong.
>>
>>
>> Also can you please provide some pointers for further debugging? Would it
>> be configuration issue? Note that, I am using VF in VM.
>>
>>
> In theory  you should be able to run ipsec-secgw sample app with QAT VF as
> long as the setup is right.
> That the autotest fails would likely point to some possible issues with
> the setup.
>
> Have you tried to run the autotest in the host?
>
> Sergio
>
>
>> Thank you once again for your prompt response and great support.
>>
>>
>> Regards,
>>
>> Chinmaya
>>
>>
>> On Mon, Jun 20, 2016 at 3:04 PM, Sergio Gonzalez Monroy <
>> sergio.gonzalez.monroy@intel.com <mailto:sergio.gonzalez.monroy@intel.com>>
>> wrote:
>>
>>     On 20/06/2016 10:08, Chinmaya Dwibedy wrote:
>>
>>
>>         Hi Sergio,
>>
>>
>>         Agreed.  We might not dequeue the same amount of crypto ops we
>>         just previously enqueued, it's asynchronous. But in this case,
>>         I have sent just one UDP packet. So there will be one crypto
>>         ops. Right?  Also I put a sleep (50) after the
>>         rte_crypto_enqueue_burst() function in ipsec_processing()
>>         (ipsec.c) , so as to allow more time  ( for QAT device) for
>>         processing. Still getting the same result i.e., the
>>         rte_crypto_dequeue_burst () functionreturns zero.
>>
>>
>>         In case of S/W crypto device (i.e., AESNI),   the VM gets
>>         inbound UDP packets on Port 1/eth1, encapsulates (after
>>         consulting its SPD) in an IPsec ESP packet and sends to its
>>         peer through Port 0/eth0 interface.
>>
>>
>>         Yes, the security policy, security association and Routing
>>         entries/configurations are exactly same. Please feel free to
>>         let me know if you need additional information.
>>
>>
>>     Could you try to run 'app/test' application then run
>>     'cryptodev_qat_autotest' ? That is a functional test for cryptodev
>>     QAT PMD.
>>
>>     Sergio
>>
>>
>>
>

Re: [dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode)

[Chinmaya Dwibedy]

Hi All,

 On host, I installed dpdk-2.2.0 and then run
'cryptodev_qat_autotest’. I found this test case to be failed in
dequeue burst API ().Upon looking into the below system logs (#dmesg),
it seems, when this function is called, the QAT

hardware fails to write the crypto packet to the DMA buffer.  When I
passed intel_iommu=off command in boot line, it works. I think, when
we have no IOMMU, "physical" address space is accessed directly by
hardware, so code works.

The issue has been observed if the system runs with the kernel
parameter “intel_iommu=on”.



[root@stag48 ~]# dmesg

[  794.746134] dmar: DRHD: handling fault status reg 2

[  794.746141] dmar: DMAR:[DMA Read] Request device [83:01.0] fault addr
3398c000

DMAR:[fault reason 02] Present bit in context entry is clear

[  794.746158] icp_qa_al err: adf_dh895xcc_adf_isr_handleUncoInterrupt:
Uncorrectable Push/Pull Misc Error

memory status: No errors occurred - Transaction Id 0x0 - Error type reserved

Bus Operation Type Push - Id 0x80000

[  794.746167] Reset needed for device: icp_dev0

[  794.746167] Auto Reset disabled. Please reset device0

[root@stag48 ~]#



*Note: *The QAT VF (*83:01.0*) is bound to the DPDK UIO driver.



83:01.0 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Function

        Subsystem: Intel Corporation Device 0000

        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-

        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 0

        Region 0: [virtual] Memory at c8280000 (64-bit, non-prefetchable)
[size=4K]

        Region 2: [virtual] Memory at c82a0000 (64-bit, non-prefetchable)
[size=4K]

        Capabilities: [90] MSI: Enable- Count=1/1 Maskable+ 64bit+

                Address: 0000000000000000  Data: 0000

                Masking: 00000000  Pending: 00000000

        Kernel driver in use: igb_uio





Now testing the same on Guest and finding the same issue (i.e.,
failing on dequeue burst API ()). Upon looking into error message (in
dmesg) on host/hypervisor it says i.e., [28475.040313]

vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state. The
PCI device 88:04.6 is the QAT VF, being assigned to Guest. I am trying
to figure out what causes “buffer not found” issue .

Please suggest if anyone has an idea on it.



Here goes the output of dmesg



[27086.464014] vfio-pci 0000:88:04.6: enabling device (0000 -> 0002)

[27086.464040] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state

[27086.564610] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state

[27086.574052] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state

[27089.786623] kvm: zapping shadow pages for mmio generation wraparound

[27106.320042] vfio-pci 0000:88:04.6: irq 136 for MSI/MSI-X

[28474.784373] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state

[28475.040313] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state

[28493.271950] vfio-pci 0000:88:04.6: irq 136 for MSI/MSI-X





Regards,

Chinmaya







On Wed, Jun 22, 2016 at 1:39 PM, Chinmaya Dwibedy <ckdwibedy@gmail.com>
wrote:

> Hi Sergio,
>
>
> Thank your for clarification. I am using dpdk-2.2.0. Here goes our
> configuration on VM. Can you please have a look into the same and let me
> know if anything is wrong. Will try to run the autotest, ipsec-segw on host
> as per your suggestion.
>
>
> [root@vpn-s ~]# uname -r
>
> 3.12.9-301.fc20.x86_64
>
> [root@vpn-s ~]#
>
>
> [root@vpn-s ~]# cat /etc/redhat-release
>
> Fedora release 20 (Heisenbug)
>
> [root@vpn-s ~]#
>
>
>
>
> [root@vpn-s ~]# cat /proc/cmdline
>
> BOOT_IMAGE=/boot/vmlinuz-3.12.9-301.fc20.x86_64
> root=UUID=ab47cbc9-68ee-403c-96a5-184e68238e65 ro console=ttyS0,115200n8
> iommu=pt intel_iommu=on
>
> [root@vpn-s ~]#
>
>
> [root@vpn-s QAT]# grep -i "iommu.*enabled" /var/log/messages
>
> Jun 22 07:31:17 vpn-s kernel: [    0.000000] Intel-IOMMU: enabled
>
>
> [root@vpn-s QAT]#
>
> [root@vpn-s dpdk-2.2.0]# lspci -nn | grep 0443
>
> 00:06.0 Co-processor [0b40]: Intel Corporation Device [8086:0443]
>
> [root@vpn-s dpdk-2.2.0]#
>
>
> Installed Intel qat driver installation on VM (downloaded from
> https://01.org/packet-processing/intel%C2%AE-quickassist-technology-drivers-and-patches)
> and followed the
>
> https://01.org/sites/default/files/page/330750-005_qat_gsg.pdf guide.
> Note, used the following installation options i.e., ./installer.sh install
> QAT1.6 guest. After this , it detects QAT VF.
>
>
> [root@vpn-s QAT]# service qat_service status
>
> There is 1 acceleration device(s) in the system:
>
>  icp_dev0 - type=dh895xcc, inst_id=0, node_id=0,  bdf=00:06:0, #accel=1,
> #engines=1, state=up
>
> [root@vpn-s QAT]#
>
>
> [root@vpn-s QAT]# lsmod | grep qa
>
> icp_qa_al_vf          813442  1
>
> [root@vpn-s QAT]#
>
>
>
> The below are DPDK configuration options in
> config/defconfig_x86_64-native-linuxapp-gcc.
>
>
> include "common_linuxapp"
>
>
>
> CONFIG_RTE_MACHINE="native"
>
> CONFIG_RTE_ARCH="x86_64"
>
> CONFIG_RTE_ARCH_X86_64=y
>
> CONFIG_RTE_ARCH_64=y
>
> CONFIG_RTE_TOOLCHAIN="gcc"
>
> CONFIG_RTE_TOOLCHAIN_GCC=y
>
> CONFIG_RTE_LIBRTE_PMD_AESNI_MB=y
>
> CONFIG_RTE_LIBRTE_CRYPTODEV_DEBUG=y
>
> CONFIG_RTE_LIBRTE_VIRTIO_PMD=y
>
> CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=y
>
> CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=y
>
> CONFIG_RTE_LIBRTE_PMD_QAT=y
>
> CONFIG_RTE_QAT_PMD_MAX_NB_SESSIONS=128
>
> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_INIT=y
>
> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_TX=y
>
> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_RX=y
>
> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_DRIVER=y
>
>
>
> The below is my setup.sh
>
>
> export RTE_SDK=/opt/dpdk-2.2.0
>
> export RTE_TARGET=x86_64-native-linuxapp-gcc
>
> export RTE_ARCH=x86_64
>
> export AESNI_MULTI_BUFFER_LIB_PATH=/opt/code
>
> make config T=x86_64-native-linuxapp-gcc
>
> sed -ri 's,(PMD_PCAP=).*,\1y,' build/.config
>
> make install T=x86_64-native-linuxapp-gcc
>
> mkdir -p /mnt/huge
>
> mount -t hugetlbfs nodev /mnt/huge
>
> echo 2048 >
> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
>
> sudo modprobe uio_pci_generic
>
> sudo modprobe uio
>
> rmmod igb_uio
>
> insmod
> ./x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.ko
>
> echo -n "0000:00:06.0" > /sys/bus/pci/devices/0000\:00\:06.0/driver/unbind
>
> echo "8086 0443" > /sys/bus/pci/drivers/igb_uio/new_id
>
> lspci -vvd:443
>
> ./tools/dpdk_nic_bind.py --bind=uio_pci_generic eth1
>
> ./tools/dpdk_nic_bind.py --bind=uio_pci_generic 00:03.0
>
> ./tools/dpdk_nic_bind.py -s
>
>
> Also modified the cryptodevs_init() ( examples/ipsec-secgw/ipsec-secgw.c)
> to rte_cryptodev_start(cdev_id); after rte_cryptodev_queue_pair_setup().
> The purpose of calling this to start the transmit and the receive units
> of the QAT device. It returns success. But still getting the same issue
> i.e. rte_cryptodev_dequeue_burst () does not return crypto packet. Note
> that, when I call rte_cryptodev_queue_pair_setup () with 3rd argument as
> NULL (in which case default configuration will be used) , it crashes. Is
> it a known issue?
>
>
>
>
>
> Regards,
>
> Chinmaya
>
> On Tue, Jun 21, 2016 at 9:10 PM, Sergio Gonzalez Monroy <
> sergio.gonzalez.monroy@intel.com> wrote:
>
>> On 21/06/2016 08:26, Chinmaya Dwibedy wrote:
>>
>>>
>>> Hi Sergio,
>>>
>>>
>>> As suggested by you, run ‘app/test' application and then run
>>> 'cryptodev_qat_autotest' . Found the same issue i.e., It halts in
>>> rte_cryptodev_dequeue_burst () (while loop) in process_crypto_request ()
>>> function (app/test/test_cryptodev.c).
>>>
>>> Debugged the qat_crypto_pkt_rx_burst() (in
>>> drivers/crypto/qat/qat_crypto.c) and found that , the value of resp_msg is
>>> 0x7F7F7F7F (i.e., ADF_RING_EMPTY_SIG) . Thus it is not getting crypto
>>> packet (i.e., rte_mbuf) from the RX queue.  But I find that, in
>>> qat_crypto_pkt_tx_burst(), the  qat_alg_write_mbuf_entry() is being called.
>>> The qat_alg_write_mbuf_entry () prints qat_req  using rte_hexdump () and
>>> returns zero.
>>>
>>>
>>> One question, the base_addr + tail of tx_q (in
>>> qat_crypto_pkt_tx_burst()) should be same as base_addr + head (in
>>> qat_crypto_pkt_rx_burst()). Right ? Please feel free to correct me if I am
>>> wrong.
>>>
>>>
>>> Also can you please provide some pointers for further debugging? Would
>>> it be configuration issue? Note that, I am using VF in VM.
>>>
>>>
>> In theory  you should be able to run ipsec-secgw sample app with QAT VF
>> as long as the setup is right.
>> That the autotest fails would likely point to some possible issues with
>> the setup.
>>
>> Have you tried to run the autotest in the host?
>>
>> Sergio
>>
>>
>>> Thank you once again for your prompt response and great support.
>>>
>>>
>>> Regards,
>>>
>>> Chinmaya
>>>
>>>
>>> On Mon, Jun 20, 2016 at 3:04 PM, Sergio Gonzalez Monroy <
>>> sergio.gonzalez.monroy@intel.com <mailto:
>>> sergio.gonzalez.monroy@intel.com>> wrote:
>>>
>>>     On 20/06/2016 10:08, Chinmaya Dwibedy wrote:
>>>
>>>
>>>         Hi Sergio,
>>>
>>>
>>>         Agreed.  We might not dequeue the same amount of crypto ops we
>>>         just previously enqueued, it's asynchronous. But in this case,
>>>         I have sent just one UDP packet. So there will be one crypto
>>>         ops. Right?  Also I put a sleep (50) after the
>>>         rte_crypto_enqueue_burst() function in ipsec_processing()
>>>         (ipsec.c) , so as to allow more time  ( for QAT device) for
>>>         processing. Still getting the same result i.e., the
>>>         rte_crypto_dequeue_burst () functionreturns zero.
>>>
>>>
>>>         In case of S/W crypto device (i.e., AESNI),   the VM gets
>>>         inbound UDP packets on Port 1/eth1, encapsulates (after
>>>         consulting its SPD) in an IPsec ESP packet and sends to its
>>>         peer through Port 0/eth0 interface.
>>>
>>>
>>>         Yes, the security policy, security association and Routing
>>>         entries/configurations are exactly same. Please feel free to
>>>         let me know if you need additional information.
>>>
>>>
>>>     Could you try to run 'app/test' application then run
>>>     'cryptodev_qat_autotest' ? That is a functional test for cryptodev
>>>     QAT PMD.
>>>
>>>     Sergio
>>>
>>>
>>>
>>
>

Re: [dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode)

[Chinmaya Dwibedy]

Hi Sergio/All,


Upon further investigation found that, on Guest VM if I remove
“intel_iommu=on” boot line parameters and run app/test and then
crypto_qat_autotest tool, it shows all the test cases are passed.
Thereafter when I run ipsec-segw application, I got the below error in
dmesg on host. Is it an issue with crypto device configuration and queue
pair setup? Please suggest.



[root@stag48 ~(keystone_admin)]# dmesg -c

[ 1264.266965] dmar: DRHD: handling fault status reg 2

[ 1264.266972] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr
ff53f000f000

DMAR:[fault reason 06] PTE Read access is not set

[ 1264.266978] icp_qa_al err: adf_dh895xcc_adf_isr_LogAE_Int: AE #5 Error

Uncorrectable Error: Yes

Control Store Error: No

Control Store information: Address 0x2A4C - syndrome 0x0

Register ECC Error No - Parity Error: Yes

Register information: address 0x2 - type Transfer

[ 1264.266980] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr
ff53f000e000

DMAR:[fault reason 06] PTE Read access is not set

[ 1264.266986] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr
ff53f000e000

DMAR:[fault reason 06] PTE Read access is not set

[ 1264.266990] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr
ff53f000f000

DMAR:[fault reason 06] PTE Read access is not set

[ 1264.266993] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr
ff53f000e000

DMAR:[fault reason 06] PTE Read access is not set

[ 1264.266997] icp_qa_al err: adf_dh895xcc_adf_isr_LogAE_Int: AE #7 Error

Uncorrectable Error: Yes

Control Store Error: No

Control Store information: Address 0x3FFD - syndrome 0x0

Register ECC Error No - Parity Error: Yes

Register information: address 0x22 - type Transfer

[ 1264.266999] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr
ff53f000e000

DMAR:[fault reason 06] PTE Read access is not set

[ 1264.267003] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr
ff53f000e000

DMAR:[fault reason 06] PTE Read access is not set

[ 1264.267007] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr
ff53f000f000

DMAR:[fault reason 06] PTE Read access is not set

[ 1264.267014] icp_qa_al err: adf_dh895xcc_adf_isr_LogAE_Int: AE #8 Error

Uncorrectable Error: Yes

Control Store Error: No

Control Store information: Address 0x2102 - syndrome 0x0

Register ECC Error No - Parity Error: Yes

Register information: address 0x2 - type Transfer

[ 1264.267020] icp_qa_al err: adf_dh895xcc_adf_isr_LogAE_Int: AE #9 Error

Uncorrectable Error: Yes

Control Store Error: No

Control Store information: Address 0x0 - syndrome 0x0

Register ECC Error No - Parity Error: Yes

Register information: address 0x22 - type Transfer

[ 1264.267026] icp_qa_al err: adf_dh895xcc_adf_isr_LogAE_Int: AE #b Error

Uncorrectable Error: Yes

Control Store Error: No

Control Store information: Address 0x3FFF - syndrome 0x0

Register ECC Error No - Parity Error: Yes

Register information: address 0x2 - type Transfer

[ 1264.267032] icp_qa_al err: adf_dh895xcc_adf_isr_handleUncoInterrupt:
Uncorrectable Push/Pull Misc Error

memory status: No errors occurred - Transaction Id 0x0 - Error type reserved

Bus Operation Type Push - Id 0x3859C020

[ 1264.267038] Reset needed for device: icp_dev1

[ 1264.267039] Auto Reset disabled. Please reset device1

[root@stag48 ~(keystone_admin)]#







Regards,

Chinmaya

On Thu, Jun 23, 2016 at 5:18 PM, Chinmaya Dwibedy <ckdwibedy@gmail.com>
wrote:

> Hi All,
>
>  On host, I installed dpdk-2.2.0 and then run 'cryptodev_qat_autotest’. I found this test case to be failed in dequeue burst API ().Upon looking into the below system logs (#dmesg), it seems, when this function is called, the QAT
>
> hardware fails to write the crypto packet to the DMA buffer.  When I passed intel_iommu=off command in boot line, it works. I think, when we have no IOMMU, "physical" address space is accessed directly by hardware, so code works.
>
> The issue has been observed if the system runs with the kernel parameter “intel_iommu=on”.
>
>
>
> [root@stag48 ~]# dmesg
>
> [  794.746134] dmar: DRHD: handling fault status reg 2
>
> [  794.746141] dmar: DMAR:[DMA Read] Request device [83:01.0] fault addr
> 3398c000
>
> DMAR:[fault reason 02] Present bit in context entry is clear
>
> [  794.746158] icp_qa_al err: adf_dh895xcc_adf_isr_handleUncoInterrupt:
> Uncorrectable Push/Pull Misc Error
>
> memory status: No errors occurred - Transaction Id 0x0 - Error type
> reserved
>
> Bus Operation Type Push - Id 0x80000
>
> [  794.746167] Reset needed for device: icp_dev0
>
> [  794.746167] Auto Reset disabled. Please reset device0
>
> [root@stag48 ~]#
>
>
>
> *Note: *The QAT VF (*83:01.0*) is bound to the DPDK UIO driver.
>
>
>
> 83:01.0 Co-processor: Intel Corporation DH895XCC Series QAT Virtual
> Function
>
>         Subsystem: Intel Corporation Device 0000
>
>         Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
>         Latency: 0
>
>         Region 0: [virtual] Memory at c8280000 (64-bit, non-prefetchable)
> [size=4K]
>
>         Region 2: [virtual] Memory at c82a0000 (64-bit, non-prefetchable)
> [size=4K]
>
>         Capabilities: [90] MSI: Enable- Count=1/1 Maskable+ 64bit+
>
>                 Address: 0000000000000000  Data: 0000
>
>                 Masking: 00000000  Pending: 00000000
>
>         Kernel driver in use: igb_uio
>
>
>
>
>
> Now testing the same on Guest and finding the same issue (i.e., failing on dequeue burst API ()). Upon looking into error message (in dmesg) on host/hypervisor it says i.e., [28475.040313]
>
> vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state. The PCI device 88:04.6 is the QAT VF, being assigned to Guest. I am trying to figure out what causes “buffer not found” issue .
>
> Please suggest if anyone has an idea on it.
>
>
>
> Here goes the output of dmesg
>
>
>
> [27086.464014] vfio-pci 0000:88:04.6: enabling device (0000 -> 0002)
>
> [27086.464040] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state
>
> [27086.564610] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state
>
> [27086.574052] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state
>
> [27089.786623] kvm: zapping shadow pages for mmio generation wraparound
>
> [27106.320042] vfio-pci 0000:88:04.6: irq 136 for MSI/MSI-X
>
> [28474.784373] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state
>
> [28475.040313] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state
>
> [28493.271950] vfio-pci 0000:88:04.6: irq 136 for MSI/MSI-X
>
>
>
>
>
> Regards,
>
> Chinmaya
>
>
>
>
>
>
>
> On Wed, Jun 22, 2016 at 1:39 PM, Chinmaya Dwibedy <ckdwibedy@gmail.com>
> wrote:
>
>> Hi Sergio,
>>
>>
>> Thank your for clarification. I am using dpdk-2.2.0. Here goes our
>> configuration on VM. Can you please have a look into the same and let me
>> know if anything is wrong. Will try to run the autotest, ipsec-segw on host
>> as per your suggestion.
>>
>>
>> [root@vpn-s ~]# uname -r
>>
>> 3.12.9-301.fc20.x86_64
>>
>> [root@vpn-s ~]#
>>
>>
>> [root@vpn-s ~]# cat /etc/redhat-release
>>
>> Fedora release 20 (Heisenbug)
>>
>> [root@vpn-s ~]#
>>
>>
>>
>>
>> [root@vpn-s ~]# cat /proc/cmdline
>>
>> BOOT_IMAGE=/boot/vmlinuz-3.12.9-301.fc20.x86_64
>> root=UUID=ab47cbc9-68ee-403c-96a5-184e68238e65 ro console=ttyS0,115200n8
>> iommu=pt intel_iommu=on
>>
>> [root@vpn-s ~]#
>>
>>
>> [root@vpn-s QAT]# grep -i "iommu.*enabled" /var/log/messages
>>
>> Jun 22 07:31:17 vpn-s kernel: [    0.000000] Intel-IOMMU: enabled
>>
>>
>> [root@vpn-s QAT]#
>>
>> [root@vpn-s dpdk-2.2.0]# lspci -nn | grep 0443
>>
>> 00:06.0 Co-processor [0b40]: Intel Corporation Device [8086:0443]
>>
>> [root@vpn-s dpdk-2.2.0]#
>>
>>
>> Installed Intel qat driver installation on VM (downloaded from
>> https://01.org/packet-processing/intel%C2%AE-quickassist-technology-drivers-and-patches)
>> and followed the
>>
>> https://01.org/sites/default/files/page/330750-005_qat_gsg.pdf guide.
>> Note, used the following installation options i.e., ./installer.sh install
>> QAT1.6 guest. After this , it detects QAT VF.
>>
>>
>> [root@vpn-s QAT]# service qat_service status
>>
>> There is 1 acceleration device(s) in the system:
>>
>>  icp_dev0 - type=dh895xcc, inst_id=0, node_id=0,  bdf=00:06:0, #accel=1,
>> #engines=1, state=up
>>
>> [root@vpn-s QAT]#
>>
>>
>> [root@vpn-s QAT]# lsmod | grep qa
>>
>> icp_qa_al_vf          813442  1
>>
>> [root@vpn-s QAT]#
>>
>>
>>
>> The below are DPDK configuration options in
>> config/defconfig_x86_64-native-linuxapp-gcc.
>>
>>
>> include "common_linuxapp"
>>
>>
>>
>> CONFIG_RTE_MACHINE="native"
>>
>> CONFIG_RTE_ARCH="x86_64"
>>
>> CONFIG_RTE_ARCH_X86_64=y
>>
>> CONFIG_RTE_ARCH_64=y
>>
>> CONFIG_RTE_TOOLCHAIN="gcc"
>>
>> CONFIG_RTE_TOOLCHAIN_GCC=y
>>
>> CONFIG_RTE_LIBRTE_PMD_AESNI_MB=y
>>
>> CONFIG_RTE_LIBRTE_CRYPTODEV_DEBUG=y
>>
>> CONFIG_RTE_LIBRTE_VIRTIO_PMD=y
>>
>> CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=y
>>
>> CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=y
>>
>> CONFIG_RTE_LIBRTE_PMD_QAT=y
>>
>> CONFIG_RTE_QAT_PMD_MAX_NB_SESSIONS=128
>>
>> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_INIT=y
>>
>> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_TX=y
>>
>> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_RX=y
>>
>> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_DRIVER=y
>>
>>
>>
>> The below is my setup.sh
>>
>>
>> export RTE_SDK=/opt/dpdk-2.2.0
>>
>> export RTE_TARGET=x86_64-native-linuxapp-gcc
>>
>> export RTE_ARCH=x86_64
>>
>> export AESNI_MULTI_BUFFER_LIB_PATH=/opt/code
>>
>> make config T=x86_64-native-linuxapp-gcc
>>
>> sed -ri 's,(PMD_PCAP=).*,\1y,' build/.config
>>
>> make install T=x86_64-native-linuxapp-gcc
>>
>> mkdir -p /mnt/huge
>>
>> mount -t hugetlbfs nodev /mnt/huge
>>
>> echo 2048 >
>> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
>>
>> sudo modprobe uio_pci_generic
>>
>> sudo modprobe uio
>>
>> rmmod igb_uio
>>
>> insmod
>> ./x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.ko
>>
>> echo -n "0000:00:06.0" > /sys/bus/pci/devices/0000\:00\:06.0/driver/unbind
>>
>> echo "8086 0443" > /sys/bus/pci/drivers/igb_uio/new_id
>>
>> lspci -vvd:443
>>
>> ./tools/dpdk_nic_bind.py --bind=uio_pci_generic eth1
>>
>> ./tools/dpdk_nic_bind.py --bind=uio_pci_generic 00:03.0
>>
>> ./tools/dpdk_nic_bind.py -s
>>
>>
>> Also modified the cryptodevs_init() ( examples/ipsec-secgw/ipsec-secgw.c)
>> to rte_cryptodev_start(cdev_id); after rte_cryptodev_queue_pair_setup().
>> The purpose of calling this to start the transmit and the receive units
>> of the QAT device. It returns success. But still getting the same issue
>> i.e. rte_cryptodev_dequeue_burst () does not return crypto packet. Note
>> that, when I call rte_cryptodev_queue_pair_setup () with 3rd argument as
>> NULL (in which case default configuration will be used) , it crashes. Is
>> it a known issue?
>>
>>
>>
>>
>>
>> Regards,
>>
>> Chinmaya
>>
>> On Tue, Jun 21, 2016 at 9:10 PM, Sergio Gonzalez Monroy <
>> sergio.gonzalez.monroy@intel.com> wrote:
>>
>>> On 21/06/2016 08:26, Chinmaya Dwibedy wrote:
>>>
>>>>
>>>> Hi Sergio,
>>>>
>>>>
>>>> As suggested by you, run ‘app/test' application and then run
>>>> 'cryptodev_qat_autotest' . Found the same issue i.e., It halts in
>>>> rte_cryptodev_dequeue_burst () (while loop) in process_crypto_request ()
>>>> function (app/test/test_cryptodev.c).
>>>>
>>>> Debugged the qat_crypto_pkt_rx_burst() (in
>>>> drivers/crypto/qat/qat_crypto.c) and found that , the value of resp_msg is
>>>> 0x7F7F7F7F (i.e., ADF_RING_EMPTY_SIG) . Thus it is not getting crypto
>>>> packet (i.e., rte_mbuf) from the RX queue.  But I find that, in
>>>> qat_crypto_pkt_tx_burst(), the  qat_alg_write_mbuf_entry() is being called.
>>>> The qat_alg_write_mbuf_entry () prints qat_req  using rte_hexdump () and
>>>> returns zero.
>>>>
>>>>
>>>> One question, the base_addr + tail of tx_q (in
>>>> qat_crypto_pkt_tx_burst()) should be same as base_addr + head (in
>>>> qat_crypto_pkt_rx_burst()). Right ? Please feel free to correct me if I am
>>>> wrong.
>>>>
>>>>
>>>> Also can you please provide some pointers for further debugging? Would
>>>> it be configuration issue? Note that, I am using VF in VM.
>>>>
>>>>
>>> In theory  you should be able to run ipsec-secgw sample app with QAT VF
>>> as long as the setup is right.
>>> That the autotest fails would likely point to some possible issues with
>>> the setup.
>>>
>>> Have you tried to run the autotest in the host?
>>>
>>> Sergio
>>>
>>>
>>>> Thank you once again for your prompt response and great support.
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Chinmaya
>>>>
>>>>
>>>> On Mon, Jun 20, 2016 at 3:04 PM, Sergio Gonzalez Monroy <
>>>> sergio.gonzalez.monroy@intel.com <mailto:
>>>> sergio.gonzalez.monroy@intel.com>> wrote:
>>>>
>>>>     On 20/06/2016 10:08, Chinmaya Dwibedy wrote:
>>>>
>>>>
>>>>         Hi Sergio,
>>>>
>>>>
>>>>         Agreed.  We might not dequeue the same amount of crypto ops we
>>>>         just previously enqueued, it's asynchronous. But in this case,
>>>>         I have sent just one UDP packet. So there will be one crypto
>>>>         ops. Right?  Also I put a sleep (50) after the
>>>>         rte_crypto_enqueue_burst() function in ipsec_processing()
>>>>         (ipsec.c) , so as to allow more time  ( for QAT device) for
>>>>         processing. Still getting the same result i.e., the
>>>>         rte_crypto_dequeue_burst () functionreturns zero.
>>>>
>>>>
>>>>         In case of S/W crypto device (i.e., AESNI),   the VM gets
>>>>         inbound UDP packets on Port 1/eth1, encapsulates (after
>>>>         consulting its SPD) in an IPsec ESP packet and sends to its
>>>>         peer through Port 0/eth0 interface.
>>>>
>>>>
>>>>         Yes, the security policy, security association and Routing
>>>>         entries/configurations are exactly same. Please feel free to
>>>>         let me know if you need additional information.
>>>>
>>>>
>>>>     Could you try to run 'app/test' application then run
>>>>     'cryptodev_qat_autotest' ? That is a functional test for cryptodev
>>>>     QAT PMD.
>>>>
>>>>     Sergio
>>>>
>>>>
>>>>
>>>
>>
>