* [dpdk-users] Issue with Physical Function passthrough on VM Guest (Fedora 20) using qatmux.l.2.6.0-60
@ 2016-06-10 7:16 Chinmaya Dwibedy
2016-06-10 7:33 ` SAKTHIVEL ANAND S
0 siblings, 1 reply; 3+ messages in thread
From: Chinmaya Dwibedy @ 2016-06-10 7:16 UTC (permalink / raw)
To: users
Hi,
I have assigned Intel QAT device to VMs as a full device (PF - physical
function, PCI passthrough). Below is the output of lspci in host (Cent OS
7.2).
[root@localhost nova(keystone_admin)]# lspci -nn | grep 0435
83:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [
8086:0435]
88:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [
8086:0435]
[root@localhost nova(keystone_admin)]#
[root@localhost ~]# lspci -vnd 8086:0435
83:00.0 0b40: 8086:0435
Subsystem: 8086:35c5
Physical Slot: 0-1
Flags: bus master, fast devsel, latency 0, IRQ 35
Memory at 387fffd00000 (64-bit, prefetchable) [size=512K]
Memory at c8200000 (64-bit, non-prefetchable) [size=256K]
Memory at c8240000 (64-bit, non-prefetchable) [size=256K]
Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [60] MSI-X: Enable- Count=33 Masked-
Capabilities: [6c] Power Management version 3
Capabilities: [74] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [138] Alternative Routing-ID Interpretation (ARI)
Capabilities: [140] Single Root I/O Virtualization (SR-IOV)
Kernel driver in use: vfio-pci
88:00.0 0b40: 8086:0435
Subsystem: 8086:35c5
Physical Slot: 0-2
Flags: fast devsel, IRQ 39
Memory at 387fffe00000 (64-bit, prefetchable) [size=512K]
Memory at c8000000 (64-bit, non-prefetchable) [size=256K]
Memory at c8040000 (64-bit, non-prefetchable) [size=256K]
Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [60] MSI-X: Enable- Count=33 Masked-
Capabilities: [6c] Power Management version 3
Capabilities: [74] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [138] Alternative Routing-ID Interpretation (ARI)
Capabilities: [140] Single Root I/O Virtualization (SR-IOV)
Kernel driver in use: vfio-pci
[root@localhost ~]#
In VM, the Intel QAT device shows up in lspci as "Intel Corporation Coleto
Creek PCIe Endpoint".
[root@test1 QAT]# lspci -nn | grep 0435
00:05.0 Co-processor [0b40]: Intel Corporation Coleto Creek PCIe Endpoint
[8086:0435]
[root@test1 QAT]#
Afterward I installed Intel® QuickAssist Technology Software i.e.,
qatmux.l.2.6.0-60.tgz ( downloaded at
https://01.org/packet-processing/intel%C2%AE-quickassist-technology-drivers-and-patches)
on the Guest VM . Here are the details of VM.
1) Fedora release 20 (Heisenbug).
2) Linux Kernel version: 3.12.9-301.fc20.x86_64.
3) BOOT_IMAGE=/boot/vmlinuz-3.12.9-301.fc20.x86_64
root=UUID=ab47cbc9-68ee-403c-96a5-184e68238e65 ro console=ttyS0,115200n8
intel_iommu=off
*Notes: *
1) We are not using SR-IOV and are instead passing through a PF for
acceleration services on guest only. Hence we have not installed the
Intel® QuickAssist Technology Software package on the host.
2) Found an Intel QAT driver with this kernel . Upon doing #lsmod |
grep qa, showed icp_qa_al_vf. Thus did rmmod icp_qa_al_vf and service
qat_service shutdown. This was done before building Intel QuickAssist
Technology Software package.
For passthrough of PF, opted the below Install Acceleration option
a) export ICP_SRIOV=0
b) export ICP_WITHOUT_IOMMU=1
c) ./installer.sh build_accel QAT1.6
d) ./installer.sh install QAT1.6 guest
e) insmod /root/QAT /QAT1.6/build/icp_qa_al.ko
When I issue the below command to start the acceleration software, it fails
to start the driver.
[root@test1 ~]# service qat_service start
Processing file: /etc/dh895xcc_qa_dev0.conf
ADF_CONFIG_CTL err: adf_set_bank_core_affinity: Invalid core affinity
settings for accelerator 0 bank 8. Max cpu number is 3. Defaulting to 0
ADF_CONFIG_CTL err: adf_set_bank_core_affinity: Invalid core affinity
settings for accelerator 0 bank 9. Max cpu number is 3. Defaulting to 0
Parity err reporting is disabled.
Error: failed to start device 0
There is 1 acceleration device(s) in the system:
icp_dev0 - type=dh895xcc, inst_id=0, node_id=0, bdf=00:05:0, #accel=6,
#engines=12, state=down
[root@test1 ~]#
Here are the kernel logs
[root@test1 ~]# dmesg -c
[ 3104.979666] Reading config file.
[ 3104.980250] Starting acceleration device icp_dev0.
[ 3104.980731] Resetting device icp_dev0
[ 3105.126920] qat_1_6_adf 0000:00:05.0: irq 45 for MSI/MSI-X
[ 3105.126939] qat_1_6_adf 0000:00:05.0: irq 46 for MSI/MSI-X
[ 3105.126956] qat_1_6_adf 0000:00:05.0: irq 47 for MSI/MSI-X
[ 3105.126973] qat_1_6_adf 0000:00:05.0: irq 48 for MSI/MSI-X
[ 3105.126989] qat_1_6_adf 0000:00:05.0: irq 49 for MSI/MSI-X
[ 3105.127029] qat_1_6_adf 0000:00:05.0: irq 50 for MSI/MSI-X
[ 3105.127048] qat_1_6_adf 0000:00:05.0: irq 51 for MSI/MSI-X
[ 3105.127065] qat_1_6_adf 0000:00:05.0: irq 52 for MSI/MSI-X
[ 3105.127082] qat_1_6_adf 0000:00:05.0: irq 53 for MSI/MSI-X
[ 3105.127098] qat_1_6_adf 0000:00:05.0: irq 54 for MSI/MSI-X
[ 3105.127115] qat_1_6_adf 0000:00:05.0: irq 55 for MSI/MSI-X
[ 3105.127131] qat_1_6_adf 0000:00:05.0: irq 56 for MSI/MSI-X
[ 3105.127148] qat_1_6_adf 0000:00:05.0: irq 57 for MSI/MSI-X
[ 3105.127164] qat_1_6_adf 0000:00:05.0: irq 58 for MSI/MSI-X
[ 3105.127181] qat_1_6_adf 0000:00:05.0: irq 59 for MSI/MSI-X
[ 3105.127198] qat_1_6_adf 0000:00:05.0: irq 60 for MSI/MSI-X
[ 3105.127215] qat_1_6_adf 0000:00:05.0: irq 61 for MSI/MSI-X
[ 3105.127231] qat_1_6_adf 0000:00:05.0: irq 62 for MSI/MSI-X
[ 3105.127248] qat_1_6_adf 0000:00:05.0: irq 63 for MSI/MSI-X
[ 3105.127265] qat_1_6_adf 0000:00:05.0: irq 64 for MSI/MSI-X
[ 3105.127282] qat_1_6_adf 0000:00:05.0: irq 65 for MSI/MSI-X
[ 3105.127298] qat_1_6_adf 0000:00:05.0: irq 66 for MSI/MSI-X
[ 3105.127315] qat_1_6_adf 0000:00:05.0: irq 67 for MSI/MSI-X
[ 3105.127331] qat_1_6_adf 0000:00:05.0: irq 68 for MSI/MSI-X
[ 3105.127348] qat_1_6_adf 0000:00:05.0: irq 69 for MSI/MSI-X
[ 3105.127364] qat_1_6_adf 0000:00:05.0: irq 70 for MSI/MSI-X
[ 3105.127380] qat_1_6_adf 0000:00:05.0: irq 71 for MSI/MSI-X
[ 3105.127397] qat_1_6_adf 0000:00:05.0: irq 72 for MSI/MSI-X
[ 3105.127413] qat_1_6_adf 0000:00:05.0: irq 73 for MSI/MSI-X
[ 3105.127429] qat_1_6_adf 0000:00:05.0: irq 74 for MSI/MSI-X
[ 3105.127446] qat_1_6_adf 0000:00:05.0: irq 75 for MSI/MSI-X
[ 3105.127462] qat_1_6_adf 0000:00:05.0: irq 76 for MSI/MSI-X
[ 3105.127487] qat_1_6_adf 0000:00:05.0: irq 77 for MSI/MSI-X
[ 3105.157973] icp_qa_al err: adf_subsystemInit: Invalid SRIOV_Enabled
value, driver is built with SRIOV.
[ 3105.158891] icp_qa_al err: adf_do_init: adf_subsystemInit error,
stopping and shutting down
[ 3106.162962] icp_qa_al err: adf_init_all: failed to initialize accelerator
[ 3106.163670] icp_qa_al err: adf_init_devices: failed init accel 0
[ 3106.164256] icp_qa_al err: adf_ctl_ioctl_dev_start: failed call to
adf_init_devices
[root@test1 ~]#
Can anyone please let me know what might be the issue and its remedy ?
please feel free to correct me if I am wrong. If it is not the right forum
, please let me know the correct one. Thank you in advance for time and
support.
Regards,
Chinmaya
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [dpdk-users] Issue with Physical Function passthrough on VM Guest (Fedora 20) using qatmux.l.2.6.0-60
2016-06-10 7:16 [dpdk-users] Issue with Physical Function passthrough on VM Guest (Fedora 20) using qatmux.l.2.6.0-60 Chinmaya Dwibedy
@ 2016-06-10 7:33 ` SAKTHIVEL ANAND S
2016-06-10 8:57 ` Chinmaya Dwibedy
0 siblings, 1 reply; 3+ messages in thread
From: SAKTHIVEL ANAND S @ 2016-06-10 7:33 UTC (permalink / raw)
To: Chinmaya Dwibedy; +Cc: users
If a failed to start device
error is shown during the Acceleration
installation, ensure that the kernel option
intel_iommu=off
has been
configured as specified in
Updating Grub Configuration File
on page 13. If
this kernel parameter is not specified, acceleration services are only
available in a guest operating system.
The above lines are from QAT official document.. so if u r using QAT in VM,
you have to make intel_iommu=on in the grub config.
On Fri, Jun 10, 2016 at 12:46 PM, Chinmaya Dwibedy <ckdwibedy@gmail.com>
wrote:
> Hi,
>
>
> I have assigned Intel QAT device to VMs as a full device (PF - physical
> function, PCI passthrough). Below is the output of lspci in host (Cent OS
> 7.2).
>
>
> [root@localhost nova(keystone_admin)]# lspci -nn | grep 0435
>
> 83:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [
> 8086:0435]
>
> 88:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [
> 8086:0435]
>
> [root@localhost nova(keystone_admin)]#
>
> [root@localhost ~]# lspci -vnd 8086:0435
>
> 83:00.0 0b40: 8086:0435
>
> Subsystem: 8086:35c5
>
> Physical Slot: 0-1
>
> Flags: bus master, fast devsel, latency 0, IRQ 35
>
> Memory at 387fffd00000 (64-bit, prefetchable) [size=512K]
>
> Memory at c8200000 (64-bit, non-prefetchable) [size=256K]
>
> Memory at c8240000 (64-bit, non-prefetchable) [size=256K]
>
> Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+
>
> Capabilities: [60] MSI-X: Enable- Count=33 Masked-
>
> Capabilities: [6c] Power Management version 3
>
> Capabilities: [74] Express Endpoint, MSI 00
>
> Capabilities: [100] Advanced Error Reporting
>
> Capabilities: [138] Alternative Routing-ID Interpretation (ARI)
>
> Capabilities: [140] Single Root I/O Virtualization (SR-IOV)
>
> Kernel driver in use: vfio-pci
>
>
>
> 88:00.0 0b40: 8086:0435
>
> Subsystem: 8086:35c5
>
> Physical Slot: 0-2
>
> Flags: fast devsel, IRQ 39
>
> Memory at 387fffe00000 (64-bit, prefetchable) [size=512K]
>
> Memory at c8000000 (64-bit, non-prefetchable) [size=256K]
>
> Memory at c8040000 (64-bit, non-prefetchable) [size=256K]
>
> Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+
>
> Capabilities: [60] MSI-X: Enable- Count=33 Masked-
>
> Capabilities: [6c] Power Management version 3
>
> Capabilities: [74] Express Endpoint, MSI 00
>
> Capabilities: [100] Advanced Error Reporting
>
> Capabilities: [138] Alternative Routing-ID Interpretation (ARI)
>
> Capabilities: [140] Single Root I/O Virtualization (SR-IOV)
>
> Kernel driver in use: vfio-pci
>
>
>
> [root@localhost ~]#
>
>
> In VM, the Intel QAT device shows up in lspci as "Intel Corporation Coleto
> Creek PCIe Endpoint".
>
>
> [root@test1 QAT]# lspci -nn | grep 0435
>
> 00:05.0 Co-processor [0b40]: Intel Corporation Coleto Creek PCIe Endpoint
> [8086:0435]
>
> [root@test1 QAT]#
>
>
> Afterward I installed Intel® QuickAssist Technology Software i.e.,
> qatmux.l.2.6.0-60.tgz ( downloaded at
>
> https://01.org/packet-processing/intel%C2%AE-quickassist-technology-drivers-and-patches
> )
> on the Guest VM . Here are the details of VM.
>
> 1) Fedora release 20 (Heisenbug).
>
> 2) Linux Kernel version: 3.12.9-301.fc20.x86_64.
>
> 3) BOOT_IMAGE=/boot/vmlinuz-3.12.9-301.fc20.x86_64
> root=UUID=ab47cbc9-68ee-403c-96a5-184e68238e65 ro console=ttyS0,115200n8
> intel_iommu=off
>
>
>
> *Notes: *
>
> 1) We are not using SR-IOV and are instead passing through a PF for
> acceleration services on guest only. Hence we have not installed the
> Intel® QuickAssist Technology Software package on the host.
>
> 2) Found an Intel QAT driver with this kernel . Upon doing #lsmod |
> grep qa, showed icp_qa_al_vf. Thus did rmmod icp_qa_al_vf and service
> qat_service shutdown. This was done before building Intel QuickAssist
> Technology Software package.
>
>
>
> For passthrough of PF, opted the below Install Acceleration option
>
> a) export ICP_SRIOV=0
>
> b) export ICP_WITHOUT_IOMMU=1
>
> c) ./installer.sh build_accel QAT1.6
>
> d) ./installer.sh install QAT1.6 guest
>
> e) insmod /root/QAT /QAT1.6/build/icp_qa_al.ko
>
>
>
> When I issue the below command to start the acceleration software, it fails
> to start the driver.
>
>
> [root@test1 ~]# service qat_service start
>
> Processing file: /etc/dh895xcc_qa_dev0.conf
>
> ADF_CONFIG_CTL err: adf_set_bank_core_affinity: Invalid core affinity
> settings for accelerator 0 bank 8. Max cpu number is 3. Defaulting to 0
>
> ADF_CONFIG_CTL err: adf_set_bank_core_affinity: Invalid core affinity
> settings for accelerator 0 bank 9. Max cpu number is 3. Defaulting to 0
>
> Parity err reporting is disabled.
>
> Error: failed to start device 0
>
> There is 1 acceleration device(s) in the system:
>
> icp_dev0 - type=dh895xcc, inst_id=0, node_id=0, bdf=00:05:0, #accel=6,
> #engines=12, state=down
>
> [root@test1 ~]#
>
>
>
> Here are the kernel logs
>
>
>
> [root@test1 ~]# dmesg -c
>
> [ 3104.979666] Reading config file.
>
> [ 3104.980250] Starting acceleration device icp_dev0.
>
> [ 3104.980731] Resetting device icp_dev0
>
> [ 3105.126920] qat_1_6_adf 0000:00:05.0: irq 45 for MSI/MSI-X
>
> [ 3105.126939] qat_1_6_adf 0000:00:05.0: irq 46 for MSI/MSI-X
>
> [ 3105.126956] qat_1_6_adf 0000:00:05.0: irq 47 for MSI/MSI-X
>
> [ 3105.126973] qat_1_6_adf 0000:00:05.0: irq 48 for MSI/MSI-X
>
> [ 3105.126989] qat_1_6_adf 0000:00:05.0: irq 49 for MSI/MSI-X
>
> [ 3105.127029] qat_1_6_adf 0000:00:05.0: irq 50 for MSI/MSI-X
>
> [ 3105.127048] qat_1_6_adf 0000:00:05.0: irq 51 for MSI/MSI-X
>
> [ 3105.127065] qat_1_6_adf 0000:00:05.0: irq 52 for MSI/MSI-X
>
> [ 3105.127082] qat_1_6_adf 0000:00:05.0: irq 53 for MSI/MSI-X
>
> [ 3105.127098] qat_1_6_adf 0000:00:05.0: irq 54 for MSI/MSI-X
>
> [ 3105.127115] qat_1_6_adf 0000:00:05.0: irq 55 for MSI/MSI-X
>
> [ 3105.127131] qat_1_6_adf 0000:00:05.0: irq 56 for MSI/MSI-X
>
> [ 3105.127148] qat_1_6_adf 0000:00:05.0: irq 57 for MSI/MSI-X
>
> [ 3105.127164] qat_1_6_adf 0000:00:05.0: irq 58 for MSI/MSI-X
>
> [ 3105.127181] qat_1_6_adf 0000:00:05.0: irq 59 for MSI/MSI-X
>
> [ 3105.127198] qat_1_6_adf 0000:00:05.0: irq 60 for MSI/MSI-X
>
> [ 3105.127215] qat_1_6_adf 0000:00:05.0: irq 61 for MSI/MSI-X
>
> [ 3105.127231] qat_1_6_adf 0000:00:05.0: irq 62 for MSI/MSI-X
>
> [ 3105.127248] qat_1_6_adf 0000:00:05.0: irq 63 for MSI/MSI-X
>
> [ 3105.127265] qat_1_6_adf 0000:00:05.0: irq 64 for MSI/MSI-X
>
> [ 3105.127282] qat_1_6_adf 0000:00:05.0: irq 65 for MSI/MSI-X
>
> [ 3105.127298] qat_1_6_adf 0000:00:05.0: irq 66 for MSI/MSI-X
>
> [ 3105.127315] qat_1_6_adf 0000:00:05.0: irq 67 for MSI/MSI-X
>
> [ 3105.127331] qat_1_6_adf 0000:00:05.0: irq 68 for MSI/MSI-X
>
> [ 3105.127348] qat_1_6_adf 0000:00:05.0: irq 69 for MSI/MSI-X
>
> [ 3105.127364] qat_1_6_adf 0000:00:05.0: irq 70 for MSI/MSI-X
>
> [ 3105.127380] qat_1_6_adf 0000:00:05.0: irq 71 for MSI/MSI-X
>
> [ 3105.127397] qat_1_6_adf 0000:00:05.0: irq 72 for MSI/MSI-X
>
> [ 3105.127413] qat_1_6_adf 0000:00:05.0: irq 73 for MSI/MSI-X
>
> [ 3105.127429] qat_1_6_adf 0000:00:05.0: irq 74 for MSI/MSI-X
>
> [ 3105.127446] qat_1_6_adf 0000:00:05.0: irq 75 for MSI/MSI-X
>
> [ 3105.127462] qat_1_6_adf 0000:00:05.0: irq 76 for MSI/MSI-X
>
> [ 3105.127487] qat_1_6_adf 0000:00:05.0: irq 77 for MSI/MSI-X
>
> [ 3105.157973] icp_qa_al err: adf_subsystemInit: Invalid SRIOV_Enabled
> value, driver is built with SRIOV.
>
> [ 3105.158891] icp_qa_al err: adf_do_init: adf_subsystemInit error,
> stopping and shutting down
>
> [ 3106.162962] icp_qa_al err: adf_init_all: failed to initialize
> accelerator
>
> [ 3106.163670] icp_qa_al err: adf_init_devices: failed init accel 0
>
> [ 3106.164256] icp_qa_al err: adf_ctl_ioctl_dev_start: failed call to
> adf_init_devices
>
> [root@test1 ~]#
>
>
>
> Can anyone please let me know what might be the issue and its remedy ?
> please feel free to correct me if I am wrong. If it is not the right forum
> , please let me know the correct one. Thank you in advance for time and
> support.
>
>
>
> Regards,
>
> Chinmaya
>
--
Thanks
Sakthivel S OM
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [dpdk-users] Issue with Physical Function passthrough on VM Guest (Fedora 20) using qatmux.l.2.6.0-60
2016-06-10 7:33 ` SAKTHIVEL ANAND S
@ 2016-06-10 8:57 ` Chinmaya Dwibedy
0 siblings, 0 replies; 3+ messages in thread
From: Chinmaya Dwibedy @ 2016-06-10 8:57 UTC (permalink / raw)
To: SAKTHIVEL ANAND S; +Cc: users
Hi SAKTHIVEL,
Thank your for your response . On VM, added “intel_iommu=on” in
/boot/grub2/grub.cfg and rebooted the VM. Still getting the same issue
i.e., failed to start device. Here are the logs. I have done “export
ICP_SRIOV=0” to build driver without SRIOV. Please let me know if I am
missing anything.
[root@test1 QAT]# service qat_service start
Processing file: /etc/dh895xcc_qa_dev0.conf
ADF_CONFIG_CTL err: adf_set_bank_core_affinity: Invalid core affinity
settings for accelerator 0 bank 8. Max cpu number is 3. Defaulting to 0
ADF_CONFIG_CTL err: adf_set_bank_core_affinity: Invalid core affinity
settings for accelerator 0 bank 9. Max cpu number is 3. Defaulting to 0
Parity err reporting is disabled.
Error: failed to start device 0
There is 1 acceleration device(s) in the system:
icp_dev0 - type=dh895xcc, inst_id=0, node_id=0, bdf=00:05:0, #accel=6,
#engines=12, state=down
[root@test1 QAT]#
[root@test1 QAT]# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.12.9-301.fc20.x86_64
root=UUID=ab47cbc9-68ee-403c-96a5-184e68238e65 ro console=ttyS0,115200n8
intel_iommu=on
[root@test1 QAT]#
*Kernel logs (#dmesg )*
[ 693.776133] icp_qa_al err: adf_subsystemInit: Invalid SRIOV_Enabled
value, driver is built with SRIOV.
[ 693.777043] icp_qa_al err: adf_do_init: adf_subsystemInit error,
stopping and shutting down
[ 694.780830] icp_qa_al err: adf_init_all: failed to initialize accelerator
[ 694.781554] icp_qa_al err: adf_init_devices: failed init accel 0
[ 694.782153] icp_qa_al err: adf_ctl_ioctl_dev_start: failed call to
adf_init_devices
[root@test1 QAT]#
Regards,
Chinmaya
On Fri, Jun 10, 2016 at 1:03 PM, SAKTHIVEL ANAND S <anand.sa88@gmail.com>
wrote:
> If a failed to start device
> error is shown during the Acceleration
> installation, ensure that the kernel option
> intel_iommu=off
> has been
> configured as specified in
> Updating Grub Configuration File
> on page 13. If
> this kernel parameter is not specified, acceleration services are only
> available in a guest operating system.
>
> The above lines are from QAT official document.. so if u r using QAT in
> VM, you have to make intel_iommu=on in the grub config.
>
>
>
> On Fri, Jun 10, 2016 at 12:46 PM, Chinmaya Dwibedy <ckdwibedy@gmail.com>
> wrote:
>
>> Hi,
>>
>>
>> I have assigned Intel QAT device to VMs as a full device (PF - physical
>> function, PCI passthrough). Below is the output of lspci in host (Cent OS
>> 7.2).
>>
>>
>> [root@localhost nova(keystone_admin)]# lspci -nn | grep 0435
>>
>> 83:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [
>> 8086:0435]
>>
>> 88:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [
>> 8086:0435]
>>
>> [root@localhost nova(keystone_admin)]#
>>
>> [root@localhost ~]# lspci -vnd 8086:0435
>>
>> 83:00.0 0b40: 8086:0435
>>
>> Subsystem: 8086:35c5
>>
>> Physical Slot: 0-1
>>
>> Flags: bus master, fast devsel, latency 0, IRQ 35
>>
>> Memory at 387fffd00000 (64-bit, prefetchable) [size=512K]
>>
>> Memory at c8200000 (64-bit, non-prefetchable) [size=256K]
>>
>> Memory at c8240000 (64-bit, non-prefetchable) [size=256K]
>>
>> Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+
>>
>> Capabilities: [60] MSI-X: Enable- Count=33 Masked-
>>
>> Capabilities: [6c] Power Management version 3
>>
>> Capabilities: [74] Express Endpoint, MSI 00
>>
>> Capabilities: [100] Advanced Error Reporting
>>
>> Capabilities: [138] Alternative Routing-ID Interpretation (ARI)
>>
>> Capabilities: [140] Single Root I/O Virtualization (SR-IOV)
>>
>> Kernel driver in use: vfio-pci
>>
>>
>>
>> 88:00.0 0b40: 8086:0435
>>
>> Subsystem: 8086:35c5
>>
>> Physical Slot: 0-2
>>
>> Flags: fast devsel, IRQ 39
>>
>> Memory at 387fffe00000 (64-bit, prefetchable) [size=512K]
>>
>> Memory at c8000000 (64-bit, non-prefetchable) [size=256K]
>>
>> Memory at c8040000 (64-bit, non-prefetchable) [size=256K]
>>
>> Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+
>>
>> Capabilities: [60] MSI-X: Enable- Count=33 Masked-
>>
>> Capabilities: [6c] Power Management version 3
>>
>> Capabilities: [74] Express Endpoint, MSI 00
>>
>> Capabilities: [100] Advanced Error Reporting
>>
>> Capabilities: [138] Alternative Routing-ID Interpretation (ARI)
>>
>> Capabilities: [140] Single Root I/O Virtualization (SR-IOV)
>>
>> Kernel driver in use: vfio-pci
>>
>>
>>
>> [root@localhost ~]#
>>
>>
>> In VM, the Intel QAT device shows up in lspci as "Intel Corporation
>> Coleto
>> Creek PCIe Endpoint".
>>
>>
>> [root@test1 QAT]# lspci -nn | grep 0435
>>
>> 00:05.0 Co-processor [0b40]: Intel Corporation Coleto Creek PCIe Endpoint
>> [8086:0435]
>>
>> [root@test1 QAT]#
>>
>>
>> Afterward I installed Intel® QuickAssist Technology Software i.e.,
>> qatmux.l.2.6.0-60.tgz ( downloaded at
>>
>> https://01.org/packet-processing/intel%C2%AE-quickassist-technology-drivers-and-patches
>> )
>> on the Guest VM . Here are the details of VM.
>>
>> 1) Fedora release 20 (Heisenbug).
>>
>> 2) Linux Kernel version: 3.12.9-301.fc20.x86_64.
>>
>> 3) BOOT_IMAGE=/boot/vmlinuz-3.12.9-301.fc20.x86_64
>> root=UUID=ab47cbc9-68ee-403c-96a5-184e68238e65 ro console=ttyS0,115200n8
>> intel_iommu=off
>>
>>
>>
>> *Notes: *
>>
>>
>> 1) We are not using SR-IOV and are instead passing through a PF for
>> acceleration services on guest only. Hence we have not installed the
>> Intel® QuickAssist Technology Software package on the host.
>>
>> 2) Found an Intel QAT driver with this kernel . Upon doing #lsmod |
>> grep qa, showed icp_qa_al_vf. Thus did rmmod icp_qa_al_vf and service
>> qat_service shutdown. This was done before building Intel QuickAssist
>> Technology Software package.
>>
>>
>>
>> For passthrough of PF, opted the below Install Acceleration option
>>
>> a) export ICP_SRIOV=0
>>
>> b) export ICP_WITHOUT_IOMMU=1
>>
>> c) ./installer.sh build_accel QAT1.6
>>
>> d) ./installer.sh install QAT1.6 guest
>>
>> e) insmod /root/QAT /QAT1.6/build/icp_qa_al.ko
>>
>>
>>
>> When I issue the below command to start the acceleration software, it
>> fails
>> to start the driver.
>>
>>
>> [root@test1 ~]# service qat_service start
>>
>> Processing file: /etc/dh895xcc_qa_dev0.conf
>>
>> ADF_CONFIG_CTL err: adf_set_bank_core_affinity: Invalid core affinity
>> settings for accelerator 0 bank 8. Max cpu number is 3. Defaulting to 0
>>
>> ADF_CONFIG_CTL err: adf_set_bank_core_affinity: Invalid core affinity
>> settings for accelerator 0 bank 9. Max cpu number is 3. Defaulting to 0
>>
>> Parity err reporting is disabled.
>>
>> Error: failed to start device 0
>>
>> There is 1 acceleration device(s) in the system:
>>
>> icp_dev0 - type=dh895xcc, inst_id=0, node_id=0, bdf=00:05:0, #accel=6,
>> #engines=12, state=down
>>
>> [root@test1 ~]#
>>
>>
>>
>> Here are the kernel logs
>>
>>
>>
>> [root@test1 ~]# dmesg -c
>>
>> [ 3104.979666] Reading config file.
>>
>> [ 3104.980250] Starting acceleration device icp_dev0.
>>
>> [ 3104.980731] Resetting device icp_dev0
>>
>> [ 3105.126920] qat_1_6_adf 0000:00:05.0: irq 45 for MSI/MSI-X
>>
>> [ 3105.126939] qat_1_6_adf 0000:00:05.0: irq 46 for MSI/MSI-X
>>
>> [ 3105.126956] qat_1_6_adf 0000:00:05.0: irq 47 for MSI/MSI-X
>>
>> [ 3105.126973] qat_1_6_adf 0000:00:05.0: irq 48 for MSI/MSI-X
>>
>> [ 3105.126989] qat_1_6_adf 0000:00:05.0: irq 49 for MSI/MSI-X
>>
>> [ 3105.127029] qat_1_6_adf 0000:00:05.0: irq 50 for MSI/MSI-X
>>
>> [ 3105.127048] qat_1_6_adf 0000:00:05.0: irq 51 for MSI/MSI-X
>>
>> [ 3105.127065] qat_1_6_adf 0000:00:05.0: irq 52 for MSI/MSI-X
>>
>> [ 3105.127082] qat_1_6_adf 0000:00:05.0: irq 53 for MSI/MSI-X
>>
>> [ 3105.127098] qat_1_6_adf 0000:00:05.0: irq 54 for MSI/MSI-X
>>
>> [ 3105.127115] qat_1_6_adf 0000:00:05.0: irq 55 for MSI/MSI-X
>>
>> [ 3105.127131] qat_1_6_adf 0000:00:05.0: irq 56 for MSI/MSI-X
>>
>> [ 3105.127148] qat_1_6_adf 0000:00:05.0: irq 57 for MSI/MSI-X
>>
>> [ 3105.127164] qat_1_6_adf 0000:00:05.0: irq 58 for MSI/MSI-X
>>
>> [ 3105.127181] qat_1_6_adf 0000:00:05.0: irq 59 for MSI/MSI-X
>>
>> [ 3105.127198] qat_1_6_adf 0000:00:05.0: irq 60 for MSI/MSI-X
>>
>> [ 3105.127215] qat_1_6_adf 0000:00:05.0: irq 61 for MSI/MSI-X
>>
>> [ 3105.127231] qat_1_6_adf 0000:00:05.0: irq 62 for MSI/MSI-X
>>
>> [ 3105.127248] qat_1_6_adf 0000:00:05.0: irq 63 for MSI/MSI-X
>>
>> [ 3105.127265] qat_1_6_adf 0000:00:05.0: irq 64 for MSI/MSI-X
>>
>> [ 3105.127282] qat_1_6_adf 0000:00:05.0: irq 65 for MSI/MSI-X
>>
>> [ 3105.127298] qat_1_6_adf 0000:00:05.0: irq 66 for MSI/MSI-X
>>
>> [ 3105.127315] qat_1_6_adf 0000:00:05.0: irq 67 for MSI/MSI-X
>>
>> [ 3105.127331] qat_1_6_adf 0000:00:05.0: irq 68 for MSI/MSI-X
>>
>> [ 3105.127348] qat_1_6_adf 0000:00:05.0: irq 69 for MSI/MSI-X
>>
>> [ 3105.127364] qat_1_6_adf 0000:00:05.0: irq 70 for MSI/MSI-X
>>
>> [ 3105.127380] qat_1_6_adf 0000:00:05.0: irq 71 for MSI/MSI-X
>>
>> [ 3105.127397] qat_1_6_adf 0000:00:05.0: irq 72 for MSI/MSI-X
>>
>> [ 3105.127413] qat_1_6_adf 0000:00:05.0: irq 73 for MSI/MSI-X
>>
>> [ 3105.127429] qat_1_6_adf 0000:00:05.0: irq 74 for MSI/MSI-X
>>
>> [ 3105.127446] qat_1_6_adf 0000:00:05.0: irq 75 for MSI/MSI-X
>>
>> [ 3105.127462] qat_1_6_adf 0000:00:05.0: irq 76 for MSI/MSI-X
>>
>> [ 3105.127487] qat_1_6_adf 0000:00:05.0: irq 77 for MSI/MSI-X
>>
>> [ 3105.157973] icp_qa_al err: adf_subsystemInit: Invalid SRIOV_Enabled
>> value, driver is built with SRIOV.
>>
>> [ 3105.158891] icp_qa_al err: adf_do_init: adf_subsystemInit error,
>> stopping and shutting down
>>
>> [ 3106.162962] icp_qa_al err: adf_init_all: failed to initialize
>> accelerator
>>
>> [ 3106.163670] icp_qa_al err: adf_init_devices: failed init accel 0
>>
>> [ 3106.164256] icp_qa_al err: adf_ctl_ioctl_dev_start: failed call to
>> adf_init_devices
>>
>> [root@test1 ~]#
>>
>>
>>
>> Can anyone please let me know what might be the issue and its remedy ?
>> please feel free to correct me if I am wrong. If it is not the right
>> forum
>> , please let me know the correct one. Thank you in advance for time and
>> support.
>>
>>
>>
>> Regards,
>>
>> Chinmaya
>>
>
>
>
> --
> Thanks
> Sakthivel S OM
>
>
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2016-06-10 8:57 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-06-10 7:16 [dpdk-users] Issue with Physical Function passthrough on VM Guest (Fedora 20) using qatmux.l.2.6.0-60 Chinmaya Dwibedy
2016-06-10 7:33 ` SAKTHIVEL ANAND S
2016-06-10 8:57 ` Chinmaya Dwibedy
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).