vfio module crash when i used dpdk secondary process send pkts

vfio module crash when i used dpdk secondary process send pkts

[jinag]

[-- Attachment #1: Type: text/plain, Size: 1288 bytes --]

when I use dpdk to call rte_eth_tx_burst function for sending data from the secondary process, vfio will crash:


PID: 60699 TASK: ffff8f0152235df00 CPU: 14  COMMAND: "testlstack02"
 #0 [ffffa7d8cecc39a8] machine_kexec at ffffffff9045d67b
 #1 [ffffa7d8ceec3a00] __crash_kexec at ffffffff90562e92
 #2 [ffffa7d8ceec3ac0] panic at ffffffff904b9b79
 #3 [ffffa7d8ceec3b48] oops_end at ffffffff904231fc
 #4 [ffffa7d8ceec3b70] remap_pfn_range at ffffffff90664772
 #5 [ffffa7d8ceec3c58] remap_pfn_range at ffffffff90664772
 #6 [ffffa7d8ceec3da0] vfio_pci_mmap_fault at ffffffffc-bda821 [vfio_pci] 
 #7 [ffffa7d8ceec3dc0] __do_fault at ffffffff90662ee9
 #8 [ffffa7d8ceec3df0] do fault at ffffffff90663b2c
 #9 [ffffa7d8ceec3e90] __handle_mm_fault at ffffffff90663c9c
#10 [ffffa7d8ceec3ec0] handle_mm_fault at ffffffff9047103b
#11 [ffffa7d8ceec3ee0] __do_page_fault at ffffffff904712e1
#12 [ffffa7d8ceec3f20] do_page_fault at ffffffff9047122e
#13 [ffffa7d8ceec3f50] page_fault at ffffffff90e012de


The crash occurs when the secondary queue of the secondary process send pkts. I have tested that the primary process and the first queue of the secondary process do not crash. I use i40e X710 nic.


Has anyone ever encountered simlilar issue? please provide some ideas for fixing the issue.
Thanks!


[-- Attachment #2: Type: text/html, Size: 2087 bytes --]

Re: vfio module crash when i used dpdk secondary process send pkts

[Stephen Hemminger]

On Tue, 22 Aug 2023 09:33:39 +0800 (CST)
jinag <15720603159@163.com> wrote:

> when I use dpdk to call rte_eth_tx_burst function for sending data from the secondary process, vfio will crash:
> 
> 
> PID: 60699 TASK: ffff8f0152235df00 CPU: 14  COMMAND: "testlstack02"
>  #0 [ffffa7d8cecc39a8] machine_kexec at ffffffff9045d67b
>  #1 [ffffa7d8ceec3a00] __crash_kexec at ffffffff90562e92
>  #2 [ffffa7d8ceec3ac0] panic at ffffffff904b9b79
>  #3 [ffffa7d8ceec3b48] oops_end at ffffffff904231fc
>  #4 [ffffa7d8ceec3b70] remap_pfn_range at ffffffff90664772
>  #5 [ffffa7d8ceec3c58] remap_pfn_range at ffffffff90664772
>  #6 [ffffa7d8ceec3da0] vfio_pci_mmap_fault at ffffffffc-bda821 [vfio_pci] 
>  #7 [ffffa7d8ceec3dc0] __do_fault at ffffffff90662ee9
>  #8 [ffffa7d8ceec3df0] do fault at ffffffff90663b2c
>  #9 [ffffa7d8ceec3e90] __handle_mm_fault at ffffffff90663c9c
> #10 [ffffa7d8ceec3ec0] handle_mm_fault at ffffffff9047103b
> #11 [ffffa7d8ceec3ee0] __do_page_fault at ffffffff904712e1
> #12 [ffffa7d8ceec3f20] do_page_fault at ffffffff9047122e
> #13 [ffffa7d8ceec3f50] page_fault at ffffffff90e012de
> 
> 
> The crash occurs when the secondary queue of the secondary process send pkts. I have tested that the primary process and the first queue of the secondary process do not crash. I use i40e X710 nic.
> 
> 
> Has anyone ever encountered simlilar issue? please provide some ideas for fixing the issue.
> Thanks!
> 
If this is a kernel crash, it is a kernel bug. No matter what application does VFIO in kernel
should not panic.  What kernel version are you using? Is it an upstream long-term-stable kernel?

Re:Re: vfio module crash when i used dpdk secondary process send pkts

[jinag]

[-- Attachment #1: Type: text/plain, Size: 1993 bytes --]

This is definitely a bug in the kernel, I am looking to see if there's a bugfix patch in the kernel community.  but maybe the app passed an incorrect parameter that caused kernel crash?
The kernel version is 4.18.0-147.5.1.9, it is based on upstream linux at version 4.18.0
At 2023-08-23 20:41:38, "Stephen Hemminger" <stephen@networkplumber.org> wrote:
>On Tue, 22 Aug 2023 09:33:39 +0800 (CST)
>jinag <15720603159@163.com> wrote:
>
>> when I use dpdk to call rte_eth_tx_burst function for sending data from the secondary process, vfio will crash:
>> 
>> 
>> PID: 60699 TASK: ffff8f0152235df00 CPU: 14  COMMAND: "testlstack02"
>>  #0 [ffffa7d8cecc39a8] machine_kexec at ffffffff9045d67b
>>  #1 [ffffa7d8ceec3a00] __crash_kexec at ffffffff90562e92
>>  #2 [ffffa7d8ceec3ac0] panic at ffffffff904b9b79
>>  #3 [ffffa7d8ceec3b48] oops_end at ffffffff904231fc
>>  #4 [ffffa7d8ceec3b70] remap_pfn_range at ffffffff90664772
>>  #5 [ffffa7d8ceec3c58] remap_pfn_range at ffffffff90664772
>>  #6 [ffffa7d8ceec3da0] vfio_pci_mmap_fault at ffffffffc-bda821 [vfio_pci] 
>>  #7 [ffffa7d8ceec3dc0] __do_fault at ffffffff90662ee9
>>  #8 [ffffa7d8ceec3df0] do fault at ffffffff90663b2c
>>  #9 [ffffa7d8ceec3e90] __handle_mm_fault at ffffffff90663c9c
>> #10 [ffffa7d8ceec3ec0] handle_mm_fault at ffffffff9047103b
>> #11 [ffffa7d8ceec3ee0] __do_page_fault at ffffffff904712e1
>> #12 [ffffa7d8ceec3f20] do_page_fault at ffffffff9047122e
>> #13 [ffffa7d8ceec3f50] page_fault at ffffffff90e012de
>> 
>> 
>> The crash occurs when the secondary queue of the secondary process send pkts. I have tested that the primary process and the first queue of the secondary process do not crash. I use i40e X710 nic.
>> 
>> 
>> Has anyone ever encountered simlilar issue? please provide some ideas for fixing the issue.
>> Thanks!
>> 
>If this is a kernel crash, it is a kernel bug. No matter what application does VFIO in kernel
>should not panic.  What kernel version are you using? Is it an upstream long-term-stable kernel?

[-- Attachment #2: Type: text/html, Size: 2332 bytes --]

Re: vfio module crash know when i used dpdk secondary process send pkts

[wangpcmic]

[-- Attachment #1: Type: text/plain, Size: 1553 bytes --]

作业写完没.



| |
wangpcmic
|
|
邮箱：wangpcmic@163.com
|




---- Replied Message ----
| From | jinag<15720603159@163.com> |
| Date | 08/23/2023 19:20 |
| To | users@dpdk.org |
| Cc | |
| Subject | vfio module crash when i used dpdk secondary process send pkts |
when I use dpdk to call rte_eth_tx_burst function for sending data from the secondary process, vfio will crash:


PID: 60699 TASK: ffff8f0152235df00 CPU: 14  COMMAND: "testlstack02"
 #0 [ffffa7d8cecc39a8] machine_kexec at ffffffff9045d67b
 #1 [ffffa7d8ceec3a00] __crash_kexec at ffffffff90562e92
 #2 [ffffa7d8ceec3ac0] panic at ffffffff904b9b79
 #3 [ffffa7d8ceec3b48] oops_end at ffffffff904231fc
 #4 [ffffa7d8ceec3b70] remap_pfn_range at ffffffff90664772
 #5 [ffffa7d8ceec3c58] remap_pfn_range at ffffffff90664772
 #6 [ffffa7d8ceec3da0] vfio_pci_mmap_fault at ffffffffc-bda821 [vfio_pci] 
 #7 [ffffa7d8ceec3dc0] __do_fault at ffffffff90662ee9
 #8 [ffffa7d8ceec3df0] do fault at ffffffff90663b2c
 #9 [ffffa7d8ceec3e90] __handle_mm_fault at ffffffff90663c9c
#10 [ffffa7d8ceec3ec0] handle_mm_fault at ffffffff9047103b
#11 [ffffa7d8ceec3ee0] __do_page_fault at ffffffff904712e1
#12 [ffffa7d8ceec3f20] do_page_fault at ffffffff9047122e
#13 [ffffa7d8ceec3f50] page_fault at ffffffff90e012de


The crash occurs when the secondary queue of the secondary process send pkts. I have tested that the primary process and the first queue of the secondary process do not crash. I use i40e X710 nic.


Has anyone ever encountered simlilar issue? please provide some ideas for fixing the issue.
Thanks!


[-- Attachment #2: Type: text/html, Size: 6880 bytes --]

RE: vfio module crash know when i used dpdk secondary process send pkts

[Pathak, Pravin]

[-- Attachment #1: Type: text/plain, Size: 2077 bytes --]

I am not 100% sure if this might be the cause, but please try disabling Linux ASLR.

$  echo 0 | sudo tee /proc/sys/kernel/randomize_va_space


From: wangpcmic@163.com <wangpcmic@163.com>
Sent: Sunday, August 27, 2023 9:23 PM
To: 15720603159 <15720603159@163.com>
Cc: users <users@dpdk.org>
Subject: Re: vfio module crash know when i used dpdk secondary process send pkts

作业写完没.

[https://mail-online.nosdn.127.net/qiyelogo/defaultAvatar.png]
wangpcmic
邮箱：wangpcmic@163.com<mailto:wangpcmic@163.com>


---- Replied Message ----
From
jinag<15720603159@163.com><mailto:15720603159@163.com>
Date
08/23/2023 19:20
To
users@dpdk.org<mailto:users@dpdk.org>
Cc
Subject
vfio module crash when i used dpdk secondary process send pkts
when I use dpdk to call rte_eth_tx_burst function for sending data from the secondary process, vfio will crash:

PID: 60699 TASK: ffff8f0152235df00 CPU: 14  COMMAND: "testlstack02"
 #0 [ffffa7d8cecc39a8] machine_kexec at ffffffff9045d67b
 #1 [ffffa7d8ceec3a00] __crash_kexec at ffffffff90562e92
 #2 [ffffa7d8ceec3ac0] panic at ffffffff904b9b79
 #3 [ffffa7d8ceec3b48] oops_end at ffffffff904231fc
 #4 [ffffa7d8ceec3b70] remap_pfn_range at ffffffff90664772
 #5 [ffffa7d8ceec3c58] remap_pfn_range at ffffffff90664772
 #6 [ffffa7d8ceec3da0] vfio_pci_mmap_fault at ffffffffc-bda821 [vfio_pci]
 #7 [ffffa7d8ceec3dc0] __do_fault at ffffffff90662ee9
 #8 [ffffa7d8ceec3df0] do fault at ffffffff90663b2c
 #9 [ffffa7d8ceec3e90] __handle_mm_fault at ffffffff90663c9c
#10 [ffffa7d8ceec3ec0] handle_mm_fault at ffffffff9047103b
#11 [ffffa7d8ceec3ee0] __do_page_fault at ffffffff904712e1
#12 [ffffa7d8ceec3f20] do_page_fault at ffffffff9047122e
#13 [ffffa7d8ceec3f50] page_fault at ffffffff90e012de

The crash occurs when the secondary queue of the secondary process send pkts. I have tested that the primary process and the first queue of the secondary process do not crash. I use i40e X710 nic.

Has anyone ever encountered simlilar issue? please provide some ideas for fixing the issue.
Thanks!


[-- Attachment #2: Type: text/html, Size: 12555 bytes --]

Re: vfio module crash when i used dpdk secondary process send pkts

[Stephen Hemminger]

On Thu, 24 Aug 2023 09:06:44 +0800 (CST)
jinag  <15720603159@163.com> wrote:

> This is definitely a bug in the kernel, I am looking to see if there's a bugfix patch in the kernel community.  but maybe the app passed an incorrect parameter that caused kernel crash?
> The kernel version is 4.18.0-147.5.1.9, it is based on upstream linux at version 4.18.0

Is this RHEL?
4.18 was end of life in November 2018.

Re:Re: vfio module crash when i used dpdk secondary process send pkts

[jinag]

[-- Attachment #1: Type: text/plain, Size: 594 bytes --]

Hi .
The crash has been fixed through backport some patches.of vfio module.


Thanks a lot.




At 2023-09-06 07:25:18, "Stephen Hemminger" <stephen@networkplumber.org> wrote:
>On Thu, 24 Aug 2023 09:06:44 +0800 (CST)
>jinag  <15720603159@163.com> wrote:
>
>> This is definitely a bug in the kernel, I am looking to see if there's a bugfix patch in the kernel community.  but maybe the app passed an incorrect parameter that caused kernel crash?
>> The kernel version is 4.18.0-147.5.1.9, it is based on upstream linux at version 4.18.0
>
>Is this RHEL?
>4.18 was end of life in November 2018.

[-- Attachment #2: Type: text/html, Size: 882 bytes --]