[dpdk-dev] kernel panic when stop my test demo

[dpdk-dev] kernel panic when stop my test demo

[安宏奎]

[-- Attachment #1: Type: text/plain, Size: 2785 bytes --]

 

Hi ereryone, I have a problem when I stop my test demo with ‘CTRL+C’, and
then kernel panic.

 

Any suggestions? 

 

BTW, I’m using the version 1.4.1-7 in RHEL6.4, kernel version is 3.2.28

 

 

 

uio device registered with irq 33

general protection fault: 0000 [#1] SMP 

CPU 2 

Modules linked in: igb_uio(O) [last unloaded: scsi_wait_scan]

 

Pid: 6072, comm: test Tainted: G           O 3.2.28 #12 SinoGrid Inc.
WiseGrid ADC2000/To be filled by O.E.M.

RIP: 0010:[<ffffffff81553afe>]  [<ffffffff81553afe>]
sock_def_wakeup+0x19/0x3e

RSP: 0018:ffff88022fa37c48  EFLAGS: 00010206

RAX: 0bd943e9b9b967f6 RBX: ffff880235946d20 RCX: ffff8802365a25c8

RDX: ffffffff81553ae5 RSI: ffff8802365a25c8 RDI: 0bd943e9b9b967ee

RBP: ffff880235a460c0 R08: 0000000000000000 R09: 0000000000000000

R10: 0000000000000000 R11: ffff88023502d800 R12: ffff8802365a25c8

R13: ffff8802354c9cc0 R14: ffff88023528bc90 R15: ffff880235ee4a80

FS:  0000000000000000(0000) GS:ffff88023fd00000(0000) knlGS:0000000000000000

CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b

CR2: 0000003ec3413000 CR3: 0000000001a3f000 CR4: 00000000000406a0

DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000

DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

Process test (pid: 6072, threadinfo ffff88022fa36000, task ffff88023537e8e0)

Stack:

ffff880229908cc0 ffffffff814bb41b 0000000000000008 ffff8802365a25c8

ffff88023528bc80 ffff88023528bc80 0000000000000008 ffffffff8109aff3

0000000000000000 ffff88023528bc80 ffff880235022140 0000000000000000

Call Trace:

[<ffffffff814bb41b>] ? uio_release+0x2a/0x4a

[<ffffffff8109aff3>] ? fput+0xea/0x1a5

[<ffffffff81098418>] ? filp_close+0x59/0x61

[<ffffffff8102d03c>] ? put_files_struct+0x67/0xc5

[<ffffffff8102e64f>] ? do_exit+0x221/0x678

[<ffffffff8102eb19>] ? do_group_exit+0x73/0x9d

[<ffffffff81038558>] ? get_signal_to_deliver+0x406/0x422

[<ffffffff810015bd>] ? do_signal+0x36/0x620

[<ffffffff8104396a>] ? hrtimer_cancel+0xc/0x16

[<ffffffff81694dd5>] ? do_nanosleep+0x77/0xb0

[<ffffffff81043def>] ? hrtimer_nanosleep+0x65/0x108

[<ffffffff81043e26>] ? hrtimer_nanosleep+0x9c/0x108

[<ffffffff81043173>] ? update_rmtp+0x5d/0x5d

[<ffffffff81001bb4>] ? do_notify_resume+0xd/0x28

[<ffffffff8169631b>] ? int_signal+0x12/0x17

Code: ff ff c7 43 54 01 00 00 00 fe 43 50 5b e9 0b c3 ad ff 48 83 ec 08 48
8b bf b8 00 00 00 0f ae f0 31 c0 48 85 ff 74 0e 48 8d 47 08 <48> 39 47 08 0f
95 c0 0f b6 c0 85 c0 74 12 31 c9 31 d2 be 01 00 

RIP  [<ffffffff81553afe>] sock_def_wakeup+0x19/0x3e

RSP <ffff88022fa37c48>

---[ end trace 807c8a5f11d131ba ]---

Fixing recursive fault but reboot is needed!

[root@localhost DPDK]#


[-- Attachment #2: Type: text/html, Size: 7902 bytes --]

Re: [dpdk-dev] kernel panic when stop my test demo

[Stephen Hemminger]

Does your application have a signal handler and stop the device
on shutdown?  Since DPDK app's have raw access to the hardware it
is critical to stop the hardware on exit. Other wise the Ethernet
device can overwrite random memory.

Perhaps the igb_uio driver should also have a fail safe and
write to the status register to reset the device on close.

[dpdk-dev] 答复: kernel panic when stop my test demo

[安宏奎]

I see, thanks.


-----邮件原件-----
发件人: Stephen Hemminger [mailto:stephen@networkplumber.org] 
发送时间: 2013年8月14日 13:58
收件人: 安宏奎
抄送: dev@dpdk.org
主题: Re: [dpdk-dev] kernel panic when stop my test demo

Does your application have a signal handler and stop the device on shutdown?
Since DPDK app's have raw access to the hardware it is critical to stop the
hardware on exit. Other wise the Ethernet device can overwrite random
memory.

Perhaps the igb_uio driver should also have a fail safe and write to the
status register to reset the device on close.

Re: [dpdk-dev] kernel panic when stop my test demo

[Lilijun]

Hi Stephen and all,

I have a same problem as this older email describes on Aug 14, 2013.
Any help will be appreciated.

The details is shown as follows.
The key step implementation of my demo is:
1. Firstly, call rte_eal_init() to do some initialization.
2. Switch the driver of my Intel  82599 NIC from ixgbe.ko to igb_uio.ko
like tools/dpdk_nic_bind.py written in C source code.
3. Configure rte_dev and start it.
4. Do some rx/tx tests.
5. call rte_eth_dev_stop(dpdk_port_id) to stop the hardware as your history emails.
6. Switch the driver of the NIC from igb_uio.ko to ixgbe.ko.
7. Kill the demo using commands: kill -9.

Then kernel panics at random points when do something later.
One of them as follows:
general protection fault: 0000 [#1] SMP
 Modules linked in: tun igb_uio(OF) uio mlx4_ib ib_sa
task: ffff881fd0d6a220 ti: ffff881fd0cf8000 task.ti: ffff881fd0cf8000
RIP: 0010:[<ffffffff8183d85f>]  [<ffffffff8183d85f>] dcbnl_rtnl_policy+0x1b111f/0x1e3f70
RSP: 0000:ffff881fd0cf9c38  EFLAGS: 00013282
RAX: ffffffff819770a0 RBX: ffff881fe7b49fa0 RCX: 0000000000000001
RDX: ffffffff8183d851 RSI: ffff881fe7f20da1 RDI: ffffffff819770a0
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
R10: ffff881fe7f20da0 R11: ffff881fd0da0310 R12: ffff881fe811e6c0
R13: 0000000000000000 R14: ffff881fe6d14180 R15: ffff881fe7af4a20
FS:  0000000000000000(0000) GS:ffff88203fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f2702d3f9e0 CR3: 00000000018ce000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
 ffffffffa0235524 ffff881fd0da0300 0000000000000008 ffff881fe7f20da0
 ffff881fd0cf9ca0 ffffffff811b1329 ffff881fe7f20da0 ffff881fd0da0310
 ffff881fd0da0200 ffffffff81c9be80 ffff881fd0d6a9f0 ffff881fd0d6a220
Call Trace:
 [<ffffffffa0235524>] ? uio_release+0x34/0x60 [uio]
 [<ffffffff811b1329>] ? __fput+0xe9/0x270
 [<ffffffff811b15fe>] ? ____fput+0xe/0x10
 [<ffffffff810823e4>] ? task_work_run+0xc4/0xe0
 [<ffffffff81063deb>] ? do_exit+0x2cb/0xa60
 [<ffffffff815e965b>] ? _raw_spin_unlock_irqrestore+0x1b/0x40
 [<ffffffff810645ff>] ? do_group_exit+0x3f/0xa0
 [<ffffffff81074010>] ? get_signal_to_deliver+0x1d0/0x6e0
 [<ffffffffa0289224>] ? tun_chr_aio_read+0xa4/0xc0 [tun]
 [<ffffffff81012437>] ? do_signal+0x57/0x600
 [<ffffffff815f0ae0>] ? kprobe_flush_task+0xd0/0x170
 [<ffffffff81092e3a>] ? finish_task_switch+0x14a/0x170
 [<ffffffff81012a49>] ? do_notify_resume+0x69/0xb0
 [<ffffffff815e9c7c>] ? retint_signal+0x48/0x8c


Thanks,
Jerry

Re: [dpdk-dev] kernel panic when stop my test demo

[Lilijun]

Hi all,

After adding unmap uio resources operations in process signal handler functions,
An new error was found as follows:
Call Trace:
 [<ffffffffa01cd530>] uio_release+0x40/0x60 [uio]
 [<ffffffff811b1329>] __fput+0xe9/0x270
 [<ffffffff811b15fe>] ____fput+0xe/0x10
 [<ffffffff810823c7>] task_work_run+0xa7/0xe0
 [<ffffffff81012a77>] do_notify_resume+0x97/0xb0
 [<ffffffff815f2a92>] int_signal+0x12/0x17

The code for unmap uio resources is shown:
static void pci_dev_uio_unmap(struct rte_pci_device *pci_dev, uint8_t port_id)
{
        int i;

        RTE_LOG(INFO, EAL, "begin unmap port %d uio resource! \n", port_id);
        if (NULL == pci_dev)
        {
                RTE_LOG(ERR, EAL, "begin unmap port %d uio resource! \n", port_id);
                return;
        }

        for (i = 0; i != PCI_MAX_RESOURCE; i++)
        {
                /* skip empty BAR */
                if (0 == pci_dev->mem_resource[i].phys_addr)
                        continue;
                if (munmap(pci_dev->mem_resource[i].addr, pci_dev->mem_resource[i].len)
                                                                        == -1){
                        RTE_LOG(ERR, EAL, "Error with munmap\n");
                        return;
                }
        }
        if (close(pci_dev->intr_handle.fd) == -1){
                RTE_LOG(ERR, EAL, "Error closing interrupt handle\n");
                return;
        }
        pci_dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
        RTE_LOG(INFO, EAL, "unmap port %d uio resource successfully!\n", port_id);
}

Does anyone has some ideas?

Thanks for any help.
Jerry

On 2014/10/14 19:58, Lilijun wrote:
> Hi Stephen and all,
> 
> I have a same problem as this older email describes on Aug 14, 2013.
> Any help will be appreciated.
> 
> The details is shown as follows.
> The key step implementation of my demo is:
> 1. Firstly, call rte_eal_init() to do some initialization.
> 2. Switch the driver of my Intel  82599 NIC from ixgbe.ko to igb_uio.ko
> like tools/dpdk_nic_bind.py written in C source code.
> 3. Configure rte_dev and start it.
> 4. Do some rx/tx tests.
> 5. call rte_eth_dev_stop(dpdk_port_id) to stop the hardware as your history emails.
> 6. Switch the driver of the NIC from igb_uio.ko to ixgbe.ko.
> 7. Kill the demo using commands: kill -9.
> 
> Then kernel panics at random points when do something later.
> One of them as follows:
> general protection fault: 0000 [#1] SMP
>  Modules linked in: tun igb_uio(OF) uio mlx4_ib ib_sa
> task: ffff881fd0d6a220 ti: ffff881fd0cf8000 task.ti: ffff881fd0cf8000
> RIP: 0010:[<ffffffff8183d85f>]  [<ffffffff8183d85f>] dcbnl_rtnl_policy+0x1b111f/0x1e3f70
> RSP: 0000:ffff881fd0cf9c38  EFLAGS: 00013282
> RAX: ffffffff819770a0 RBX: ffff881fe7b49fa0 RCX: 0000000000000001
> RDX: ffffffff8183d851 RSI: ffff881fe7f20da1 RDI: ffffffff819770a0
> RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
> R10: ffff881fe7f20da0 R11: ffff881fd0da0310 R12: ffff881fe811e6c0
> R13: 0000000000000000 R14: ffff881fe6d14180 R15: ffff881fe7af4a20
> FS:  0000000000000000(0000) GS:ffff88203fc00000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 00007f2702d3f9e0 CR3: 00000000018ce000 CR4: 00000000000007e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Stack:
>  ffffffffa0235524 ffff881fd0da0300 0000000000000008 ffff881fe7f20da0
>  ffff881fd0cf9ca0 ffffffff811b1329 ffff881fe7f20da0 ffff881fd0da0310
>  ffff881fd0da0200 ffffffff81c9be80 ffff881fd0d6a9f0 ffff881fd0d6a220
> Call Trace:
>  [<ffffffffa0235524>] ? uio_release+0x34/0x60 [uio]
>  [<ffffffff811b1329>] ? __fput+0xe9/0x270
>  [<ffffffff811b15fe>] ? ____fput+0xe/0x10
>  [<ffffffff810823e4>] ? task_work_run+0xc4/0xe0
>  [<ffffffff81063deb>] ? do_exit+0x2cb/0xa60
>  [<ffffffff815e965b>] ? _raw_spin_unlock_irqrestore+0x1b/0x40
>  [<ffffffff810645ff>] ? do_group_exit+0x3f/0xa0
>  [<ffffffff81074010>] ? get_signal_to_deliver+0x1d0/0x6e0
>  [<ffffffffa0289224>] ? tun_chr_aio_read+0xa4/0xc0 [tun]
>  [<ffffffff81012437>] ? do_signal+0x57/0x600
>  [<ffffffff815f0ae0>] ? kprobe_flush_task+0xd0/0x170
>  [<ffffffff81092e3a>] ? finish_task_switch+0x14a/0x170
>  [<ffffffff81012a49>] ? do_notify_resume+0x69/0xb0
>  [<ffffffff815e9c7c>] ? retint_signal+0x48/0x8c
> 
> 
> Thanks,
> Jerry
> 
> 
> .
> 

Re: [dpdk-dev] kernel panic when stop my test demo

[Richardson, Bruce]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Lilijun
> Sent: Wednesday, October 15, 2014 10:43 AM
> To: dev@dpdk.org; stephen@networkplumber.org
> Subject: Re: [dpdk-dev] kernel panic when stop my test demo
> 
> Hi all,
> 
> After adding unmap uio resources operations in process signal handler functions,
> An new error was found as follows:
> Call Trace:
>  [<ffffffffa01cd530>] uio_release+0x40/0x60 [uio]
>  [<ffffffff811b1329>] __fput+0xe9/0x270
>  [<ffffffff811b15fe>] ____fput+0xe/0x10
>  [<ffffffff810823c7>] task_work_run+0xa7/0xe0
>  [<ffffffff81012a77>] do_notify_resume+0x97/0xb0
>  [<ffffffff815f2a92>] int_signal+0x12/0x17
> 
> The code for unmap uio resources is shown:
> static void pci_dev_uio_unmap(struct rte_pci_device *pci_dev, uint8_t port_id)
> {
>         int i;
> 
>         RTE_LOG(INFO, EAL, "begin unmap port %d uio resource! \n", port_id);
>         if (NULL == pci_dev)
>         {
>                 RTE_LOG(ERR, EAL, "begin unmap port %d uio resource! \n", port_id);
>                 return;
>         }
> 
>         for (i = 0; i != PCI_MAX_RESOURCE; i++)
>         {
>                 /* skip empty BAR */
>                 if (0 == pci_dev->mem_resource[i].phys_addr)
>                         continue;
>                 if (munmap(pci_dev->mem_resource[i].addr, pci_dev-
> >mem_resource[i].len)
>                                                                         == -1){
>                         RTE_LOG(ERR, EAL, "Error with munmap\n");
>                         return;
>                 }
>         }
>         if (close(pci_dev->intr_handle.fd) == -1){
>                 RTE_LOG(ERR, EAL, "Error closing interrupt handle\n");
>                 return;
>         }
>         pci_dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
>         RTE_LOG(INFO, EAL, "unmap port %d uio resource successfully!\n",
> port_id);
> }
> 
> Does anyone has some ideas?
> 
> Thanks for any help.
> Jerry
> 
> On 2014/10/14 19:58, Lilijun wrote:
> > Hi Stephen and all,
> >
> > I have a same problem as this older email describes on Aug 14, 2013.
> > Any help will be appreciated.
> >
> > The details is shown as follows.
> > The key step implementation of my demo is:
> > 1. Firstly, call rte_eal_init() to do some initialization.
> > 2. Switch the driver of my Intel  82599 NIC from ixgbe.ko to igb_uio.ko
> > like tools/dpdk_nic_bind.py written in C source code.
> > 3. Configure rte_dev and start it.
> > 4. Do some rx/tx tests.
> > 5. call rte_eth_dev_stop(dpdk_port_id) to stop the hardware as your history
> emails.
> > 6. Switch the driver of the NIC from igb_uio.ko to ixgbe.ko.
> > 7. Kill the demo using commands: kill -9.

Just to clarify one point - you have an application running which was using the NICs with DPDK while you remove the uio driver and replace it with ixgbe? I would expect doing such a thing to cause problems as stopping the device does not cause the NIC BAR memory to be unmapped from the DPDK process. Therefore removing the driver providing that memory map and getting another driver to start using those same BARs would not be recommended.

/Bruce

Re: [dpdk-dev] kernel panic when stop my test demo

[Lilijun]

On 2014/10/15 18:08, Richardson, Bruce wrote:
> 
> 
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Lilijun
>> Sent: Wednesday, October 15, 2014 10:43 AM
>> To: dev@dpdk.org; stephen@networkplumber.org
>> Subject: Re: [dpdk-dev] kernel panic when stop my test demo
>>
>> Hi all,
>>
>> After adding unmap uio resources operations in process signal handler functions,
>> An new error was found as follows:
>> Call Trace:
>>  [<ffffffffa01cd530>] uio_release+0x40/0x60 [uio]
>>  [<ffffffff811b1329>] __fput+0xe9/0x270
>>  [<ffffffff811b15fe>] ____fput+0xe/0x10
>>  [<ffffffff810823c7>] task_work_run+0xa7/0xe0
>>  [<ffffffff81012a77>] do_notify_resume+0x97/0xb0
>>  [<ffffffff815f2a92>] int_signal+0x12/0x17
>>
>> The code for unmap uio resources is shown:
>> static void pci_dev_uio_unmap(struct rte_pci_device *pci_dev, uint8_t port_id)
>> {
>>         int i;
>>
>>         RTE_LOG(INFO, EAL, "begin unmap port %d uio resource! \n", port_id);
>>         if (NULL == pci_dev)
>>         {
>>                 RTE_LOG(ERR, EAL, "begin unmap port %d uio resource! \n", port_id);
>>                 return;
>>         }
>>
>>         for (i = 0; i != PCI_MAX_RESOURCE; i++)
>>         {
>>                 /* skip empty BAR */
>>                 if (0 == pci_dev->mem_resource[i].phys_addr)
>>                         continue;
>>                 if (munmap(pci_dev->mem_resource[i].addr, pci_dev-
>>> mem_resource[i].len)
>>                                                                         == -1){
>>                         RTE_LOG(ERR, EAL, "Error with munmap\n");
>>                         return;
>>                 }
>>         }
>>         if (close(pci_dev->intr_handle.fd) == -1){
>>                 RTE_LOG(ERR, EAL, "Error closing interrupt handle\n");
>>                 return;
>>         }
>>         pci_dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
>>         RTE_LOG(INFO, EAL, "unmap port %d uio resource successfully!\n",
>> port_id);
>> }
>>
>> Does anyone has some ideas?
>>
>> Thanks for any help.
>> Jerry
>>
>> On 2014/10/14 19:58, Lilijun wrote:
>>> Hi Stephen and all,
>>>
>>> I have a same problem as this older email describes on Aug 14, 2013.
>>> Any help will be appreciated.
>>>
>>> The details is shown as follows.
>>> The key step implementation of my demo is:
>>> 1. Firstly, call rte_eal_init() to do some initialization.
>>> 2. Switch the driver of my Intel  82599 NIC from ixgbe.ko to igb_uio.ko
>>> like tools/dpdk_nic_bind.py written in C source code.
>>> 3. Configure rte_dev and start it.
>>> 4. Do some rx/tx tests.
>>> 5. call rte_eth_dev_stop(dpdk_port_id) to stop the hardware as your history
>> emails.
>>> 6. Switch the driver of the NIC from igb_uio.ko to ixgbe.ko.
>>> 7. Kill the demo using commands: kill -9.
> 
> Just to clarify one point - you have an application running which was using the NICs with DPDK while you remove the uio driver and replace it with ixgbe? 
I would expect doing such a thing to cause problems as stopping the device does not cause the NIC BAR memory to be unmapped from the DPDK process.
Therefore removing the driver providing that memory map and getting another driver to start using those same BARs would not be recommended.
> 

Thanks for your reply.
Yes, I want to change the NIC driver by replacing the uio driver with ixgbe in order to recover the NIC to origin kernel ether-net devices while keeping the application running.
Then my application can use the NICs with DPDK or with kernel ixgbe driver on demand.
I am confusing with how to release all uio resources when stop my application.

Would you like to give me any suggestions for my requirements?

> /Bruce
> 
> 
> .
> 

Re: [dpdk-dev] kernel panic when stop my test demo

[Richardson, Bruce]

> -----Original Message-----
> From: Lilijun [mailto:jerry.lilijun@huawei.com]
> Sent: Thursday, October 16, 2014 3:40 AM
> To: Richardson, Bruce; dev@dpdk.org; stephen@networkplumber.org
> Subject: Re: [dpdk-dev] kernel panic when stop my test demo
> 
> On 2014/10/15 18:08, Richardson, Bruce wrote:
> >
> >
> >> -----Original Message-----
> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Lilijun
> >> Sent: Wednesday, October 15, 2014 10:43 AM
> >> To: dev@dpdk.org; stephen@networkplumber.org
> >> Subject: Re: [dpdk-dev] kernel panic when stop my test demo
> >>
> >> Hi all,
> >>
> >> After adding unmap uio resources operations in process signal handler
> functions,
> >> An new error was found as follows:
> >> Call Trace:
> >>  [<ffffffffa01cd530>] uio_release+0x40/0x60 [uio]
> >>  [<ffffffff811b1329>] __fput+0xe9/0x270
> >>  [<ffffffff811b15fe>] ____fput+0xe/0x10
> >>  [<ffffffff810823c7>] task_work_run+0xa7/0xe0
> >>  [<ffffffff81012a77>] do_notify_resume+0x97/0xb0
> >>  [<ffffffff815f2a92>] int_signal+0x12/0x17
> >>
> >> The code for unmap uio resources is shown:
> >> static void pci_dev_uio_unmap(struct rte_pci_device *pci_dev, uint8_t
> port_id)
> >> {
> >>         int i;
> >>
> >>         RTE_LOG(INFO, EAL, "begin unmap port %d uio resource! \n", port_id);
> >>         if (NULL == pci_dev)
> >>         {
> >>                 RTE_LOG(ERR, EAL, "begin unmap port %d uio resource! \n",
> port_id);
> >>                 return;
> >>         }
> >>
> >>         for (i = 0; i != PCI_MAX_RESOURCE; i++)
> >>         {
> >>                 /* skip empty BAR */
> >>                 if (0 == pci_dev->mem_resource[i].phys_addr)
> >>                         continue;
> >>                 if (munmap(pci_dev->mem_resource[i].addr, pci_dev-
> >>> mem_resource[i].len)
> >>                                                                         == -1){
> >>                         RTE_LOG(ERR, EAL, "Error with munmap\n");
> >>                         return;
> >>                 }
> >>         }
> >>         if (close(pci_dev->intr_handle.fd) == -1){
> >>                 RTE_LOG(ERR, EAL, "Error closing interrupt handle\n");
> >>                 return;
> >>         }
> >>         pci_dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
> >>         RTE_LOG(INFO, EAL, "unmap port %d uio resource successfully!\n",
> >> port_id);
> >> }
> >>
> >> Does anyone has some ideas?
> >>
> >> Thanks for any help.
> >> Jerry
> >>
> >> On 2014/10/14 19:58, Lilijun wrote:
> >>> Hi Stephen and all,
> >>>
> >>> I have a same problem as this older email describes on Aug 14, 2013.
> >>> Any help will be appreciated.
> >>>
> >>> The details is shown as follows.
> >>> The key step implementation of my demo is:
> >>> 1. Firstly, call rte_eal_init() to do some initialization.
> >>> 2. Switch the driver of my Intel  82599 NIC from ixgbe.ko to igb_uio.ko
> >>> like tools/dpdk_nic_bind.py written in C source code.
> >>> 3. Configure rte_dev and start it.
> >>> 4. Do some rx/tx tests.
> >>> 5. call rte_eth_dev_stop(dpdk_port_id) to stop the hardware as your history
> >> emails.
> >>> 6. Switch the driver of the NIC from igb_uio.ko to ixgbe.ko.
> >>> 7. Kill the demo using commands: kill -9.
> >
> > Just to clarify one point - you have an application running which was using the
> NICs with DPDK while you remove the uio driver and replace it with ixgbe?
> I would expect doing such a thing to cause problems as stopping the device does
> not cause the NIC BAR memory to be unmapped from the DPDK process.
> Therefore removing the driver providing that memory map and getting another
> driver to start using those same BARs would not be recommended.
> >
> 
> Thanks for your reply.
> Yes, I want to change the NIC driver by replacing the uio driver with ixgbe in
> order to recover the NIC to origin kernel ether-net devices while keeping the
> application running.
> Then my application can use the NICs with DPDK or with kernel ixgbe driver on
> demand.
> I am confusing with how to release all uio resources when stop my application.
> 
> Would you like to give me any suggestions for my requirements?
> 

Right now, there is no way to do this without changing the internals of the DPDK itself. The BARs from the NIC are mmapped permanently into the processes address space on initialization of the application, and are never released. You'd basically need to write code to un-initialize the DPDK and then reinitialize it at a later point.
Might an alternative be to actually have two separate applications or binaries that appear as one, or work as one? Then you could shut down the dpdk binary before removing the uio driver, and switch over to the ixgbe driver and use the other application. However, I realise that getting a seamless transition could be difficult there.

/Bruce