Re: [dpdk-dev] kernel panic when stop my test demo

["Richardson, Bruce" <bruce.richardson@intel.com>]

> -----Original Message-----
> From: Lilijun [mailto:jerry.lilijun@huawei.com]
> Sent: Thursday, October 16, 2014 3:40 AM
> To: Richardson, Bruce; dev@dpdk.org; stephen@networkplumber.org
> Subject: Re: [dpdk-dev] kernel panic when stop my test demo
> 
> On 2014/10/15 18:08, Richardson, Bruce wrote:
> >
> >
> >> -----Original Message-----
> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Lilijun
> >> Sent: Wednesday, October 15, 2014 10:43 AM
> >> To: dev@dpdk.org; stephen@networkplumber.org
> >> Subject: Re: [dpdk-dev] kernel panic when stop my test demo
> >>
> >> Hi all,
> >>
> >> After adding unmap uio resources operations in process signal handler
> functions,
> >> An new error was found as follows:
> >> Call Trace:
> >>  [<ffffffffa01cd530>] uio_release+0x40/0x60 [uio]
> >>  [<ffffffff811b1329>] __fput+0xe9/0x270
> >>  [<ffffffff811b15fe>] ____fput+0xe/0x10
> >>  [<ffffffff810823c7>] task_work_run+0xa7/0xe0
> >>  [<ffffffff81012a77>] do_notify_resume+0x97/0xb0
> >>  [<ffffffff815f2a92>] int_signal+0x12/0x17
> >>
> >> The code for unmap uio resources is shown:
> >> static void pci_dev_uio_unmap(struct rte_pci_device *pci_dev, uint8_t
> port_id)
> >> {
> >>         int i;
> >>
> >>         RTE_LOG(INFO, EAL, "begin unmap port %d uio resource! \n", port_id);
> >>         if (NULL == pci_dev)
> >>         {
> >>                 RTE_LOG(ERR, EAL, "begin unmap port %d uio resource! \n",
> port_id);
> >>                 return;
> >>         }
> >>
> >>         for (i = 0; i != PCI_MAX_RESOURCE; i++)
> >>         {
> >>                 /* skip empty BAR */
> >>                 if (0 == pci_dev->mem_resource[i].phys_addr)
> >>                         continue;
> >>                 if (munmap(pci_dev->mem_resource[i].addr, pci_dev-
> >>> mem_resource[i].len)
> >>                                                                         == -1){
> >>                         RTE_LOG(ERR, EAL, "Error with munmap\n");
> >>                         return;
> >>                 }
> >>         }
> >>         if (close(pci_dev->intr_handle.fd) == -1){
> >>                 RTE_LOG(ERR, EAL, "Error closing interrupt handle\n");
> >>                 return;
> >>         }
> >>         pci_dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
> >>         RTE_LOG(INFO, EAL, "unmap port %d uio resource successfully!\n",
> >> port_id);
> >> }
> >>
> >> Does anyone has some ideas?
> >>
> >> Thanks for any help.
> >> Jerry
> >>
> >> On 2014/10/14 19:58, Lilijun wrote:
> >>> Hi Stephen and all,
> >>>
> >>> I have a same problem as this older email describes on Aug 14, 2013.
> >>> Any help will be appreciated.
> >>>
> >>> The details is shown as follows.
> >>> The key step implementation of my demo is:
> >>> 1. Firstly, call rte_eal_init() to do some initialization.
> >>> 2. Switch the driver of my Intel  82599 NIC from ixgbe.ko to igb_uio.ko
> >>> like tools/dpdk_nic_bind.py written in C source code.
> >>> 3. Configure rte_dev and start it.
> >>> 4. Do some rx/tx tests.
> >>> 5. call rte_eth_dev_stop(dpdk_port_id) to stop the hardware as your history
> >> emails.
> >>> 6. Switch the driver of the NIC from igb_uio.ko to ixgbe.ko.
> >>> 7. Kill the demo using commands: kill -9.
> >
> > Just to clarify one point - you have an application running which was using the
> NICs with DPDK while you remove the uio driver and replace it with ixgbe?
> I would expect doing such a thing to cause problems as stopping the device does
> not cause the NIC BAR memory to be unmapped from the DPDK process.
> Therefore removing the driver providing that memory map and getting another
> driver to start using those same BARs would not be recommended.
> >
> 
> Thanks for your reply.
> Yes, I want to change the NIC driver by replacing the uio driver with ixgbe in
> order to recover the NIC to origin kernel ether-net devices while keeping the
> application running.
> Then my application can use the NICs with DPDK or with kernel ixgbe driver on
> demand.
> I am confusing with how to release all uio resources when stop my application.
> 
> Would you like to give me any suggestions for my requirements?
> 

Right now, there is no way to do this without changing the internals of the DPDK itself. The BARs from the NIC are mmapped permanently into the processes address space on initialization of the application, and are never released. You'd basically need to write code to un-initialize the DPDK and then reinitialize it at a later point.
Might an alternative be to actually have two separate applications or binaries that appear as one, or work as one? Then you could shut down the dpdk binary before removing the uio driver, and switch over to the ixgbe driver and use the other application. However, I realise that getting a seamless transition could be difficult there.

/Bruce