Re: [dpdk-users] Error in rte_eal_init using dpdk 16.04 and Qlogic (Broadcom) bnx2x driver in a SECONDARY process

Re: [dpdk-users] Error in rte_eal_init using dpdk 16.04 and Qlogic (Broadcom) bnx2x driver in a SECONDARY process

[dawid_jurek]

Hello David,

By looking on driver sources (bnx2x_ethdev.c) assertion `sc->bar[0].base_addr' fail is caused by pci_dev->mem_resource[0].addr == NULL. It means that MMIO mapping of region nr. 0 failed. 
The reason of this may be print reported on ring level some time before crash:

"EAL: memzone_reserve_aligned_thread_unsafe(): memzone <RG_MP_log_history>
already exists
RING: Cannot reserve memory."

I suspect some memory allocation issue but not 100% sure.

Regards,
Dawid


> Hello,
> 
> We are trying to run the
> "dpdk-16.04\examples\multi_process\client_server_mp" sample application 
> on a RHEL6.7 linux server, with DPDK 16.04 and the following network
> interfaces:
> 
> Network devices using DPDK-compatible driver
> ============================================
> 0000:10:00.0 'BCM57840 NetXtreme II 10 Gigabit Ethernet' drv=igb_uio
> unused=bnx2x
> 0000:10:00.1 'BCM57840 NetXtreme II 10 Gigabit Ethernet' drv=igb_uio
> unused=bnx2x
> ===================================
> 
> PMD: bnx2x_print_adapter_info():         ASIC : 0x168d1010
> PMD: bnx2x_print_adapter_info():          Rev : B1
> PMD: bnx2x_print_adapter_info():     Bus PCIe : 8,
> PMD: bnx2x_print_adapter_info():                5 Gbps
> PMD: bnx2x_print_adapter_info():        Flags :
> PMD: bnx2x_print_adapter_info():                MSI
> PMD: bnx2x_print_adapter_info(): |
> PMD: bnx2x_print_adapter_info():                MSI-X
> PMD: bnx2x_print_adapter_info():       Queues :
> PMD: bnx2x_print_adapter_info():                RSS : 1
> PMD: bnx2x_print_adapter_info():         DPDK : DPDK 16.04.0
> PMD: bnx2x_print_adapter_info():       Driver : BNX2X PMD 1.78.18_1.0.0
> PMD: bnx2x_print_adapter_info():     Firmware : 7.2.51
> PMD: bnx2x_print_adapter_info():     Bootcode : 7.13.24
> PMD: bnx2x_print_adapter_info():
> 
> We can run the server process, without problems, with the following command.
> 
>       ./mp_server -c 6 -n 4 -- -p 3 -n 1
> 
>  .than, in a second shell, we run the client process with the following
> command .
> 
>       ./mp_client -c 8 -n 4 --proc-type=auto -- -n 0
>       
> .and we always get this output:
> 
> EAL: Detected lcore 0 as core 0 on socket 0
> EAL: Detected lcore 1 as core 1 on socket 0
> EAL: Detected lcore 2 as core 2 on socket 0
> EAL: Detected lcore 3 as core 3 on socket 0
> EAL: Detected lcore 4 as core 4 on socket 0
> EAL: Detected lcore 5 as core 5 on socket 0
> EAL: Detected lcore 6 as core 0 on socket 1
> EAL: Detected lcore 7 as core 1 on socket 1
> EAL: Detected lcore 8 as core 2 on socket 1
> EAL: Detected lcore 9 as core 3 on socket 1
> EAL: Detected lcore 10 as core 4 on socket 1
> EAL: Detected lcore 11 as core 5 on socket 1
> EAL: Support maximum 128 logical core(s) by configuration.
> EAL: Detected 12 lcore(s)
> EAL: Auto-detected process type: SECONDARY
> EAL: Setting up physically contiguous memory...
> EAL: Analysing 16 files
> EAL: Mapped segment 0 of size 0x200000000
> EAL: Mapped segment 1 of size 0x200000000
> EAL: memzone_reserve_aligned_thread_unsafe(): memzone <RG_MP_log_history>
> already exists
> RING: Cannot reserve memory
> EAL: TSC frequency is ~2400000 KHz
> EAL: Master lcore 3 is ready (tid=f7fec880;cpuset=[3])
> PMD: rte_bnx2x_pmd_init():  >>
> PMD: rte_bnx2xvf_pmd_init():  >>
> EAL: PCI device 0000:10:00.0 on NUMA socket 0
> EAL:   probe driver: 14e4:16a1 rte_bnx2x_pmd
> EAL:   PCI memory mapped at 0x7fffc0000000
> EAL:   PCI memory mapped at 0x7fffc0800000
> EAL:   PCI memory mapped at 0x7fffc1000000
> PMD: eth_bnx2x_dev_init():  >>
> PMD: bnx2x_common_dev_init():  >>
> mp_client: /root/dpdk/dpdk-16.04/drivers/net/bnx2x/bnx2x_ethdev.c:430:
> bnx2x_common_dev_init: Assertion `sc->bar[0].base_addr' failed.
> Aborted (core dumped)
> 
> and this is the call stack from dump file:
> 
> (gdb) bt
> #0  0x000000363c032625 in raise () from /lib64/libc.so.6
> #1  0x000000363c033e05 in abort () from /lib64/libc.so.6
> #2  0x000000363c02b74e in __assert_fail_base () from /lib64/libc.so.6
> #3  0x000000363c02b810 in __assert_fail () from /lib64/libc.so.6
> #4  0x00000000004b8ed3 in bnx2x_common_dev_init ()
> #5  0x0000000000473bfa in rte_eth_dev_init ()
> #6  0x0000000000482422 in rte_eal_pci_probe ()
> #7  0x00000000004779b8 in rte_eal_init ()
> #8  0x0000000000420e7c in main ()
>       
> All this steps on different server with Intel NICs make the application
> (server side e client side) run with no problems at all.
> 
> Is there someone that could help us to understand this issue with Qlogic
> (Broadcom) driver?
> 
> Thanks in advance,
> David Coen
> 
> 
> 

[dpdk-users] R: Error in rte_eal_init using dpdk 16.04 and Qlogic (Broadcom) bnx2x driver in a SECONDARY process

[David Coen]

Hello Dawid,

thank you for your answer.

Probably you are right, but I think that the issue is one or more steps before.

Looking at Intel and Mellanox driver sources ("ixgbe_ethdev.c" and "mlx4.c" respectively), 
I found that, in the driver function called by rte_eal_init to initialize pci devs, is mandatory
to distinguish if the process is running as SECONDARY (--proc-type=secondary in EAL prameters).

For example:
eth_ixgbe_dev_init() function of the ixgbe Intel driver, at line 988 :

	/*
	 * For secondary processes, we don't initialise any further as primary
	 * has already done this work. Only check we don't need a different
	 * RX and TX function.
	 */
	if (rte_eal_process_type() != RTE_PROC_PRIMARY){
		... some code lines here ...
		return 0;
	}

While in Mellanox mlx4 driver there is a check in ALL driver functions that makes them return
immediatelly if the running process is of type "secondary".

That means that a driver has to take into account that the process could be a SECONDARY process...
but in Qlogic (Broadcom) bnx2x driver I didn't find anything about that.

Do you think I've missed something ?

Thank you for your help,

David

> -----Messaggio originale-----
> Da: dawid_jurek [mailto:dawid_jurek@vp.pl]
> Inviato: mercoledì 4 maggio 2016 21:08
> A: users@dpdk.org; David Coen
> Oggetto: Re: [dpdk-users] Error in rte_eal_init using dpdk 16.04 and Qlogic
> (Broadcom) bnx2x driver in a SECONDARY process
> 
> Hello David,
> 
> By looking on driver sources (bnx2x_ethdev.c) assertion `sc-
> >bar[0].base_addr' fail is caused by pci_dev->mem_resource[0].addr ==
> NULL. It means that MMIO mapping of region nr. 0 failed.
> The reason of this may be print reported on ring level some time before
> crash:
> 
> "EAL: memzone_reserve_aligned_thread_unsafe(): memzone
> <RG_MP_log_history> already exists
> RING: Cannot reserve memory."
> 
> I suspect some memory allocation issue but not 100% sure.
> 
> Regards,
> Dawid
> 
> 
> > Hello,
> >
> > We are trying to run the
> > "dpdk-16.04\examples\multi_process\client_server_mp" sample
> > application on a RHEL6.7 linux server, with DPDK 16.04 and the
> > following network
> > interfaces:
> >
> > Network devices using DPDK-compatible driver
> > ============================================
> > 0000:10:00.0 'BCM57840 NetXtreme II 10 Gigabit Ethernet' drv=igb_uio
> > unused=bnx2x
> > 0000:10:00.1 'BCM57840 NetXtreme II 10 Gigabit Ethernet' drv=igb_uio
> > unused=bnx2x ===================================
> >
> > PMD: bnx2x_print_adapter_info():         ASIC : 0x168d1010
> > PMD: bnx2x_print_adapter_info():          Rev : B1
> > PMD: bnx2x_print_adapter_info():     Bus PCIe : 8,
> > PMD: bnx2x_print_adapter_info():                5 Gbps
> > PMD: bnx2x_print_adapter_info():        Flags :
> > PMD: bnx2x_print_adapter_info():                MSI
> > PMD: bnx2x_print_adapter_info(): |
> > PMD: bnx2x_print_adapter_info():                MSI-X
> > PMD: bnx2x_print_adapter_info():       Queues :
> > PMD: bnx2x_print_adapter_info():                RSS : 1
> > PMD: bnx2x_print_adapter_info():         DPDK : DPDK 16.04.0
> > PMD: bnx2x_print_adapter_info():       Driver : BNX2X PMD 1.78.18_1.0.0
> > PMD: bnx2x_print_adapter_info():     Firmware : 7.2.51
> > PMD: bnx2x_print_adapter_info():     Bootcode : 7.13.24
> > PMD: bnx2x_print_adapter_info():
> >
> > We can run the server process, without problems, with the following
> command.
> >
> >       ./mp_server -c 6 -n 4 -- -p 3 -n 1
> >
> >  .than, in a second shell, we run the client process with the
> > following command .
> >
> >       ./mp_client -c 8 -n 4 --proc-type=auto -- -n 0
> >
> > .and we always get this output:
> >
> > EAL: Detected lcore 0 as core 0 on socket 0
> > EAL: Detected lcore 1 as core 1 on socket 0
> > EAL: Detected lcore 2 as core 2 on socket 0
> > EAL: Detected lcore 3 as core 3 on socket 0
> > EAL: Detected lcore 4 as core 4 on socket 0
> > EAL: Detected lcore 5 as core 5 on socket 0
> > EAL: Detected lcore 6 as core 0 on socket 1
> > EAL: Detected lcore 7 as core 1 on socket 1
> > EAL: Detected lcore 8 as core 2 on socket 1
> > EAL: Detected lcore 9 as core 3 on socket 1
> > EAL: Detected lcore 10 as core 4 on socket 1
> > EAL: Detected lcore 11 as core 5 on socket 1
> > EAL: Support maximum 128 logical core(s) by configuration.
> > EAL: Detected 12 lcore(s)
> > EAL: Auto-detected process type: SECONDARY
> > EAL: Setting up physically contiguous memory...
> > EAL: Analysing 16 files
> > EAL: Mapped segment 0 of size 0x200000000
> > EAL: Mapped segment 1 of size 0x200000000
> > EAL: memzone_reserve_aligned_thread_unsafe(): memzone
> > <RG_MP_log_history> already exists
> > RING: Cannot reserve memory
> > EAL: TSC frequency is ~2400000 KHz
> > EAL: Master lcore 3 is ready (tid=f7fec880;cpuset=[3])
> > PMD: rte_bnx2x_pmd_init():  >>
> > PMD: rte_bnx2xvf_pmd_init():  >>
> > EAL: PCI device 0000:10:00.0 on NUMA socket 0
> > EAL:   probe driver: 14e4:16a1 rte_bnx2x_pmd
> > EAL:   PCI memory mapped at 0x7fffc0000000
> > EAL:   PCI memory mapped at 0x7fffc0800000
> > EAL:   PCI memory mapped at 0x7fffc1000000
> > PMD: eth_bnx2x_dev_init():  >>
> > PMD: bnx2x_common_dev_init():  >>
> > mp_client: /root/dpdk/dpdk-
> 16.04/drivers/net/bnx2x/bnx2x_ethdev.c:430:
> > bnx2x_common_dev_init: Assertion `sc->bar[0].base_addr' failed.
> > Aborted (core dumped)
> >
> > and this is the call stack from dump file:
> >
> > (gdb) bt
> > #0  0x000000363c032625 in raise () from /lib64/libc.so.6
> > #1  0x000000363c033e05 in abort () from /lib64/libc.so.6
> > #2  0x000000363c02b74e in __assert_fail_base () from /lib64/libc.so.6
> > #3  0x000000363c02b810 in __assert_fail () from /lib64/libc.so.6
> > #4  0x00000000004b8ed3 in bnx2x_common_dev_init ()
> > #5  0x0000000000473bfa in rte_eth_dev_init ()
> > #6  0x0000000000482422 in rte_eal_pci_probe ()
> > #7  0x00000000004779b8 in rte_eal_init ()
> > #8  0x0000000000420e7c in main ()
> >
> > All this steps on different server with Intel NICs make the
> > application (server side e client side) run with no problems at all.
> >
> > Is there someone that could help us to understand this issue with
> > Qlogic
> > (Broadcom) driver?
> >
> > Thanks in advance,
> > David Coen
> >
> >
> >
> 
> 

Re: [dpdk-users] R: Error in rte_eal_init using dpdk 16.04 and Qlogic (Broadcom) bnx2x driver in a SECONDARY process

[Thomas Monjalon]

2016-05-05 13:02, David Coen:
> Looking at Intel and Mellanox driver sources ("ixgbe_ethdev.c" and "mlx4.c" respectively), 
> I found that, in the driver function called by rte_eal_init to initialize pci devs, is mandatory
> to distinguish if the process is running as SECONDARY (--proc-type=secondary in EAL prameters).
> 
> For example:
> eth_ixgbe_dev_init() function of the ixgbe Intel driver, at line 988 :
> 
> 	/*
> 	 * For secondary processes, we don't initialise any further as primary
> 	 * has already done this work. Only check we don't need a different
> 	 * RX and TX function.
> 	 */
> 	if (rte_eal_process_type() != RTE_PROC_PRIMARY){
> 		... some code lines here ...
> 		return 0;
> 	}
> 
> While in Mellanox mlx4 driver there is a check in ALL driver functions that makes them return
> immediatelly if the running process is of type "secondary".
> 
> That means that a driver has to take into account that the process could be a SECONDARY process...

This is called "Multiprocess aware" in doc/guides/nics/overview.rst.

> but in Qlogic (Broadcom) bnx2x driver I didn't find anything about that.

It seems the driver do not comply with this multiprocess requirement.

> Do you think I've missed something ?

You are probably right :)