[dpdk-users] [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen

[dpdk-users] [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen

[longtb5]

Hi, 

I'm running Ubuntu 16.04 on my VM with vmxnet3 as the NIC. I have successfully installed DPDK
and managed to run a few sample apps. However when I tried pktgen the kernel threw a SIGSEGV
without me doing anything yet.

$ sudo .app/x86_64-native-linuxapp-gcc/pktgen

Pktgen:/> 
====== Pktgen got a Segment Fault

Obtained 11 stack frames.
app/x86_64-native-linuxapp-gcc/pktgen() [0x468ba5]
/lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f1e618de4b0]
app/x86_64-native-linuxapp-gcc/pktgen() [0x6c7c70]
app/x86_64-native-linuxapp-gcc/pktgen(rte_eth_stats_get+0x79) [0x4a3fe9]
app/x86_64-native-linuxapp-gcc/pktgen(pktgen_process_stats+0xa9) [0x46f0f9]
app/x86_64-native-linuxapp-gcc/pktgen(rte_timer_manage+0x526) [0x48ca96]
app/x86_64-native-linuxapp-gcc/pktgen(cli_start+0x26d) [0x71276d]
app/x86_64-native-linuxapp-gcc/pktgen(pktgen_cli_start+0xb) [0x4509bb]
app/x86_64-native-linuxapp-gcc/pktgen(main+0x7a6) [0x447b86]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f1e618c9830]
app/x86_64-native-linuxapp-gcc/pktgen(_start+0x29) [0x44a639]

# gdb .app/x86_64-native-linuxapp-gcc/pktgen

Pktgen:/> 
Thread 1 "pktgen" received signal SIGSEGV, Segmentation fault.
0x00000000006c7c70 in vmxnet3_dev_stats_get ()

I tried using older versions of DPDK and pktgen but nothing worked so far.

Is it a problem on my end? Like, the way I set up the NICs in my VM? Or is it a bug?

BR,

Long.

Re: [dpdk-users] [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen

[longtb5]

Hi,

I have tested another app (proc_info) which also uses vmxnet3_dev_stats_get() but it ran just fine.

I have found out that this function, which is called inside vmxnet3_dev_stats_get(), causes the kernel to return a SIGSEGV when running pktgen:

drivers/net/vmxnet3/vmxnet3_ethdev.c:898
vmxnet3_hw_tx_stats_get(hw, i, &txStats);

Should be related to how pktgen called this function? Because the other app didn't crash.

Thanks,
Long


> ----- Original Message -----
> From: longtb5@viettel.com.vn
> To: users@dpdk.org
> Sent: Thursday, September 7, 2017 7:51:46 AM
> Subject: [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen
> 
> Hi, 
> 
> I'm running Ubuntu 16.04 on my VM with vmxnet3 as the NIC. I have successfully installed DPDK
> and managed to run a few sample apps. However when I tried pktgen the kernel threw a SIGSEGV
> without me doing anything yet.
> 
> $ sudo .app/x86_64-native-linuxapp-gcc/pktgen
> 
> Pktgen:/> 
> ====== Pktgen got a Segment Fault
> 
> Obtained 11 stack frames.
> app/x86_64-native-linuxapp-gcc/pktgen() [0x468ba5]
> /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f1e618de4b0]
> app/x86_64-native-linuxapp-gcc/pktgen() [0x6c7c70]
> app/x86_64-native-linuxapp-gcc/pktgen(rte_eth_stats_get+0x79) [0x4a3fe9]
> app/x86_64-native-linuxapp-gcc/pktgen(pktgen_process_stats+0xa9) [0x46f0f9]
> app/x86_64-native-linuxapp-gcc/pktgen(rte_timer_manage+0x526) [0x48ca96]
> app/x86_64-native-linuxapp-gcc/pktgen(cli_start+0x26d) [0x71276d]
> app/x86_64-native-linuxapp-gcc/pktgen(pktgen_cli_start+0xb) [0x4509bb]
> app/x86_64-native-linuxapp-gcc/pktgen(main+0x7a6) [0x447b86]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f1e618c9830]
> app/x86_64-native-linuxapp-gcc/pktgen(_start+0x29) [0x44a639]
> 
> # gdb .app/x86_64-native-linuxapp-gcc/pktgen
> 
> Pktgen:/> 
> Thread 1 "pktgen" received signal SIGSEGV, Segmentation fault.
> 0x00000000006c7c70 in vmxnet3_dev_stats_get ()
> 
> I tried using older versions of DPDK and pktgen but nothing worked so far.
> 
> Is it a problem on my end? Like, the way I set up the NICs in my VM? Or is it a bug?
> 
> BR,
> 
> Long.

Re: [dpdk-users] [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen

[longtb5]

Hi,

Apparently the problem was that I started CLI _with_ timers. When I directly called cli_start(); instead of
cli_start_with_timers() in pktgen_cli_start() every works OK (except it threw another segfault after quit, but
that's not important right now).

I have check and see that HPET is enabled, but is not the current clk source. Maybe that's the problem.

Regards,
Long.




----- Original Message -----
From: longtb5@viettel.com.vn
To: users@dpdk.org
Cc: "keith wiles" <keith.wiles@intel.com>, skhare@vmware.com
Sent: Thursday, September 7, 2017 2:42:06 PM
Subject: Re: [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen

Hi,

I have tested another app (proc_info) which also uses vmxnet3_dev_stats_get() but it ran just fine.

I have found out that this function, which is called inside vmxnet3_dev_stats_get(), causes the kernel to return a SIGSEGV when running pktgen:

drivers/net/vmxnet3/vmxnet3_ethdev.c:898
vmxnet3_hw_tx_stats_get(hw, i, &txStats);

Should be related to how pktgen called this function? Because the other app didn't crash.

Thanks,
Long


> ----- Original Message -----
> From: longtb5@viettel.com.vn
> To: users@dpdk.org
> Sent: Thursday, September 7, 2017 7:51:46 AM
> Subject: [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen
> 
> Hi, 
> 
> I'm running Ubuntu 16.04 on my VM with vmxnet3 as the NIC. I have successfully installed DPDK
> and managed to run a few sample apps. However when I tried pktgen the kernel threw a SIGSEGV
> without me doing anything yet.
> 
> $ sudo .app/x86_64-native-linuxapp-gcc/pktgen
> 
> Pktgen:/&gt; 
> ====== Pktgen got a Segment Fault
> 
> Obtained 11 stack frames.
> app/x86_64-native-linuxapp-gcc/pktgen() [0x468ba5]
> /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f1e618de4b0]
> app/x86_64-native-linuxapp-gcc/pktgen() [0x6c7c70]
> app/x86_64-native-linuxapp-gcc/pktgen(rte_eth_stats_get+0x79) [0x4a3fe9]
> app/x86_64-native-linuxapp-gcc/pktgen(pktgen_process_stats+0xa9) [0x46f0f9]
> app/x86_64-native-linuxapp-gcc/pktgen(rte_timer_manage+0x526) [0x48ca96]
> app/x86_64-native-linuxapp-gcc/pktgen(cli_start+0x26d) [0x71276d]
> app/x86_64-native-linuxapp-gcc/pktgen(pktgen_cli_start+0xb) [0x4509bb]
> app/x86_64-native-linuxapp-gcc/pktgen(main+0x7a6) [0x447b86]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f1e618c9830]
> app/x86_64-native-linuxapp-gcc/pktgen(_start+0x29) [0x44a639]
> 
> # gdb .app/x86_64-native-linuxapp-gcc/pktgen
> 
> Pktgen:/&gt; 
> Thread 1 "pktgen" received signal SIGSEGV, Segmentation fault.
> 0x00000000006c7c70 in vmxnet3_dev_stats_get ()
> 
> I tried using older versions of DPDK and pktgen but nothing worked so far.
> 
> Is it a problem on my end? Like, the way I set up the NICs in my VM? Or is it a bug?
> 
> BR,
> 
> Long.

Re: [dpdk-users] [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen

[Stephen Hemminger]

On Thu,  7 Sep 2017 07:51:46 +0700 (ICT)
longtb5@viettel.com.vn wrote:

> Hi, 
> 
> I'm running Ubuntu 16.04 on my VM with vmxnet3 as the NIC. I have successfully installed DPDK
> and managed to run a few sample apps. However when I tried pktgen the kernel threw a SIGSEGV
> without me doing anything yet.
> 
> $ sudo .app/x86_64-native-linuxapp-gcc/pktgen
> 
> Pktgen:/> 
> ====== Pktgen got a Segment Fault
> 
> Obtained 11 stack frames.
> app/x86_64-native-linuxapp-gcc/pktgen() [0x468ba5]
> /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f1e618de4b0]
> app/x86_64-native-linuxapp-gcc/pktgen() [0x6c7c70]
> app/x86_64-native-linuxapp-gcc/pktgen(rte_eth_stats_get+0x79) [0x4a3fe9]
> app/x86_64-native-linuxapp-gcc/pktgen(pktgen_process_stats+0xa9) [0x46f0f9]
> app/x86_64-native-linuxapp-gcc/pktgen(rte_timer_manage+0x526) [0x48ca96]
> app/x86_64-native-linuxapp-gcc/pktgen(cli_start+0x26d) [0x71276d]
> app/x86_64-native-linuxapp-gcc/pktgen(pktgen_cli_start+0xb) [0x4509bb]
> app/x86_64-native-linuxapp-gcc/pktgen(main+0x7a6) [0x447b86]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f1e618c9830]
> app/x86_64-native-linuxapp-gcc/pktgen(_start+0x29) [0x44a639]
> 
> # gdb .app/x86_64-native-linuxapp-gcc/pktgen
> 
> Pktgen:/> 
> Thread 1 "pktgen" received signal SIGSEGV, Segmentation fault.
> 0x00000000006c7c70 in vmxnet3_dev_stats_get ()
> 
> I tried using older versions of DPDK and pktgen but nothing worked so far.
> 
> Is it a problem on my end? Like, the way I set up the NICs in my VM? Or is it a bug?
> 
> BR,
> 
> Long.

If you modify the Makefiles for pktgen, ethdev, and vmxnet3 to have -g flag
then you will get more symbolic data which makes better backtrace.

Re: [dpdk-users] [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen

[Wiles, Keith]

> On Sep 7, 2017, at 7:58 AM, Stephen Hemminger <stephen@networkplumber.org> wrote:
> 
> On Thu,  7 Sep 2017 07:51:46 +0700 (ICT)
> longtb5@viettel.com.vn wrote:
> 
>> Hi, 
>> 
>> I'm running Ubuntu 16.04 on my VM with vmxnet3 as the NIC. I have successfully installed DPDK
>> and managed to run a few sample apps. However when I tried pktgen the kernel threw a SIGSEGV
>> without me doing anything yet.
>> 
>> $ sudo .app/x86_64-native-linuxapp-gcc/pktgen
>> 
>> Pktgen:/&gt; 
>> ====== Pktgen got a Segment Fault
>> 
>> Obtained 11 stack frames.
>> app/x86_64-native-linuxapp-gcc/pktgen() [0x468ba5]
>> /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f1e618de4b0]
>> app/x86_64-native-linuxapp-gcc/pktgen() [0x6c7c70]
>> app/x86_64-native-linuxapp-gcc/pktgen(rte_eth_stats_get+0x79) [0x4a3fe9]
>> app/x86_64-native-linuxapp-gcc/pktgen(pktgen_process_stats+0xa9) [0x46f0f9]
>> app/x86_64-native-linuxapp-gcc/pktgen(rte_timer_manage+0x526) [0x48ca96]
>> app/x86_64-native-linuxapp-gcc/pktgen(cli_start+0x26d) [0x71276d]
>> app/x86_64-native-linuxapp-gcc/pktgen(pktgen_cli_start+0xb) [0x4509bb]
>> app/x86_64-native-linuxapp-gcc/pktgen(main+0x7a6) [0x447b86]
>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f1e618c9830]
>> app/x86_64-native-linuxapp-gcc/pktgen(_start+0x29) [0x44a639]
>> 
>> # gdb .app/x86_64-native-linuxapp-gcc/pktgen
>> 
>> Pktgen:/&gt; 
>> Thread 1 "pktgen" received signal SIGSEGV, Segmentation fault.
>> 0x00000000006c7c70 in vmxnet3_dev_stats_get ()
>> 
>> I tried using older versions of DPDK and pktgen but nothing worked so far.
>> 
>> Is it a problem on my end? Like, the way I set up the NICs in my VM? Or is it a bug?
>> 
>> BR,
>> 
>> Long.
> 
> If you modify the Makefiles for pktgen, ethdev, and vmxnet3 to have -g flag
> then you will get more symbolic data which makes better backtrace.

I use the option EXTRA_CFLAGS on the command line to improve debug.

make install T=x86_64-native-linuxapp-gcc EXTRA_CFLAGS=“-g -O0”

Using -O0 for gdb is good for non-performance testing.


Regards,
Keith

Re: [dpdk-users] [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen

[longtb5]

>----- Original Message -----
>From: "keith wiles" <keith.wiles@intel.com>
>To: "Stephen Hemminger" <stephen@networkplumber.org>
>Cc: longtb5@viettel.com.vn, users@dpdk.org
>Sent: Thursday, September 7, 2017 11:49:39 PM
>Subject: Re: [dpdk-users] [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen
>
>
>I use the option EXTRA_CFLAGS on the command line to improve debug.
>
>make install T=x86_64-native-linuxapp-gcc EXTRA_CFLAGS=“-g -O0”
>
>Using -O0 for gdb is good for non-performance testing.
>
>
>Regards,
>Keith

Hi,

Thanks Keith and Stephen for the pointer. I have enabled the debug flags and acquired some more infos.

(gdb) r
Thread 1 "pktgen" received signal SIGSEGV, Segmentation fault.
0x000000000075aa6c in vmxnet3_hw_tx_stats_get (hw=0x7fffb55b61c0, q=0, res=0x7fffffffdf10) at /home/tester/dpdk/drivers/net/vmxnet3/vmxnet3_ethdev.c:905
905		VMXNET3_UPDATE_TX_STAT(hw, q, ucastPktsTxOK, res);
(gdb) backtrace
#0  0x000000000075aa6c in vmxnet3_hw_tx_stats_get (hw=0x7fffb55b61c0, q=0, res=0x7fffffffdf10) at /home/tester/dpdk/drivers/net/vmxnet3/vmxnet3_ethdev.c:905
#1  0x000000000075b3c8 in vmxnet3_dev_stats_get (dev=0xba34c0 <rte_eth_devices>, stats=0x7fffffffe000) at /home/tester/dpdk/drivers/net/vmxnet3/vmxnet3_ethdev.c:1049
#2  0x00000000004a2f76 in rte_eth_stats_get (port_id=0 '\000', stats=0x7fffffffe000) at /home/tester/dpdk/lib/librte_ether/rte_ethdev.c:1340
#3  0x0000000000467059 in pktgen_process_stats (tim=<optimized out>, arg=<optimized out>) at /home/tester/pktgen-dpdk/app/pktgen-stats.c:468
#4  0x000000000048568a in rte_timer_manage () at /home/tester/dpdk/lib/librte_timer/rte_timer.c:593
#5  0x00000000007b47dd in cli_start ()
#6  0x000000000044891b in pktgen_cli_start () at /home/tester/pktgen-dpdk/app/cli-functions.c:1434
#7  0x00000000004423f6 in main (argc=<optimized out>, argv=<optimized out>) at /home/tester/pktgen-dpdk/app/pktgen-main.c:470
(gdb) l
900	{
901	#define VMXNET3_UPDATE_TX_STAT(h, i, f, r)		\
902			((r)->f = (h)->tqd_start[(i)].stats.f +	\
903				(h)->saved_tx_stats[(i)].f)
904	
905		VMXNET3_UPDATE_TX_STAT(hw, q, ucastPktsTxOK, res);
906		VMXNET3_UPDATE_TX_STAT(hw, q, mcastPktsTxOK, res);
907		VMXNET3_UPDATE_TX_STAT(hw, q, bcastPktsTxOK, res);
908		VMXNET3_UPDATE_TX_STAT(hw, q, ucastBytesTxOK, res);
909		VMXNET3_UPDATE_TX_STAT(hw, q, mcastBytesTxOK, res);
(gdb) p hw->tqd_start
$1 = (Vmxnet3_TxQueueDesc *) 0x0

The problem was on line 902.
This tqd_start is the "start address of all tx queue desc" according to comments in source code. Looks like it was not initialized properly.
I don't have enough knowledge to fix this. Any advice?

Regards,
BL.

Re: [dpdk-users] [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen

[Wiles, Keith]

> On Sep 7, 2017, at 7:07 PM, longtb5@viettel.com.vn wrote:
> 
>> ----- Original Message -----
>> From: "keith wiles" <keith.wiles@intel.com>
>> To: "Stephen Hemminger" <stephen@networkplumber.org>
>> Cc: longtb5@viettel.com.vn, users@dpdk.org
>> Sent: Thursday, September 7, 2017 11:49:39 PM
>> Subject: Re: [dpdk-users] [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen
>> 
>> 
>> I use the option EXTRA_CFLAGS on the command line to improve debug.
>> 
>> make install T=x86_64-native-linuxapp-gcc EXTRA_CFLAGS=“-g -O0”
>> 
>> Using -O0 for gdb is good for non-performance testing.
>> 
>> 
>> Regards,
>> Keith
> 
> Hi,
> 
> Thanks Keith and Stephen for the pointer. I have enabled the debug flags and acquired some more infos.
> 
> (gdb) r
> Thread 1 "pktgen" received signal SIGSEGV, Segmentation fault.
> 0x000000000075aa6c in vmxnet3_hw_tx_stats_get (hw=0x7fffb55b61c0, q=0, res=0x7fffffffdf10) at /home/tester/dpdk/drivers/net/vmxnet3/vmxnet3_ethdev.c:905
> 905		VMXNET3_UPDATE_TX_STAT(hw, q, ucastPktsTxOK, res);
> (gdb) backtrace
> #0  0x000000000075aa6c in vmxnet3_hw_tx_stats_get (hw=0x7fffb55b61c0, q=0, res=0x7fffffffdf10) at /home/tester/dpdk/drivers/net/vmxnet3/vmxnet3_ethdev.c:905
> #1  0x000000000075b3c8 in vmxnet3_dev_stats_get (dev=0xba34c0 <rte_eth_devices>, stats=0x7fffffffe000) at /home/tester/dpdk/drivers/net/vmxnet3/vmxnet3_ethdev.c:1049
> #2  0x00000000004a2f76 in rte_eth_stats_get (port_id=0 '\000', stats=0x7fffffffe000) at /home/tester/dpdk/lib/librte_ether/rte_ethdev.c:1340
> #3  0x0000000000467059 in pktgen_process_stats (tim=<optimized out>, arg=<optimized out>) at /home/tester/pktgen-dpdk/app/pktgen-stats.c:468
> #4  0x000000000048568a in rte_timer_manage () at /home/tester/dpdk/lib/librte_timer/rte_timer.c:593
> #5  0x00000000007b47dd in cli_start ()
> #6  0x000000000044891b in pktgen_cli_start () at /home/tester/pktgen-dpdk/app/cli-functions.c:1434
> #7  0x00000000004423f6 in main (argc=<optimized out>, argv=<optimized out>) at /home/tester/pktgen-dpdk/app/pktgen-main.c:470
> (gdb) l
> 900	{
> 901	#define VMXNET3_UPDATE_TX_STAT(h, i, f, r)		\
> 902			((r)->f = (h)->tqd_start[(i)].stats.f +	\
> 903				(h)->saved_tx_stats[(i)].f)
> 904	
> 905		VMXNET3_UPDATE_TX_STAT(hw, q, ucastPktsTxOK, res);
> 906		VMXNET3_UPDATE_TX_STAT(hw, q, mcastPktsTxOK, res);
> 907		VMXNET3_UPDATE_TX_STAT(hw, q, bcastPktsTxOK, res);
> 908		VMXNET3_UPDATE_TX_STAT(hw, q, ucastBytesTxOK, res);
> 909		VMXNET3_UPDATE_TX_STAT(hw, q, mcastBytesTxOK, res);
> (gdb) p hw->tqd_start
> $1 = (Vmxnet3_TxQueueDesc *) 0x0
> 
> The problem was on line 902.
> This tqd_start is the "start address of all tx queue desc" according to comments in source code. Looks like it was not initialized properly.
> I don't have enough knowledge to fix this. Any advice?

I am not able to fix this problem, but it seems like it could be pktgen command line configuration issue. Normally these types of problems are because the application tried to access a port/queue that is not setup. The tqd_start[] is an internal array to the VMXNET3 driver, so I do not know how it relates to the DPDK  API requests for port/queue.

Sorry, if you provided this before, but what is the command line you are using for Pktgen?

> 
> Regards,
> BL.

Regards,
Keith

Re: [dpdk-users] [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen

[longtb5]

// Sorry I forgot to CC

I tested these:
$ sudo ./app/x86_64-native-linuxapp-gcc/pktgen
$ sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 0-3 -n 2 
$ sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 0-3 -n 2 -- -P -m "1.0, 2.1"
$ sudo ./tools/dpdk-run.py default

This is my default.cfg
# Run command and options
run = {
    'exec': (
        'sudo',
        '-E'
        ),

    # Application name and use app_path to help locate the app
    'app_name': 'pktgen',

    # using (sdk) or (target) for specific variables
    # add (app_name) of the application
    # Each path is tested for the application
    'app_path': (
        './app/%(target)s/%(app_name)s',
        '%(sdk)s/%(target)s/app/%(app_name)s',
        ),

        'dpdk': (
                '-l 0-1',
                '-n 2',
                '--proc-type auto',
                '--log-level 7',
                #'--socket-mem 2048,2048',
                #'--file-prefix pg'
                ),
        

        'app': (
                '-T',
                '-P',
                '--crc-strip',
                '-m 1.[2/5]'
                ),
        
        'misc': (
                #'-f',
                'themes/black-yellow.theme'
                )
        }

$ sudo ~/dev/dpdk/usertools/cpu_layout.py
cores =  [0, 1]
sockets =  [0, 1]

       Socket 0   Socket 1  
       --------   --------  
Core 0 [0]        [2]       
Core 1 [1]        [3]  

$ sudo ./dpdk/usertools/dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:03:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=
0000:04:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=
0000:0b:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=
0000:0c:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=
0000:13:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=
0000:14:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=
0000:1b:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=
0000:1c:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=

Network devices using kernel driver
===================================
0000:05:00.0 'VMXNET3 Ethernet Controller 07b0' if=ens162 drv=vmxnet3 unused=igb_uio *Active*
0000:0d:00.0 'VMXNET3 Ethernet Controller 07b0' if=ens194 drv=vmxnet3 unused=igb_uio *Active*


>
>----- Original Message -----
>From: "keith wiles" <keith.wiles@intel.com>
>To: longtb5@viettel.com.vn
>Cc: "Stephen Hemminger" <stephen@networkplumber.org>, users@dpdk.org, skhare@vmware.com
>Sent: Friday, September 8, 2017 9:22:25 AM
>Subject: Re: [dpdk-users] [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen
>
>
>> On Sep 7, 2017, at 7:07 PM, longtb5@viettel.com.vn wrote:
>> 
>>> ----- Original Message -----
>>> From: "keith wiles" <keith.wiles@intel.com>
>>> To: "Stephen Hemminger" <stephen@networkplumber.org>
>>> Cc: longtb5@viettel.com.vn, users@dpdk.org
>>> Sent: Thursday, September 7, 2017 11:49:39 PM
>>> Subject: Re: [dpdk-users] [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen
>>> 
>>> 
>>> I use the option EXTRA_CFLAGS on the command line to improve debug.
>>> 
>>> make install T=x86_64-native-linuxapp-gcc EXTRA_CFLAGS=“-g -O0”
>>> 
>>> Using -O0 for gdb is good for non-performance testing.
>>> 
>>> 
>>> Regards,
>>> Keith
>> 
>> Hi,
>> 
>> Thanks Keith and Stephen for the pointer. I have enabled the debug flags and acquired some more infos.
>> 
>> (gdb) r
>> Thread 1 "pktgen" received signal SIGSEGV, Segmentation fault.
>> 0x000000000075aa6c in vmxnet3_hw_tx_stats_get (hw=0x7fffb55b61c0, q=0, res=0x7fffffffdf10) at /home/tester/dpdk/drivers/net/vmxnet3/vmxnet3_ethdev.c:905
>> 905                VMXNET3_UPDATE_TX_STAT(hw, q, ucastPktsTxOK, res);
>> (gdb) backtrace
>> #0  0x000000000075aa6c in vmxnet3_hw_tx_stats_get (hw=0x7fffb55b61c0, q=0, res=0x7fffffffdf10) at /home/tester/dpdk/drivers/net/vmxnet3/vmxnet3_ethdev.c:905
>> #1  0x000000000075b3c8 in vmxnet3_dev_stats_get (dev=0xba34c0 <rte_eth_devices>, stats=0x7fffffffe000) at /home/tester/dpdk/drivers/net/vmxnet3/vmxnet3_ethdev.c:1049
>> #2  0x00000000004a2f76 in rte_eth_stats_get (port_id=0 '\000', stats=0x7fffffffe000) at /home/tester/dpdk/lib/librte_ether/rte_ethdev.c:1340
>> #3  0x0000000000467059 in pktgen_process_stats (tim=<optimized out>, arg=<optimized out>) at /home/tester/pktgen-dpdk/app/pktgen-stats.c:468
>> #4  0x000000000048568a in rte_timer_manage () at /home/tester/dpdk/lib/librte_timer/rte_timer.c:593
>> #5  0x00000000007b47dd in cli_start ()
>> #6  0x000000000044891b in pktgen_cli_start () at /home/tester/pktgen-dpdk/app/cli-functions.c:1434
>> #7  0x00000000004423f6 in main (argc=<optimized out>, argv=<optimized out>) at /home/tester/pktgen-dpdk/app/pktgen-main.c:470
>> (gdb) l
>> 900        {
>> 901        #define VMXNET3_UPDATE_TX_STAT(h, i, f, r)                \
>> 902                        ((r)->f = (h)->tqd_start[(i)].stats.f +        \
>> 903                                (h)->saved_tx_stats[(i)].f)
>> 904        
>> 905                VMXNET3_UPDATE_TX_STAT(hw, q, ucastPktsTxOK, res);
>> 906                VMXNET3_UPDATE_TX_STAT(hw, q, mcastPktsTxOK, res);
>> 907                VMXNET3_UPDATE_TX_STAT(hw, q, bcastPktsTxOK, res);
>> 908                VMXNET3_UPDATE_TX_STAT(hw, q, ucastBytesTxOK, res);
>> 909                VMXNET3_UPDATE_TX_STAT(hw, q, mcastBytesTxOK, res);
>> (gdb) p hw->tqd_start
>> $1 = (Vmxnet3_TxQueueDesc *) 0x0
>> 
>> The problem was on line 902.
>> This tqd_start is the "start address of all tx queue desc" according to comments in source code. Looks like it was not initialized properly.
>> I don't have enough knowledge to fix this. Any advice?
>
>I am not able to fix this problem, but it seems like it could be pktgen command line configuration issue. Normally these types of problems are because the application tried to access a port/queue that is not >setup. The tqd_start[] is an internal array to the VMXNET3 driver, so I do not know how it relates to the DPDK  API requests for port/queue.>
>
>Sorry, if you provided this before, but what is the command line you are using for Pktgen?
>
>> 
>> Regards,
>> BL.
>
>Regards,
>Keith

Re: [dpdk-users] [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen

[longtb5]

Hi Keith,

All of the test runs that I listed gave me the same segfault from the same function in the driver. 
The config was not for any purposes other than trying to start pktgen. Maybe it is indeed a 
driver problem then.

I was able to run pktgen by replacing
cli_start_with_timers(NULL);
in pktgen_cli_start() in app/cli-functions.c, with
cli_start(NULL);

effectively disabled this flag setup 
this_cli->flags |= CLI_USE_TIMERS

With that I was able to run pktgen, but segfault still happens when I quit or want to access the stats page.

Do you have any ideas why this might happen?

Regards,
BL

----- Original Message -----
From: "keith wiles" <keith.wiles@intel.com>
To: longtb5@viettel.com.vn
Sent: Friday, September 8, 2017 10:28:46 AM
Subject: Re: [dpdk-users] [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen


> On Sep 7, 2017, at 7:54 PM, longtb5@viettel.com.vn wrote:
> 
> I tested these:
> $ sudo ./app/x86_64-native-linuxapp-gcc/pktgen
> $ sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 0-3 -n 2

The above will not work as you do not have the — arguments.

> $ sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 0-3 -n 2 -- -P -m "1.0, 2.1”

These above look fine except you use 4 cores and only use one for pktgen and 1 for each port for a total of three, you could have used -l 0-2

If the above still give the segfault then I do not know what is happening here and it does look like a driver problem??

> $ sudo ./tools/dpdk-run.py default

This one seem odd to me.
> 
> This is my default.cfg
> # Run command and options
> run = {
>    'exec': (
>        'sudo',
>        '-E'
>        ),
> 
>    # Application name and use app_path to help locate the app
>    'app_name': 'pktgen',
> 
>    # using (sdk) or (target) for specific variables
>    # add (app_name) of the application
>    # Each path is tested for the application
>    'app_path': (
>        './app/%(target)s/%(app_name)s',
>        '%(sdk)s/%(target)s/app/%(app_name)s',
>        ),
> 
> 	'dpdk': (
> 		'-l 0-1',
> 		'-n 2',
> 		'--proc-type auto',
> 		'--log-level 7',
> 		#'--socket-mem 2048,2048',
> 		#'--file-prefix pg'
> 		),
> 	
> 
> 	'app': (
> 		'-T',
> 		'-P',
> 		'--crc-strip',
> 		'-m 1.[2/5]’

Only doing one port trying to control port 2 and port 5?? Should this not be just 1.0 for one port.

> 		),
> 	
> 	'misc': (
> 		#'-f',
> 		'themes/black-yellow.theme'
> 		)
> 	}
> 
> $ sudo ~/dev/dpdk/usertools/cpu_layout.py
> cores =  [0, 1]
> sockets =  [0, 1]
> 
>       Socket 0   Socket 1  
>       --------   --------  
> Core 0 [0]        [2]       
> Core 1 [1]        [3]  
> 
> $ sudo ./dpdk/usertools/dpdk-devbind.py -s
> 
> Network devices using DPDK-compatible driver
> ============================================
> 0000:03:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=
> 0000:04:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=
> 0000:0b:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=
> 0000:0c:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=
> 0000:13:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=
> 0000:14:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=
> 0000:1b:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=
> 0000:1c:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=
> 
> Network devices using kernel driver
> ===================================
> 0000:05:00.0 'VMXNET3 Ethernet Controller 07b0' if=ens162 drv=vmxnet3 unused=igb_uio *Active*
> 0000:0d:00.0 'VMXNET3 Ethernet Controller 07b0' if=ens194 drv=vmxnet3 unused=igb_uio *Active*
> 
> 
>> 
>> ----- Original Message -----
>> From: "keith wiles" <keith.wiles@intel.com>
>> To: longtb5@viettel.com.vn
>> Cc: "Stephen Hemminger" <stephen@networkplumber.org>, users@dpdk.org, skhare@vmware.com
>> Sent: Friday, September 8, 2017 9:22:25 AM
>> Subject: Re: [dpdk-users] [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen
>> 
>> 
>>> On Sep 7, 2017, at 7:07 PM, longtb5@viettel.com.vn wrote:
>>> 
>>>> ----- Original Message -----
>>>> From: "keith wiles" <keith.wiles@intel.com>
>>>> To: "Stephen Hemminger" <stephen@networkplumber.org>
>>>> Cc: longtb5@viettel.com.vn, users@dpdk.org
>>>> Sent: Thursday, September 7, 2017 11:49:39 PM
>>>> Subject: Re: [dpdk-users] [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen
>>>> 
>>>> 
>>>> I use the option EXTRA_CFLAGS on the command line to improve debug.
>>>> 
>>>> make install T=x86_64-native-linuxapp-gcc EXTRA_CFLAGS=“-g -O0”
>>>> 
>>>> Using -O0 for gdb is good for non-performance testing.
>>>> 
>>>> 
>>>> Regards,
>>>> Keith
>>> 
>>> Hi,
>>> 
>>> Thanks Keith and Stephen for the pointer. I have enabled the debug flags and acquired some more infos.
>>> 
>>> (gdb) r
>>> Thread 1 "pktgen" received signal SIGSEGV, Segmentation fault.
>>> 0x000000000075aa6c in vmxnet3_hw_tx_stats_get (hw=0x7fffb55b61c0, q=0, res=0x7fffffffdf10) at /home/tester/dpdk/drivers/net/vmxnet3/vmxnet3_ethdev.c:905
>>> 905		VMXNET3_UPDATE_TX_STAT(hw, q, ucastPktsTxOK, res);
>>> (gdb) backtrace
>>> #0  0x000000000075aa6c in vmxnet3_hw_tx_stats_get (hw=0x7fffb55b61c0, q=0, res=0x7fffffffdf10) at /home/tester/dpdk/drivers/net/vmxnet3/vmxnet3_ethdev.c:905
>>> #1  0x000000000075b3c8 in vmxnet3_dev_stats_get (dev=0xba34c0 <rte_eth_devices>, stats=0x7fffffffe000) at /home/tester/dpdk/drivers/net/vmxnet3/vmxnet3_ethdev.c:1049
>>> #2  0x00000000004a2f76 in rte_eth_stats_get (port_id=0 '\000', stats=0x7fffffffe000) at /home/tester/dpdk/lib/librte_ether/rte_ethdev.c:1340
>>> #3  0x0000000000467059 in pktgen_process_stats (tim=<optimized out>, arg=<optimized out>) at /home/tester/pktgen-dpdk/app/pktgen-stats.c:468
>>> #4  0x000000000048568a in rte_timer_manage () at /home/tester/dpdk/lib/librte_timer/rte_timer.c:593
>>> #5  0x00000000007b47dd in cli_start ()
>>> #6  0x000000000044891b in pktgen_cli_start () at /home/tester/pktgen-dpdk/app/cli-functions.c:1434
>>> #7  0x00000000004423f6 in main (argc=<optimized out>, argv=<optimized out>) at /home/tester/pktgen-dpdk/app/pktgen-main.c:470
>>> (gdb) l
>>> 900	{
>>> 901	#define VMXNET3_UPDATE_TX_STAT(h, i, f, r)		\
>>> 902			((r)->f = (h)->tqd_start[(i)].stats.f +	\
>>> 903				(h)->saved_tx_stats[(i)].f)
>>> 904	
>>> 905		VMXNET3_UPDATE_TX_STAT(hw, q, ucastPktsTxOK, res);
>>> 906		VMXNET3_UPDATE_TX_STAT(hw, q, mcastPktsTxOK, res);
>>> 907		VMXNET3_UPDATE_TX_STAT(hw, q, bcastPktsTxOK, res);
>>> 908		VMXNET3_UPDATE_TX_STAT(hw, q, ucastBytesTxOK, res);
>>> 909		VMXNET3_UPDATE_TX_STAT(hw, q, mcastBytesTxOK, res);
>>> (gdb) p hw->tqd_start
>>> $1 = (Vmxnet3_TxQueueDesc *) 0x0
>>> 
>>> The problem was on line 902.
>>> This tqd_start is the "start address of all tx queue desc" according to comments in source code. Looks like it was not initialized properly.
>>> I don't have enough knowledge to fix this. Any advice?
>> 
>> I am not able to fix this problem, but it seems like it could be pktgen command line configuration issue. Normally these types of problems are because the application tried to access a port/queue that is not >setup. The tqd_start[] is an internal array to the VMXNET3 driver, so I do not know how it relates to the DPDK  API requests for port/queue.>
>> 
>> Sorry, if you provided this before, but what is the command line you are using for Pktgen?
>> 
>>> 
>>> Regards,
>>> BL.
>> 
>> Regards,
>> Keith
> 

Regards,
Keith

Re: [dpdk-users] [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen

[longtb5]

Hi all,

Somehow this issue is gone for me. All I did was removing then re-adding new ethernet adapters in VMware, make a clean rebuild of both dpdk and pktgen.
My coworker told me he needed to do something similar (reset the host adapter) to fix a problem that he had.

Regards,
BL

Re: [dpdk-users] [VMXNET3] vmxnet3_dev_stats_get() segfault when called by pktgen

[longtb5]

I found the problem. It was because pktgen tried to get stats for all dpdk-enabled ports, including ones not used by pktgen.

Since I used VMplayer, I added few too many devices and bound all of them to igb_uio. Pktgen tried to get stats from all of them.

Solution: unbind all unused ports from DPDK.

Regards,
BL