Re: [dpdk-users] [ovs-discuss] ovs-dpdk crash when use vhost-user in docker

Re: [dpdk-users] [ovs-discuss] ovs-dpdk crash when use vhost-user in docker

[Loftus, Ciara]

Hi,

I am cc-ing the DPDK users’ list as the SEGV originates in the DPDK vHost code and somebody there might be able to help too.
Could you provide more information about your environment please? eg. OVS & DPDK versions, hugepage configuration, etc.

Thanks,
Ciara

From: ovs-discuss-bounces@openvswitch.org [mailto:ovs-discuss-bounces@openvswitch.org] On Behalf Of ???
Sent: Monday, August 20, 2018 12:06 PM
To: ovs-discuss@openvswitch.org
Subject: [ovs-discuss] ovs-dpdk crash when use vhost-user in docker

Hi,

   I used ovs-dpdk  as bridge  and l2fwd  as container. When l2fwd was runned ,the ovs-dpdk was crashed.

My command is :

    docker run -it --privileged --name=dpdk-docker  -v /dev/hugepages:/mnt/huge -v /usr/local/var/run/openvswitch:/var/run/openvswitch dpdk-docker

./l2fwd -c 0x06 -n 4  --socket-mem=1024  --no-pci --vdev=net_virtio_user0,mac=00:00:00:00:00:05,path=/var/run/openvswitch/vhost-user0  --vdev=net_virtio_user1,mac=00:00:00:00:00:01,path=/var/run/openvswitch/vhost-user1 -- -p 0x3



The crash log



Program terminated with signal 11, Segmentation fault.

#0  0x0000000000445828 in malloc_elem_alloc ()

Missing separate debuginfos, use: debuginfo-install glibc-2.17-196.el7_4.2.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-8.el7.x86_64 libcap-ng-0.7.5-4.el7.x86_64 libcom_err-1.42.9-10.el7.x86_64 libgcc-4.8.5-16.el7_4.1.x86_64 libpcap-1.5.3-9.el7.x86_64 libselinux-2.5-12.el7.x86_64 numactl-libs-2.0.9-6.el7_2.x86_64 openssl-libs-1.0.2k-8.el7.x86_64 pcre-8.32-17.el7.x86_64 zlib-1.2.7-17.el7.x86_64

(gdb) bt

#0  0x0000000000445828 in malloc_elem_alloc ()

#1  0x0000000000445e5d in malloc_heap_alloc ()

#2  0x0000000000444c74 in rte_zmalloc ()

#3  0x00000000006c16bf in vhost_new_device ()

#4  0x00000000006bfaf4 in vhost_user_add_connection ()

#5  0x00000000006beb88 in fdset_event_dispatch ()

#6  0x00007f613b288e25 in start_thread () from /usr/lib64/libpthread.so.0

#7  0x00007f613a86b34d in clone () from /usr/lib64/libc.so.6



My OVS  version is 2.9.1 , DPDK version is 17.11.3





Thanks

Re: [dpdk-users] [ovs-discuss] ovs-dpdk crash when use vhost-user in docker

[O Mahony, Billy]

Hi,

One thing to look out for with DPDK < 18.05 is that you need to used 1GB huge pages (and no more than eight of them) to use virtio. I’m not sure if that is the issue you have as I think it I don’t remember it causing a seg fault. But is certainly worth checking.

If that does not work please send the info Ciara refers to as well as the ovs-vsctl interface config for the ovs vhost backend.

Thanks,
Billy

From: ovs-discuss-bounces@openvswitch.org [mailto:ovs-discuss-bounces@openvswitch.org] On Behalf Of Loftus, Ciara
Sent: Tuesday, August 21, 2018 9:06 AM
To: gmzhang76@gmail.com; ovs-discuss@openvswitch.org
Cc: users@dpdk.org
Subject: Re: [ovs-discuss] ovs-dpdk crash when use vhost-user in docker

Hi,

I am cc-ing the DPDK users’ list as the SEGV originates in the DPDK vHost code and somebody there might be able to help too.
Could you provide more information about your environment please? eg. OVS & DPDK versions, hugepage configuration, etc.

Thanks,
Ciara

From: ovs-discuss-bounces@openvswitch.org<mailto:ovs-discuss-bounces@openvswitch.org> [mailto:ovs-discuss-bounces@openvswitch.org] On Behalf Of ???
Sent: Monday, August 20, 2018 12:06 PM
To: ovs-discuss@openvswitch.org<mailto:ovs-discuss@openvswitch.org>
Subject: [ovs-discuss] ovs-dpdk crash when use vhost-user in docker

Hi,

   I used ovs-dpdk  as bridge  and l2fwd  as container. When l2fwd was runned ,the ovs-dpdk was crashed.

My command is :

    docker run -it --privileged --name=dpdk-docker  -v /dev/hugepages:/mnt/huge -v /usr/local/var/run/openvswitch:/var/run/openvswitch dpdk-docker

./l2fwd -c 0x06 -n 4  --socket-mem=1024  --no-pci --vdev=net_virtio_user0,mac=00:00:00:00:00:05,path=/var/run/openvswitch/vhost-user0  --vdev=net_virtio_user1,mac=00:00:00:00:00:01,path=/var/run/openvswitch/vhost-user1 -- -p 0x3



The crash log



Program terminated with signal 11, Segmentation fault.

#0  0x0000000000445828 in malloc_elem_alloc ()

Missing separate debuginfos, use: debuginfo-install glibc-2.17-196.el7_4.2.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-8.el7.x86_64 libcap-ng-0.7.5-4.el7.x86_64 libcom_err-1.42.9-10.el7.x86_64 libgcc-4.8.5-16.el7_4.1.x86_64 libpcap-1.5.3-9.el7.x86_64 libselinux-2.5-12.el7.x86_64 numactl-libs-2.0.9-6.el7_2.x86_64 openssl-libs-1.0.2k-8.el7.x86_64 pcre-8.32-17.el7.x86_64 zlib-1.2.7-17.el7.x86_64

(gdb) bt

#0  0x0000000000445828 in malloc_elem_alloc ()

#1  0x0000000000445e5d in malloc_heap_alloc ()

#2  0x0000000000444c74 in rte_zmalloc ()

#3  0x00000000006c16bf in vhost_new_device ()

#4  0x00000000006bfaf4 in vhost_user_add_connection ()

#5  0x00000000006beb88 in fdset_event_dispatch ()

#6  0x00007f613b288e25 in start_thread () from /usr/lib64/libpthread.so.0

#7  0x00007f613a86b34d in clone () from /usr/lib64/libc.so.6



My OVS  version is 2.9.1 , DPDK version is 17.11.3





Thanks

Re: [dpdk-users] [ovs-discuss] ovs-dpdk crash when use vhost-user in docker

[张广明]

Hi,  Ciara and  Billy

Thanks for your reply

 The default huge page size that i used  is 1GB .
root@localhost openvswitch]# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-514.el7.x86_64 root=/dev/mapper/centos-root ro
crashkernel=auto iommu=pt intel_iommu=on default_hugepagesz=1G
hugepagesz=1G hugepages=2 rd.lvm.lv=centos/root rd.lvm.lv=centos/swap
rd.lvm.lv=centos/usr rhgb

The huge page number is 4
[root@localhost openvswitch]# cat /proc/meminfo | grep Huge
AnonHugePages:     14336 kB
HugePages_Total:       4
HugePages_Free:        2
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB


My ovs dpdk configure is
[root@localhost openvswitch]# ovs-vsctl --no-wait get Open_vSwitch .
other_config
{dpdk-init="true", dpdk-socket-mem="2048,0", pmd-cpu-mask="0x01"}

My ovs configure
[root@localhost openvswitch]# ovs-vsctl show
d2b6062a-4d6f-46f6-8fa4-66dca6b06c96
    Manager "tcp:192.168.15.18:6640"
        is_connected: true
    Bridge br-router
        Port "p2p1"
            Interface "p2p1"
                type: dpdk
                options: {dpdk-devargs="0000:01:00.0"}
        Port patch-gtp
            Interface patch-gtp
                type: patch
                options: {peer=patch-router}
        Port br-router
            Interface br-router
                type: internal
    Bridge "br0"
        Controller "tcp:192.168.15.18:6633"
            is_connected: true
        fail_mode: secure
        Port "p1p1"
            Interface "p1p1"
                type: dpdk
                options: {dpdk-devargs="0000:03:00.0"}
        Port patch-router
            Interface patch-router
                type: patch
                options: {peer=patch-gtp}
        Port "br0"
            Interface "br0"
                type: internal
        Port "vhost-user1"
            Interface "vhost-user1"
                type: dpdkvhostuser
        Port "vhost-user0"
            Interface "vhost-user0"
                type: dpdkvhostuser
    Bridge br-vxlan
        Port br-vxlan
            Interface br-vxlan
                type: internal


Docker running command is

   docker run -it --privileged --name=dpdk-docker  -v
/dev/hugepages:/mnt/huge -v
/usr/local/var/run/openvswitch:/var/run/openvswitch dpdk-docker

./l2fwd -c 0x06 -n 4  --socket-mem=1024  --no-pci
--vdev=net_virtio_user0,mac=00:00:00:00:00:05,path=/var/run/openvswitch/vhost-user0
 --vdev=net_virtio_user1,mac=00:00:00:00:00:01,path=/var/run/openvswitch/vhost-user1
-- -p 0x3
more detail core dump message

Program terminated with signal 11, Segmentation fault.
#0  0x0000000000443c9c in find_suitable_element (bound=0, align=64,
flags=0, size=6272, heap=0x7fbc461f2a1c) at
/home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_eal/common/malloc_heap.c:134
134 if (check_hugepage_sz(flags, elem->ms->hugepage_sz))
Missing separate debuginfos, use: debuginfo-install
glibc-2.17-196.el7_4.2.x86_64 keyutils-libs-1.5.8-3.el7.x86_64
krb5-libs-1.15.1-8.el7.x86_64 libcap-ng-0.7.5-4.el7.x86_64
libcom_err-1.42.9-10.el7.x86_64 libgcc-4.8.5-16.el7_4.1.x86_64
libpcap-1.5.3-9.el7.x86_64 libselinux-2.5-12.el7.x86_64
numactl-libs-2.0.9-6.el7_2.x86_64 openssl-libs-1.0.2k-8.el7.x86_64
pcre-8.32-17.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0  0x0000000000443c9c in find_suitable_element (bound=0, align=64,
flags=0, size=6272, heap=0x7fbc461f2a1c) at
/home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_eal/common/malloc_heap.c:134
#1  malloc_heap_alloc (heap=heap@entry=0x7fbc461f2a1c, type=type@entry=0x0,
size=size@entry=6272, flags=flags@entry=0, align=64, align@entry=1,
bound=bound@entry=0) at
/home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_eal/common/malloc_heap.c:166
#2  0x000000000044312a in rte_malloc_socket (type=type@entry=0x0,
size=size@entry=6272, align=align@entry=0, socket_arg=<optimized out>,
socket_arg@entry=-1) at
/home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_eal/common/rte_malloc.c:91
#3  0x00000000004431d1 in rte_zmalloc_socket (socket=-1, align=0,
size=6272, type=0x0) at
/home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_eal/common/rte_malloc.c:126
#4  rte_zmalloc (type=type@entry=0x0, size=size@entry=6272,
align=align@entry=0) at
/home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_eal/common/rte_malloc.c:135
#5  0x00000000006bec48 in vhost_new_device () at
/home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_vhost/vhost.c:311
#6  0x00000000006bd685 in vhost_user_add_connection (fd=fd@entry=66,
vsocket=vsocket@entry=0x1197560) at
/home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_vhost/socket.c:224
#7  0x00000000006bdbf6 in vhost_user_server_new_connection (fd=66, fd@entry=54,
dat=dat@entry=0x1197560, remove=remove@entry=0x7fbbafffe9dc) at
/home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_vhost/socket.c:284
#8  0x00000000006bc48c in fdset_event_dispatch (arg=0xc1ace0
<vhost_user+8192>) at
/home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_vhost/fd_man.c:308
#9  0x00007fbc450fee25 in start_thread () from /usr/lib64/libpthread.so.0
#10 0x00007fbc446e134d in clone () from /usr/lib64/libc.so.6
(gdb) fr 0
#0  0x0000000000443c9c in find_suitable_element (bound=0, align=64,
flags=0, size=6272, heap=0x7fbc461f2a1c) at
/home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_eal/common/malloc_heap.c:134
134 if (check_hugepage_sz(flags, elem->ms->hugepage_sz))
(gdb) p elem->ms
$1 = (const struct rte_memseg *) 0x7fa4f3ebb01c
(gdb) p *elem->ms
Cannot access memory at address 0x7fa4f3ebb01c
(gdb) p *elem
$2 = {heap = 0x7fa4f3eeda1c, prev = 0x0, free_list = {le_next = 0x0,
le_prev = 0x7fa4f3eeda7c}, ms = 0x7fa4f3ebb01c, state = ELEM_FREE, pad = 0,
size = 1073439232}
(gdb)  disassemble 0x0000000000443c9c
Dump of assembler code for function malloc_heap_alloc:
=> 0x0000000000443c9c <+156>: mov    0x18(%rax),%rax
   0x0000000000443ca0 <+160>: test   %r15d,%r15d
   0x0000000000443ca3 <+163>: je     0x443d7c <malloc_heap_alloc+380>
   0x0000000000443ca9 <+169>: cmp    $0x10000000,%rax
   0x0000000000443caf <+175>: je     0x443d25 <malloc_heap_alloc+293>
---Type <return> to continue, or q <return> to quit---q
Quit
(gdb) info reg rax
rax            0x7fa4f3ebb01c 140346443673628

Is  the   dpdk-socket-mem    too small ?

Thanks



O Mahony, Billy <billy.o.mahony@intel.com> 于2018年8月21日周二 下午4:17写道：

> Hi,
>
>
>
> One thing to look out for with DPDK < 18.05 is that you need to used 1GB
> huge pages (and no more than eight of them) to use virtio. I’m not sure if
> that is the issue you have as I think it I don’t remember it causing a seg
> fault. But is certainly worth checking.
>
>
>
> If that does not work please send the info Ciara refers to as well as the
> ovs-vsctl interface config for the ovs vhost backend.
>
>
>
> Thanks,
>
> Billy
>
>
>
> *From:* ovs-discuss-bounces@openvswitch.org [mailto:
> ovs-discuss-bounces@openvswitch.org] *On Behalf Of *Loftus, Ciara
> *Sent:* Tuesday, August 21, 2018 9:06 AM
> *To:* gmzhang76@gmail.com; ovs-discuss@openvswitch.org
> *Cc:* users@dpdk.org
> *Subject:* Re: [ovs-discuss] ovs-dpdk crash when use vhost-user in docker
>
>
>
> Hi,
>
>
>
> I am cc-ing the DPDK users’ list as the SEGV originates in the DPDK vHost
> code and somebody there might be able to help too.
>
> Could you provide more information about your environment please? eg. OVS
> & DPDK versions, hugepage configuration, etc.
>
>
>
> Thanks,
>
> Ciara
>
>
>
> *From:* ovs-discuss-bounces@openvswitch.org [
> mailto:ovs-discuss-bounces@openvswitch.org
> <ovs-discuss-bounces@openvswitch.org>] *On Behalf Of *???
> *Sent:* Monday, August 20, 2018 12:06 PM
> *To:* ovs-discuss@openvswitch.org
> *Subject:* [ovs-discuss] ovs-dpdk crash when use vhost-user in docker
>
>
>
> Hi,
>
>
>
>    I used ovs-dpdk  as bridge  and l2fwd  as container. When l2fwd was
> runned ,the ovs-dpdk was crashed.
>
>
>
> My command is :
>
>
>
>     docker run -it --privileged --name=dpdk-docker  -v
> /dev/hugepages:/mnt/huge -v
> /usr/local/var/run/openvswitch:/var/run/openvswitch dpdk-docker
>
> ./l2fwd -c 0x06 -n 4  --socket-mem=1024  --no-pci
> --vdev=net_virtio_user0,mac=00:00:00:00:00:05,path=/var/run/openvswitch/vhost-user0
>  --vdev=net_virtio_user1,mac=00:00:00:00:00:01,path=/var/run/openvswitch/vhost-user1
> -- -p 0x3
>
>
>
> The crash log
>
>
>
> Program terminated with signal 11, Segmentation fault.
>
> #0  0x0000000000445828 in malloc_elem_alloc ()
>
> Missing separate debuginfos, use: debuginfo-install
> glibc-2.17-196.el7_4.2.x86_64 keyutils-libs-1.5.8-3.el7.x86_64
> krb5-libs-1.15.1-8.el7.x86_64 libcap-ng-0.7.5-4.el7.x86_64
> libcom_err-1.42.9-10.el7.x86_64 libgcc-4.8.5-16.el7_4.1.x86_64
> libpcap-1.5.3-9.el7.x86_64 libselinux-2.5-12.el7.x86_64
> numactl-libs-2.0.9-6.el7_2.x86_64 openssl-libs-1.0.2k-8.el7.x86_64
> pcre-8.32-17.el7.x86_64 zlib-1.2.7-17.el7.x86_64
>
> (gdb) bt
>
> #0  0x0000000000445828 in malloc_elem_alloc ()
>
> #1  0x0000000000445e5d in malloc_heap_alloc ()
>
> #2  0x0000000000444c74 in rte_zmalloc ()
>
> #3  0x00000000006c16bf in vhost_new_device ()
>
> #4  0x00000000006bfaf4 in vhost_user_add_connection ()
>
> #5  0x00000000006beb88 in fdset_event_dispatch ()
>
> #6  0x00007f613b288e25 in start_thread () from /usr/lib64/libpthread.so.0
>
> #7  0x00007f613a86b34d in clone () from /usr/lib64/libc.so.6
>
>
>
> My OVS  version is 2.9.1 , DPDK version is 17.11.3
>
>
>
>
>
> Thanks
>
>
>
>
>
>
>
>
>

Re: [dpdk-users] [ovs-discuss] ovs-dpdk crash when use vhost-user in docker

[张广明]

Hi,

     This issue was resolved. The cause is i miss a  parameter --file-prefix
when run l2fwd

Thanks Billy and  Ciara




张广明 <gmzhang76@gmail.com> 于2018年8月21日周二 下午4:59写道：

> Hi,  Ciara and  Billy
>
> Thanks for your reply
>
>  The default huge page size that i used  is 1GB .
> root@localhost openvswitch]# cat /proc/cmdline
> BOOT_IMAGE=/vmlinuz-3.10.0-514.el7.x86_64 root=/dev/mapper/centos-root ro
> crashkernel=auto iommu=pt intel_iommu=on default_hugepagesz=1G
> hugepagesz=1G hugepages=2 rd.lvm.lv=centos/root rd.lvm.lv=centos/swap
> rd.lvm.lv=centos/usr rhgb
>
> The huge page number is 4
> [root@localhost openvswitch]# cat /proc/meminfo | grep Huge
> AnonHugePages:     14336 kB
> HugePages_Total:       4
> HugePages_Free:        2
> HugePages_Rsvd:        0
> HugePages_Surp:        0
> Hugepagesize:    1048576 kB
>
>
> My ovs dpdk configure is
> [root@localhost openvswitch]# ovs-vsctl --no-wait get Open_vSwitch .
> other_config
> {dpdk-init="true", dpdk-socket-mem="2048,0", pmd-cpu-mask="0x01"}
>
> My ovs configure
> [root@localhost openvswitch]# ovs-vsctl show
> d2b6062a-4d6f-46f6-8fa4-66dca6b06c96
>     Manager "tcp:192.168.15.18:6640"
>         is_connected: true
>     Bridge br-router
>         Port "p2p1"
>             Interface "p2p1"
>                 type: dpdk
>                 options: {dpdk-devargs="0000:01:00.0"}
>         Port patch-gtp
>             Interface patch-gtp
>                 type: patch
>                 options: {peer=patch-router}
>         Port br-router
>             Interface br-router
>                 type: internal
>     Bridge "br0"
>         Controller "tcp:192.168.15.18:6633"
>             is_connected: true
>         fail_mode: secure
>         Port "p1p1"
>             Interface "p1p1"
>                 type: dpdk
>                 options: {dpdk-devargs="0000:03:00.0"}
>         Port patch-router
>             Interface patch-router
>                 type: patch
>                 options: {peer=patch-gtp}
>         Port "br0"
>             Interface "br0"
>                 type: internal
>         Port "vhost-user1"
>             Interface "vhost-user1"
>                 type: dpdkvhostuser
>         Port "vhost-user0"
>             Interface "vhost-user0"
>                 type: dpdkvhostuser
>     Bridge br-vxlan
>         Port br-vxlan
>             Interface br-vxlan
>                 type: internal
>
>
> Docker running command is
>
>    docker run -it --privileged --name=dpdk-docker  -v
> /dev/hugepages:/mnt/huge -v
> /usr/local/var/run/openvswitch:/var/run/openvswitch dpdk-docker
>
> ./l2fwd -c 0x06 -n 4  --socket-mem=1024  --no-pci
> --vdev=net_virtio_user0,mac=00:00:00:00:00:05,path=/var/run/openvswitch/vhost-user0
>  --vdev=net_virtio_user1,mac=00:00:00:00:00:01,path=/var/run/openvswitch/vhost-user1
> -- -p 0x3
> more detail core dump message
>
> Program terminated with signal 11, Segmentation fault.
> #0  0x0000000000443c9c in find_suitable_element (bound=0, align=64,
> flags=0, size=6272, heap=0x7fbc461f2a1c) at
> /home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_eal/common/malloc_heap.c:134
> 134 if (check_hugepage_sz(flags, elem->ms->hugepage_sz))
> Missing separate debuginfos, use: debuginfo-install
> glibc-2.17-196.el7_4.2.x86_64 keyutils-libs-1.5.8-3.el7.x86_64
> krb5-libs-1.15.1-8.el7.x86_64 libcap-ng-0.7.5-4.el7.x86_64
> libcom_err-1.42.9-10.el7.x86_64 libgcc-4.8.5-16.el7_4.1.x86_64
> libpcap-1.5.3-9.el7.x86_64 libselinux-2.5-12.el7.x86_64
> numactl-libs-2.0.9-6.el7_2.x86_64 openssl-libs-1.0.2k-8.el7.x86_64
> pcre-8.32-17.el7.x86_64 zlib-1.2.7-17.el7.x86_64
> (gdb) bt
> #0  0x0000000000443c9c in find_suitable_element (bound=0, align=64,
> flags=0, size=6272, heap=0x7fbc461f2a1c) at
> /home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_eal/common/malloc_heap.c:134
> #1  malloc_heap_alloc (heap=heap@entry=0x7fbc461f2a1c, type=type@entry=0x0,
> size=size@entry=6272, flags=flags@entry=0, align=64, align@entry=1,
> bound=bound@entry=0) at
> /home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_eal/common/malloc_heap.c:166
> #2  0x000000000044312a in rte_malloc_socket (type=type@entry=0x0,
> size=size@entry=6272, align=align@entry=0, socket_arg=<optimized out>,
> socket_arg@entry=-1) at
> /home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_eal/common/rte_malloc.c:91
> #3  0x00000000004431d1 in rte_zmalloc_socket (socket=-1, align=0,
> size=6272, type=0x0) at
> /home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_eal/common/rte_malloc.c:126
> #4  rte_zmalloc (type=type@entry=0x0, size=size@entry=6272,
> align=align@entry=0) at
> /home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_eal/common/rte_malloc.c:135
> #5  0x00000000006bec48 in vhost_new_device () at
> /home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_vhost/vhost.c:311
> #6  0x00000000006bd685 in vhost_user_add_connection (fd=fd@entry=66,
> vsocket=vsocket@entry=0x1197560) at
> /home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_vhost/socket.c:224
> #7  0x00000000006bdbf6 in vhost_user_server_new_connection (fd=66, fd@entry=54,
> dat=dat@entry=0x1197560, remove=remove@entry=0x7fbbafffe9dc) at
> /home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_vhost/socket.c:284
> #8  0x00000000006bc48c in fdset_event_dispatch (arg=0xc1ace0
> <vhost_user+8192>) at
> /home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_vhost/fd_man.c:308
> #9  0x00007fbc450fee25 in start_thread () from /usr/lib64/libpthread.so.0
> #10 0x00007fbc446e134d in clone () from /usr/lib64/libc.so.6
> (gdb) fr 0
> #0  0x0000000000443c9c in find_suitable_element (bound=0, align=64,
> flags=0, size=6272, heap=0x7fbc461f2a1c) at
> /home/gmzhang/work/dpdk-stable-17.11.3/lib/librte_eal/common/malloc_heap.c:134
> 134 if (check_hugepage_sz(flags, elem->ms->hugepage_sz))
> (gdb) p elem->ms
> $1 = (const struct rte_memseg *) 0x7fa4f3ebb01c
> (gdb) p *elem->ms
> Cannot access memory at address 0x7fa4f3ebb01c
> (gdb) p *elem
> $2 = {heap = 0x7fa4f3eeda1c, prev = 0x0, free_list = {le_next = 0x0,
> le_prev = 0x7fa4f3eeda7c}, ms = 0x7fa4f3ebb01c, state = ELEM_FREE, pad = 0,
> size = 1073439232}
> (gdb)  disassemble 0x0000000000443c9c
> Dump of assembler code for function malloc_heap_alloc:
> => 0x0000000000443c9c <+156>: mov    0x18(%rax),%rax
>    0x0000000000443ca0 <+160>: test   %r15d,%r15d
>    0x0000000000443ca3 <+163>: je     0x443d7c <malloc_heap_alloc+380>
>    0x0000000000443ca9 <+169>: cmp    $0x10000000,%rax
>    0x0000000000443caf <+175>: je     0x443d25 <malloc_heap_alloc+293>
> ---Type <return> to continue, or q <return> to quit---q
> Quit
> (gdb) info reg rax
> rax            0x7fa4f3ebb01c 140346443673628
>
> Is  the   dpdk-socket-mem    too small ?
>
> Thanks
>
>
>
> O Mahony, Billy <billy.o.mahony@intel.com> 于2018年8月21日周二 下午4:17写道：
>
>> Hi,
>>
>>
>>
>> One thing to look out for with DPDK < 18.05 is that you need to used 1GB
>> huge pages (and no more than eight of them) to use virtio. I’m not sure if
>> that is the issue you have as I think it I don’t remember it causing a seg
>> fault. But is certainly worth checking.
>>
>>
>>
>> If that does not work please send the info Ciara refers to as well as the
>> ovs-vsctl interface config for the ovs vhost backend.
>>
>>
>>
>> Thanks,
>>
>> Billy
>>
>>
>>
>> *From:* ovs-discuss-bounces@openvswitch.org [mailto:
>> ovs-discuss-bounces@openvswitch.org] *On Behalf Of *Loftus, Ciara
>> *Sent:* Tuesday, August 21, 2018 9:06 AM
>> *To:* gmzhang76@gmail.com; ovs-discuss@openvswitch.org
>> *Cc:* users@dpdk.org
>> *Subject:* Re: [ovs-discuss] ovs-dpdk crash when use vhost-user in docker
>>
>>
>>
>> Hi,
>>
>>
>>
>> I am cc-ing the DPDK users’ list as the SEGV originates in the DPDK vHost
>> code and somebody there might be able to help too.
>>
>> Could you provide more information about your environment please? eg. OVS
>> & DPDK versions, hugepage configuration, etc.
>>
>>
>>
>> Thanks,
>>
>> Ciara
>>
>>
>>
>> *From:* ovs-discuss-bounces@openvswitch.org [
>> mailto:ovs-discuss-bounces@openvswitch.org
>> <ovs-discuss-bounces@openvswitch.org>] *On Behalf Of *???
>> *Sent:* Monday, August 20, 2018 12:06 PM
>> *To:* ovs-discuss@openvswitch.org
>> *Subject:* [ovs-discuss] ovs-dpdk crash when use vhost-user in docker
>>
>>
>>
>> Hi,
>>
>>
>>
>>    I used ovs-dpdk  as bridge  and l2fwd  as container. When l2fwd was
>> runned ,the ovs-dpdk was crashed.
>>
>>
>>
>> My command is :
>>
>>
>>
>>     docker run -it --privileged --name=dpdk-docker  -v
>> /dev/hugepages:/mnt/huge -v
>> /usr/local/var/run/openvswitch:/var/run/openvswitch dpdk-docker
>>
>> ./l2fwd -c 0x06 -n 4  --socket-mem=1024  --no-pci
>> --vdev=net_virtio_user0,mac=00:00:00:00:00:05,path=/var/run/openvswitch/vhost-user0
>>  --vdev=net_virtio_user1,mac=00:00:00:00:00:01,path=/var/run/openvswitch/vhost-user1
>> -- -p 0x3
>>
>>
>>
>> The crash log
>>
>>
>>
>> Program terminated with signal 11, Segmentation fault.
>>
>> #0  0x0000000000445828 in malloc_elem_alloc ()
>>
>> Missing separate debuginfos, use: debuginfo-install
>> glibc-2.17-196.el7_4.2.x86_64 keyutils-libs-1.5.8-3.el7.x86_64
>> krb5-libs-1.15.1-8.el7.x86_64 libcap-ng-0.7.5-4.el7.x86_64
>> libcom_err-1.42.9-10.el7.x86_64 libgcc-4.8.5-16.el7_4.1.x86_64
>> libpcap-1.5.3-9.el7.x86_64 libselinux-2.5-12.el7.x86_64
>> numactl-libs-2.0.9-6.el7_2.x86_64 openssl-libs-1.0.2k-8.el7.x86_64
>> pcre-8.32-17.el7.x86_64 zlib-1.2.7-17.el7.x86_64
>>
>> (gdb) bt
>>
>> #0  0x0000000000445828 in malloc_elem_alloc ()
>>
>> #1  0x0000000000445e5d in malloc_heap_alloc ()
>>
>> #2  0x0000000000444c74 in rte_zmalloc ()
>>
>> #3  0x00000000006c16bf in vhost_new_device ()
>>
>> #4  0x00000000006bfaf4 in vhost_user_add_connection ()
>>
>> #5  0x00000000006beb88 in fdset_event_dispatch ()
>>
>> #6  0x00007f613b288e25 in start_thread () from /usr/lib64/libpthread.so.0
>>
>> #7  0x00007f613a86b34d in clone () from /usr/lib64/libc.so.6
>>
>>
>>
>> My OVS  version is 2.9.1 , DPDK version is 17.11.3
>>
>>
>>
>>
>>
>> Thanks
>>
>>
>>
>>
>>
>>
>>
>>
>>
>