[dpdk-users] dpdk with snort 2.9.9 receive from DPDK ring

[dpdk-users] dpdk with snort 2.9.9 receive from DPDK ring

[Ming Fu]

I am making a snort DAQ module to receive packet from DPDK capture through a ring. The snort version is 2.9.9 and DPDK version is 17.08. The code is very similar to the multi_process/client_server_mp example. The DPDK is initialized in daq initialize function and mbuf received from daq acquire function. Snort inspects the mbuf but does not re-inject back to the network, so daq free the mbuf by rte_pktmbuf_free(mbuf) in the same thread as it calls rte_ring_dequeue_burst();
The snort successfully received the first burst of 32 packets, but it fails on the first rte_pktmbuf_free().

Seems some internal dpdk function pointer is NULL.
#0  0x0000000000000000 in ?? ()
#1  0x00000000004f7a8d in rte_mempool_ops_enqueue_bulk (n=<optimized out>, obj_table=<optimized out>, mp=<optimized out>)
    at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:497
#2  __mempool_generic_put (cache=<optimized out>, n=1, obj_table=0x7fffffffe6e8, mp=<optimized out>) at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1069
#3  rte_mempool_generic_put (flags=<optimized out>, cache=<optimized out>, n=1, obj_table=0x7fffffffe6e8, mp=<optimized out>)
    at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1109
#4  rte_mempool_put_bulk (n=1, obj_table=0x7fffffffe6e8, mp=<optimized out>) at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1132
#5  rte_mempool_put (obj=<optimized out>, mp=<optimized out>) at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1150
#6  rte_mbuf_raw_free (m=<optimized out>) at /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:853
#7  rte_pktmbuf_free_seg (m=<optimized out>) at /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:1349
#8  rte_pktmbuf_free (m=<optimized out>) at /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:1369
#9  dpdk_daq_acquire (handle=0x1af6910, cnt=0, callback=<optimized out>, metaback=<optimized out>, user=<optimized out>) at daq_dpdk.c:234
#10 0x00000000004551d3 in DAQ_Acquire ()
#11 0x0000000000437828 in SnortMain ()
#12 0x00007ffff61af830 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#13 0x0000000000406d29 in _start ()

I notice that snort has three threads,
(gdb) info thread
  Id   Target Id         Frame
* 1    Thread 0x7ffff7fdf8c0 (LWP 42472) "snort" 0x0000000000000000 in ?? ()
  2    Thread 0x7fffdff5a700 (LWP 42481) "eal-intr-thread" 0x00007ffff62969d3 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
  3    Thread 0x7fffdf4d5700 (LWP 42482) "snort" 0x00007ffff625b30d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6

Would it be a problem if the rte_pktmbuf_free() is not called from the eal thread?

Thanks
Ming

Re: [dpdk-users] dpdk with snort 2.9.9 receive from DPDK ring

[Padam Jeet Singh]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii, Size: 4123 bytes --]


> On 30-Aug-2017, at 2:21 AM, Ming Fu <Ming.Fu@esentire.com> wrote:
>
> I am making a snort DAQ module to receive packet from DPDK capture through a ring. The snort version is 2.9.9 and DPDK version is 17.08. The code is very similar to the multi_process/client_server_mp example. The DPDK is initialized in daq initialize function and mbuf received from daq acquire function. Snort inspects the mbuf but does not re-inject back to the network, so daq free the mbuf by rte_pktmbuf_free(mbuf) in the same thread as it calls rte_ring_dequeue_burst();
> The snort successfully received the first burst of 32 packets, but it fails on the first rte_pktmbuf_free().
>
> Seems some internal dpdk function pointer is NULL.
> #0  0x0000000000000000 in ?? ()
> #1  0x00000000004f7a8d in rte_mempool_ops_enqueue_bulk (n=<optimized out>, obj_table=<optimized out>, mp=<optimized out>)
>    at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:497
> #2  __mempool_generic_put (cache=<optimized out>, n=1, obj_table=0x7fffffffe6e8, mp=<optimized out>) at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1069
> #3  rte_mempool_generic_put (flags=<optimized out>, cache=<optimized out>, n=1, obj_table=0x7fffffffe6e8, mp=<optimized out>)
>    at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1109
> #4  rte_mempool_put_bulk (n=1, obj_table=0x7fffffffe6e8, mp=<optimized out>) at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1132
> #5  rte_mempool_put (obj=<optimized out>, mp=<optimized out>) at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1150
> #6  rte_mbuf_raw_free (m=<optimized out>) at /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:853
> #7  rte_pktmbuf_free_seg (m=<optimized out>) at /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:1349
> #8  rte_pktmbuf_free (m=<optimized out>) at /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:1369
> #9  dpdk_daq_acquire (handle=0x1af6910, cnt=0, callback=<optimized out>, metaback=<optimized out>, user=<optimized out>) at daq_dpdk.c:234
> #10 0x00000000004551d3 in DAQ_Acquire ()
> #11 0x0000000000437828 in SnortMain ()
> #12 0x00007ffff61af830 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
> #13 0x0000000000406d29 in _start ()
>
> I notice that snort has three threads,
> (gdb) info thread
>  Id   Target Id         Frame
> * 1    Thread 0x7ffff7fdf8c0 (LWP 42472) "snort" 0x0000000000000000 in ?? ()
>  2    Thread 0x7fffdff5a700 (LWP 42481) "eal-intr-thread" 0x00007ffff62969d3 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
>  3    Thread 0x7fffdf4d5700 (LWP 42482) "snort" 0x00007ffff625b30d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
>
> Would it be a problem if the rte_pktmbuf_free() is not called from the eal thread?
>

The rte_mbuf_free internally calls rte_mempool put calls - which should only be called from a EAL thread - so yes, only call rte_pktmbuf_free from an EAL thread.

> Thanks
> Ming
>
>
>
>
>



-------------------------------------------------------------------------------------------------------------------------------------------------------
NOTICE

Please Consider the Environment before printing this Email.

This email was sent from within Inventum Technologies Private Limited (https://www.inventum.net). This email (and any attachments or hyperlinks within it) may contain information that is confidential, legally privileged or otherwise protected from disclosure. If you are not the intended recipient of this email, you are not entitled to use, disclose, distribute, copy, print, disseminate or rely on this email in any way. If you have received this email in error, please notify the sender immediately by telephone or email and destroy it, and all copies of it.

We have taken steps to ensure that this email (and any attachments) are free from computer viruses and the like. However, it is the recipient's responsibility to ensure that it is actually virus free. Any emails that you send to us may be monitored for the purposes of ascertaining whether the communication complies with the law and our policies.

Re: [dpdk-users] dpdk with snort 2.9.9 receive from DPDK ring

[Andriy Berestovskyy]

Hey,
1. There is a non-EAL thread support since DPDK 2.0, so any thread can
call rte_pktmbuf_free()

2. From the traceback it looks like the memory pool enqueue() op is
NULL for a reason. Note that to free an mbuf, the mbuf header should
be valid and point to a valid memory pool mbuf belongs.

Try to compile DPDK in debug mode and you might see the reason.

Andriy

On Wed, Aug 30, 2017 at 7:15 AM, Padam Jeet Singh
<padam.singh@inventum.net> wrote:
>
>> On 30-Aug-2017, at 2:21 AM, Ming Fu <Ming.Fu@esentire.com> wrote:
>>
>> I am making a snort DAQ module to receive packet from DPDK capture through a ring. The snort version is 2.9.9 and DPDK version is 17.08. The code is very similar to the multi_process/client_server_mp example. The DPDK is initialized in daq initialize function and mbuf received from daq acquire function. Snort inspects the mbuf but does not re-inject back to the network, so daq free the mbuf by rte_pktmbuf_free(mbuf) in the same thread as it calls rte_ring_dequeue_burst();
>> The snort successfully received the first burst of 32 packets, but it fails on the first rte_pktmbuf_free().
>>
>> Seems some internal dpdk function pointer is NULL.
>> #0  0x0000000000000000 in ?? ()
>> #1  0x00000000004f7a8d in rte_mempool_ops_enqueue_bulk (n=<optimized out>, obj_table=<optimized out>, mp=<optimized out>)
>>    at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:497
>> #2  __mempool_generic_put (cache=<optimized out>, n=1, obj_table=0x7fffffffe6e8, mp=<optimized out>) at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1069
>> #3  rte_mempool_generic_put (flags=<optimized out>, cache=<optimized out>, n=1, obj_table=0x7fffffffe6e8, mp=<optimized out>)
>>    at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1109
>> #4  rte_mempool_put_bulk (n=1, obj_table=0x7fffffffe6e8, mp=<optimized out>) at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1132
>> #5  rte_mempool_put (obj=<optimized out>, mp=<optimized out>) at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1150
>> #6  rte_mbuf_raw_free (m=<optimized out>) at /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:853
>> #7  rte_pktmbuf_free_seg (m=<optimized out>) at /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:1349
>> #8  rte_pktmbuf_free (m=<optimized out>) at /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:1369
>> #9  dpdk_daq_acquire (handle=0x1af6910, cnt=0, callback=<optimized out>, metaback=<optimized out>, user=<optimized out>) at daq_dpdk.c:234
>> #10 0x00000000004551d3 in DAQ_Acquire ()
>> #11 0x0000000000437828 in SnortMain ()
>> #12 0x00007ffff61af830 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
>> #13 0x0000000000406d29 in _start ()
>>
>> I notice that snort has three threads,
>> (gdb) info thread
>>  Id   Target Id         Frame
>> * 1    Thread 0x7ffff7fdf8c0 (LWP 42472) "snort" 0x0000000000000000 in ?? ()
>>  2    Thread 0x7fffdff5a700 (LWP 42481) "eal-intr-thread" 0x00007ffff62969d3 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
>>  3    Thread 0x7fffdf4d5700 (LWP 42482) "snort" 0x00007ffff625b30d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
>>
>> Would it be a problem if the rte_pktmbuf_free() is not called from the eal thread?
>>
>
> The rte_mbuf_free internally calls rte_mempool put calls - which should only be called from a EAL thread - so yes, only call rte_pktmbuf_free from an EAL thread.
>
>> Thanks
>> Ming
>>
>>
>>
>>
>>
>
>
>
> -------------------------------------------------------------------------------------------------------------------------------------------------------
> NOTICE
>
> Please Consider the Environment before printing this Email.
>
> This email was sent from within Inventum Technologies Private Limited (https://www.inventum.net). This email (and any attachments or hyperlinks within it) may contain information that is confidential, legally privileged or otherwise protected from disclosure. If you are not the intended recipient of this email, you are not entitled to use, disclose, distribute, copy, print, disseminate or rely on this email in any way. If you have received this email in error, please notify the sender immediately by telephone or email and destroy it, and all copies of it.
>
> We have taken steps to ensure that this email (and any attachments) are free from computer viruses and the like. However, it is the recipient's responsibility to ensure that it is actually virus free. Any emails that you send to us may be monitored for the purposes of ascertaining whether the communication complies with the law and our policies.
>



-- 
Andriy Berestovskyy

Re: [dpdk-users] dpdk with snort 2.9.9 receive from DPDK ring

[tom.barbette]

Hi all,

We have a proof of concept but working implementation of DPDK ring in snort available here : https://github.com/IurmanJ/daq-2.0.6/. It is a clone of the daq-2.0.6, patched for DPDK and DPDK ring support. 
The interesting file would be https://github.com/IurmanJ/daq-2.0.6/blob/master/os-daq-modules/daq_dpdkring.c . I guess you can use it to compare the difference, or use that one.

For autorship notice, it has been made by Justin Iurman for his master thesis.

Cheers,

Tom Barbette 
PhD Student @ Université de Liège 

----- Mail original -----
> De: "Andriy Berestovskyy" <aber@semihalf.com>
> À: "Padam Jeet Singh" <padam.singh@inventum.net>
> Cc: "Ming Fu" <Ming.Fu@esentire.com>, users@dpdk.org
> Envoyé: Mercredi 30 Août 2017 11:18:39
> Objet: Re: [dpdk-users] dpdk with snort 2.9.9 receive from DPDK ring

> Hey,
> 1. There is a non-EAL thread support since DPDK 2.0, so any thread can
> call rte_pktmbuf_free()
> 
> 2. From the traceback it looks like the memory pool enqueue() op is
> NULL for a reason. Note that to free an mbuf, the mbuf header should
> be valid and point to a valid memory pool mbuf belongs.
> 
> Try to compile DPDK in debug mode and you might see the reason.
> 
> Andriy
> 
> On Wed, Aug 30, 2017 at 7:15 AM, Padam Jeet Singh
> <padam.singh@inventum.net> wrote:
>>
>>> On 30-Aug-2017, at 2:21 AM, Ming Fu <Ming.Fu@esentire.com> wrote:
>>>
>>> I am making a snort DAQ module to receive packet from DPDK capture through a
>>> ring. The snort version is 2.9.9 and DPDK version is 17.08. The code is very
>>> similar to the multi_process/client_server_mp example. The DPDK is initialized
>>> in daq initialize function and mbuf received from daq acquire function. Snort
>>> inspects the mbuf but does not re-inject back to the network, so daq free the
>>> mbuf by rte_pktmbuf_free(mbuf) in the same thread as it calls
>>> rte_ring_dequeue_burst();
>>> The snort successfully received the first burst of 32 packets, but it fails on
>>> the first rte_pktmbuf_free().
>>>
>>> Seems some internal dpdk function pointer is NULL.
>>> #0  0x0000000000000000 in ?? ()
>>> #1  0x00000000004f7a8d in rte_mempool_ops_enqueue_bulk (n=<optimized out>,
>>> obj_table=<optimized out>, mp=<optimized out>)
>>>    at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:497
>>> #2  __mempool_generic_put (cache=<optimized out>, n=1, obj_table=0x7fffffffe6e8,
>>> mp=<optimized out>) at
>>> /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1069
>>> #3  rte_mempool_generic_put (flags=<optimized out>, cache=<optimized out>, n=1,
>>> obj_table=0x7fffffffe6e8, mp=<optimized out>)
>>>    at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1109
>>> #4  rte_mempool_put_bulk (n=1, obj_table=0x7fffffffe6e8, mp=<optimized out>) at
>>> /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1132
>>> #5  rte_mempool_put (obj=<optimized out>, mp=<optimized out>) at
>>> /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1150
>>> #6  rte_mbuf_raw_free (m=<optimized out>) at
>>> /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:853
>>> #7  rte_pktmbuf_free_seg (m=<optimized out>) at
>>> /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:1349
>>> #8  rte_pktmbuf_free (m=<optimized out>) at
>>> /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:1369
>>> #9  dpdk_daq_acquire (handle=0x1af6910, cnt=0, callback=<optimized out>,
>>> metaback=<optimized out>, user=<optimized out>) at daq_dpdk.c:234
>>> #10 0x00000000004551d3 in DAQ_Acquire ()
>>> #11 0x0000000000437828 in SnortMain ()
>>> #12 0x00007ffff61af830 in __libc_start_main () from
>>> /lib/x86_64-linux-gnu/libc.so.6
>>> #13 0x0000000000406d29 in _start ()
>>>
>>> I notice that snort has three threads,
>>> (gdb) info thread
>>>  Id   Target Id         Frame
>>> * 1    Thread 0x7ffff7fdf8c0 (LWP 42472) "snort" 0x0000000000000000 in ?? ()
>>>  2    Thread 0x7fffdff5a700 (LWP 42481) "eal-intr-thread" 0x00007ffff62969d3 in
>>>  epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
>>>  3    Thread 0x7fffdf4d5700 (LWP 42482) "snort" 0x00007ffff625b30d in nanosleep
>>>  () from /lib/x86_64-linux-gnu/libc.so.6
>>>
>>> Would it be a problem if the rte_pktmbuf_free() is not called from the eal
>>> thread?
>>>
>>
>> The rte_mbuf_free internally calls rte_mempool put calls - which should only be
>> called from a EAL thread - so yes, only call rte_pktmbuf_free from an EAL
>> thread.
>>
>>> Thanks
>>> Ming
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>> -------------------------------------------------------------------------------------------------------------------------------------------------------
>> NOTICE
>>
>> Please Consider the Environment before printing this Email.
>>
>> This email was sent from within Inventum Technologies Private Limited
>> (https://www.inventum.net). This email (and any attachments or hyperlinks
>> within it) may contain information that is confidential, legally privileged or
>> otherwise protected from disclosure. If you are not the intended recipient of
>> this email, you are not entitled to use, disclose, distribute, copy, print,
>> disseminate or rely on this email in any way. If you have received this email
>> in error, please notify the sender immediately by telephone or email and
>> destroy it, and all copies of it.
>>
>> We have taken steps to ensure that this email (and any attachments) are free
>> from computer viruses and the like. However, it is the recipient's
>> responsibility to ensure that it is actually virus free. Any emails that you
>> send to us may be monitored for the purposes of ascertaining whether the
>> communication complies with the law and our policies.
>>
> 
> 
> 
> --
> Andriy Berestovskyy

Re: [dpdk-users] dpdk with snort 2.9.9 receive from DPDK ring

[Ming Fu]

Hi Tom,

Thanks for share with me the dpdkring DAQ. I tried to use it with dpdk-17.08. It seems to be coded for an older version of DPDK, some API call has missing argument. I assume these are extras added to the newer version of DPDK.

Conceptually, my code is the very similar to yours. However, I haven't manage to include 
	include $(RTE_SDK)/mk/rte.vars.mk
	include $(RTE_SDK)/mk/rte.extlib.mk
into the daq Makefile for dpdkring. Do you know if you have to include any DPDK mk include file from DPDK mk directory? I didn't find any such change to the DAQ Makefile from your git repo, but the DPDK doc seems to suggest such.

Ming

-----Original Message-----
From: tom.barbette@ulg.ac.be [mailto:tom.barbette@ulg.ac.be] 
Sent: August-30-17 7:55 AM
To: Andriy Berestovskyy <aber@semihalf.com>
Cc: Padam Jeet Singh <padam.singh@inventum.net>; Ming Fu <Ming.Fu@esentire.com>; users@dpdk.org
Subject: Re: [dpdk-users] dpdk with snort 2.9.9 receive from DPDK ring

Hi all,

We have a proof of concept but working implementation of DPDK ring in snort available here : https://github.com/IurmanJ/daq-2.0.6/. It is a clone of the daq-2.0.6, patched for DPDK and DPDK ring support. 
The interesting file would be https://github.com/IurmanJ/daq-2.0.6/blob/master/os-daq-modules/daq_dpdkring.c . I guess you can use it to compare the difference, or use that one.

For autorship notice, it has been made by Justin Iurman for his master thesis.

Cheers,

Tom Barbette
PhD Student @ Université de Liège 

----- Mail original -----
> De: "Andriy Berestovskyy" <aber@semihalf.com>
> À: "Padam Jeet Singh" <padam.singh@inventum.net>
> Cc: "Ming Fu" <Ming.Fu@esentire.com>, users@dpdk.org
> Envoyé: Mercredi 30 Août 2017 11:18:39
> Objet: Re: [dpdk-users] dpdk with snort 2.9.9 receive from DPDK ring

> Hey,
> 1. There is a non-EAL thread support since DPDK 2.0, so any thread can 
> call rte_pktmbuf_free()
> 
> 2. From the traceback it looks like the memory pool enqueue() op is 
> NULL for a reason. Note that to free an mbuf, the mbuf header should 
> be valid and point to a valid memory pool mbuf belongs.
> 
> Try to compile DPDK in debug mode and you might see the reason.
> 
> Andriy
> 
> On Wed, Aug 30, 2017 at 7:15 AM, Padam Jeet Singh 
> <padam.singh@inventum.net> wrote:
>>
>>> On 30-Aug-2017, at 2:21 AM, Ming Fu <Ming.Fu@esentire.com> wrote:
>>>
>>> I am making a snort DAQ module to receive packet from DPDK capture 
>>> through a ring. The snort version is 2.9.9 and DPDK version is 
>>> 17.08. The code is very similar to the 
>>> multi_process/client_server_mp example. The DPDK is initialized in 
>>> daq initialize function and mbuf received from daq acquire function. 
>>> Snort inspects the mbuf but does not re-inject back to the network, 
>>> so daq free the mbuf by rte_pktmbuf_free(mbuf) in the same thread as 
>>> it calls rte_ring_dequeue_burst(); The snort successfully received 
>>> the first burst of 32 packets, but it fails on the first rte_pktmbuf_free().
>>>
>>> Seems some internal dpdk function pointer is NULL.
>>> #0  0x0000000000000000 in ?? ()
>>> #1  0x00000000004f7a8d in rte_mempool_ops_enqueue_bulk (n=<optimized 
>>> out>, obj_table=<optimized out>, mp=<optimized out>)
>>>    at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:497
>>> #2  __mempool_generic_put (cache=<optimized out>, n=1, 
>>> obj_table=0x7fffffffe6e8, mp=<optimized out>) at
>>> /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1069
>>> #3  rte_mempool_generic_put (flags=<optimized out>, cache=<optimized 
>>> out>, n=1, obj_table=0x7fffffffe6e8, mp=<optimized out>)
>>>    at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1109
>>> #4  rte_mempool_put_bulk (n=1, obj_table=0x7fffffffe6e8, 
>>> mp=<optimized out>) at
>>> /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1132
>>> #5  rte_mempool_put (obj=<optimized out>, mp=<optimized out>) at
>>> /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1150
>>> #6  rte_mbuf_raw_free (m=<optimized out>) at
>>> /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:853
>>> #7  rte_pktmbuf_free_seg (m=<optimized out>) at
>>> /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:1349
>>> #8  rte_pktmbuf_free (m=<optimized out>) at
>>> /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:1369
>>> #9  dpdk_daq_acquire (handle=0x1af6910, cnt=0, callback=<optimized 
>>> out>, metaback=<optimized out>, user=<optimized out>) at 
>>> daq_dpdk.c:234
>>> #10 0x00000000004551d3 in DAQ_Acquire ()
>>> #11 0x0000000000437828 in SnortMain ()
>>> #12 0x00007ffff61af830 in __libc_start_main () from
>>> /lib/x86_64-linux-gnu/libc.so.6
>>> #13 0x0000000000406d29 in _start ()
>>>
>>> I notice that snort has three threads,
>>> (gdb) info thread
>>>  Id   Target Id         Frame
>>> * 1    Thread 0x7ffff7fdf8c0 (LWP 42472) "snort" 0x0000000000000000 in ?? ()
>>>  2    Thread 0x7fffdff5a700 (LWP 42481) "eal-intr-thread" 0x00007ffff62969d3 in
>>>  epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
>>>  3    Thread 0x7fffdf4d5700 (LWP 42482) "snort" 0x00007ffff625b30d in nanosleep
>>>  () from /lib/x86_64-linux-gnu/libc.so.6
>>>
>>> Would it be a problem if the rte_pktmbuf_free() is not called from 
>>> the eal thread?
>>>
>>
>> The rte_mbuf_free internally calls rte_mempool put calls - which 
>> should only be called from a EAL thread - so yes, only call 
>> rte_pktmbuf_free from an EAL thread.
>>
>>> Thanks
>>> Ming
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> ---------------------------------------------------------------------
>> -------------
>> NOTICE
>>
>> Please Consider the Environment before printing this Email.
>>
>> This email was sent from within Inventum Technologies Private Limited 
>> (https://www.inventum.net). This email (and any attachments or 
>> hyperlinks within it) may contain information that is confidential, 
>> legally privileged or otherwise protected from disclosure. If you are 
>> not the intended recipient of this email, you are not entitled to 
>> use, disclose, distribute, copy, print, disseminate or rely on this 
>> email in any way. If you have received this email in error, please 
>> notify the sender immediately by telephone or email and destroy it, and all copies of it.
>>
>> We have taken steps to ensure that this email (and any attachments) 
>> are free from computer viruses and the like. However, it is the 
>> recipient's responsibility to ensure that it is actually virus free. 
>> Any emails that you send to us may be monitored for the purposes of 
>> ascertaining whether the communication complies with the law and our policies.
>>
> 
> 
> 
> --
> Andriy Berestovskyy

Re: [dpdk-users] dpdk with snort 2.9.9 receive from DPDK ring

[tom.barbette]

Hi,

Yes it was done with DPDK 17.02, it is still fairly recent though.

DPDK compile process is very rigid and intended to be somehow "the master", adding some files using the propose technique. It is very difficult to hook into it for big programs like Snort. We managed to do it with the Click Modular Router, but that leads to a lot of maintainance efforts, backporting changes from some rte.*.mk with each versions.

The solution in our Snort DPDK RING DAQ (based on someone else's DPDK DAQ) is to lib against Click compiled in combined library mode, simply using -ldpdk. I think it is also what OVS does. They definitely did it before but now they seem to use some perl script, not sure if they still do that or live-patch the compile rte*mk.

Tom

----- Mail original -----
> De: "Ming Fu" <Ming.Fu@esentire.com>
> À: "tom barbette" <tom.barbette@ulg.ac.be>
> Cc: users@dpdk.org
> Envoyé: Vendredi 1 Septembre 2017 16:17:11
> Objet: RE: [dpdk-users] dpdk with snort 2.9.9 receive from DPDK ring

> Hi Tom,
> 
> Thanks for share with me the dpdkring DAQ. I tried to use it with dpdk-17.08. It
> seems to be coded for an older version of DPDK, some API call has missing
> argument. I assume these are extras added to the newer version of DPDK.
> 
> Conceptually, my code is the very similar to yours. However, I haven't manage to
> include
>	include $(RTE_SDK)/mk/rte.vars.mk
>	include $(RTE_SDK)/mk/rte.extlib.mk
> into the daq Makefile for dpdkring. Do you know if you have to include any DPDK
> mk include file from DPDK mk directory? I didn't find any such change to the
> DAQ Makefile from your git repo, but the DPDK doc seems to suggest such.
> 
> Ming
> 
> -----Original Message-----
> From: tom.barbette@ulg.ac.be [mailto:tom.barbette@ulg.ac.be]
> Sent: August-30-17 7:55 AM
> To: Andriy Berestovskyy <aber@semihalf.com>
> Cc: Padam Jeet Singh <padam.singh@inventum.net>; Ming Fu <Ming.Fu@esentire.com>;
> users@dpdk.org
> Subject: Re: [dpdk-users] dpdk with snort 2.9.9 receive from DPDK ring
> 
> Hi all,
> 
> We have a proof of concept but working implementation of DPDK ring in snort
> available here : https://github.com/IurmanJ/daq-2.0.6/. It is a clone of the
> daq-2.0.6, patched for DPDK and DPDK ring support.
> The interesting file would be
> https://github.com/IurmanJ/daq-2.0.6/blob/master/os-daq-modules/daq_dpdkring.c
> . I guess you can use it to compare the difference, or use that one.
> 
> For autorship notice, it has been made by Justin Iurman for his master thesis.
> 
> Cheers,
> 
> Tom Barbette
> PhD Student @ Université de Liège
> 
> ----- Mail original -----
>> De: "Andriy Berestovskyy" <aber@semihalf.com>
>> À: "Padam Jeet Singh" <padam.singh@inventum.net>
>> Cc: "Ming Fu" <Ming.Fu@esentire.com>, users@dpdk.org
>> Envoyé: Mercredi 30 Août 2017 11:18:39
>> Objet: Re: [dpdk-users] dpdk with snort 2.9.9 receive from DPDK ring
> 
>> Hey,
>> 1. There is a non-EAL thread support since DPDK 2.0, so any thread can
>> call rte_pktmbuf_free()
>> 
>> 2. From the traceback it looks like the memory pool enqueue() op is
>> NULL for a reason. Note that to free an mbuf, the mbuf header should
>> be valid and point to a valid memory pool mbuf belongs.
>> 
>> Try to compile DPDK in debug mode and you might see the reason.
>> 
>> Andriy
>> 
>> On Wed, Aug 30, 2017 at 7:15 AM, Padam Jeet Singh
>> <padam.singh@inventum.net> wrote:
>>>
>>>> On 30-Aug-2017, at 2:21 AM, Ming Fu <Ming.Fu@esentire.com> wrote:
>>>>
>>>> I am making a snort DAQ module to receive packet from DPDK capture
>>>> through a ring. The snort version is 2.9.9 and DPDK version is
>>>> 17.08. The code is very similar to the
>>>> multi_process/client_server_mp example. The DPDK is initialized in
>>>> daq initialize function and mbuf received from daq acquire function.
>>>> Snort inspects the mbuf but does not re-inject back to the network,
>>>> so daq free the mbuf by rte_pktmbuf_free(mbuf) in the same thread as
>>>> it calls rte_ring_dequeue_burst(); The snort successfully received
>>>> the first burst of 32 packets, but it fails on the first rte_pktmbuf_free().
>>>>
>>>> Seems some internal dpdk function pointer is NULL.
>>>> #0  0x0000000000000000 in ?? ()
>>>> #1  0x00000000004f7a8d in rte_mempool_ops_enqueue_bulk (n=<optimized
>>>> out>, obj_table=<optimized out>, mp=<optimized out>)
>>>>    at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:497
>>>> #2  __mempool_generic_put (cache=<optimized out>, n=1,
>>>> obj_table=0x7fffffffe6e8, mp=<optimized out>) at
>>>> /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1069
>>>> #3  rte_mempool_generic_put (flags=<optimized out>, cache=<optimized
>>>> out>, n=1, obj_table=0x7fffffffe6e8, mp=<optimized out>)
>>>>    at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1109
>>>> #4  rte_mempool_put_bulk (n=1, obj_table=0x7fffffffe6e8,
>>>> mp=<optimized out>) at
>>>> /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1132
>>>> #5  rte_mempool_put (obj=<optimized out>, mp=<optimized out>) at
>>>> /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1150
>>>> #6  rte_mbuf_raw_free (m=<optimized out>) at
>>>> /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:853
>>>> #7  rte_pktmbuf_free_seg (m=<optimized out>) at
>>>> /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:1349
>>>> #8  rte_pktmbuf_free (m=<optimized out>) at
>>>> /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:1369
>>>> #9  dpdk_daq_acquire (handle=0x1af6910, cnt=0, callback=<optimized
>>>> out>, metaback=<optimized out>, user=<optimized out>) at
>>>> daq_dpdk.c:234
>>>> #10 0x00000000004551d3 in DAQ_Acquire ()
>>>> #11 0x0000000000437828 in SnortMain ()
>>>> #12 0x00007ffff61af830 in __libc_start_main () from
>>>> /lib/x86_64-linux-gnu/libc.so.6
>>>> #13 0x0000000000406d29 in _start ()
>>>>
>>>> I notice that snort has three threads,
>>>> (gdb) info thread
>>>>  Id   Target Id         Frame
>>>> * 1    Thread 0x7ffff7fdf8c0 (LWP 42472) "snort" 0x0000000000000000 in ?? ()
>>>>  2    Thread 0x7fffdff5a700 (LWP 42481) "eal-intr-thread" 0x00007ffff62969d3 in
>>>>  epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
>>>>  3    Thread 0x7fffdf4d5700 (LWP 42482) "snort" 0x00007ffff625b30d in nanosleep
>>>>  () from /lib/x86_64-linux-gnu/libc.so.6
>>>>
>>>> Would it be a problem if the rte_pktmbuf_free() is not called from
>>>> the eal thread?
>>>>
>>>
>>> The rte_mbuf_free internally calls rte_mempool put calls - which
>>> should only be called from a EAL thread - so yes, only call
>>> rte_pktmbuf_free from an EAL thread.
>>>
>>>> Thanks
>>>> Ming
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> ---------------------------------------------------------------------
>>> -------------
>>> NOTICE
>>>
>>> Please Consider the Environment before printing this Email.
>>>
>>> This email was sent from within Inventum Technologies Private Limited
>>> (https://www.inventum.net). This email (and any attachments or
>>> hyperlinks within it) may contain information that is confidential,
>>> legally privileged or otherwise protected from disclosure. If you are
>>> not the intended recipient of this email, you are not entitled to
>>> use, disclose, distribute, copy, print, disseminate or rely on this
>>> email in any way. If you have received this email in error, please
>>> notify the sender immediately by telephone or email and destroy it, and all
>>> copies of it.
>>>
>>> We have taken steps to ensure that this email (and any attachments)
>>> are free from computer viruses and the like. However, it is the
>>> recipient's responsibility to ensure that it is actually virus free.
>>> Any emails that you send to us may be monitored for the purposes of
>>> ascertaining whether the communication complies with the law and our policies.
>>>
>> 
>> 
>> 
>> --
> > Andriy Berestovskyy

Re: [dpdk-users] dpdk with snort 2.9.9 receive from DPDK ring

[Ming Fu]

Hi Andriy,

Thanks for the suggestion.

I turned on the RTE_LIBRTE_MBUF_DEBUG and RTE_LIBRTE_MEMPOOL_DEBUG. There is no panic from the debug code and I can gdb trace the successive passes of rte_mbuf_sanity_check().  A similar skeleton code without link to snort works fine. I even tried to skip  the snort daq call back that passes the mbuf data to snort, so that I can rule out the possibility that snort corrupted the mbuf. It still has the same NULL function pointer.

Ming

-----Original Message-----
From: Andriy Berestovskyy [mailto:aber@semihalf.com] 
Sent: August-30-17 5:19 AM
To: Padam Jeet Singh <padam.singh@inventum.net>
Cc: Ming Fu <Ming.Fu@esentire.com>; users@dpdk.org
Subject: Re: [dpdk-users] dpdk with snort 2.9.9 receive from DPDK ring

Hey,
1. There is a non-EAL thread support since DPDK 2.0, so any thread can call rte_pktmbuf_free()

2. From the traceback it looks like the memory pool enqueue() op is NULL for a reason. Note that to free an mbuf, the mbuf header should be valid and point to a valid memory pool mbuf belongs.

Try to compile DPDK in debug mode and you might see the reason.

Andriy

On Wed, Aug 30, 2017 at 7:15 AM, Padam Jeet Singh <padam.singh@inventum.net> wrote:
>
>> On 30-Aug-2017, at 2:21 AM, Ming Fu <Ming.Fu@esentire.com> wrote:
>>
>> I am making a snort DAQ module to receive packet from DPDK capture 
>> through a ring. The snort version is 2.9.9 and DPDK version is 17.08. The code is very similar to the multi_process/client_server_mp example. The DPDK is initialized in daq initialize function and mbuf received from daq acquire function. Snort inspects the mbuf but does not re-inject back to the network, so daq free the mbuf by rte_pktmbuf_free(mbuf) in the same thread as it calls rte_ring_dequeue_burst(); The snort successfully received the first burst of 32 packets, but it fails on the first rte_pktmbuf_free().
>>
>> Seems some internal dpdk function pointer is NULL.
>> #0  0x0000000000000000 in ?? ()
>> #1  0x00000000004f7a8d in rte_mempool_ops_enqueue_bulk (n=<optimized out>, obj_table=<optimized out>, mp=<optimized out>)
>>    at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:497
>> #2  __mempool_generic_put (cache=<optimized out>, n=1, 
>> obj_table=0x7fffffffe6e8, mp=<optimized out>) at 
>> /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1069
>> #3  rte_mempool_generic_put (flags=<optimized out>, cache=<optimized out>, n=1, obj_table=0x7fffffffe6e8, mp=<optimized out>)
>>    at /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1109
>> #4  rte_mempool_put_bulk (n=1, obj_table=0x7fffffffe6e8, 
>> mp=<optimized out>) at 
>> /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1132
>> #5  rte_mempool_put (obj=<optimized out>, mp=<optimized out>) at 
>> /home/mfu/work/dpdk-17.08/build/include/rte_mempool.h:1150
>> #6  rte_mbuf_raw_free (m=<optimized out>) at 
>> /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:853
>> #7  rte_pktmbuf_free_seg (m=<optimized out>) at 
>> /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:1349
>> #8  rte_pktmbuf_free (m=<optimized out>) at 
>> /home/mfu/work/dpdk-17.08/build/include/rte_mbuf.h:1369
>> #9  dpdk_daq_acquire (handle=0x1af6910, cnt=0, callback=<optimized 
>> out>, metaback=<optimized out>, user=<optimized out>) at 
>> daq_dpdk.c:234
>> #10 0x00000000004551d3 in DAQ_Acquire ()
>> #11 0x0000000000437828 in SnortMain ()
>> #12 0x00007ffff61af830 in __libc_start_main () from 
>> /lib/x86_64-linux-gnu/libc.so.6
>> #13 0x0000000000406d29 in _start ()
>>
>> I notice that snort has three threads,
>> (gdb) info thread
>>  Id   Target Id         Frame
>> * 1    Thread 0x7ffff7fdf8c0 (LWP 42472) "snort" 0x0000000000000000 in ?? ()
>>  2    Thread 0x7fffdff5a700 (LWP 42481) "eal-intr-thread" 0x00007ffff62969d3 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
>>  3    Thread 0x7fffdf4d5700 (LWP 42482) "snort" 0x00007ffff625b30d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
>>
>> Would it be a problem if the rte_pktmbuf_free() is not called from the eal thread?
>>
>
> The rte_mbuf_free internally calls rte_mempool put calls - which should only be called from a EAL thread - so yes, only call rte_pktmbuf_free from an EAL thread.
>
>> Thanks
>> Ming
>>
>>
>>
>>
>>
>
>
>
> ----------------------------------------------------------------------
> ----------------------------------------------------------------------
> -----------
> NOTICE
>
> Please Consider the Environment before printing this Email.
>
> This email was sent from within Inventum Technologies Private Limited (https://www.inventum.net). This email (and any attachments or hyperlinks within it) may contain information that is confidential, legally privileged or otherwise protected from disclosure. If you are not the intended recipient of this email, you are not entitled to use, disclose, distribute, copy, print, disseminate or rely on this email in any way. If you have received this email in error, please notify the sender immediately by telephone or email and destroy it, and all copies of it.
>
> We have taken steps to ensure that this email (and any attachments) are free from computer viruses and the like. However, it is the recipient's responsibility to ensure that it is actually virus free. Any emails that you send to us may be monitored for the purposes of ascertaining whether the communication complies with the law and our policies.
>



--
Andriy Berestovskyy