[dpdk-users] dpdk concurrency

[dpdk-users] dpdk concurrency

[andrei]

Hi,


I had run into a deadlock by using DPDK, and I am out of ideas on how to
debug the issue.


Scenario: (The application is more complicated this is the simplified
version)

4 mempools, 4 txBuffers, 3 threads, (CPU 4 cores; irrelevant).

One thread extracts buffers(rte_mbuff) from a random memory(rte_mempool)
pool and place them into a ring buffer.

Other thread (Sender) extracts the buffers from the ring populates them
with data and place them into a rte_eth_dev_tx_buffer by calling
rte_eth_tx_buffer();

One thread (Flusher) is used as a flusher. It goes trough the
rte_eth_dev_tx_buffer's and calls rte_eth_tx_buffer_flush.


The deadlock occurs, in my opinion, when the Sender and the Flusher
threads try to place back buffers into the same memory pool.

This is a fragment of the core dump.


Thread 2 (Thread 0x7f5932e69700 (LWP 14014)):
#0  0x00007f59388e933a in common_ring_mp_enqueue () from
/usr/local/lib/librte_mempool.so.2.1
#1  0x00007f59386b27e0 in ixgbe_xmit_pkts () from
/usr/local/lib/librte_pmd_ixgbe.so.1.1
#2  0x00007f593d00aab7 in rte_eth_tx_burst (nb_pkts=<optimized out>,
tx_pkts=<optimized out>,
    queue_id=0, port_id=<optimized out>) at
/usr/local/include/dpdk/rte_ethdev.h:2858
#3  rte_eth_tx_buffer_flush (buffer=<optimized out>, queue_id=0,
port_id=<optimized out>)
    at /usr/local/include/dpdk/rte_ethdev.h:3040
#4  rte_eth_tx_buffer (tx_pkt=<optimized out>, buffer=<optimized out>,
queue_id=0,
    port_id=<optimized out>) at /usr/local/include/dpdk/rte_ethdev.h:3090


Thread 30 (Thread 0x7f5933175700 (LWP 13958)):
#0  0x00007f59388e91cc in common_ring_mp_enqueue () from
/usr/local/lib/librte_mempool.so.2.1
#1  0x00007f59386b27e0 in ixgbe_xmit_pkts () from
/usr/local/lib/librte_pmd_ixgbe.so.1.1
#2  0x00007f593d007dfd in rte_eth_tx_burst (nb_pkts=<optimized out>,
tx_pkts=0x7f587a410358,
    queue_id=<optimized out>, port_id=0 '\000') at
/usr/local/include/dpdk/rte_ethdev.h:2858
#3  rte_eth_tx_buffer_flush (buffer=0x7f587a410340, queue_id=<optimized
out>, port_id=0 '\000')
    at /usr/local/include/dpdk/rte_ethdev.h:3040


Questions:

       1. I am using DPDK 17.02.01. Was a  bug solved in newer releases
that can be mapped to this behavior?

       2. If two threads try to place buffers into the same pool, the
operation should be synchronized by DPDK or by application?


Thank you,

Andrei Comandatu

Re: [dpdk-users] dpdk concurrency

[Wiles, Keith]

> On Feb 26, 2018, at 8:36 AM, andrei <andrei@null.ro> wrote:
> 
> Hi,
> 
> 
> I had run into a deadlock by using DPDK, and I am out of ideas on how to
> debug the issue.
> 
> 
> Scenario: (The application is more complicated this is the simplified
> version)
> 
> 4 mempools, 4 txBuffers, 3 threads, (CPU 4 cores; irrelevant).
> 
> One thread extracts buffers(rte_mbuff) from a random memory(rte_mempool)
> pool and place them into a ring buffer.
> 
> Other thread (Sender) extracts the buffers from the ring populates them
> with data and place them into a rte_eth_dev_tx_buffer by calling
> rte_eth_tx_buffer();
> 
> One thread (Flusher) is used as a flusher. It goes trough the
> rte_eth_dev_tx_buffer's and calls rte_eth_tx_buffer_flush.
> 
> 
> The deadlock occurs, in my opinion, when the Sender and the Flusher
> threads try to place back buffers into the same memory pool.
> 
> This is a fragment of the core dump.
> 
> 
> Thread 2 (Thread 0x7f5932e69700 (LWP 14014)):
> #0  0x00007f59388e933a in common_ring_mp_enqueue () from
> /usr/local/lib/librte_mempool.so.2.1
> #1  0x00007f59386b27e0 in ixgbe_xmit_pkts () from
> /usr/local/lib/librte_pmd_ixgbe.so.1.1
> #2  0x00007f593d00aab7 in rte_eth_tx_burst (nb_pkts=<optimized out>,
> tx_pkts=<optimized out>,
>   queue_id=0, port_id=<optimized out>) at
> /usr/local/include/dpdk/rte_ethdev.h:2858
> #3  rte_eth_tx_buffer_flush (buffer=<optimized out>, queue_id=0,
> port_id=<optimized out>)
>   at /usr/local/include/dpdk/rte_ethdev.h:3040
> #4  rte_eth_tx_buffer (tx_pkt=<optimized out>, buffer=<optimized out>,
> queue_id=0,
>   port_id=<optimized out>) at /usr/local/include/dpdk/rte_ethdev.h:3090
> 
> 
> Thread 30 (Thread 0x7f5933175700 (LWP 13958)):
> #0  0x00007f59388e91cc in common_ring_mp_enqueue () from
> /usr/local/lib/librte_mempool.so.2.1
> #1  0x00007f59386b27e0 in ixgbe_xmit_pkts () from
> /usr/local/lib/librte_pmd_ixgbe.so.1.1
> #2  0x00007f593d007dfd in rte_eth_tx_burst (nb_pkts=<optimized out>,
> tx_pkts=0x7f587a410358,
>   queue_id=<optimized out>, port_id=0 '\000') at
> /usr/local/include/dpdk/rte_ethdev.h:2858
> #3  rte_eth_tx_buffer_flush (buffer=0x7f587a410340, queue_id=<optimized
> out>, port_id=0 '\000')
>   at /usr/local/include/dpdk/rte_ethdev.h:3040
> 
> 
> Questions:
> 
>      1. I am using DPDK 17.02.01. Was a  bug solved in newer releases
> that can be mapped to this behavior?
> 
>      2. If two threads try to place buffers into the same pool, the
> operation should be synchronized by DPDK or by application?

Not sure this will help, but if you are running more then one thread per core then you can have problems. I assume you have a mempool cache here and the per lcore caches are not thread safe (If I remember correctly), the main cache in the mempool is thread safe. If you are not using multiple threads per lcore, then I expect something else is going on here as this is the normal mode of operation for mempools. If you turn off the mempool caches you should not see the problem, but then your performance will drop some.

I had a similar problem and made sure only one thread puts or gets buffers from the same mempool.

> 
> 
> Thank you,
> 
> Andrei Comandatu
> 
> 

Regards,
Keith

Re: [dpdk-users] dpdk concurrency

[andrei]

Hi Keith,


Thank you very much for your advice. I will try to redesign the access
to the mempool's.

Regards,
Andrei Comandatu

On 02/26/2018 05:08 PM, Wiles, Keith wrote:
>
>> On Feb 26, 2018, at 8:36 AM, andrei <andrei@null.ro> wrote:
>>
>> Hi,
>>
>>
>> I had run into a deadlock by using DPDK, and I am out of ideas on how to
>> debug the issue.
>>
>>
>> Scenario: (The application is more complicated this is the simplified
>> version)
>>
>> 4 mempools, 4 txBuffers, 3 threads, (CPU 4 cores; irrelevant).
>>
>> One thread extracts buffers(rte_mbuff) from a random memory(rte_mempool)
>> pool and place them into a ring buffer.
>>
>> Other thread (Sender) extracts the buffers from the ring populates them
>> with data and place them into a rte_eth_dev_tx_buffer by calling
>> rte_eth_tx_buffer();
>>
>> One thread (Flusher) is used as a flusher. It goes trough the
>> rte_eth_dev_tx_buffer's and calls rte_eth_tx_buffer_flush.
>>
>>
>> The deadlock occurs, in my opinion, when the Sender and the Flusher
>> threads try to place back buffers into the same memory pool.
>>
>> This is a fragment of the core dump.
>>
>>
>> Thread 2 (Thread 0x7f5932e69700 (LWP 14014)):
>> #0  0x00007f59388e933a in common_ring_mp_enqueue () from
>> /usr/local/lib/librte_mempool.so.2.1
>> #1  0x00007f59386b27e0 in ixgbe_xmit_pkts () from
>> /usr/local/lib/librte_pmd_ixgbe.so.1.1
>> #2  0x00007f593d00aab7 in rte_eth_tx_burst (nb_pkts=<optimized out>,
>> tx_pkts=<optimized out>,
>>   queue_id=0, port_id=<optimized out>) at
>> /usr/local/include/dpdk/rte_ethdev.h:2858
>> #3  rte_eth_tx_buffer_flush (buffer=<optimized out>, queue_id=0,
>> port_id=<optimized out>)
>>   at /usr/local/include/dpdk/rte_ethdev.h:3040
>> #4  rte_eth_tx_buffer (tx_pkt=<optimized out>, buffer=<optimized out>,
>> queue_id=0,
>>   port_id=<optimized out>) at /usr/local/include/dpdk/rte_ethdev.h:3090
>>
>>
>> Thread 30 (Thread 0x7f5933175700 (LWP 13958)):
>> #0  0x00007f59388e91cc in common_ring_mp_enqueue () from
>> /usr/local/lib/librte_mempool.so.2.1
>> #1  0x00007f59386b27e0 in ixgbe_xmit_pkts () from
>> /usr/local/lib/librte_pmd_ixgbe.so.1.1
>> #2  0x00007f593d007dfd in rte_eth_tx_burst (nb_pkts=<optimized out>,
>> tx_pkts=0x7f587a410358,
>>   queue_id=<optimized out>, port_id=0 '\000') at
>> /usr/local/include/dpdk/rte_ethdev.h:2858
>> #3  rte_eth_tx_buffer_flush (buffer=0x7f587a410340, queue_id=<optimized
>> out>, port_id=0 '\000')
>>   at /usr/local/include/dpdk/rte_ethdev.h:3040
>>
>>
>> Questions:
>>
>>      1. I am using DPDK 17.02.01. Was a  bug solved in newer releases
>> that can be mapped to this behavior?
>>
>>      2. If two threads try to place buffers into the same pool, the
>> operation should be synchronized by DPDK or by application?
> Not sure this will help, but if you are running more then one thread per core then you can have problems. I assume you have a mempool cache here and the per lcore caches are not thread safe (If I remember correctly), the main cache in the mempool is thread safe. If you are not using multiple threads per lcore, then I expect something else is going on here as this is the normal mode of operation for mempools. If you turn off the mempool caches you should not see the problem, but then your performance will drop some.
>
> I had a similar problem and made sure only one thread puts or gets buffers from the same mempool.
>
>>
>> Thank you,
>>
>> Andrei Comandatu
>>
>>
> Regards,
> Keith
>