Re: [dts] Packets reorder in the second pass on mempool

[Masoud Moshref Javadi <masood.moshref.j@gmail.com>]

The NIC is E10G42BT X520-T2 and the rx queue has 256 and tx queue has 
512 entries.

I found out the problem was that the memory pool become empty. I thought 
based on the description of rte_eth_tx_burst, after the sending, the 
rte_mbufs should be freed and returned into the mempool.

Now I use rte_mempool_put_bulk() just after draining the batch of 
packets to get back mbufs into the pool and it works.

Thanks

On 7/27/2015 1:06 AM, Liu, Yong wrote:
> Hi Javadi,
> I think it's fine to send out issue to dev mailing list.
> Maybe there's no reply from people in community is just because of can't reproduce your issue.
> We need the NIC type and related rx/tx queue settings for reproduce your problem.
>
>> -----Original Message-----
>> From: dts [mailto:dts-bounces@dpdk.org] On Behalf Of Masoud Moshref Javadi
>> Sent: Saturday, July 25, 2015 6:04 AM
>> To: dts@dpdk.org
>> Subject: [dts] Packets reorder in the second pass on mempool
>>
>> Hi
>>
>> I'm new to dpdk and I hope you can solve my weird problem (I think I
>> sent this message to dev list by mistake. sorry)
>>
>> I see packet reordering correlated with the batchsize and mempool size.
>>
>> I have a very simple setting of a sender and a receiver connected with a
>> simple 10G switch.
>> The sender sends udp packets at 14.88MPPS and just puts a 32bit sequence
>> number in the udp packets.
>> The receiver reads packets and expects the packets to be in order.
>> However, the receiver sees packets in a reordered way like this (note
>> that it starts from 7808 in different runs):
>>
>> expected, seen , seen-expected
>> 7808, 7936 = 128
>> 7978, 8106 = 128
>> 8145, 8273 = 128
>> 8308, 8436 = 128
>> 8448, 8320 = -128
>> 8391, 8519 = 128
>> 8576, 8448 = -128
>> 8518, 8646 = 128
>> 8704, 8576 = -128
>> ....
>>
>> The batchsize at sender is 128.
>> The configuration of ports and mempools are similar to basicfwd.c in
>> basic forwarding example of dpdk.
>> Interestingly if I change the size of mempool, the beginning of packet
>> reordering changes from 7808.
>>
>> #define NUM_MBUFS 8192
>> #define MBUF_SIZE (1600 + sizeof(struct rte_mbuf) + RTE_PKTMBUF_HEADROOM)
>> #define MBUF_CACHE_SIZE 250
>> #define BURST_SIZE 128
>>
>> mbuf_pool = rte_mempool_create("MBUF_POOL",
>>                                           NUM_MBUFS * nb_ports,
>>                                           MBUF_SIZE,
>>                                           MBUF_CACHE_SIZE,
>>                                           sizeof(struct
>> rte_pktmbuf_pool_private),
>>                                           rte_pktmbuf_pool_init, NULL,
>>                                           rte_pktmbuf_init,      NULL,
>>                                           1,
>>                                           0);
>>
>> The main sender loop is as follows:
>>
>>     for (;pkts_num<target;) {
>>                            /* Get burst of RX packets, from first port of
>> pair. */
>>                            rte_mempool_sc_get_bulk(mbuf_pool,
>> (void**)&bufs, BURST_SIZE);
>>                            /* Send burst of TX packets, to second port of
>> pair. */
>>                            for (i=0; i< BURST_SIZE; i++){
>>                                    m = bufs[i];
>>                                    m->data_len = 60;
>>                                    m->pkt_len  = 60;
>>                                    rte_memcpy((uint8_t *)m->buf_addr +
>> m->data_off,bufpkt, 60);
>>                                    uint32_t * data = (uint32_t
>> *)(rte_ctrlmbuf_data(m) + data_offset);
>>                                    *data = pkts_num+i;
>>                            }
>>
>>                            uint16_t nb_tx = 0;
>>                            while (nb_tx < BURST_SIZE){
>>                                    nb_tx+= rte_eth_tx_burst(port, 0, bufs
>> + nb_tx, BURST_SIZE - nb_tx);
>>                            }
>>                            pkts_num += nb_tx;
>> }
>>
>> Regards