* [dts] Packets reorder in the second pass on mempool
@ 2015-07-24 22:03 Masoud Moshref Javadi
2015-07-27 8:06 ` Liu, Yong
0 siblings, 1 reply; 3+ messages in thread
From: Masoud Moshref Javadi @ 2015-07-24 22:03 UTC (permalink / raw)
To: dts
Hi
I'm new to dpdk and I hope you can solve my weird problem (I think I
sent this message to dev list by mistake. sorry)
I see packet reordering correlated with the batchsize and mempool size.
I have a very simple setting of a sender and a receiver connected with a
simple 10G switch.
The sender sends udp packets at 14.88MPPS and just puts a 32bit sequence
number in the udp packets.
The receiver reads packets and expects the packets to be in order.
However, the receiver sees packets in a reordered way like this (note
that it starts from 7808 in different runs):
expected, seen , seen-expected
7808, 7936 = 128
7978, 8106 = 128
8145, 8273 = 128
8308, 8436 = 128
8448, 8320 = -128
8391, 8519 = 128
8576, 8448 = -128
8518, 8646 = 128
8704, 8576 = -128
....
The batchsize at sender is 128.
The configuration of ports and mempools are similar to basicfwd.c in
basic forwarding example of dpdk.
Interestingly if I change the size of mempool, the beginning of packet
reordering changes from 7808.
#define NUM_MBUFS 8192
#define MBUF_SIZE (1600 + sizeof(struct rte_mbuf) + RTE_PKTMBUF_HEADROOM)
#define MBUF_CACHE_SIZE 250
#define BURST_SIZE 128
mbuf_pool = rte_mempool_create("MBUF_POOL",
NUM_MBUFS * nb_ports,
MBUF_SIZE,
MBUF_CACHE_SIZE,
sizeof(struct
rte_pktmbuf_pool_private),
rte_pktmbuf_pool_init, NULL,
rte_pktmbuf_init, NULL,
1,
0);
The main sender loop is as follows:
for (;pkts_num<target;) {
/* Get burst of RX packets, from first port of
pair. */
rte_mempool_sc_get_bulk(mbuf_pool,
(void**)&bufs, BURST_SIZE);
/* Send burst of TX packets, to second port of
pair. */
for (i=0; i< BURST_SIZE; i++){
m = bufs[i];
m->data_len = 60;
m->pkt_len = 60;
rte_memcpy((uint8_t *)m->buf_addr +
m->data_off,bufpkt, 60);
uint32_t * data = (uint32_t
*)(rte_ctrlmbuf_data(m) + data_offset);
*data = pkts_num+i;
}
uint16_t nb_tx = 0;
while (nb_tx < BURST_SIZE){
nb_tx+= rte_eth_tx_burst(port, 0, bufs
+ nb_tx, BURST_SIZE - nb_tx);
}
pkts_num += nb_tx;
}
Regards
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: [dts] Packets reorder in the second pass on mempool
2015-07-24 22:03 [dts] Packets reorder in the second pass on mempool Masoud Moshref Javadi
@ 2015-07-27 8:06 ` Liu, Yong
2015-07-27 17:59 ` Masoud Moshref Javadi
0 siblings, 1 reply; 3+ messages in thread
From: Liu, Yong @ 2015-07-27 8:06 UTC (permalink / raw)
To: Masoud Moshref Javadi, dts
Hi Javadi,
I think it's fine to send out issue to dev mailing list.
Maybe there's no reply from people in community is just because of can't reproduce your issue.
We need the NIC type and related rx/tx queue settings for reproduce your problem.
> -----Original Message-----
> From: dts [mailto:dts-bounces@dpdk.org] On Behalf Of Masoud Moshref Javadi
> Sent: Saturday, July 25, 2015 6:04 AM
> To: dts@dpdk.org
> Subject: [dts] Packets reorder in the second pass on mempool
>
> Hi
>
> I'm new to dpdk and I hope you can solve my weird problem (I think I
> sent this message to dev list by mistake. sorry)
>
> I see packet reordering correlated with the batchsize and mempool size.
>
> I have a very simple setting of a sender and a receiver connected with a
> simple 10G switch.
> The sender sends udp packets at 14.88MPPS and just puts a 32bit sequence
> number in the udp packets.
> The receiver reads packets and expects the packets to be in order.
> However, the receiver sees packets in a reordered way like this (note
> that it starts from 7808 in different runs):
>
> expected, seen , seen-expected
> 7808, 7936 = 128
> 7978, 8106 = 128
> 8145, 8273 = 128
> 8308, 8436 = 128
> 8448, 8320 = -128
> 8391, 8519 = 128
> 8576, 8448 = -128
> 8518, 8646 = 128
> 8704, 8576 = -128
> ....
>
> The batchsize at sender is 128.
> The configuration of ports and mempools are similar to basicfwd.c in
> basic forwarding example of dpdk.
> Interestingly if I change the size of mempool, the beginning of packet
> reordering changes from 7808.
>
> #define NUM_MBUFS 8192
> #define MBUF_SIZE (1600 + sizeof(struct rte_mbuf) + RTE_PKTMBUF_HEADROOM)
> #define MBUF_CACHE_SIZE 250
> #define BURST_SIZE 128
>
> mbuf_pool = rte_mempool_create("MBUF_POOL",
> NUM_MBUFS * nb_ports,
> MBUF_SIZE,
> MBUF_CACHE_SIZE,
> sizeof(struct
> rte_pktmbuf_pool_private),
> rte_pktmbuf_pool_init, NULL,
> rte_pktmbuf_init, NULL,
> 1,
> 0);
>
> The main sender loop is as follows:
>
> for (;pkts_num<target;) {
> /* Get burst of RX packets, from first port of
> pair. */
> rte_mempool_sc_get_bulk(mbuf_pool,
> (void**)&bufs, BURST_SIZE);
> /* Send burst of TX packets, to second port of
> pair. */
> for (i=0; i< BURST_SIZE; i++){
> m = bufs[i];
> m->data_len = 60;
> m->pkt_len = 60;
> rte_memcpy((uint8_t *)m->buf_addr +
> m->data_off,bufpkt, 60);
> uint32_t * data = (uint32_t
> *)(rte_ctrlmbuf_data(m) + data_offset);
> *data = pkts_num+i;
> }
>
> uint16_t nb_tx = 0;
> while (nb_tx < BURST_SIZE){
> nb_tx+= rte_eth_tx_burst(port, 0, bufs
> + nb_tx, BURST_SIZE - nb_tx);
> }
> pkts_num += nb_tx;
> }
>
> Regards
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: [dts] Packets reorder in the second pass on mempool
2015-07-27 8:06 ` Liu, Yong
@ 2015-07-27 17:59 ` Masoud Moshref Javadi
0 siblings, 0 replies; 3+ messages in thread
From: Masoud Moshref Javadi @ 2015-07-27 17:59 UTC (permalink / raw)
To: Liu, Yong, dts
The NIC is E10G42BT X520-T2 and the rx queue has 256 and tx queue has
512 entries.
I found out the problem was that the memory pool become empty. I thought
based on the description of rte_eth_tx_burst, after the sending, the
rte_mbufs should be freed and returned into the mempool.
Now I use rte_mempool_put_bulk() just after draining the batch of
packets to get back mbufs into the pool and it works.
Thanks
On 7/27/2015 1:06 AM, Liu, Yong wrote:
> Hi Javadi,
> I think it's fine to send out issue to dev mailing list.
> Maybe there's no reply from people in community is just because of can't reproduce your issue.
> We need the NIC type and related rx/tx queue settings for reproduce your problem.
>
>> -----Original Message-----
>> From: dts [mailto:dts-bounces@dpdk.org] On Behalf Of Masoud Moshref Javadi
>> Sent: Saturday, July 25, 2015 6:04 AM
>> To: dts@dpdk.org
>> Subject: [dts] Packets reorder in the second pass on mempool
>>
>> Hi
>>
>> I'm new to dpdk and I hope you can solve my weird problem (I think I
>> sent this message to dev list by mistake. sorry)
>>
>> I see packet reordering correlated with the batchsize and mempool size.
>>
>> I have a very simple setting of a sender and a receiver connected with a
>> simple 10G switch.
>> The sender sends udp packets at 14.88MPPS and just puts a 32bit sequence
>> number in the udp packets.
>> The receiver reads packets and expects the packets to be in order.
>> However, the receiver sees packets in a reordered way like this (note
>> that it starts from 7808 in different runs):
>>
>> expected, seen , seen-expected
>> 7808, 7936 = 128
>> 7978, 8106 = 128
>> 8145, 8273 = 128
>> 8308, 8436 = 128
>> 8448, 8320 = -128
>> 8391, 8519 = 128
>> 8576, 8448 = -128
>> 8518, 8646 = 128
>> 8704, 8576 = -128
>> ....
>>
>> The batchsize at sender is 128.
>> The configuration of ports and mempools are similar to basicfwd.c in
>> basic forwarding example of dpdk.
>> Interestingly if I change the size of mempool, the beginning of packet
>> reordering changes from 7808.
>>
>> #define NUM_MBUFS 8192
>> #define MBUF_SIZE (1600 + sizeof(struct rte_mbuf) + RTE_PKTMBUF_HEADROOM)
>> #define MBUF_CACHE_SIZE 250
>> #define BURST_SIZE 128
>>
>> mbuf_pool = rte_mempool_create("MBUF_POOL",
>> NUM_MBUFS * nb_ports,
>> MBUF_SIZE,
>> MBUF_CACHE_SIZE,
>> sizeof(struct
>> rte_pktmbuf_pool_private),
>> rte_pktmbuf_pool_init, NULL,
>> rte_pktmbuf_init, NULL,
>> 1,
>> 0);
>>
>> The main sender loop is as follows:
>>
>> for (;pkts_num<target;) {
>> /* Get burst of RX packets, from first port of
>> pair. */
>> rte_mempool_sc_get_bulk(mbuf_pool,
>> (void**)&bufs, BURST_SIZE);
>> /* Send burst of TX packets, to second port of
>> pair. */
>> for (i=0; i< BURST_SIZE; i++){
>> m = bufs[i];
>> m->data_len = 60;
>> m->pkt_len = 60;
>> rte_memcpy((uint8_t *)m->buf_addr +
>> m->data_off,bufpkt, 60);
>> uint32_t * data = (uint32_t
>> *)(rte_ctrlmbuf_data(m) + data_offset);
>> *data = pkts_num+i;
>> }
>>
>> uint16_t nb_tx = 0;
>> while (nb_tx < BURST_SIZE){
>> nb_tx+= rte_eth_tx_burst(port, 0, bufs
>> + nb_tx, BURST_SIZE - nb_tx);
>> }
>> pkts_num += nb_tx;
>> }
>>
>> Regards
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2015-07-27 17:57 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-07-24 22:03 [dts] Packets reorder in the second pass on mempool Masoud Moshref Javadi
2015-07-27 8:06 ` Liu, Yong
2015-07-27 17:59 ` Masoud Moshref Javadi
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).