From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <masood.moshref.j@gmail.com>
Received: from mail-pd0-f174.google.com (mail-pd0-f174.google.com
 [209.85.192.174]) by dpdk.org (Postfix) with ESMTP id 2F2E0C494
 for <dts@dpdk.org>; Sat, 25 Jul 2015 00:02:17 +0200 (CEST)
Received: by pdbbh15 with SMTP id bh15so19337091pdb.1
 for <dts@dpdk.org>; Fri, 24 Jul 2015 15:02:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=message-id:date:from:user-agent:mime-version:to:subject
 :content-type:content-transfer-encoding;
 bh=3dmfRw/X6GxYiezz2Z/flnVF5EYBdRCCwnq2jqBo3XM=;
 b=k4em9dBYuVECCRqLNoGKLIifIiturateKDg/lgtHfzLvx37kzM2BR4cw50hi/osHoC
 RYxF0WAAd7MzUz5OhMP5E94g4V4W6F5oznMdPtdUzeNxeyV1VyG2lFvOCusiB2hy77qF
 yfRAnG/T9T7evc7nSunlqbtxvoAo394Tyv+2NpMGskaAdzo4pmHlvOC/Mj3jcUzdfd0U
 Nv83vecaIfFSNK4Cbjtixa9t2ZyGr9MKbZkSNTUWSpISM0SSCfyjboyADFRKCYWw95oz
 9h3iKPfCKg69mwS2Wr29KvKxFuda8d9mLD9528XklGZp1dXvq+OoCjrSnb9DE8Wi1WK6
 RHug==
X-Received: by 10.70.48.34 with SMTP id i2mr35319347pdn.125.1437775336529;
 Fri, 24 Jul 2015 15:02:16 -0700 (PDT)
Received: from [204.57.7.4] ([204.57.7.4])
 by smtp.googlemail.com with ESMTPSA id o7sm16329585pdi.16.2015.07.24.15.02.15
 for <dts@dpdk.org>
 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
 Fri, 24 Jul 2015 15:02:15 -0700 (PDT)
Message-ID: <55B2B641.9060504@gmail.com>
Date: Fri, 24 Jul 2015 15:03:45 -0700
From: Masoud Moshref Javadi <masood.moshref.j@gmail.com>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64;
 rv:31.0) Gecko/20100101 Thunderbird/31.7.0
MIME-Version: 1.0
To: dts@dpdk.org
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Subject: [dts] Packets reorder in the second pass on mempool
X-BeenThere: dts@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: test suite reviews and discussions <dts.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dts>,
 <mailto:dts-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dts/>
List-Post: <mailto:dts@dpdk.org>
List-Help: <mailto:dts-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dts>,
 <mailto:dts-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Fri, 24 Jul 2015 22:02:17 -0000

Hi

I'm new to dpdk and I hope you can solve my weird problem (I think I 
sent this message to dev list by mistake. sorry)

I see packet reordering correlated with the batchsize and mempool size.

I have a very simple setting of a sender and a receiver connected with a
simple 10G switch.
The sender sends udp packets at 14.88MPPS and just puts a 32bit sequence
number in the udp packets.
The receiver reads packets and expects the packets to be in order.
However, the receiver sees packets in a reordered way like this (note
that it starts from 7808 in different runs):

expected, seen , seen-expected
7808, 7936 = 128
7978, 8106 = 128
8145, 8273 = 128
8308, 8436 = 128
8448, 8320 = -128
8391, 8519 = 128
8576, 8448 = -128
8518, 8646 = 128
8704, 8576 = -128
....

The batchsize at sender is 128.
The configuration of ports and mempools are similar to basicfwd.c in
basic forwarding example of dpdk.
Interestingly if I change the size of mempool, the beginning of packet
reordering changes from 7808.

#define NUM_MBUFS 8192
#define MBUF_SIZE (1600 + sizeof(struct rte_mbuf) + RTE_PKTMBUF_HEADROOM)
#define MBUF_CACHE_SIZE 250
#define BURST_SIZE 128

mbuf_pool = rte_mempool_create("MBUF_POOL",
                                         NUM_MBUFS * nb_ports,
                                         MBUF_SIZE,
                                         MBUF_CACHE_SIZE,
                                         sizeof(struct
rte_pktmbuf_pool_private),
                                         rte_pktmbuf_pool_init, NULL,
                                         rte_pktmbuf_init,      NULL,
                                         1,
                                         0);

The main sender loop is as follows:

   for (;pkts_num<target;) {
                          /* Get burst of RX packets, from first port of
pair. */
                          rte_mempool_sc_get_bulk(mbuf_pool,
(void**)&bufs, BURST_SIZE);
                          /* Send burst of TX packets, to second port of
pair. */
                          for (i=0; i< BURST_SIZE; i++){
                                  m = bufs[i];
                                  m->data_len = 60;
                                  m->pkt_len  = 60;
                                  rte_memcpy((uint8_t *)m->buf_addr +
m->data_off,bufpkt, 60);
                                  uint32_t * data = (uint32_t
*)(rte_ctrlmbuf_data(m) + data_offset);
                                  *data = pkts_num+i;
                          }

                          uint16_t nb_tx = 0;
                          while (nb_tx < BURST_SIZE){
                                  nb_tx+= rte_eth_tx_burst(port, 0, bufs
+ nb_tx, BURST_SIZE - nb_tx);
                          }
                          pkts_num += nb_tx;
}

Regards