From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f52.google.com (mail-pa0-f52.google.com [209.85.220.52]) by dpdk.org (Postfix) with ESMTP id 1E39DC5AC for ; Mon, 27 Jul 2015 19:57:51 +0200 (CEST) Received: by pachj5 with SMTP id hj5so54959271pac.3 for ; Mon, 27 Jul 2015 10:57:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=yuiIB9DwauHHLyx7JkPvc7DnnhoV4hl7HbabHBEY9DU=; b=a/K8g61k8NRXx4ZKMvyMjUvoGewsCqRiU0c73ZJ1Qr8lP7nuHSdfi28hEX1dRFyu3z fvb5nRJGnid4xQNrbwYBdyC/5b85rAyXq6sz6pA8VAlYnKTVneX6QpYwkui+Ew9Va5mh D6ZvPN6jdr75TDP7jEW7b4RYkHlvh3+KHmb1jOlEIUbD6g0uiO2YZH/zGKJFjIs82Gc2 QiH0ok4CydRZxvHQTT4UmgUFs9alLUp/5x6jOMKQ6T2YokWijfZh3xZXif+S02Fepe/u C/TyCAUojr0t/7VZrQA2Q8I+qm8lDOfyt2X+WDGuAxE+3mG1fLRj3jHQz3/eI38z0GJD xqUw== X-Received: by 10.66.176.202 with SMTP id ck10mr60784477pac.15.1438019870349; Mon, 27 Jul 2015 10:57:50 -0700 (PDT) Received: from [204.57.7.4] ([204.57.7.4]) by smtp.googlemail.com with ESMTPSA id pe3sm30628601pdb.55.2015.07.27.10.57.48 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 27 Jul 2015 10:57:49 -0700 (PDT) Message-ID: <55B67178.1060805@gmail.com> Date: Mon, 27 Jul 2015 10:59:20 -0700 From: Masoud Moshref Javadi User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: "Liu, Yong" , "dts@dpdk.org" References: <55B2B641.9060504@gmail.com> <86228AFD5BCD8E4EBFD2B90117B5E81E10EAD7ED@SHSMSX103.ccr.corp.intel.com> In-Reply-To: <86228AFD5BCD8E4EBFD2B90117B5E81E10EAD7ED@SHSMSX103.ccr.corp.intel.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [dts] Packets reorder in the second pass on mempool X-BeenThere: dts@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: test suite reviews and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Jul 2015 17:57:51 -0000 The NIC is E10G42BT X520-T2 and the rx queue has 256 and tx queue has 512 entries. I found out the problem was that the memory pool become empty. I thought based on the description of rte_eth_tx_burst, after the sending, the rte_mbufs should be freed and returned into the mempool. Now I use rte_mempool_put_bulk() just after draining the batch of packets to get back mbufs into the pool and it works. Thanks On 7/27/2015 1:06 AM, Liu, Yong wrote: > Hi Javadi, > I think it's fine to send out issue to dev mailing list. > Maybe there's no reply from people in community is just because of can't reproduce your issue. > We need the NIC type and related rx/tx queue settings for reproduce your problem. > >> -----Original Message----- >> From: dts [mailto:dts-bounces@dpdk.org] On Behalf Of Masoud Moshref Javadi >> Sent: Saturday, July 25, 2015 6:04 AM >> To: dts@dpdk.org >> Subject: [dts] Packets reorder in the second pass on mempool >> >> Hi >> >> I'm new to dpdk and I hope you can solve my weird problem (I think I >> sent this message to dev list by mistake. sorry) >> >> I see packet reordering correlated with the batchsize and mempool size. >> >> I have a very simple setting of a sender and a receiver connected with a >> simple 10G switch. >> The sender sends udp packets at 14.88MPPS and just puts a 32bit sequence >> number in the udp packets. >> The receiver reads packets and expects the packets to be in order. >> However, the receiver sees packets in a reordered way like this (note >> that it starts from 7808 in different runs): >> >> expected, seen , seen-expected >> 7808, 7936 = 128 >> 7978, 8106 = 128 >> 8145, 8273 = 128 >> 8308, 8436 = 128 >> 8448, 8320 = -128 >> 8391, 8519 = 128 >> 8576, 8448 = -128 >> 8518, 8646 = 128 >> 8704, 8576 = -128 >> .... >> >> The batchsize at sender is 128. >> The configuration of ports and mempools are similar to basicfwd.c in >> basic forwarding example of dpdk. >> Interestingly if I change the size of mempool, the beginning of packet >> reordering changes from 7808. >> >> #define NUM_MBUFS 8192 >> #define MBUF_SIZE (1600 + sizeof(struct rte_mbuf) + RTE_PKTMBUF_HEADROOM) >> #define MBUF_CACHE_SIZE 250 >> #define BURST_SIZE 128 >> >> mbuf_pool = rte_mempool_create("MBUF_POOL", >> NUM_MBUFS * nb_ports, >> MBUF_SIZE, >> MBUF_CACHE_SIZE, >> sizeof(struct >> rte_pktmbuf_pool_private), >> rte_pktmbuf_pool_init, NULL, >> rte_pktmbuf_init, NULL, >> 1, >> 0); >> >> The main sender loop is as follows: >> >> for (;pkts_num> /* Get burst of RX packets, from first port of >> pair. */ >> rte_mempool_sc_get_bulk(mbuf_pool, >> (void**)&bufs, BURST_SIZE); >> /* Send burst of TX packets, to second port of >> pair. */ >> for (i=0; i< BURST_SIZE; i++){ >> m = bufs[i]; >> m->data_len = 60; >> m->pkt_len = 60; >> rte_memcpy((uint8_t *)m->buf_addr + >> m->data_off,bufpkt, 60); >> uint32_t * data = (uint32_t >> *)(rte_ctrlmbuf_data(m) + data_offset); >> *data = pkts_num+i; >> } >> >> uint16_t nb_tx = 0; >> while (nb_tx < BURST_SIZE){ >> nb_tx+= rte_eth_tx_burst(port, 0, bufs >> + nb_tx, BURST_SIZE - nb_tx); >> } >> pkts_num += nb_tx; >> } >> >> Regards