From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 288332C09 for ; Tue, 25 Oct 2016 03:05:31 +0200 (CEST) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga102.fm.intel.com with ESMTP; 24 Oct 2016 18:05:29 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.31,544,1473145200"; d="scan'208,217";a="1058570061" Received: from fmsmsx107.amr.corp.intel.com ([10.18.124.205]) by fmsmga001.fm.intel.com with ESMTP; 24 Oct 2016 18:05:29 -0700 Received: from fmsmsx125.amr.corp.intel.com (10.18.125.40) by fmsmsx107.amr.corp.intel.com (10.18.124.205) with Microsoft SMTP Server (TLS) id 14.3.248.2; Mon, 24 Oct 2016 18:05:29 -0700 Received: from shsmsx101.ccr.corp.intel.com (10.239.4.153) by FMSMSX125.amr.corp.intel.com (10.18.125.40) with Microsoft SMTP Server (TLS) id 14.3.248.2; Mon, 24 Oct 2016 18:05:29 -0700 Received: from shsmsx103.ccr.corp.intel.com ([169.254.4.139]) by SHSMSX101.ccr.corp.intel.com ([169.254.1.104]) with mapi id 14.03.0248.002; Tue, 25 Oct 2016 09:05:26 +0800 From: "Zhang, Helin" To: yingzhi CC: "dev@dpdk.org" Thread-Topic: RE: [dpdk-dev] rte_kni_tx_burst() hangs because of no freedescriptors Thread-Index: AQHSLeLbDJ4emPtN7UK2ndyA/Yz7Z6C4W3cg Date: Tue, 25 Oct 2016 01:05:26 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] rte_kni_tx_burst() hangs because of no freedescriptors X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Oct 2016 01:05:32 -0000 From: yingzhi [mailto:kaitoy@qq.com] Sent: Monday, October 24, 2016 6:39 PM To: Zhang, Helin Cc: dev@dpdk.org Subject: Re: RE: [dpdk-dev] rte_kni_tx_burst() hangs because of no freedesc= riptors Hi Helin, Thanks for your response, to answer your questions: 1. we send only one packet each time calling rte_kni_tx_burst(), which mean= s the last argument is 1. 2. it returns 0 because the free mbuf function inside tx_burst will not fre= e any mbuf: if (txq->nb_tx_free < txq->tx_free_thresh) ixgbe_tx_free_bufs(txq); after this operation, the txq->nb_tx_free is not increased and keeps to be = "0" eventually. I did some tests today, I commented out this section of ixgbe_rxtx_vec_comm= on.h -> ixgbe_tx_free_bufs status =3D txq->tx_ring[txq->tx_next_dd].wb.status; if (!(status & IXGBE_ADVTXD_STAT_DD)) return 0; After ignoring DD bit check, our app runs for about 6 hours without issue. = I suspect there is something wrong in my program set the DD bit somewhere. = One of the possible cause currently I suspect is as far as I know, rte_pktm= buf_free(mbuf.array[k]) will free the mbuf of the packet and any fragmented= packets following by it. But in our application such as below code snippet= : auto nb_tx =3D rte_eth_tx_burst(port, queue, mbuf.array, (uint16_t)= nb_rx); if (unlikely(nb_tx < nb_rx)) { for (unsigned k =3D nb_tx; k < nb_rx; k++) { rte_pktmbuf_free(mbuf.array[k]); } } [Zhang, Helin] it seems above code piece has memory leak, if the buffer is = chained. After all memory leaked, then the issue comes. Please try to check= if this is the root cause! In this case if there are fragmented packets and failed transmission, may c= ause a mbuf be freed multiple times. Above is just my suspect, need to do some tests later today or tomorrow. Thanks ------------------ Original ------------------ From: "Zhang, Helin";; Date: Mon, Oct 24, 2016 11:33 AM To: "yingzhi"; Cc: "dev@dpdk.org"; Subject: RE: [dpdk-dev] rte_kni_tx_burst() hangs because of no freedescrip= tors Hi Yingzhi Thank you for the reporting! The description is not so clear at least for m= e. Please help to narrown down the issue by youself. How many packets would it have for calling TX function? Why it would return 0 after calling TX function? No memory? Or return from = else? Have you found anything? Regards, Helin > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of yingzhi > Sent: Sunday, October 23, 2016 9:30 PM > To: users; dev@dpdk.org > Subject: [dpdk-dev] rte_kni_tx_burst() hangs because of no free descripto= rs > > - > Hi Experts, > > Background: > > We are using DPDK to develop a LoadBalancer following below logic: When > a new packet is received: > 1. if the dst_addr is management IP, forward to KNI. 2. if the dst_addr = is in > VIP list, select backend and forward(modify dst mac address). 3. otherwis= e > drop the packet. > > At this stage, we use one single thread for KNI forwarding and another fo= r > VIP forwarding(forward to eth). > > DPDK version: 16.07 > NIC: 82599ES 10-Gigabit SFI/SFP+ Network Connection > Linux: 14.04.1-Ubuntu x64 > > Promblem description: > > The program runs correctly for sometime(around 2 hours for 400Mb traffic)= . > But it it will hang. When problem happens, rte_eth_tx_burst() will not ab= le to > send out any packets(always returns 0). We tracked into that function and > noticed it is actually calling ixgbe driver's ixgbe_xmit_pkts_vec() funct= ion in > our environment, because we use default tx queue configuration, after > printing some info, we found if the free function works fine: > tx_rs_thresh: 32, tx_free_thresh: 32, nb_tx_free: 31 > > it will trigger free and make 32 more free descriptors: > tx_rs_thresh: 32, tx_free_thresh: 32, nb_tx_free: 62 > > but when something going wrong, it will no longer free anything: > tx_rs_thresh: 32, tx_free_thresh: 32, nb_tx_free: 0 tx_rs_thresh: 32, > tx_free_thresh: 32, nb_tx_free: 0 > > It may related with the DD flag of the descriptor but we are not quite su= re. > > Our program logic: > > create two mbuf pools on socket 0, one for rx_queue and one for kni. (all > lcore threads runs on socket0) > > init kni interface with rte_kni_alloc() > > > init one NIC interface with > rte_eth_dev_configure(); rte_eth_rx_queue_setup(); > rte_eth_tx_queue_setup(); rte_eth_dev_start(); > > > > in the eth main loop: (code is simplified) > while(1) { n =3D rte_eth_rx_burst(packets); for (i =3D 0; i < n;= ++i) { if > (SEND_TO_KNI) { m =3D rte_kni_tx_burst(packets[i]); = if (m !=3D 1)) > { rte_pktmbuf_free(packets[i]); } } = if (SEND_TO_ETH) > { // after modify the packet m =3D rte_eth_tx_bur= st(packets[i]); > if (m !=3D 1)) { rte_pktmbuf_free(packets[i]); = } } // > otherwise drop the packet rte_pktmbuf_free(packets[i]); } } > > > Please advise if I'm using DPDK in a wrong way. Sorry if I missed somethi= ng > basic, I'm new to DPDK. > > Thanks in advance > Best regards