From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id EDE225B02 for ; Fri, 23 Jan 2015 12:52:58 +0100 (CET) Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga102.fm.intel.com with ESMTP; 23 Jan 2015 03:52:54 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.09,453,1418112000"; d="scan'208";a="674742708" Received: from bricha3-mobl3.ger.corp.intel.com ([10.243.20.38]) by orsmga002.jf.intel.com with SMTP; 23 Jan 2015 03:52:46 -0800 Received: by (sSMTP sendmail emulation); Fri, 23 Jan 2015 11:52:44 +0025 Date: Fri, 23 Jan 2015 11:52:44 +0000 From: Bruce Richardson To: Martin Weiser Message-ID: <20150123115244.GA10808@bricha3-MOBL3> References: <54BCDBF1.8020909@allegro-packets.com> <54BE3047.9060909@allegro-packets.com> <20150121134921.GA2592@bricha3-MOBL3> <54C23265.8090403@allegro-packets.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <54C23265.8090403@allegro-packets.com> Organization: Intel Shannon Ltd. User-Agent: Mutt/1.5.23 (2014-03-12) Cc: dev@dpdk.org Subject: Re: [dpdk-dev] Segmentation fault in ixgbe_rxtx_vec.c:444 with 1.8.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jan 2015 11:52:59 -0000 On Fri, Jan 23, 2015 at 12:37:09PM +0100, Martin Weiser wrote: > Hi Bruce, > > I now had the chance to reproduce the issue we are seeing with a DPDK > example app. > I started out with a vanilla DPDK 1.8.0 and only made the following changes: > > diff --git a/examples/l2fwd/main.c b/examples/l2fwd/main.c > index e684234..48e6b7c 100644 > --- a/examples/l2fwd/main.c > +++ b/examples/l2fwd/main.c > @@ -118,8 +118,9 @@ static const struct rte_eth_conf port_conf = { > .header_split = 0, /**< Header Split disabled */ > .hw_ip_checksum = 0, /**< IP checksum offload disabled */ > .hw_vlan_filter = 0, /**< VLAN filtering disabled */ > - .jumbo_frame = 0, /**< Jumbo Frame Support disabled */ > + .jumbo_frame = 1, /**< Jumbo Frame Support disabled */ > .hw_strip_crc = 0, /**< CRC stripped by hardware */ > + .max_rx_pkt_len = 9000, > }, > .txmode = { > .mq_mode = ETH_MQ_TX_NONE, > diff --git a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c > b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c > index b54cb19..dfaccee 100644 > --- a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c > +++ b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c > @@ -402,10 +402,10 @@ reassemble_packets(struct igb_rx_queue *rxq, > struct rte_mbuf **rx_bufs, > struct rte_mbuf *pkts[RTE_IXGBE_VPMD_RX_BURST]; /*finished pkts*/ > struct rte_mbuf *start = rxq->pkt_first_seg; > struct rte_mbuf *end = rxq->pkt_last_seg; > - unsigned pkt_idx = 0, buf_idx = 0; > + unsigned pkt_idx, buf_idx; > > > - while (buf_idx < nb_bufs) { > + for (buf_idx = 0, pkt_idx = 0; buf_idx < nb_bufs; buf_idx++) { > if (end != NULL) { > /* processing a split packet */ > end->next = rx_bufs[buf_idx]; > @@ -448,7 +448,6 @@ reassemble_packets(struct igb_rx_queue *rxq, struct > rte_mbuf **rx_bufs, > rx_bufs[buf_idx]->data_len += rxq->crc_len; > rx_bufs[buf_idx]->pkt_len += rxq->crc_len; > } > - buf_idx++; > } > > /* save the partial packet for next time */ > > > This includes your previously posted fix and makes a small modification > to the l2fwd example app to enable jumbo frames of up to 9000 bytes. > The system is equipped with a two port Intel 82599 card and both ports > are hooked up to a packet generator. The packet generator produces > simple Ethernet/IPv4/UDP packets. > I started the l2fwd app with the following command line: > > $ ./build/l2fwd -c f -n 4 -- -q 8 -p 3 > > Both build variants that I have tested (CONFIG_RTE_IXGBE_INC_VECTOR=y > and CONFIG_RTE_IXGBE_INC_VECTOR=n) now give me the same result: > As long as the packet size is <= 2048 bytes the application behaves > normally and all packets are forwarded as expected. > As soon as the packet size exceeds 2048 bytes the application will only > forward some packets and then stop forwarding altogether. Even small > packets will not be forwarded anymore. > > If you want me to try out anything else just let me know. > > > Best regards, > Martin > I think the txq flags are at fault here. The default txq flags setting for the l2fwd sample application includes the flag ETH_TXQ_FLAGS_NOMULTSEGS which disables support for sending packets with multiple segments i.e. jumbo frames in this case. If you change l2fwd to explicitly pass a txqflags parameter in as part of the port setup (as was the case in previous releases), and set txqflags to 0, does the problem go away? /Bruce > > > On 21.01.15 14:49, Bruce Richardson wrote: > > On Tue, Jan 20, 2015 at 11:39:03AM +0100, Martin Weiser wrote: > >> Hi again, > >> > >> I did some further testing and it seems like this issue is linked to > >> jumbo frames. I think a similar issue has already been reported by > >> Prashant Upadhyaya with the subject 'Packet Rx issue with DPDK1.8'. > >> In our application we use the following rxmode port configuration: > >> > >> .mq_mode = ETH_MQ_RX_RSS, > >> .split_hdr_size = 0, > >> .header_split = 0, > >> .hw_ip_checksum = 1, > >> .hw_vlan_filter = 0, > >> .jumbo_frame = 1, > >> .hw_strip_crc = 1, > >> .max_rx_pkt_len = 9000, > >> > >> and the mbuf size is calculated like the following: > >> > >> (2048 + sizeof(struct rte_mbuf) + RTE_PKTMBUF_HEADROOM) > >> > >> This works fine with DPDK 1.7 and jumbo frames are split into buffer > >> chains and can be forwarded on another port without a problem. > >> With DPDK 1.8 and the default configuration (CONFIG_RTE_IXGBE_INC_VECTOR > >> enabled) the application sometimes crashes like described in my first > >> mail and sometimes packet receiving stops with subsequently arriving > >> packets counted as rx errors. When CONFIG_RTE_IXGBE_INC_VECTOR is > >> disabled the packet processing also comes to a halt as soon as jumbo > >> frames arrive with a the slightly different effect that now > >> rte_eth_tx_burst refuses to send any previously received packets. > >> > >> Is there anything special to consider regarding jumbo frames when moving > >> from DPDK 1.7 to 1.8 that we might have missed? > >> > >> Martin > >> > >> > >> > >> On 19.01.15 11:26, Martin Weiser wrote: > >>> Hi everybody, > >>> > >>> we quite recently updated one of our applications to DPDK 1.8.0 and are > >>> now seeing a segmentation fault in ixgbe_rxtx_vec.c:444 after a few minutes. > >>> I just did some quick debugging and I only have a very limited > >>> understanding of the code in question but it seems that the 'continue' > >>> in line 445 without increasing 'buf_idx' might cause the problem. In one > >>> debugging session when the crash occurred the value of 'buf_idx' was 2 > >>> and the value of 'pkt_idx' was 8965. > >>> Any help with this issue would be greatly appreciated. If you need any > >>> further information just let me know. > >>> > >>> Martin > >>> > >>> > > Hi Martin, Prashant, > > > > I've managed to reproduce the issue here and had a look at it. Could you > > both perhaps try the proposed change below and see if it fixes the problem for > > you and gives you a working system? If so, I'll submit this as a patch fix > > officially - or go back to the drawing board, if not. :-) > > > > diff --git a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c > > index b54cb19..dfaccee 100644 > > --- a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c > > +++ b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c > > @@ -402,10 +402,10 @@ reassemble_packets(struct igb_rx_queue *rxq, struct rte_mbuf **rx_bufs, > > struct rte_mbuf *pkts[RTE_IXGBE_VPMD_RX_BURST]; /*finished pkts*/ > > struct rte_mbuf *start = rxq->pkt_first_seg; > > struct rte_mbuf *end = rxq->pkt_last_seg; > > - unsigned pkt_idx = 0, buf_idx = 0; > > + unsigned pkt_idx, buf_idx; > > > > > > - while (buf_idx < nb_bufs) { > > + for (buf_idx = 0, pkt_idx = 0; buf_idx < nb_bufs; buf_idx++) { > > if (end != NULL) { > > /* processing a split packet */ > > end->next = rx_bufs[buf_idx]; > > @@ -448,7 +448,6 @@ reassemble_packets(struct igb_rx_queue *rxq, struct rte_mbuf **rx_bufs, > > rx_bufs[buf_idx]->data_len += rxq->crc_len; > > rx_bufs[buf_idx]->pkt_len += rxq->crc_len; > > } > > - buf_idx++; > > } > > > > /* save the partial packet for next time */ > > > > > > Regards, > > /Bruce > > >