Re: [dpdk-dev] Segmentation fault in ixgbe_rxtx_vec.c:444 with 1.8.0

[Martin Weiser <martin.weiser@allegro-packets.com>]

Hi Bruce,

I now had the chance to reproduce the issue we are seeing with a DPDK
example app.
I started out with a vanilla DPDK 1.8.0 and only made the following changes:

diff --git a/examples/l2fwd/main.c b/examples/l2fwd/main.c
index e684234..48e6b7c 100644
--- a/examples/l2fwd/main.c
+++ b/examples/l2fwd/main.c
@@ -118,8 +118,9 @@ static const struct rte_eth_conf port_conf = {
                .header_split   = 0, /**< Header Split disabled */
                .hw_ip_checksum = 0, /**< IP checksum offload disabled */
                .hw_vlan_filter = 0, /**< VLAN filtering disabled */
-               .jumbo_frame    = 0, /**< Jumbo Frame Support disabled */
+               .jumbo_frame    = 1, /**< Jumbo Frame Support disabled */
                .hw_strip_crc   = 0, /**< CRC stripped by hardware */
+               .max_rx_pkt_len = 9000,
        },
        .txmode = {
                .mq_mode = ETH_MQ_TX_NONE,
diff --git a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
index b54cb19..dfaccee 100644
--- a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
+++ b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
@@ -402,10 +402,10 @@ reassemble_packets(struct igb_rx_queue *rxq,
struct rte_mbuf **rx_bufs,
        struct rte_mbuf *pkts[RTE_IXGBE_VPMD_RX_BURST]; /*finished pkts*/
        struct rte_mbuf *start = rxq->pkt_first_seg;
        struct rte_mbuf *end =  rxq->pkt_last_seg;
-       unsigned pkt_idx = 0, buf_idx = 0;
+       unsigned pkt_idx, buf_idx;
 
 
-       while (buf_idx < nb_bufs) {
+       for (buf_idx = 0, pkt_idx = 0; buf_idx < nb_bufs; buf_idx++) {
                if (end != NULL) {
                        /* processing a split packet */
                        end->next = rx_bufs[buf_idx];
@@ -448,7 +448,6 @@ reassemble_packets(struct igb_rx_queue *rxq, struct
rte_mbuf **rx_bufs,
                        rx_bufs[buf_idx]->data_len += rxq->crc_len;
                        rx_bufs[buf_idx]->pkt_len += rxq->crc_len;
                }
-               buf_idx++;
        }
 
        /* save the partial packet for next time */


This includes your previously posted fix and makes a small modification
to the l2fwd example app to enable jumbo frames of up to 9000 bytes.
The system is equipped with a two port Intel 82599 card and both ports
are hooked up to a packet generator. The packet generator produces
simple Ethernet/IPv4/UDP packets.
I started the l2fwd app with the following command line:

$ ./build/l2fwd -c f -n 4 -- -q 8 -p 3

Both build variants that I have tested (CONFIG_RTE_IXGBE_INC_VECTOR=y
and CONFIG_RTE_IXGBE_INC_VECTOR=n) now give me the same result:
As long as the packet size is <= 2048 bytes the application behaves
normally and all packets are forwarded as expected.
As soon as the packet size exceeds 2048 bytes the application will only
forward some packets and then stop forwarding altogether. Even small
packets will not be forwarded anymore.

If you want me to try out anything else just let me know.


Best regards,
Martin




On 21.01.15 14:49, Bruce Richardson wrote:
> On Tue, Jan 20, 2015 at 11:39:03AM +0100, Martin Weiser wrote:
>> Hi again,
>>
>> I did some further testing and it seems like this issue is linked to
>> jumbo frames. I think a similar issue has already been reported by
>> Prashant Upadhyaya with the subject 'Packet Rx issue with DPDK1.8'.
>> In our application we use the following rxmode port configuration:
>>
>> .mq_mode    = ETH_MQ_RX_RSS,
>> .split_hdr_size = 0,
>> .header_split   = 0,
>> .hw_ip_checksum = 1,
>> .hw_vlan_filter = 0,
>> .jumbo_frame    = 1,
>> .hw_strip_crc   = 1,
>> .max_rx_pkt_len = 9000,
>>
>> and the mbuf size is calculated like the following:
>>
>> (2048 + sizeof(struct rte_mbuf) + RTE_PKTMBUF_HEADROOM)
>>
>> This works fine with DPDK 1.7 and jumbo frames are split into buffer
>> chains and can be forwarded on another port without a problem.
>> With DPDK 1.8 and the default configuration (CONFIG_RTE_IXGBE_INC_VECTOR
>> enabled) the application sometimes crashes like described in my first
>> mail and sometimes packet receiving stops with subsequently arriving
>> packets counted as rx errors. When CONFIG_RTE_IXGBE_INC_VECTOR is
>> disabled the packet processing also comes to a halt as soon as jumbo
>> frames arrive with a the slightly different effect that now
>> rte_eth_tx_burst refuses to send any previously received packets.
>>
>> Is there anything special to consider regarding jumbo frames when moving
>> from DPDK 1.7 to 1.8 that we might have missed?
>>
>> Martin
>>
>>
>>
>> On 19.01.15 11:26, Martin Weiser wrote:
>>> Hi everybody,
>>>
>>> we quite recently updated one of our applications to DPDK 1.8.0 and are
>>> now seeing a segmentation fault in ixgbe_rxtx_vec.c:444 after a few minutes.
>>> I just did some quick debugging and I only have a very limited
>>> understanding of the code in question but it seems that the 'continue'
>>> in line 445 without increasing 'buf_idx' might cause the problem. In one
>>> debugging session when the crash occurred the value of 'buf_idx' was 2
>>> and the value of 'pkt_idx' was 8965.
>>> Any help with this issue would be greatly appreciated. If you need any
>>> further information just let me know.
>>>
>>> Martin
>>>
>>>
> Hi Martin, Prashant,
>
> I've managed to reproduce the issue here and had a look at it. Could you
> both perhaps try the proposed change below and see if it fixes the problem for
> you and gives you a working system? If so, I'll submit this as a patch fix 
> officially - or go back to the drawing board, if not. :-)
>
> diff --git a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
> index b54cb19..dfaccee 100644
> --- a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
> +++ b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
> @@ -402,10 +402,10 @@ reassemble_packets(struct igb_rx_queue *rxq, struct rte_mbuf **rx_bufs,
>         struct rte_mbuf *pkts[RTE_IXGBE_VPMD_RX_BURST]; /*finished pkts*/
>         struct rte_mbuf *start = rxq->pkt_first_seg;
>         struct rte_mbuf *end =  rxq->pkt_last_seg;
> -       unsigned pkt_idx = 0, buf_idx = 0;
> +       unsigned pkt_idx, buf_idx;
>
>
> -       while (buf_idx < nb_bufs) {
> +       for (buf_idx = 0, pkt_idx = 0; buf_idx < nb_bufs; buf_idx++) {
>                 if (end != NULL) {
>                         /* processing a split packet */
>                         end->next = rx_bufs[buf_idx];
> @@ -448,7 +448,6 @@ reassemble_packets(struct igb_rx_queue *rxq, struct rte_mbuf **rx_bufs,
>                         rx_bufs[buf_idx]->data_len += rxq->crc_len;
>                         rx_bufs[buf_idx]->pkt_len += rxq->crc_len;
>                 }
> -               buf_idx++;
>         }
>
>         /* save the partial packet for next time */
>
>
> Regards,
> /Bruce
>