Re: [dpdk-dev] Segmentation fault in ixgbe_rxtx_vec.c:444 with 1.8.0

[Martin Weiser <martin.weiser@allegro-packets.com>]

Hi Bruce,

yes, you are absolutely right. That resolves the problem.
I was really happy to see that DPDK 1.8 includes proper default
configurations for each driver and I made use of this. But unfortunately
I was not aware that the default configuration did include the
ETH_TXQ_FLAGS_NOMULTSEGS flag for ixgbe and i40e.
I now use rte_eth_dev_info_get to get the default config for the port
and then modify the txq_flags to not not include ETH_TXQ_FLAGS_NOMULTSEGS.

With your fix this now works for CONFIG_RTE_IXGBE_INC_VECTOR=y, too.

Sorry for missing this and thanks for the quick help.

Best regards,
Martin


On 23.01.15 12:52, Bruce Richardson wrote:
> On Fri, Jan 23, 2015 at 12:37:09PM +0100, Martin Weiser wrote:
>> Hi Bruce,
>>
>> I now had the chance to reproduce the issue we are seeing with a DPDK
>> example app.
>> I started out with a vanilla DPDK 1.8.0 and only made the following changes:
>>
>> diff --git a/examples/l2fwd/main.c b/examples/l2fwd/main.c
>> index e684234..48e6b7c 100644
>> --- a/examples/l2fwd/main.c
>> +++ b/examples/l2fwd/main.c
>> @@ -118,8 +118,9 @@ static const struct rte_eth_conf port_conf = {
>>                 .header_split   = 0, /**< Header Split disabled */
>>                 .hw_ip_checksum = 0, /**< IP checksum offload disabled */
>>                 .hw_vlan_filter = 0, /**< VLAN filtering disabled */
>> -               .jumbo_frame    = 0, /**< Jumbo Frame Support disabled */
>> +               .jumbo_frame    = 1, /**< Jumbo Frame Support disabled */
>>                 .hw_strip_crc   = 0, /**< CRC stripped by hardware */
>> +               .max_rx_pkt_len = 9000,
>>         },
>>         .txmode = {
>>                 .mq_mode = ETH_MQ_TX_NONE,
>> diff --git a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
>> b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
>> index b54cb19..dfaccee 100644
>> --- a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
>> +++ b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
>> @@ -402,10 +402,10 @@ reassemble_packets(struct igb_rx_queue *rxq,
>> struct rte_mbuf **rx_bufs,
>>         struct rte_mbuf *pkts[RTE_IXGBE_VPMD_RX_BURST]; /*finished pkts*/
>>         struct rte_mbuf *start = rxq->pkt_first_seg;
>>         struct rte_mbuf *end =  rxq->pkt_last_seg;
>> -       unsigned pkt_idx = 0, buf_idx = 0;
>> +       unsigned pkt_idx, buf_idx;
>>  
>>  
>> -       while (buf_idx < nb_bufs) {
>> +       for (buf_idx = 0, pkt_idx = 0; buf_idx < nb_bufs; buf_idx++) {
>>                 if (end != NULL) {
>>                         /* processing a split packet */
>>                         end->next = rx_bufs[buf_idx];
>> @@ -448,7 +448,6 @@ reassemble_packets(struct igb_rx_queue *rxq, struct
>> rte_mbuf **rx_bufs,
>>                         rx_bufs[buf_idx]->data_len += rxq->crc_len;
>>                         rx_bufs[buf_idx]->pkt_len += rxq->crc_len;
>>                 }
>> -               buf_idx++;
>>         }
>>  
>>         /* save the partial packet for next time */
>>
>>
>> This includes your previously posted fix and makes a small modification
>> to the l2fwd example app to enable jumbo frames of up to 9000 bytes.
>> The system is equipped with a two port Intel 82599 card and both ports
>> are hooked up to a packet generator. The packet generator produces
>> simple Ethernet/IPv4/UDP packets.
>> I started the l2fwd app with the following command line:
>>
>> $ ./build/l2fwd -c f -n 4 -- -q 8 -p 3
>>
>> Both build variants that I have tested (CONFIG_RTE_IXGBE_INC_VECTOR=y
>> and CONFIG_RTE_IXGBE_INC_VECTOR=n) now give me the same result:
>> As long as the packet size is <= 2048 bytes the application behaves
>> normally and all packets are forwarded as expected.
>> As soon as the packet size exceeds 2048 bytes the application will only
>> forward some packets and then stop forwarding altogether. Even small
>> packets will not be forwarded anymore.
>>
>> If you want me to try out anything else just let me know.
>>
>>
>> Best regards,
>> Martin
>>
> I think the txq flags are at fault here. The default txq flags setting for
> the l2fwd sample application includes the flag ETH_TXQ_FLAGS_NOMULTSEGS which
> disables support for sending packets with multiple segments i.e. jumbo frames
> in this case. If you change l2fwd to explicitly pass a txqflags parameter in
> as part of the port setup (as was the case in previous releases), and set txqflags
> to 0, does the problem go away?
>
> /Bruce
>
>>
>> On 21.01.15 14:49, Bruce Richardson wrote:
>>> On Tue, Jan 20, 2015 at 11:39:03AM +0100, Martin Weiser wrote:
>>>> Hi again,
>>>>
>>>> I did some further testing and it seems like this issue is linked to
>>>> jumbo frames. I think a similar issue has already been reported by
>>>> Prashant Upadhyaya with the subject 'Packet Rx issue with DPDK1.8'.
>>>> In our application we use the following rxmode port configuration:
>>>>
>>>> .mq_mode    = ETH_MQ_RX_RSS,
>>>> .split_hdr_size = 0,
>>>> .header_split   = 0,
>>>> .hw_ip_checksum = 1,
>>>> .hw_vlan_filter = 0,
>>>> .jumbo_frame    = 1,
>>>> .hw_strip_crc   = 1,
>>>> .max_rx_pkt_len = 9000,
>>>>
>>>> and the mbuf size is calculated like the following:
>>>>
>>>> (2048 + sizeof(struct rte_mbuf) + RTE_PKTMBUF_HEADROOM)
>>>>
>>>> This works fine with DPDK 1.7 and jumbo frames are split into buffer
>>>> chains and can be forwarded on another port without a problem.
>>>> With DPDK 1.8 and the default configuration (CONFIG_RTE_IXGBE_INC_VECTOR
>>>> enabled) the application sometimes crashes like described in my first
>>>> mail and sometimes packet receiving stops with subsequently arriving
>>>> packets counted as rx errors. When CONFIG_RTE_IXGBE_INC_VECTOR is
>>>> disabled the packet processing also comes to a halt as soon as jumbo
>>>> frames arrive with a the slightly different effect that now
>>>> rte_eth_tx_burst refuses to send any previously received packets.
>>>>
>>>> Is there anything special to consider regarding jumbo frames when moving
>>>> from DPDK 1.7 to 1.8 that we might have missed?
>>>>
>>>> Martin
>>>>
>>>>
>>>>
>>>> On 19.01.15 11:26, Martin Weiser wrote:
>>>>> Hi everybody,
>>>>>
>>>>> we quite recently updated one of our applications to DPDK 1.8.0 and are
>>>>> now seeing a segmentation fault in ixgbe_rxtx_vec.c:444 after a few minutes.
>>>>> I just did some quick debugging and I only have a very limited
>>>>> understanding of the code in question but it seems that the 'continue'
>>>>> in line 445 without increasing 'buf_idx' might cause the problem. In one
>>>>> debugging session when the crash occurred the value of 'buf_idx' was 2
>>>>> and the value of 'pkt_idx' was 8965.
>>>>> Any help with this issue would be greatly appreciated. If you need any
>>>>> further information just let me know.
>>>>>
>>>>> Martin
>>>>>
>>>>>
>>> Hi Martin, Prashant,
>>>
>>> I've managed to reproduce the issue here and had a look at it. Could you
>>> both perhaps try the proposed change below and see if it fixes the problem for
>>> you and gives you a working system? If so, I'll submit this as a patch fix 
>>> officially - or go back to the drawing board, if not. :-)
>>>
>>> diff --git a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
>>> index b54cb19..dfaccee 100644
>>> --- a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
>>> +++ b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
>>> @@ -402,10 +402,10 @@ reassemble_packets(struct igb_rx_queue *rxq, struct rte_mbuf **rx_bufs,
>>>         struct rte_mbuf *pkts[RTE_IXGBE_VPMD_RX_BURST]; /*finished pkts*/
>>>         struct rte_mbuf *start = rxq->pkt_first_seg;
>>>         struct rte_mbuf *end =  rxq->pkt_last_seg;
>>> -       unsigned pkt_idx = 0, buf_idx = 0;
>>> +       unsigned pkt_idx, buf_idx;
>>>
>>>
>>> -       while (buf_idx < nb_bufs) {
>>> +       for (buf_idx = 0, pkt_idx = 0; buf_idx < nb_bufs; buf_idx++) {
>>>                 if (end != NULL) {
>>>                         /* processing a split packet */
>>>                         end->next = rx_bufs[buf_idx];
>>> @@ -448,7 +448,6 @@ reassemble_packets(struct igb_rx_queue *rxq, struct rte_mbuf **rx_bufs,
>>>                         rx_bufs[buf_idx]->data_len += rxq->crc_len;
>>>                         rx_bufs[buf_idx]->pkt_len += rxq->crc_len;
>>>                 }
>>> -               buf_idx++;
>>>         }
>>>
>>>         /* save the partial packet for next time */
>>>
>>>
>>> Regards,
>>> /Bruce
>>>