Re: [dpdk-dev] Intel 82599ES send packets failed on dpdk-17.05.1 after traffic testing

[zimeiw <zimeiw@163.com>]

hi,


I have tested below dpdk version, after dpdk-17.05 version, ixgbe nic would send fail. the logs is same as below.


dpdk-17.02    ---- ok
dpdk-17.02.1  ---- ok 


dpdk-17.05  ---- nok 

dpdk-17.05.1  ---- nok 

dpdk-17.08   ---- nok 


PMD: ixgbe_xmit_pkts(): port_id=0 queue_id=0 tx_tail=63 nb_tx=11
PMD: ixgbe_xmit_cleanup(): TX descriptor   95 is not done(port=0 queue=0)
PMD: ixgbe_xmit_pkts(): port_id=0 queue_id=0 pktlen=205 tx_first=63 tx_last=63
PMD: ixgbe_xmit_pkts(): Not enough free TX descriptors nb_used=   1 nb_free=   0 (port=0 queue=0)
PMD: ixgbe_xmit_cleanup(): TX descriptor   95 is not done(port=0 queue=0)
tx burst failed, ret 0 , n 16,  Success 
PMD: ixgbe_xmit_cleanup(): TX descriptor   95 is not done(port=0 queue=0)
PMD: ixgbe_xmit_pkts(): port_id=0 queue_id=0 pktlen=205 tx_first=63 tx_last=63
PMD: ixgbe_xmit_pkts(): Not enough free TX descriptors nb_used=   1 nb_free=   0 (port=0 queue=0)
PMD: ixgbe_xmit_cleanup(): TX descriptor   95 is not done(port=0 queue=0)
tx burst failed, ret 0 , n 16,  Success 
PMD: ixgbe_xmit_cleanup(): TX descriptor   95 is not done(port=0 queue=0)
PMD: ixgbe_xmit_pkts(): port_id=0 queue_id=0 pktlen=54 tx_first=63 tx_last=63
PMD: ixgbe_xmit_pkts(): Not enough free TX descriptors nb_used=   1 nb_free=   0 (port=0 queue=0)
PMD: ixgbe_xmit_cleanup(): TX descriptor   95 is not done(port=0 queue=0)
tx burst failed, ret 0 , n 5, Success 
PMD: ixgbe_xmit_cleanup(): TX descriptor   95 is not done(port=0 queue=0)



--

Best Regards,
zimeiw



At 2017-08-23 17:04:44, "Ferruh Yigit" <ferruh.yigit@intel.com> wrote:
>On 8/23/2017 6:23 AM, zimeiw wrote:
>> hi,
>> 
>> Any comments?
>> 
>> is it possible NIC haredware issue? or NIC firmware version isn't match
>> dpdk driver?
>
>AFAIK for there isn't this kind of dependency for ixgbe. ixgbe driver
>maintainer cc'ed for more information.
>
>> 
>> --
>> Best Regards,
>> zimeiw
>> 
>> At 2017-08-23 08:52:38, "zimeiw" <zimeiw@163.com> wrote:
>> 
>>     hi,
>> 
>>     "disable tso" logs were printed by my app. 
>> 
>>     dpdk-17.05.1 is download from dpdk website, it is stable release.
>> 
>> 
>>     --
>>     Best Regards,
>>     zimeiw
>> 
>> 
>>     At 2017-08-22 22:23:54, "Ferruh Yigit" <ferruh.yigit@intel.com <mailto:ferruh.yigit@intel.com>> wrote:
>>     >On 8/22/2017 2:51 PM, zimeiw wrote:
>>     >> hi,
>>     >> =====
>>     >> check the dpdk default config,  CONFIG_RTE_ETHDEV_TX_PREPARE_NOOP isn't
>>     >> enable.
>>     >> #
>>     >> # Turn off Tx preparation stage
>>     >> #
>>     >> # Warning: rte_ethdev_tx_prepare() can be safely disabled only if using a
>>     >> # driver which do not implement any Tx preparation.
>>     >> #
>>     >> CONFIG_RTE_ETHDEV_TX_PREPARE_NOOP=n
>>     >> 
>>     >> ====
>>     >> I enable IXGBE debug for dpdk-17.05.1 and dpdk-17.02. the logs are in
>>     >> the attachment.
>>     >> app with dpdk-17.02 works ok.
>>     >> app with dpdk-17.05.1 has below issue. it seems TX descriptor   95 can't
>>     >> be handled by NIC?
>>     >
>>     >This is interesting, no idea what is going on, trying dpdk-17.05 can be
>>     >an option, to be sure if this issue is in the original release or in the
>>     >stable tree?
>>     >
>>     >btw, in 17.05.01, there are "disable tso" logs, do you know where they
>>     >are coming from?
>>     >
>>     >> 
>>     >> 
>>     >> PMD: ixgbe_xmit_pkts(): port_id=0 queue_id=0 tx_tail=63 nb_tx=11
>>     >> PMD: ixgbe_xmit_cleanup(): TX descriptor   95 is not done(port=0 queue=0)
>>     >
>>     >Descriptor Done bit is not set, HW says descriptor is still in use, so
>>     >cleaning descriptor fails.
>>     >
>>     >> PMD: ixgbe_xmit_pkts(): port_id=0 queue_id=0 pktlen=205 tx_first=63
>>     >> tx_last=63
>>     >> PMD: ixgbe_xmit_pkts(): Not enough free TX descriptors nb_used=   1
>>     >> nb_free=   0 (port=0 queue=0)
>>     >
>>     >And after a while there is no free descriptor left and Tx fails.
>>     >
>>     >> PMD: ixgbe_xmit_cleanup(): TX descriptor   95 is not done(port=0 queue=0)
>>     >> tx burst failed, ret 0 , n 16, nb_prep 0 , Success 
>>     >> PMD: ixgbe_xmit_cleanup(): TX descriptor   95 is not done(port=0 queue=0)
>>     >> PMD: ixgbe_xmit_pkts(): port_id=0 queue_id=0 pktlen=205 tx_first=63
>>     >> tx_last=63
>>     >> PMD: ixgbe_xmit_pkts(): Not enough free TX descriptors nb_used=   1
>>     >> nb_free=   0 (port=0 queue=0)
>>     >> PMD: ixgbe_xmit_cleanup(): TX descriptor   95 is not done(port=0 queue=0)
>>     >> tx burst failed, ret 0 , n 16, nb_prep 0 , Success 
>>     >> PMD: ixgbe_xmit_cleanup(): TX descriptor   95 is not done(port=0 queue=0)
>>     >> PMD: ixgbe_xmit_pkts(): port_id=0 queue_id=0 pktlen=54 tx_first=63
>>     >> tx_last=63
>>     >> PMD: ixgbe_xmit_pkts(): Not enough free TX descriptors nb_used=   1
>>     >> nb_free=   0 (port=0 queue=0)
>>     >> PMD: ixgbe_xmit_cleanup(): TX descriptor   95 is not done(port=0 queue=0)
>>     >> tx burst failed, ret 0 , n 5, nb_prep 0 , Success 
>>     >> 
>>     >> 
>>     >> 
>>     >> --
>>     >> Best Regards,
>>     >> zimeiw
>>     >> 
>>     >> 
>>     >> At 2017-08-22 20:13:57, "Ferruh Yigit" <ferruh.yigit@intel.com <mailto:ferruh.yigit@intel.com>> wrote:
>>     >>>On 8/22/2017 12:00 PM, zimeiw wrote:
>>     >>>> hi,
>>     >>>> 
>>     >>>> i set .txq_flags = 0.
>>     >>>> and change the "send burst" code as below.
>>     >>>> and printf some logs.
>>     >>>> according to the logs, rte_eth_tx_burst() return 0, no error happen in
>>     >>>> rte_eth_tx_burst().
>>     >>>> 
>>     >>>> ====================
>>     >>>>    struct rte_mbuf **m_table;
>>     >>>>     int ret;
>>     >>>>     uint16_t queueid;
>>     >>>>     uint16_t nb_prep;
>>     >>>> 
>>     >>>>     queueid = qconf->tx_queue_id[port];
>>     >>>>     m_table = (struct rte_mbuf **)qconf->tx_mbufs[port].m_table;
>>     >>>> 
>>     >>>>     nb_prep = rte_eth_tx_prepare(port, queueid, m_table, n);
>>     >>>>     if (nb_prep != n)
>>     >>>>         printf("Preparing packet burst %d  to transmit failed: %s\n",
>>     >>>> nb_prep, rte_strerror(rte_errno));
>>     >>>
>>     >>>m->ol_flags should request offloading (PKT_TX_...), otherwise prepare
>>     >>>does not do the work.
>>     >>>
>>     >>>but m->ol_flags is not changed, if this is working with 17.02, issue can
>>     >>>be something else.
>>     >>>
>>     >>>You may try debugging ixgbe PMD to figure out at which point it gives an
>>     >>>error.
>>     >>>
>>     >>>> 
>>     >>>>     ret = rte_eth_tx_burst(port, queueid, m_table, nb_prep);
>>     >>>>     if (unlikely(ret < n))
>>     >>>>     {
>>     >>>>         printf("tx burst failed, ret %d , n %d, nb_prep %d , %s \n",
>>     >>>> ret, n, nb_prep, rte_strerror(rte_errno) );
>>     >>>>         do
>>     >>>>         {
>>     >>>>             rte_pktmbuf_free(m_table[ret]);
>>     >>>>         } while (++ret < n);
>>     >>>>     }
>>     >>>> =======================
>>     >>>> 
>>     >>>> =====log======
>>     >>>> Checking link status done
>>     >>>> Port 0 Link Up - speed 10000 Mbps - full-duplex
>>     >>>> USER8: main loop on lcore 1
>>     >>>> USER8:  -- lcoreid=1 portid=0 rxqueueid=0
>>     >>>> nb ports 1 hz: 1698027100
>>     >>>> tx burst failed, ret 2 , n 16, nb_prep 16 , Success
>>     >>>> tx burst failed, ret 0 , n 16, nb_prep 16 , Success
>>     >>>> tx burst failed, ret 0 , n 16, nb_prep 16 , Success
>>     >>>> tx burst failed, ret 0 , n 16, nb_prep 16 , Success
>>     >>>> tx burst failed, ret 0 , n 16, nb_prep 16 , Success
>>     >>>> tx burst failed, ret 0 , n 16, nb_prep 16 , Success
>>     >>>> tx burst failed, ret 0 , n 16, nb_prep 16 , Success
>>     >>>> tx burst failed, ret 0 , n 16, nb_prep 16 , Success
>>     >>>> tx burst failed, ret 0 , n 16, nb_prep 16 , Success
>>     >>>> tx burst failed, ret 0 , n 16, nb_prep 16 , Success
>>     >>>> 
>>     >>>> ==========NIC firware version =======
>>     >>>> # ethtool -i enp2s0f1
>>     >>>> driver: ixgbe
>>     >>>> version: 4.2.1-k
>>     >>>> firmware-version: 0x61bd0001
>>     >>>> expansion-rom-version:
>>     >>>> bus-info: 0000:02:00.1
>>     >>>> supports-statistics: yes
>>     >>>> supports-test: yes
>>     >>>> supports-eeprom-access: yes
>>     >>>> supports-register-dump: yes
>>     >>>> supports-priv-flags: no
>>     >>>> 
>>     >>>> 
>>     >>>> --
>>     >>>> Best Regards,
>>     >>>> zimeiw
>>     >>>> 
>>     >>>> 
>>     >>>> At 2017-08-22 17:31:56, "Ferruh Yigit" <ferruh.yigit@intel.com <mailto:ferruh.yigit@intel.com>> wrote:
>>     >>>>>On 8/22/2017 9:30 AM, zimeiw wrote:
>>     >>>>>> hi,
>>     >>>>>> 
>>     >>>>>> no use rte_eth_tx_prepare()?
>>     >>>>>> 
>>     >>>>>> I set txq_flags value as below,  may it cause this issue?
>>     >>>>>> 
>>     >>>>>>   .txq_flags = ~ETH_TXQ_FLAGS_NOXSUMS 
>>     >>>>>
>>     >>>>>So you are enabling all csum offloading, for Tx path this requires
>>     >>>>>packets to be prepared before sending to the NIC.
>>     >>>>>
>>     >>>>>Can you please test with rte_eth_tx_prepare()? testpmd has the sample usage.
>>     >>>>>
>>     >>>>>Thanks,
>>     >>>>>ferruh
>>     >>>>>
>>     >>>>>> 
>>     >>>>>> --
>>     >>>>>> Best Regards,
>>     >>>>>> zimeiw
>>     >>>>>> 
>>     >>>>>> 
>>     >>>>>> At 2017-08-22 15:49:56, "Ferruh Yigit" <ferruh.yigit@intel.com <mailto:ferruh.yigit@intel.com>> wrote:
>>     >>>>>>>On 8/22/2017 5:45 AM, zimeiw wrote:
>>     >>>>>>>> hi,
>>     >>>>>>>> My test env is dell T430 server, NIC is intel 82599ES.
>>     >>>>>>>> I integrate my app with dpdk-17.05.1 version. after do some tcp traffic testing, app can receive packets from the 82599 NIC, but 82599 NIC send packets failed.
>>     >>>>>>>> But my app with dpdk-17.02 works well. 
>>     >>>>>>>> 
>>     >>>>>>>> 
>>     >>>>>>>> If any change with ixgbe driver? or need some additional configuration for the NIC?
>>     >>>>>>>
>>     >>>>>>>tx_pkt_prepare added but it has been added on 17.02 which seems working
>>     >>>>>>>for your case.
>>     >>>>>>>
>>     >>>>>>>Just to double check, are you using rte_eth_tx_prepare()?
>>     >>>>>>>
>>     >>>>>>>> 
>>     >>>>>>>> 
>>     >>>>>>>> 
>>     >>>>>>>> Thanks.
>>     >>>>>>>> 
>>     >>>>>>>> 
>>     >>>>>>>> --
>>     >>>>>>>> 
>>     >>>>>>>> Best Regards,
>>     >>>>>>>> zimeiw
>>     >>>>>>>> 
>>     >>>>>>>
>>     >>>>>> 
>>     >>>>>> 
>>     >>>>>> 
>>     >>>>>>  
>>     >>>>>> 
>>     >>>>>
>>     >>>> 
>>     >>>> 
>>     >>>> 
>>     >>>>  
>>     >>>> 
>>     >>>
>>     >> 
>>     >> 
>>     >> 
>>     >>  
>>     >> 
>>     >
>> 
>> 
>> 
>>      
>> 
>> 
>> 
>>  
>> 
>